== replay-dual test 26: dbench and tar with mds failover ========================================================== 17:28:38 (1743456518) Starting client oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre Started clients oleg345-client.virtnet: 192.168.203.145@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 4099 Started dbench loop with 4100 striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Mar 31 17:28:39 EDT 2025 waiting for dbench pid 4141 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 223 8.62 MB/sec warmup 1 sec latency 27.002 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2304 1285384 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1285640 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26120 3555952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7161436 1% /mnt/lustre 1 464 8.08 MB/sec warmup 2 sec latency 23.605 ms 1 674 7.06 MB/sec warmup 3 sec latency 400.802 ms 1 926 5.44 MB/sec warmup 4 sec latency 17.926 ms test_26 fail mds1 1 times 1 1242 4.62 MB/sec warmup 5 sec latency 15.837 ms Failing mds1 on oleg345-server Stopping /mnt/lustre-mds1 (opts:) on oleg345-server 1 1526 4.35 MB/sec warmup 6 sec latency 260.492 ms 1 1526 3.73 MB/sec warmup 7 sec latency 1260.687 ms 1 1526 3.26 MB/sec warmup 8 sec latency 2260.897 ms 1 1526 2.90 MB/sec warmup 9 sec latency 3261.115 ms 1 1526 2.61 MB/sec warmup 10 sec latency 4261.291 ms 1 1526 2.37 MB/sec warmup 11 sec latency 5261.406 ms 17:28:51 (1743456531) shut down facet: mds1 facet_host: oleg345-server facet_failover_host: oleg345-server 1 1526 2.17 MB/sec warmup 12 sec latency 6261.551 ms 1 1526 2.01 MB/sec warmup 13 sec latency 7261.716 ms 1 1526 1.86 MB/sec warmup 14 sec latency 8261.937 ms 1 1526 1.74 MB/sec warmup 15 sec latency 9262.128 ms 1 1526 1.63 MB/sec warmup 16 sec latency 10262.315 ms 1 1526 1.53 MB/sec warmup 17 sec latency 11262.512 ms 1 1526 1.45 MB/sec warmup 18 sec latency 12262.694 ms 1 1526 1.37 MB/sec warmup 19 sec latency 13262.880 ms 1 1526 0.00 MB/sec execute 1 sec latency 15263.199 ms Failover mds1 to oleg345-server mount facets: mds1 1 1526 0.00 MB/sec execute 2 sec latency 16263.397 ms 1 1526 0.00 MB/sec execute 3 sec latency 17266.649 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1526 0.00 MB/sec execute 4 sec latency 18268.270 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 1 1526 0.00 MB/sec execute 5 sec latency 19268.452 ms Started lustre-MDT0000 17:29:05 (1743456545) targets are mounted 17:29:05 (1743456545) facet_failover done 1 1526 0.00 MB/sec execute 6 sec latency 20268.624 ms 1 1526 0.00 MB/sec execute 7 sec latency 21268.857 ms 1 1526 0.00 MB/sec execute 8 sec latency 22269.071 ms 1 1526 0.00 MB/sec execute 9 sec latency 23269.186 ms 1 1665 0.01 MB/sec execute 10 sec latency 23841.085 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2000 0.04 MB/sec execute 11 sec latency 19.498 ms 1 2273 0.09 MB/sec execute 12 sec latency 29.280 ms 1 2571 0.22 MB/sec execute 13 sec latency 45.860 ms 1 3170 0.57 MB/sec execute 14 sec latency 22.021 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2896 1284792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2656 1285032 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 32920 3563360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6448 3591476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39368 7154836 1% /mnt/lustre 1 3629 0.83 MB/sec execute 15 sec latency 18.668 ms 1 3881 0.85 MB/sec execute 16 sec latency 16.203 ms 1 4117 0.82 MB/sec execute 17 sec latency 16.305 ms 1 4421 0.81 MB/sec execute 18 sec latency 40.868 ms test_26 fail mds2 2 times Failing mds2 on oleg345-server Stopping /mnt/lustre-mds2 (opts:) on oleg345-server 1 4697 0.84 MB/sec execute 19 sec latency 58.848 ms 17:29:19 (1743456559) shut down facet: mds2 facet_host: oleg345-server facet_failover_host: oleg345-server 1 4697 0.80 MB/sec execute 20 sec latency 1059.067 ms 1 4697 0.76 MB/sec execute 21 sec latency 2059.252 ms 1 4697 0.73 MB/sec execute 22 sec latency 3059.420 ms 1 4697 0.70 MB/sec execute 23 sec latency 4059.616 ms 1 4697 0.67 MB/sec execute 24 sec latency 5059.817 ms 1 4697 0.64 MB/sec execute 25 sec latency 6059.999 ms 1 4697 0.62 MB/sec execute 26 sec latency 7060.238 ms 1 4697 0.59 MB/sec execute 27 sec latency 8060.426 ms 1 4697 0.57 MB/sec execute 28 sec latency 9060.605 ms 1 4697 0.55 MB/sec execute 29 sec latency 10060.767 ms Failover mds2 to oleg345-server mount facets: mds2 1 4697 0.53 MB/sec execute 30 sec latency 11060.893 ms 1 4697 0.52 MB/sec execute 31 sec latency 12061.068 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4697 0.50 MB/sec execute 32 sec latency 13061.251 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all 1 4697 0.49 MB/sec execute 33 sec latency 14061.458 ms pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 17:29:33 (1743456573) targets are mounted 17:29:33 (1743456573) facet_failover done 1 4697 0.47 MB/sec execute 34 sec latency 15061.650 ms 1 4697 0.46 MB/sec execute 35 sec latency 16061.856 ms 1 4697 0.44 MB/sec execute 36 sec latency 17062.065 ms 1 4697 0.43 MB/sec execute 37 sec latency 18062.244 ms 1 4697 0.42 MB/sec execute 38 sec latency 19062.366 ms 1 4697 0.41 MB/sec execute 39 sec latency 20062.484 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5025 0.48 MB/sec execute 40 sec latency 20110.063 ms 1 5284 0.47 MB/sec execute 41 sec latency 31.139 ms 1 5577 0.47 MB/sec execute 42 sec latency 15.138 ms 1 5936 0.49 MB/sec execute 43 sec latency 13.894 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2728 1284960 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3176 1284512 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38800 3557996 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12024 3586800 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50824 7144796 1% /mnt/lustre 1 6338 0.57 MB/sec execute 44 sec latency 36.681 ms 1 6922 0.63 MB/sec execute 45 sec latency 16.429 ms 1 7336 0.71 MB/sec execute 46 sec latency 15.422 ms 1 7559 0.70 MB/sec execute 47 sec latency 26.068 ms test_26 fail mds1 3 times Failing mds1 on oleg345-server 1 7824 0.69 MB/sec execute 48 sec latency 44.358 ms Stopping /mnt/lustre-mds1 (opts:) on oleg345-server 1 8078 0.69 MB/sec execute 49 sec latency 46.225 ms 17:29:49 (1743456589) shut down facet: mds1 facet_host: oleg345-server facet_failover_host: oleg345-server 1 8106 0.68 MB/sec execute 50 sec latency 853.732 ms 1 8106 0.67 MB/sec execute 51 sec latency 1853.876 ms 1 8106 0.65 MB/sec execute 52 sec latency 2854.008 ms 1 8106 0.64 MB/sec execute 53 sec latency 3854.121 ms 1 8106 0.63 MB/sec execute 54 sec latency 4854.246 ms 1 8106 0.62 MB/sec execute 55 sec latency 5854.387 ms 1 8106 0.61 MB/sec execute 56 sec latency 6854.544 ms 1 8106 0.60 MB/sec execute 57 sec latency 7854.671 ms 1 8106 0.59 MB/sec execute 58 sec latency 8854.799 ms 1 8106 0.58 MB/sec execute 59 sec latency 9854.944 ms Failover mds1 to oleg345-server mount facets: mds1 1 8106 0.57 MB/sec execute 60 sec latency 10855.145 ms 1 8106 0.56 MB/sec execute 61 sec latency 11855.333 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 8106 0.55 MB/sec execute 62 sec latency 12855.470 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 17:30:02 (1743456602) targets are mounted 17:30:02 (1743456602) facet_failover done 1 8106 0.54 MB/sec execute 63 sec latency 13855.675 ms 1 8106 0.53 MB/sec execute 64 sec latency 14855.827 ms 1 8106 0.52 MB/sec execute 65 sec latency 15855.978 ms 1 8106 0.52 MB/sec execute 66 sec latency 16856.178 ms 1 8106 0.51 MB/sec execute 67 sec latency 17856.303 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8384 0.52 MB/sec execute 68 sec latency 17952.583 ms 1 8718 0.55 MB/sec execute 69 sec latency 16.391 ms 1 8979 0.55 MB/sec execute 70 sec latency 18.109 ms 1 9306 0.55 MB/sec execute 71 sec latency 14.992 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3052 1284636 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4608 1283080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 51848 3539748 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15184 3575712 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 67032 7115460 1% /mnt/lustre 1 9603 0.57 MB/sec execute 72 sec latency 17.301 ms 1 10108 0.63 MB/sec execute 73 sec latency 16.114 ms 1 10637 0.68 MB/sec execute 74 sec latency 20.169 ms 1 10917 0.69 MB/sec execute 75 sec latency 17.302 ms test_26 fail mds2 4 times Failing mds2 on oleg345-server 1 11136 0.69 MB/sec execute 76 sec latency 21.372 ms Stopping /mnt/lustre-mds2 (opts:) on oleg345-server 1 11310 0.68 MB/sec execute 77 sec latency 401.602 ms 17:30:17 (1743456617) shut down facet: mds2 facet_host: oleg345-server facet_failover_host: oleg345-server 1 11310 0.67 MB/sec execute 78 sec latency 1401.756 ms 1 11310 0.66 MB/sec execute 79 sec latency 2401.933 ms 1 11310 0.65 MB/sec execute 80 sec latency 3402.122 ms 1 11310 0.65 MB/sec execute 81 sec latency 4402.296 ms 1 11310 0.64 MB/sec execute 82 sec latency 5402.502 ms 1 11310 0.63 MB/sec execute 83 sec latency 6402.703 ms 1 11310 0.62 MB/sec execute 84 sec latency 7402.877 ms 1 11310 0.62 MB/sec execute 85 sec latency 8403.040 ms 1 11310 0.61 MB/sec execute 86 sec latency 9403.210 ms 1 11310 0.60 MB/sec execute 87 sec latency 10403.325 ms Failover mds2 to oleg345-server mount facets: mds2 1 11310 0.59 MB/sec execute 88 sec latency 11403.442 ms 1 11310 0.59 MB/sec execute 89 sec latency 12403.573 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 11310 0.58 MB/sec execute 90 sec latency 13403.780 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all 1 11310 0.57 MB/sec execute 91 sec latency 14403.993 ms pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 17:30:31 (1743456631) targets are mounted 17:30:31 (1743456631) facet_failover done 1 11310 0.57 MB/sec execute 92 sec latency 15404.184 ms 1 11310 0.56 MB/sec execute 93 sec latency 16404.407 ms 1 11310 0.56 MB/sec execute 94 sec latency 17404.613 ms 1 11310 0.55 MB/sec execute 95 sec latency 18404.797 ms 1 11315 0.54 MB/sec execute 96 sec latency 19374.046 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11563 0.55 MB/sec execute 97 sec latency 22.196 ms 1 11846 0.55 MB/sec execute 98 sec latency 16.065 ms 1 12190 0.58 MB/sec execute 99 sec latency 18.832 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1908 28.497 19374.034 Close 1407 1.595 5.773 Rename 79 14.223 22.003 Unlink 376 4.531 16.057 Qpathinfo 1742 14.010 20110.046 Qfileinfo 301 0.443 2.308 Qfsinfo 306 0.910 13.941 Sfileinfo 150 8.232 13.910 Find 657 38.707 23841.068 WriteX 924 2.005 7.444 ReadX 2932 0.115 31.964 LockX 6 1.504 1.653 UnlockX 6 1.571 1.684 Flush 131 8.161 44.344 Throughput 0.579316 MB/sec 1 clients 1 procs max_latency=23841.085 ms stopping dbench on /mnt/lustre at Mon Mar 31 17:30:40 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3740 1283948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5240 1282448 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 55920 3546192 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21148 3582664 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 77068 7128856 2% /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Mar 31 17:30:42 EDT 2025 waiting for dbench pid 8692 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 22.646 ms 1 517 8.90 MB/sec warmup 2 sec latency 17.576 ms test_26 fail mds1 5 times Failing mds1 on oleg345-server Stopping /mnt/lustre-mds1 (opts:) on oleg345-server 1 808 7.16 MB/sec warmup 3 sec latency 27.981 ms 1 853 5.39 MB/sec warmup 4 sec latency 858.511 ms 1 853 4.31 MB/sec warmup 5 sec latency 1858.664 ms 1 853 3.60 MB/sec warmup 6 sec latency 2858.857 ms 1 853 3.08 MB/sec warmup 7 sec latency 3859.019 ms 1 853 2.70 MB/sec warmup 8 sec latency 4859.219 ms 1 853 2.40 MB/sec warmup 9 sec latency 5859.394 ms 17:30:51 (1743456651) shut down facet: mds1 facet_host: oleg345-server facet_failover_host: oleg345-server 1 853 2.16 MB/sec warmup 10 sec latency 6859.609 ms 1 853 1.96 MB/sec warmup 11 sec latency 7859.801 ms 1 853 1.80 MB/sec warmup 12 sec latency 8859.982 ms 1 853 1.66 MB/sec warmup 13 sec latency 9860.202 ms 1 853 1.54 MB/sec warmup 14 sec latency 10860.383 ms 1 853 1.44 MB/sec warmup 15 sec latency 11860.570 ms 1 853 1.35 MB/sec warmup 16 sec latency 12860.740 ms 1 853 1.27 MB/sec warmup 17 sec latency 13860.929 ms 1 853 1.20 MB/sec warmup 18 sec latency 14861.153 ms 1 853 1.14 MB/sec warmup 19 sec latency 15861.347 ms Failover mds1 to oleg345-server mount facets: mds1 1 853 0.00 MB/sec execute 1 sec latency 17861.684 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 853 0.00 MB/sec execute 2 sec latency 18861.810 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 17:31:05 (1743456665) targets are mounted 17:31:05 (1743456665) facet_failover done 1 853 0.00 MB/sec execute 3 sec latency 19861.955 ms 1 853 0.00 MB/sec execute 4 sec latency 20862.072 ms 1 853 0.00 MB/sec execute 5 sec latency 21862.240 ms 1 853 0.00 MB/sec execute 6 sec latency 22862.411 ms 1 853 0.00 MB/sec execute 7 sec latency 23862.541 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1008 0.09 MB/sec execute 8 sec latency 24115.062 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1343 0.37 MB/sec execute 9 sec latency 45.409 ms 1 1655 0.46 MB/sec execute 10 sec latency 30.480 ms 1 1989 0.44 MB/sec execute 11 sec latency 16.999 ms 1 2347 0.53 MB/sec execute 12 sec latency 26.866 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2636 1285052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4120 1283568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 48480 3549008 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12672 3583800 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 61152 7132808 1% /mnt/lustre 1 2601 0.57 MB/sec execute 13 sec latency 40.467 ms 1 3210 0.91 MB/sec execute 14 sec latency 16.560 ms 1 3677 1.14 MB/sec execute 15 sec latency 11.077 ms test_26 fail mds2 6 times 1 3947 1.16 MB/sec execute 16 sec latency 13.900 ms Failing mds2 on oleg345-server Stopping /mnt/lustre-mds2 (opts:) on oleg345-server 1 4152 1.09 MB/sec execute 17 sec latency 60.379 ms 1 4152 1.03 MB/sec execute 18 sec latency 1060.508 ms 1 4152 0.97 MB/sec execute 19 sec latency 2060.626 ms 1 4152 0.93 MB/sec execute 20 sec latency 3060.827 ms 1 4152 0.88 MB/sec execute 21 sec latency 4061.018 ms 1 4152 0.84 MB/sec execute 22 sec latency 5061.234 ms 1 4152 0.80 MB/sec execute 23 sec latency 6061.345 ms 17:31:25 (1743456685) shut down facet: mds2 facet_host: oleg345-server facet_failover_host: oleg345-server 1 4152 0.77 MB/sec execute 24 sec latency 7061.517 ms 1 4152 0.74 MB/sec execute 25 sec latency 8061.754 ms 1 4152 0.71 MB/sec execute 26 sec latency 9061.985 ms 1 4152 0.69 MB/sec execute 27 sec latency 10062.220 ms 1 4152 0.66 MB/sec execute 28 sec latency 11062.370 ms 1 4152 0.64 MB/sec execute 29 sec latency 12062.526 ms 1 4152 0.62 MB/sec execute 30 sec latency 13062.690 ms 1 4152 0.60 MB/sec execute 31 sec latency 14062.903 ms 1 4152 0.58 MB/sec execute 32 sec latency 15063.136 ms 1 4152 0.56 MB/sec execute 33 sec latency 16063.280 ms Failover mds2 to oleg345-server mount facets: mds2 1 4152 0.54 MB/sec execute 34 sec latency 17063.441 ms 1 4152 0.53 MB/sec execute 35 sec latency 18063.632 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4152 0.51 MB/sec execute 36 sec latency 19063.751 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 17:31:39 (1743456699) targets are mounted 17:31:39 (1743456699) facet_failover done 1 4152 0.50 MB/sec execute 37 sec latency 20063.927 ms 1 4152 0.49 MB/sec execute 38 sec latency 21064.032 ms 1 4152 0.47 MB/sec execute 39 sec latency 22064.192 ms 1 4152 0.46 MB/sec execute 40 sec latency 23064.388 ms 1 4152 0.45 MB/sec execute 41 sec latency 24064.540 ms 1 4152 0.44 MB/sec execute 42 sec latency 25064.664 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4376 0.44 MB/sec execute 43 sec latency 25175.090 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4663 0.47 MB/sec execute 44 sec latency 16.811 ms 1 5033 0.52 MB/sec execute 45 sec latency 17.543 ms 1 5292 0.52 MB/sec execute 46 sec latency 43.436 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3156 1284532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 49444 3556744 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17876 3588500 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 67320 7145244 1% /mnt/lustre 1 5583 0.51 MB/sec execute 47 sec latency 29.745 ms 1 5946 0.53 MB/sec execute 48 sec latency 14.000 ms 1 6161 0.54 MB/sec execute 49 sec latency 318.769 ms 1 6831 0.66 MB/sec execute 50 sec latency 15.665 ms test_26 fail mds1 7 times 1 7254 0.71 MB/sec execute 51 sec latency 13.714 ms Failing mds1 on oleg345-server Stopping /mnt/lustre-mds1 (opts:) on oleg345-server 1 7408 0.72 MB/sec execute 52 sec latency 434.589 ms 17:31:54 (1743456714) shut down facet: mds1 facet_host: oleg345-server facet_failover_host: oleg345-server 1 7408 0.70 MB/sec execute 53 sec latency 1434.712 ms 1 7408 0.69 MB/sec execute 54 sec latency 2434.841 ms 1 7408 0.68 MB/sec execute 55 sec latency 3435.012 ms 1 7408 0.67 MB/sec execute 56 sec latency 4435.186 ms 1 7408 0.65 MB/sec execute 57 sec latency 5435.369 ms 1 7408 0.64 MB/sec execute 58 sec latency 6435.558 ms 1 7408 0.63 MB/sec execute 59 sec latency 7435.695 ms 1 7408 0.62 MB/sec execute 60 sec latency 8435.889 ms 1 7408 0.61 MB/sec execute 61 sec latency 9436.049 ms 1 7408 0.60 MB/sec execute 62 sec latency 10436.228 ms Failover mds1 to oleg345-server mount facets: mds1 1 7408 0.59 MB/sec execute 63 sec latency 11436.346 ms 1 7408 0.58 MB/sec execute 64 sec latency 12436.490 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7408 0.57 MB/sec execute 65 sec latency 13436.639 ms oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 1 7408 0.57 MB/sec execute 66 sec latency 14436.735 ms 17:32:08 (1743456728) targets are mounted 17:32:08 (1743456728) facet_failover done 1 7408 0.56 MB/sec execute 67 sec latency 15436.893 ms 1 7408 0.55 MB/sec execute 68 sec latency 16437.063 ms 1 7408 0.54 MB/sec execute 69 sec latency 17437.195 ms 1 7408 0.53 MB/sec execute 70 sec latency 18437.313 ms oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7572 0.53 MB/sec execute 71 sec latency 18655.917 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7808 0.53 MB/sec execute 72 sec latency 19.930 ms striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 8040 0.53 MB/sec execute 73 sec latency 55.339 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Mar 31 17:32:16 EDT 2025 with return code 0 8692 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 8692 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished