== replay-dual test 26: dbench and tar with mds failover ========================================================== 11:03:18 (1736265798) Starting client oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre Started clients oleg420-client.virtnet: 192.168.204.120@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 6487 Started dbench loop with 6488 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jan 7 11:03:20 EST 2025 waiting for dbench pid 6523 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 19.623 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25984 3555980 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27372 7161612 1% /mnt/lustre 1 501 8.73 MB/sec warmup 2 sec latency 21.076 ms 1 738 7.10 MB/sec warmup 3 sec latency 368.840 ms 1 1080 5.73 MB/sec warmup 4 sec latency 14.909 ms test_26 fail mds1 1 times 1 1489 5.21 MB/sec warmup 5 sec latency 15.302 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 1787 4.38 MB/sec warmup 6 sec latency 242.840 ms reboot facets: mds1 1 1787 3.76 MB/sec warmup 7 sec latency 1243.070 ms 1 1787 3.29 MB/sec warmup 8 sec latency 2243.256 ms 1 1787 2.92 MB/sec warmup 9 sec latency 3243.441 ms 1 1787 2.63 MB/sec warmup 10 sec latency 4243.627 ms 1 1787 2.39 MB/sec warmup 11 sec latency 5243.905 ms 1 1787 2.19 MB/sec warmup 12 sec latency 6244.155 ms 1 1787 2.02 MB/sec warmup 13 sec latency 7244.383 ms 1 1787 1.88 MB/sec warmup 14 sec latency 8244.584 ms 1 1787 1.75 MB/sec warmup 15 sec latency 9244.771 ms 1 1787 1.64 MB/sec warmup 16 sec latency 10244.989 ms Failover mds1 to oleg420-server mount facets: mds1 1 1787 1.55 MB/sec warmup 17 sec latency 11245.152 ms 1 1787 1.46 MB/sec warmup 18 sec latency 12245.268 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1787 1.38 MB/sec warmup 19 sec latency 13245.421 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1787 0.00 MB/sec execute 1 sec latency 15245.855 ms 1 1787 0.00 MB/sec execute 2 sec latency 16246.004 ms 1 1787 0.00 MB/sec execute 3 sec latency 17246.175 ms 1 1787 0.00 MB/sec execute 4 sec latency 18246.420 ms 1 1787 0.00 MB/sec execute 5 sec latency 19246.554 ms 1 2146 0.06 MB/sec execute 6 sec latency 19323.695 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2479 0.34 MB/sec execute 7 sec latency 14.431 ms 1 3169 0.98 MB/sec execute 8 sec latency 16.821 ms 1 3673 1.38 MB/sec execute 9 sec latency 14.259 ms 1 3954 1.38 MB/sec execute 10 sec latency 15.150 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2760 1284928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 34824 3567336 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13544 3583316 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 48368 7150652 1% /mnt/lustre 1 4249 1.27 MB/sec execute 11 sec latency 25.898 ms 1 4549 1.23 MB/sec execute 12 sec latency 15.245 ms 1 4985 1.45 MB/sec execute 13 sec latency 14.490 ms 1 5337 1.36 MB/sec execute 14 sec latency 14.823 ms test_26 fail mds2 2 times Failing mds2 on oleg420-server Stopping /mnt/lustre-mds2 (opts:) on oleg420-server 1 5719 1.30 MB/sec execute 15 sec latency 11.656 ms reboot facets: mds2 1 5820 1.22 MB/sec execute 16 sec latency 715.934 ms 1 5820 1.15 MB/sec execute 17 sec latency 1716.097 ms 1 5820 1.09 MB/sec execute 18 sec latency 2716.296 ms 1 5820 1.03 MB/sec execute 19 sec latency 3716.559 ms 1 5820 0.98 MB/sec execute 20 sec latency 4716.773 ms 1 5820 0.93 MB/sec execute 21 sec latency 5716.942 ms 1 5820 0.89 MB/sec execute 22 sec latency 6717.116 ms 1 5820 0.85 MB/sec execute 23 sec latency 7717.275 ms 1 5820 0.82 MB/sec execute 24 sec latency 8717.425 ms 1 5820 0.78 MB/sec execute 25 sec latency 9717.612 ms Failover mds2 to oleg420-server 1 5820 0.75 MB/sec execute 26 sec latency 10717.740 ms mount facets: mds2 1 5820 0.73 MB/sec execute 27 sec latency 11717.841 ms 1 5820 0.70 MB/sec execute 28 sec latency 12717.954 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 5820 0.68 MB/sec execute 29 sec latency 13718.124 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 1 5820 0.65 MB/sec execute 30 sec latency 14718.256 ms Started lustre-MDT0001 1 5820 0.63 MB/sec execute 31 sec latency 15718.404 ms 1 5820 0.61 MB/sec execute 32 sec latency 16718.559 ms 1 5820 0.59 MB/sec execute 33 sec latency 17718.770 ms 1 5820 0.58 MB/sec execute 34 sec latency 18718.916 ms 1 5820 0.56 MB/sec execute 35 sec latency 19719.042 ms 1 5976 0.58 MB/sec execute 36 sec latency 20265.132 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 6486 0.71 MB/sec execute 37 sec latency 17.618 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7092 0.82 MB/sec execute 38 sec latency 13.397 ms 1 7400 0.84 MB/sec execute 39 sec latency 17.635 ms 1 7662 0.82 MB/sec execute 40 sec latency 14.222 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2640 1285048 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3356 1284332 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38564 3568112 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20916 3585772 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59480 7153884 1% /mnt/lustre 1 8009 0.82 MB/sec execute 41 sec latency 36.221 ms 1 8360 0.83 MB/sec execute 42 sec latency 14.760 ms 1 8795 0.89 MB/sec execute 43 sec latency 13.688 ms 1 9196 0.88 MB/sec execute 44 sec latency 13.637 ms test_26 fail mds1 3 times 1 9571 0.90 MB/sec execute 45 sec latency 50.053 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 10118 0.99 MB/sec execute 46 sec latency 18.065 ms reboot facets: mds1 1 10748 1.08 MB/sec execute 47 sec latency 16.853 ms 1 11029 1.08 MB/sec execute 48 sec latency 15.061 ms 1 11174 1.06 MB/sec execute 49 sec latency 425.860 ms 1 11174 1.04 MB/sec execute 50 sec latency 1425.964 ms 1 11174 1.02 MB/sec execute 51 sec latency 2426.129 ms 1 11174 1.00 MB/sec execute 52 sec latency 3426.242 ms 1 11174 0.98 MB/sec execute 53 sec latency 4426.427 ms 1 11174 0.96 MB/sec execute 54 sec latency 5426.591 ms 1 11174 0.95 MB/sec execute 55 sec latency 6426.770 ms 1 11174 0.93 MB/sec execute 56 sec latency 7426.966 ms Failover mds1 to oleg420-server mount facets: mds1 1 11174 0.91 MB/sec execute 57 sec latency 8427.100 ms 1 11174 0.90 MB/sec execute 58 sec latency 9427.233 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11174 0.88 MB/sec execute 59 sec latency 10427.400 ms 1 11174 0.87 MB/sec execute 60 sec latency 11427.576 ms 1 11174 0.85 MB/sec execute 61 sec latency 12427.747 ms 1 11174 0.84 MB/sec execute 62 sec latency 13427.902 ms 1 11174 0.83 MB/sec execute 63 sec latency 14428.120 ms 1 11174 0.81 MB/sec execute 64 sec latency 15428.238 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 1 11174 0.80 MB/sec execute 65 sec latency 16428.385 ms 1 11174 0.79 MB/sec execute 66 sec latency 17428.535 ms 1 11174 0.78 MB/sec execute 67 sec latency 18428.718 ms 1 11174 0.76 MB/sec execute 68 sec latency 19428.892 ms 1 11241 0.75 MB/sec execute 69 sec latency 20257.900 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 11577 0.75 MB/sec execute 70 sec latency 14.746 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11913 0.76 MB/sec execute 71 sec latency 16.860 ms 1 12310 0.80 MB/sec execute 72 sec latency 16.316 ms 1 12688 0.79 MB/sec execute 73 sec latency 14.556 ms 1 13078 0.80 MB/sec execute 74 sec latency 14.527 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3020 1284668 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4744 1282944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 52616 3553196 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24272 3581400 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 76888 7134596 2% /mnt/lustre 1 13585 0.86 MB/sec execute 75 sec latency 15.885 ms 1 14007 0.87 MB/sec execute 76 sec latency 362.392 ms 1 14424 0.92 MB/sec execute 77 sec latency 14.515 ms test_26 fail mds2 4 times 1 14654 0.91 MB/sec execute 78 sec latency 23.035 ms Failing mds2 on oleg420-server Stopping /mnt/lustre-mds2 (opts:) on oleg420-server 1 14935 0.90 MB/sec execute 79 sec latency 26.248 ms reboot facets: mds2 1 15200 0.90 MB/sec execute 80 sec latency 89.218 ms 1 15200 0.89 MB/sec execute 81 sec latency 1089.366 ms 1 15200 0.88 MB/sec execute 82 sec latency 2089.519 ms 1 15200 0.87 MB/sec execute 83 sec latency 3089.634 ms 1 15200 0.86 MB/sec execute 84 sec latency 4089.798 ms 1 15200 0.85 MB/sec execute 85 sec latency 5089.942 ms 1 15200 0.84 MB/sec execute 86 sec latency 6090.072 ms 1 15200 0.83 MB/sec execute 87 sec latency 7090.211 ms 1 15200 0.82 MB/sec execute 88 sec latency 8090.403 ms 1 15200 0.81 MB/sec execute 89 sec latency 9090.580 ms Failover mds2 to oleg420-server mount facets: mds2 1 15200 0.80 MB/sec execute 90 sec latency 10090.729 ms 1 15200 0.79 MB/sec execute 91 sec latency 11090.847 ms 1 15200 0.78 MB/sec execute 92 sec latency 12090.981 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 15200 0.78 MB/sec execute 93 sec latency 13091.129 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 1 15200 0.77 MB/sec execute 94 sec latency 14091.264 ms 1 15200 0.76 MB/sec execute 95 sec latency 15091.412 ms 1 15200 0.75 MB/sec execute 96 sec latency 16091.588 ms 1 15200 0.74 MB/sec execute 97 sec latency 17091.763 ms 1 15200 0.74 MB/sec execute 98 sec latency 18091.898 ms 1 15416 0.74 MB/sec execute 99 sec latency 18478.020 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 100 sec 0 cleanup 100 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2413 31.856 20265.121 Close 1757 1.446 36.197 Rename 103 11.015 16.840 Unlink 493 4.003 13.590 Qpathinfo 2208 11.139 20257.883 Qfileinfo 381 0.364 2.226 Qfsinfo 408 0.685 10.926 Sfileinfo 200 5.316 9.501 Find 853 1.571 19.135 WriteX 1199 2.016 7.579 ReadX 3823 0.032 2.925 LockX 8 1.265 1.398 UnlockX 8 1.340 1.731 Flush 170 9.715 362.378 Throughput 0.739666 MB/sec 1 clients 1 procs max_latency=20265.132 ms stopping dbench on /mnt/lustre at Tue Jan 7 11:05:20 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jan 7 11:05:21 EST 2025 waiting for dbench pid 11355 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 264 9.92 MB/sec warmup 1 sec latency 19.052 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3644 1284044 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5272 1282416 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19580 3586996 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38392 3542636 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 57972 7129632 1% /mnt/lustre 1 513 8.86 MB/sec warmup 2 sec latency 26.859 ms 1 884 7.22 MB/sec warmup 3 sec latency 16.618 ms 1 1238 5.77 MB/sec warmup 4 sec latency 14.010 ms 1 1676 5.24 MB/sec warmup 5 sec latency 26.879 ms test_26 fail mds1 5 times 1 2060 4.44 MB/sec warmup 6 sec latency 41.287 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 2330 3.98 MB/sec warmup 7 sec latency 414.682 ms 1 2330 3.48 MB/sec warmup 8 sec latency 1414.826 ms 1 2330 3.09 MB/sec warmup 9 sec latency 2415.063 ms 1 2330 2.78 MB/sec warmup 10 sec latency 3415.319 ms 1 2330 2.53 MB/sec warmup 11 sec latency 4415.508 ms 1 2330 2.32 MB/sec warmup 12 sec latency 5415.667 ms reboot facets: mds1 1 2330 2.14 MB/sec warmup 13 sec latency 6415.781 ms 1 2330 1.99 MB/sec warmup 14 sec latency 7415.921 ms 1 2330 1.86 MB/sec warmup 15 sec latency 8416.128 ms 1 2330 1.74 MB/sec warmup 16 sec latency 9416.295 ms 1 2330 1.64 MB/sec warmup 17 sec latency 10416.526 ms 1 2330 1.55 MB/sec warmup 18 sec latency 11416.658 ms 1 2330 1.46 MB/sec warmup 19 sec latency 12416.805 ms 1 2330 0.00 MB/sec execute 1 sec latency 14417.124 ms 1 2330 0.00 MB/sec execute 2 sec latency 15417.284 ms Failover mds1 to oleg420-server mount facets: mds1 1 2330 0.00 MB/sec execute 3 sec latency 16417.411 ms 1 2330 0.00 MB/sec execute 4 sec latency 17417.573 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 2330 0.00 MB/sec execute 5 sec latency 18417.729 ms 1 2330 0.00 MB/sec execute 6 sec latency 19417.917 ms 1 2330 0.00 MB/sec execute 7 sec latency 20418.082 ms 1 2330 0.00 MB/sec execute 8 sec latency 21418.204 ms 1 2330 0.00 MB/sec execute 9 sec latency 22418.330 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 1 2330 0.00 MB/sec execute 10 sec latency 23418.479 ms Started lustre-MDT0000 1 2330 0.00 MB/sec execute 11 sec latency 24418.646 ms 1 2330 0.00 MB/sec execute 12 sec latency 25418.820 ms 1 2330 0.00 MB/sec execute 13 sec latency 26418.988 ms 1 2330 0.00 MB/sec execute 14 sec latency 27419.114 ms 1 2435 0.02 MB/sec execute 15 sec latency 27972.161 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2918 0.35 MB/sec execute 16 sec latency 40.083 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3492 0.62 MB/sec execute 17 sec latency 14.785 ms 1 3827 0.66 MB/sec execute 18 sec latency 15.044 ms 1 4085 0.65 MB/sec execute 19 sec latency 14.265 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2644 1285044 1% /mnt/lustre[MDT:0] 1 4379 0.64 MB/sec execute 20 sec latency 34.210 ms lustre-MDT0001_UUID 1414116 3988 1283700 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19500 3584860 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47516 3558236 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 67016 7143096 1% /mnt/lustre 1 4714 0.68 MB/sec execute 21 sec latency 22.318 ms 1 5091 0.79 MB/sec execute 22 sec latency 23.870 ms 1 5410 0.77 MB/sec execute 23 sec latency 21.534 ms test_26 fail mds2 6 times 1 5814 0.75 MB/sec execute 24 sec latency 29.033 ms Failing mds2 on oleg420-server Stopping /mnt/lustre-mds2 (opts:) on oleg420-server 1 6153 0.81 MB/sec execute 25 sec latency 50.033 ms reboot facets: mds2 1 6830 1.02 MB/sec execute 26 sec latency 14.651 ms 1 7304 1.15 MB/sec execute 27 sec latency 14.479 ms 1 7553 1.12 MB/sec execute 28 sec latency 14.579 ms 1 7627 1.08 MB/sec execute 29 sec latency 731.768 ms 1 7627 1.05 MB/sec execute 30 sec latency 1731.969 ms 1 7627 1.01 MB/sec execute 31 sec latency 2732.193 ms 1 7627 0.98 MB/sec execute 32 sec latency 3732.423 ms 1 7627 0.95 MB/sec execute 33 sec latency 4732.629 ms 1 7627 0.92 MB/sec execute 34 sec latency 5732.749 ms 1 7627 0.90 MB/sec execute 35 sec latency 6732.918 ms Failover mds2 to oleg420-server mount facets: mds2 1 7627 0.87 MB/sec execute 36 sec latency 7733.052 ms 1 7627 0.85 MB/sec execute 37 sec latency 8733.181 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 7627 0.83 MB/sec execute 38 sec latency 9733.393 ms 1 7627 0.80 MB/sec execute 39 sec latency 10733.520 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 1 7627 0.78 MB/sec execute 40 sec latency 11733.684 ms 1 7627 0.77 MB/sec execute 41 sec latency 12733.825 ms 1 7627 0.75 MB/sec execute 42 sec latency 13733.982 ms 1 7627 0.73 MB/sec execute 43 sec latency 14734.147 ms 1 7627 0.71 MB/sec execute 44 sec latency 15734.261 ms 1 7812 0.70 MB/sec execute 45 sec latency 16237.512 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8109 0.70 MB/sec execute 46 sec latency 15.333 ms 1 8537 0.77 MB/sec execute 47 sec latency 15.737 ms 1 8839 0.76 MB/sec execute 48 sec latency 27.622 ms 1 9228 0.76 MB/sec execute 49 sec latency 26.312 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2552 1285136 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19792 3586520 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44672 3561228 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 64464 7147748 1% /mnt/lustre 1 9578 0.78 MB/sec execute 50 sec latency 26.266 ms 1 10099 0.86 MB/sec execute 51 sec latency 86.216 ms 1 10590 0.94 MB/sec execute 52 sec latency 430.931 ms striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar 1 10901 0.95 MB/sec execute 53 sec latency 39.392 ms test_26 fail mds1 7 times 1 11145 0.93 MB/sec execute 54 sec latency 15.083 ms Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 1 11507 0.93 MB/sec execute 55 sec latency 17.470 ms reboot facets: mds1 1 11653 0.92 MB/sec execute 56 sec latency 484.353 ms 1 11653 0.90 MB/sec execute 57 sec latency 1484.593 ms 1 11653 0.89 MB/sec execute 58 sec latency 2484.787 ms 1 11653 0.87 MB/sec execute 59 sec latency 3485.016 ms 1 11653 0.86 MB/sec execute 60 sec latency 4485.263 ms 1 11653 0.84 MB/sec execute 61 sec latency 5485.415 ms 1 11653 0.83 MB/sec execute 62 sec latency 6485.623 ms 1 11653 0.82 MB/sec execute 63 sec latency 7485.849 ms 1 11653 0.80 MB/sec execute 64 sec latency 8486.068 ms 1 11653 0.79 MB/sec execute 65 sec latency 9486.274 ms Failover mds1 to oleg420-server mount facets: mds1 1 11653 0.78 MB/sec execute 66 sec latency 10486.436 ms 1 11653 0.77 MB/sec execute 67 sec latency 11486.607 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11653 0.76 MB/sec execute 68 sec latency 12486.778 ms 1 11653 0.75 MB/sec execute 69 sec latency 13486.918 ms 1 11653 0.73 MB/sec execute 70 sec latency 14487.070 ms 1 11653 0.72 MB/sec execute 71 sec latency 15487.220 ms 1 11653 0.71 MB/sec execute 72 sec latency 16487.347 ms oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 1 11653 0.70 MB/sec execute 73 sec latency 17487.472 ms Started lustre-MDT0000 1 11653 0.69 MB/sec execute 74 sec latency 18487.629 ms 1 11653 0.69 MB/sec execute 75 sec latency 19487.778 ms 1 11653 0.68 MB/sec execute 76 sec latency 20487.915 ms 1 11653 0.67 MB/sec execute 77 sec latency 21488.036 ms 1 11902 0.67 MB/sec execute 78 sec latency 21780.752 ms oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 12334 0.71 MB/sec execute 79 sec latency 14.808 ms 1 12713 0.70 MB/sec execute 80 sec latency 17.235 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 13099 0.71 MB/sec execute 81 sec latency 13.001 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jan 7 11:07:03 EST 2025 with return code 0 11355 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 11355 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished