== replay-dual test 26: dbench and tar with mds failover ========================================================== 11:03:19 (1736265799) Starting client oleg404-client.virtnet: -o user_xattr,flock oleg404-server@tcp:/lustre /mnt/lustre Started clients oleg404-client.virtnet: 192.168.204.104@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 6496 Started dbench loop with 6497 striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jan 7 11:03:21 EST 2025 waiting for dbench pid 6532 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 279 10.44 MB/sec warmup 1 sec latency 17.377 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2216 1285472 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25984 3556056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27372 7161688 1% /mnt/lustre 1 542 9.55 MB/sec warmup 2 sec latency 19.187 ms 1 791 7.15 MB/sec warmup 3 sec latency 354.058 ms 1 1164 5.75 MB/sec warmup 4 sec latency 12.837 ms test_26 fail mds1 1 times 1 1616 5.23 MB/sec warmup 5 sec latency 14.670 ms Failing mds1 on oleg404-server Stopping /mnt/lustre-mds1 (opts:) on oleg404-server 1 1877 4.39 MB/sec warmup 6 sec latency 320.064 ms reboot facets: mds1 1 1877 3.77 MB/sec warmup 7 sec latency 1320.268 ms 1 1877 3.30 MB/sec warmup 8 sec latency 2320.404 ms 1 1877 2.93 MB/sec warmup 9 sec latency 3320.594 ms 1 1877 2.64 MB/sec warmup 10 sec latency 4320.996 ms 1 1877 2.40 MB/sec warmup 11 sec latency 5321.234 ms 1 1877 2.20 MB/sec warmup 12 sec latency 6321.408 ms 1 1877 2.03 MB/sec warmup 13 sec latency 7321.536 ms 1 1877 1.88 MB/sec warmup 14 sec latency 8321.739 ms 1 1877 1.76 MB/sec warmup 15 sec latency 9321.905 ms 1 1877 1.65 MB/sec warmup 16 sec latency 10322.122 ms Failover mds1 to oleg404-server mount facets: mds1 1 1877 1.55 MB/sec warmup 17 sec latency 11322.280 ms 1 1877 1.46 MB/sec warmup 18 sec latency 12322.407 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1877 1.39 MB/sec warmup 19 sec latency 13322.682 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1877 0.00 MB/sec execute 1 sec latency 15322.983 ms 1 1877 0.00 MB/sec execute 2 sec latency 16323.144 ms 1 1877 0.00 MB/sec execute 3 sec latency 17323.355 ms 1 1877 0.00 MB/sec execute 4 sec latency 18323.574 ms 1 1877 0.00 MB/sec execute 5 sec latency 19323.689 ms 1 2156 0.06 MB/sec execute 6 sec latency 19560.387 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2558 0.36 MB/sec execute 7 sec latency 13.473 ms 1 3209 0.99 MB/sec execute 8 sec latency 13.921 ms 1 3721 1.38 MB/sec execute 9 sec latency 13.721 ms 1 3996 1.37 MB/sec execute 10 sec latency 15.473 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2820 1284868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 29836 3570684 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5316 3583408 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 35152 7154092 1% /mnt/lustre 1 4318 1.28 MB/sec execute 11 sec latency 13.814 ms 1 4663 1.31 MB/sec execute 12 sec latency 17.723 ms 1 5075 1.45 MB/sec execute 13 sec latency 15.161 ms 1 5420 1.37 MB/sec execute 14 sec latency 13.857 ms test_26 fail mds2 2 times Failing mds2 on oleg404-server 1 5905 1.38 MB/sec execute 15 sec latency 17.725 ms Stopping /mnt/lustre-mds2 (opts:) on oleg404-server reboot facets: mds2 1 5988 1.30 MB/sec execute 16 sec latency 701.288 ms 1 5988 1.23 MB/sec execute 17 sec latency 1701.462 ms 1 5988 1.16 MB/sec execute 18 sec latency 2701.770 ms 1 5988 1.10 MB/sec execute 19 sec latency 3702.010 ms 1 5988 1.04 MB/sec execute 20 sec latency 4702.233 ms 1 5988 0.99 MB/sec execute 21 sec latency 5702.456 ms 1 5988 0.95 MB/sec execute 22 sec latency 6702.622 ms 1 5988 0.91 MB/sec execute 23 sec latency 7702.911 ms 1 5988 0.87 MB/sec execute 24 sec latency 8703.039 ms 1 5988 0.83 MB/sec execute 25 sec latency 9703.194 ms Failover mds2 to oleg404-server 1 5988 0.80 MB/sec execute 26 sec latency 10703.330 ms mount facets: mds2 1 5988 0.77 MB/sec execute 27 sec latency 11703.551 ms 1 5988 0.74 MB/sec execute 28 sec latency 12703.731 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 5988 0.72 MB/sec execute 29 sec latency 13703.825 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 1 5988 0.70 MB/sec execute 30 sec latency 14703.926 ms Started lustre-MDT0001 1 5988 0.67 MB/sec execute 31 sec latency 15704.073 ms 1 5988 0.65 MB/sec execute 32 sec latency 16704.268 ms 1 5988 0.63 MB/sec execute 33 sec latency 17704.430 ms 1 5988 0.61 MB/sec execute 34 sec latency 18704.555 ms 1 5988 0.60 MB/sec execute 35 sec latency 19704.651 ms 1 6112 0.60 MB/sec execute 36 sec latency 20401.179 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 6726 0.73 MB/sec execute 37 sec latency 22.076 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7199 0.83 MB/sec execute 38 sec latency 15.694 ms 1 7492 0.84 MB/sec execute 39 sec latency 14.860 ms 1 7823 0.83 MB/sec execute 40 sec latency 13.977 ms 1 8161 0.85 MB/sec execute 41 sec latency 23.181 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2636 1285052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3404 1284284 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37420 3567668 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22080 3583060 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59500 7150728 1% /mnt/lustre 1 8601 0.90 MB/sec execute 42 sec latency 13.702 ms 1 8929 0.89 MB/sec execute 43 sec latency 80.072 ms 1 9387 0.89 MB/sec execute 44 sec latency 8.594 ms test_26 fail mds1 3 times 1 9820 0.97 MB/sec execute 45 sec latency 37.699 ms Failing mds1 on oleg404-server Stopping /mnt/lustre-mds1 (opts:) on oleg404-server 1 10453 1.03 MB/sec execute 46 sec latency 18.799 ms reboot facets: mds1 1 10898 1.10 MB/sec execute 47 sec latency 14.377 ms 1 11155 1.08 MB/sec execute 48 sec latency 13.770 ms 1 11174 1.06 MB/sec execute 49 sec latency 941.263 ms 1 11174 1.04 MB/sec execute 50 sec latency 1941.376 ms 1 11174 1.02 MB/sec execute 51 sec latency 2941.472 ms 1 11174 1.00 MB/sec execute 52 sec latency 3941.632 ms 1 11174 0.98 MB/sec execute 53 sec latency 4941.814 ms 1 11174 0.96 MB/sec execute 54 sec latency 5942.110 ms 1 11174 0.94 MB/sec execute 55 sec latency 6942.364 ms 1 11174 0.93 MB/sec execute 56 sec latency 7942.503 ms Failover mds1 to oleg404-server mount facets: mds1 1 11174 0.91 MB/sec execute 57 sec latency 8942.621 ms 1 11174 0.90 MB/sec execute 58 sec latency 9942.769 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11174 0.88 MB/sec execute 59 sec latency 10942.954 ms 1 11174 0.87 MB/sec execute 60 sec latency 11943.136 ms 1 11174 0.85 MB/sec execute 61 sec latency 12943.268 ms 1 11174 0.84 MB/sec execute 62 sec latency 13943.413 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 1 11174 0.82 MB/sec execute 63 sec latency 14943.556 ms pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-MDT0000 1 11174 0.81 MB/sec execute 64 sec latency 15943.749 ms 1 11174 0.80 MB/sec execute 65 sec latency 16943.956 ms 1 11174 0.79 MB/sec execute 66 sec latency 17944.104 ms 1 11174 0.78 MB/sec execute 67 sec latency 18944.236 ms 1 11520 0.77 MB/sec execute 68 sec latency 18991.510 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11841 0.78 MB/sec execute 69 sec latency 16.524 ms 1 12258 0.82 MB/sec execute 70 sec latency 16.275 ms 1 12640 0.81 MB/sec execute 71 sec latency 13.958 ms 1 13049 0.82 MB/sec execute 72 sec latency 13.208 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3076 1284612 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4748 1282940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 40548 3565524 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36272 3568692 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 76820 7134216 2% /mnt/lustre 1 13566 0.88 MB/sec execute 73 sec latency 13.888 ms 1 13932 0.90 MB/sec execute 74 sec latency 430.045 ms 1 14399 0.94 MB/sec execute 75 sec latency 11.327 ms 1 14645 0.93 MB/sec execute 76 sec latency 18.742 ms test_26 fail mds2 4 times Failing mds2 on oleg404-server Stopping /mnt/lustre-mds2 (opts:) on oleg404-server 1 14942 0.93 MB/sec execute 77 sec latency 17.096 ms reboot facets: mds2 1 15200 0.92 MB/sec execute 78 sec latency 107.206 ms 1 15200 0.91 MB/sec execute 79 sec latency 1107.379 ms 1 15200 0.90 MB/sec execute 80 sec latency 2107.557 ms 1 15200 0.89 MB/sec execute 81 sec latency 3107.726 ms 1 15200 0.88 MB/sec execute 82 sec latency 4107.837 ms 1 15200 0.87 MB/sec execute 83 sec latency 5107.996 ms 1 15200 0.86 MB/sec execute 84 sec latency 6108.153 ms 1 15200 0.85 MB/sec execute 85 sec latency 7108.342 ms 1 15200 0.84 MB/sec execute 86 sec latency 8108.499 ms 1 15200 0.83 MB/sec execute 87 sec latency 9108.640 ms Failover mds2 to oleg404-server 1 15200 0.82 MB/sec execute 88 sec latency 10108.810 ms mount facets: mds2 1 15200 0.81 MB/sec execute 89 sec latency 11108.937 ms 1 15200 0.80 MB/sec execute 90 sec latency 12109.057 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 15200 0.79 MB/sec execute 91 sec latency 13109.258 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-MDT0001 1 15200 0.78 MB/sec execute 92 sec latency 14109.435 ms 1 15200 0.77 MB/sec execute 93 sec latency 15109.565 ms 1 15200 0.77 MB/sec execute 94 sec latency 16109.784 ms 1 15200 0.76 MB/sec execute 95 sec latency 17110.003 ms 1 15200 0.75 MB/sec execute 96 sec latency 18110.140 ms 1 15404 0.75 MB/sec execute 97 sec latency 18513.090 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 15809 0.78 MB/sec execute 98 sec latency 15.125 ms 1 16187 0.77 MB/sec execute 99 sec latency 16.157 ms 1 cleanup 100 sec 0 cleanup 100 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2538 22.551 19560.373 Close 1893 1.400 18.773 Rename 106 203.010 20401.164 Unlink 505 3.857 13.536 Qpathinfo 2366 9.908 18991.489 Qfileinfo 405 0.367 2.332 Qfsinfo 414 0.710 9.113 Sfileinfo 207 5.225 13.015 Find 895 1.584 20.332 WriteX 1267 2.010 22.888 ReadX 3938 0.035 3.861 LockX 8 1.338 1.688 UnlockX 8 1.274 1.409 Flush 177 10.119 430.031 Throughput 0.773469 MB/sec 1 clients 1 procs max_latency=20401.179 ms stopping dbench on /mnt/lustre at Tue Jan 7 11:05:21 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3584 1284104 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5216 1282472 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 40772 3537256 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36112 3565188 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 76884 7102444 2% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jan 7 11:05:23 EST 2025 waiting for dbench pid 11411 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 271 10.18 MB/sec warmup 1 sec latency 19.595 ms 1 576 9.71 MB/sec warmup 2 sec latency 16.956 ms 1 923 7.26 MB/sec warmup 3 sec latency 27.481 ms test_26 fail mds1 5 times Failing mds1 on oleg404-server 1 1296 5.87 MB/sec warmup 4 sec latency 16.794 ms Stopping /mnt/lustre-mds1 (opts:) on oleg404-server 1 1497 5.21 MB/sec warmup 5 sec latency 581.788 ms 1 1497 4.34 MB/sec warmup 6 sec latency 1581.906 ms 1 1497 3.72 MB/sec warmup 7 sec latency 2581.998 ms 1 1497 3.26 MB/sec warmup 8 sec latency 3582.085 ms 1 1497 2.89 MB/sec warmup 9 sec latency 4582.174 ms 1 1497 2.60 MB/sec warmup 10 sec latency 5582.285 ms reboot facets: mds1 1 1497 2.37 MB/sec warmup 11 sec latency 6582.383 ms 1 1497 2.17 MB/sec warmup 12 sec latency 7582.505 ms 1 1497 2.00 MB/sec warmup 13 sec latency 8582.613 ms 1 1497 1.86 MB/sec warmup 14 sec latency 9582.719 ms 1 1497 1.74 MB/sec warmup 15 sec latency 10582.848 ms 1 1497 1.63 MB/sec warmup 16 sec latency 11582.960 ms 1 1497 1.53 MB/sec warmup 17 sec latency 12583.071 ms 1 1497 1.45 MB/sec warmup 18 sec latency 13583.235 ms 1 1497 1.37 MB/sec warmup 19 sec latency 14583.360 ms 1 1497 0.00 MB/sec execute 1 sec latency 16584.754 ms Failover mds1 to oleg404-server mount facets: mds1 1 1497 0.00 MB/sec execute 2 sec latency 17584.857 ms 1 1497 0.00 MB/sec execute 3 sec latency 18585.037 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1497 0.00 MB/sec execute 4 sec latency 19585.248 ms 1 1497 0.00 MB/sec execute 5 sec latency 20585.436 ms 1 1497 0.00 MB/sec execute 6 sec latency 21585.602 ms 1 1497 0.00 MB/sec execute 7 sec latency 22585.703 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 1 1497 0.00 MB/sec execute 8 sec latency 23585.808 ms Started lustre-MDT0000 1 1497 0.00 MB/sec execute 9 sec latency 24586.016 ms 1 1497 0.00 MB/sec execute 10 sec latency 25586.232 ms 1 1497 0.00 MB/sec execute 11 sec latency 26586.427 ms 1 1497 0.00 MB/sec execute 12 sec latency 27586.534 ms 1 1643 0.01 MB/sec execute 13 sec latency 28057.883 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2010 0.03 MB/sec execute 14 sec latency 21.049 ms 1 2405 0.13 MB/sec execute 15 sec latency 34.524 ms 1 2926 0.46 MB/sec execute 16 sec latency 15.694 ms striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 3563 0.73 MB/sec execute 17 sec latency 28.993 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2392 1285296 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37684 3566468 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11760 3594656 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49444 7161124 1% /mnt/lustre 1 3890 0.77 MB/sec execute 18 sec latency 13.843 ms 1 4165 0.74 MB/sec execute 19 sec latency 30.675 ms 1 4386 0.73 MB/sec execute 20 sec latency 351.844 ms 1 4727 0.77 MB/sec execute 21 sec latency 15.167 ms test_26 fail mds2 6 times Failing mds2 on oleg404-server 1 5143 0.87 MB/sec execute 22 sec latency 14.682 ms Stopping /mnt/lustre-mds2 (opts:) on oleg404-server 1 5490 0.85 MB/sec execute 23 sec latency 13.886 ms reboot facets: mds2 1 5820 0.83 MB/sec execute 24 sec latency 286.791 ms 1 5820 0.79 MB/sec execute 25 sec latency 1287.010 ms 1 5820 0.76 MB/sec execute 26 sec latency 2287.188 ms 1 5820 0.74 MB/sec execute 27 sec latency 3287.357 ms 1 5820 0.71 MB/sec execute 28 sec latency 4287.497 ms 1 5820 0.68 MB/sec execute 29 sec latency 5287.659 ms 1 5820 0.66 MB/sec execute 30 sec latency 6287.880 ms 1 5820 0.64 MB/sec execute 31 sec latency 7288.030 ms 1 5820 0.62 MB/sec execute 32 sec latency 8288.181 ms 1 5820 0.60 MB/sec execute 33 sec latency 9288.331 ms Failover mds2 to oleg404-server mount facets: mds2 1 5820 0.58 MB/sec execute 34 sec latency 10288.469 ms 1 5820 0.57 MB/sec execute 35 sec latency 11288.591 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 5820 0.55 MB/sec execute 36 sec latency 12288.722 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 1 5820 0.54 MB/sec execute 37 sec latency 13288.883 ms pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-MDT0001 1 5820 0.52 MB/sec execute 38 sec latency 14289.031 ms 1 5820 0.51 MB/sec execute 39 sec latency 15289.209 ms 1 5820 0.50 MB/sec execute 40 sec latency 16289.383 ms 1 5820 0.48 MB/sec execute 41 sec latency 17289.521 ms 1 5994 0.50 MB/sec execute 42 sec latency 17778.667 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6542 0.62 MB/sec execute 43 sec latency 13.931 ms 1 7164 0.72 MB/sec execute 44 sec latency 15.129 ms 1 7464 0.74 MB/sec execute 45 sec latency 14.038 ms 1 7778 0.72 MB/sec execute 46 sec latency 14.927 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2880 1284808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2944 1284744 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37956 3560416 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12144 3587100 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50100 7147516 1% /mnt/lustre 1 8082 0.72 MB/sec execute 47 sec latency 42.779 ms 1 8519 0.80 MB/sec execute 48 sec latency 15.444 ms 1 8839 0.78 MB/sec execute 49 sec latency 87.973 ms 1 9276 0.78 MB/sec execute 50 sec latency 8.639 ms test_26 fail mds1 7 times Failing mds1 on oleg404-server 1 9679 0.81 MB/sec execute 51 sec latency 36.180 ms Stopping /mnt/lustre-mds1 (opts:) on oleg404-server 1 10359 0.91 MB/sec execute 52 sec latency 15.751 ms reboot facets: mds1 1 10858 0.98 MB/sec execute 53 sec latency 11.748 ms 1 11126 0.97 MB/sec execute 54 sec latency 14.078 ms 1 11174 0.95 MB/sec execute 55 sec latency 847.945 ms 1 11174 0.93 MB/sec execute 56 sec latency 1848.046 ms 1 11174 0.92 MB/sec execute 57 sec latency 2848.133 ms 1 11174 0.90 MB/sec execute 58 sec latency 3848.254 ms 1 11174 0.89 MB/sec execute 59 sec latency 4848.403 ms 1 11174 0.87 MB/sec execute 60 sec latency 5848.515 ms 1 11174 0.86 MB/sec execute 61 sec latency 6848.620 ms 1 11174 0.84 MB/sec execute 62 sec latency 7848.719 ms Failover mds1 to oleg404-server mount facets: mds1 1 11174 0.83 MB/sec execute 63 sec latency 8848.819 ms 1 11174 0.82 MB/sec execute 64 sec latency 9848.942 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11174 0.80 MB/sec execute 65 sec latency 10849.026 ms 1 11174 0.79 MB/sec execute 66 sec latency 11849.117 ms 1 11174 0.78 MB/sec execute 67 sec latency 12849.242 ms 1 11174 0.77 MB/sec execute 68 sec latency 13849.344 ms 1 11174 0.76 MB/sec execute 69 sec latency 14849.487 ms oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all 8 1 11174 0.75 MB/sec execute 70 sec latency 15849.601 ms pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-MDT0000 1 11174 0.74 MB/sec execute 71 sec latency 16849.705 ms 1 11174 0.73 MB/sec execute 72 sec latency 17849.882 ms 1 11174 0.72 MB/sec execute 73 sec latency 18850.110 ms 1 11174 0.71 MB/sec execute 74 sec latency 19850.265 ms 1 11451 0.70 MB/sec execute 75 sec latency 20087.912 ms oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11810 0.71 MB/sec execute 76 sec latency 13.354 ms 1 12225 0.75 MB/sec execute 77 sec latency 14.105 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 12608 0.74 MB/sec execute 78 sec latency 13.597 ms 1 13048 0.75 MB/sec execute 79 sec latency 14.482 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jan 7 11:07:02 EST 2025 with return code 0 11411 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 11411 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished