== replay-dual test 26: dbench and tar with mds failover ========================================================== 21:51:33 (1716256293) Starting client oleg131-client.virtnet: -o user_xattr,flock oleg131-server@tcp:/lustre /mnt/lustre Started clients oleg131-client.virtnet: 192.168.201.131@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 6404 Started dbench loop with 6405 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon May 20 21:51:34 EDT 2024 waiting for dbench pid 6446 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 155 6.54 MB/sec warmup 1 sec latency 37.981 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2204 1285484 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1980 1285708 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26120 3580704 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1524 3605496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27644 7186200 1% /mnt/lustre 1 298 5.48 MB/sec warmup 2 sec latency 38.678 ms 1 437 5.13 MB/sec warmup 3 sec latency 32.645 ms 1 640 5.26 MB/sec warmup 4 sec latency 51.394 ms 1 799 4.29 MB/sec warmup 5 sec latency 25.133 ms test_26 fail mds1 1 times 1 960 3.64 MB/sec warmup 6 sec latency 24.975 ms Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 1 1134 3.28 MB/sec warmup 7 sec latency 210.017 ms 1 1134 2.87 MB/sec warmup 8 sec latency 1210.240 ms 1 1134 2.55 MB/sec warmup 9 sec latency 2210.445 ms 1 1134 2.30 MB/sec warmup 10 sec latency 3210.716 ms 1 1134 2.09 MB/sec warmup 11 sec latency 4210.938 ms 1 1134 1.92 MB/sec warmup 12 sec latency 5211.158 ms 1 1134 1.77 MB/sec warmup 13 sec latency 6211.366 ms 21:51:48 (1716256308) shut down 1 1134 1.64 MB/sec warmup 14 sec latency 7211.637 ms 1 1134 1.53 MB/sec warmup 15 sec latency 8211.856 ms 1 1134 1.44 MB/sec warmup 16 sec latency 9212.108 ms 1 1134 1.35 MB/sec warmup 17 sec latency 10212.388 ms 1 1134 1.28 MB/sec warmup 18 sec latency 11212.683 ms 1 1134 1.21 MB/sec warmup 19 sec latency 12212.929 ms 1 1134 0.00 MB/sec execute 1 sec latency 14213.353 ms 1 1134 0.00 MB/sec execute 2 sec latency 15213.582 ms 1 1134 0.00 MB/sec execute 3 sec latency 16213.837 ms Failover mds1 to oleg131-server mount facets: mds1 1 1134 0.00 MB/sec execute 4 sec latency 17214.085 ms 1 1134 0.00 MB/sec execute 5 sec latency 18214.241 ms 1 1134 0.00 MB/sec execute 6 sec latency 19214.444 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1134 0.00 MB/sec execute 7 sec latency 20214.648 ms oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 1 1134 0.00 MB/sec execute 8 sec latency 21214.830 ms pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 21:52:03 (1716256323) targets are mounted 21:52:03 (1716256323) facet_failover done 1 1134 0.00 MB/sec execute 9 sec latency 22215.041 ms 1 1134 0.00 MB/sec execute 10 sec latency 23215.286 ms 1 1134 0.00 MB/sec execute 11 sec latency 24215.474 ms 1 1134 0.00 MB/sec execute 12 sec latency 25215.689 ms 1 1143 0.00 MB/sec execute 13 sec latency 26157.920 ms oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1406 0.22 MB/sec execute 14 sec latency 24.327 ms 1 1575 0.21 MB/sec execute 15 sec latency 22.414 ms 1 1755 0.21 MB/sec execute 16 sec latency 26.597 ms 1 1963 0.20 MB/sec execute 17 sec latency 22.930 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2796 1284892 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36592 3566868 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9216 3595052 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 45808 7161920 1% /mnt/lustre 1 2145 0.21 MB/sec execute 18 sec latency 22.554 ms 1 2368 0.26 MB/sec execute 19 sec latency 21.059 ms 1 2457 0.26 MB/sec execute 20 sec latency 377.173 ms 1 2876 0.44 MB/sec execute 21 sec latency 25.284 ms 1 3263 0.56 MB/sec execute 22 sec latency 20.435 ms test_26 fail mds2 2 times Failing mds2 on oleg131-server 1 3665 0.68 MB/sec execute 23 sec latency 15.575 ms Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 1 3760 0.70 MB/sec execute 24 sec latency 655.848 ms 21:52:19 (1716256339) shut down 1 3760 0.67 MB/sec execute 25 sec latency 1656.013 ms 1 3760 0.65 MB/sec execute 26 sec latency 2656.198 ms 1 3760 0.62 MB/sec execute 27 sec latency 3656.394 ms 1 3760 0.60 MB/sec execute 28 sec latency 4656.608 ms 1 3760 0.58 MB/sec execute 29 sec latency 5656.760 ms 1 3760 0.56 MB/sec execute 30 sec latency 6656.873 ms 1 3760 0.54 MB/sec execute 31 sec latency 7657.044 ms 1 3760 0.52 MB/sec execute 32 sec latency 8657.190 ms 1 3760 0.51 MB/sec execute 33 sec latency 9657.353 ms 1 3760 0.49 MB/sec execute 34 sec latency 10657.504 ms Failover mds2 to oleg131-server mount facets: mds2 1 3760 0.48 MB/sec execute 35 sec latency 11657.786 ms 1 3760 0.47 MB/sec execute 36 sec latency 12657.916 ms 1 3760 0.45 MB/sec execute 37 sec latency 13658.201 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 3760 0.44 MB/sec execute 38 sec latency 14658.358 ms oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all 1 3760 0.43 MB/sec execute 39 sec latency 15658.517 ms pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 21:52:34 (1716256354) targets are mounted 21:52:34 (1716256354) facet_failover done 1 3760 0.42 MB/sec execute 40 sec latency 16658.683 ms 1 3760 0.41 MB/sec execute 41 sec latency 17658.819 ms 1 3760 0.40 MB/sec execute 42 sec latency 18658.954 ms 1 3760 0.39 MB/sec execute 43 sec latency 19659.115 ms oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 3826 0.38 MB/sec execute 44 sec latency 20187.279 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3960 0.38 MB/sec execute 45 sec latency 23.416 ms 1 4095 0.37 MB/sec execute 46 sec latency 23.889 ms 1 4318 0.37 MB/sec execute 47 sec latency 39.265 ms 1 4511 0.37 MB/sec execute 48 sec latency 21.752 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4032 1283656 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2652 1285036 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39724 3554980 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12728 3579484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 52452 7134464 1% /mnt/lustre 1 4734 0.39 MB/sec execute 49 sec latency 35.832 ms 1 4985 0.44 MB/sec execute 50 sec latency 24.112 ms 1 5153 0.44 MB/sec execute 51 sec latency 30.128 ms 1 5327 0.43 MB/sec execute 52 sec latency 26.479 ms test_26 fail mds1 3 times 1 5515 0.43 MB/sec execute 53 sec latency 36.413 ms Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 1 5716 0.42 MB/sec execute 54 sec latency 35.871 ms 21:52:49 (1716256369) shut down 1 5716 0.42 MB/sec execute 55 sec latency 1036.089 ms 1 5716 0.41 MB/sec execute 56 sec latency 2036.285 ms 1 5716 0.40 MB/sec execute 57 sec latency 3036.517 ms 1 5716 0.39 MB/sec execute 58 sec latency 4036.672 ms 1 5716 0.39 MB/sec execute 59 sec latency 5036.859 ms 1 5716 0.38 MB/sec execute 60 sec latency 6037.027 ms 1 5716 0.37 MB/sec execute 61 sec latency 7037.208 ms 1 5716 0.37 MB/sec execute 62 sec latency 8037.406 ms 1 5716 0.36 MB/sec execute 63 sec latency 9037.641 ms 1 5716 0.36 MB/sec execute 64 sec latency 10037.881 ms Failover mds1 to oleg131-server mount facets: mds1 1 5716 0.35 MB/sec execute 65 sec latency 11038.081 ms 1 5716 0.35 MB/sec execute 66 sec latency 12038.258 ms 1 5716 0.34 MB/sec execute 67 sec latency 13038.441 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5716 0.34 MB/sec execute 68 sec latency 14038.600 ms 1 5716 0.33 MB/sec execute 69 sec latency 15038.848 ms oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 1 5716 0.33 MB/sec execute 70 sec latency 16039.063 ms Started lustre-MDT0000 21:53:05 (1716256385) targets are mounted 21:53:05 (1716256385) facet_failover done 1 5716 0.32 MB/sec execute 71 sec latency 17039.227 ms 1 5716 0.32 MB/sec execute 72 sec latency 18039.415 ms 1 5716 0.31 MB/sec execute 73 sec latency 19039.668 ms 1 5716 0.31 MB/sec execute 74 sec latency 20039.843 ms 1 5716 0.30 MB/sec execute 75 sec latency 21040.043 ms 1 5716 0.30 MB/sec execute 76 sec latency 22040.172 ms oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5945 0.31 MB/sec execute 77 sec latency 22095.868 ms 1 6140 0.32 MB/sec execute 78 sec latency 24.549 ms 1 6486 0.37 MB/sec execute 79 sec latency 23.925 ms 1 6860 0.39 MB/sec execute 80 sec latency 24.796 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4328 1283360 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2564 1285124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 43960 3551864 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18896 3582524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 62856 7134388 1% /mnt/lustre 1 7164 0.43 MB/sec execute 81 sec latency 22.952 ms 1 7336 0.44 MB/sec execute 82 sec latency 24.258 ms 1 7467 0.44 MB/sec execute 83 sec latency 25.537 ms 1 7595 0.43 MB/sec execute 84 sec latency 24.102 ms 1 7785 0.43 MB/sec execute 85 sec latency 43.846 ms test_26 fail mds2 4 times Failing mds2 on oleg131-server 1 7960 0.43 MB/sec execute 86 sec latency 28.069 ms Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 1 8106 0.43 MB/sec execute 87 sec latency 70.486 ms 21:53:22 (1716256402) shut down 1 8106 0.42 MB/sec execute 88 sec latency 1070.659 ms 1 8106 0.42 MB/sec execute 89 sec latency 2070.917 ms 1 8106 0.41 MB/sec execute 90 sec latency 3071.161 ms 1 8106 0.41 MB/sec execute 91 sec latency 4071.370 ms 1 8106 0.40 MB/sec execute 92 sec latency 5071.602 ms 1 8106 0.40 MB/sec execute 93 sec latency 6071.881 ms 1 8106 0.40 MB/sec execute 94 sec latency 7072.151 ms 1 8106 0.39 MB/sec execute 95 sec latency 8072.386 ms 1 8106 0.39 MB/sec execute 96 sec latency 9072.613 ms 1 8106 0.38 MB/sec execute 97 sec latency 10072.867 ms Failover mds2 to oleg131-server mount facets: mds2 1 8106 0.38 MB/sec execute 98 sec latency 11073.084 ms 1 8106 0.38 MB/sec execute 99 sec latency 12073.280 ms 1 cleanup 100 sec Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 cleanup 101 sec 1 cleanup 102 sec oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 21:53:37 (1716256417) targets are mounted 21:53:37 (1716256417) facet_failover done 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 cleanup 107 sec 0 cleanup 107 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1211 64.703 22095.767 Close 883 2.732 5.555 Rename 51 530.721 26157.867 Unlink 246 6.613 22.173 Qpathinfo 1106 3.720 24.823 Qfileinfo 191 0.617 4.243 Qfsinfo 195 1.202 6.725 Sfileinfo 96 10.009 18.457 Find 422 3.975 16.676 WriteX 585 3.050 19.725 ReadX 1893 0.176 17.088 LockX 4 2.957 3.113 UnlockX 4 3.037 3.216 Flush 86 12.630 43.834 Throughput 0.375297 MB/sec 1 clients 1 procs max_latency=26157.920 ms stopping dbench on /mnt/lustre at Mon May 20 21:53:42 EDT 2024 with return code 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon May 20 21:53:44 EDT 2024 waiting for dbench pid 10878 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 158 6.54 MB/sec warmup 1 sec latency 36.409 ms 1 349 6.26 MB/sec warmup 2 sec latency 27.737 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4696 1282992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2696 1284992 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7268 3594628 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 34764 3530144 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42032 7124772 1% /mnt/lustre 1 503 5.82 MB/sec warmup 3 sec latency 31.013 ms 1 712 5.30 MB/sec warmup 4 sec latency 119.260 ms 1 848 4.31 MB/sec warmup 5 sec latency 388.884 ms 1 985 3.65 MB/sec warmup 6 sec latency 32.329 ms test_26 fail mds1 5 times 1 1195 3.29 MB/sec warmup 7 sec latency 23.439 ms Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 1 1439 3.25 MB/sec warmup 8 sec latency 83.441 ms 1 1439 2.89 MB/sec warmup 9 sec latency 1083.585 ms 1 1439 2.60 MB/sec warmup 10 sec latency 2083.756 ms 1 1439 2.36 MB/sec warmup 11 sec latency 3084.008 ms 1 1439 2.17 MB/sec warmup 12 sec latency 4084.247 ms 1 1439 2.00 MB/sec warmup 13 sec latency 5084.445 ms 21:53:58 (1716256438) shut down 1 1439 1.86 MB/sec warmup 14 sec latency 6084.627 ms 1 1439 1.73 MB/sec warmup 15 sec latency 7084.818 ms 1 1439 1.63 MB/sec warmup 16 sec latency 8085.010 ms 1 1439 1.53 MB/sec warmup 17 sec latency 9085.243 ms 1 1439 1.45 MB/sec warmup 18 sec latency 10085.432 ms 1 1439 1.37 MB/sec warmup 19 sec latency 11085.630 ms 1 1439 0.00 MB/sec execute 1 sec latency 13085.962 ms 1 1439 0.00 MB/sec execute 2 sec latency 14086.168 ms 1 1439 0.00 MB/sec execute 3 sec latency 15086.358 ms Failover mds1 to oleg131-server mount facets: mds1 1 1439 0.00 MB/sec execute 4 sec latency 16086.552 ms 1 1439 0.00 MB/sec execute 5 sec latency 17086.721 ms 1 1439 0.00 MB/sec execute 6 sec latency 18086.918 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1439 0.00 MB/sec execute 7 sec latency 19087.113 ms 1 1439 0.00 MB/sec execute 8 sec latency 20087.279 ms oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 1 1439 0.00 MB/sec execute 9 sec latency 21087.452 ms Started lustre-MDT0000 21:54:13 (1716256453) targets are mounted 21:54:13 (1716256453) facet_failover done 1 1439 0.00 MB/sec execute 10 sec latency 22087.638 ms 1 1439 0.00 MB/sec execute 11 sec latency 23087.831 ms 1 1439 0.00 MB/sec execute 12 sec latency 24088.098 ms 1 1439 0.00 MB/sec execute 13 sec latency 25088.254 ms oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1545 0.01 MB/sec execute 14 sec latency 25466.764 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1729 0.02 MB/sec execute 15 sec latency 26.868 ms 1 1954 0.03 MB/sec execute 16 sec latency 19.529 ms 1 2156 0.04 MB/sec execute 17 sec latency 17.280 ms 1 2377 0.11 MB/sec execute 18 sec latency 20.784 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5164 1282524 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3200 1284488 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15448 3590100 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 63276 3542000 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 78724 7132100 2% /mnt/lustre 1 2571 0.15 MB/sec execute 19 sec latency 23.967 ms 1 2922 0.37 MB/sec execute 20 sec latency 49.650 ms 1 3015 0.36 MB/sec execute 21 sec latency 604.390 ms 1 3432 0.48 MB/sec execute 22 sec latency 24.336 ms test_26 fail mds2 6 times 1 3696 0.55 MB/sec execute 23 sec latency 23.469 ms Failing mds2 on oleg131-server Stopping /mnt/lustre-mds2 (opts:) on oleg131-server 1 3857 0.57 MB/sec execute 24 sec latency 23.240 ms 21:54:29 (1716256469) shut down 1 3884 0.55 MB/sec execute 25 sec latency 817.666 ms 1 3884 0.53 MB/sec execute 26 sec latency 1817.898 ms 1 3884 0.51 MB/sec execute 27 sec latency 2818.112 ms 1 3884 0.49 MB/sec execute 28 sec latency 3818.352 ms 1 3884 0.47 MB/sec execute 29 sec latency 4818.610 ms 1 3884 0.46 MB/sec execute 30 sec latency 5818.839 ms 1 3884 0.44 MB/sec execute 31 sec latency 6819.060 ms 1 3884 0.43 MB/sec execute 32 sec latency 7819.263 ms 1 3884 0.42 MB/sec execute 33 sec latency 8819.498 ms 1 3884 0.40 MB/sec execute 34 sec latency 9819.732 ms Failover mds2 to oleg131-server 1 3884 0.39 MB/sec execute 35 sec latency 10819.900 ms mount facets: mds2 1 3884 0.38 MB/sec execute 36 sec latency 11820.068 ms 1 3884 0.37 MB/sec execute 37 sec latency 12820.248 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 3884 0.36 MB/sec execute 38 sec latency 13820.425 ms 1 3884 0.35 MB/sec execute 39 sec latency 14820.643 ms oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 1 3884 0.34 MB/sec execute 40 sec latency 15820.830 ms Started lustre-MDT0001 21:54:44 (1716256484) targets are mounted 21:54:44 (1716256484) facet_failover done 1 3884 0.34 MB/sec execute 41 sec latency 16821.009 ms 1 3884 0.33 MB/sec execute 42 sec latency 17821.247 ms 1 3884 0.32 MB/sec execute 43 sec latency 18821.440 ms 1 3884 0.31 MB/sec execute 44 sec latency 19821.648 ms oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 3949 0.31 MB/sec execute 45 sec latency 20354.060 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4076 0.31 MB/sec execute 46 sec latency 21.878 ms 1 4236 0.30 MB/sec execute 47 sec latency 41.561 ms 1 4393 0.30 MB/sec execute 48 sec latency 42.451 ms 1 4548 0.31 MB/sec execute 49 sec latency 26.493 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5344 1282344 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3408 1284280 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15804 3586928 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 64660 3539040 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 80464 7125968 2% /mnt/lustre 1 4759 0.32 MB/sec execute 50 sec latency 24.664 ms 1 5013 0.37 MB/sec execute 51 sec latency 25.919 ms 1 5123 0.37 MB/sec execute 52 sec latency 530.339 ms 1 5320 0.37 MB/sec execute 53 sec latency 23.752 ms 1 5547 0.36 MB/sec execute 54 sec latency 23.862 ms test_26 fail mds1 7 times Failing mds1 on oleg131-server Stopping /mnt/lustre-mds1 (opts:) on oleg131-server 1 5806 0.36 MB/sec execute 55 sec latency 14.341 ms 1 6007 0.38 MB/sec execute 56 sec latency 30.241 ms 1 6339 0.44 MB/sec execute 57 sec latency 24.357 ms 1 6767 0.47 MB/sec execute 58 sec latency 25.348 ms 1 7109 0.53 MB/sec execute 59 sec latency 18.910 ms 1 7323 0.55 MB/sec execute 60 sec latency 24.839 ms 1 7473 0.54 MB/sec execute 61 sec latency 24.007 ms 21:55:06 (1716256506) shut down 1 7627 0.54 MB/sec execute 62 sec latency 46.928 ms 1 7627 0.53 MB/sec execute 63 sec latency 1047.068 ms 1 7627 0.52 MB/sec execute 64 sec latency 2047.242 ms 1 7627 0.51 MB/sec execute 65 sec latency 3047.419 ms 1 7627 0.50 MB/sec execute 66 sec latency 4047.662 ms 1 7627 0.50 MB/sec execute 67 sec latency 5047.859 ms 1 7627 0.49 MB/sec execute 68 sec latency 6048.037 ms 1 7627 0.48 MB/sec execute 69 sec latency 7048.238 ms 1 7627 0.47 MB/sec execute 70 sec latency 8048.608 ms 1 7627 0.47 MB/sec execute 71 sec latency 9048.876 ms Failover mds1 to oleg131-server mount facets: mds1 1 7627 0.46 MB/sec execute 72 sec latency 10049.094 ms 1 7627 0.45 MB/sec execute 73 sec latency 11049.309 ms 1 7627 0.45 MB/sec execute 74 sec latency 12049.546 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.44 MB/sec execute 75 sec latency 13049.753 ms 1 7627 0.44 MB/sec execute 76 sec latency 14050.031 ms oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 1 7627 0.43 MB/sec execute 77 sec latency 15050.245 ms Started lustre-MDT0000 21:55:21 (1716256521) targets are mounted 21:55:21 (1716256521) facet_failover done 1 7627 0.43 MB/sec execute 78 sec latency 16050.588 ms 1 7627 0.42 MB/sec execute 79 sec latency 17050.884 ms 1 7627 0.41 MB/sec execute 80 sec latency 18051.119 ms 1 7627 0.41 MB/sec execute 81 sec latency 19051.359 ms oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7777 0.41 MB/sec execute 82 sec latency 19319.371 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7933 0.41 MB/sec execute 83 sec latency 29.884 ms 1 8096 0.41 MB/sec execute 84 sec latency 33.722 ms 1 8301 0.41 MB/sec execute 85 sec latency 23.934 ms 1 8556 0.44 MB/sec execute 86 sec latency 24.465 ms 1 8696 0.44 MB/sec execute 87 sec latency 45.770 ms 1 8878 0.44 MB/sec execute 88 sec latency 26.908 ms 1 9074 0.43 MB/sec execute 89 sec latency 32.272 ms 1 9270 0.43 MB/sec execute 90 sec latency 20.930 ms 1 9482 0.44 MB/sec execute 91 sec latency 44.407 ms 1 9676 0.45 MB/sec execute 92 sec latency 24.728 ms 1 10028 0.49 MB/sec execute 93 sec latency 30.755 ms 1 10376 0.51 MB/sec execute 94 sec latency 40.240 ms 1 10673 0.53 MB/sec execute 95 sec latency 19.309 ms 1 10867 0.54 MB/sec execute 96 sec latency 23.352 ms 1 11008 0.54 MB/sec execute 97 sec latency 25.242 ms 1 11132 0.53 MB/sec execute 98 sec latency 25.229 ms 1 11315 0.53 MB/sec execute 99 sec latency 54.645 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1753 24.963 20354.003 Close 1269 22.924 25466.749 Rename 70 19.329 36.563 Unlink 360 7.236 44.459 Qpathinfo 1604 3.724 54.625 Qfileinfo 277 0.627 5.038 Qfsinfo 287 1.336 9.398 Sfileinfo 126 11.035 30.217 Find 611 35.179 19319.351 WriteX 853 3.194 12.402 ReadX 2724 0.104 2.780 LockX 6 2.955 4.100 UnlockX 6 2.753 3.040 Flush 123 22.296 604.376 Throughput 0.529251 MB/sec 1 clients 1 procs max_latency=25466.764 ms stopping dbench on /mnt/lustre at Mon May 20 21:55:45 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon May 20 21:55:47 EDT 2024 waiting for dbench pid 14355 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 151 6.35 MB/sec warmup 1 sec latency 37.872 ms 1 294 5.42 MB/sec warmup 2 sec latency 34.117 ms 1 445 5.21 MB/sec warmup 3 sec latency 29.939 ms 1 669 5.29 MB/sec warmup 4 sec latency 27.261 ms 1 845 4.31 MB/sec warmup 5 sec latency 23.756 ms 1 980 3.65 MB/sec warmup 6 sec latency 56.422 ms 1 1193 3.29 MB/sec warmup 7 sec latency 26.241 ms 1 1447 3.25 MB/sec warmup 8 sec latency 25.890 ms 1 1632 2.91 MB/sec warmup 9 sec latency 22.289 ms 1 1838 2.64 MB/sec warmup 10 sec latency 28.752 ms 1 2030 2.42 MB/sec warmup 11 sec latency 48.768 ms 1 2251 2.23 MB/sec warmup 12 sec latency 16.288 ms 1 2440 2.16 MB/sec warmup 13 sec latency 41.975 ms 1 2746 2.28 MB/sec warmup 14 sec latency 31.135 ms 1 3012 2.24 MB/sec warmup 15 sec latency 29.978 ms 1 3378 2.22 MB/sec warmup 16 sec latency 26.193 ms 1 3685 2.28 MB/sec warmup 17 sec latency 18.595 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon May 20 21:56:05 EDT 2024 with return code 0 14355 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 14355 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished