== replay-dual test 26: dbench and tar with mds failover ========================================================== 19:18:10 (1736986690) Starting client oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre Started clients oleg210-client.virtnet: 192.168.202.110@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15898 Started dbench loop with 15899 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jan 15 19:18:12 EST 2025 waiting for dbench pid 15925 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 295 10.90 MB/sec warmup 1 sec latency 18.070 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3641344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3707904 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7349248 1% /mnt/lustre 1 652 10.53 MB/sec warmup 2 sec latency 27.004 ms 1 916 7.26 MB/sec warmup 3 sec latency 59.021 ms 1 1405 6.50 MB/sec warmup 4 sec latency 21.542 ms test_26 fail mds1 1 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 1 1698 5.25 MB/sec warmup 5 sec latency 127.903 ms reboot facets: mds1 1 1698 4.37 MB/sec warmup 6 sec latency 1128.061 ms 1 1698 3.75 MB/sec warmup 7 sec latency 2128.323 ms 1 1698 3.28 MB/sec warmup 8 sec latency 3128.509 ms 1 1698 2.91 MB/sec warmup 9 sec latency 4128.733 ms 1 1698 2.62 MB/sec warmup 10 sec latency 5128.886 ms 1 1698 2.38 MB/sec warmup 11 sec latency 6129.060 ms 1 1698 2.19 MB/sec warmup 12 sec latency 7129.260 ms 1 1698 2.02 MB/sec warmup 13 sec latency 8129.514 ms 1 1698 1.87 MB/sec warmup 14 sec latency 9129.739 ms 1 1698 1.75 MB/sec warmup 15 sec latency 10129.909 ms Failover mds1 to oleg210-server mount facets: mds1 1 1698 1.64 MB/sec warmup 16 sec latency 11130.087 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1698 1.54 MB/sec warmup 17 sec latency 12130.286 ms 1 1698 1.46 MB/sec warmup 18 sec latency 13130.488 ms 1 1698 1.38 MB/sec warmup 19 sec latency 14130.729 ms oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 1 1698 0.00 MB/sec execute 1 sec latency 16130.930 ms Started lustre-MDT0000 1 1698 0.00 MB/sec execute 2 sec latency 17131.226 ms 1 1698 0.00 MB/sec execute 3 sec latency 18131.439 ms 1 1698 0.00 MB/sec execute 4 sec latency 19131.655 ms 1 1698 0.00 MB/sec execute 5 sec latency 20131.903 ms 1 1698 0.00 MB/sec execute 6 sec latency 21132.131 ms 1 1698 0.00 MB/sec execute 7 sec latency 22132.294 ms 1 1698 0.00 MB/sec execute 8 sec latency 23132.451 ms 1 1698 0.00 MB/sec execute 9 sec latency 24132.651 ms 1 1698 0.00 MB/sec execute 10 sec latency 25132.919 ms 1 1698 0.00 MB/sec execute 11 sec latency 26133.156 ms 1 1698 0.00 MB/sec execute 12 sec latency 27133.301 ms 1 1698 0.00 MB/sec execute 13 sec latency 28133.506 ms 1 1698 0.00 MB/sec execute 14 sec latency 29133.657 ms 1 1698 0.00 MB/sec execute 15 sec latency 30133.811 ms 1 1855 0.01 MB/sec execute 16 sec latency 30666.021 ms oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2198 0.03 MB/sec execute 17 sec latency 42.525 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2474 0.13 MB/sec execute 18 sec latency 58.231 ms 1 2987 0.38 MB/sec execute 19 sec latency 37.949 ms 1 3543 0.61 MB/sec execute 20 sec latency 33.159 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4992 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3661824 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3689472 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 60416 7351296 1% /mnt/lustre 1 3838 0.64 MB/sec execute 21 sec latency 17.193 ms 1 4068 0.63 MB/sec execute 22 sec latency 16.588 ms 1 4349 0.62 MB/sec execute 23 sec latency 53.388 ms test_26 fail mds1 2 times 1 4619 0.66 MB/sec execute 24 sec latency 36.930 ms Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 1 4946 0.76 MB/sec execute 25 sec latency 279.420 ms 1 4946 0.73 MB/sec execute 26 sec latency 1279.606 ms 1 4946 0.70 MB/sec execute 27 sec latency 2279.798 ms 1 4946 0.67 MB/sec execute 28 sec latency 3279.968 ms 1 4946 0.65 MB/sec execute 29 sec latency 4280.140 ms 1 4946 0.63 MB/sec execute 30 sec latency 5280.303 ms 1 4946 0.61 MB/sec execute 31 sec latency 6280.501 ms reboot facets: mds1 1 4946 0.59 MB/sec execute 32 sec latency 7280.706 ms 1 4946 0.57 MB/sec execute 33 sec latency 8280.906 ms 1 4946 0.56 MB/sec execute 34 sec latency 9281.081 ms 1 4946 0.54 MB/sec execute 35 sec latency 10281.265 ms 1 4946 0.52 MB/sec execute 36 sec latency 11281.490 ms 1 4946 0.51 MB/sec execute 37 sec latency 12281.697 ms 1 4946 0.50 MB/sec execute 38 sec latency 13281.871 ms 1 4946 0.48 MB/sec execute 39 sec latency 14282.000 ms 1 4946 0.47 MB/sec execute 40 sec latency 15282.134 ms 1 4946 0.46 MB/sec execute 41 sec latency 16282.317 ms Failover mds1 to oleg210-server mount facets: mds1 1 4946 0.45 MB/sec execute 42 sec latency 17282.521 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4946 0.44 MB/sec execute 43 sec latency 18282.709 ms 1 4946 0.43 MB/sec execute 44 sec latency 19282.906 ms 1 4946 0.42 MB/sec execute 45 sec latency 20283.081 ms 1 4946 0.41 MB/sec execute 46 sec latency 21283.307 ms 1 4946 0.40 MB/sec execute 47 sec latency 22283.497 ms 1 4946 0.39 MB/sec execute 48 sec latency 23283.654 ms 1 4946 0.39 MB/sec execute 49 sec latency 24283.809 ms oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4946 0.38 MB/sec execute 50 sec latency 25283.969 ms 1 4946 0.37 MB/sec execute 51 sec latency 26284.099 ms 1 4947 0.36 MB/sec execute 52 sec latency 27266.724 ms 1 5218 0.36 MB/sec execute 53 sec latency 54.536 ms oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5565 0.36 MB/sec execute 54 sec latency 37.252 ms 1 5937 0.38 MB/sec execute 55 sec latency 41.103 ms 1 6337 0.44 MB/sec execute 56 sec latency 33.426 ms 1 6860 0.50 MB/sec execute 57 sec latency 41.609 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6784 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3553280 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3569664 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7122944 1% /mnt/lustre 1 7241 0.55 MB/sec execute 58 sec latency 20.414 ms 1 7464 0.56 MB/sec execute 59 sec latency 21.062 ms 1 7681 0.55 MB/sec execute 60 sec latency 18.585 ms test_26 fail mds1 3 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 1 7902 0.55 MB/sec execute 61 sec latency 73.300 ms reboot facets: mds1 1 7962 0.54 MB/sec execute 62 sec latency 765.851 ms 1 7962 0.53 MB/sec execute 63 sec latency 1766.058 ms 1 7962 0.52 MB/sec execute 64 sec latency 2766.265 ms 1 7962 0.52 MB/sec execute 65 sec latency 3766.437 ms 1 7962 0.51 MB/sec execute 66 sec latency 4766.605 ms 1 7962 0.50 MB/sec execute 67 sec latency 5766.760 ms 1 7962 0.49 MB/sec execute 68 sec latency 6766.945 ms 1 7962 0.49 MB/sec execute 69 sec latency 7767.106 ms 1 7962 0.48 MB/sec execute 70 sec latency 8767.287 ms 1 7962 0.47 MB/sec execute 71 sec latency 9767.515 ms Failover mds1 to oleg210-server 1 7962 0.47 MB/sec execute 72 sec latency 10767.630 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7962 0.46 MB/sec execute 73 sec latency 11767.746 ms 1 7962 0.45 MB/sec execute 74 sec latency 12767.930 ms 1 7962 0.45 MB/sec execute 75 sec latency 13768.167 ms 1 7962 0.44 MB/sec execute 76 sec latency 14768.450 ms 1 7962 0.44 MB/sec execute 77 sec latency 15768.651 ms 1 7962 0.43 MB/sec execute 78 sec latency 16768.839 ms 1 7962 0.43 MB/sec execute 79 sec latency 17769.036 ms 1 7962 0.42 MB/sec execute 80 sec latency 18769.272 ms oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 1 7962 0.41 MB/sec execute 81 sec latency 19769.579 ms 1 7962 0.41 MB/sec execute 82 sec latency 20769.770 ms 1 7962 0.40 MB/sec execute 83 sec latency 21769.963 ms 1 7962 0.40 MB/sec execute 84 sec latency 22770.136 ms 1 7962 0.40 MB/sec execute 85 sec latency 23770.326 ms 1 7962 0.39 MB/sec execute 86 sec latency 24770.493 ms 1 7962 0.39 MB/sec execute 87 sec latency 25770.652 ms 1 7962 0.38 MB/sec execute 88 sec latency 26770.837 ms 1 7962 0.38 MB/sec execute 89 sec latency 27771.043 ms 1 7962 0.37 MB/sec execute 90 sec latency 28771.218 ms 1 7962 0.37 MB/sec execute 91 sec latency 29771.422 ms 1 7962 0.36 MB/sec execute 92 sec latency 30771.617 ms 1 7962 0.36 MB/sec execute 93 sec latency 31771.795 ms 1 8066 0.36 MB/sec execute 94 sec latency 32312.458 ms oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 8382 0.37 MB/sec execute 95 sec latency 28.805 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8742 0.40 MB/sec execute 96 sec latency 49.723 ms 1 9076 0.40 MB/sec execute 97 sec latency 45.064 ms 1 9449 0.41 MB/sec execute 98 sec latency 22.427 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8448 2200064 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3697664 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3731456 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7429120 1% /mnt/lustre 1 9723 0.41 MB/sec execute 99 sec latency 42.656 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1445 27.584 27266.709 Close 1077 31.624 32312.437 Rename 63 11.614 19.571 Unlink 275 4.346 14.103 Qpathinfo 1379 24.458 30665.989 Qfileinfo 250 0.367 2.593 Qfsinfo 235 0.403 2.586 Sfileinfo 116 5.515 9.119 Find 506 1.100 24.161 WriteX 748 1.890 8.635 ReadX 2470 0.060 28.640 LockX 6 1.636 2.186 UnlockX 6 1.517 1.801 Flush 98 25.713 73.288 Throughput 0.414946 MB/sec 1 clients 1 procs max_latency=32312.458 ms stopping dbench on /mnt/lustre at Wed Jan 15 19:20:12 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jan 15 19:20:14 EST 2025 waiting for dbench pid 19418 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec test_26 fail mds1 4 times 1 of 1 processes prepared for launch 0 sec releasing clients Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 1 215 8.36 MB/sec warmup 1 sec latency 178.146 ms reboot facets: mds1 1 215 4.18 MB/sec warmup 2 sec latency 1178.332 ms 1 215 2.79 MB/sec warmup 3 sec latency 2178.485 ms 1 215 2.09 MB/sec warmup 4 sec latency 3178.614 ms 1 215 1.67 MB/sec warmup 5 sec latency 4178.758 ms 1 215 1.39 MB/sec warmup 6 sec latency 5178.903 ms 1 215 1.19 MB/sec warmup 7 sec latency 6179.053 ms 1 215 1.05 MB/sec warmup 8 sec latency 7179.207 ms 1 215 0.93 MB/sec warmup 9 sec latency 8179.323 ms 1 215 0.84 MB/sec warmup 10 sec latency 9179.458 ms 1 215 0.76 MB/sec warmup 11 sec latency 10179.595 ms Failover mds1 to oleg210-server mount facets: mds1 1 215 0.70 MB/sec warmup 12 sec latency 11179.730 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 215 0.64 MB/sec warmup 13 sec latency 12179.869 ms 1 215 0.60 MB/sec warmup 14 sec latency 13180.017 ms 1 215 0.56 MB/sec warmup 15 sec latency 14180.186 ms 1 215 0.52 MB/sec warmup 16 sec latency 15180.303 ms oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 1 215 0.49 MB/sec warmup 17 sec latency 16180.405 ms 1 279 0.58 MB/sec warmup 18 sec latency 16944.248 ms 1 525 0.95 MB/sec warmup 19 sec latency 20.557 ms oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1121 1.49 MB/sec execute 1 sec latency 47.664 ms 1 1497 2.29 MB/sec execute 2 sec latency 40.025 ms 1 1873 1.63 MB/sec execute 3 sec latency 32.815 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10112 2198400 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3770368 43008 3368960 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3393536 1% /mnt/lustre[OST:1] filesystem_summary: 7541760 68608 6762496 2% /mnt/lustre 1 2334 1.59 MB/sec execute 4 sec latency 34.438 ms 1 2770 2.10 MB/sec execute 5 sec latency 37.096 ms 1 3374 2.33 MB/sec execute 6 sec latency 28.091 ms test_26 fail mds1 5 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 1 3811 2.61 MB/sec execute 7 sec latency 21.273 ms reboot facets: mds1 1 3855 2.29 MB/sec execute 8 sec latency 766.461 ms 1 3855 2.03 MB/sec execute 9 sec latency 1766.622 ms 1 3855 1.83 MB/sec execute 10 sec latency 2766.774 ms 1 3855 1.66 MB/sec execute 11 sec latency 3766.974 ms 1 3855 1.52 MB/sec execute 12 sec latency 4767.150 ms 1 3855 1.41 MB/sec execute 13 sec latency 5767.314 ms 1 3855 1.31 MB/sec execute 14 sec latency 6767.489 ms 1 3855 1.22 MB/sec execute 15 sec latency 7767.660 ms 1 3855 1.14 MB/sec execute 16 sec latency 8767.821 ms 1 3855 1.08 MB/sec execute 17 sec latency 9767.985 ms Failover mds1 to oleg210-server 1 3855 1.02 MB/sec execute 18 sec latency 10768.110 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3855 0.96 MB/sec execute 19 sec latency 11768.263 ms 1 3855 0.91 MB/sec execute 20 sec latency 12768.446 ms 1 3855 0.87 MB/sec execute 21 sec latency 13768.628 ms 1 3855 0.83 MB/sec execute 22 sec latency 14768.800 ms 1 3855 0.80 MB/sec execute 23 sec latency 15769.012 ms 1 3855 0.76 MB/sec execute 24 sec latency 16769.144 ms oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 1 3855 0.73 MB/sec execute 25 sec latency 17769.249 ms 1 3855 0.70 MB/sec execute 26 sec latency 18769.359 ms 1 3855 0.68 MB/sec execute 27 sec latency 19769.527 ms 1 3855 0.65 MB/sec execute 28 sec latency 20769.667 ms 1 3855 0.63 MB/sec execute 29 sec latency 21769.913 ms 1 3855 0.61 MB/sec execute 30 sec latency 22770.048 ms 1 3855 0.59 MB/sec execute 31 sec latency 23770.237 ms 1 3855 0.57 MB/sec execute 32 sec latency 24770.372 ms 1 3855 0.55 MB/sec execute 33 sec latency 25770.595 ms 1 3855 0.54 MB/sec execute 34 sec latency 26770.761 ms 1 3855 0.52 MB/sec execute 35 sec latency 27770.979 ms 1 3855 0.51 MB/sec execute 36 sec latency 28771.089 ms 1 3865 0.49 MB/sec execute 37 sec latency 29726.153 ms oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4151 0.49 MB/sec execute 38 sec latency 13.982 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4433 0.49 MB/sec execute 39 sec latency 51.558 ms 1 4751 0.52 MB/sec execute 40 sec latency 43.177 ms 1 5152 0.58 MB/sec execute 41 sec latency 61.224 ms 1 5504 0.57 MB/sec execute 42 sec latency 53.649 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11136 2197376 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 55296 3680256 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 35840 3709952 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 91136 7390208 2% /mnt/lustre 1 5916 0.59 MB/sec execute 43 sec latency 27.524 ms 1 6321 0.67 MB/sec execute 44 sec latency 68.186 ms 1 6875 0.73 MB/sec execute 45 sec latency 45.665 ms test_26 fail mds1 6 times Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 1 7283 0.80 MB/sec execute 46 sec latency 70.544 ms reboot facets: mds1 1 7283 0.78 MB/sec execute 47 sec latency 1070.803 ms 1 7283 0.77 MB/sec execute 48 sec latency 2071.025 ms 1 7283 0.75 MB/sec execute 49 sec latency 3071.890 ms 1 7283 0.74 MB/sec execute 50 sec latency 4072.107 ms 1 7283 0.72 MB/sec execute 51 sec latency 5072.288 ms 1 7283 0.71 MB/sec execute 52 sec latency 6072.460 ms 1 7283 0.70 MB/sec execute 53 sec latency 7072.684 ms 1 7283 0.68 MB/sec execute 54 sec latency 8072.845 ms 1 7283 0.67 MB/sec execute 55 sec latency 9073.067 ms 1 7283 0.66 MB/sec execute 56 sec latency 10073.268 ms Failover mds1 to oleg210-server mount facets: mds1 1 7283 0.65 MB/sec execute 57 sec latency 11073.393 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7283 0.64 MB/sec execute 58 sec latency 12073.576 ms 1 7283 0.62 MB/sec execute 59 sec latency 13073.789 ms 1 7283 0.61 MB/sec execute 60 sec latency 14073.968 ms 1 7283 0.60 MB/sec execute 61 sec latency 15074.148 ms 1 7283 0.59 MB/sec execute 62 sec latency 16074.327 ms oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 1 7283 0.58 MB/sec execute 63 sec latency 17074.465 ms Started lustre-MDT0000 1 7283 0.58 MB/sec execute 64 sec latency 18074.618 ms 1 7283 0.57 MB/sec execute 65 sec latency 19074.796 ms 1 7283 0.56 MB/sec execute 66 sec latency 20074.987 ms 1 7283 0.55 MB/sec execute 67 sec latency 21075.182 ms 1 7283 0.54 MB/sec execute 68 sec latency 22075.392 ms 1 7283 0.53 MB/sec execute 69 sec latency 23075.581 ms 1 7283 0.53 MB/sec execute 70 sec latency 24075.781 ms 1 7283 0.52 MB/sec execute 71 sec latency 25075.956 ms 1 7283 0.51 MB/sec execute 72 sec latency 26076.059 ms 1 7283 0.50 MB/sec execute 73 sec latency 27076.210 ms 1 7283 0.50 MB/sec execute 74 sec latency 28076.387 ms 1 7283 0.49 MB/sec execute 75 sec latency 29076.554 ms 1 7283 0.48 MB/sec execute 76 sec latency 30076.806 ms 1 7283 0.48 MB/sec execute 77 sec latency 31076.983 ms 1 7283 0.47 MB/sec execute 78 sec latency 32077.110 ms 1 7467 0.48 MB/sec execute 79 sec latency 32346.222 ms oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7722 0.47 MB/sec execute 80 sec latency 15.358 ms 1 7963 0.47 MB/sec execute 81 sec latency 101.633 ms 1 8263 0.48 MB/sec execute 82 sec latency 50.424 ms 1 8668 0.52 MB/sec execute 83 sec latency 68.556 ms 1 8938 0.51 MB/sec execute 84 sec latency 61.208 ms 1 9399 0.52 MB/sec execute 85 sec latency 29.654 ms 1 9711 0.53 MB/sec execute 86 sec latency 49.169 ms 1 10363 0.60 MB/sec execute 87 sec latency 52.436 ms 1 10850 0.64 MB/sec execute 88 sec latency 12.826 ms 1 11132 0.64 MB/sec execute 89 sec latency 14.572 ms 1 11434 0.64 MB/sec execute 90 sec latency 91.933 ms 1 11709 0.65 MB/sec execute 91 sec latency 37.508 ms 1 12124 0.67 MB/sec execute 92 sec latency 14.162 ms 1 12398 0.67 MB/sec execute 93 sec latency 68.010 ms 1 12810 0.67 MB/sec execute 94 sec latency 37.454 ms 1 13206 0.68 MB/sec execute 95 sec latency 50.443 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Jan 15 19:22:09 EST 2025 with return code 0 19418 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 19418 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished