== replay-dual test 26: dbench and tar with mds failover ========================================================== 11:46:29 (1737132389) Starting client oleg459-client.virtnet: -o user_xattr,flock oleg459-server@tcp:/lustre /mnt/lustre Started clients oleg459-client.virtnet: 192.168.204.159@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15863 Started dbench loop with 15864 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 17 11:46:30 EST 2025 waiting for dbench pid 15888 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 239 9.14 MB/sec warmup 1 sec latency 26.425 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3641344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3723264 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7364608 1% /mnt/lustre 1 509 8.83 MB/sec warmup 2 sec latency 16.839 ms 1 848 7.19 MB/sec warmup 3 sec latency 35.384 ms 1 1076 5.73 MB/sec warmup 4 sec latency 51.642 ms test_26 fail mds1 1 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 1451 5.20 MB/sec warmup 5 sec latency 15.347 ms reboot facets: mds1 1 1459 4.34 MB/sec warmup 6 sec latency 970.149 ms 1 1459 3.72 MB/sec warmup 7 sec latency 1970.365 ms 1 1459 3.25 MB/sec warmup 8 sec latency 2970.596 ms 1 1459 2.89 MB/sec warmup 9 sec latency 3970.833 ms 1 1459 2.60 MB/sec warmup 10 sec latency 4971.151 ms 1 1459 2.37 MB/sec warmup 11 sec latency 5971.369 ms 1 1459 2.17 MB/sec warmup 12 sec latency 6971.581 ms 1 1459 2.00 MB/sec warmup 13 sec latency 7971.798 ms 1 1459 1.86 MB/sec warmup 14 sec latency 8972.054 ms 1 1459 1.74 MB/sec warmup 15 sec latency 9972.465 ms Failover mds1 to oleg459-server 1 1459 1.63 MB/sec warmup 16 sec latency 10972.715 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1459 1.53 MB/sec warmup 17 sec latency 11972.935 ms 1 1459 1.45 MB/sec warmup 18 sec latency 12973.144 ms 1 1459 1.37 MB/sec warmup 19 sec latency 13973.393 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 1 1459 0.00 MB/sec execute 1 sec latency 15973.718 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1459 0.00 MB/sec execute 2 sec latency 16973.921 ms 1 1459 0.00 MB/sec execute 3 sec latency 17974.144 ms 1 1459 0.00 MB/sec execute 4 sec latency 18974.438 ms 1 1459 0.00 MB/sec execute 5 sec latency 19974.731 ms 1 1459 0.00 MB/sec execute 6 sec latency 20975.031 ms 1 1459 0.00 MB/sec execute 7 sec latency 21975.448 ms 1 1459 0.00 MB/sec execute 8 sec latency 22975.708 ms 1 1459 0.00 MB/sec execute 9 sec latency 23975.991 ms 1 1459 0.00 MB/sec execute 10 sec latency 24976.388 ms 1 1459 0.00 MB/sec execute 11 sec latency 25976.612 ms 1 1459 0.00 MB/sec execute 12 sec latency 26976.848 ms 1 1459 0.00 MB/sec execute 13 sec latency 27977.102 ms 1 1459 0.00 MB/sec execute 14 sec latency 28977.237 ms 1 1459 0.00 MB/sec execute 15 sec latency 29977.388 ms 1 1735 0.02 MB/sec execute 16 sec latency 30135.079 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2104 0.04 MB/sec execute 17 sec latency 40.045 ms 1 2426 0.11 MB/sec execute 18 sec latency 44.665 ms 1 2917 0.39 MB/sec execute 19 sec latency 31.079 ms 1 3365 0.47 MB/sec execute 20 sec latency 61.419 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4864 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3649536 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3613696 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7263232 1% /mnt/lustre 1 3809 0.65 MB/sec execute 21 sec latency 13.700 ms 1 4024 0.64 MB/sec execute 22 sec latency 18.216 ms 1 4314 0.63 MB/sec execute 23 sec latency 65.392 ms test_26 fail mds1 2 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 4614 0.67 MB/sec execute 24 sec latency 49.362 ms reboot facets: mds1 1 4701 0.64 MB/sec execute 25 sec latency 798.327 ms 1 4701 0.62 MB/sec execute 26 sec latency 1798.542 ms 1 4701 0.60 MB/sec execute 27 sec latency 2798.744 ms 1 4701 0.57 MB/sec execute 28 sec latency 3798.912 ms 1 4701 0.55 MB/sec execute 29 sec latency 4799.061 ms 1 4701 0.54 MB/sec execute 30 sec latency 5799.212 ms 1 4701 0.52 MB/sec execute 31 sec latency 6799.459 ms 1 4701 0.50 MB/sec execute 32 sec latency 7799.751 ms 1 4701 0.49 MB/sec execute 33 sec latency 8799.935 ms 1 4701 0.47 MB/sec execute 34 sec latency 9800.133 ms Failover mds1 to oleg459-server 1 4701 0.46 MB/sec execute 35 sec latency 10800.346 ms mount facets: mds1 1 4701 0.45 MB/sec execute 36 sec latency 11800.593 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4701 0.43 MB/sec execute 37 sec latency 12800.707 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 1 4701 0.42 MB/sec execute 38 sec latency 13801.001 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4701 0.41 MB/sec execute 39 sec latency 14801.395 ms 1 4701 0.40 MB/sec execute 40 sec latency 15801.600 ms 1 4701 0.39 MB/sec execute 41 sec latency 16801.748 ms 1 5047 0.46 MB/sec execute 42 sec latency 16945.475 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5362 0.45 MB/sec execute 43 sec latency 34.530 ms 1 5761 0.45 MB/sec execute 44 sec latency 33.013 ms 1 6050 0.48 MB/sec execute 45 sec latency 38.903 ms 1 6484 0.58 MB/sec execute 46 sec latency 50.052 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6656 2201856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3683328 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3708928 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7392256 1% /mnt/lustre 1 7056 0.67 MB/sec execute 47 sec latency 46.160 ms 1 7373 0.68 MB/sec execute 48 sec latency 18.863 ms 1 7648 0.68 MB/sec execute 49 sec latency 13.355 ms test_26 fail mds1 3 times Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 7904 0.67 MB/sec execute 50 sec latency 101.231 ms reboot facets: mds1 1 7976 0.66 MB/sec execute 51 sec latency 793.724 ms 1 7976 0.65 MB/sec execute 52 sec latency 1793.954 ms 1 7976 0.64 MB/sec execute 53 sec latency 2794.119 ms 1 7976 0.63 MB/sec execute 54 sec latency 3794.296 ms 1 7976 0.61 MB/sec execute 55 sec latency 4794.521 ms 1 7976 0.60 MB/sec execute 56 sec latency 5794.683 ms 1 7976 0.59 MB/sec execute 57 sec latency 6794.845 ms 1 7976 0.58 MB/sec execute 58 sec latency 7795.013 ms 1 7976 0.57 MB/sec execute 59 sec latency 8795.200 ms 1 7976 0.56 MB/sec execute 60 sec latency 9795.430 ms Failover mds1 to oleg459-server 1 7976 0.55 MB/sec execute 61 sec latency 10795.552 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7976 0.54 MB/sec execute 62 sec latency 11795.660 ms 1 7976 0.54 MB/sec execute 63 sec latency 12795.810 ms 1 7976 0.53 MB/sec execute 64 sec latency 13795.977 ms 1 7976 0.52 MB/sec execute 65 sec latency 14796.089 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 1 7976 0.51 MB/sec execute 66 sec latency 15796.265 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 1 7976 0.50 MB/sec execute 67 sec latency 16796.468 ms 1 7976 0.50 MB/sec execute 68 sec latency 17796.712 ms 1 7976 0.49 MB/sec execute 69 sec latency 18796.887 ms 1 7976 0.48 MB/sec execute 70 sec latency 19797.052 ms 1 7976 0.48 MB/sec execute 71 sec latency 20797.274 ms 1 7976 0.47 MB/sec execute 72 sec latency 21797.444 ms 1 7976 0.46 MB/sec execute 73 sec latency 22797.604 ms 1 7976 0.46 MB/sec execute 74 sec latency 23797.768 ms 1 7976 0.45 MB/sec execute 75 sec latency 24797.958 ms 1 7976 0.44 MB/sec execute 76 sec latency 25798.287 ms 1 7976 0.44 MB/sec execute 77 sec latency 26798.521 ms 1 7976 0.43 MB/sec execute 78 sec latency 27798.760 ms 1 7976 0.43 MB/sec execute 79 sec latency 28798.984 ms 1 7976 0.42 MB/sec execute 80 sec latency 29799.146 ms 1 8083 0.42 MB/sec execute 81 sec latency 30382.547 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8512 0.47 MB/sec execute 82 sec latency 29.490 ms 1 8792 0.46 MB/sec execute 83 sec latency 50.296 ms 1 9169 0.46 MB/sec execute 84 sec latency 33.472 ms 1 9557 0.47 MB/sec execute 85 sec latency 39.106 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8704 2199808 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3704832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3718144 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7422976 1% /mnt/lustre 1 10099 0.53 MB/sec execute 86 sec latency 31.679 ms 1 10709 0.58 MB/sec execute 87 sec latency 25.769 ms 1 10987 0.59 MB/sec execute 88 sec latency 16.639 ms test_26 fail mds1 4 times 1 11250 0.59 MB/sec execute 89 sec latency 16.325 ms Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 11400 0.58 MB/sec execute 90 sec latency 401.087 ms reboot facets: mds1 1 11400 0.58 MB/sec execute 91 sec latency 1401.314 ms 1 11400 0.57 MB/sec execute 92 sec latency 2401.512 ms 1 11400 0.57 MB/sec execute 93 sec latency 3401.765 ms 1 11400 0.56 MB/sec execute 94 sec latency 4402.131 ms 1 11400 0.55 MB/sec execute 95 sec latency 5402.461 ms 1 11400 0.55 MB/sec execute 96 sec latency 6402.725 ms 1 11400 0.54 MB/sec execute 97 sec latency 7402.995 ms 1 11400 0.54 MB/sec execute 98 sec latency 8403.222 ms 1 11400 0.53 MB/sec execute 99 sec latency 9403.559 ms 1 cleanup 100 sec Failover mds1 to oleg459-server mount facets: mds1 1 cleanup 101 sec Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 102 sec oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 1 cleanup 103 sec Started lustre-MDT0000 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 0 cleanup 107 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1732 52.702 30382.537 Close 1251 1.495 7.620 Rename 69 256.308 16945.441 Unlink 355 4.046 13.957 Qpathinfo 1594 2.086 28.850 Qfileinfo 275 0.372 2.413 Qfsinfo 282 0.387 3.330 Sfileinfo 122 5.156 11.666 Find 604 1.028 10.535 WriteX 842 1.801 7.259 ReadX 2687 0.076 26.465 LockX 6 1.340 1.604 UnlockX 6 1.304 1.893 Flush 117 23.196 101.219 Throughput 0.53166 MB/sec 1 clients 1 procs max_latency=30382.547 ms stopping dbench on /mnt/lustre at Fri Jan 17 11:48:37 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 17 11:48:38 EST 2025 waiting for dbench pid 20289 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 289 10.70 MB/sec warmup 1 sec latency 17.161 ms 1 542 9.55 MB/sec warmup 2 sec latency 25.589 ms 1 849 7.19 MB/sec warmup 3 sec latency 83.947 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10240 2198272 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3682304 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 54272 3614720 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7297024 1% /mnt/lustre 1 1172 5.75 MB/sec warmup 4 sec latency 31.152 ms 1 1616 5.23 MB/sec warmup 5 sec latency 25.430 ms 1 1989 4.41 MB/sec warmup 6 sec latency 33.076 ms test_26 fail mds1 5 times Failing mds1 on oleg459-server 1 2376 3.99 MB/sec warmup 7 sec latency 40.769 ms Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 2440 3.51 MB/sec warmup 8 sec latency 725.903 ms reboot facets: mds1 1 2440 3.12 MB/sec warmup 9 sec latency 1726.071 ms 1 2440 2.81 MB/sec warmup 10 sec latency 2726.256 ms 1 2440 2.55 MB/sec warmup 11 sec latency 3726.490 ms 1 2440 2.34 MB/sec warmup 12 sec latency 4726.752 ms 1 2440 2.16 MB/sec warmup 13 sec latency 5726.989 ms 1 2440 2.01 MB/sec warmup 14 sec latency 6727.290 ms 1 2440 1.87 MB/sec warmup 15 sec latency 7727.567 ms 1 2440 1.76 MB/sec warmup 16 sec latency 8727.810 ms 1 2440 1.65 MB/sec warmup 17 sec latency 9728.065 ms 1 2440 1.56 MB/sec warmup 18 sec latency 10728.344 ms Failover mds1 to oleg459-server mount facets: mds1 1 2440 1.48 MB/sec warmup 19 sec latency 11728.588 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2440 0.00 MB/sec execute 1 sec latency 13729.086 ms 1 2440 0.00 MB/sec execute 2 sec latency 14729.359 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 1 2440 0.00 MB/sec execute 3 sec latency 15729.557 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2440 0.00 MB/sec execute 4 sec latency 16729.796 ms 1 2440 0.00 MB/sec execute 5 sec latency 17730.008 ms 1 2440 0.00 MB/sec execute 6 sec latency 18730.279 ms 1 2440 0.00 MB/sec execute 7 sec latency 19730.520 ms 1 2440 0.00 MB/sec execute 8 sec latency 20730.802 ms 1 2440 0.00 MB/sec execute 9 sec latency 21730.948 ms 1 2440 0.00 MB/sec execute 10 sec latency 22731.138 ms 1 2440 0.00 MB/sec execute 11 sec latency 23731.351 ms 1 2440 0.00 MB/sec execute 12 sec latency 24731.623 ms 1 2440 0.00 MB/sec execute 13 sec latency 25732.027 ms 1 2440 0.00 MB/sec execute 14 sec latency 26732.338 ms 1 2440 0.00 MB/sec execute 15 sec latency 27732.638 ms 1 2440 0.00 MB/sec execute 16 sec latency 28732.992 ms 1 2440 0.00 MB/sec execute 17 sec latency 29733.154 ms 1 2475 0.03 MB/sec execute 18 sec latency 30610.903 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3019 0.29 MB/sec execute 19 sec latency 37.786 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3644 0.53 MB/sec execute 20 sec latency 29.064 ms 1 3951 0.57 MB/sec execute 21 sec latency 15.049 ms 1 4238 0.55 MB/sec execute 22 sec latency 106.720 ms 1 4491 0.56 MB/sec execute 23 sec latency 57.582 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11136 2197376 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 27648 3670016 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 64512 3639296 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 92160 7309312 2% /mnt/lustre 1 4839 0.59 MB/sec execute 24 sec latency 29.582 ms 1 5237 0.69 MB/sec execute 25 sec latency 59.602 ms test_26 fail mds1 6 times 1 5622 0.68 MB/sec execute 26 sec latency 25.201 ms Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 5965 0.71 MB/sec execute 27 sec latency 75.478 ms reboot facets: mds1 1 5965 0.68 MB/sec execute 28 sec latency 1075.718 ms 1 5965 0.66 MB/sec execute 29 sec latency 2075.931 ms 1 5965 0.63 MB/sec execute 30 sec latency 3076.044 ms 1 5965 0.61 MB/sec execute 31 sec latency 4076.252 ms 1 5965 0.59 MB/sec execute 32 sec latency 5076.511 ms 1 5965 0.58 MB/sec execute 33 sec latency 6076.807 ms 1 5965 0.56 MB/sec execute 34 sec latency 7076.989 ms 1 5965 0.54 MB/sec execute 35 sec latency 8077.258 ms 1 5965 0.53 MB/sec execute 36 sec latency 9077.489 ms 1 5965 0.51 MB/sec execute 37 sec latency 10077.687 ms Failover mds1 to oleg459-server 1 5965 0.50 MB/sec execute 38 sec latency 11077.938 ms mount facets: mds1 1 5965 0.49 MB/sec execute 39 sec latency 12078.168 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5965 0.48 MB/sec execute 40 sec latency 13078.416 ms 1 5965 0.46 MB/sec execute 41 sec latency 14078.623 ms 1 5965 0.45 MB/sec execute 42 sec latency 15078.875 ms 1 5965 0.44 MB/sec execute 43 sec latency 16079.149 ms 1 5965 0.43 MB/sec execute 44 sec latency 17079.396 ms 1 5965 0.42 MB/sec execute 45 sec latency 18079.632 ms 1 5965 0.41 MB/sec execute 46 sec latency 19079.763 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 1 5965 0.41 MB/sec execute 47 sec latency 20079.966 ms 1 5965 0.40 MB/sec execute 48 sec latency 21080.161 ms 1 5965 0.39 MB/sec execute 49 sec latency 22080.318 ms 1 5965 0.38 MB/sec execute 50 sec latency 23080.490 ms 1 5965 0.37 MB/sec execute 51 sec latency 24080.655 ms 1 5965 0.37 MB/sec execute 52 sec latency 25080.801 ms 1 5965 0.36 MB/sec execute 53 sec latency 26080.953 ms 1 5965 0.35 MB/sec execute 54 sec latency 27081.144 ms 1 5965 0.35 MB/sec execute 55 sec latency 28081.338 ms 1 5965 0.34 MB/sec execute 56 sec latency 29081.565 ms 1 5965 0.33 MB/sec execute 57 sec latency 30081.779 ms 1 5965 0.33 MB/sec execute 58 sec latency 31081.991 ms 1 5965 0.32 MB/sec execute 59 sec latency 32082.119 ms 1 6049 0.33 MB/sec execute 60 sec latency 32703.823 ms 1 6524 0.40 MB/sec execute 61 sec latency 72.341 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6915 0.43 MB/sec execute 62 sec latency 56.091 ms 1 7187 0.47 MB/sec execute 63 sec latency 29.988 ms 1 7312 0.48 MB/sec execute 64 sec latency 37.650 ms 1 7377 0.47 MB/sec execute 65 sec latency 49.296 ms 1 7449 0.47 MB/sec execute 66 sec latency 56.597 ms 1 7506 0.46 MB/sec execute 67 sec latency 116.389 ms 1 7563 0.46 MB/sec execute 68 sec latency 42.378 ms 1 7637 0.45 MB/sec execute 69 sec latency 60.952 ms 1 7763 0.45 MB/sec execute 70 sec latency 41.782 ms 1 7831 0.44 MB/sec execute 71 sec latency 170.258 ms 1 7902 0.44 MB/sec execute 72 sec latency 117.492 ms 1 7985 0.44 MB/sec execute 73 sec latency 101.285 ms 1 8030 0.43 MB/sec execute 74 sec latency 180.107 ms 1 8101 0.43 MB/sec execute 75 sec latency 157.602 ms 1 8235 0.44 MB/sec execute 76 sec latency 54.878 ms 1 8333 0.43 MB/sec execute 77 sec latency 48.713 ms 1 8508 0.46 MB/sec execute 78 sec latency 63.547 ms 1 8595 0.46 MB/sec execute 79 sec latency 212.028 ms 1 8686 0.45 MB/sec execute 80 sec latency 61.987 ms 1 8781 0.45 MB/sec execute 81 sec latency 101.025 ms 1 8891 0.44 MB/sec execute 82 sec latency 37.614 ms 1 8986 0.44 MB/sec execute 83 sec latency 44.047 ms 1 9123 0.44 MB/sec execute 84 sec latency 130.126 ms 1 9283 0.43 MB/sec execute 85 sec latency 31.835 ms 1 9477 0.44 MB/sec execute 86 sec latency 101.156 ms 1 9569 0.45 MB/sec execute 87 sec latency 137.556 ms 1 9710 0.44 MB/sec execute 88 sec latency 103.154 ms 1 9876 0.47 MB/sec execute 89 sec latency 354.510 ms 1 10012 0.48 MB/sec execute 90 sec latency 88.364 ms 1 10036 0.48 MB/sec execute 91 sec latency 448.611 ms 1 10095 0.47 MB/sec execute 92 sec latency 280.814 ms 1 10269 0.48 MB/sec execute 93 sec latency 252.851 ms 1 10336 0.47 MB/sec execute 94 sec latency 274.810 ms 1 10456 0.48 MB/sec execute 95 sec latency 103.345 ms 1 10631 0.51 MB/sec execute 96 sec latency 55.065 ms 1 10725 0.50 MB/sec execute 97 sec latency 57.643 ms 1 10793 0.50 MB/sec execute 98 sec latency 66.053 ms 1 10870 0.50 MB/sec execute 99 sec latency 205.426 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 102 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1394 39.848 30610.885 Close 1023 35.602 32703.806 Rename 64 22.390 79.610 Unlink 284 8.505 44.154 Qpathinfo 1278 4.288 63.087 Qfileinfo 231 0.539 12.190 Qfsinfo 258 0.888 20.904 Sfileinfo 106 10.412 27.186 Find 514 1.663 34.772 WriteX 723 3.292 41.689 ReadX 2495 0.047 9.754 LockX 6 6.367 16.054 UnlockX 6 4.666 12.538 Flush 96 67.848 448.599 Throughput 0.503681 MB/sec 1 clients 1 procs max_latency=32703.823 ms stopping dbench on /mnt/lustre at Fri Jan 17 11:50:41 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 17 11:50:45 EST 2025 waiting for dbench pid 22852 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 121 5.50 MB/sec warmup 1 sec latency 59.472 ms 1 255 4.83 MB/sec warmup 2 sec latency 44.103 ms 1 414 4.87 MB/sec warmup 3 sec latency 39.776 ms 1 664 5.29 MB/sec warmup 4 sec latency 35.630 ms 1 733 4.26 MB/sec warmup 5 sec latency 120.853 ms 1 806 3.58 MB/sec warmup 6 sec latency 139.943 ms 1 904 3.10 MB/sec warmup 7 sec latency 128.397 ms 1 983 2.74 MB/sec warmup 8 sec latency 133.327 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Jan 17 11:50:53 EST 2025 with return code 0 22852 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 22852 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished