== replay-dual test 26: dbench and tar with mds failover ========================================================== 23:41:49 (1722483709) Starting client oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre Started clients oleg414-client.virtnet: 192.168.204.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15844 Started dbench loop with 15845 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jul 31 23:41:51 EDT 2024 waiting for dbench pid 15869 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 199 7.84 MB/sec warmup 1 sec latency 35.457 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3703808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3715072 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7418880 1% /mnt/lustre 1 437 7.69 MB/sec warmup 2 sec latency 29.717 ms 1 730 7.10 MB/sec warmup 3 sec latency 55.243 ms 1 980 5.47 MB/sec warmup 4 sec latency 46.539 ms test_26 fail mds1 1 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 1403 5.20 MB/sec warmup 5 sec latency 119.997 ms reboot facets: mds1 1 1403 4.34 MB/sec warmup 6 sec latency 1120.163 ms 1 1403 3.72 MB/sec warmup 7 sec latency 2120.309 ms 1 1403 3.25 MB/sec warmup 8 sec latency 3120.463 ms 1 1403 2.89 MB/sec warmup 9 sec latency 4120.689 ms 1 1403 2.60 MB/sec warmup 10 sec latency 5120.830 ms 1 1403 2.36 MB/sec warmup 11 sec latency 6120.980 ms 1 1403 2.17 MB/sec warmup 12 sec latency 7121.198 ms 1 1403 2.00 MB/sec warmup 13 sec latency 8121.431 ms 1 1403 1.86 MB/sec warmup 14 sec latency 9121.666 ms 1 1403 1.73 MB/sec warmup 15 sec latency 10121.877 ms Failover mds1 to oleg414-server mount facets: mds1 1 1403 1.63 MB/sec warmup 16 sec latency 11122.040 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1403 1.53 MB/sec warmup 17 sec latency 12122.173 ms 1 1403 1.45 MB/sec warmup 18 sec latency 13122.357 ms 1 1403 1.37 MB/sec warmup 19 sec latency 14122.593 ms 1 1403 0.00 MB/sec execute 1 sec latency 16122.942 ms 1 1403 0.00 MB/sec execute 2 sec latency 17123.144 ms 1 1403 0.00 MB/sec execute 3 sec latency 18123.306 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 1 1403 0.00 MB/sec execute 4 sec latency 19123.489 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1403 0.00 MB/sec execute 5 sec latency 20123.630 ms 1 1403 0.00 MB/sec execute 6 sec latency 21123.814 ms 1 1403 0.00 MB/sec execute 7 sec latency 22123.951 ms 1 1403 0.00 MB/sec execute 8 sec latency 23124.174 ms 1 1403 0.00 MB/sec execute 9 sec latency 24124.348 ms 1 1403 0.00 MB/sec execute 10 sec latency 25124.526 ms 1 1403 0.00 MB/sec execute 11 sec latency 26124.656 ms 1 1403 0.00 MB/sec execute 12 sec latency 27124.771 ms 1 1403 0.00 MB/sec execute 13 sec latency 28125.005 ms 1 1403 0.00 MB/sec execute 14 sec latency 29125.287 ms 1 1403 0.00 MB/sec execute 15 sec latency 30125.450 ms 1 1441 0.00 MB/sec execute 16 sec latency 31006.237 ms 1 1653 0.01 MB/sec execute 17 sec latency 59.826 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1946 0.02 MB/sec execute 18 sec latency 47.114 ms 1 2238 0.04 MB/sec execute 19 sec latency 13.993 ms 1 2489 0.14 MB/sec execute 20 sec latency 64.591 ms 1 2922 0.36 MB/sec execute 21 sec latency 45.598 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4864 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3702784 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3635200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 59392 7337984 1% /mnt/lustre 1 3305 0.43 MB/sec execute 22 sec latency 52.698 ms 1 3666 0.55 MB/sec execute 23 sec latency 20.704 ms 1 3836 0.57 MB/sec execute 24 sec latency 27.112 ms test_26 fail mds1 2 times 1 3978 0.56 MB/sec execute 25 sec latency 27.176 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 4162 0.54 MB/sec execute 26 sec latency 25.282 ms reboot facets: mds1 1 4187 0.52 MB/sec execute 27 sec latency 914.063 ms 1 4187 0.50 MB/sec execute 28 sec latency 1914.196 ms 1 4187 0.49 MB/sec execute 29 sec latency 2914.419 ms 1 4187 0.47 MB/sec execute 30 sec latency 3914.600 ms 1 4187 0.45 MB/sec execute 31 sec latency 4914.801 ms 1 4187 0.44 MB/sec execute 32 sec latency 5915.027 ms 1 4187 0.43 MB/sec execute 33 sec latency 6915.291 ms 1 4187 0.41 MB/sec execute 34 sec latency 7915.440 ms 1 4187 0.40 MB/sec execute 35 sec latency 8915.616 ms 1 4187 0.39 MB/sec execute 36 sec latency 9915.856 ms 1 4187 0.38 MB/sec execute 37 sec latency 10916.049 ms Failover mds1 to oleg414-server mount facets: mds1 1 4187 0.37 MB/sec execute 38 sec latency 11916.271 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4187 0.36 MB/sec execute 39 sec latency 12916.522 ms 1 4187 0.35 MB/sec execute 40 sec latency 13916.671 ms 1 4187 0.34 MB/sec execute 41 sec latency 14916.855 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 1 4187 0.34 MB/sec execute 42 sec latency 15917.074 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4187 0.33 MB/sec execute 43 sec latency 16917.227 ms 1 4187 0.32 MB/sec execute 44 sec latency 17917.412 ms 1 4187 0.31 MB/sec execute 45 sec latency 18917.586 ms 1 4187 0.31 MB/sec execute 46 sec latency 19917.721 ms 1 4187 0.30 MB/sec execute 47 sec latency 20917.895 ms 1 4187 0.29 MB/sec execute 48 sec latency 21918.096 ms 1 4187 0.29 MB/sec execute 49 sec latency 22918.348 ms 1 4187 0.28 MB/sec execute 50 sec latency 23918.520 ms 1 4187 0.28 MB/sec execute 51 sec latency 24918.669 ms 1 4187 0.27 MB/sec execute 52 sec latency 25918.907 ms 1 4187 0.27 MB/sec execute 53 sec latency 26919.218 ms 1 4187 0.26 MB/sec execute 54 sec latency 27919.499 ms 1 4187 0.26 MB/sec execute 55 sec latency 28919.837 ms 1 4187 0.25 MB/sec execute 56 sec latency 29920.007 ms 1 4187 0.25 MB/sec execute 57 sec latency 30920.150 ms 1 4292 0.25 MB/sec execute 58 sec latency 31409.574 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4511 0.25 MB/sec execute 59 sec latency 54.527 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4764 0.27 MB/sec execute 60 sec latency 24.905 ms 1 5040 0.31 MB/sec execute 61 sec latency 24.927 ms 1 5225 0.31 MB/sec execute 62 sec latency 62.013 ms 1 5567 0.31 MB/sec execute 63 sec latency 42.541 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6144 2202368 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3650560 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3619840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 7270400 1% /mnt/lustre 1 5960 0.33 MB/sec execute 64 sec latency 27.953 ms 1 6337 0.39 MB/sec execute 65 sec latency 43.620 ms test_26 fail mds1 3 times 1 6752 0.41 MB/sec execute 66 sec latency 45.241 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 7126 0.47 MB/sec execute 67 sec latency 59.515 ms reboot facets: mds1 1 7126 0.46 MB/sec execute 68 sec latency 1029.933 ms 1 7126 0.46 MB/sec execute 69 sec latency 2030.125 ms 1 7126 0.45 MB/sec execute 70 sec latency 3030.387 ms 1 7126 0.44 MB/sec execute 71 sec latency 4030.643 ms 1 7126 0.44 MB/sec execute 72 sec latency 5030.850 ms 1 7126 0.43 MB/sec execute 73 sec latency 6031.068 ms 1 7126 0.43 MB/sec execute 74 sec latency 7031.288 ms 1 7126 0.42 MB/sec execute 75 sec latency 8031.521 ms 1 7126 0.41 MB/sec execute 76 sec latency 9031.724 ms 1 7126 0.41 MB/sec execute 77 sec latency 10031.960 ms Failover mds1 to oleg414-server 1 7126 0.40 MB/sec execute 78 sec latency 11032.224 ms mount facets: mds1 1 7126 0.40 MB/sec execute 79 sec latency 12032.456 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7126 0.39 MB/sec execute 80 sec latency 13032.627 ms 1 7126 0.39 MB/sec execute 81 sec latency 14032.816 ms 1 7126 0.38 MB/sec execute 82 sec latency 15033.013 ms 1 7126 0.38 MB/sec execute 83 sec latency 16033.221 ms 1 7126 0.38 MB/sec execute 84 sec latency 17033.370 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 1 7126 0.37 MB/sec execute 85 sec latency 18033.530 ms pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 7126 0.37 MB/sec execute 86 sec latency 19033.675 ms 1 7126 0.36 MB/sec execute 87 sec latency 20033.795 ms 1 7126 0.36 MB/sec execute 88 sec latency 21033.971 ms 1 7126 0.35 MB/sec execute 89 sec latency 22034.265 ms 1 7126 0.35 MB/sec execute 90 sec latency 23034.463 ms 1 7126 0.35 MB/sec execute 91 sec latency 24034.596 ms 1 7126 0.34 MB/sec execute 92 sec latency 25034.721 ms 1 7126 0.34 MB/sec execute 93 sec latency 26034.859 ms 1 7126 0.34 MB/sec execute 94 sec latency 27035.009 ms 1 7126 0.33 MB/sec execute 95 sec latency 28035.179 ms 1 7126 0.33 MB/sec execute 96 sec latency 29035.342 ms 1 7126 0.33 MB/sec execute 97 sec latency 30035.463 ms 1 7126 0.32 MB/sec execute 98 sec latency 31035.573 ms 1 7224 0.32 MB/sec execute 99 sec latency 31795.804 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 100 sec mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1045 70.434 31795.796 Close 776 42.437 31409.538 Rename 42 14.103 22.320 Unlink 191 5.183 17.167 Qpathinfo 971 2.740 16.463 Qfileinfo 176 0.447 3.911 Qfsinfo 168 0.519 2.854 Sfileinfo 66 6.315 11.732 Find 366 1.189 13.229 WriteX 524 2.222 8.613 ReadX 1695 0.138 28.319 LockX 4 2.120 2.392 UnlockX 4 2.476 2.726 Flush 62 30.375 64.578 Throughput 0.321414 MB/sec 1 clients 1 procs max_latency=31795.804 ms stopping dbench on /mnt/lustre at Wed Jul 31 23:43:51 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jul 31 23:43:53 EDT 2024 waiting for dbench pid 19307 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 243 9.21 MB/sec warmup 1 sec latency 27.584 ms 1 477 8.34 MB/sec warmup 2 sec latency 44.494 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 7808 2200832 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3494912 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 28672 3718144 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7213056 1% /mnt/lustre 1 779 7.12 MB/sec warmup 3 sec latency 45.934 ms 1 1020 5.62 MB/sec warmup 4 sec latency 48.363 ms test_26 fail mds1 4 times 1 1419 5.20 MB/sec warmup 5 sec latency 18.373 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 1690 4.37 MB/sec warmup 6 sec latency 185.365 ms reboot facets: mds1 1 1690 3.75 MB/sec warmup 7 sec latency 1185.482 ms 1 1690 3.28 MB/sec warmup 8 sec latency 2185.657 ms 1 1690 2.91 MB/sec warmup 9 sec latency 3185.882 ms 1 1690 2.62 MB/sec warmup 10 sec latency 4186.081 ms 1 1690 2.38 MB/sec warmup 11 sec latency 5186.344 ms 1 1690 2.19 MB/sec warmup 12 sec latency 6186.517 ms 1 1690 2.02 MB/sec warmup 13 sec latency 7186.732 ms 1 1690 1.87 MB/sec warmup 14 sec latency 8186.940 ms 1 1690 1.75 MB/sec warmup 15 sec latency 9187.146 ms 1 1690 1.64 MB/sec warmup 16 sec latency 10187.387 ms Failover mds1 to oleg414-server mount facets: mds1 1 1690 1.54 MB/sec warmup 17 sec latency 11187.551 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1690 1.46 MB/sec warmup 18 sec latency 12187.781 ms 1 1690 1.38 MB/sec warmup 19 sec latency 13188.033 ms 1 1690 0.00 MB/sec execute 1 sec latency 15188.509 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 1 1690 0.00 MB/sec execute 2 sec latency 16188.646 ms Started lustre-MDT0000 1 1690 0.00 MB/sec execute 3 sec latency 17188.805 ms 1 1858 0.03 MB/sec execute 4 sec latency 17512.145 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2160 0.10 MB/sec execute 5 sec latency 80.715 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2406 0.30 MB/sec execute 6 sec latency 75.070 ms 1 2608 0.40 MB/sec execute 7 sec latency 53.348 ms 1 3019 0.91 MB/sec execute 8 sec latency 75.856 ms 1 3399 1.03 MB/sec execute 9 sec latency 51.654 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 9344 2199296 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3136512 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3731456 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 6867968 1% /mnt/lustre 1 3705 1.26 MB/sec execute 10 sec latency 17.301 ms 1 3917 1.26 MB/sec execute 11 sec latency 24.086 ms 1 4061 1.15 MB/sec execute 12 sec latency 25.961 ms test_26 fail mds1 5 times Failing mds1 on oleg414-server 1 4304 1.09 MB/sec execute 13 sec latency 69.132 ms Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 1 4373 1.03 MB/sec execute 14 sec latency 752.548 ms 1 4373 0.96 MB/sec execute 15 sec latency 1752.727 ms 1 4373 0.90 MB/sec execute 16 sec latency 2752.907 ms 1 4373 0.85 MB/sec execute 17 sec latency 3753.093 ms 1 4373 0.80 MB/sec execute 18 sec latency 4753.342 ms 1 4373 0.76 MB/sec execute 19 sec latency 5753.550 ms 1 4373 0.72 MB/sec execute 20 sec latency 6753.726 ms 1 4373 0.68 MB/sec execute 21 sec latency 7753.970 ms 1 4373 0.65 MB/sec execute 22 sec latency 8754.170 ms 1 4373 0.62 MB/sec execute 23 sec latency 9754.386 ms Failover mds1 to oleg414-server 1 4373 0.60 MB/sec execute 24 sec latency 10754.550 ms mount facets: mds1 1 4373 0.57 MB/sec execute 25 sec latency 11754.776 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4373 0.55 MB/sec execute 26 sec latency 12754.942 ms 1 4373 0.53 MB/sec execute 27 sec latency 13755.075 ms 1 4373 0.51 MB/sec execute 28 sec latency 14755.203 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4373 0.50 MB/sec execute 29 sec latency 15755.341 ms 1 4373 0.48 MB/sec execute 30 sec latency 16755.484 ms 1 4373 0.46 MB/sec execute 31 sec latency 17755.640 ms 1 4373 0.45 MB/sec execute 32 sec latency 18755.840 ms 1 4373 0.44 MB/sec execute 33 sec latency 19756.027 ms 1 4373 0.42 MB/sec execute 34 sec latency 20756.168 ms 1 4373 0.41 MB/sec execute 35 sec latency 21756.314 ms 1 4373 0.40 MB/sec execute 36 sec latency 22756.475 ms 1 4373 0.39 MB/sec execute 37 sec latency 23756.620 ms 1 4373 0.38 MB/sec execute 38 sec latency 24756.831 ms 1 4373 0.37 MB/sec execute 39 sec latency 25757.008 ms 1 4373 0.36 MB/sec execute 40 sec latency 26757.170 ms 1 4373 0.35 MB/sec execute 41 sec latency 27757.314 ms 1 4373 0.34 MB/sec execute 42 sec latency 28757.499 ms 1 4470 0.34 MB/sec execute 43 sec latency 29440.327 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4703 0.36 MB/sec execute 44 sec latency 42.233 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5019 0.42 MB/sec execute 45 sec latency 18.193 ms 1 5225 0.42 MB/sec execute 46 sec latency 62.655 ms 1 5649 0.42 MB/sec execute 47 sec latency 26.179 ms 1 5946 0.43 MB/sec execute 48 sec latency 61.125 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10752 2197888 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 63488 3562496 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3622912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 88064 7185408 2% /mnt/lustre 1 6140 0.45 MB/sec execute 49 sec latency 61.595 ms 1 6552 0.53 MB/sec execute 50 sec latency 54.772 ms 1 6925 0.56 MB/sec execute 51 sec latency 50.464 ms test_26 fail mds1 6 times 1 7240 0.61 MB/sec execute 52 sec latency 18.632 ms Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server 1 7376 0.62 MB/sec execute 53 sec latency 399.141 ms reboot facets: mds1 1 7376 0.60 MB/sec execute 54 sec latency 1399.374 ms 1 7376 0.59 MB/sec execute 55 sec latency 2399.674 ms 1 7376 0.58 MB/sec execute 56 sec latency 3399.931 ms 1 7376 0.57 MB/sec execute 57 sec latency 4400.232 ms 1 7376 0.56 MB/sec execute 58 sec latency 5400.426 ms 1 7376 0.55 MB/sec execute 59 sec latency 6400.696 ms 1 7376 0.54 MB/sec execute 60 sec latency 7400.997 ms 1 7376 0.54 MB/sec execute 61 sec latency 8401.242 ms 1 7376 0.53 MB/sec execute 62 sec latency 9401.449 ms 1 7376 0.52 MB/sec execute 63 sec latency 10401.645 ms Failover mds1 to oleg414-server mount facets: mds1 1 7376 0.51 MB/sec execute 64 sec latency 11401.842 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7376 0.50 MB/sec execute 65 sec latency 12402.037 ms 1 7376 0.49 MB/sec execute 66 sec latency 13402.230 ms 1 7376 0.49 MB/sec execute 67 sec latency 14402.408 ms 1 7376 0.48 MB/sec execute 68 sec latency 15402.534 ms 1 7376 0.47 MB/sec execute 69 sec latency 16402.720 ms 1 7376 0.47 MB/sec execute 70 sec latency 17403.012 ms 1 7376 0.46 MB/sec execute 71 sec latency 18403.170 ms oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 1 7376 0.45 MB/sec execute 72 sec latency 19403.320 ms 1 7376 0.45 MB/sec execute 73 sec latency 20403.453 ms 1 7376 0.44 MB/sec execute 74 sec latency 21403.690 ms 1 7376 0.44 MB/sec execute 75 sec latency 22403.934 ms 1 7376 0.43 MB/sec execute 76 sec latency 23404.182 ms 1 7376 0.42 MB/sec execute 77 sec latency 24404.430 ms 1 7376 0.42 MB/sec execute 78 sec latency 25404.578 ms 1 7376 0.41 MB/sec execute 79 sec latency 26404.792 ms 1 7376 0.41 MB/sec execute 80 sec latency 27404.969 ms 1 7376 0.40 MB/sec execute 81 sec latency 28405.146 ms 1 7376 0.40 MB/sec execute 82 sec latency 29405.369 ms 1 7376 0.39 MB/sec execute 83 sec latency 30405.561 ms 1 7376 0.39 MB/sec execute 84 sec latency 31405.735 ms 1 7543 0.39 MB/sec execute 85 sec latency 31685.771 ms oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7812 0.39 MB/sec execute 86 sec latency 74.934 ms 1 8081 0.39 MB/sec execute 87 sec latency 39.838 ms 1 8530 0.43 MB/sec execute 88 sec latency 38.948 ms 1 8806 0.43 MB/sec execute 89 sec latency 80.984 ms 1 9201 0.43 MB/sec execute 90 sec latency 60.574 ms 1 9567 0.45 MB/sec execute 91 sec latency 51.835 ms 1 10083 0.49 MB/sec execute 92 sec latency 65.418 ms 1 10715 0.54 MB/sec execute 93 sec latency 34.633 ms 1 11013 0.55 MB/sec execute 94 sec latency 13.255 ms 1 11325 0.55 MB/sec execute 95 sec latency 24.851 ms 1 11532 0.55 MB/sec execute 96 sec latency 83.847 ms 1 11752 0.56 MB/sec execute 97 sec latency 63.952 ms 1 12149 0.58 MB/sec execute 98 sec latency 50.335 ms 1 12372 0.58 MB/sec execute 99 sec latency 65.772 ms 1 cleanup 100 sec 0 cleanup 100 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1901 26.068 31685.759 Close 1405 35.248 29440.310 Rename 78 13.178 24.830 Unlink 375 4.468 17.399 Qpathinfo 1753 2.519 17.552 Qfileinfo 301 0.399 2.409 Qfsinfo 306 0.461 5.368 Sfileinfo 150 5.839 13.506 Find 659 1.154 13.816 WriteX 924 2.078 7.855 ReadX 2927 0.037 3.838 LockX 6 1.528 2.352 UnlockX 6 1.740 2.466 Flush 132 30.177 83.834 Throughput 0.579533 MB/sec 1 clients 1 procs max_latency=31685.771 ms stopping dbench on /mnt/lustre at Wed Jul 31 23:45:53 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jul 31 23:45:54 EDT 2024 waiting for dbench pid 22835 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 26.187 ms 1 540 9.51 MB/sec warmup 2 sec latency 19.802 ms 1 802 7.16 MB/sec warmup 3 sec latency 98.876 ms 1 1056 5.72 MB/sec warmup 4 sec latency 67.961 ms 1 1420 5.20 MB/sec warmup 5 sec latency 26.649 ms 1 1757 4.38 MB/sec warmup 6 sec latency 58.038 ms 1 2138 3.81 MB/sec warmup 7 sec latency 41.297 ms 1 2428 3.51 MB/sec warmup 8 sec latency 68.424 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Jul 31 23:46:03 EDT 2024 with return code 0 22835 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 22835 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished