== replay-dual test 26: dbench and tar with mds failover ========================================================== 12:48:58 (1721407738) Starting client oleg460-client.virtnet: -o user_xattr,flock oleg460-server@tcp:/lustre /mnt/lustre Started clients oleg460-client.virtnet: 192.168.204.160@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15808 Started dbench loop with 15809 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jul 19 12:49:00 EDT 2024 waiting for dbench pid 15834 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 297 10.96 MB/sec warmup 1 sec latency 17.491 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3456 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3623936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3689472 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7313408 1% /mnt/lustre 1 606 10.48 MB/sec warmup 2 sec latency 20.357 ms 1 880 7.22 MB/sec warmup 3 sec latency 44.679 ms 1 1229 5.77 MB/sec warmup 4 sec latency 34.130 ms test_26 fail mds1 1 times Failing mds1 on oleg460-server Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 1 1643 5.24 MB/sec warmup 5 sec latency 56.309 ms reboot facets: mds1 1 1650 4.37 MB/sec warmup 6 sec latency 970.954 ms 1 1650 3.74 MB/sec warmup 7 sec latency 1971.150 ms 1 1650 3.28 MB/sec warmup 8 sec latency 2971.358 ms 1 1650 2.91 MB/sec warmup 9 sec latency 3971.599 ms 1 1650 2.62 MB/sec warmup 10 sec latency 4971.787 ms 1 1650 2.38 MB/sec warmup 11 sec latency 5972.026 ms 1 1650 2.18 MB/sec warmup 12 sec latency 6972.268 ms 1 1650 2.02 MB/sec warmup 13 sec latency 7972.453 ms 1 1650 1.87 MB/sec warmup 14 sec latency 8972.699 ms 1 1650 1.75 MB/sec warmup 15 sec latency 9972.945 ms Failover mds1 to oleg460-server 1 1650 1.64 MB/sec warmup 16 sec latency 10973.157 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1650 1.54 MB/sec warmup 17 sec latency 11973.368 ms 1 1650 1.46 MB/sec warmup 18 sec latency 12973.589 ms 1 1650 1.38 MB/sec warmup 19 sec latency 13973.768 ms 1 1650 0.00 MB/sec execute 1 sec latency 15974.197 ms 1 1650 0.00 MB/sec execute 2 sec latency 16974.428 ms 1 1650 0.00 MB/sec execute 3 sec latency 17974.590 ms oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 1 1650 0.00 MB/sec execute 4 sec latency 18974.691 ms Started lustre-MDT0000 1 1650 0.00 MB/sec execute 5 sec latency 19974.830 ms 1 1650 0.00 MB/sec execute 6 sec latency 20975.205 ms 1 1650 0.00 MB/sec execute 7 sec latency 21975.476 ms 1 1650 0.00 MB/sec execute 8 sec latency 22975.718 ms 1 1650 0.00 MB/sec execute 9 sec latency 23975.894 ms 1 1650 0.00 MB/sec execute 10 sec latency 24976.090 ms 1 1650 0.00 MB/sec execute 11 sec latency 25976.390 ms 1 1650 0.00 MB/sec execute 12 sec latency 26976.677 ms 1 1650 0.00 MB/sec execute 13 sec latency 27976.919 ms 1 1650 0.00 MB/sec execute 14 sec latency 28977.156 ms 1 1650 0.00 MB/sec execute 15 sec latency 29977.432 ms 1 1650 0.00 MB/sec execute 16 sec latency 30977.678 ms 1 1650 0.00 MB/sec execute 17 sec latency 31977.872 ms 1 1650 0.00 MB/sec execute 18 sec latency 32979.380 ms 1 1810 0.01 MB/sec execute 19 sec latency 33506.408 ms oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2199 0.03 MB/sec execute 20 sec latency 48.265 ms 1 2559 0.13 MB/sec execute 21 sec latency 42.772 ms 1 3204 0.37 MB/sec execute 22 sec latency 41.897 ms 1 3665 0.54 MB/sec execute 23 sec latency 16.849 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4736 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 44032 3659776 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3700736 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 60416 7360512 1% /mnt/lustre 1 3932 0.58 MB/sec execute 24 sec latency 18.919 ms 1 4236 0.56 MB/sec execute 25 sec latency 26.251 ms 1 4484 0.56 MB/sec execute 26 sec latency 62.718 ms test_26 fail mds1 2 times 1 4839 0.59 MB/sec execute 27 sec latency 35.294 ms Failing mds1 on oleg460-server Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 1 5089 0.68 MB/sec execute 28 sec latency 465.538 ms reboot facets: mds1 1 5089 0.66 MB/sec execute 29 sec latency 1465.756 ms 1 5089 0.63 MB/sec execute 30 sec latency 2465.984 ms 1 5089 0.61 MB/sec execute 31 sec latency 3466.269 ms 1 5089 0.59 MB/sec execute 32 sec latency 4466.481 ms 1 5089 0.58 MB/sec execute 33 sec latency 5466.677 ms 1 5089 0.56 MB/sec execute 34 sec latency 6466.874 ms 1 5089 0.54 MB/sec execute 35 sec latency 7467.048 ms 1 5089 0.53 MB/sec execute 36 sec latency 8467.238 ms 1 5089 0.51 MB/sec execute 37 sec latency 9467.428 ms 1 5089 0.50 MB/sec execute 38 sec latency 10467.607 ms Failover mds1 to oleg460-server mount facets: mds1 1 5089 0.49 MB/sec execute 39 sec latency 11467.782 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5089 0.48 MB/sec execute 40 sec latency 12467.930 ms oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 1 5089 0.46 MB/sec execute 41 sec latency 13468.141 ms Started lustre-MDT0000 1 5089 0.45 MB/sec execute 42 sec latency 14468.354 ms 1 5089 0.44 MB/sec execute 43 sec latency 15468.537 ms 1 5089 0.43 MB/sec execute 44 sec latency 16468.774 ms 1 5360 0.43 MB/sec execute 45 sec latency 16631.519 ms oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5746 0.43 MB/sec execute 46 sec latency 48.565 ms 1 6049 0.46 MB/sec execute 47 sec latency 60.768 ms 1 6457 0.54 MB/sec execute 48 sec latency 53.214 ms 1 6894 0.58 MB/sec execute 49 sec latency 55.428 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6912 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 47104 3684352 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3706880 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7391232 1% /mnt/lustre 1 7304 0.65 MB/sec execute 50 sec latency 20.985 ms 1 7530 0.65 MB/sec execute 51 sec latency 17.817 ms 1 7712 0.63 MB/sec execute 52 sec latency 24.356 ms test_26 fail mds1 3 times Failing mds1 on oleg460-server 1 7929 0.63 MB/sec execute 53 sec latency 86.873 ms Stopping /mnt/lustre-mds1 (opts:) on oleg460-server reboot facets: mds1 1 8012 0.62 MB/sec execute 54 sec latency 754.152 ms 1 8012 0.61 MB/sec execute 55 sec latency 1754.351 ms 1 8012 0.60 MB/sec execute 56 sec latency 2754.615 ms 1 8012 0.59 MB/sec execute 57 sec latency 3754.927 ms 1 8012 0.58 MB/sec execute 58 sec latency 4755.190 ms 1 8012 0.57 MB/sec execute 59 sec latency 5755.446 ms 1 8012 0.56 MB/sec execute 60 sec latency 6755.709 ms 1 8012 0.55 MB/sec execute 61 sec latency 7755.927 ms 1 8012 0.54 MB/sec execute 62 sec latency 8756.102 ms 1 8012 0.54 MB/sec execute 63 sec latency 9756.371 ms 1 8012 0.53 MB/sec execute 64 sec latency 10756.554 ms Failover mds1 to oleg460-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8012 0.52 MB/sec execute 65 sec latency 11756.694 ms 1 8012 0.51 MB/sec execute 66 sec latency 12756.828 ms 1 8012 0.50 MB/sec execute 67 sec latency 13756.997 ms 1 8012 0.50 MB/sec execute 68 sec latency 14757.153 ms oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 1 8012 0.49 MB/sec execute 69 sec latency 15757.284 ms Started lustre-MDT0000 1 8012 0.48 MB/sec execute 70 sec latency 16757.480 ms 1 8012 0.47 MB/sec execute 71 sec latency 17757.686 ms 1 8012 0.47 MB/sec execute 72 sec latency 18757.811 ms 1 8012 0.46 MB/sec execute 73 sec latency 19758.006 ms 1 8012 0.46 MB/sec execute 74 sec latency 20758.270 ms 1 8012 0.45 MB/sec execute 75 sec latency 21758.515 ms 1 8012 0.44 MB/sec execute 76 sec latency 22758.755 ms 1 8012 0.44 MB/sec execute 77 sec latency 23758.923 ms 1 8012 0.43 MB/sec execute 78 sec latency 24759.178 ms 1 8012 0.43 MB/sec execute 79 sec latency 25759.497 ms 1 8012 0.42 MB/sec execute 80 sec latency 26759.746 ms 1 8012 0.42 MB/sec execute 81 sec latency 27759.998 ms 1 8012 0.41 MB/sec execute 82 sec latency 28760.211 ms 1 8012 0.41 MB/sec execute 83 sec latency 29760.389 ms 1 8209 0.42 MB/sec execute 84 sec latency 30129.628 ms oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8619 0.45 MB/sec execute 85 sec latency 52.047 ms 1 8942 0.45 MB/sec execute 86 sec latency 41.021 ms 1 9254 0.45 MB/sec execute 87 sec latency 44.568 ms 1 9520 0.46 MB/sec execute 88 sec latency 68.499 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8576 2200064 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3567616 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3604480 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7172096 1% /mnt/lustre 1 10002 0.50 MB/sec execute 89 sec latency 46.986 ms 1 10465 0.53 MB/sec execute 90 sec latency 45.942 ms 1 10852 0.57 MB/sec execute 91 sec latency 22.191 ms test_26 fail mds1 4 times Failing mds1 on oleg460-server 1 11045 0.57 MB/sec execute 92 sec latency 24.789 ms Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 1 11126 0.56 MB/sec execute 93 sec latency 573.397 ms reboot facets: mds1 1 11126 0.55 MB/sec execute 94 sec latency 1573.639 ms 1 11126 0.55 MB/sec execute 95 sec latency 2573.921 ms 1 11126 0.54 MB/sec execute 96 sec latency 3574.159 ms 1 11126 0.54 MB/sec execute 97 sec latency 4574.427 ms 1 11126 0.53 MB/sec execute 98 sec latency 5574.593 ms 1 11126 0.53 MB/sec execute 99 sec latency 6574.801 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec Failover mds1 to oleg460-server mount facets: mds1 1 cleanup 104 sec Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 105 sec oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 0 cleanup 110 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1632 37.238 30129.612 Close 1199 29.538 33506.389 Rename 68 11.166 28.842 Unlink 323 4.298 18.538 Qpathinfo 1497 13.339 16631.492 Qfileinfo 261 0.358 2.139 Qfsinfo 271 0.404 2.620 Sfileinfo 116 5.265 10.518 Find 570 1.073 13.047 WriteX 800 1.904 7.496 ReadX 2626 0.058 29.477 LockX 6 1.657 2.324 UnlockX 6 2.000 2.807 Flush 102 27.450 86.859 Throughput 0.526175 MB/sec 1 clients 1 procs max_latency=33506.408 ms stopping dbench on /mnt/lustre at Fri Jul 19 12:51:10 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jul 19 12:51:11 EDT 2024 waiting for dbench pid 20252 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 271 10.18 MB/sec warmup 1 sec latency 24.342 ms 1 584 9.98 MB/sec warmup 2 sec latency 20.326 ms 1 848 7.19 MB/sec warmup 3 sec latency 121.804 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10112 2198528 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 28672 3528704 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3533824 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 68608 7062528 1% /mnt/lustre 1 1173 5.75 MB/sec warmup 4 sec latency 32.630 ms 1 1619 5.23 MB/sec warmup 5 sec latency 29.339 ms 1 1984 4.41 MB/sec warmup 6 sec latency 36.268 ms test_26 fail mds1 5 times Failing mds1 on oleg460-server Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 1 2358 3.98 MB/sec warmup 7 sec latency 53.370 ms reboot facets: mds1 1 2412 3.50 MB/sec warmup 8 sec latency 806.718 ms 1 2412 3.11 MB/sec warmup 9 sec latency 1806.898 ms 1 2412 2.80 MB/sec warmup 10 sec latency 2807.078 ms 1 2412 2.55 MB/sec warmup 11 sec latency 3807.400 ms 1 2412 2.34 MB/sec warmup 12 sec latency 4807.724 ms 1 2412 2.16 MB/sec warmup 13 sec latency 5807.987 ms 1 2412 2.00 MB/sec warmup 14 sec latency 6808.222 ms 1 2412 1.87 MB/sec warmup 15 sec latency 7808.508 ms 1 2412 1.75 MB/sec warmup 16 sec latency 8808.756 ms 1 2412 1.65 MB/sec warmup 17 sec latency 9809.017 ms Failover mds1 to oleg460-server 1 2412 1.56 MB/sec warmup 18 sec latency 10809.208 ms mount facets: mds1 1 2412 1.47 MB/sec warmup 19 sec latency 11809.490 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2412 0.00 MB/sec execute 1 sec latency 13809.867 ms 1 2412 0.00 MB/sec execute 2 sec latency 14810.115 ms oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all 8 1 2412 0.00 MB/sec execute 3 sec latency 15810.407 ms pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2412 0.00 MB/sec execute 4 sec latency 16810.678 ms 1 2412 0.00 MB/sec execute 5 sec latency 17811.013 ms 1 2412 0.00 MB/sec execute 6 sec latency 18811.337 ms 1 2412 0.00 MB/sec execute 7 sec latency 19811.564 ms 1 2412 0.00 MB/sec execute 8 sec latency 20811.813 ms 1 2412 0.00 MB/sec execute 9 sec latency 21812.109 ms 1 2412 0.00 MB/sec execute 10 sec latency 22812.416 ms 1 2412 0.00 MB/sec execute 11 sec latency 23812.647 ms 1 2412 0.00 MB/sec execute 12 sec latency 24812.931 ms 1 2412 0.00 MB/sec execute 13 sec latency 25813.177 ms 1 2412 0.00 MB/sec execute 14 sec latency 26813.352 ms 1 2412 0.00 MB/sec execute 15 sec latency 27813.581 ms 1 2412 0.00 MB/sec execute 16 sec latency 28813.858 ms 1 2412 0.00 MB/sec execute 17 sec latency 29814.002 ms 1 2412 0.00 MB/sec execute 18 sec latency 30814.191 ms 1 2887 0.23 MB/sec execute 19 sec latency 30980.054 ms oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3364 0.37 MB/sec execute 20 sec latency 40.453 ms 1 3755 0.56 MB/sec execute 21 sec latency 15.075 ms 1 3927 0.55 MB/sec execute 22 sec latency 20.080 ms 1 4076 0.52 MB/sec execute 23 sec latency 23.354 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 11008 2197632 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3703808 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3704832 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 91136 7408640 2% /mnt/lustre 1 4297 0.52 MB/sec execute 24 sec latency 95.888 ms 1 4518 0.52 MB/sec execute 25 sec latency 55.986 ms 1 4808 0.55 MB/sec execute 26 sec latency 54.267 ms test_26 fail mds1 6 times Failing mds1 on oleg460-server 1 5186 0.64 MB/sec execute 27 sec latency 59.345 ms Stopping /mnt/lustre-mds1 (opts:) on oleg460-server reboot facets: mds1 1 5261 0.62 MB/sec execute 28 sec latency 706.787 ms 1 5261 0.60 MB/sec execute 29 sec latency 1707.066 ms 1 5261 0.58 MB/sec execute 30 sec latency 2707.454 ms 1 5261 0.56 MB/sec execute 31 sec latency 3707.707 ms 1 5261 0.54 MB/sec execute 32 sec latency 4708.011 ms 1 5261 0.52 MB/sec execute 33 sec latency 5708.353 ms 1 5261 0.51 MB/sec execute 34 sec latency 6708.612 ms 1 5261 0.49 MB/sec execute 35 sec latency 7708.909 ms 1 5261 0.48 MB/sec execute 36 sec latency 8709.212 ms 1 5261 0.47 MB/sec execute 37 sec latency 9709.557 ms 1 5261 0.46 MB/sec execute 38 sec latency 10709.791 ms Failover mds1 to oleg460-server mount facets: mds1 1 5261 0.44 MB/sec execute 39 sec latency 11709.976 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5261 0.43 MB/sec execute 40 sec latency 12710.231 ms 1 5261 0.42 MB/sec execute 41 sec latency 13710.562 ms 1 5261 0.41 MB/sec execute 42 sec latency 14710.924 ms 1 5261 0.40 MB/sec execute 43 sec latency 15711.227 ms 1 5261 0.39 MB/sec execute 44 sec latency 16711.536 ms 1 5261 0.38 MB/sec execute 45 sec latency 17711.901 ms 1 5261 0.38 MB/sec execute 46 sec latency 18712.088 ms oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 1 5261 0.37 MB/sec execute 47 sec latency 19712.344 ms Started lustre-MDT0000 1 5261 0.36 MB/sec execute 48 sec latency 20712.690 ms 1 5261 0.35 MB/sec execute 49 sec latency 21713.018 ms 1 5261 0.35 MB/sec execute 50 sec latency 22713.256 ms 1 5261 0.34 MB/sec execute 51 sec latency 23713.557 ms 1 5261 0.33 MB/sec execute 52 sec latency 24713.832 ms 1 5261 0.33 MB/sec execute 53 sec latency 25714.071 ms 1 5261 0.32 MB/sec execute 54 sec latency 26714.334 ms 1 5261 0.31 MB/sec execute 55 sec latency 27714.592 ms 1 5261 0.31 MB/sec execute 56 sec latency 28714.856 ms 1 5261 0.30 MB/sec execute 57 sec latency 29715.085 ms 1 5261 0.30 MB/sec execute 58 sec latency 30715.294 ms 1 5261 0.29 MB/sec execute 59 sec latency 31715.573 ms 1 5261 0.29 MB/sec execute 60 sec latency 32715.769 ms 1 5422 0.29 MB/sec execute 61 sec latency 32841.462 ms oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5691 0.29 MB/sec execute 62 sec latency 76.630 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5937 0.30 MB/sec execute 63 sec latency 68.304 ms 1 6162 0.31 MB/sec execute 64 sec latency 72.326 ms 1 6787 0.39 MB/sec execute 65 sec latency 61.557 ms 1 7240 0.45 MB/sec execute 66 sec latency 19.376 ms 1 7478 0.46 MB/sec execute 67 sec latency 18.410 ms 1 7722 0.46 MB/sec execute 68 sec latency 21.823 ms 1 7960 0.46 MB/sec execute 69 sec latency 115.458 ms 1 8255 0.47 MB/sec execute 70 sec latency 49.191 ms 1 8600 0.51 MB/sec execute 71 sec latency 50.255 ms 1 8814 0.51 MB/sec execute 72 sec latency 63.300 ms 1 9133 0.50 MB/sec execute 73 sec latency 54.246 ms 1 9493 0.52 MB/sec execute 74 sec latency 73.525 ms 1 9819 0.56 MB/sec execute 75 sec latency 65.007 ms 1 10290 0.58 MB/sec execute 76 sec latency 58.804 ms 1 10736 0.63 MB/sec execute 77 sec latency 60.519 ms 1 11038 0.64 MB/sec execute 78 sec latency 20.716 ms 1 11231 0.64 MB/sec execute 79 sec latency 22.458 ms 1 11468 0.64 MB/sec execute 80 sec latency 94.131 ms 1 11790 0.65 MB/sec execute 81 sec latency 40.279 ms 1 12097 0.67 MB/sec execute 82 sec latency 22.454 ms 1 12344 0.67 MB/sec execute 83 sec latency 65.059 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Jul 19 12:52:54 EDT 2024 with return code 0 20252 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20252 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished