== replay-dual test 26: dbench and tar with mds failover ========================================================== 08:19:52 (1743509992) Starting client oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre Started clients oleg453-client.virtnet: 192.168.204.153@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 14957 Started dbench loop with 14958 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Apr 1 08:19:54 EDT 2025 waiting for dbench pid 14989 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 271 10.18 MB/sec warmup 1 sec latency 19.842 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3712 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3679232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3689472 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7368704 1% /mnt/lustre 1 522 9.03 MB/sec warmup 2 sec latency 17.539 ms 1 799 7.16 MB/sec warmup 3 sec latency 42.437 ms test_26 fail mds1 1 times 1 1119 5.74 MB/sec warmup 4 sec latency 30.685 ms Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 1 1410 5.20 MB/sec warmup 5 sec latency 297.850 ms 08:19:59 (1743509999) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server 1 1410 4.34 MB/sec warmup 6 sec latency 1298.035 ms 1 1410 3.72 MB/sec warmup 7 sec latency 2298.268 ms 1 1410 3.25 MB/sec warmup 8 sec latency 3298.502 ms 1 1410 2.89 MB/sec warmup 9 sec latency 4298.689 ms 1 1410 2.60 MB/sec warmup 10 sec latency 5298.874 ms 1 1410 2.36 MB/sec warmup 11 sec latency 6299.077 ms 1 1410 2.17 MB/sec warmup 12 sec latency 7299.313 ms 1 1410 2.00 MB/sec warmup 13 sec latency 8299.504 ms 1 1410 1.86 MB/sec warmup 14 sec latency 9299.689 ms 1 1410 1.73 MB/sec warmup 15 sec latency 10299.896 ms Failover mds1 to oleg453-server mount facets: mds1 1 1410 1.63 MB/sec warmup 16 sec latency 11300.079 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1410 1.53 MB/sec warmup 17 sec latency 12300.313 ms 1 1410 1.45 MB/sec warmup 18 sec latency 13300.573 ms oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:20:13 (1743510013) targets are mounted 08:20:13 (1743510013) facet_failover done 1 1410 1.37 MB/sec warmup 19 sec latency 14300.856 ms 1 1410 0.00 MB/sec execute 1 sec latency 16301.105 ms 1 1457 0.01 MB/sec execute 2 sec latency 17112.411 ms oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1698 0.07 MB/sec execute 3 sec latency 49.350 ms 1 2026 0.13 MB/sec execute 4 sec latency 26.051 ms 1 2358 0.38 MB/sec execute 5 sec latency 42.665 ms 1 2566 0.48 MB/sec execute 6 sec latency 55.592 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3575808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3553280 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7129088 1% /mnt/lustre 1 2922 1.07 MB/sec execute 7 sec latency 56.183 ms 1 3341 1.17 MB/sec execute 8 sec latency 49.242 ms 1 3647 1.40 MB/sec execute 9 sec latency 21.972 ms test_26 fail mds1 2 times 1 3816 1.37 MB/sec execute 10 sec latency 25.468 ms Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 1 3955 1.28 MB/sec execute 11 sec latency 420.651 ms 08:20:25 (1743510025) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server 1 3955 1.17 MB/sec execute 12 sec latency 1420.750 ms 1 3955 1.08 MB/sec execute 13 sec latency 2421.012 ms 1 3955 1.00 MB/sec execute 14 sec latency 3421.212 ms 1 3955 0.94 MB/sec execute 15 sec latency 4421.392 ms 1 3955 0.88 MB/sec execute 16 sec latency 5421.688 ms 1 3955 0.83 MB/sec execute 17 sec latency 6421.990 ms 1 3955 0.78 MB/sec execute 18 sec latency 7422.238 ms 1 3955 0.74 MB/sec execute 19 sec latency 8422.508 ms 1 3955 0.70 MB/sec execute 20 sec latency 9422.715 ms 1 3955 0.67 MB/sec execute 21 sec latency 10422.955 ms Failover mds1 to oleg453-server mount facets: mds1 1 3955 0.64 MB/sec execute 22 sec latency 11423.177 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3955 0.61 MB/sec execute 23 sec latency 12423.413 ms oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:20:38 (1743510038) targets are mounted 08:20:38 (1743510038) facet_failover done 1 3955 0.59 MB/sec execute 24 sec latency 13423.658 ms 1 3955 0.56 MB/sec execute 25 sec latency 14423.829 ms 1 3955 0.54 MB/sec execute 26 sec latency 15424.060 ms 1 3955 0.52 MB/sec execute 27 sec latency 16424.286 ms 1 3955 0.50 MB/sec execute 28 sec latency 17424.479 ms 1 3955 0.48 MB/sec execute 29 sec latency 18424.708 ms 1 3955 0.47 MB/sec execute 30 sec latency 19424.939 ms 1 3955 0.45 MB/sec execute 31 sec latency 20425.107 ms 1 3955 0.44 MB/sec execute 32 sec latency 21425.275 ms oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4133 0.43 MB/sec execute 33 sec latency 21583.577 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4333 0.43 MB/sec execute 34 sec latency 77.055 ms 1 4538 0.43 MB/sec execute 35 sec latency 36.235 ms 1 4839 0.45 MB/sec execute 36 sec latency 31.879 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6272 2202240 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3524608 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3481600 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 62464 7006208 1% /mnt/lustre 1 5185 0.52 MB/sec execute 37 sec latency 40.997 ms 1 5426 0.51 MB/sec execute 38 sec latency 44.938 ms 1 5722 0.51 MB/sec execute 39 sec latency 33.357 ms test_26 fail mds1 3 times 1 5949 0.53 MB/sec execute 40 sec latency 49.852 ms Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 1 6132 0.54 MB/sec execute 41 sec latency 229.561 ms 08:20:55 (1743510055) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server 1 6132 0.53 MB/sec execute 42 sec latency 1229.719 ms 1 6132 0.51 MB/sec execute 43 sec latency 2229.982 ms 1 6132 0.50 MB/sec execute 44 sec latency 3230.187 ms 1 6132 0.49 MB/sec execute 45 sec latency 4230.367 ms 1 6132 0.48 MB/sec execute 46 sec latency 5230.546 ms 1 6132 0.47 MB/sec execute 47 sec latency 6230.775 ms 1 6132 0.46 MB/sec execute 48 sec latency 7230.986 ms 1 6132 0.45 MB/sec execute 49 sec latency 8231.196 ms 1 6132 0.44 MB/sec execute 50 sec latency 9231.481 ms 1 6132 0.43 MB/sec execute 51 sec latency 10231.745 ms Failover mds1 to oleg453-server mount facets: mds1 1 6132 0.42 MB/sec execute 52 sec latency 11231.918 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6132 0.42 MB/sec execute 53 sec latency 12232.159 ms 1 6132 0.41 MB/sec execute 54 sec latency 13232.409 ms oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:21:09 (1743510069) targets are mounted 08:21:09 (1743510069) facet_failover done 1 6132 0.40 MB/sec execute 55 sec latency 14232.636 ms 1 6132 0.39 MB/sec execute 56 sec latency 15232.786 ms 1 6132 0.39 MB/sec execute 57 sec latency 16232.989 ms 1 6132 0.38 MB/sec execute 58 sec latency 17233.292 ms 1 6132 0.37 MB/sec execute 59 sec latency 18233.547 ms 1 6132 0.37 MB/sec execute 60 sec latency 19233.805 ms 1 6132 0.36 MB/sec execute 61 sec latency 20233.973 ms 1 6132 0.36 MB/sec execute 62 sec latency 21234.245 ms oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 6471 0.42 MB/sec execute 63 sec latency 21520.995 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6920 0.45 MB/sec execute 64 sec latency 28.610 ms 1 7187 0.49 MB/sec execute 65 sec latency 23.833 ms 1 7356 0.50 MB/sec execute 66 sec latency 26.903 ms 1 7492 0.49 MB/sec execute 67 sec latency 25.562 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7168 2201344 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3450880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3430400 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 65536 6881280 1% /mnt/lustre 1 7617 0.49 MB/sec execute 68 sec latency 25.844 ms 1 7794 0.48 MB/sec execute 69 sec latency 65.180 ms 1 7914 0.48 MB/sec execute 70 sec latency 68.490 ms test_26 fail mds1 4 times Failing mds1 on oleg453-server 1 8075 0.48 MB/sec execute 71 sec latency 53.777 ms Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 1 8184 0.49 MB/sec execute 72 sec latency 472.390 ms 08:21:26 (1743510086) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server 1 8184 0.48 MB/sec execute 73 sec latency 1472.601 ms 1 8184 0.47 MB/sec execute 74 sec latency 2472.825 ms 1 8184 0.47 MB/sec execute 75 sec latency 3473.052 ms 1 8184 0.46 MB/sec execute 76 sec latency 4473.337 ms 1 8184 0.46 MB/sec execute 77 sec latency 5473.590 ms 1 8184 0.45 MB/sec execute 78 sec latency 6473.812 ms 1 8184 0.44 MB/sec execute 79 sec latency 7474.074 ms 1 8184 0.44 MB/sec execute 80 sec latency 8474.356 ms 1 8184 0.43 MB/sec execute 81 sec latency 9474.565 ms 1 8184 0.43 MB/sec execute 82 sec latency 10474.797 ms Failover mds1 to oleg453-server mount facets: mds1 1 8184 0.42 MB/sec execute 83 sec latency 11475.035 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8184 0.42 MB/sec execute 84 sec latency 12475.326 ms oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all 1 8184 0.41 MB/sec execute 85 sec latency 13475.532 ms pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:21:39 (1743510099) targets are mounted 08:21:39 (1743510099) facet_failover done 1 8184 0.41 MB/sec execute 86 sec latency 14475.733 ms 1 8184 0.40 MB/sec execute 87 sec latency 15475.865 ms 1 8184 0.40 MB/sec execute 88 sec latency 16476.085 ms 1 8184 0.39 MB/sec execute 89 sec latency 17476.327 ms 1 8184 0.39 MB/sec execute 90 sec latency 18476.498 ms 1 8184 0.39 MB/sec execute 91 sec latency 19476.653 ms 1 8184 0.38 MB/sec execute 92 sec latency 20476.903 ms oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 8378 0.38 MB/sec execute 93 sec latency 20501.348 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8699 0.41 MB/sec execute 94 sec latency 40.612 ms 1 8893 0.41 MB/sec execute 95 sec latency 38.106 ms 1 9087 0.40 MB/sec execute 96 sec latency 70.483 ms 1 9274 0.40 MB/sec execute 97 sec latency 28.890 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8832 2199680 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3569664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3561472 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7131136 1% /mnt/lustre 1 9581 0.42 MB/sec execute 98 sec latency 47.422 ms 1 10099 0.46 MB/sec execute 99 sec latency 27.597 ms test_26 fail mds1 5 times 1 cleanup 100 sec Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1550 63.244 21583.569 Close 1164 2.034 7.196 Rename 64 18.255 59.170 Unlink 288 5.577 19.824 Qpathinfo 1464 3.280 27.746 Qfileinfo 268 0.503 3.815 Qfsinfo 255 0.653 3.723 Sfileinfo 116 10.639 38.228 Find 570 1.272 16.447 WriteX 793 2.206 8.531 ReadX 2607 0.148 30.822 LockX 6 2.039 2.997 UnlockX 6 1.963 2.632 Flush 105 27.048 77.040 Throughput 0.462138 MB/sec 1 clients 1 procs max_latency=21583.577 ms stopping dbench on /mnt/lustre at Tue Apr 1 08:21:54 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre 08:21:55 (1743510115) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:22:08 (1743510128) targets are mounted 08:22:08 (1743510128) facet_failover done removed 'client.txt' /mnt/lustre dbench successfully finished oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Apr 1 08:22:17 EDT 2025 waiting for dbench pid 20067 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 270 10.18 MB/sec warmup 1 sec latency 24.697 ms 1 460 8.08 MB/sec warmup 2 sec latency 24.899 ms 1 695 7.06 MB/sec warmup 3 sec latency 93.767 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10240 2198272 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3336192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 50176 3407872 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 6744064 2% /mnt/lustre 1 899 5.42 MB/sec warmup 4 sec latency 40.405 ms 1 1126 4.60 MB/sec warmup 5 sec latency 52.436 ms 1 1474 4.34 MB/sec warmup 6 sec latency 18.736 ms test_26 fail mds1 6 times Failing mds1 on oleg453-server 1 1748 3.75 MB/sec warmup 7 sec latency 37.468 ms Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 1 1834 3.29 MB/sec warmup 8 sec latency 694.822 ms 08:22:25 (1743510145) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server 1 1834 2.93 MB/sec warmup 9 sec latency 1695.044 ms 1 1834 2.64 MB/sec warmup 10 sec latency 2695.271 ms 1 1834 2.40 MB/sec warmup 11 sec latency 3695.497 ms 1 1834 2.20 MB/sec warmup 12 sec latency 4695.791 ms 1 1834 2.03 MB/sec warmup 13 sec latency 5696.040 ms 1 1834 1.88 MB/sec warmup 14 sec latency 6696.251 ms 1 1834 1.76 MB/sec warmup 15 sec latency 7696.552 ms 1 1834 1.65 MB/sec warmup 16 sec latency 8696.798 ms 1 1834 1.55 MB/sec warmup 17 sec latency 9697.074 ms 1 1834 1.46 MB/sec warmup 18 sec latency 10697.372 ms Failover mds1 to oleg453-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1834 1.39 MB/sec warmup 19 sec latency 11697.659 ms oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:22:38 (1743510158) targets are mounted 08:22:38 (1743510158) facet_failover done 1 1834 0.00 MB/sec execute 1 sec latency 13698.033 ms 1 1834 0.00 MB/sec execute 2 sec latency 14698.270 ms 1 1834 0.00 MB/sec execute 3 sec latency 15698.543 ms 1 1834 0.00 MB/sec execute 4 sec latency 16698.797 ms 1 1834 0.00 MB/sec execute 5 sec latency 17699.067 ms 1 1834 0.00 MB/sec execute 6 sec latency 18699.409 ms 1 1834 0.00 MB/sec execute 7 sec latency 19699.663 ms 1 1834 0.00 MB/sec execute 8 sec latency 20699.951 ms 1 1836 0.00 MB/sec execute 9 sec latency 21697.923 ms oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2145 0.03 MB/sec execute 10 sec latency 29.027 ms 1 2399 0.15 MB/sec execute 11 sec latency 79.772 ms 1 2622 0.23 MB/sec execute 12 sec latency 38.413 ms 1 3000 0.55 MB/sec execute 13 sec latency 47.404 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 11392 2197120 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 53248 3423232 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3476480 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 6899712 2% /mnt/lustre 1 3562 0.86 MB/sec execute 14 sec latency 42.143 ms 1 3816 0.89 MB/sec execute 15 sec latency 22.569 ms 1 3986 0.86 MB/sec execute 16 sec latency 19.057 ms test_26 fail mds1 7 times 1 4227 0.81 MB/sec execute 17 sec latency 19.356 ms Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 1 4350 0.78 MB/sec execute 18 sec latency 369.067 ms 08:22:55 (1743510175) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server 1 4350 0.74 MB/sec execute 19 sec latency 1369.276 ms 1 4350 0.71 MB/sec execute 20 sec latency 2369.517 ms 1 4350 0.67 MB/sec execute 21 sec latency 3369.749 ms 1 4350 0.64 MB/sec execute 22 sec latency 4369.969 ms 1 4350 0.61 MB/sec execute 23 sec latency 5370.203 ms 1 4350 0.59 MB/sec execute 24 sec latency 6370.464 ms 1 4350 0.57 MB/sec execute 25 sec latency 7370.683 ms 1 4350 0.54 MB/sec execute 26 sec latency 8370.882 ms 1 4350 0.52 MB/sec execute 27 sec latency 9371.099 ms 1 4350 0.50 MB/sec execute 28 sec latency 10371.351 ms Failover mds1 to oleg453-server mount facets: mds1 1 4350 0.49 MB/sec execute 29 sec latency 11371.587 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4350 0.47 MB/sec execute 30 sec latency 12371.778 ms oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all 1 4350 0.46 MB/sec execute 31 sec latency 13372.002 ms pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 08:23:09 (1743510189) targets are mounted 08:23:09 (1743510189) facet_failover done 1 4350 0.44 MB/sec execute 32 sec latency 14372.237 ms 1 4350 0.43 MB/sec execute 33 sec latency 15372.476 ms 1 4350 0.42 MB/sec execute 34 sec latency 16372.700 ms 1 4350 0.40 MB/sec execute 35 sec latency 17372.882 ms 1 4350 0.39 MB/sec execute 36 sec latency 18373.059 ms 1 4350 0.38 MB/sec execute 37 sec latency 19373.191 ms 1 4350 0.37 MB/sec execute 38 sec latency 20373.347 ms oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4448 0.37 MB/sec execute 39 sec latency 20830.182 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4686 0.39 MB/sec execute 40 sec latency 42.570 ms 1 5025 0.46 MB/sec execute 41 sec latency 20.958 ms 1 5292 0.45 MB/sec execute 42 sec latency 47.240 ms 1 5469 0.45 MB/sec execute 43 sec latency 55.685 ms 1 5712 0.44 MB/sec execute 44 sec latency 22.579 ms 1 5927 0.46 MB/sec execute 45 sec latency 90.739 ms 1 6131 0.47 MB/sec execute 46 sec latency 51.180 ms 1 6477 0.56 MB/sec execute 47 sec latency 72.577 ms 1 6787 0.56 MB/sec execute 48 sec latency 65.885 ms 1 7134 0.64 MB/sec execute 49 sec latency 22.304 ms 1 7318 0.65 MB/sec execute 50 sec latency 25.380 ms 1 7479 0.64 MB/sec execute 51 sec latency 25.816 ms 1 7603 0.63 MB/sec execute 52 sec latency 24.672 ms 1 7788 0.62 MB/sec execute 53 sec latency 93.951 ms 1 7937 0.62 MB/sec execute 54 sec latency 66.364 ms 1 8091 0.61 MB/sec execute 55 sec latency 60.318 ms 1 8289 0.62 MB/sec execute 56 sec latency 52.309 ms 1 8519 0.66 MB/sec execute 57 sec latency 30.373 ms 1 8680 0.66 MB/sec execute 58 sec latency 88.959 ms 1 8883 0.65 MB/sec execute 59 sec latency 51.306 ms 1 9104 0.64 MB/sec execute 60 sec latency 36.871 ms 1 9477 0.65 MB/sec execute 61 sec latency 40.253 ms 1 9886 0.71 MB/sec execute 62 sec latency 39.556 ms 1 10460 0.75 MB/sec execute 63 sec latency 43.912 ms 1 10871 0.81 MB/sec execute 64 sec latency 12.735 ms 1 11081 0.80 MB/sec execute 65 sec latency 18.877 ms 1 11315 0.79 MB/sec execute 66 sec latency 24.671 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Apr 1 08:23:43 EDT 2025 with return code 0 20067 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20067 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished