== replay-dual test 26: dbench and tar with mds failover ========================================================== 03:48:32 (1711093712) Starting client oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre Started clients oleg119-client.virtnet: 192.168.201.119@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 6403 Started dbench loop with 6405 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Mar 22 03:48:35 EDT 2024 waiting for dbench pid 6445 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2092 1285596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1868 1285820 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1816 3605204 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3580364 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3612 7185568 1% /mnt/lustre 1 113 5.23 MB/sec warmup 1 sec latency 79.646 ms 1 199 3.92 MB/sec warmup 2 sec latency 77.167 ms 1 287 3.56 MB/sec warmup 3 sec latency 53.483 ms 1 377 3.34 MB/sec warmup 4 sec latency 53.943 ms 1 511 3.54 MB/sec warmup 5 sec latency 41.144 ms test_26 fail mds1 1 times 1 678 3.53 MB/sec warmup 6 sec latency 48.134 ms Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 1 778 3.05 MB/sec warmup 7 sec latency 44.021 ms 1 815 2.68 MB/sec warmup 8 sec latency 687.516 ms 03:48:43 (1711093723) shut down 1 815 2.38 MB/sec warmup 9 sec latency 1688.886 ms 1 815 2.15 MB/sec warmup 10 sec latency 2689.201 ms 1 815 1.95 MB/sec warmup 11 sec latency 3689.845 ms 1 815 1.79 MB/sec warmup 12 sec latency 4690.045 ms 1 815 1.65 MB/sec warmup 13 sec latency 5690.273 ms 1 815 1.53 MB/sec warmup 14 sec latency 6690.475 ms 1 815 1.43 MB/sec warmup 15 sec latency 7691.346 ms 1 815 1.34 MB/sec warmup 16 sec latency 8691.527 ms 1 815 1.26 MB/sec warmup 17 sec latency 9695.129 ms 1 815 1.19 MB/sec warmup 18 sec latency 10695.303 ms Failover mds1 to oleg119-server mount facets: mds1 1 815 1.13 MB/sec warmup 19 sec latency 11695.485 ms 1 815 0.00 MB/sec execute 1 sec latency 13695.835 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 815 0.00 MB/sec execute 2 sec latency 14696.010 ms oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all 1 815 0.00 MB/sec execute 3 sec latency 15697.859 ms pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 03:48:59 (1711093739) targets are mounted 03:48:59 (1711093739) facet_failover done 1 815 0.00 MB/sec execute 4 sec latency 16699.615 ms 1 815 0.00 MB/sec execute 5 sec latency 17700.581 ms 1 815 0.00 MB/sec execute 6 sec latency 18702.148 ms 1 815 0.00 MB/sec execute 7 sec latency 19704.878 ms 1 818 0.00 MB/sec execute 8 sec latency 20567.132 ms oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 903 0.02 MB/sec execute 9 sec latency 45.101 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 981 0.04 MB/sec execute 10 sec latency 48.851 ms 1 1114 0.13 MB/sec execute 11 sec latency 38.214 ms 1 1214 0.13 MB/sec execute 12 sec latency 73.206 ms 1 1393 0.35 MB/sec execute 13 sec latency 45.478 ms 1 1492 0.33 MB/sec execute 14 sec latency 43.735 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2636 1285052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2404 1285284 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6888 3592076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31800 3565496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38688 7157572 1% /mnt/lustre 1 1567 0.31 MB/sec execute 15 sec latency 50.348 ms 1 1642 0.29 MB/sec execute 16 sec latency 54.923 ms 1 1729 0.28 MB/sec execute 17 sec latency 79.102 ms 1 1849 0.27 MB/sec execute 18 sec latency 41.569 ms 1 1982 0.26 MB/sec execute 19 sec latency 36.344 ms test_26 fail mds2 2 times 1 2060 0.26 MB/sec execute 20 sec latency 50.823 ms Failing mds2 on oleg119-server 1 2158 0.25 MB/sec execute 21 sec latency 76.718 ms Stopping /mnt/lustre-mds2 (opts:) on oleg119-server 1 2259 0.24 MB/sec execute 22 sec latency 114.117 ms 03:49:18 (1711093758) shut down 1 2259 0.23 MB/sec execute 23 sec latency 1116.255 ms 1 2259 0.22 MB/sec execute 24 sec latency 2118.448 ms 1 2259 0.21 MB/sec execute 25 sec latency 3118.675 ms 1 2259 0.20 MB/sec execute 26 sec latency 4119.010 ms 1 2259 0.20 MB/sec execute 27 sec latency 5119.212 ms 1 2259 0.19 MB/sec execute 28 sec latency 6120.694 ms 1 2259 0.18 MB/sec execute 29 sec latency 7122.101 ms 1 2259 0.18 MB/sec execute 30 sec latency 8122.293 ms 1 2259 0.17 MB/sec execute 31 sec latency 9122.536 ms 1 2259 0.17 MB/sec execute 32 sec latency 10122.722 ms Failover mds2 to oleg119-server mount facets: mds2 1 2259 0.16 MB/sec execute 33 sec latency 11126.322 ms 1 2259 0.16 MB/sec execute 34 sec latency 12127.223 ms 1 2259 0.15 MB/sec execute 35 sec latency 13127.464 ms 1 2259 0.15 MB/sec execute 36 sec latency 14127.694 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.14 MB/sec execute 37 sec latency 15128.277 ms 1 2259 0.14 MB/sec execute 38 sec latency 16128.482 ms 1 2259 0.14 MB/sec execute 39 sec latency 17128.991 ms oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 1 2259 0.13 MB/sec execute 40 sec latency 18129.191 ms Started lustre-MDT0001 03:49:36 (1711093776) targets are mounted 03:49:36 (1711093776) facet_failover done 1 2259 0.13 MB/sec execute 41 sec latency 19129.865 ms 1 2259 0.13 MB/sec execute 42 sec latency 20130.020 ms 1 2259 0.12 MB/sec execute 43 sec latency 21130.708 ms 1 2264 0.12 MB/sec execute 44 sec latency 22070.377 ms oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2417 0.15 MB/sec execute 45 sec latency 69.846 ms 1 2574 0.16 MB/sec execute 46 sec latency 37.437 ms 1 2902 0.24 MB/sec execute 47 sec latency 37.258 ms 1 3257 0.28 MB/sec execute 48 sec latency 25.996 ms 1 3579 0.35 MB/sec execute 49 sec latency 19.707 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2824 1284864 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2576 1285112 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12460 3592248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38040 3564504 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50500 7156752 1% /mnt/lustre 1 3828 0.37 MB/sec execute 50 sec latency 17.010 ms 1 3995 0.36 MB/sec execute 51 sec latency 26.192 ms 1 4236 0.36 MB/sec execute 52 sec latency 17.895 ms 1 4473 0.36 MB/sec execute 53 sec latency 28.301 ms test_26 fail mds1 3 times Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 1 4703 0.38 MB/sec execute 54 sec latency 19.535 ms 03:49:50 (1711093790) shut down 1 4756 0.38 MB/sec execute 55 sec latency 748.703 ms 1 4756 0.37 MB/sec execute 56 sec latency 1749.327 ms 1 4756 0.36 MB/sec execute 57 sec latency 2749.620 ms 1 4756 0.36 MB/sec execute 58 sec latency 3749.901 ms 1 4756 0.35 MB/sec execute 59 sec latency 4750.167 ms 1 4756 0.34 MB/sec execute 60 sec latency 5750.346 ms 1 4756 0.34 MB/sec execute 61 sec latency 6750.511 ms 1 4756 0.33 MB/sec execute 62 sec latency 7750.727 ms 1 4756 0.33 MB/sec execute 63 sec latency 8751.041 ms 1 4756 0.32 MB/sec execute 64 sec latency 9751.358 ms 1 4756 0.32 MB/sec execute 65 sec latency 10754.970 ms Failover mds1 to oleg119-server mount facets: mds1 1 4756 0.31 MB/sec execute 66 sec latency 11755.181 ms 1 4756 0.31 MB/sec execute 67 sec latency 12755.693 ms 1 4756 0.30 MB/sec execute 68 sec latency 13757.200 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4756 0.30 MB/sec execute 69 sec latency 14758.034 ms 1 4756 0.30 MB/sec execute 70 sec latency 15759.024 ms 1 4756 0.29 MB/sec execute 71 sec latency 16759.266 ms oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 1 4756 0.29 MB/sec execute 72 sec latency 17759.418 ms Started lustre-MDT0000 03:50:07 (1711093807) targets are mounted 03:50:07 (1711093807) facet_failover done 1 4756 0.28 MB/sec execute 73 sec latency 18759.623 ms 1 4756 0.28 MB/sec execute 74 sec latency 19759.902 ms 1 4756 0.28 MB/sec execute 75 sec latency 20760.086 ms 1 4756 0.27 MB/sec execute 76 sec latency 21760.227 ms 1 4756 0.27 MB/sec execute 77 sec latency 22761.097 ms 1 4756 0.26 MB/sec execute 78 sec latency 23761.909 ms 1 4758 0.26 MB/sec execute 79 sec latency 24744.011 ms oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4957 0.30 MB/sec execute 80 sec latency 69.252 ms 1 4996 0.29 MB/sec execute 81 sec latency 449.995 ms 1 4996 0.29 MB/sec execute 82 sec latency 1450.272 ms 1 4996 0.28 MB/sec execute 83 sec latency 2450.482 ms 1 4996 0.28 MB/sec execute 84 sec latency 3450.705 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3012 1284676 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12516 3585376 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38152 3562160 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 50668 7147536 1% /mnt/lustre 1 4996 0.28 MB/sec execute 85 sec latency 4451.310 ms 1 4996 0.27 MB/sec execute 86 sec latency 5451.824 ms 1 4996 0.27 MB/sec execute 87 sec latency 6452.192 ms 1 4996 0.27 MB/sec execute 88 sec latency 7452.983 ms 1 4996 0.27 MB/sec execute 89 sec latency 8453.189 ms test_26 fail mds2 4 times 1 4996 0.26 MB/sec execute 90 sec latency 9460.295 ms Failing mds2 on oleg119-server 1 4996 0.26 MB/sec execute 91 sec latency 10462.313 ms Stopping /mnt/lustre-mds2 (opts:) on oleg119-server 1 4999 0.26 MB/sec execute 92 sec latency 11086.213 ms 1 4999 0.25 MB/sec execute 93 sec latency 1370.218 ms 03:50:28 (1711093828) shut down 1 4999 0.25 MB/sec execute 94 sec latency 2372.739 ms 1 4999 0.25 MB/sec execute 95 sec latency 3373.139 ms 1 4999 0.25 MB/sec execute 96 sec latency 4375.949 ms 1 4999 0.24 MB/sec execute 97 sec latency 5376.117 ms 1 4999 0.24 MB/sec execute 98 sec latency 6376.278 ms 1 4999 0.24 MB/sec execute 99 sec latency 7376.439 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec Failover mds2 to oleg119-server mount facets: mds2 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 cleanup 107 sec 1 cleanup 108 sec oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all 1 cleanup 109 sec pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 03:50:45 (1711093845) targets are mounted 03:50:45 (1711093845) facet_failover done 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 cleanup 114 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 0 cleanup 114 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 703 84.457 24743.996 Close 529 65.132 21052.892 Rename 34 23.088 73.178 Unlink 136 6.994 20.669 Qpathinfo 667 35.934 20567.109 Qfileinfo 111 0.762 6.336 Qfsinfo 121 1.553 24.945 Sfileinfo 75 14.577 69.241 Find 251 6.886 33.440 WriteX 360 4.061 37.962 ReadX 1145 0.395 28.390 LockX 2 2.436 2.557 UnlockX 2 3.149 3.195 Flush 49 16.726 41.992 Throughput 0.238744 MB/sec 1 clients 1 procs max_latency=24744.011 ms stopping dbench on /mnt/lustre at Fri Mar 22 03:50:49 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Mar 22 03:50:53 EDT 2024 waiting for dbench pid 10885 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 126 5.63 MB/sec warmup 1 sec latency 70.959 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3052 1284636 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23928 3549184 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31020 3569580 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 54948 7118764 1% /mnt/lustre 1 213 4.14 MB/sec warmup 2 sec latency 72.743 ms 1 298 3.65 MB/sec warmup 3 sec latency 71.443 ms 1 381 3.39 MB/sec warmup 4 sec latency 66.225 ms 1 503 3.49 MB/sec warmup 5 sec latency 53.983 ms 1 678 3.53 MB/sec warmup 6 sec latency 48.674 ms test_26 fail mds1 5 times 1 794 3.06 MB/sec warmup 7 sec latency 37.219 ms Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 1 902 2.71 MB/sec warmup 8 sec latency 116.677 ms 03:51:02 (1711093862) shut down 1 902 2.41 MB/sec warmup 9 sec latency 1117.583 ms 1 902 2.17 MB/sec warmup 10 sec latency 2117.721 ms 1 902 1.97 MB/sec warmup 11 sec latency 3118.211 ms 1 902 1.81 MB/sec warmup 12 sec latency 4118.434 ms 1 902 1.67 MB/sec warmup 13 sec latency 5119.028 ms 1 902 1.55 MB/sec warmup 14 sec latency 6123.559 ms 1 902 1.44 MB/sec warmup 15 sec latency 7123.784 ms 1 902 1.35 MB/sec warmup 16 sec latency 8124.508 ms 1 902 1.27 MB/sec warmup 17 sec latency 9124.703 ms 1 902 1.20 MB/sec warmup 18 sec latency 10124.942 ms 1 902 1.14 MB/sec warmup 19 sec latency 11125.076 ms Failover mds1 to oleg119-server mount facets: mds1 1 902 0.00 MB/sec execute 1 sec latency 13126.635 ms 1 902 0.00 MB/sec execute 2 sec latency 14126.965 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 902 0.00 MB/sec execute 3 sec latency 15128.234 ms 1 902 0.00 MB/sec execute 4 sec latency 16128.413 ms oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 1 902 0.00 MB/sec execute 5 sec latency 17128.955 ms Started lustre-MDT0000 03:51:18 (1711093878) targets are mounted 03:51:18 (1711093878) facet_failover done 1 902 0.00 MB/sec execute 6 sec latency 18129.302 ms 1 902 0.00 MB/sec execute 7 sec latency 19129.515 ms 1 902 0.00 MB/sec execute 8 sec latency 20129.864 ms 1 918 0.01 MB/sec execute 9 sec latency 20860.738 ms oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1028 0.12 MB/sec execute 10 sec latency 44.202 ms 1 1149 0.12 MB/sec execute 11 sec latency 43.030 ms 1 1257 0.12 MB/sec execute 12 sec latency 40.887 ms 1 1267 0.11 MB/sec execute 13 sec latency 863.957 ms 1 1267 0.10 MB/sec execute 14 sec latency 1866.228 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3036 1284652 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33132 3563256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5728 3589944 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38860 7153200 1% /mnt/lustre 1 1267 0.10 MB/sec execute 15 sec latency 2866.418 ms 1 1267 0.09 MB/sec execute 16 sec latency 3876.585 ms 1 1267 0.09 MB/sec execute 17 sec latency 4876.771 ms 1 1267 0.08 MB/sec execute 18 sec latency 5877.174 ms 1 1267 0.08 MB/sec execute 19 sec latency 6877.347 ms test_26 fail mds2 6 times 1 1267 0.07 MB/sec execute 20 sec latency 7877.472 ms Failing mds2 on oleg119-server 1 1267 0.07 MB/sec execute 21 sec latency 8877.797 ms Stopping /mnt/lustre-mds2 (opts:) on oleg119-server 1 1267 0.07 MB/sec execute 22 sec latency 9878.108 ms 03:51:35 (1711093895) shut down 1 1267 0.06 MB/sec execute 23 sec latency 10878.309 ms 1 1278 0.06 MB/sec execute 24 sec latency 11020.215 ms 1 1278 0.06 MB/sec execute 25 sec latency 1732.776 ms 1 1278 0.06 MB/sec execute 26 sec latency 2732.949 ms 1 1278 0.05 MB/sec execute 27 sec latency 3733.335 ms 1 1278 0.05 MB/sec execute 28 sec latency 4733.627 ms 1 1278 0.05 MB/sec execute 29 sec latency 5733.943 ms 1 1278 0.05 MB/sec execute 30 sec latency 6734.121 ms 1 1278 0.05 MB/sec execute 31 sec latency 7734.300 ms 1 1278 0.05 MB/sec execute 32 sec latency 8734.486 ms Failover mds2 to oleg119-server mount facets: mds2 1 1278 0.04 MB/sec execute 33 sec latency 9734.733 ms 1 1278 0.04 MB/sec execute 34 sec latency 10735.174 ms 1 1278 0.04 MB/sec execute 35 sec latency 11735.358 ms 1 1278 0.04 MB/sec execute 36 sec latency 12735.551 ms 1 1278 0.04 MB/sec execute 37 sec latency 13735.744 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1278 0.04 MB/sec execute 38 sec latency 14735.941 ms 1 1278 0.04 MB/sec execute 39 sec latency 15736.118 ms oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 1 1278 0.04 MB/sec execute 40 sec latency 16737.176 ms Started lustre-MDT0001 03:51:53 (1711093913) targets are mounted 03:51:53 (1711093913) facet_failover done 1 1278 0.04 MB/sec execute 41 sec latency 17737.409 ms 1 1278 0.03 MB/sec execute 42 sec latency 18737.655 ms 1 1278 0.03 MB/sec execute 43 sec latency 19737.811 ms oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1451 0.10 MB/sec execute 44 sec latency 19840.650 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1549 0.10 MB/sec execute 45 sec latency 62.223 ms 1 1634 0.10 MB/sec execute 46 sec latency 46.853 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1777 0.10 MB/sec execute 47 sec latency 48.830 ms 1 1877 0.10 MB/sec execute 48 sec latency 55.491 ms 1 2014 0.10 MB/sec execute 49 sec latency 47.924 ms 1 2141 0.10 MB/sec execute 50 sec latency 39.247 ms 1 2305 0.12 MB/sec execute 51 sec latency 42.508 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Mar 22 03:52:04 EDT 2024 with return code 0 10885 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 10885 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished