== replay-dual test 26: dbench and tar with mds failover ========================================================== 06:33:54 (1744022034) Starting client oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre Started clients oleg350-client.virtnet: 192.168.203.150@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 4105 Started dbench loop with 4106 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Apr 7 06:33:56 EDT 2025 waiting for dbench pid 4147 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 166 6.80 MB/sec warmup 1 sec latency 32.555 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2040 1285648 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26112 3580684 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7186160 1% /mnt/lustre 1 333 6.00 MB/sec warmup 2 sec latency 33.742 ms 1 451 5.28 MB/sec warmup 3 sec latency 467.743 ms 1 674 5.29 MB/sec warmup 4 sec latency 30.574 ms 1 848 4.31 MB/sec warmup 5 sec latency 29.817 ms test_26 fail mds1 1 times Failing mds1 on oleg350-server 1 1018 3.73 MB/sec warmup 6 sec latency 26.034 ms Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 1 1112 3.28 MB/sec warmup 7 sec latency 700.224 ms 1 1112 2.87 MB/sec warmup 8 sec latency 1700.438 ms 1 1112 2.55 MB/sec warmup 9 sec latency 2700.701 ms 1 1112 2.30 MB/sec warmup 10 sec latency 3700.880 ms 1 1112 2.09 MB/sec warmup 11 sec latency 4701.064 ms 1 1112 1.91 MB/sec warmup 12 sec latency 5701.243 ms 06:34:08 (1744022048) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server 1 1112 1.77 MB/sec warmup 13 sec latency 6701.459 ms 1 1112 1.64 MB/sec warmup 14 sec latency 7701.697 ms 1 1112 1.53 MB/sec warmup 15 sec latency 8701.879 ms 1 1112 1.44 MB/sec warmup 16 sec latency 9702.122 ms 1 1112 1.35 MB/sec warmup 17 sec latency 10702.340 ms 1 1112 1.28 MB/sec warmup 18 sec latency 11702.535 ms 1 1112 1.21 MB/sec warmup 19 sec latency 12702.691 ms 1 1112 0.00 MB/sec execute 1 sec latency 14703.012 ms 1 1112 0.00 MB/sec execute 2 sec latency 15703.205 ms Failover mds1 to oleg350-server mount facets: mds1 1 1112 0.00 MB/sec execute 3 sec latency 16703.366 ms 1 1112 0.00 MB/sec execute 4 sec latency 17703.516 ms 1 1112 0.00 MB/sec execute 5 sec latency 18703.691 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1112 0.00 MB/sec execute 6 sec latency 19703.879 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:34:23 (1744022063) targets are mounted 06:34:23 (1744022063) facet_failover done 1 1112 0.00 MB/sec execute 7 sec latency 20704.020 ms 1 1112 0.00 MB/sec execute 8 sec latency 21704.144 ms 1 1112 0.00 MB/sec execute 9 sec latency 22704.253 ms 1 1112 0.00 MB/sec execute 10 sec latency 23704.548 ms 1 1112 0.00 MB/sec execute 11 sec latency 24704.698 ms oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1378 0.25 MB/sec execute 12 sec latency 24977.452 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1609 0.25 MB/sec execute 13 sec latency 19.750 ms 1 1928 0.25 MB/sec execute 14 sec latency 18.220 ms 1 2117 0.25 MB/sec execute 15 sec latency 21.956 ms 1 2332 0.30 MB/sec execute 16 sec latency 32.929 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2932 1284756 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2612 1285076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6624 3594348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 32564 3559184 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39188 7153532 1% /mnt/lustre 1 2603 0.36 MB/sec execute 17 sec latency 25.593 ms 1 2920 0.58 MB/sec execute 18 sec latency 464.606 ms 1 3541 0.81 MB/sec execute 19 sec latency 16.409 ms test_26 fail mds2 2 times 1 3789 0.84 MB/sec execute 20 sec latency 20.054 ms Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 1 3975 0.81 MB/sec execute 21 sec latency 120.413 ms 06:34:37 (1744022077) shut down facet: mds2 facet_host: oleg350-server facet_failover_host: oleg350-server 1 3975 0.78 MB/sec execute 22 sec latency 1120.577 ms 1 3975 0.74 MB/sec execute 23 sec latency 2120.776 ms 1 3975 0.71 MB/sec execute 24 sec latency 3121.088 ms 1 3975 0.68 MB/sec execute 25 sec latency 4121.255 ms 1 3975 0.66 MB/sec execute 26 sec latency 5121.442 ms 1 3975 0.63 MB/sec execute 27 sec latency 6121.630 ms 1 3975 0.61 MB/sec execute 28 sec latency 7121.801 ms 1 3975 0.59 MB/sec execute 29 sec latency 8122.048 ms 1 3975 0.57 MB/sec execute 30 sec latency 9122.217 ms 1 3975 0.55 MB/sec execute 31 sec latency 10122.379 ms Failover mds2 to oleg350-server mount facets: mds2 1 3975 0.53 MB/sec execute 32 sec latency 11122.483 ms 1 3975 0.52 MB/sec execute 33 sec latency 12122.586 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 3975 0.50 MB/sec execute 34 sec latency 13122.672 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all 1 3975 0.49 MB/sec execute 35 sec latency 14122.774 ms pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:34:51 (1744022091) targets are mounted 06:34:51 (1744022091) facet_failover done 1 3975 0.47 MB/sec execute 36 sec latency 15122.890 ms 1 3975 0.46 MB/sec execute 37 sec latency 16122.986 ms 1 3975 0.45 MB/sec execute 38 sec latency 17123.084 ms 1 3975 0.44 MB/sec execute 39 sec latency 18123.188 ms 1 4042 0.43 MB/sec execute 40 sec latency 18826.596 ms oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4286 0.42 MB/sec execute 41 sec latency 21.529 ms 1 4548 0.43 MB/sec execute 42 sec latency 18.068 ms 1 4828 0.45 MB/sec execute 43 sec latency 16.533 ms 1 5167 0.51 MB/sec execute 44 sec latency 25.287 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4472 1283216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9056 3583012 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 42788 3544844 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51844 7127856 1% /mnt/lustre 1 5422 0.50 MB/sec execute 45 sec latency 25.543 ms 1 5724 0.50 MB/sec execute 46 sec latency 17.812 ms 1 6020 0.52 MB/sec execute 47 sec latency 20.814 ms 1 6462 0.60 MB/sec execute 48 sec latency 22.315 ms test_26 fail mds1 3 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 1 6950 0.65 MB/sec execute 49 sec latency 20.608 ms 1 7324 0.72 MB/sec execute 50 sec latency 25.156 ms 1 7499 0.71 MB/sec execute 51 sec latency 24.464 ms 1 7627 0.70 MB/sec execute 52 sec latency 101.679 ms 1 7627 0.68 MB/sec execute 53 sec latency 1101.815 ms 1 7627 0.67 MB/sec execute 54 sec latency 2102.040 ms 1 7627 0.66 MB/sec execute 55 sec latency 3102.269 ms 06:35:11 (1744022111) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server 1 7627 0.65 MB/sec execute 56 sec latency 4102.471 ms 1 7627 0.64 MB/sec execute 57 sec latency 5102.638 ms 1 7627 0.62 MB/sec execute 58 sec latency 6102.811 ms 1 7627 0.61 MB/sec execute 59 sec latency 7103.007 ms 1 7627 0.60 MB/sec execute 60 sec latency 8103.280 ms 1 7627 0.59 MB/sec execute 61 sec latency 9103.439 ms 1 7627 0.58 MB/sec execute 62 sec latency 10103.611 ms 1 7627 0.58 MB/sec execute 63 sec latency 11103.824 ms 1 7627 0.57 MB/sec execute 64 sec latency 12104.056 ms 1 7627 0.56 MB/sec execute 65 sec latency 13104.287 ms Failover mds1 to oleg350-server mount facets: mds1 1 7627 0.55 MB/sec execute 66 sec latency 14104.437 ms 1 7627 0.54 MB/sec execute 67 sec latency 15104.573 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.53 MB/sec execute 68 sec latency 16104.791 ms 1 7627 0.53 MB/sec execute 69 sec latency 17104.962 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:35:26 (1744022126) targets are mounted 06:35:26 (1744022126) facet_failover done 1 7627 0.52 MB/sec execute 70 sec latency 18105.147 ms 1 7627 0.51 MB/sec execute 71 sec latency 19105.422 ms 1 7627 0.50 MB/sec execute 72 sec latency 20105.871 ms 1 7627 0.50 MB/sec execute 73 sec latency 21106.115 ms 1 7627 0.49 MB/sec execute 74 sec latency 22106.315 ms 1 7695 0.48 MB/sec execute 75 sec latency 22871.574 ms oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7937 0.48 MB/sec execute 76 sec latency 28.172 ms 1 8173 0.50 MB/sec execute 77 sec latency 27.722 ms 1 8487 0.53 MB/sec execute 78 sec latency 20.286 ms 1 8686 0.52 MB/sec execute 79 sec latency 19.645 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5016 1282672 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3044 1284644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23344 3578076 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 50328 3553136 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 73672 7131212 2% /mnt/lustre 1 8895 0.52 MB/sec execute 80 sec latency 21.819 ms 1 9111 0.52 MB/sec execute 81 sec latency 23.970 ms 1 9405 0.52 MB/sec execute 82 sec latency 18.453 ms 1 9578 0.53 MB/sec execute 83 sec latency 31.880 ms test_26 fail mds2 4 times 1 10033 0.58 MB/sec execute 84 sec latency 22.357 ms Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 1 10345 0.59 MB/sec execute 85 sec latency 510.356 ms 06:35:41 (1744022141) shut down facet: mds2 facet_host: oleg350-server facet_failover_host: oleg350-server 1 10345 0.58 MB/sec execute 86 sec latency 1510.576 ms 1 10345 0.58 MB/sec execute 87 sec latency 2510.802 ms 1 10345 0.57 MB/sec execute 88 sec latency 3511.072 ms 1 10345 0.56 MB/sec execute 89 sec latency 4511.259 ms 1 10345 0.56 MB/sec execute 90 sec latency 5511.384 ms 1 10345 0.55 MB/sec execute 91 sec latency 6511.568 ms 1 10345 0.54 MB/sec execute 92 sec latency 7511.819 ms 1 10345 0.54 MB/sec execute 93 sec latency 8512.083 ms 1 10345 0.53 MB/sec execute 94 sec latency 9512.345 ms 1 10345 0.53 MB/sec execute 95 sec latency 10512.707 ms Failover mds2 to oleg350-server mount facets: mds2 1 10345 0.52 MB/sec execute 96 sec latency 11512.998 ms 1 10345 0.52 MB/sec execute 97 sec latency 12513.177 ms 1 10345 0.51 MB/sec execute 98 sec latency 13513.381 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 10345 0.51 MB/sec execute 99 sec latency 14513.572 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:35:56 (1744022156) targets are mounted 06:35:56 (1744022156) facet_failover done 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 cleanup 105 sec 0 cleanup 105 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1554 35.050 19860.592 Close 1167 23.308 24977.432 Rename 69 15.924 31.851 Unlink 285 4.873 15.292 Qpathinfo 1478 3.027 23.942 Qfileinfo 271 0.507 3.258 Qfsinfo 238 0.935 6.501 Sfileinfo 130 9.628 22.820 Find 534 45.457 22871.563 WriteX 787 2.290 12.370 ReadX 2608 0.115 13.396 LockX 6 1.910 2.652 UnlockX 6 1.933 2.707 Flush 101 13.357 464.589 Throughput 0.505406 MB/sec 1 clients 1 procs max_latency=24977.452 ms stopping dbench on /mnt/lustre at Mon Apr 7 06:36:01 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Apr 7 06:36:03 EDT 2025 waiting for dbench pid 8577 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 165 6.80 MB/sec warmup 1 sec latency 32.725 ms 1 370 6.59 MB/sec warmup 2 sec latency 35.741 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5408 1282280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3484 1284204 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 44624 3557916 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12084 3590860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 56708 7148776 1% /mnt/lustre 1 689 7.06 MB/sec warmup 3 sec latency 41.178 ms 1 789 5.36 MB/sec warmup 4 sec latency 420.478 ms 1 921 4.35 MB/sec warmup 5 sec latency 602.784 ms 1 1091 3.82 MB/sec warmup 6 sec latency 32.468 ms test_26 fail mds1 5 times Failing mds1 on oleg350-server 1 1405 3.72 MB/sec warmup 7 sec latency 31.157 ms Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 1 1509 3.26 MB/sec warmup 8 sec latency 606.111 ms 06:36:11 (1744022171) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server 1 1509 2.89 MB/sec warmup 9 sec latency 1606.311 ms 1 1509 2.60 MB/sec warmup 10 sec latency 2606.619 ms 1 1509 2.37 MB/sec warmup 11 sec latency 3606.919 ms 1 1509 2.17 MB/sec warmup 12 sec latency 4607.183 ms 1 1509 2.00 MB/sec warmup 13 sec latency 5607.459 ms 1 1509 1.86 MB/sec warmup 14 sec latency 6607.761 ms 1 1509 1.74 MB/sec warmup 15 sec latency 7608.077 ms 1 1509 1.63 MB/sec warmup 16 sec latency 8608.283 ms 1 1509 1.53 MB/sec warmup 17 sec latency 9608.501 ms 1 1509 1.45 MB/sec warmup 18 sec latency 10608.738 ms Failover mds1 to oleg350-server mount facets: mds1 1 1509 1.37 MB/sec warmup 19 sec latency 11608.936 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1509 0.00 MB/sec execute 1 sec latency 13609.147 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:36:25 (1744022185) targets are mounted 06:36:25 (1744022185) facet_failover done 1 1509 0.00 MB/sec execute 2 sec latency 14609.299 ms 1 1509 0.00 MB/sec execute 3 sec latency 15609.415 ms 1 1509 0.00 MB/sec execute 4 sec latency 16609.583 ms 1 1509 0.00 MB/sec execute 5 sec latency 17609.772 ms 1 1509 0.00 MB/sec execute 6 sec latency 18609.895 ms oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1752 0.03 MB/sec execute 7 sec latency 18862.438 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2069 0.07 MB/sec execute 8 sec latency 17.395 ms 1 2400 0.21 MB/sec execute 9 sec latency 40.334 ms 1 2896 0.68 MB/sec execute 10 sec latency 29.087 ms 1 3380 0.85 MB/sec execute 11 sec latency 21.731 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4384 1283304 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2688 1285000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37144 3555576 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12472 3583120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7138696 1% /mnt/lustre 1 3792 1.14 MB/sec execute 12 sec latency 32.233 ms 1 4014 1.08 MB/sec execute 13 sec latency 17.194 ms 1 4270 1.02 MB/sec execute 14 sec latency 28.319 ms test_26 fail mds2 6 times 1 4524 0.99 MB/sec execute 15 sec latency 27.763 ms Failing mds2 on oleg350-server Stopping /mnt/lustre-mds2 (opts:) on oleg350-server 1 4810 1.01 MB/sec execute 16 sec latency 20.789 ms 1 4851 0.95 MB/sec execute 17 sec latency 846.411 ms 1 4851 0.90 MB/sec execute 18 sec latency 1846.550 ms 1 4851 0.85 MB/sec execute 19 sec latency 2846.657 ms 1 4851 0.81 MB/sec execute 20 sec latency 3846.782 ms 1 4851 0.77 MB/sec execute 21 sec latency 4846.887 ms 06:36:44 (1744022204) shut down facet: mds2 facet_host: oleg350-server facet_failover_host: oleg350-server 1 4851 0.74 MB/sec execute 22 sec latency 5847.017 ms 1 4851 0.70 MB/sec execute 23 sec latency 6847.146 ms 1 4851 0.68 MB/sec execute 24 sec latency 7847.265 ms 1 4851 0.65 MB/sec execute 25 sec latency 8847.378 ms 1 4851 0.62 MB/sec execute 26 sec latency 9847.471 ms 1 4851 0.60 MB/sec execute 27 sec latency 10847.614 ms 1 4851 0.58 MB/sec execute 28 sec latency 11847.742 ms 1 4851 0.56 MB/sec execute 29 sec latency 12847.890 ms 1 4851 0.54 MB/sec execute 30 sec latency 13848.083 ms 1 4851 0.52 MB/sec execute 31 sec latency 14848.280 ms Failover mds2 to oleg350-server mount facets: mds2 1 4851 0.51 MB/sec execute 32 sec latency 15848.433 ms 1 4851 0.49 MB/sec execute 33 sec latency 16848.598 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4851 0.48 MB/sec execute 34 sec latency 17848.716 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all 1 4851 0.46 MB/sec execute 35 sec latency 18848.908 ms pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 06:36:58 (1744022218) targets are mounted 06:36:58 (1744022218) facet_failover done 1 4851 0.45 MB/sec execute 36 sec latency 19849.032 ms 1 4851 0.44 MB/sec execute 37 sec latency 20849.219 ms 1 4851 0.43 MB/sec execute 38 sec latency 21849.421 ms 1 4851 0.42 MB/sec execute 39 sec latency 22849.643 ms 1 4978 0.48 MB/sec execute 40 sec latency 23627.916 ms oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar 1 5227 0.47 MB/sec execute 41 sec latency 15.987 ms 1 5489 0.46 MB/sec execute 42 sec latency 22.939 ms 1 5820 0.46 MB/sec execute 43 sec latency 14.399 ms 1 6140 0.50 MB/sec execute 44 sec latency 19.017 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2760 1284928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37732 3568028 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11788 3594332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49520 7162360 1% /mnt/lustre 1 6752 0.61 MB/sec execute 45 sec latency 27.286 ms 1 7191 0.69 MB/sec execute 46 sec latency 16.234 ms 1 7459 0.70 MB/sec execute 47 sec latency 15.498 ms 1 7707 0.69 MB/sec execute 48 sec latency 17.002 ms test_26 fail mds1 7 times Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 1 7990 0.69 MB/sec execute 49 sec latency 23.182 ms 1 8021 0.68 MB/sec execute 50 sec latency 865.133 ms 1 8021 0.67 MB/sec execute 51 sec latency 1865.347 ms 1 8021 0.65 MB/sec execute 52 sec latency 2865.554 ms 1 8021 0.64 MB/sec execute 53 sec latency 3865.714 ms 1 8021 0.63 MB/sec execute 54 sec latency 4865.861 ms 1 8021 0.62 MB/sec execute 55 sec latency 5866.024 ms 06:37:18 (1744022238) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server 1 8021 0.61 MB/sec execute 56 sec latency 6866.208 ms 1 8021 0.60 MB/sec execute 57 sec latency 7866.373 ms 1 8021 0.59 MB/sec execute 58 sec latency 8866.551 ms 1 8021 0.58 MB/sec execute 59 sec latency 9866.749 ms 1 8021 0.57 MB/sec execute 60 sec latency 10866.914 ms 1 8021 0.56 MB/sec execute 61 sec latency 11867.105 ms 1 8021 0.55 MB/sec execute 62 sec latency 12867.263 ms 1 8021 0.54 MB/sec execute 63 sec latency 13867.415 ms 1 8021 0.53 MB/sec execute 64 sec latency 14867.585 ms 1 8021 0.52 MB/sec execute 65 sec latency 15867.711 ms Failover mds1 to oleg350-server mount facets: mds1 1 8021 0.51 MB/sec execute 66 sec latency 16867.864 ms 1 8021 0.51 MB/sec execute 67 sec latency 17868.003 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 8021 0.50 MB/sec execute 68 sec latency 18868.165 ms oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all 1 8021 0.49 MB/sec execute 69 sec latency 19868.323 ms pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 06:37:32 (1744022252) targets are mounted 06:37:32 (1744022252) facet_failover done 1 8021 0.48 MB/sec execute 70 sec latency 20868.463 ms 1 8021 0.48 MB/sec execute 71 sec latency 21868.652 ms 1 8021 0.47 MB/sec execute 72 sec latency 22868.829 ms 1 8021 0.46 MB/sec execute 73 sec latency 23869.026 ms 1 8035 0.46 MB/sec execute 74 sec latency 24763.716 ms oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8329 0.47 MB/sec execute 75 sec latency 16.637 ms 1 8665 0.50 MB/sec execute 76 sec latency 23.532 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 8907 0.50 MB/sec execute 77 sec latency 18.168 ms 1 9236 0.50 MB/sec execute 78 sec latency 20.171 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Apr 7 06:37:41 EDT 2025 with return code 0 8577 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 8577 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished