== replay-dual test 26: dbench and tar with mds failover ========================================================== 12:57:20 (1752512240) Starting client oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre Started clients oleg154-client.virtnet: 192.168.201.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 64889 Started dbench loop with 64890 striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3588732 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3080 7194216 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 14 12:57:23 EDT 2025 waiting for dbench pid 64970 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 113 5.24 MB/sec warmup 1 sec latency 42.372 ms 1 222 4.31 MB/sec warmup 2 sec latency 56.565 ms 1 339 4.09 MB/sec warmup 3 sec latency 34.964 ms 1 436 3.85 MB/sec warmup 4 sec latency 86.423 ms test_26 fail mds1 1 times Failing mds1 on oleg154-server 1 534 3.73 MB/sec warmup 5 sec latency 95.719 ms Stopping /mnt/lustre-mds1 (opts:) on oleg154-server 1 606 3.49 MB/sec warmup 6 sec latency 124.040 ms 12:57:30 (1752512250) shut down facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server 1 606 3.00 MB/sec warmup 7 sec latency 1124.241 ms 1 606 2.62 MB/sec warmup 8 sec latency 2124.405 ms 1 606 2.33 MB/sec warmup 9 sec latency 3124.593 ms 1 606 2.10 MB/sec warmup 10 sec latency 4124.790 ms 1 606 1.91 MB/sec warmup 11 sec latency 5125.011 ms 1 606 1.75 MB/sec warmup 12 sec latency 6125.292 ms 1 606 1.61 MB/sec warmup 13 sec latency 7125.602 ms 1 606 1.50 MB/sec warmup 14 sec latency 8125.809 ms 1 606 1.40 MB/sec warmup 15 sec latency 9125.998 ms 1 606 1.31 MB/sec warmup 16 sec latency 10126.218 ms Failover mds1 to oleg154-server mount facets: mds1 1 606 1.23 MB/sec warmup 17 sec latency 11126.447 ms 1 606 1.16 MB/sec warmup 18 sec latency 12126.595 ms 1 606 1.10 MB/sec warmup 19 sec latency 13127.111 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 606 0.00 MB/sec execute 1 sec latency 15127.477 ms oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all 1 606 0.00 MB/sec execute 2 sec latency 16127.627 ms pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 12:57:46 (1752512266) targets are mounted 12:57:46 (1752512266) facet_failover done 1 606 0.00 MB/sec execute 3 sec latency 17127.827 ms 1 606 0.00 MB/sec execute 4 sec latency 18128.029 ms 1 606 0.00 MB/sec execute 5 sec latency 19128.259 ms 1 606 0.00 MB/sec execute 6 sec latency 20128.423 ms 1 676 0.03 MB/sec execute 7 sec latency 20567.385 ms oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 781 0.05 MB/sec execute 8 sec latency 51.330 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 898 0.08 MB/sec execute 9 sec latency 41.073 ms 1 1008 0.13 MB/sec execute 10 sec latency 34.904 ms 1 1148 0.18 MB/sec execute 11 sec latency 72.177 ms 1 1268 0.18 MB/sec execute 12 sec latency 39.087 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1285188 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2216 1285472 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39640 3556900 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 6784 3590476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 46424 7147376 1% /mnt/lustre 1 1447 0.39 MB/sec execute 13 sec latency 49.878 ms 1 1553 0.37 MB/sec execute 14 sec latency 83.588 ms 1 1614 0.35 MB/sec execute 15 sec latency 59.257 ms 1 1767 0.33 MB/sec execute 16 sec latency 37.797 ms 1 1861 0.32 MB/sec execute 17 sec latency 58.692 ms test_26 fail mds2 2 times Failing mds2 on oleg154-server 1 2006 0.31 MB/sec execute 18 sec latency 57.441 ms Stopping /mnt/lustre-mds2 (opts:) on oleg154-server 1 2162 0.30 MB/sec execute 19 sec latency 48.708 ms 1 2259 0.29 MB/sec execute 20 sec latency 639.585 ms 1 2259 0.28 MB/sec execute 21 sec latency 1639.805 ms 1 2259 0.26 MB/sec execute 22 sec latency 2639.978 ms 1 2259 0.25 MB/sec execute 23 sec latency 3640.157 ms 1 2259 0.24 MB/sec execute 24 sec latency 4640.336 ms 1 2259 0.23 MB/sec execute 25 sec latency 5640.541 ms 12:58:08 (1752512288) shut down facet: mds2 facet_host: oleg154-server facet_failover_host: oleg154-server 1 2259 0.22 MB/sec execute 26 sec latency 6640.804 ms 1 2259 0.22 MB/sec execute 27 sec latency 7641.053 ms 1 2259 0.21 MB/sec execute 28 sec latency 8641.330 ms 1 2259 0.20 MB/sec execute 29 sec latency 9641.547 ms 1 2259 0.19 MB/sec execute 30 sec latency 10641.794 ms 1 2259 0.19 MB/sec execute 31 sec latency 11642.021 ms 1 2259 0.18 MB/sec execute 32 sec latency 12642.211 ms 1 2259 0.18 MB/sec execute 33 sec latency 13642.364 ms 1 2259 0.17 MB/sec execute 34 sec latency 14642.546 ms 1 2259 0.17 MB/sec execute 35 sec latency 15642.786 ms Failover mds2 to oleg154-server mount facets: mds2 1 2259 0.16 MB/sec execute 36 sec latency 16643.029 ms 1 2259 0.16 MB/sec execute 37 sec latency 17643.246 ms 1 2259 0.15 MB/sec execute 38 sec latency 18643.429 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.15 MB/sec execute 39 sec latency 19643.646 ms 1 2259 0.15 MB/sec execute 40 sec latency 20643.833 ms oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 1 2259 0.14 MB/sec execute 41 sec latency 21644.010 ms Started lustre-MDT0001 12:58:24 (1752512304) targets are mounted 12:58:25 (1752512305) facet_failover done 1 2259 0.14 MB/sec execute 42 sec latency 22644.199 ms 1 2259 0.14 MB/sec execute 43 sec latency 23644.393 ms 1 2259 0.13 MB/sec execute 44 sec latency 24644.576 ms 1 2350 0.15 MB/sec execute 45 sec latency 25091.393 ms oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2452 0.16 MB/sec execute 46 sec latency 41.912 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2596 0.17 MB/sec execute 47 sec latency 49.136 ms 1 2901 0.25 MB/sec execute 48 sec latency 65.745 ms 1 3188 0.27 MB/sec execute 49 sec latency 38.245 ms 1 3366 0.29 MB/sec execute 50 sec latency 49.892 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2852 1284836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 56280 3547960 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14196 3590384 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 70476 7138344 1% /mnt/lustre 1 3626 0.35 MB/sec execute 51 sec latency 39.188 ms 1 3773 0.36 MB/sec execute 52 sec latency 38.624 ms 1 3907 0.36 MB/sec execute 53 sec latency 34.846 ms 1 4023 0.35 MB/sec execute 54 sec latency 65.571 ms 1 4142 0.35 MB/sec execute 55 sec latency 35.364 ms test_26 fail mds1 3 times Failing mds1 on oleg154-server 1 4279 0.35 MB/sec execute 56 sec latency 63.654 ms Stopping /mnt/lustre-mds1 (opts:) on oleg154-server 1 4349 0.34 MB/sec execute 57 sec latency 444.158 ms 12:58:41 (1752512321) shut down facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server 1 4349 0.34 MB/sec execute 58 sec latency 1444.386 ms 1 4349 0.33 MB/sec execute 59 sec latency 2444.658 ms 1 4349 0.33 MB/sec execute 60 sec latency 3444.916 ms 1 4349 0.32 MB/sec execute 61 sec latency 4445.142 ms 1 4349 0.31 MB/sec execute 62 sec latency 5445.400 ms 1 4349 0.31 MB/sec execute 63 sec latency 6445.656 ms 1 4349 0.30 MB/sec execute 64 sec latency 7445.905 ms 1 4349 0.30 MB/sec execute 65 sec latency 8446.178 ms 1 4349 0.30 MB/sec execute 66 sec latency 9446.431 ms 1 4349 0.29 MB/sec execute 67 sec latency 10446.689 ms Failover mds1 to oleg154-server mount facets: mds1 1 4349 0.29 MB/sec execute 68 sec latency 11446.891 ms 1 4349 0.28 MB/sec execute 69 sec latency 12447.072 ms 1 4349 0.28 MB/sec execute 70 sec latency 13447.309 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4349 0.27 MB/sec execute 71 sec latency 14447.554 ms 1 4349 0.27 MB/sec execute 72 sec latency 15447.752 ms oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 1 4349 0.27 MB/sec execute 73 sec latency 16447.952 ms Started lustre-MDT0000 12:58:56 (1752512336) targets are mounted 12:58:56 (1752512336) facet_failover done 1 4349 0.26 MB/sec execute 74 sec latency 17448.170 ms 1 4349 0.26 MB/sec execute 75 sec latency 18448.448 ms 1 4349 0.26 MB/sec execute 76 sec latency 19448.685 ms 1 4349 0.25 MB/sec execute 77 sec latency 20448.816 ms oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4470 0.25 MB/sec execute 78 sec latency 20561.510 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4600 0.27 MB/sec execute 79 sec latency 31.570 ms 1 4768 0.26 MB/sec execute 80 sec latency 34.568 ms 1 4984 0.30 MB/sec execute 81 sec latency 30.675 ms 1 5153 0.30 MB/sec execute 82 sec latency 30.350 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3388 1284300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3144 1284544 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 58232 3548084 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14724 3591244 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 72956 7139328 2% /mnt/lustre 1 5296 0.29 MB/sec execute 83 sec latency 45.731 ms 1 5474 0.29 MB/sec execute 84 sec latency 31.880 ms 1 5716 0.29 MB/sec execute 85 sec latency 24.061 ms 1 5916 0.30 MB/sec execute 86 sec latency 41.190 ms 1 6050 0.31 MB/sec execute 87 sec latency 31.684 ms test_26 fail mds2 4 times Failing mds2 on oleg154-server 1 6332 0.34 MB/sec execute 88 sec latency 38.992 ms Stopping /mnt/lustre-mds2 (opts:) on oleg154-server 1 6537 0.36 MB/sec execute 89 sec latency 410.991 ms 12:59:13 (1752512353) shut down facet: mds2 facet_host: oleg154-server facet_failover_host: oleg154-server 1 6537 0.35 MB/sec execute 90 sec latency 1411.228 ms 1 6537 0.35 MB/sec execute 91 sec latency 2411.398 ms 1 6537 0.34 MB/sec execute 92 sec latency 3411.561 ms 1 6537 0.34 MB/sec execute 93 sec latency 4411.756 ms 1 6537 0.34 MB/sec execute 94 sec latency 5411.962 ms 1 6537 0.33 MB/sec execute 95 sec latency 6412.133 ms 1 6537 0.33 MB/sec execute 96 sec latency 7412.361 ms 1 6537 0.33 MB/sec execute 97 sec latency 8412.574 ms 1 6537 0.32 MB/sec execute 98 sec latency 9412.830 ms 1 6537 0.32 MB/sec execute 99 sec latency 10413.080 ms Failover mds2 to oleg154-server mount facets: mds2 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 cleanup 103 sec 1 cleanup 104 sec oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all 1 cleanup 105 sec pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0001 12:59:29 (1752512369) targets are mounted 12:59:29 (1752512369) facet_failover done 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 0 cleanup 111 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1002 42.197 25091.352 Close 773 3.233 16.004 Rename 45 485.605 20561.489 Unlink 169 10.740 65.541 Qpathinfo 948 27.317 20567.359 Qfileinfo 179 0.976 3.765 Qfsinfo 153 0.619 4.734 Sfileinfo 100 18.222 48.625 Find 345 5.459 25.907 WriteX 535 4.479 35.994 ReadX 1595 0.426 25.293 LockX 4 3.098 3.595 UnlockX 4 2.840 3.210 Flush 80 274.587 20740.040 Throughput 0.31982 MB/sec 1 clients 1 procs max_latency=25091.393 ms stopping dbench on /mnt/lustre at Mon Jul 14 12:59:34 EDT 2025 with return code 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Jul 14 12:59:38 EDT 2025 waiting for dbench pid 69126 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3884 1283804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3692 1283996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 31248 3549532 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 4124 3602728 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 35372 7152260 1% /mnt/lustre 1 113 5.24 MB/sec warmup 1 sec latency 52.125 ms 1 224 4.31 MB/sec warmup 2 sec latency 48.530 ms 1 342 4.09 MB/sec warmup 3 sec latency 36.730 ms 1 485 4.24 MB/sec warmup 4 sec latency 33.188 ms test_26 fail mds1 5 times 1 664 4.23 MB/sec warmup 5 sec latency 62.677 ms Failing mds1 on oleg154-server 1 740 3.55 MB/sec warmup 6 sec latency 115.016 ms Stopping /mnt/lustre-mds1 (opts:) on oleg154-server 1 897 3.10 MB/sec warmup 7 sec latency 57.506 ms 12:59:46 (1752512386) shut down facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server 1 995 2.74 MB/sec warmup 8 sec latency 384.826 ms 1 995 2.43 MB/sec warmup 9 sec latency 1385.017 ms 1 995 2.19 MB/sec warmup 10 sec latency 2385.201 ms 1 995 1.99 MB/sec warmup 11 sec latency 3385.349 ms 1 995 1.83 MB/sec warmup 12 sec latency 4385.582 ms 1 995 1.69 MB/sec warmup 13 sec latency 5385.804 ms 1 995 1.56 MB/sec warmup 14 sec latency 6386.003 ms 1 995 1.46 MB/sec warmup 15 sec latency 7386.181 ms 1 995 1.37 MB/sec warmup 16 sec latency 8386.376 ms 1 995 1.29 MB/sec warmup 17 sec latency 9386.567 ms Failover mds1 to oleg154-server mount facets: mds1 1 995 1.22 MB/sec warmup 18 sec latency 10386.818 ms 1 995 1.15 MB/sec warmup 19 sec latency 11387.047 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 1 sec latency 13387.409 ms 1 995 0.00 MB/sec execute 2 sec latency 14387.661 ms oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 1 995 0.00 MB/sec execute 3 sec latency 15387.821 ms Started lustre-MDT0000 13:00:02 (1752512402) targets are mounted 13:00:02 (1752512402) facet_failover done 1 995 0.00 MB/sec execute 4 sec latency 16387.993 ms 1 995 0.00 MB/sec execute 5 sec latency 17388.153 ms 1 995 0.00 MB/sec execute 6 sec latency 18388.340 ms 1 995 0.00 MB/sec execute 7 sec latency 19388.467 ms 1 1018 0.06 MB/sec execute 8 sec latency 20039.011 ms oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1163 0.12 MB/sec execute 9 sec latency 39.717 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1281 0.12 MB/sec execute 10 sec latency 37.453 ms 1 1505 0.38 MB/sec execute 11 sec latency 34.758 ms 1 1656 0.36 MB/sec execute 12 sec latency 47.528 ms 1 1832 0.34 MB/sec execute 13 sec latency 37.712 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3684 1284004 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3452 1284236 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 59048 3545932 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14548 3591788 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73596 7137720 2% /mnt/lustre 1 1982 0.32 MB/sec execute 14 sec latency 80.311 ms 1 2160 0.32 MB/sec execute 15 sec latency 28.895 ms 1 2356 0.37 MB/sec execute 16 sec latency 63.266 ms 1 2477 0.40 MB/sec execute 17 sec latency 39.602 ms 1 2753 0.56 MB/sec execute 18 sec latency 32.047 ms test_26 fail mds2 6 times 1 2932 0.61 MB/sec execute 19 sec latency 221.750 ms Failing mds2 on oleg154-server Stopping /mnt/lustre-mds2 (opts:) on oleg154-server 1 3237 0.63 MB/sec execute 20 sec latency 54.688 ms 1 3556 0.79 MB/sec execute 21 sec latency 19.245 ms 1 3780 0.81 MB/sec execute 22 sec latency 21.440 ms 1 3963 0.79 MB/sec execute 23 sec latency 43.076 ms 1 4083 0.76 MB/sec execute 24 sec latency 324.305 ms 1 4083 0.73 MB/sec execute 25 sec latency 1324.560 ms 1 4083 0.70 MB/sec execute 26 sec latency 2324.703 ms 13:00:24 (1752512424) shut down facet: mds2 facet_host: oleg154-server facet_failover_host: oleg154-server 1 4083 0.67 MB/sec execute 27 sec latency 3324.946 ms 1 4083 0.65 MB/sec execute 28 sec latency 4325.130 ms 1 4083 0.63 MB/sec execute 29 sec latency 5325.313 ms 1 4083 0.61 MB/sec execute 30 sec latency 6325.538 ms 1 4083 0.59 MB/sec execute 31 sec latency 7325.721 ms 1 4083 0.57 MB/sec execute 32 sec latency 8325.936 ms 1 4083 0.55 MB/sec execute 33 sec latency 9326.110 ms 1 4083 0.53 MB/sec execute 34 sec latency 10326.300 ms 1 4083 0.52 MB/sec execute 35 sec latency 11326.480 ms 1 4083 0.50 MB/sec execute 36 sec latency 12326.658 ms Failover mds2 to oleg154-server mount facets: mds2 1 4083 0.49 MB/sec execute 37 sec latency 13326.846 ms 1 4083 0.48 MB/sec execute 38 sec latency 14327.011 ms 1 4083 0.47 MB/sec execute 39 sec latency 15327.141 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.45 MB/sec execute 40 sec latency 16327.334 ms oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all 1 4083 0.44 MB/sec execute 41 sec latency 17327.464 ms pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0001 13:00:40 (1752512440) targets are mounted 13:00:40 (1752512440) facet_failover done 1 4083 0.43 MB/sec execute 42 sec latency 18327.632 ms 1 4083 0.42 MB/sec execute 43 sec latency 19327.808 ms 1 4083 0.41 MB/sec execute 44 sec latency 20327.986 ms 1 4083 0.40 MB/sec execute 45 sec latency 21328.213 ms 1 4162 0.40 MB/sec execute 46 sec latency 21706.003 ms oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4282 0.39 MB/sec execute 47 sec latency 62.080 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4377 0.39 MB/sec execute 48 sec latency 83.591 ms 1 4477 0.39 MB/sec execute 49 sec latency 61.758 ms 1 4552 0.38 MB/sec execute 50 sec latency 58.435 ms 1 4644 0.39 MB/sec execute 51 sec latency 166.050 ms 1 4738 0.39 MB/sec execute 52 sec latency 62.399 ms 1 4808 0.38 MB/sec execute 53 sec latency 48.490 ms 1 4950 0.43 MB/sec execute 54 sec latency 74.198 ms 1 5013 0.42 MB/sec execute 55 sec latency 67.406 ms 1 5063 0.42 MB/sec execute 56 sec latency 159.289 ms 1 5129 0.41 MB/sec execute 57 sec latency 77.110 ms 1 5190 0.40 MB/sec execute 58 sec latency 60.762 ms 1 5251 0.40 MB/sec execute 59 sec latency 164.453 ms 1 5292 0.39 MB/sec execute 60 sec latency 92.099 ms 1 5350 0.39 MB/sec execute 61 sec latency 172.992 ms 1 5399 0.38 MB/sec execute 62 sec latency 70.454 ms 1 5454 0.37 MB/sec execute 63 sec latency 94.887 ms 1 5545 0.37 MB/sec execute 64 sec latency 69.329 ms 1 5604 0.37 MB/sec execute 65 sec latency 68.180 ms 1 5700 0.36 MB/sec execute 66 sec latency 68.697 ms 1 5760 0.36 MB/sec execute 67 sec latency 68.763 ms 1 5857 0.37 MB/sec execute 68 sec latency 99.215 ms 1 5918 0.36 MB/sec execute 69 sec latency 120.096 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Jul 14 13:01:08 EDT 2025 with return code 0 69126 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69126 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished