== replay-dual test 26: dbench and tar with mds failover ========================================================== 15:04:20 (1758740660) Starting client oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre Started clients oleg247-client.virtnet: 192.168.202.147@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65039 Started dbench loop with 65040 striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2056 1285632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13832 3576624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15368 7182108 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Sep 24 15:04:25 EDT 2025 waiting for dbench pid 65119 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 51 2.10 MB/sec warmup 1 sec latency 94.735 ms 1 127 2.83 MB/sec warmup 2 sec latency 100.390 ms 1 181 2.43 MB/sec warmup 3 sec latency 122.535 ms 1 247 2.34 MB/sec warmup 4 sec latency 102.601 ms 1 313 2.29 MB/sec warmup 5 sec latency 69.871 ms 1 373 2.21 MB/sec warmup 6 sec latency 106.348 ms 1 442 2.21 MB/sec warmup 7 sec latency 79.375 ms 1 503 2.18 MB/sec warmup 8 sec latency 75.891 ms 1 608 2.33 MB/sec warmup 9 sec latency 80.433 ms 1 705 2.12 MB/sec warmup 10 sec latency 52.123 ms test_26 fail mds1 1 times 1 780 1.94 MB/sec warmup 11 sec latency 64.402 ms Failing mds1 on oleg247-server 1 854 1.80 MB/sec warmup 12 sec latency 67.398 ms Stopping /mnt/lustre-mds1 (opts:) on oleg247-server 1 902 1.67 MB/sec warmup 13 sec latency 74.845 ms 1 909 1.55 MB/sec warmup 14 sec latency 868.028 ms 15:04:40 (1758740680) shut down facet: mds1 facet_host: oleg247-server facet_failover_host: oleg247-server 1 909 1.45 MB/sec warmup 15 sec latency 1870.338 ms 1 909 1.36 MB/sec warmup 16 sec latency 2872.270 ms 1 909 1.28 MB/sec warmup 17 sec latency 3872.562 ms 1 909 1.21 MB/sec warmup 18 sec latency 4874.506 ms 1 909 1.14 MB/sec warmup 19 sec latency 5875.886 ms 1 909 0.00 MB/sec execute 1 sec latency 7879.492 ms 1 909 0.00 MB/sec execute 2 sec latency 8880.686 ms 1 909 0.00 MB/sec execute 3 sec latency 9880.918 ms 1 909 0.00 MB/sec execute 4 sec latency 10881.166 ms Failover mds1 to oleg247-server 1 909 0.00 MB/sec execute 5 sec latency 11882.796 ms mount facets: mds1 1 909 0.00 MB/sec execute 6 sec latency 12884.631 ms 1 909 0.00 MB/sec execute 7 sec latency 13884.830 ms 1 909 0.00 MB/sec execute 8 sec latency 14885.355 ms 1 909 0.00 MB/sec execute 9 sec latency 15888.304 ms 1 909 0.00 MB/sec execute 10 sec latency 16888.505 ms 1 909 0.00 MB/sec execute 11 sec latency 17892.416 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 909 0.00 MB/sec execute 12 sec latency 18894.665 ms 1 909 0.00 MB/sec execute 13 sec latency 19895.658 ms 1 909 0.00 MB/sec execute 14 sec latency 20895.944 ms 1 909 0.00 MB/sec execute 15 sec latency 21896.164 ms oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all 1 909 0.00 MB/sec execute 16 sec latency 22898.893 ms pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 15:05:02 (1758740702) targets are mounted 15:05:02 (1758740702) facet_failover done 1 909 0.00 MB/sec execute 17 sec latency 23899.197 ms 1 909 0.00 MB/sec execute 18 sec latency 24900.460 ms 1 909 0.00 MB/sec execute 19 sec latency 25902.230 ms 1 921 0.00 MB/sec execute 20 sec latency 26452.592 ms 1 977 0.01 MB/sec execute 21 sec latency 75.467 ms oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1064 0.05 MB/sec execute 22 sec latency 57.237 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1139 0.05 MB/sec execute 23 sec latency 54.977 ms 1 1201 0.05 MB/sec execute 24 sec latency 130.324 ms 1 1259 0.05 MB/sec execute 25 sec latency 82.021 ms 1 1405 0.16 MB/sec execute 26 sec latency 64.873 ms 1 1497 0.16 MB/sec execute 27 sec latency 56.899 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1285276 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1285528 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 47992 3558076 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12024 3593788 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 60016 7151864 1% /mnt/lustre 1 1565 0.16 MB/sec execute 28 sec latency 61.445 ms 1 1634 0.15 MB/sec execute 29 sec latency 59.489 ms 1 1692 0.15 MB/sec execute 30 sec latency 71.949 ms 1 1793 0.15 MB/sec execute 31 sec latency 57.192 ms 1 1864 0.14 MB/sec execute 32 sec latency 67.089 ms 1 1972 0.14 MB/sec execute 33 sec latency 61.716 ms 1 2028 0.14 MB/sec execute 34 sec latency 66.172 ms test_26 fail mds2 2 times 1 2135 0.14 MB/sec execute 35 sec latency 52.894 ms Failing mds2 on oleg247-server Stopping /mnt/lustre-mds2 (opts:) on oleg247-server 1 2191 0.14 MB/sec execute 36 sec latency 61.048 ms 1 2259 0.14 MB/sec execute 37 sec latency 305.152 ms 1 2259 0.13 MB/sec execute 38 sec latency 1305.394 ms 15:05:23 (1758740723) shut down facet: mds2 facet_host: oleg247-server facet_failover_host: oleg247-server 1 2259 0.13 MB/sec execute 39 sec latency 2305.679 ms 1 2259 0.13 MB/sec execute 40 sec latency 3305.899 ms 1 2259 0.12 MB/sec execute 41 sec latency 4306.110 ms 1 2259 0.12 MB/sec execute 42 sec latency 5306.308 ms 1 2259 0.12 MB/sec execute 43 sec latency 6309.565 ms 1 2259 0.11 MB/sec execute 44 sec latency 7311.353 ms 1 2259 0.11 MB/sec execute 45 sec latency 8311.563 ms 1 2259 0.11 MB/sec execute 46 sec latency 9315.319 ms 1 2259 0.11 MB/sec execute 47 sec latency 10315.495 ms 1 2259 0.10 MB/sec execute 48 sec latency 11319.819 ms Failover mds2 to oleg247-server mount facets: mds2 1 2259 0.10 MB/sec execute 49 sec latency 12321.048 ms 1 2259 0.10 MB/sec execute 50 sec latency 13321.234 ms 1 2259 0.10 MB/sec execute 51 sec latency 14323.154 ms 1 2259 0.10 MB/sec execute 52 sec latency 15324.903 ms 1 2259 0.09 MB/sec execute 53 sec latency 16325.801 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2259 0.09 MB/sec execute 54 sec latency 17326.785 ms 1 2259 0.09 MB/sec execute 55 sec latency 18328.772 ms 1 2259 0.09 MB/sec execute 56 sec latency 19330.628 ms 1 2259 0.09 MB/sec execute 57 sec latency 20330.787 ms oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all 1 2259 0.09 MB/sec execute 58 sec latency 21332.005 ms pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 15:05:44 (1758740744) targets are mounted 15:05:44 (1758740744) facet_failover done 1 2259 0.09 MB/sec execute 59 sec latency 22333.110 ms 1 2259 0.08 MB/sec execute 60 sec latency 23333.251 ms 1 2356 0.10 MB/sec execute 61 sec latency 23446.291 ms oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 2417 0.10 MB/sec execute 62 sec latency 66.281 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2477 0.11 MB/sec execute 63 sec latency 73.243 ms 1 2577 0.11 MB/sec execute 64 sec latency 64.546 ms 1 2765 0.16 MB/sec execute 65 sec latency 66.876 ms 1 2922 0.18 MB/sec execute 66 sec latency 54.106 ms 1 3010 0.18 MB/sec execute 67 sec latency 67.652 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2744 1284944 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 48816 3555480 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20744 3583948 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 69560 7139428 1% /mnt/lustre 1 3204 0.18 MB/sec execute 68 sec latency 84.981 ms 1 3297 0.20 MB/sec execute 69 sec latency 114.245 ms 1 3386 0.20 MB/sec execute 70 sec latency 58.802 ms 1 3552 0.23 MB/sec execute 71 sec latency 55.232 ms 1 3638 0.23 MB/sec execute 72 sec latency 55.945 ms 1 3707 0.23 MB/sec execute 73 sec latency 55.982 ms 1 3789 0.24 MB/sec execute 74 sec latency 76.701 ms test_26 fail mds1 3 times 1 3856 0.24 MB/sec execute 75 sec latency 65.190 ms Failing mds1 on oleg247-server 1 3927 0.24 MB/sec execute 76 sec latency 59.243 ms Stopping /mnt/lustre-mds1 (opts:) on oleg247-server 1 3986 0.24 MB/sec execute 77 sec latency 66.020 ms 1 4042 0.23 MB/sec execute 78 sec latency 75.966 ms 15:06:04 (1758740764) shut down facet: mds1 facet_host: oleg247-server facet_failover_host: oleg247-server 1 4083 0.23 MB/sec execute 79 sec latency 354.935 ms 1 4083 0.23 MB/sec execute 80 sec latency 1356.381 ms 1 4083 0.23 MB/sec execute 81 sec latency 2358.702 ms 1 4083 0.22 MB/sec execute 82 sec latency 3358.917 ms 1 4083 0.22 MB/sec execute 83 sec latency 4359.230 ms 1 4083 0.22 MB/sec execute 84 sec latency 5360.798 ms 1 4083 0.22 MB/sec execute 85 sec latency 6362.675 ms 1 4083 0.21 MB/sec execute 86 sec latency 7362.907 ms 1 4083 0.21 MB/sec execute 87 sec latency 8363.119 ms 1 4083 0.21 MB/sec execute 88 sec latency 9363.472 ms Failover mds1 to oleg247-server mount facets: mds1 1 4083 0.21 MB/sec execute 89 sec latency 10365.353 ms 1 4083 0.20 MB/sec execute 90 sec latency 11365.435 ms 1 4083 0.20 MB/sec execute 91 sec latency 12366.192 ms 1 4083 0.20 MB/sec execute 92 sec latency 13366.704 ms 1 4083 0.20 MB/sec execute 93 sec latency 14367.729 ms 1 4083 0.19 MB/sec execute 94 sec latency 15367.956 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.19 MB/sec execute 95 sec latency 16368.722 ms 1 4083 0.19 MB/sec execute 96 sec latency 17368.907 ms 1 4083 0.19 MB/sec execute 97 sec latency 18370.734 ms oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all 1 4083 0.19 MB/sec execute 98 sec latency 19371.418 ms pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 15:06:24 (1758740784) targets are mounted 15:06:24 (1758740784) facet_failover done 1 4083 0.18 MB/sec execute 99 sec latency 20372.476 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 562 72.885 23446.277 Close 404 73.808 26452.568 Rename 23 55.611 130.283 Unlink 89 16.226 38.906 Qpathinfo 514 56.265 23558.290 Qfileinfo 89 1.016 7.385 Qfsinfo 87 1.649 21.372 Sfileinfo 37 38.326 58.811 Find 187 6.880 39.692 WriteX 263 8.453 53.643 ReadX 893 0.605 34.797 LockX 2 7.857 10.776 UnlockX 2 7.833 8.301 Flush 23 40.514 61.699 Throughput 0.184962 MB/sec 1 clients 1 procs max_latency=26452.592 ms stopping dbench on /mnt/lustre at Wed Sep 24 15:06:30 EDT 2025 with return code 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Sep 24 15:06:35 EDT 2025 waiting for dbench pid 68245 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3020 1284668 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2720 1284968 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13664 3591232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 35960 3570324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7161556 1% /mnt/lustre 1 43 1.64 MB/sec warmup 1 sec latency 64.534 ms 1 121 2.75 MB/sec warmup 2 sec latency 97.353 ms 1 173 2.35 MB/sec warmup 3 sec latency 92.467 ms 1 231 2.22 MB/sec warmup 4 sec latency 101.161 ms 1 300 2.19 MB/sec warmup 5 sec latency 72.174 ms 1 357 2.13 MB/sec warmup 6 sec latency 89.469 ms 1 438 2.20 MB/sec warmup 7 sec latency 57.730 ms test_26 fail mds2 4 times 1 507 2.19 MB/sec warmup 8 sec latency 81.582 ms Failing mds2 on oleg247-server Stopping /mnt/lustre-mds2 (opts:) on oleg247-server 1 639 2.34 MB/sec warmup 9 sec latency 81.625 ms 1 710 2.12 MB/sec warmup 10 sec latency 69.463 ms 1 778 1.94 MB/sec warmup 11 sec latency 74.123 ms 1 870 1.80 MB/sec warmup 12 sec latency 57.750 ms 1 939 1.67 MB/sec warmup 13 sec latency 64.263 ms 1 995 1.56 MB/sec warmup 14 sec latency 158.893 ms 1 995 1.46 MB/sec warmup 15 sec latency 1160.239 ms 1 995 1.37 MB/sec warmup 16 sec latency 2160.472 ms 15:06:51 (1758740811) shut down facet: mds2 facet_host: oleg247-server facet_failover_host: oleg247-server 1 995 1.29 MB/sec warmup 17 sec latency 3161.237 ms 1 995 1.22 MB/sec warmup 18 sec latency 4161.465 ms 1 995 1.15 MB/sec warmup 19 sec latency 5161.753 ms 1 995 0.00 MB/sec execute 1 sec latency 7162.172 ms 1 995 0.00 MB/sec execute 2 sec latency 8162.652 ms 1 995 0.00 MB/sec execute 3 sec latency 9162.867 ms 1 995 0.00 MB/sec execute 4 sec latency 10163.437 ms 1 995 0.00 MB/sec execute 5 sec latency 11163.610 ms 1 995 0.00 MB/sec execute 6 sec latency 12165.842 ms Failover mds2 to oleg247-server mount facets: mds2 1 995 0.00 MB/sec execute 7 sec latency 13167.068 ms 1 995 0.00 MB/sec execute 8 sec latency 14167.259 ms 1 995 0.00 MB/sec execute 9 sec latency 15167.480 ms 1 995 0.00 MB/sec execute 10 sec latency 16169.204 ms 1 995 0.00 MB/sec execute 11 sec latency 17169.577 ms 1 995 0.00 MB/sec execute 12 sec latency 18169.753 ms Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.00 MB/sec execute 13 sec latency 19170.439 ms 1 995 0.00 MB/sec execute 14 sec latency 20170.660 ms 1 995 0.00 MB/sec execute 15 sec latency 21172.124 ms oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all 1 995 0.00 MB/sec execute 16 sec latency 22172.880 ms pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 15:07:12 (1758740832) targets are mounted 15:07:12 (1758740832) facet_failover done 1 995 0.00 MB/sec execute 17 sec latency 23176.063 ms 1 995 0.00 MB/sec execute 18 sec latency 24176.350 ms 1 995 0.00 MB/sec execute 19 sec latency 25176.622 ms 1 995 0.00 MB/sec execute 20 sec latency 26179.388 ms 1 1057 0.05 MB/sec execute 21 sec latency 26400.035 ms oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1145 0.05 MB/sec execute 22 sec latency 52.608 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1205 0.05 MB/sec execute 23 sec latency 68.167 ms 1 1265 0.05 MB/sec execute 24 sec latency 80.695 ms 1 1426 0.16 MB/sec execute 25 sec latency 72.366 ms 1 1508 0.16 MB/sec execute 26 sec latency 62.411 ms 1 1617 0.16 MB/sec execute 27 sec latency 55.023 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3416 1284272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3212 1284476 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26096 3579896 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47040 3559308 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 73136 7139204 2% /mnt/lustre 1 1698 0.15 MB/sec execute 28 sec latency 53.669 ms 1 1819 0.15 MB/sec execute 29 sec latency 45.011 ms 1 1904 0.15 MB/sec execute 30 sec latency 68.054 ms 1 1999 0.15 MB/sec execute 31 sec latency 62.520 ms 1 2050 0.15 MB/sec execute 32 sec latency 346.897 ms 1 2148 0.14 MB/sec execute 33 sec latency 65.253 ms 1 2226 0.14 MB/sec execute 34 sec latency 58.361 ms test_26 fail mds1 5 times 1 2358 0.17 MB/sec execute 35 sec latency 52.053 ms Failing mds1 on oleg247-server Stopping /mnt/lustre-mds1 (opts:) on oleg247-server 1 2417 0.17 MB/sec execute 36 sec latency 84.615 ms 1 2477 0.18 MB/sec execute 37 sec latency 60.675 ms 1 2576 0.19 MB/sec execute 38 sec latency 69.877 ms 15:07:33 (1758740853) shut down facet: mds1 facet_host: oleg247-server facet_failover_host: oleg247-server 1 2768 0.26 MB/sec execute 39 sec latency 63.295 ms 1 2932 0.29 MB/sec execute 40 sec latency 55.230 ms 1 3176 0.30 MB/sec execute 41 sec latency 62.206 ms 1 3306 0.32 MB/sec execute 42 sec latency 55.464 ms 1 3504 0.38 MB/sec execute 43 sec latency 43.336 ms 1 3618 0.38 MB/sec execute 44 sec latency 43.112 ms 1 3716 0.37 MB/sec execute 45 sec latency 46.073 ms 1 3820 0.39 MB/sec execute 46 sec latency 46.693 ms 1 3910 0.39 MB/sec execute 47 sec latency 69.557 ms 1 4006 0.38 MB/sec execute 48 sec latency 46.677 ms Failover mds1 to oleg247-server mount facets: mds1 1 4083 0.37 MB/sec execute 49 sec latency 53.762 ms 1 4083 0.36 MB/sec execute 50 sec latency 1017.292 ms 1 4083 0.36 MB/sec execute 51 sec latency 2017.780 ms 1 4083 0.35 MB/sec execute 52 sec latency 3018.112 ms 1 4083 0.34 MB/sec execute 53 sec latency 4018.819 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.34 MB/sec execute 54 sec latency 5020.840 ms 1 4083 0.33 MB/sec execute 55 sec latency 6022.129 ms 1 4083 0.32 MB/sec execute 56 sec latency 7024.907 ms oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all 1 4083 0.32 MB/sec execute 57 sec latency 8025.576 ms pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 15:07:53 (1758740873) targets are mounted 15:07:53 (1758740873) facet_failover done 1 4083 0.31 MB/sec execute 58 sec latency 9025.755 ms 1 4083 0.31 MB/sec execute 59 sec latency 10027.343 ms 1 4083 0.30 MB/sec execute 60 sec latency 11028.714 ms 1 4091 0.30 MB/sec execute 61 sec latency 11859.621 ms 1 4184 0.29 MB/sec execute 62 sec latency 43.274 ms oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4263 0.29 MB/sec execute 63 sec latency 77.490 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4316 0.29 MB/sec execute 64 sec latency 79.335 ms 1 4379 0.29 MB/sec execute 65 sec latency 76.250 ms kill: sending signal to 65069 failed: No such process 1 4455 0.29 MB/sec execute 66 sec latency 64.095 ms 1 4521 0.28 MB/sec execute 67 sec latency 55.095 ms 1 4597 0.30 MB/sec execute 68 sec latency 60.469 ms 1 4691 0.29 MB/sec execute 69 sec latency 59.073 ms 1 4747 0.29 MB/sec execute 70 sec latency 81.699 ms 1 4812 0.29 MB/sec execute 71 sec latency 66.795 ms 1 4957 0.32 MB/sec execute 72 sec latency 81.032 ms 1 5023 0.32 MB/sec execute 73 sec latency 71.895 ms 1 5086 0.31 MB/sec execute 74 sec latency 106.078 ms 1 5153 0.31 MB/sec execute 75 sec latency 56.551 ms 1 5227 0.31 MB/sec execute 76 sec latency 101.136 ms 1 5292 0.30 MB/sec execute 77 sec latency 64.865 ms 1 5364 0.30 MB/sec execute 78 sec latency 49.074 ms 1 5442 0.30 MB/sec execute 79 sec latency 121.434 ms 1 5551 0.30 MB/sec execute 80 sec latency 78.619 ms 1 5633 0.29 MB/sec execute 81 sec latency 52.492 ms 1 5732 0.29 MB/sec execute 82 sec latency 56.069 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Sep 24 15:08:18 EDT 2025 with return code 0 68245 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 68245 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished