== replay-dual test 26: dbench and tar with mds failover ========================================================== 22:54:45 (1789440885) Starting client oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre Started clients oleg646-client.virtnet: 192.168.206.146@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45210 Started dbench loop with 45211 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3732480 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 20480 7494656 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Sep 14 22:54:52 EDT 2026 waiting for dbench pid 45279 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 14.611 ms 1 71 1.57 MB/sec warmup 2 sec latency 209.230 ms 1 113 1.75 MB/sec warmup 3 sec latency 172.296 ms 1 147 1.57 MB/sec warmup 4 sec latency 170.722 ms 1 182 1.46 MB/sec warmup 5 sec latency 163.088 ms test_26 fail mds1 1 times 1 207 1.35 MB/sec warmup 6 sec latency 220.668 ms 1 242 1.30 MB/sec warmup 7 sec latency 171.869 ms Failing mds1 on oleg646-server 1 280 1.30 MB/sec warmup 8 sec latency 111.612 ms Stopping /mnt/lustre-mds1 (opts:) on oleg646-server 1 313 1.27 MB/sec warmup 9 sec latency 147.402 ms 1 339 1.23 MB/sec warmup 10 sec latency 203.644 ms 1 339 1.11 MB/sec warmup 11 sec latency 1203.871 ms 1 339 1.02 MB/sec warmup 12 sec latency 2204.146 ms 22:55:04 (1789440904) shut down facet: mds1 facet_host: oleg646-server facet_failover_host: oleg646-server 1 339 0.94 MB/sec warmup 13 sec latency 3204.520 ms 1 339 0.88 MB/sec warmup 14 sec latency 4204.822 ms 1 339 0.82 MB/sec warmup 15 sec latency 5205.058 ms 1 339 0.77 MB/sec warmup 16 sec latency 6205.430 ms 1 339 0.72 MB/sec warmup 17 sec latency 7205.815 ms 1 339 0.68 MB/sec warmup 18 sec latency 8206.066 ms 1 339 0.64 MB/sec warmup 19 sec latency 9208.011 ms 1 339 0.00 MB/sec execute 1 sec latency 11211.731 ms 1 339 0.00 MB/sec execute 2 sec latency 12212.073 ms 1 339 0.00 MB/sec execute 3 sec latency 13212.259 ms Failover mds1 to oleg646-server mount facets: mds1 1 339 0.00 MB/sec execute 4 sec latency 14212.567 ms 1 339 0.00 MB/sec execute 5 sec latency 15214.815 ms 1 339 0.00 MB/sec execute 6 sec latency 16216.582 ms 1 339 0.00 MB/sec execute 7 sec latency 17217.698 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 339 0.00 MB/sec execute 8 sec latency 18220.317 ms 1 339 0.00 MB/sec execute 9 sec latency 19220.587 ms 1 339 0.00 MB/sec execute 10 sec latency 20220.777 ms 1 339 0.00 MB/sec execute 11 sec latency 21221.806 ms 1 339 0.00 MB/sec execute 12 sec latency 22222.052 ms 1 339 0.00 MB/sec execute 13 sec latency 23222.976 ms 1 339 0.00 MB/sec execute 14 sec latency 24223.292 ms 1 339 0.00 MB/sec execute 15 sec latency 25225.492 ms 1 339 0.00 MB/sec execute 16 sec latency 26225.731 ms 1 339 0.00 MB/sec execute 17 sec latency 27227.522 ms 1 339 0.00 MB/sec execute 18 sec latency 28227.708 ms 1 339 0.00 MB/sec execute 19 sec latency 29227.985 ms 1 339 0.00 MB/sec execute 20 sec latency 30228.209 ms 1 339 0.00 MB/sec execute 21 sec latency 31230.602 ms 1 339 0.00 MB/sec execute 22 sec latency 32231.724 ms 1 339 0.00 MB/sec execute 23 sec latency 33232.043 ms oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all 1 339 0.00 MB/sec execute 24 sec latency 34232.400 ms 1 339 0.00 MB/sec execute 25 sec latency 35232.890 ms pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 1 339 0.00 MB/sec execute 26 sec latency 36233.106 ms Started lustre-MDT0000 22:55:38 (1789440938) targets are mounted 22:55:38 (1789440938) facet_failover done 1 339 0.00 MB/sec execute 27 sec latency 37233.733 ms 1 339 0.00 MB/sec execute 28 sec latency 38233.929 ms 1 339 0.00 MB/sec execute 29 sec latency 39234.991 ms 1 339 0.00 MB/sec execute 30 sec latency 40235.406 ms 1 339 0.00 MB/sec execute 31 sec latency 41237.717 ms 1 339 0.00 MB/sec execute 32 sec latency 42237.955 ms 1 357 0.02 MB/sec execute 33 sec latency 42715.981 ms 1 398 0.05 MB/sec execute 34 sec latency 106.943 ms 1 430 0.08 MB/sec execute 35 sec latency 131.126 ms 1 469 0.12 MB/sec execute 36 sec latency 119.714 ms oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 503 0.14 MB/sec execute 37 sec latency 116.114 ms 1 542 0.18 MB/sec execute 38 sec latency 204.100 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 612 0.22 MB/sec execute 39 sec latency 111.383 ms 1 664 0.22 MB/sec execute 40 sec latency 103.273 ms 1 676 0.22 MB/sec execute 41 sec latency 365.957 ms 1 708 0.21 MB/sec execute 42 sec latency 179.195 ms 1 726 0.21 MB/sec execute 43 sec latency 295.741 ms 1 747 0.21 MB/sec execute 44 sec latency 129.173 ms 1 780 0.20 MB/sec execute 45 sec latency 130.769 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3726336 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 35840 3702784 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 50176 7429120 1% /mnt/lustre 1 799 0.20 MB/sec execute 46 sec latency 231.122 ms 1 821 0.20 MB/sec execute 47 sec latency 141.920 ms 1 870 0.20 MB/sec execute 48 sec latency 121.078 ms 1 892 0.19 MB/sec execute 49 sec latency 274.870 ms 1 913 0.19 MB/sec execute 50 sec latency 246.294 ms 1 949 0.19 MB/sec execute 51 sec latency 277.447 ms test_26 fail mds1 2 times 1 981 0.19 MB/sec execute 52 sec latency 181.999 ms 1 1037 0.20 MB/sec execute 53 sec latency 122.947 ms Failing mds1 on oleg646-server 1 1080 0.20 MB/sec execute 54 sec latency 90.376 ms Stopping /mnt/lustre-mds1 (opts:) on oleg646-server 1 1128 0.19 MB/sec execute 55 sec latency 98.890 ms 1 1159 0.19 MB/sec execute 56 sec latency 195.012 ms 1 1159 0.19 MB/sec execute 57 sec latency 1195.235 ms 1 1159 0.19 MB/sec execute 58 sec latency 2195.390 ms 22:56:11 (1789440971) shut down 1 1159 0.18 MB/sec execute 59 sec latency 3195.989 ms facet: mds1 facet_host: oleg646-server facet_failover_host: oleg646-server 1 1159 0.18 MB/sec execute 60 sec latency 4200.296 ms 1 1159 0.18 MB/sec execute 61 sec latency 5200.539 ms 1 1159 0.17 MB/sec execute 62 sec latency 6202.201 ms 1 1159 0.17 MB/sec execute 63 sec latency 7202.518 ms 1 1159 0.17 MB/sec execute 64 sec latency 8202.827 ms 1 1159 0.17 MB/sec execute 65 sec latency 9203.113 ms 1 1159 0.16 MB/sec execute 66 sec latency 10203.337 ms 1 1159 0.16 MB/sec execute 67 sec latency 11203.523 ms 1 1159 0.16 MB/sec execute 68 sec latency 12203.729 ms 1 1159 0.16 MB/sec execute 69 sec latency 13204.353 ms Failover mds1 to oleg646-server mount facets: mds1 1 1159 0.15 MB/sec execute 70 sec latency 14204.596 ms 1 1159 0.15 MB/sec execute 71 sec latency 15204.802 ms 1 1159 0.15 MB/sec execute 72 sec latency 16204.959 ms 1 1159 0.15 MB/sec execute 73 sec latency 17205.324 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1159 0.15 MB/sec execute 74 sec latency 18206.031 ms 1 1159 0.14 MB/sec execute 75 sec latency 19206.436 ms 1 1159 0.14 MB/sec execute 76 sec latency 20206.839 ms 1 1159 0.14 MB/sec execute 77 sec latency 21207.162 ms 1 1159 0.14 MB/sec execute 78 sec latency 22211.788 ms 1 1159 0.14 MB/sec execute 79 sec latency 23211.950 ms 1 1193 0.13 MB/sec execute 80 sec latency 23396.731 ms oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all 1 1232 0.13 MB/sec execute 81 sec latency 112.381 ms pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 1 1267 0.13 MB/sec execute 82 sec latency 108.326 ms Started lustre-MDT0000 22:56:35 (1789440995) targets are mounted 22:56:35 (1789440995) facet_failover done 1 1395 0.17 MB/sec execute 83 sec latency 123.328 ms 1 1426 0.16 MB/sec execute 84 sec latency 121.731 ms 1 1481 0.16 MB/sec execute 85 sec latency 114.717 ms 1 1542 0.16 MB/sec execute 86 sec latency 200.034 ms 1 1597 0.16 MB/sec execute 87 sec latency 79.908 ms 1 1634 0.16 MB/sec execute 88 sec latency 250.025 ms 1 1670 0.16 MB/sec execute 89 sec latency 172.643 ms oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1709 0.16 MB/sec execute 90 sec latency 137.806 ms 1 1771 0.15 MB/sec execute 91 sec latency 105.676 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1825 0.15 MB/sec execute 92 sec latency 78.675 ms 1 1871 0.15 MB/sec execute 93 sec latency 120.939 ms 1 1924 0.15 MB/sec execute 94 sec latency 267.671 ms 1 1980 0.15 MB/sec execute 95 sec latency 112.283 ms 1 2016 0.15 MB/sec execute 96 sec latency 124.357 ms 1 2081 0.15 MB/sec execute 97 sec latency 103.708 ms 1 2145 0.15 MB/sec execute 98 sec latency 96.466 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3699712 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3687424 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 7387136 1% /mnt/lustre 1 2166 0.15 MB/sec execute 99 sec latency 147.299 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 338 187.970 42715.964 Close 279 13.432 34.852 Rename 11 117.207 295.724 Unlink 28 31.261 52.220 Qpathinfo 275 101.140 23396.710 Qfileinfo 87 2.021 31.415 Qfsinfo 33 4.541 25.580 Sfileinfo 34 64.512 112.367 Find 91 6.598 89.948 WriteX 320 13.823 67.499 ReadX 346 1.883 116.956 Flush 30 161.868 365.942 Throughput 0.145829 MB/sec 1 clients 1 procs max_latency=42715.981 ms stopping dbench on /mnt/lustre at Mon Sep 14 22:56:56 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished test_26 fail mds1 3 times looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt Failing mds1 on oleg646-server Stopping /mnt/lustre-mds1 (opts:) on oleg646-server 22:57:03 (1789441023) shut down facet: mds1 facet_host: oleg646-server facet_failover_host: oleg646-server Failover mds1 to oleg646-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0000 22:57:26 (1789441046) targets are mounted 22:57:26 (1789441046) facet_failover done '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Sep 14 22:57:42 EDT 2026 waiting for dbench pid 47789 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 50 2.10 MB/sec warmup 1 sec latency 88.980 ms 1 113 2.62 MB/sec warmup 2 sec latency 184.700 ms 1 155 2.18 MB/sec warmup 3 sec latency 138.726 ms 1 181 1.83 MB/sec warmup 4 sec latency 231.515 ms oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 218 1.67 MB/sec warmup 5 sec latency 180.369 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 262 1.65 MB/sec warmup 6 sec latency 130.049 ms 1 289 1.52 MB/sec warmup 7 sec latency 141.166 ms 1 320 1.45 MB/sec warmup 8 sec latency 166.925 ms 1 349 1.39 MB/sec warmup 9 sec latency 176.323 ms 1 377 1.33 MB/sec warmup 10 sec latency 234.576 ms 1 437 1.40 MB/sec warmup 11 sec latency 95.469 ms 1 485 1.41 MB/sec warmup 12 sec latency 105.978 ms 1 517 1.36 MB/sec warmup 13 sec latency 175.055 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 29696 3707904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 34816 3694592 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 7402496 1% /mnt/lustre 1 596 1.48 MB/sec warmup 14 sec latency 179.555 ms 1 649 1.40 MB/sec warmup 15 sec latency 122.338 ms 1 671 1.32 MB/sec warmup 16 sec latency 351.277 ms 1 702 1.24 MB/sec warmup 17 sec latency 121.916 ms 1 730 1.18 MB/sec warmup 18 sec latency 155.742 ms 1 773 1.12 MB/sec warmup 19 sec latency 122.139 ms test_26 fail mds1 4 times 1 821 0.01 MB/sec execute 1 sec latency 193.742 ms Failing mds1 on oleg646-server 1 864 0.05 MB/sec execute 2 sec latency 158.132 ms Stopping /mnt/lustre-mds1 (opts:) on oleg646-server 1 878 0.06 MB/sec execute 3 sec latency 322.020 ms 1 903 0.05 MB/sec execute 4 sec latency 165.413 ms 1 909 0.06 MB/sec execute 5 sec latency 755.149 ms 1 909 0.05 MB/sec execute 6 sec latency 1757.396 ms 1 909 0.04 MB/sec execute 7 sec latency 2757.532 ms 1 909 0.04 MB/sec execute 8 sec latency 3759.738 ms 1 909 0.03 MB/sec execute 9 sec latency 4759.921 ms 22:58:12 (1789441092) shut down facet: mds1 facet_host: oleg646-server facet_failover_host: oleg646-server 1 909 0.03 MB/sec execute 10 sec latency 5760.176 ms 1 909 0.03 MB/sec execute 11 sec latency 6762.795 ms 1 909 0.02 MB/sec execute 12 sec latency 7763.197 ms 1 909 0.02 MB/sec execute 13 sec latency 8764.780 ms 1 909 0.02 MB/sec execute 14 sec latency 9765.230 ms 1 909 0.02 MB/sec execute 15 sec latency 10765.464 ms 1 909 0.02 MB/sec execute 16 sec latency 11765.662 ms 1 909 0.02 MB/sec execute 17 sec latency 12767.552 ms 1 909 0.02 MB/sec execute 18 sec latency 13767.748 ms 1 909 0.01 MB/sec execute 19 sec latency 14768.312 ms 1 909 0.01 MB/sec execute 20 sec latency 15770.602 ms Failover mds1 to oleg646-server mount facets: mds1 1 909 0.01 MB/sec execute 21 sec latency 16775.234 ms 1 909 0.01 MB/sec execute 22 sec latency 17775.433 ms 1 909 0.01 MB/sec execute 23 sec latency 18776.986 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 909 0.01 MB/sec execute 24 sec latency 19777.220 ms 1 909 0.01 MB/sec execute 25 sec latency 20777.946 ms 1 909 0.01 MB/sec execute 26 sec latency 21778.329 ms 1 909 0.01 MB/sec execute 27 sec latency 22778.624 ms 1 909 0.01 MB/sec execute 28 sec latency 23778.940 ms 1 909 0.01 MB/sec execute 29 sec latency 24779.540 ms oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all 1 909 0.01 MB/sec execute 30 sec latency 25779.715 ms pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 1 909 0.01 MB/sec execute 31 sec latency 26780.087 ms Started lustre-MDT0000 22:58:33 (1789441113) targets are mounted 22:58:33 (1789441113) facet_failover done 1 909 0.01 MB/sec execute 32 sec latency 27781.759 ms 1 909 0.01 MB/sec execute 33 sec latency 28783.094 ms 1 909 0.01 MB/sec execute 34 sec latency 29783.272 ms 1 909 0.01 MB/sec execute 35 sec latency 30784.502 ms 1 909 0.01 MB/sec execute 36 sec latency 31784.687 ms 1 909 0.01 MB/sec execute 37 sec latency 32784.877 ms 1 909 0.01 MB/sec execute 38 sec latency 33787.014 ms 1 909 0.01 MB/sec execute 39 sec latency 34787.205 ms 1 909 0.01 MB/sec execute 40 sec latency 35789.292 ms 1 909 0.01 MB/sec execute 41 sec latency 36791.522 ms 1 909 0.01 MB/sec execute 42 sec latency 37791.727 ms 1 909 0.01 MB/sec execute 43 sec latency 38792.676 ms 1 909 0.01 MB/sec execute 44 sec latency 39795.429 ms 1 909 0.01 MB/sec execute 45 sec latency 40796.264 ms 1 919 0.01 MB/sec execute 46 sec latency 41214.878 ms 1 964 0.01 MB/sec execute 47 sec latency 112.067 ms 1 984 0.01 MB/sec execute 48 sec latency 193.326 ms 1 1041 0.03 MB/sec execute 49 sec latency 95.376 ms 1 1085 0.03 MB/sec execute 50 sec latency 102.993 ms 1 1136 0.03 MB/sec execute 51 sec latency 89.065 ms oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1173 0.03 MB/sec execute 52 sec latency 148.121 ms 1 1200 0.03 MB/sec execute 53 sec latency 132.121 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1230 0.03 MB/sec execute 54 sec latency 171.177 ms 1 1262 0.03 MB/sec execute 55 sec latency 151.866 ms 1 1387 0.08 MB/sec execute 56 sec latency 106.196 ms 1 1418 0.08 MB/sec execute 57 sec latency 108.019 ms 1 1451 0.08 MB/sec execute 58 sec latency 130.984 ms 1 1486 0.08 MB/sec execute 59 sec latency 216.544 ms 1 1532 0.08 MB/sec execute 60 sec latency 113.676 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 1580 0.08 MB/sec execute 61 sec latency 78.790 ms 1 1619 0.08 MB/sec execute 62 sec latency 109.113 ms 1 1653 0.07 MB/sec execute 63 sec latency 312.545 ms 1 1700 0.07 MB/sec execute 64 sec latency 101.623 ms 1 1773 0.07 MB/sec execute 65 sec latency 90.330 ms 1 1820 0.07 MB/sec execute 66 sec latency 119.836 ms 1 1856 0.07 MB/sec execute 67 sec latency 127.787 ms 1 1884 0.07 MB/sec execute 68 sec latency 360.402 ms 1 1956 0.07 MB/sec execute 69 sec latency 426.985 ms 1 2018 0.07 MB/sec execute 70 sec latency 87.648 ms 1 2098 0.07 MB/sec execute 71 sec latency 113.474 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Sep 14 22:59:14 EDT 2026 with return code 0 1 2152 0.07 MB/sec execute 72 sec latency 139.082 ms 47789 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47789 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished