== replay-dual test 26: dbench and tar with mds failover ========================================================== 08:18:37 (1764940717) Starting client oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre Started clients oleg614-client.virtnet: 192.168.206.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar loop with pid 66842 Started dbench loop with 66843 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Dec 5 08:18:40 EST 2025 waiting for dbench pid 66887 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1984 1285704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3579820 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2796 7185432 1% /mnt/lustre 1 91 4.19 MB/sec warmup 1 sec latency 58.907 ms 1 157 3.27 MB/sec warmup 2 sec latency 104.069 ms 1 224 2.87 MB/sec warmup 3 sec latency 74.148 ms 1 305 2.80 MB/sec warmup 4 sec latency 68.324 ms 1 378 2.69 MB/sec warmup 5 sec latency 67.501 ms test_26 fail mds1 1 times 1 449 2.62 MB/sec warmup 6 sec latency 87.976 ms Failing mds1 on oleg614-server 1 521 2.58 MB/sec warmup 7 sec latency 69.258 ms Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 1 661 2.64 MB/sec warmup 8 sec latency 113.860 ms 1 759 2.37 MB/sec warmup 9 sec latency 60.439 ms 1 865 2.16 MB/sec warmup 10 sec latency 45.817 ms 1 954 1.98 MB/sec warmup 11 sec latency 52.108 ms 1 995 1.82 MB/sec warmup 12 sec latency 419.899 ms 1 995 1.68 MB/sec warmup 13 sec latency 1420.109 ms 1 995 1.56 MB/sec warmup 14 sec latency 2420.281 ms reboot facets: mds1 1 995 1.46 MB/sec warmup 15 sec latency 3421.521 ms 1 995 1.37 MB/sec warmup 16 sec latency 4425.520 ms 1 995 1.29 MB/sec warmup 17 sec latency 5426.059 ms 1 995 1.22 MB/sec warmup 18 sec latency 6426.240 ms 1 995 1.15 MB/sec warmup 19 sec latency 7427.479 ms 1 995 0.00 MB/sec execute 1 sec latency 9429.602 ms 1 995 0.00 MB/sec execute 2 sec latency 10430.539 ms 1 995 0.00 MB/sec execute 3 sec latency 11430.734 ms 1 995 0.00 MB/sec execute 4 sec latency 12431.557 ms Failover mds1 to oleg614-server 1 995 0.00 MB/sec execute 5 sec latency 13432.508 ms mount facets: mds1 1 995 0.00 MB/sec execute 6 sec latency 14432.699 ms 1 995 0.00 MB/sec execute 7 sec latency 15432.830 ms 1 995 0.00 MB/sec execute 8 sec latency 16435.658 ms 1 995 0.00 MB/sec execute 9 sec latency 17435.856 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 10 sec latency 18436.054 ms 1 995 0.00 MB/sec execute 11 sec latency 19436.240 ms 1 995 0.00 MB/sec execute 12 sec latency 20436.575 ms 1 995 0.00 MB/sec execute 13 sec latency 21436.801 ms 1 995 0.00 MB/sec execute 14 sec latency 22437.596 ms 1 995 0.00 MB/sec execute 15 sec latency 23440.435 ms 1 995 0.00 MB/sec execute 16 sec latency 24441.595 ms 1 995 0.00 MB/sec execute 17 sec latency 25441.901 ms 1 995 0.00 MB/sec execute 18 sec latency 26442.091 ms 1 995 0.00 MB/sec execute 19 sec latency 27442.304 ms 1 995 0.00 MB/sec execute 20 sec latency 28442.556 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 1 995 0.00 MB/sec execute 21 sec latency 29443.119 ms Started lustre-MDT0000 1 995 0.00 MB/sec execute 22 sec latency 30443.627 ms 1 995 0.00 MB/sec execute 23 sec latency 31444.664 ms 1 995 0.00 MB/sec execute 24 sec latency 32445.460 ms 1 1041 0.04 MB/sec execute 25 sec latency 33018.848 ms 1 1172 0.04 MB/sec execute 26 sec latency 48.120 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1261 0.04 MB/sec execute 27 sec latency 45.329 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1447 0.15 MB/sec execute 28 sec latency 50.508 ms 1 1591 0.15 MB/sec execute 29 sec latency 40.590 ms 1 1698 0.14 MB/sec execute 30 sec latency 40.480 ms 1 1857 0.14 MB/sec execute 31 sec latency 35.107 ms 1 2018 0.14 MB/sec execute 32 sec latency 31.570 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1285380 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22744 3583024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 35920 3570464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 58664 7153488 1% /mnt/lustre 1 2158 0.15 MB/sec execute 33 sec latency 50.777 ms 1 2333 0.17 MB/sec execute 34 sec latency 33.601 ms 1 2426 0.18 MB/sec execute 35 sec latency 44.506 ms 1 2560 0.19 MB/sec execute 36 sec latency 50.967 ms 1 2763 0.27 MB/sec execute 37 sec latency 56.099 ms test_26 fail mds2 2 times 1 2973 0.31 MB/sec execute 38 sec latency 49.512 ms Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server 1 3209 0.32 MB/sec execute 39 sec latency 53.791 ms 1 3378 0.34 MB/sec execute 40 sec latency 45.546 ms reboot facets: mds2 1 3621 0.41 MB/sec execute 41 sec latency 35.244 ms 1 3760 0.42 MB/sec execute 42 sec latency 34.035 ms 1 3857 0.41 MB/sec execute 43 sec latency 41.694 ms 1 3952 0.41 MB/sec execute 44 sec latency 63.947 ms 1 4037 0.40 MB/sec execute 45 sec latency 65.202 ms 1 4083 0.39 MB/sec execute 46 sec latency 479.502 ms 1 4083 0.39 MB/sec execute 47 sec latency 1480.434 ms 1 4083 0.38 MB/sec execute 48 sec latency 2480.996 ms 1 4083 0.37 MB/sec execute 49 sec latency 3481.442 ms 1 4083 0.36 MB/sec execute 50 sec latency 4482.648 ms Failover mds2 to oleg614-server 1 4083 0.36 MB/sec execute 51 sec latency 5482.832 ms mount facets: mds2 1 4083 0.35 MB/sec execute 52 sec latency 6483.596 ms 1 4083 0.34 MB/sec execute 53 sec latency 7483.805 ms 1 4083 0.34 MB/sec execute 54 sec latency 8484.061 ms 1 4083 0.33 MB/sec execute 55 sec latency 9486.323 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.32 MB/sec execute 56 sec latency 10486.567 ms 1 4083 0.32 MB/sec execute 57 sec latency 11487.626 ms 1 4083 0.31 MB/sec execute 58 sec latency 12490.338 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 1 4083 0.31 MB/sec execute 59 sec latency 13490.536 ms pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 1 4083 0.30 MB/sec execute 60 sec latency 14490.719 ms 1 4083 0.30 MB/sec execute 61 sec latency 15491.643 ms 1 4083 0.29 MB/sec execute 62 sec latency 16491.751 ms 1 4173 0.29 MB/sec execute 63 sec latency 16707.616 ms 1 4279 0.29 MB/sec execute 64 sec latency 66.920 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4350 0.29 MB/sec execute 65 sec latency 67.554 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4443 0.29 MB/sec execute 66 sec latency 57.042 ms 1 4546 0.28 MB/sec execute 67 sec latency 44.904 ms 1 4693 0.30 MB/sec execute 68 sec latency 36.229 ms 1 4793 0.29 MB/sec execute 69 sec latency 50.471 ms 1 4971 0.33 MB/sec execute 70 sec latency 42.617 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2764 1284924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24392 3579988 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44292 3560632 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 68684 7140620 1% /mnt/lustre 1 5068 0.33 MB/sec execute 71 sec latency 56.022 ms 1 5170 0.32 MB/sec execute 72 sec latency 51.440 ms 1 5280 0.32 MB/sec execute 73 sec latency 48.691 ms 1 5370 0.32 MB/sec execute 74 sec latency 45.210 ms 1 5493 0.32 MB/sec execute 75 sec latency 40.990 ms 1 5627 0.31 MB/sec execute 76 sec latency 45.761 ms test_26 fail mds1 3 times 1 5759 0.31 MB/sec execute 77 sec latency 39.368 ms Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 1 5912 0.32 MB/sec execute 78 sec latency 44.895 ms 1 5912 0.32 MB/sec execute 79 sec latency 1023.304 ms reboot facets: mds1 1 5912 0.31 MB/sec execute 80 sec latency 2024.067 ms 1 5912 0.31 MB/sec execute 81 sec latency 3025.295 ms 1 5912 0.31 MB/sec execute 82 sec latency 4025.503 ms 1 5912 0.30 MB/sec execute 83 sec latency 5025.779 ms 1 5912 0.30 MB/sec execute 84 sec latency 6026.811 ms 1 5912 0.30 MB/sec execute 85 sec latency 7027.009 ms 1 5912 0.29 MB/sec execute 86 sec latency 8027.326 ms 1 5912 0.29 MB/sec execute 87 sec latency 9027.630 ms 1 5912 0.29 MB/sec execute 88 sec latency 10027.826 ms 1 5912 0.28 MB/sec execute 89 sec latency 11029.316 ms Failover mds1 to oleg614-server 1 5912 0.28 MB/sec execute 90 sec latency 12029.493 ms mount facets: mds1 1 5912 0.28 MB/sec execute 91 sec latency 13029.680 ms 1 5912 0.27 MB/sec execute 92 sec latency 14030.712 ms 1 5912 0.27 MB/sec execute 93 sec latency 15030.861 ms 1 5912 0.27 MB/sec execute 94 sec latency 16031.369 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5912 0.26 MB/sec execute 95 sec latency 17033.074 ms 1 5912 0.26 MB/sec execute 96 sec latency 18033.255 ms 1 5912 0.26 MB/sec execute 97 sec latency 19033.576 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 1 5912 0.26 MB/sec execute 98 sec latency 20034.021 ms Started lustre-MDT0000 1 5912 0.25 MB/sec execute 99 sec latency 21034.259 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 864 60.068 33018.834 Close 667 40.451 23607.361 Rename 33 31.627 53.766 Unlink 145 10.512 22.809 Qpathinfo 849 25.207 16707.594 Qfileinfo 142 0.838 8.127 Qfsinfo 122 0.856 10.740 Sfileinfo 70 18.443 54.652 Find 291 4.720 38.415 WriteX 424 5.497 35.790 ReadX 1257 0.466 66.417 LockX 2 4.266 4.635 UnlockX 2 4.151 5.330 Flush 50 26.224 56.452 Throughput 0.253432 MB/sec 1 clients 1 procs max_latency=33018.848 ms stopping dbench on /mnt/lustre at Fri Dec 5 08:20:43 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Dec 5 08:20:47 EST 2025 waiting for dbench pid 70276 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 114 5.24 MB/sec warmup 1 sec latency 58.714 ms 1 197 3.92 MB/sec warmup 2 sec latency 59.700 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3268 1284420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3092 1284596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37716 3553056 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11380 3594228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49096 7147284 1% /mnt/lustre 1 279 3.48 MB/sec warmup 3 sec latency 63.559 ms 1 358 3.19 MB/sec warmup 4 sec latency 55.417 ms 1 446 3.12 MB/sec warmup 5 sec latency 83.164 ms 1 526 3.02 MB/sec warmup 6 sec latency 57.247 ms 1 689 3.02 MB/sec warmup 7 sec latency 43.533 ms test_26 fail mds2 4 times 1 789 2.68 MB/sec warmup 8 sec latency 42.775 ms Failing mds2 on oleg614-server 1 892 2.41 MB/sec warmup 9 sec latency 48.154 ms Stopping /mnt/lustre-mds2 (opts:) on oleg614-server 1 981 2.19 MB/sec warmup 10 sec latency 52.132 ms reboot facets: mds2 1 995 1.99 MB/sec warmup 11 sec latency 834.480 ms 1 995 1.82 MB/sec warmup 12 sec latency 1835.040 ms 1 995 1.68 MB/sec warmup 13 sec latency 2835.742 ms 1 995 1.56 MB/sec warmup 14 sec latency 3835.940 ms 1 995 1.46 MB/sec warmup 15 sec latency 4836.315 ms 1 995 1.37 MB/sec warmup 16 sec latency 5838.333 ms 1 995 1.29 MB/sec warmup 17 sec latency 6840.576 ms 1 995 1.22 MB/sec warmup 18 sec latency 7840.752 ms 1 995 1.15 MB/sec warmup 19 sec latency 8841.099 ms Failover mds2 to oleg614-server 1 995 0.00 MB/sec execute 1 sec latency 10842.861 ms mount facets: mds2 1 995 0.00 MB/sec execute 2 sec latency 11843.067 ms 1 995 0.00 MB/sec execute 3 sec latency 12843.227 ms 1 995 0.00 MB/sec execute 4 sec latency 13843.404 ms 1 995 0.00 MB/sec execute 5 sec latency 14844.356 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.00 MB/sec execute 6 sec latency 15844.620 ms 1 995 0.00 MB/sec execute 7 sec latency 16845.529 ms 1 995 0.00 MB/sec execute 8 sec latency 17845.715 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 1 995 0.00 MB/sec execute 9 sec latency 18851.466 ms pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 1 995 0.00 MB/sec execute 10 sec latency 19852.524 ms 1 995 0.00 MB/sec execute 11 sec latency 20852.652 ms 1 995 0.00 MB/sec execute 12 sec latency 21853.617 ms 1 1026 0.07 MB/sec execute 13 sec latency 22454.731 ms 1 1169 0.08 MB/sec execute 14 sec latency 39.007 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1285 0.08 MB/sec execute 15 sec latency 38.332 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1492 0.26 MB/sec execute 16 sec latency 47.104 ms 1 1634 0.25 MB/sec execute 17 sec latency 41.460 ms 1 1773 0.24 MB/sec execute 18 sec latency 43.268 ms 1 1948 0.24 MB/sec execute 19 sec latency 33.433 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3696 1283992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3644 1284044 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 50648 3555976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22484 3584080 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73132 7140056 2% /mnt/lustre 1 2097 0.24 MB/sec execute 20 sec latency 33.602 ms 1 2290 0.28 MB/sec execute 21 sec latency 29.653 ms 1 2417 0.28 MB/sec execute 22 sec latency 39.452 ms 1 2559 0.30 MB/sec execute 23 sec latency 40.415 ms 1 2785 0.42 MB/sec execute 24 sec latency 38.244 ms 1 2997 0.46 MB/sec execute 25 sec latency 45.016 ms test_26 fail mds1 5 times Failing mds1 on oleg614-server 1 3275 0.52 MB/sec execute 26 sec latency 46.083 ms Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 1 3399 0.50 MB/sec execute 27 sec latency 76.770 ms 1 3627 0.60 MB/sec execute 28 sec latency 38.133 ms reboot facets: mds1 1 3760 0.62 MB/sec execute 29 sec latency 35.007 ms 1 3851 0.59 MB/sec execute 30 sec latency 56.272 ms 1 3945 0.59 MB/sec execute 31 sec latency 65.930 ms 1 4049 0.57 MB/sec execute 32 sec latency 45.321 ms 1 4083 0.55 MB/sec execute 33 sec latency 735.743 ms 1 4083 0.53 MB/sec execute 34 sec latency 1735.909 ms 1 4083 0.52 MB/sec execute 35 sec latency 2736.153 ms 1 4083 0.50 MB/sec execute 36 sec latency 3736.343 ms 1 4083 0.49 MB/sec execute 37 sec latency 4736.601 ms 1 4083 0.48 MB/sec execute 38 sec latency 5737.727 ms Failover mds1 to oleg614-server 1 4083 0.47 MB/sec execute 39 sec latency 6739.912 ms mount facets: mds1 1 4083 0.45 MB/sec execute 40 sec latency 7740.592 ms 1 4083 0.44 MB/sec execute 41 sec latency 8740.858 ms 1 4083 0.43 MB/sec execute 42 sec latency 9741.038 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.42 MB/sec execute 43 sec latency 10741.259 ms 1 4083 0.41 MB/sec execute 44 sec latency 11741.419 ms 1 4083 0.40 MB/sec execute 45 sec latency 12741.631 ms 1 4083 0.39 MB/sec execute 46 sec latency 13741.874 ms 1 4083 0.39 MB/sec execute 47 sec latency 14742.332 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 1 4083 0.38 MB/sec execute 48 sec latency 15742.877 ms pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4083 0.37 MB/sec execute 49 sec latency 16744.277 ms 1 4083 0.36 MB/sec execute 50 sec latency 17744.455 ms 1 4083 0.36 MB/sec execute 51 sec latency 18745.058 ms 1 4083 0.35 MB/sec execute 52 sec latency 19745.303 ms 1 4214 0.34 MB/sec execute 53 sec latency 19880.996 ms 1 4309 0.34 MB/sec execute 54 sec latency 69.592 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4427 0.34 MB/sec execute 55 sec latency 45.641 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4503 0.34 MB/sec execute 56 sec latency 85.123 ms 1 4647 0.35 MB/sec execute 57 sec latency 35.926 ms kill: sending signal to 66865 failed: No such process 1 4780 0.35 MB/sec execute 58 sec latency 37.902 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Dec 5 08:22:05 EST 2025 with return code 0 70276 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 70276 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished