== replay-dual test 26: dbench and tar with mds failover ========================================================== 23:27:16 (1718854036) Starting client oleg306-client.virtnet: -o user_xattr,flock oleg306-server@tcp:/lustre /mnt/lustre Started clients oleg306-client.virtnet: 192.168.203.106@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 2880 Started dbench loop with 2881 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jun 19 23:27:18 EDT 2024 waiting for dbench pid 2922 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2232 1285456 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1980 1285708 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26120 3580676 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27656 7186160 1% /mnt/lustre 1 155 6.54 MB/sec warmup 1 sec latency 36.180 ms 1 312 5.74 MB/sec warmup 2 sec latency 36.724 ms 1 421 4.95 MB/sec warmup 3 sec latency 252.746 ms 1 609 5.24 MB/sec warmup 4 sec latency 31.415 ms 1 779 4.27 MB/sec warmup 5 sec latency 28.193 ms test_26 fail mds1 1 times Failing mds1 on oleg306-server 1 933 3.63 MB/sec warmup 6 sec latency 26.351 ms Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 1 1036 3.27 MB/sec warmup 7 sec latency 472.178 ms 23:27:26 (1718854046) shut down 1 1036 2.86 MB/sec warmup 8 sec latency 1472.397 ms 1 1036 2.54 MB/sec warmup 9 sec latency 2472.617 ms 1 1036 2.29 MB/sec warmup 10 sec latency 3472.916 ms 1 1036 2.08 MB/sec warmup 11 sec latency 4473.178 ms 1 1036 1.91 MB/sec warmup 12 sec latency 5473.461 ms 1 1036 1.76 MB/sec warmup 13 sec latency 6473.716 ms 1 1036 1.63 MB/sec warmup 14 sec latency 7473.905 ms 1 1036 1.53 MB/sec warmup 15 sec latency 8474.145 ms 1 1036 1.43 MB/sec warmup 16 sec latency 9474.356 ms 1 1036 1.35 MB/sec warmup 17 sec latency 10474.618 ms Failover mds1 to oleg306-server mount facets: mds1 1 1036 1.27 MB/sec warmup 18 sec latency 11474.837 ms 1 1036 1.20 MB/sec warmup 19 sec latency 12475.000 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 1 1036 0.00 MB/sec execute 1 sec latency 14475.267 ms pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 23:27:40 (1718854060) targets are mounted 23:27:40 (1718854060) facet_failover done 1 1036 0.00 MB/sec execute 2 sec latency 15475.417 ms 1 1036 0.00 MB/sec execute 3 sec latency 16475.569 ms 1 1036 0.00 MB/sec execute 4 sec latency 17475.745 ms 1 1036 0.00 MB/sec execute 5 sec latency 18475.942 ms 1 1036 0.00 MB/sec execute 6 sec latency 19476.146 ms oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1169 0.02 MB/sec execute 7 sec latency 19743.655 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1403 0.39 MB/sec execute 8 sec latency 25.781 ms 1 1568 0.36 MB/sec execute 9 sec latency 27.414 ms 1 1705 0.33 MB/sec execute 10 sec latency 28.111 ms 1 1932 0.32 MB/sec execute 11 sec latency 21.825 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 31496 3570024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7328 3592932 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38824 7162956 1% /mnt/lustre 1 2117 0.32 MB/sec execute 12 sec latency 25.293 ms 1 2345 0.39 MB/sec execute 13 sec latency 28.817 ms 1 2406 0.37 MB/sec execute 14 sec latency 598.096 ms 1 2608 0.41 MB/sec execute 15 sec latency 25.542 ms test_26 fail mds2 2 times 1 3008 0.67 MB/sec execute 16 sec latency 29.268 ms Failing mds2 on oleg306-server Stopping /mnt/lustre-mds2 (opts:) on oleg306-server 1 3410 0.76 MB/sec execute 17 sec latency 24.383 ms 23:27:56 (1718854076) shut down 1 3530 0.86 MB/sec execute 18 sec latency 853.659 ms 1 3530 0.82 MB/sec execute 19 sec latency 1853.878 ms 1 3530 0.77 MB/sec execute 20 sec latency 2854.083 ms 1 3530 0.74 MB/sec execute 21 sec latency 3854.352 ms 1 3530 0.70 MB/sec execute 22 sec latency 4854.680 ms 1 3530 0.67 MB/sec execute 23 sec latency 5854.838 ms 1 3530 0.65 MB/sec execute 24 sec latency 6855.007 ms 1 3530 0.62 MB/sec execute 25 sec latency 7855.225 ms 1 3530 0.60 MB/sec execute 26 sec latency 8855.493 ms 1 3530 0.57 MB/sec execute 27 sec latency 9855.653 ms Failover mds2 to oleg306-server mount facets: mds2 1 3530 0.55 MB/sec execute 28 sec latency 10855.779 ms 1 3530 0.53 MB/sec execute 29 sec latency 11855.946 ms 1 3530 0.52 MB/sec execute 30 sec latency 12856.148 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 3530 0.50 MB/sec execute 31 sec latency 13856.273 ms oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 1 3530 0.48 MB/sec execute 32 sec latency 14856.514 ms Started lustre-MDT0001 23:28:11 (1718854091) targets are mounted 23:28:11 (1718854091) facet_failover done 1 3530 0.47 MB/sec execute 33 sec latency 15856.737 ms 1 3530 0.46 MB/sec execute 34 sec latency 16856.957 ms 1 3530 0.44 MB/sec execute 35 sec latency 17857.211 ms 1 3530 0.43 MB/sec execute 36 sec latency 18857.408 ms oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 3628 0.42 MB/sec execute 37 sec latency 19540.550 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3854 0.44 MB/sec execute 38 sec latency 26.763 ms 1 4017 0.44 MB/sec execute 39 sec latency 21.371 ms 1 4208 0.43 MB/sec execute 40 sec latency 25.483 ms 1 4444 0.44 MB/sec execute 41 sec latency 23.371 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2908 1284780 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2968 1284720 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 39628 3559040 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13184 3588140 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 52812 7147180 1% /mnt/lustre 1 4650 0.46 MB/sec execute 42 sec latency 39.841 ms 1 4951 0.52 MB/sec execute 43 sec latency 21.869 ms 1 5157 0.51 MB/sec execute 44 sec latency 21.640 ms 1 5367 0.50 MB/sec execute 45 sec latency 25.147 ms test_26 fail mds1 3 times Failing mds1 on oleg306-server Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 1 5659 0.50 MB/sec execute 46 sec latency 15.341 ms 23:28:25 (1718854105) shut down 1 5718 0.49 MB/sec execute 47 sec latency 780.532 ms 1 5718 0.48 MB/sec execute 48 sec latency 1780.702 ms 1 5718 0.47 MB/sec execute 49 sec latency 2780.824 ms 1 5718 0.46 MB/sec execute 50 sec latency 3780.987 ms 1 5718 0.45 MB/sec execute 51 sec latency 4781.106 ms 1 5718 0.44 MB/sec execute 52 sec latency 5781.203 ms 1 5718 0.43 MB/sec execute 53 sec latency 6781.385 ms 1 5718 0.42 MB/sec execute 54 sec latency 7781.560 ms 1 5718 0.42 MB/sec execute 55 sec latency 8781.801 ms 1 5718 0.41 MB/sec execute 56 sec latency 9782.994 ms 1 5718 0.40 MB/sec execute 57 sec latency 10783.106 ms Failover mds1 to oleg306-server mount facets: mds1 1 5718 0.40 MB/sec execute 58 sec latency 11783.240 ms 1 5718 0.39 MB/sec execute 59 sec latency 12783.349 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5718 0.38 MB/sec execute 60 sec latency 13783.484 ms oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 1 5718 0.38 MB/sec execute 61 sec latency 14783.679 ms pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 23:28:40 (1718854120) targets are mounted 23:28:40 (1718854120) facet_failover done 1 5718 0.37 MB/sec execute 62 sec latency 15783.837 ms 1 5718 0.36 MB/sec execute 63 sec latency 16783.998 ms 1 5718 0.36 MB/sec execute 64 sec latency 17784.182 ms 1 5718 0.35 MB/sec execute 65 sec latency 18784.397 ms oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5913 0.37 MB/sec execute 66 sec latency 19233.397 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6139 0.38 MB/sec execute 67 sec latency 21.984 ms 1 6548 0.44 MB/sec execute 68 sec latency 23.595 ms 1 6944 0.46 MB/sec execute 69 sec latency 23.267 ms 1 7261 0.50 MB/sec execute 70 sec latency 16.096 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2772 1284916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4248 1283440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38244 3551800 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12364 3574484 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50608 7126284 1% /mnt/lustre 1 7495 0.51 MB/sec execute 71 sec latency 24.288 ms 1 7668 0.50 MB/sec execute 72 sec latency 23.415 ms 1 7902 0.50 MB/sec execute 73 sec latency 21.602 ms 1 8134 0.51 MB/sec execute 74 sec latency 39.518 ms test_26 fail mds2 4 times Failing mds2 on oleg306-server Stopping /mnt/lustre-mds2 (opts:) on oleg306-server 1 8427 0.53 MB/sec execute 75 sec latency 18.000 ms 1 8542 0.54 MB/sec execute 76 sec latency 716.621 ms 1 8542 0.54 MB/sec execute 77 sec latency 1716.780 ms 1 8542 0.53 MB/sec execute 78 sec latency 2716.928 ms 1 8542 0.52 MB/sec execute 79 sec latency 3717.107 ms 1 8542 0.52 MB/sec execute 80 sec latency 4717.271 ms 23:28:59 (1718854139) shut down 1 8542 0.51 MB/sec execute 81 sec latency 5717.383 ms 1 8542 0.50 MB/sec execute 82 sec latency 6717.548 ms 1 8542 0.50 MB/sec execute 83 sec latency 7717.739 ms 1 8542 0.49 MB/sec execute 84 sec latency 8717.910 ms 1 8542 0.49 MB/sec execute 85 sec latency 9718.112 ms 1 8542 0.48 MB/sec execute 86 sec latency 10718.324 ms 1 8542 0.48 MB/sec execute 87 sec latency 11718.554 ms 1 8542 0.47 MB/sec execute 88 sec latency 12718.807 ms 1 8542 0.46 MB/sec execute 89 sec latency 13718.977 ms 1 8542 0.46 MB/sec execute 90 sec latency 14719.141 ms Failover mds2 to oleg306-server 1 8542 0.45 MB/sec execute 91 sec latency 15719.298 ms mount facets: mds2 1 8542 0.45 MB/sec execute 92 sec latency 16719.515 ms 1 8542 0.44 MB/sec execute 93 sec latency 17719.713 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 8542 0.44 MB/sec execute 94 sec latency 18719.947 ms oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 1 8542 0.44 MB/sec execute 95 sec latency 19720.149 ms pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0001 23:29:14 (1718854154) targets are mounted 23:29:14 (1718854154) facet_failover done 1 8542 0.43 MB/sec execute 96 sec latency 20720.378 ms 1 8542 0.43 MB/sec execute 97 sec latency 21720.657 ms 1 8542 0.42 MB/sec execute 98 sec latency 22720.972 ms 1 8542 0.42 MB/sec execute 99 sec latency 23721.278 ms 1 cleanup 100 sec 1 cleanup 101 sec oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 0 cleanup 102 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1278 41.797 19540.534 Close 948 29.468 25666.983 Rename 58 16.094 28.000 Unlink 255 5.754 17.664 Qpathinfo 1202 19.894 19743.631 Qfileinfo 205 0.590 4.999 Qfsinfo 215 1.121 7.432 Sfileinfo 116 9.310 39.832 Find 452 3.726 17.551 WriteX 631 2.647 12.384 ReadX 2053 0.171 17.897 LockX 4 1.889 1.916 UnlockX 4 1.996 2.293 Flush 86 17.895 598.082 Throughput 0.417834 MB/sec 1 clients 1 procs max_latency=23721.278 ms stopping dbench on /mnt/lustre at Wed Jun 19 23:29:20 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed Jun 19 23:29:21 EDT 2024 waiting for dbench pid 7288 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 218 8.36 MB/sec warmup 1 sec latency 25.814 ms 1 397 7.04 MB/sec warmup 2 sec latency 28.306 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3112 1284576 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4672 1283016 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 16792 3573988 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33764 3544212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50556 7118200 1% /mnt/lustre 1 682 7.06 MB/sec warmup 3 sec latency 23.681 ms 1 898 5.42 MB/sec warmup 4 sec latency 21.903 ms 1 1116 4.59 MB/sec warmup 5 sec latency 24.476 ms 1 1430 4.34 MB/sec warmup 6 sec latency 26.787 ms test_26 fail mds1 5 times Failing mds1 on oleg306-server 1 1689 3.75 MB/sec warmup 7 sec latency 23.232 ms Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 1 1841 3.29 MB/sec warmup 8 sec latency 499.724 ms 1 1841 2.93 MB/sec warmup 9 sec latency 1499.922 ms 1 1841 2.64 MB/sec warmup 10 sec latency 2500.176 ms 1 1841 2.40 MB/sec warmup 11 sec latency 3500.424 ms 1 1841 2.20 MB/sec warmup 12 sec latency 4500.625 ms 1 1841 2.03 MB/sec warmup 13 sec latency 5500.798 ms 23:29:35 (1718854175) shut down 1 1841 1.88 MB/sec warmup 14 sec latency 6500.970 ms 1 1841 1.76 MB/sec warmup 15 sec latency 7501.168 ms 1 1841 1.65 MB/sec warmup 16 sec latency 8501.314 ms 1 1841 1.55 MB/sec warmup 17 sec latency 9501.511 ms 1 1841 1.46 MB/sec warmup 18 sec latency 10501.700 ms 1 1841 1.39 MB/sec warmup 19 sec latency 11501.859 ms 1 1841 0.00 MB/sec execute 1 sec latency 13502.186 ms 1 1841 0.00 MB/sec execute 2 sec latency 14502.392 ms 1 1841 0.00 MB/sec execute 3 sec latency 15502.549 ms Failover mds1 to oleg306-server mount facets: mds1 1 1841 0.00 MB/sec execute 4 sec latency 16502.681 ms 1 1841 0.00 MB/sec execute 5 sec latency 17502.838 ms 1 1841 0.00 MB/sec execute 6 sec latency 18503.061 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1841 0.00 MB/sec execute 7 sec latency 19503.417 ms oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 1 1841 0.00 MB/sec execute 8 sec latency 20503.594 ms pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 23:29:50 (1718854190) targets are mounted 23:29:50 (1718854190) facet_failover done 1 1841 0.00 MB/sec execute 9 sec latency 21503.757 ms 1 1841 0.00 MB/sec execute 10 sec latency 22504.018 ms 1 1841 0.00 MB/sec execute 11 sec latency 23504.170 ms 1 1841 0.00 MB/sec execute 12 sec latency 24504.344 ms oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1924 0.01 MB/sec execute 13 sec latency 25256.563 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2253 0.03 MB/sec execute 14 sec latency 14.822 ms 1 2473 0.15 MB/sec execute 15 sec latency 21.450 ms 1 2897 0.40 MB/sec execute 16 sec latency 22.593 ms 1 3337 0.53 MB/sec execute 17 sec latency 33.321 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3740 1283948 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5148 1282540 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 31652 3571588 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47944 3554440 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 79596 7126028 2% /mnt/lustre 1 3702 0.69 MB/sec execute 18 sec latency 17.787 ms 1 3892 0.72 MB/sec execute 19 sec latency 20.721 ms 1 4046 0.69 MB/sec execute 20 sec latency 19.000 ms 1 4284 0.67 MB/sec execute 21 sec latency 22.438 ms test_26 fail mds2 6 times Failing mds2 on oleg306-server 1 4524 0.66 MB/sec execute 22 sec latency 23.267 ms Stopping /mnt/lustre-mds2 (opts:) on oleg306-server 23:30:05 (1718854205) shut down 1 4618 0.68 MB/sec execute 23 sec latency 625.304 ms 1 4618 0.65 MB/sec execute 24 sec latency 1625.436 ms 1 4618 0.63 MB/sec execute 25 sec latency 2625.595 ms 1 4618 0.60 MB/sec execute 26 sec latency 3625.791 ms 1 4618 0.58 MB/sec execute 27 sec latency 4625.976 ms 1 4618 0.56 MB/sec execute 28 sec latency 5626.200 ms 1 4618 0.54 MB/sec execute 29 sec latency 6626.434 ms 1 4618 0.52 MB/sec execute 30 sec latency 7626.587 ms 1 4618 0.50 MB/sec execute 31 sec latency 8626.790 ms 1 4618 0.49 MB/sec execute 32 sec latency 9627.048 ms 1 4618 0.47 MB/sec execute 33 sec latency 10627.198 ms Failover mds2 to oleg306-server mount facets: mds2 1 4618 0.46 MB/sec execute 34 sec latency 11627.411 ms 1 4618 0.45 MB/sec execute 35 sec latency 12627.591 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4618 0.43 MB/sec execute 36 sec latency 13627.782 ms oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 1 4618 0.42 MB/sec execute 37 sec latency 14627.970 ms Started lustre-MDT0001 23:30:19 (1718854219) targets are mounted 23:30:19 (1718854219) facet_failover done 1 4618 0.41 MB/sec execute 38 sec latency 15628.178 ms 1 4618 0.40 MB/sec execute 39 sec latency 16628.378 ms 1 4618 0.39 MB/sec execute 40 sec latency 17628.540 ms 1 4618 0.38 MB/sec execute 41 sec latency 18628.672 ms 1 4654 0.37 MB/sec execute 42 sec latency 19482.696 ms oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5021 0.44 MB/sec execute 43 sec latency 20.563 ms 1 5227 0.43 MB/sec execute 44 sec latency 37.024 ms 1 5431 0.42 MB/sec execute 45 sec latency 17.995 ms 1 5732 0.42 MB/sec execute 46 sec latency 23.681 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3096 1284592 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4564 1283124 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 31676 3574220 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47732 3558248 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 79408 7132468 2% /mnt/lustre 1 5978 0.44 MB/sec execute 47 sec latency 44.634 ms 1 6324 0.52 MB/sec execute 48 sec latency 146.889 ms 1 6567 0.54 MB/sec execute 49 sec latency 394.588 ms 1 6934 0.56 MB/sec execute 50 sec latency 24.639 ms test_26 fail mds1 7 times 1 7247 0.62 MB/sec execute 51 sec latency 18.086 ms Failing mds1 on oleg306-server Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 1 7403 0.63 MB/sec execute 52 sec latency 240.592 ms 23:30:34 (1718854234) shut down 1 7403 0.61 MB/sec execute 53 sec latency 1240.692 ms 1 7403 0.60 MB/sec execute 54 sec latency 2240.897 ms 1 7403 0.59 MB/sec execute 55 sec latency 3241.091 ms 1 7403 0.58 MB/sec execute 56 sec latency 4241.302 ms 1 7403 0.57 MB/sec execute 57 sec latency 5241.499 ms 1 7403 0.56 MB/sec execute 58 sec latency 6241.637 ms 1 7403 0.55 MB/sec execute 59 sec latency 7241.861 ms 1 7403 0.54 MB/sec execute 60 sec latency 8242.065 ms 1 7403 0.53 MB/sec execute 61 sec latency 9242.282 ms 1 7403 0.52 MB/sec execute 62 sec latency 10242.497 ms Failover mds1 to oleg306-server mount facets: mds1 1 7403 0.52 MB/sec execute 63 sec latency 11242.709 ms 1 7403 0.51 MB/sec execute 64 sec latency 12242.842 ms 1 7403 0.50 MB/sec execute 65 sec latency 13242.995 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7403 0.49 MB/sec execute 66 sec latency 14243.150 ms oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all 1 7403 0.49 MB/sec execute 67 sec latency 15243.320 ms pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 23:30:49 (1718854249) targets are mounted 23:30:49 (1718854249) facet_failover done 1 7403 0.48 MB/sec execute 68 sec latency 16243.449 ms 1 7403 0.47 MB/sec execute 69 sec latency 17243.652 ms 1 7403 0.46 MB/sec execute 70 sec latency 18243.843 ms 1 7403 0.46 MB/sec execute 71 sec latency 19244.541 ms oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7471 0.46 MB/sec execute 72 sec latency 19869.252 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7646 0.45 MB/sec execute 73 sec latency 26.278 ms 1 7854 0.45 MB/sec execute 74 sec latency 29.981 ms 1 8070 0.45 MB/sec execute 75 sec latency 22.541 ms 1 8315 0.46 MB/sec execute 76 sec latency 21.838 ms 1 8601 0.49 MB/sec execute 77 sec latency 37.135 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed Jun 19 23:30:59 EDT 2024 with return code 0 7288 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 7288 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished