== replay-dual test 26: dbench and tar with mds failover ========================================================== 20:15:29 (1718928929) Starting client oleg444-client.virtnet: -o user_xattr,flock oleg444-server@tcp:/lustre /mnt/lustre Started clients oleg444-client.virtnet: 192.168.204.144@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15817 Started dbench loop with 15818 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 20 20:15:31 EDT 2024 waiting for dbench pid 15843 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 207 8.10 MB/sec warmup 1 sec latency 31.416 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3456 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3719168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3662848 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 36864 7382016 1% /mnt/lustre 1 385 6.81 MB/sec warmup 2 sec latency 27.662 ms 1 664 7.05 MB/sec warmup 3 sec latency 31.765 ms 1 910 5.44 MB/sec warmup 4 sec latency 49.445 ms test_26 fail mds1 1 times Failing mds1 on oleg444-server 1 1232 4.61 MB/sec warmup 5 sec latency 44.371 ms Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 1 1421 4.34 MB/sec warmup 6 sec latency 665.487 ms 1 1421 3.72 MB/sec warmup 7 sec latency 1665.725 ms 1 1421 3.25 MB/sec warmup 8 sec latency 2665.894 ms 1 1421 2.89 MB/sec warmup 9 sec latency 3666.096 ms 1 1421 2.60 MB/sec warmup 10 sec latency 4666.337 ms 1 1421 2.36 MB/sec warmup 11 sec latency 5666.546 ms 1 1421 2.17 MB/sec warmup 12 sec latency 6666.713 ms reboot facets: mds1 1 1421 2.00 MB/sec warmup 13 sec latency 7666.917 ms 1 1421 1.86 MB/sec warmup 14 sec latency 8667.211 ms 1 1421 1.73 MB/sec warmup 15 sec latency 9667.410 ms 1 1421 1.63 MB/sec warmup 16 sec latency 10667.689 ms 1 1421 1.53 MB/sec warmup 17 sec latency 11667.892 ms 1 1421 1.45 MB/sec warmup 18 sec latency 12668.102 ms 1 1421 1.37 MB/sec warmup 19 sec latency 13668.380 ms 1 1421 0.00 MB/sec execute 1 sec latency 15668.768 ms 1 1421 0.00 MB/sec execute 2 sec latency 16668.997 ms Failover mds1 to oleg444-server mount facets: mds1 1 1421 0.00 MB/sec execute 3 sec latency 17669.217 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1421 0.00 MB/sec execute 4 sec latency 18669.420 ms 1 1421 0.00 MB/sec execute 5 sec latency 19669.571 ms 1 1421 0.00 MB/sec execute 6 sec latency 20669.756 ms 1 1421 0.00 MB/sec execute 7 sec latency 21669.965 ms 1 1421 0.00 MB/sec execute 8 sec latency 22670.205 ms 1 1421 0.00 MB/sec execute 9 sec latency 23670.387 ms oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1421 0.00 MB/sec execute 10 sec latency 24670.551 ms 1 1421 0.00 MB/sec execute 11 sec latency 25670.690 ms 1 1582 0.01 MB/sec execute 12 sec latency 26175.460 ms oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1946 0.03 MB/sec execute 13 sec latency 41.633 ms 1 2345 0.13 MB/sec execute 14 sec latency 23.603 ms 1 2592 0.20 MB/sec execute 15 sec latency 60.852 ms 1 3176 0.51 MB/sec execute 16 sec latency 47.387 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4864 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3715072 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3610624 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 60416 7325696 1% /mnt/lustre 1 3617 0.74 MB/sec execute 17 sec latency 52.038 ms 1 3868 0.76 MB/sec execute 18 sec latency 16.404 ms 1 4059 0.74 MB/sec execute 19 sec latency 23.060 ms test_26 fail mds1 2 times 1 4309 0.72 MB/sec execute 20 sec latency 89.594 ms Failing mds1 on oleg444-server Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 1 4479 0.71 MB/sec execute 21 sec latency 425.355 ms reboot facets: mds1 1 4479 0.68 MB/sec execute 22 sec latency 1425.538 ms 1 4479 0.65 MB/sec execute 23 sec latency 2425.788 ms 1 4479 0.62 MB/sec execute 24 sec latency 3426.025 ms 1 4479 0.59 MB/sec execute 25 sec latency 4426.199 ms 1 4479 0.57 MB/sec execute 26 sec latency 5426.351 ms 1 4479 0.55 MB/sec execute 27 sec latency 6426.583 ms 1 4479 0.53 MB/sec execute 28 sec latency 7426.918 ms 1 4479 0.51 MB/sec execute 29 sec latency 8427.170 ms 1 4479 0.50 MB/sec execute 30 sec latency 9427.455 ms 1 4479 0.48 MB/sec execute 31 sec latency 10427.669 ms Failover mds1 to oleg444-server mount facets: mds1 1 4479 0.46 MB/sec execute 32 sec latency 11427.875 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4479 0.45 MB/sec execute 33 sec latency 12428.069 ms 1 4479 0.44 MB/sec execute 34 sec latency 13428.268 ms 1 4479 0.42 MB/sec execute 35 sec latency 14428.448 ms oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 1 4479 0.41 MB/sec execute 36 sec latency 15428.647 ms pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4503 0.40 MB/sec execute 37 sec latency 16312.530 ms 1 4967 0.50 MB/sec execute 38 sec latency 50.576 ms oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5294 0.50 MB/sec execute 39 sec latency 49.894 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5712 0.49 MB/sec execute 40 sec latency 43.383 ms 1 6117 0.54 MB/sec execute 41 sec latency 45.024 ms 1 6787 0.65 MB/sec execute 42 sec latency 37.608 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6912 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3457024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 49152 3378176 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 6835200 1% /mnt/lustre 1 7283 0.75 MB/sec execute 43 sec latency 15.141 ms 1 7558 0.75 MB/sec execute 44 sec latency 15.982 ms 1 7875 0.75 MB/sec execute 45 sec latency 68.377 ms test_26 fail mds1 3 times Failing mds1 on oleg444-server 1 8195 0.76 MB/sec execute 46 sec latency 38.857 ms Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 1 8370 0.75 MB/sec execute 47 sec latency 555.435 ms reboot facets: mds1 1 8370 0.74 MB/sec execute 48 sec latency 1555.591 ms 1 8370 0.72 MB/sec execute 49 sec latency 2555.735 ms 1 8370 0.71 MB/sec execute 50 sec latency 3555.902 ms 1 8370 0.69 MB/sec execute 51 sec latency 4556.081 ms 1 8370 0.68 MB/sec execute 52 sec latency 5556.222 ms 1 8370 0.67 MB/sec execute 53 sec latency 6556.420 ms 1 8370 0.65 MB/sec execute 54 sec latency 7556.577 ms 1 8370 0.64 MB/sec execute 55 sec latency 8556.710 ms 1 8370 0.63 MB/sec execute 56 sec latency 9557.028 ms 1 8370 0.62 MB/sec execute 57 sec latency 10557.191 ms Failover mds1 to oleg444-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8370 0.61 MB/sec execute 58 sec latency 11557.293 ms 1 8370 0.60 MB/sec execute 59 sec latency 12557.427 ms 1 8370 0.59 MB/sec execute 60 sec latency 13557.574 ms 1 8370 0.58 MB/sec execute 61 sec latency 14558.546 ms 1 8370 0.57 MB/sec execute 62 sec latency 15558.755 ms oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 1 8370 0.56 MB/sec execute 63 sec latency 16558.935 ms Started lustre-MDT0000 1 8370 0.55 MB/sec execute 64 sec latency 17559.087 ms 1 8370 0.54 MB/sec execute 65 sec latency 18559.261 ms 1 8370 0.54 MB/sec execute 66 sec latency 19559.446 ms 1 8370 0.53 MB/sec execute 67 sec latency 20559.631 ms 1 8370 0.52 MB/sec execute 68 sec latency 21559.837 ms 1 8370 0.51 MB/sec execute 69 sec latency 22560.078 ms 1 8370 0.50 MB/sec execute 70 sec latency 23560.257 ms 1 8370 0.50 MB/sec execute 71 sec latency 24560.454 ms 1 8370 0.49 MB/sec execute 72 sec latency 25560.647 ms 1 8370 0.48 MB/sec execute 73 sec latency 26560.845 ms 1 8370 0.48 MB/sec execute 74 sec latency 27560.966 ms 1 8370 0.47 MB/sec execute 75 sec latency 28561.090 ms 1 8370 0.46 MB/sec execute 76 sec latency 29561.234 ms 1 8668 0.50 MB/sec execute 77 sec latency 29905.095 ms oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8995 0.49 MB/sec execute 78 sec latency 47.378 ms 1 9472 0.51 MB/sec execute 79 sec latency 49.113 ms 1 9998 0.56 MB/sec execute 80 sec latency 40.609 ms 1 10607 0.62 MB/sec execute 81 sec latency 37.595 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8832 2199808 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3627008 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 52224 3598336 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7225344 1% /mnt/lustre 1 10937 0.63 MB/sec execute 82 sec latency 19.386 ms 1 11223 0.63 MB/sec execute 83 sec latency 15.337 ms 1 11469 0.63 MB/sec execute 84 sec latency 80.054 ms test_26 fail mds1 4 times Failing mds1 on oleg444-server Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 1 11792 0.64 MB/sec execute 85 sec latency 46.525 ms reboot facets: mds1 1 11854 0.63 MB/sec execute 86 sec latency 810.321 ms 1 11854 0.63 MB/sec execute 87 sec latency 1810.551 ms 1 11854 0.62 MB/sec execute 88 sec latency 2810.698 ms 1 11854 0.61 MB/sec execute 89 sec latency 3810.881 ms 1 11854 0.60 MB/sec execute 90 sec latency 4811.052 ms 1 11854 0.60 MB/sec execute 91 sec latency 5811.228 ms 1 11854 0.59 MB/sec execute 92 sec latency 6811.418 ms 1 11854 0.58 MB/sec execute 93 sec latency 7811.565 ms 1 11854 0.58 MB/sec execute 94 sec latency 8811.704 ms 1 11854 0.57 MB/sec execute 95 sec latency 9811.842 ms Failover mds1 to oleg444-server 1 11854 0.57 MB/sec execute 96 sec latency 10811.941 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 11854 0.56 MB/sec execute 97 sec latency 11812.050 ms 1 11854 0.55 MB/sec execute 98 sec latency 12812.173 ms 1 11854 0.55 MB/sec execute 99 sec latency 13812.367 ms 1 cleanup 100 sec oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1812 31.319 26175.451 Close 1329 13.691 16312.505 Rename 75 10.110 20.088 Unlink 368 3.869 12.428 Qpathinfo 1662 1.948 11.960 Qfileinfo 286 0.314 2.317 Qfsinfo 301 0.344 1.990 Sfileinfo 143 214.070 29905.080 Find 633 0.961 10.667 WriteX 899 1.602 8.618 ReadX 2785 0.071 23.577 LockX 6 1.309 1.717 UnlockX 6 1.395 1.675 Flush 129 25.530 89.582 Throughput 0.549305 MB/sec 1 clients 1 procs max_latency=29905.095 ms stopping dbench on /mnt/lustre at Thu Jun 20 20:17:33 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 20 20:17:34 EDT 2024 waiting for dbench pid 20239 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 312 11.48 MB/sec warmup 1 sec latency 18.200 ms 1 664 10.58 MB/sec warmup 2 sec latency 61.532 ms 1 923 7.26 MB/sec warmup 3 sec latency 84.811 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10496 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3386368 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3477504 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 6863872 1% /mnt/lustre 1 1377 6.50 MB/sec warmup 4 sec latency 31.587 ms 1 1770 5.26 MB/sec warmup 5 sec latency 36.404 ms 1 2186 4.46 MB/sec warmup 6 sec latency 31.213 ms test_26 fail mds1 5 times Failing mds1 on oleg444-server Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 1 2590 4.14 MB/sec warmup 7 sec latency 38.309 ms reboot facets: mds1 1 2768 4.00 MB/sec warmup 8 sec latency 803.164 ms 1 2768 3.55 MB/sec warmup 9 sec latency 1803.336 ms 1 2768 3.20 MB/sec warmup 10 sec latency 2803.494 ms 1 2768 2.91 MB/sec warmup 11 sec latency 3803.655 ms 1 2768 2.66 MB/sec warmup 12 sec latency 4803.817 ms 1 2768 2.46 MB/sec warmup 13 sec latency 5803.965 ms 1 2768 2.28 MB/sec warmup 14 sec latency 6804.127 ms 1 2768 2.13 MB/sec warmup 15 sec latency 7804.292 ms 1 2768 2.00 MB/sec warmup 16 sec latency 8804.483 ms 1 2768 1.88 MB/sec warmup 17 sec latency 9804.646 ms Failover mds1 to oleg444-server 1 2768 1.78 MB/sec warmup 18 sec latency 10804.783 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2768 1.68 MB/sec warmup 19 sec latency 11804.973 ms 1 2768 0.00 MB/sec execute 1 sec latency 13805.262 ms 1 2768 0.00 MB/sec execute 2 sec latency 14805.395 ms 1 2768 0.00 MB/sec execute 3 sec latency 15805.556 ms 1 2768 0.00 MB/sec execute 4 sec latency 16805.688 ms oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 1 2768 0.00 MB/sec execute 5 sec latency 17806.526 ms Started lustre-MDT0000 1 2768 0.00 MB/sec execute 6 sec latency 18806.701 ms 1 2768 0.00 MB/sec execute 7 sec latency 19806.804 ms 1 2768 0.00 MB/sec execute 8 sec latency 20807.016 ms 1 2768 0.00 MB/sec execute 9 sec latency 21807.246 ms 1 2768 0.00 MB/sec execute 10 sec latency 22807.485 ms 1 2768 0.00 MB/sec execute 11 sec latency 23807.694 ms 1 2768 0.00 MB/sec execute 12 sec latency 24807.856 ms 1 2768 0.00 MB/sec execute 13 sec latency 25808.110 ms 1 2768 0.00 MB/sec execute 14 sec latency 26809.169 ms 1 2768 0.00 MB/sec execute 15 sec latency 27809.409 ms 1 2768 0.00 MB/sec execute 16 sec latency 28810.454 ms 1 2768 0.00 MB/sec execute 17 sec latency 29811.279 ms 1 2918 0.08 MB/sec execute 18 sec latency 30584.255 ms oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3533 0.34 MB/sec execute 19 sec latency 45.640 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3879 0.39 MB/sec execute 20 sec latency 14.840 ms 1 4211 0.39 MB/sec execute 21 sec latency 16.091 ms 1 4444 0.40 MB/sec execute 22 sec latency 113.889 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10368 2198272 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 65536 3650560 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 27648 3714048 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 93184 7364608 2% /mnt/lustre 1 4810 0.44 MB/sec execute 23 sec latency 31.180 ms 1 5197 0.56 MB/sec execute 24 sec latency 83.044 ms 1 5575 0.55 MB/sec execute 25 sec latency 54.125 ms test_26 fail mds1 6 times 1 5996 0.59 MB/sec execute 26 sec latency 39.699 ms Failing mds1 on oleg444-server Stopping /mnt/lustre-mds1 (opts:) on oleg444-server 1 6319 0.71 MB/sec execute 27 sec latency 494.100 ms reboot facets: mds1 1 6319 0.68 MB/sec execute 28 sec latency 1494.273 ms 1 6319 0.66 MB/sec execute 29 sec latency 2494.432 ms 1 6319 0.64 MB/sec execute 30 sec latency 3494.652 ms 1 6319 0.62 MB/sec execute 31 sec latency 4494.850 ms 1 6319 0.60 MB/sec execute 32 sec latency 5495.013 ms 1 6319 0.58 MB/sec execute 33 sec latency 6495.199 ms 1 6319 0.56 MB/sec execute 34 sec latency 7495.381 ms 1 6319 0.55 MB/sec execute 35 sec latency 8495.542 ms 1 6319 0.53 MB/sec execute 36 sec latency 9495.722 ms 1 6319 0.52 MB/sec execute 37 sec latency 10495.957 ms Failover mds1 to oleg444-server mount facets: mds1 1 6319 0.50 MB/sec execute 38 sec latency 11496.135 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6319 0.49 MB/sec execute 39 sec latency 12496.309 ms 1 6319 0.48 MB/sec execute 40 sec latency 13496.558 ms 1 6319 0.47 MB/sec execute 41 sec latency 14496.707 ms 1 6319 0.45 MB/sec execute 42 sec latency 15496.902 ms oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 1 6319 0.44 MB/sec execute 43 sec latency 16497.046 ms pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 1 6537 0.47 MB/sec execute 44 sec latency 17091.835 ms 1 7174 0.57 MB/sec execute 45 sec latency 27.290 ms oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7483 0.59 MB/sec execute 46 sec latency 14.623 ms 1 7778 0.58 MB/sec execute 47 sec latency 50.012 ms 1 8022 0.58 MB/sec execute 48 sec latency 93.373 ms 1 8427 0.62 MB/sec execute 49 sec latency 29.209 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6784 2201856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 61440 3677184 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3732480 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 81920 7409664 2% /mnt/lustre 1 8792 0.65 MB/sec execute 50 sec latency 72.397 ms 1 9194 0.65 MB/sec execute 51 sec latency 23.211 ms 1 9559 0.66 MB/sec execute 52 sec latency 58.517 ms test_26 fail mds1 7 times Failing mds1 on oleg444-server 1 10111 0.75 MB/sec execute 53 sec latency 46.258 ms Stopping /mnt/lustre-mds1 (opts:) on oleg444-server reboot facets: mds1 1 10428 0.77 MB/sec execute 54 sec latency 648.954 ms 1 10428 0.76 MB/sec execute 55 sec latency 1649.137 ms 1 10428 0.74 MB/sec execute 56 sec latency 2649.321 ms 1 10428 0.73 MB/sec execute 57 sec latency 3649.531 ms 1 10428 0.72 MB/sec execute 58 sec latency 4649.735 ms 1 10428 0.71 MB/sec execute 59 sec latency 5649.929 ms 1 10428 0.69 MB/sec execute 60 sec latency 6650.106 ms 1 10428 0.68 MB/sec execute 61 sec latency 7650.357 ms 1 10428 0.67 MB/sec execute 62 sec latency 8650.574 ms 1 10428 0.66 MB/sec execute 63 sec latency 9650.720 ms Failover mds1 to oleg444-server 1 10428 0.65 MB/sec execute 64 sec latency 10650.820 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 10428 0.64 MB/sec execute 65 sec latency 11650.925 ms 1 10428 0.63 MB/sec execute 66 sec latency 12651.057 ms 1 10428 0.62 MB/sec execute 67 sec latency 13651.258 ms 1 10428 0.61 MB/sec execute 68 sec latency 14651.430 ms 1 10428 0.60 MB/sec execute 69 sec latency 15651.571 ms 1 10428 0.59 MB/sec execute 70 sec latency 16651.767 ms 1 10428 0.59 MB/sec execute 71 sec latency 17651.990 ms oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 1 10428 0.58 MB/sec execute 72 sec latency 18652.227 ms 1 10428 0.57 MB/sec execute 73 sec latency 19652.405 ms 1 10428 0.56 MB/sec execute 74 sec latency 20652.580 ms 1 10428 0.56 MB/sec execute 75 sec latency 21652.804 ms 1 10428 0.55 MB/sec execute 76 sec latency 22653.015 ms 1 10428 0.54 MB/sec execute 77 sec latency 23653.254 ms 1 10428 0.53 MB/sec execute 78 sec latency 24653.460 ms 1 10428 0.53 MB/sec execute 79 sec latency 25653.665 ms 1 10428 0.52 MB/sec execute 80 sec latency 26653.865 ms 1 10428 0.51 MB/sec execute 81 sec latency 27654.041 ms 1 10428 0.51 MB/sec execute 82 sec latency 28654.271 ms 1 10428 0.50 MB/sec execute 83 sec latency 29654.422 ms 1 10428 0.50 MB/sec execute 84 sec latency 30654.619 ms 1 10428 0.49 MB/sec execute 85 sec latency 31654.748 ms 1 10593 0.52 MB/sec execute 86 sec latency 32369.455 ms oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 10937 0.53 MB/sec execute 87 sec latency 13.871 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11221 0.53 MB/sec execute 88 sec latency 13.460 ms 1 11458 0.53 MB/sec execute 89 sec latency 62.837 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Jun 20 20:19:24 EDT 2024 with return code 0 20239 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20239 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished