== replay-dual test 26: dbench and tar with mds failover ========================================================== 17:34:44 (1715463284) Starting client oleg446-client.virtnet: -o user_xattr,flock oleg446-server@tcp:/lustre /mnt/lustre Started clients oleg446-client.virtnet: 192.168.204.146@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 6421 Started dbench loop with 6422 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat May 11 17:34:45 EDT 2024 waiting for dbench pid 6463 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 247 9.40 MB/sec warmup 1 sec latency 20.144 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1285580 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25964 3555692 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27372 7161304 1% /mnt/lustre 1 511 8.86 MB/sec warmup 2 sec latency 19.830 ms 1 744 7.11 MB/sec warmup 3 sec latency 409.521 ms 1 985 5.48 MB/sec warmup 4 sec latency 19.068 ms test_26 fail mds1 1 times 1 1384 5.20 MB/sec warmup 5 sec latency 15.948 ms Failing mds1 on oleg446-server Stopping /mnt/lustre-mds1 (opts:) on oleg446-server 1 1644 4.37 MB/sec warmup 6 sec latency 324.097 ms reboot facets: mds1 1 1644 3.74 MB/sec warmup 7 sec latency 1324.210 ms 1 1644 3.27 MB/sec warmup 8 sec latency 2324.379 ms 1 1644 2.91 MB/sec warmup 9 sec latency 3324.560 ms 1 1644 2.62 MB/sec warmup 10 sec latency 4324.711 ms 1 1644 2.38 MB/sec warmup 11 sec latency 5324.869 ms 1 1644 2.18 MB/sec warmup 12 sec latency 6325.005 ms 1 1644 2.02 MB/sec warmup 13 sec latency 7325.133 ms 1 1644 1.87 MB/sec warmup 14 sec latency 8325.278 ms 1 1644 1.75 MB/sec warmup 15 sec latency 9325.454 ms 1 1644 1.64 MB/sec warmup 16 sec latency 10325.575 ms Failover mds1 to oleg446-server mount facets: mds1 1 1644 1.54 MB/sec warmup 17 sec latency 11325.686 ms 1 1644 1.46 MB/sec warmup 18 sec latency 12325.788 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1644 1.38 MB/sec warmup 19 sec latency 13325.900 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 1 1644 0.00 MB/sec execute 1 sec latency 15326.069 ms Started lustre-MDT0000 1 1644 0.00 MB/sec execute 2 sec latency 16326.262 ms 1 1644 0.00 MB/sec execute 3 sec latency 17326.383 ms 1 1644 0.00 MB/sec execute 4 sec latency 18326.486 ms 1 1644 0.00 MB/sec execute 5 sec latency 19326.589 ms 1 1868 0.03 MB/sec execute 6 sec latency 19747.932 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2358 0.24 MB/sec execute 7 sec latency 11.341 ms 1 2894 0.83 MB/sec execute 8 sec latency 15.873 ms 1 3380 1.03 MB/sec execute 9 sec latency 19.932 ms 1 3809 1.36 MB/sec execute 10 sec latency 13.644 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 4600 3590496 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30264 3568872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 34864 7159368 1% /mnt/lustre 1 4062 1.26 MB/sec execute 11 sec latency 23.646 ms 1 4292 1.18 MB/sec execute 12 sec latency 227.263 ms 1 4549 1.14 MB/sec execute 13 sec latency 266.361 ms 1 4953 1.35 MB/sec execute 14 sec latency 18.259 ms test_26 fail mds2 2 times Failing mds2 on oleg446-server Stopping /mnt/lustre-mds2 (opts:) on oleg446-server 1 5308 1.28 MB/sec execute 15 sec latency 13.731 ms reboot facets: mds2 1 5350 1.20 MB/sec execute 16 sec latency 901.245 ms 1 5350 1.13 MB/sec execute 17 sec latency 1901.375 ms 1 5350 1.07 MB/sec execute 18 sec latency 2901.545 ms 1 5350 1.01 MB/sec execute 19 sec latency 3901.750 ms 1 5350 0.96 MB/sec execute 20 sec latency 4901.955 ms 1 5350 0.92 MB/sec execute 21 sec latency 5902.107 ms 1 5350 0.87 MB/sec execute 22 sec latency 6902.236 ms 1 5350 0.84 MB/sec execute 23 sec latency 7902.376 ms 1 5350 0.80 MB/sec execute 24 sec latency 8902.482 ms 1 5350 0.77 MB/sec execute 25 sec latency 9902.598 ms Failover mds2 to oleg446-server 1 5350 0.74 MB/sec execute 26 sec latency 10902.774 ms mount facets: mds2 1 5350 0.71 MB/sec execute 27 sec latency 11902.927 ms 1 5350 0.69 MB/sec execute 28 sec latency 12903.106 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 5350 0.66 MB/sec execute 29 sec latency 13903.269 ms 1 5350 0.64 MB/sec execute 30 sec latency 14903.406 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0001 1 5350 0.62 MB/sec execute 31 sec latency 15903.652 ms 1 5350 0.60 MB/sec execute 32 sec latency 16903.846 ms 1 5350 0.58 MB/sec execute 33 sec latency 17904.053 ms 1 5350 0.57 MB/sec execute 34 sec latency 18904.243 ms 1 5581 0.56 MB/sec execute 35 sec latency 19172.013 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5948 0.58 MB/sec execute 36 sec latency 13.167 ms 1 6435 0.68 MB/sec execute 37 sec latency 18.681 ms 1 6890 0.74 MB/sec execute 38 sec latency 19.684 ms 1 7243 0.81 MB/sec execute 39 sec latency 13.529 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4488 1283200 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2424 1285264 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 20328 3584592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 43020 3563320 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 63348 7147912 1% /mnt/lustre 1 7449 0.82 MB/sec execute 40 sec latency 29.690 ms 1 7650 0.81 MB/sec execute 41 sec latency 21.545 ms 1 7823 0.79 MB/sec execute 42 sec latency 505.570 ms 1 8081 0.79 MB/sec execute 43 sec latency 19.370 ms test_26 fail mds1 3 times 1 8365 0.80 MB/sec execute 44 sec latency 23.623 ms Failing mds1 on oleg446-server Stopping /mnt/lustre-mds1 (opts:) on oleg446-server 1 8740 0.85 MB/sec execute 45 sec latency 19.570 ms reboot facets: mds1 1 8986 0.83 MB/sec execute 46 sec latency 23.319 ms 1 9364 0.83 MB/sec execute 47 sec latency 27.124 ms 1 9364 0.81 MB/sec execute 48 sec latency 1027.272 ms 1 9364 0.79 MB/sec execute 49 sec latency 2027.440 ms 1 9364 0.78 MB/sec execute 50 sec latency 3027.621 ms 1 9364 0.76 MB/sec execute 51 sec latency 4027.818 ms 1 9364 0.75 MB/sec execute 52 sec latency 5027.979 ms 1 9364 0.73 MB/sec execute 53 sec latency 6028.122 ms 1 9364 0.72 MB/sec execute 54 sec latency 7028.295 ms 1 9364 0.71 MB/sec execute 55 sec latency 8028.437 ms Failover mds1 to oleg446-server mount facets: mds1 1 9364 0.69 MB/sec execute 56 sec latency 9028.706 ms 1 9364 0.68 MB/sec execute 57 sec latency 10028.906 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 9364 0.67 MB/sec execute 58 sec latency 11029.054 ms 1 9364 0.66 MB/sec execute 59 sec latency 12029.179 ms 1 9364 0.65 MB/sec execute 60 sec latency 13029.314 ms 1 9364 0.64 MB/sec execute 61 sec latency 14029.483 ms 1 9364 0.63 MB/sec execute 62 sec latency 15029.654 ms 1 9364 0.62 MB/sec execute 63 sec latency 16029.802 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 1 9364 0.61 MB/sec execute 64 sec latency 17029.931 ms 1 9364 0.60 MB/sec execute 65 sec latency 18030.060 ms 1 9364 0.59 MB/sec execute 66 sec latency 19030.241 ms 1 9364 0.58 MB/sec execute 67 sec latency 20030.394 ms 1 9428 0.59 MB/sec execute 68 sec latency 20893.712 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 9685 0.59 MB/sec execute 69 sec latency 17.298 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 10331 0.66 MB/sec execute 70 sec latency 17.864 ms 1 10677 0.71 MB/sec execute 71 sec latency 16.590 ms 1 10970 0.72 MB/sec execute 72 sec latency 26.722 ms 1 11269 0.71 MB/sec execute 73 sec latency 16.822 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5124 1282564 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3232 1284456 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 33008 3558800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44012 3558140 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 77020 7116940 2% /mnt/lustre 1 11589 0.72 MB/sec execute 74 sec latency 16.956 ms 1 12040 0.76 MB/sec execute 75 sec latency 13.433 ms 1 12364 0.76 MB/sec execute 76 sec latency 17.626 ms test_26 fail mds2 4 times 1 12720 0.75 MB/sec execute 77 sec latency 12.293 ms Failing mds2 on oleg446-server Stopping /mnt/lustre-mds2 (opts:) on oleg446-server 1 13133 0.77 MB/sec execute 78 sec latency 15.942 ms reboot facets: mds2 1 13830 0.83 MB/sec execute 79 sec latency 13.938 ms 1 14268 0.87 MB/sec execute 80 sec latency 16.238 ms 1 14466 0.88 MB/sec execute 81 sec latency 23.644 ms 1 14673 0.87 MB/sec execute 82 sec latency 23.223 ms 1 14722 0.86 MB/sec execute 83 sec latency 810.180 ms 1 14722 0.85 MB/sec execute 84 sec latency 1810.317 ms 1 14722 0.84 MB/sec execute 85 sec latency 2810.494 ms 1 14722 0.83 MB/sec execute 86 sec latency 3810.717 ms 1 14722 0.82 MB/sec execute 87 sec latency 4810.887 ms 1 14722 0.81 MB/sec execute 88 sec latency 5810.993 ms Failover mds2 to oleg446-server mount facets: mds2 1 14722 0.80 MB/sec execute 89 sec latency 6811.119 ms 1 14722 0.79 MB/sec execute 90 sec latency 7811.206 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 14722 0.78 MB/sec execute 91 sec latency 8811.360 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 1 14722 0.77 MB/sec execute 92 sec latency 9811.483 ms pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0001 1 14722 0.77 MB/sec execute 93 sec latency 10811.580 ms 1 14722 0.76 MB/sec execute 94 sec latency 11811.729 ms 1 14722 0.75 MB/sec execute 95 sec latency 12811.873 ms 1 14722 0.74 MB/sec execute 96 sec latency 13811.991 ms 1 14722 0.73 MB/sec execute 97 sec latency 14812.107 ms 1 15064 0.73 MB/sec execute 98 sec latency 14843.094 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 15379 0.74 MB/sec execute 99 sec latency 23.168 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2407 25.144 20893.700 Close 1778 1.638 5.070 Rename 104 12.501 23.604 Unlink 494 5.024 505.548 Qpathinfo 2203 2.231 15.827 Qfileinfo 379 0.390 2.793 Qfsinfo 408 0.774 4.235 Sfileinfo 200 5.916 13.116 Find 849 42.551 19747.924 WriteX 1192 2.236 9.991 ReadX 3825 0.033 2.399 LockX 8 1.646 3.525 UnlockX 8 1.302 1.944 Flush 169 8.975 266.343 Throughput 0.740288 MB/sec 1 clients 1 procs max_latency=20893.712 ms stopping dbench on /mnt/lustre at Sat May 11 17:36:46 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat May 11 17:36:47 EDT 2024 waiting for dbench pid 11325 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4980 1282708 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3408 1284280 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 28108 3577480 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 40760 3540928 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 68868 7118408 1% /mnt/lustre 1 239 9.14 MB/sec warmup 1 sec latency 23.748 ms 1 478 8.34 MB/sec warmup 2 sec latency 23.524 ms 1 791 7.15 MB/sec warmup 3 sec latency 35.857 ms test_26 fail mds1 5 times 1 1060 5.72 MB/sec warmup 4 sec latency 18.671 ms Failing mds1 on oleg446-server Stopping /mnt/lustre-mds1 (opts:) on oleg446-server 1 1419 5.20 MB/sec warmup 5 sec latency 29.649 ms reboot facets: mds1 1 1419 4.34 MB/sec warmup 6 sec latency 1029.843 ms 1 1419 3.72 MB/sec warmup 7 sec latency 2030.075 ms 1 1419 3.25 MB/sec warmup 8 sec latency 3030.349 ms 1 1419 2.89 MB/sec warmup 9 sec latency 4030.640 ms 1 1419 2.60 MB/sec warmup 10 sec latency 5030.868 ms 1 1419 2.36 MB/sec warmup 11 sec latency 6031.093 ms 1 1419 2.17 MB/sec warmup 12 sec latency 7031.413 ms 1 1419 2.00 MB/sec warmup 13 sec latency 8031.614 ms 1 1419 1.86 MB/sec warmup 14 sec latency 9031.780 ms 1 1419 1.73 MB/sec warmup 15 sec latency 10031.996 ms Failover mds1 to oleg446-server 1 1419 1.63 MB/sec warmup 16 sec latency 11032.191 ms mount facets: mds1 1 1419 1.53 MB/sec warmup 17 sec latency 12032.382 ms 1 1419 1.45 MB/sec warmup 18 sec latency 13032.518 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1419 1.37 MB/sec warmup 19 sec latency 14032.657 ms 1 1419 0.00 MB/sec execute 1 sec latency 16032.961 ms 1 1419 0.00 MB/sec execute 2 sec latency 17033.087 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 1 1419 0.00 MB/sec execute 3 sec latency 18033.305 ms pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1419 0.00 MB/sec execute 4 sec latency 19033.433 ms 1 1419 0.00 MB/sec execute 5 sec latency 20033.617 ms 1 1419 0.00 MB/sec execute 6 sec latency 21033.819 ms 1 1419 0.00 MB/sec execute 7 sec latency 22033.982 ms 1 1419 0.00 MB/sec execute 8 sec latency 23034.104 ms 1 1519 0.01 MB/sec execute 9 sec latency 23558.766 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1808 0.03 MB/sec execute 10 sec latency 19.827 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2160 0.06 MB/sec execute 11 sec latency 21.548 ms 1 2434 0.17 MB/sec execute 12 sec latency 19.300 ms 1 2889 0.50 MB/sec execute 13 sec latency 15.973 ms 1 3305 0.67 MB/sec execute 14 sec latency 24.569 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3968 1283720 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3440 1284248 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30816 3573060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 42032 3564332 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 72848 7137392 2% /mnt/lustre 1 3715 0.85 MB/sec execute 15 sec latency 31.387 ms 1 3952 0.88 MB/sec execute 16 sec latency 25.016 ms 1 4188 0.83 MB/sec execute 17 sec latency 17.915 ms 1 4430 0.82 MB/sec execute 18 sec latency 38.222 ms test_26 fail mds2 6 times Failing mds2 on oleg446-server Stopping /mnt/lustre-mds2 (opts:) on oleg446-server 1 4671 0.84 MB/sec execute 19 sec latency 24.308 ms 1 4851 0.81 MB/sec execute 20 sec latency 291.083 ms 1 4851 0.77 MB/sec execute 21 sec latency 1291.226 ms 1 4851 0.74 MB/sec execute 22 sec latency 2291.395 ms 1 4851 0.71 MB/sec execute 23 sec latency 3291.557 ms 1 4851 0.68 MB/sec execute 24 sec latency 4291.738 ms 1 4851 0.65 MB/sec execute 25 sec latency 5291.922 ms reboot facets: mds2 1 4851 0.62 MB/sec execute 26 sec latency 6292.139 ms 1 4851 0.60 MB/sec execute 27 sec latency 7292.388 ms 1 4851 0.58 MB/sec execute 28 sec latency 8292.603 ms 1 4851 0.56 MB/sec execute 29 sec latency 9292.756 ms 1 4851 0.54 MB/sec execute 30 sec latency 10292.950 ms 1 4851 0.52 MB/sec execute 31 sec latency 11293.157 ms 1 4851 0.51 MB/sec execute 32 sec latency 12293.402 ms 1 4851 0.49 MB/sec execute 33 sec latency 13293.620 ms 1 4851 0.48 MB/sec execute 34 sec latency 14293.794 ms 1 4851 0.46 MB/sec execute 35 sec latency 15293.995 ms Failover mds2 to oleg446-server mount facets: mds2 1 4851 0.45 MB/sec execute 36 sec latency 16294.135 ms 1 4851 0.44 MB/sec execute 37 sec latency 17294.284 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4851 0.43 MB/sec execute 38 sec latency 18294.460 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 1 4851 0.42 MB/sec execute 39 sec latency 19294.591 ms pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0001 1 4851 0.41 MB/sec execute 40 sec latency 20294.714 ms 1 4851 0.40 MB/sec execute 41 sec latency 21294.882 ms 1 4851 0.39 MB/sec execute 42 sec latency 22295.029 ms 1 4851 0.38 MB/sec execute 43 sec latency 23295.329 ms 1 4955 0.43 MB/sec execute 44 sec latency 24207.139 ms 1 5249 0.43 MB/sec execute 45 sec latency 27.200 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5554 0.43 MB/sec execute 46 sec latency 22.979 ms 1 5819 0.42 MB/sec execute 47 sec latency 32.485 ms 1 6056 0.45 MB/sec execute 48 sec latency 22.539 ms 1 6486 0.54 MB/sec execute 49 sec latency 30.048 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2684 1285004 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2492 1285196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13648 3592056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38284 3566268 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51932 7158324 1% /mnt/lustre 1 7066 0.63 MB/sec execute 50 sec latency 24.868 ms 1 7369 0.64 MB/sec execute 51 sec latency 22.020 ms 1 7562 0.64 MB/sec execute 52 sec latency 21.783 ms striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar 1 7812 0.63 MB/sec execute 53 sec latency 42.081 ms test_26 fail mds1 7 times Failing mds1 on oleg446-server 1 8033 0.63 MB/sec execute 54 sec latency 31.784 ms Stopping /mnt/lustre-mds1 (opts:) on oleg446-server 1 8139 0.64 MB/sec execute 55 sec latency 658.454 ms reboot facets: mds1 1 8139 0.63 MB/sec execute 56 sec latency 1658.670 ms 1 8139 0.62 MB/sec execute 57 sec latency 2658.880 ms 1 8139 0.61 MB/sec execute 58 sec latency 3659.145 ms 1 8139 0.59 MB/sec execute 59 sec latency 4659.337 ms 1 8139 0.59 MB/sec execute 60 sec latency 5659.505 ms 1 8139 0.58 MB/sec execute 61 sec latency 6659.625 ms 1 8139 0.57 MB/sec execute 62 sec latency 7659.824 ms 1 8139 0.56 MB/sec execute 63 sec latency 8659.971 ms 1 8139 0.55 MB/sec execute 64 sec latency 9660.091 ms 1 8139 0.54 MB/sec execute 65 sec latency 10660.276 ms Failover mds1 to oleg446-server mount facets: mds1 1 8139 0.53 MB/sec execute 66 sec latency 11660.375 ms 1 8139 0.52 MB/sec execute 67 sec latency 12660.531 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 8139 0.52 MB/sec execute 68 sec latency 13660.770 ms 1 8139 0.51 MB/sec execute 69 sec latency 14660.950 ms 1 8139 0.50 MB/sec execute 70 sec latency 15661.081 ms 1 8139 0.49 MB/sec execute 71 sec latency 16661.301 ms oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all 8 1 8139 0.49 MB/sec execute 72 sec latency 17661.520 ms pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 1 8139 0.48 MB/sec execute 73 sec latency 18661.662 ms 1 8139 0.47 MB/sec execute 74 sec latency 19661.849 ms 1 8139 0.47 MB/sec execute 75 sec latency 20662.010 ms 1 8139 0.46 MB/sec execute 76 sec latency 21662.158 ms 1 8227 0.46 MB/sec execute 77 sec latency 22445.953 ms oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 8621 0.49 MB/sec execute 78 sec latency 18.641 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8947 0.49 MB/sec execute 79 sec latency 15.744 ms 1 9378 0.49 MB/sec execute 80 sec latency 10.780 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 9822 0.55 MB/sec execute 81 sec latency 13.604 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sat May 11 17:38:29 EDT 2024 with return code 0 11325 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 11325 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished