== replay-dual test 26: dbench and tar with mds failover ========================================================== 23:42:17 (1717558937) Starting client oleg148-client.virtnet: -o user_xattr,flock oleg148-server@tcp:/lustre /mnt/lustre Started clients oleg148-client.virtnet: 192.168.201.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 2859 Started dbench loop with 2860 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 4 23:42:18 EDT 2024 waiting for dbench pid 2902 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 373 13.30 MB/sec warmup 1 sec latency 21.520 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1285336 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2140 1285548 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3594060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26016 3545068 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27560 7139128 1% /mnt/lustre 1 856 10.79 MB/sec warmup 2 sec latency 20.176 ms 1 1257 7.70 MB/sec warmup 3 sec latency 13.708 ms 1 1838 6.59 MB/sec warmup 4 sec latency 12.627 ms 1 2489 5.74 MB/sec warmup 5 sec latency 12.828 ms test_26 fail mds1 1 times 1 3792 6.63 MB/sec warmup 6 sec latency 13.606 ms Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 1 4102 5.73 MB/sec warmup 7 sec latency 15.066 ms 23:42:26 (1717558946) shut down 1 4130 5.01 MB/sec warmup 8 sec latency 950.472 ms 1 4130 4.45 MB/sec warmup 9 sec latency 1950.732 ms 1 4130 4.01 MB/sec warmup 10 sec latency 2950.981 ms 1 4130 3.64 MB/sec warmup 11 sec latency 3951.278 ms 1 4130 3.34 MB/sec warmup 12 sec latency 4951.483 ms 1 4130 3.08 MB/sec warmup 13 sec latency 5951.646 ms 1 4130 2.86 MB/sec warmup 14 sec latency 6951.879 ms 1 4130 2.67 MB/sec warmup 15 sec latency 7952.145 ms 1 4130 2.51 MB/sec warmup 16 sec latency 8952.355 ms 1 4130 2.36 MB/sec warmup 17 sec latency 9952.517 ms Failover mds1 to oleg148-server mount facets: mds1 1 4130 2.23 MB/sec warmup 18 sec latency 10952.643 ms 1 4130 2.11 MB/sec warmup 19 sec latency 11952.819 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4130 0.00 MB/sec execute 1 sec latency 13953.168 ms 1 4130 0.00 MB/sec execute 2 sec latency 14953.397 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 23:42:41 (1717558961) targets are mounted 23:42:41 (1717558961) facet_failover done 1 4130 0.00 MB/sec execute 3 sec latency 15953.617 ms 1 4130 0.00 MB/sec execute 4 sec latency 16953.771 ms 1 4130 0.00 MB/sec execute 5 sec latency 17954.067 ms 1 4130 0.00 MB/sec execute 6 sec latency 18954.299 ms 1 4130 0.00 MB/sec execute 7 sec latency 19954.465 ms 1 4130 0.00 MB/sec execute 8 sec latency 20954.584 ms 1 4130 0.00 MB/sec execute 9 sec latency 21954.693 ms 1 4130 0.00 MB/sec execute 10 sec latency 22954.820 ms oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4488 0.07 MB/sec execute 11 sec latency 23438.896 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5075 0.42 MB/sec execute 12 sec latency 20.327 ms 1 5408 0.41 MB/sec execute 13 sec latency 18.560 ms 1 5907 0.49 MB/sec execute 14 sec latency 20.383 ms 1 6536 0.84 MB/sec execute 15 sec latency 13.639 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3604 1284084 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2708 1284980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7472 3583232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33444 3560008 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 40916 7143240 1% /mnt/lustre 1 7412 1.17 MB/sec execute 16 sec latency 16.848 ms 1 7891 1.15 MB/sec execute 17 sec latency 110.008 ms 1 8264 1.18 MB/sec execute 18 sec latency 15.116 ms 1 8740 1.28 MB/sec execute 19 sec latency 12.775 ms test_26 fail mds2 2 times Failing mds2 on oleg148-server 1 9129 1.23 MB/sec execute 20 sec latency 15.003 ms Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 1 9881 1.43 MB/sec execute 21 sec latency 27.025 ms 23:42:59 (1717558979) shut down 1 10909 1.72 MB/sec execute 22 sec latency 8.997 ms 1 11174 1.66 MB/sec execute 23 sec latency 243.000 ms 1 11174 1.59 MB/sec execute 24 sec latency 1243.267 ms 1 11174 1.53 MB/sec execute 25 sec latency 2243.429 ms 1 11174 1.47 MB/sec execute 26 sec latency 3243.657 ms 1 11174 1.42 MB/sec execute 27 sec latency 4243.877 ms 1 11174 1.37 MB/sec execute 28 sec latency 5244.083 ms 1 11174 1.32 MB/sec execute 29 sec latency 6244.274 ms 1 11174 1.27 MB/sec execute 30 sec latency 7244.474 ms 1 11174 1.23 MB/sec execute 31 sec latency 8244.698 ms Failover mds2 to oleg148-server mount facets: mds2 1 11174 1.19 MB/sec execute 32 sec latency 9244.894 ms 1 11174 1.16 MB/sec execute 33 sec latency 10245.088 ms 1 11174 1.12 MB/sec execute 34 sec latency 11245.259 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 11174 1.09 MB/sec execute 35 sec latency 12245.451 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all 1 11174 1.06 MB/sec execute 36 sec latency 13245.629 ms pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 23:43:14 (1717558994) targets are mounted 23:43:14 (1717558994) facet_failover done 1 11174 1.03 MB/sec execute 37 sec latency 14245.861 ms 1 11174 1.01 MB/sec execute 38 sec latency 15246.147 ms 1 11174 0.98 MB/sec execute 39 sec latency 16246.363 ms 1 11174 0.96 MB/sec execute 40 sec latency 17246.516 ms oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11620 0.95 MB/sec execute 41 sec latency 17560.747 ms 1 12326 1.04 MB/sec execute 42 sec latency 11.929 ms 1 13042 1.05 MB/sec execute 43 sec latency 12.375 ms 1 13662 1.15 MB/sec execute 44 sec latency 14.795 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5272 1282416 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3164 1284524 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26644 3560768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 40580 3551920 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 67224 7112688 1% /mnt/lustre 1 14492 1.27 MB/sec execute 45 sec latency 12.440 ms 1 15069 1.26 MB/sec execute 46 sec latency 10.838 ms 1 15591 1.33 MB/sec execute 47 sec latency 15.213 ms 1 15928 1.31 MB/sec execute 48 sec latency 16.056 ms test_26 fail mds1 3 times 1 16358 1.29 MB/sec execute 49 sec latency 14.568 ms Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 1 17188 1.40 MB/sec execute 50 sec latency 193.808 ms 23:43:28 (1717559008) shut down 1 17188 1.37 MB/sec execute 51 sec latency 1193.999 ms 1 17188 1.34 MB/sec execute 52 sec latency 2194.270 ms 1 17188 1.32 MB/sec execute 53 sec latency 3194.484 ms 1 17188 1.29 MB/sec execute 54 sec latency 4194.703 ms 1 17188 1.27 MB/sec execute 55 sec latency 5194.898 ms 1 17188 1.25 MB/sec execute 56 sec latency 6195.129 ms 1 17188 1.23 MB/sec execute 57 sec latency 7195.389 ms 1 17188 1.21 MB/sec execute 58 sec latency 8195.656 ms 1 17188 1.18 MB/sec execute 59 sec latency 9195.907 ms 1 17188 1.17 MB/sec execute 60 sec latency 10196.128 ms Failover mds1 to oleg148-server mount facets: mds1 1 17188 1.15 MB/sec execute 61 sec latency 11196.385 ms 1 17188 1.13 MB/sec execute 62 sec latency 12196.588 ms 1 17188 1.11 MB/sec execute 63 sec latency 13196.796 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 17188 1.09 MB/sec execute 64 sec latency 14196.995 ms 1 17188 1.08 MB/sec execute 65 sec latency 15197.271 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 23:43:44 (1717559024) targets are mounted 23:43:44 (1717559024) facet_failover done 1 17188 1.06 MB/sec execute 66 sec latency 16197.494 ms 1 17188 1.04 MB/sec execute 67 sec latency 17197.691 ms 1 17188 1.03 MB/sec execute 68 sec latency 18197.971 ms 1 17188 1.01 MB/sec execute 69 sec latency 19198.137 ms 1 17188 1.00 MB/sec execute 70 sec latency 20198.357 ms oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 17826 1.06 MB/sec execute 71 sec latency 20324.262 ms 1 18127 1.06 MB/sec execute 72 sec latency 15.954 ms 1 18533 1.05 MB/sec execute 73 sec latency 14.727 ms 1 19150 1.10 MB/sec execute 74 sec latency 15.678 ms 1 19612 1.09 MB/sec execute 75 sec latency 10.047 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4664 1283024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3328 1284360 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 35296 3566640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 45116 3558716 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 80412 7125356 2% /mnt/lustre 1 20232 1.10 MB/sec execute 76 sec latency 16.869 ms 1 20795 1.16 MB/sec execute 77 sec latency 17.330 ms 1 21409 1.21 MB/sec execute 78 sec latency 14.021 ms 1 21721 1.21 MB/sec execute 79 sec latency 14.995 ms test_26 fail mds2 4 times Failing mds2 on oleg148-server 1 22208 1.20 MB/sec execute 80 sec latency 12.030 ms Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 1 22479 1.21 MB/sec execute 81 sec latency 560.321 ms 23:43:59 (1717559039) shut down 1 22479 1.19 MB/sec execute 82 sec latency 1560.539 ms 1 22479 1.18 MB/sec execute 83 sec latency 2560.756 ms 1 22479 1.16 MB/sec execute 84 sec latency 3561.045 ms 1 22479 1.15 MB/sec execute 85 sec latency 4561.316 ms 1 22479 1.14 MB/sec execute 86 sec latency 5561.513 ms 1 22479 1.12 MB/sec execute 87 sec latency 6561.703 ms 1 22479 1.11 MB/sec execute 88 sec latency 7561.919 ms 1 22479 1.10 MB/sec execute 89 sec latency 8562.146 ms 1 22479 1.09 MB/sec execute 90 sec latency 9562.356 ms 1 22479 1.07 MB/sec execute 91 sec latency 10562.661 ms Failover mds2 to oleg148-server mount facets: mds2 1 22479 1.06 MB/sec execute 92 sec latency 11562.860 ms 1 22479 1.05 MB/sec execute 93 sec latency 12563.077 ms 1 22479 1.04 MB/sec execute 94 sec latency 13563.271 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 22479 1.03 MB/sec execute 95 sec latency 14563.505 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all 1 22479 1.02 MB/sec execute 96 sec latency 15563.716 ms pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0001 23:44:15 (1717559055) targets are mounted 23:44:15 (1717559055) facet_failover done 1 22479 1.01 MB/sec execute 97 sec latency 16563.906 ms 1 22479 1.00 MB/sec execute 98 sec latency 17564.042 ms 1 22479 0.99 MB/sec execute 99 sec latency 18564.219 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 3167 11.919 20324.247 Close 2344 1.236 10.933 Rename 137 153.116 19953.065 Unlink 646 1.921 14.204 Qpathinfo 2855 7.659 17560.729 Qfileinfo 497 0.095 2.043 Qfsinfo 536 0.198 3.058 Sfileinfo 273 3.649 11.099 Find 1113 22.148 23438.885 WriteX 1585 0.741 7.343 ReadX 4938 0.011 1.160 LockX 10 1.332 1.922 UnlockX 10 1.325 2.060 Flush 239 6.035 109.990 Throughput 0.9865 MB/sec 1 clients 1 procs max_latency=23438.896 ms stopping dbench on /mnt/lustre at Tue Jun 4 23:44:18 EDT 2024 with return code 0 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 4 23:44:19 EDT 2024 waiting for dbench pid 7204 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 380 13.56 MB/sec warmup 1 sec latency 22.422 ms 1 955 10.90 MB/sec warmup 2 sec latency 21.889 ms 1 1410 8.67 MB/sec warmup 3 sec latency 15.000 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2836 1284852 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11720 3584040 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 27472 3560132 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39192 7144172 1% /mnt/lustre 1 1803 6.58 MB/sec warmup 4 sec latency 20.758 ms 1 2245 5.36 MB/sec warmup 5 sec latency 11.707 ms 1 2749 5.33 MB/sec warmup 6 sec latency 18.969 ms striped dir -i0 -c2 -H crush2 /mnt/lustre/d26.replay-dual/run_tar 1 3633 5.52 MB/sec warmup 7 sec latency 12.065 ms 1 3991 5.01 MB/sec warmup 8 sec latency 16.002 ms test_26 fail mds1 5 times Failing mds1 on oleg148-server 1 4395 4.51 MB/sec warmup 9 sec latency 13.806 ms Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 1 4679 4.21 MB/sec warmup 10 sec latency 589.743 ms 1 4679 3.83 MB/sec warmup 11 sec latency 1589.940 ms 1 4679 3.51 MB/sec warmup 12 sec latency 2590.239 ms 1 4679 3.24 MB/sec warmup 13 sec latency 3590.474 ms 1 4679 3.01 MB/sec warmup 14 sec latency 4590.742 ms 1 4679 2.81 MB/sec warmup 15 sec latency 5590.922 ms 23:44:35 (1717559075) shut down 1 4679 2.63 MB/sec warmup 16 sec latency 6591.085 ms 1 4679 2.48 MB/sec warmup 17 sec latency 7591.223 ms 1 4679 2.34 MB/sec warmup 18 sec latency 8591.446 ms 1 4679 2.21 MB/sec warmup 19 sec latency 9591.748 ms 1 4679 0.00 MB/sec execute 1 sec latency 11592.065 ms 1 4679 0.00 MB/sec execute 2 sec latency 12592.323 ms 1 4679 0.00 MB/sec execute 3 sec latency 13592.499 ms 1 4679 0.00 MB/sec execute 4 sec latency 14592.726 ms 1 4679 0.00 MB/sec execute 5 sec latency 15592.901 ms Failover mds1 to oleg148-server mount facets: mds1 1 4679 0.00 MB/sec execute 6 sec latency 16593.090 ms 1 4679 0.00 MB/sec execute 7 sec latency 17593.272 ms 1 4679 0.00 MB/sec execute 8 sec latency 18593.530 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4679 0.00 MB/sec execute 9 sec latency 19593.760 ms 1 4679 0.00 MB/sec execute 10 sec latency 20593.964 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 1 4679 0.00 MB/sec execute 11 sec latency 21594.127 ms Started lustre-MDT0000 23:44:50 (1717559090) targets are mounted 23:44:50 (1717559090) facet_failover done 1 4679 0.00 MB/sec execute 12 sec latency 22594.330 ms 1 4679 0.00 MB/sec execute 13 sec latency 23594.607 ms 1 4679 0.00 MB/sec execute 14 sec latency 24594.998 ms 1 4679 0.00 MB/sec execute 15 sec latency 25595.232 ms 1 4679 0.00 MB/sec execute 16 sec latency 26595.465 ms oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4967 0.18 MB/sec execute 17 sec latency 27152.366 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5358 0.19 MB/sec execute 18 sec latency 15.776 ms 1 5764 0.20 MB/sec execute 19 sec latency 10.980 ms 1 6172 0.31 MB/sec execute 20 sec latency 16.740 ms 1 6933 0.59 MB/sec execute 21 sec latency 15.489 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3196 1284492 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2764 1284924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12128 3591784 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37860 3565224 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49988 7157008 1% /mnt/lustre 1 7388 0.76 MB/sec execute 22 sec latency 14.119 ms 1 7721 0.74 MB/sec execute 23 sec latency 15.979 ms 1 8050 0.75 MB/sec execute 24 sec latency 20.299 ms 1 8530 0.89 MB/sec execute 25 sec latency 15.259 ms 1 8870 0.86 MB/sec execute 26 sec latency 21.531 ms test_26 fail mds2 6 times Failing mds2 on oleg148-server 1 9305 0.85 MB/sec execute 27 sec latency 27.016 ms Stopping /mnt/lustre-mds2 (opts:) on oleg148-server 23:45:07 (1717559107) shut down 1 9551 0.87 MB/sec execute 28 sec latency 710.072 ms 1 9551 0.84 MB/sec execute 29 sec latency 1710.261 ms 1 9551 0.81 MB/sec execute 30 sec latency 2710.451 ms 1 9551 0.78 MB/sec execute 31 sec latency 3710.643 ms 1 9551 0.76 MB/sec execute 32 sec latency 4710.825 ms 1 9551 0.74 MB/sec execute 33 sec latency 5711.070 ms 1 9551 0.71 MB/sec execute 34 sec latency 6711.203 ms 1 9551 0.69 MB/sec execute 35 sec latency 7711.695 ms 1 9551 0.67 MB/sec execute 36 sec latency 8711.878 ms 1 9551 0.66 MB/sec execute 37 sec latency 9712.060 ms 1 9551 0.64 MB/sec execute 38 sec latency 10712.287 ms Failover mds2 to oleg148-server mount facets: mds2 1 9551 0.62 MB/sec execute 39 sec latency 11712.466 ms 1 9551 0.61 MB/sec execute 40 sec latency 12712.630 ms 1 9551 0.59 MB/sec execute 41 sec latency 13712.809 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 9551 0.58 MB/sec execute 42 sec latency 14712.956 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 1 9551 0.56 MB/sec execute 43 sec latency 15713.105 ms Started lustre-MDT0001 23:45:22 (1717559122) targets are mounted 23:45:22 (1717559122) facet_failover done 1 9551 0.55 MB/sec execute 44 sec latency 16713.294 ms 1 9551 0.54 MB/sec execute 45 sec latency 17713.576 ms 1 9551 0.53 MB/sec execute 46 sec latency 18713.844 ms 1 9551 0.52 MB/sec execute 47 sec latency 19714.014 ms oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 9757 0.55 MB/sec execute 48 sec latency 20262.469 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 10665 0.71 MB/sec execute 49 sec latency 16.138 ms 1 11260 0.72 MB/sec execute 50 sec latency 10.475 ms 1 11607 0.73 MB/sec execute 51 sec latency 14.493 ms 1 12276 0.80 MB/sec execute 52 sec latency 12.066 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5408 1282280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3244 1284444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 20512 3564444 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38112 3547024 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 58624 7111468 1% /mnt/lustre 1 12817 0.79 MB/sec execute 53 sec latency 15.961 ms 1 13572 0.90 MB/sec execute 54 sec latency 15.182 ms 1 14447 1.00 MB/sec execute 55 sec latency 12.139 ms 1 14715 0.99 MB/sec execute 56 sec latency 15.652 ms test_26 fail mds1 7 times 1 15087 0.98 MB/sec execute 57 sec latency 22.340 ms Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 1 15380 0.99 MB/sec execute 58 sec latency 248.158 ms 1 15380 0.97 MB/sec execute 59 sec latency 1248.387 ms 1 15380 0.96 MB/sec execute 60 sec latency 2248.644 ms 1 15380 0.94 MB/sec execute 61 sec latency 3248.922 ms 1 15380 0.93 MB/sec execute 62 sec latency 4249.118 ms 1 15380 0.91 MB/sec execute 63 sec latency 5249.231 ms 23:45:43 (1717559143) shut down 1 15380 0.90 MB/sec execute 64 sec latency 6249.430 ms 1 15380 0.88 MB/sec execute 65 sec latency 7249.581 ms 1 15380 0.87 MB/sec execute 66 sec latency 8249.811 ms 1 15380 0.86 MB/sec execute 67 sec latency 9250.022 ms 1 15380 0.84 MB/sec execute 68 sec latency 10250.182 ms 1 15380 0.83 MB/sec execute 69 sec latency 11250.392 ms 1 15380 0.82 MB/sec execute 70 sec latency 12250.687 ms 1 15380 0.81 MB/sec execute 71 sec latency 13250.928 ms 1 15380 0.80 MB/sec execute 72 sec latency 14251.182 ms 1 15380 0.79 MB/sec execute 73 sec latency 15251.370 ms Failover mds1 to oleg148-server mount facets: mds1 1 15380 0.78 MB/sec execute 74 sec latency 16251.560 ms 1 15380 0.77 MB/sec execute 75 sec latency 17251.685 ms 1 15380 0.76 MB/sec execute 76 sec latency 18251.891 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 15380 0.75 MB/sec execute 77 sec latency 19252.121 ms 1 15380 0.74 MB/sec execute 78 sec latency 20252.351 ms oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 1 15380 0.73 MB/sec execute 79 sec latency 21252.580 ms Started lustre-MDT0000 23:45:58 (1717559158) targets are mounted 23:45:58 (1717559158) facet_failover done 1 15380 0.72 MB/sec execute 80 sec latency 22252.760 ms 1 15380 0.71 MB/sec execute 81 sec latency 23253.045 ms 1 15380 0.70 MB/sec execute 82 sec latency 24253.287 ms 1 15380 0.69 MB/sec execute 83 sec latency 25253.516 ms oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 15586 0.72 MB/sec execute 84 sec latency 26032.811 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 16150 0.72 MB/sec execute 85 sec latency 13.255 ms 1 16584 0.72 MB/sec execute 86 sec latency 15.275 ms 1 17121 0.78 MB/sec execute 87 sec latency 16.189 ms 1 17768 0.83 MB/sec execute 88 sec latency 15.425 ms 1 18107 0.84 MB/sec execute 89 sec latency 15.460 ms 1 18495 0.83 MB/sec execute 90 sec latency 26.350 ms 1 19001 0.84 MB/sec execute 91 sec latency 15.390 ms 1 19412 0.87 MB/sec execute 92 sec latency 29.703 ms 1 19775 0.86 MB/sec execute 93 sec latency 19.650 ms 1 20191 0.87 MB/sec execute 94 sec latency 16.284 ms 1 20970 0.92 MB/sec execute 95 sec latency 15.404 ms 1 21589 0.97 MB/sec execute 96 sec latency 15.600 ms 1 21845 0.97 MB/sec execute 97 sec latency 15.338 ms 1 22209 0.96 MB/sec execute 98 sec latency 28.355 ms 1 22564 0.97 MB/sec execute 99 sec latency 15.537 ms 1 cleanup 100 sec 0 cleanup 100 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 3177 6.384 29.685 Close 2332 13.099 27152.332 Rename 135 9.162 16.717 Unlink 629 2.318 8.522 Qpathinfo 2874 17.804 26032.798 Qfileinfo 503 0.112 2.252 Qfsinfo 518 0.249 3.285 Sfileinfo 264 5.086 17.277 Find 1104 0.804 11.034 WriteX 1555 0.921 12.432 ReadX 4966 0.015 7.420 LockX 10 1.434 1.869 UnlockX 10 1.469 2.191 Flush 218 8.303 28.338 Throughput 0.96713 MB/sec 1 clients 1 procs max_latency=27152.366 ms stopping dbench on /mnt/lustre at Tue Jun 4 23:46:19 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 4 23:46:20 EDT 2024 waiting for dbench pid 10893 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 339 12.26 MB/sec warmup 1 sec latency 26.236 ms 1 733 10.66 MB/sec warmup 2 sec latency 30.714 ms 1 1063 7.63 MB/sec warmup 3 sec latency 15.956 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jun 4 23:46:24 EDT 2024 with return code 0 10893 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 10893 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished