== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:15 (1709786715) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1709786744.02 total 10.01 last 24.28) - open/close 475 (time 1709786754.77 total 20.75 last 21.59) - open/close 751 (time 1709786764.78 total 30.77 last 27.57) - open/close 912 (time 1709786774.78 total 40.77 last 16.10) - open/close 1096 (time 1709786784.90 total 50.89 last 18.18) - open/close 1379 (time 1709786795.07 total 61.06 last 27.83) - open/close 1580 (time 1709786805.40 total 71.39 last 19.46) - open/close 1762 (time 1709786815.59 total 81.58 last 17.86) - open/close 1958 (time 1709786825.67 total 91.65 last 19.46) - open/close 2119 (time 1709786836.77 total 102.76 last 14.50) - open/close 2295 (time 1709786847.25 total 113.23 last 16.80) - open/close 2551 (time 1709786857.47 total 123.46 last 25.04) - open/close 2768 (time 1709786867.48 total 133.46 last 21.68) - open/close 2972 (time 1709786877.49 total 143.48 last 20.37) - open/close 3210 (time 1709786888.48 total 154.46 last 21.67) - open/close 3278 (time 1709786904.79 total 170.77 last 4.17) - open/close 3460 (time 1709786914.80 total 180.78 last 18.19) - open/close 3695 (time 1709786925.29 total 191.27 last 22.39) - open/close 3895 (time 1709786935.30 total 201.29 last 19.97) - open/close 4083 (time 1709786945.85 total 211.84 last 17.82) - open/close 4364 (time 1709786955.85 total 221.84 last 28.10) - open/close 4619 (time 1709786965.86 total 231.84 last 25.49) - open/close 4762 (time 1709786976.21 total 242.20 last 13.81) - open/close 5028 (time 1709786986.63 total 252.61 last 25.54) - open/close 5248 (time 1709786996.84 total 262.82 last 21.56) - open/close 5399 (time 1709787006.86 total 272.85 last 15.06) - open/close 5592 (time 1709787017.86 total 283.84 last 17.56) - open/close 5936 (time 1709787028.52 total 294.51 last 32.26) - open/close 6203 (time 1709787038.52 total 304.51 last 26.70) - open/close 6455 (time 1709787048.52 total 314.51 last 25.19) - open/close 6709 (time 1709787058.55 total 324.53 last 25.34) - open/close 6907 (time 1709787069.38 total 335.36 last 18.28) - open/close 7107 (time 1709787080.36 total 346.34 last 18.22) - open/close 7300 (time 1709787090.69 total 356.68 last 18.67) - open/close 7492 (time 1709787100.69 total 366.68 last 19.20) - open/close 7679 (time 1709787111.10 total 377.08 last 17.97) - open/close 7927 (time 1709787121.74 total 387.72 last 23.30) - open/close 8050 (time 1709787132.00 total 397.99 last 11.99) - open/close 8229 (time 1709787142.72 total 408.70 last 16.70) - open/close 8391 (time 1709787153.30 total 419.28 last 15.31) - open/close 8563 (time 1709787163.90 total 429.88 last 16.23) - open/close 8736 (time 1709787173.90 total 439.88 last 17.29) - open/close 8890 (time 1709787184.15 total 450.13 last 15.03) - open/close 9121 (time 1709787194.30 total 460.29 last 22.74) - open/close 9288 (time 1709787204.32 total 470.30 last 16.68) - open/close 9440 (time 1709787215.32 total 481.30 last 13.82) - open/close 9643 (time 1709787225.32 total 491.30 last 20.30) - open/close 9816 (time 1709787235.53 total 501.51 last 16.94) - open/close 9949 (time 1709787245.93 total 511.92 last 12.78) - open/close 10000 (time 1709787249.24 total 515.23 last 15.41) - open/close 10220 (time 1709787265.04 total 531.02 last 13.93) - open/close 10778 (time 1709787275.04 total 541.03 last 55.77) - open/close 11232 (time 1709787294.76 total 560.74 last 23.03) - open/close 11764 (time 1709787304.78 total 570.77 last 53.07) - open/close 12195 (time 1709787314.80 total 580.78 last 43.03) - open/close 12652 (time 1709787324.84 total 590.82 last 45.53) - open/close 13124 (time 1709787334.86 total 600.85 last 47.08) - open/close 13574 (time 1709787344.87 total 610.86 last 44.95) - open/close 14042 (time 1709787354.88 total 620.86 last 46.78) - open/close 14496 (time 1709787364.88 total 630.87 last 45.37) - open/close 14944 (time 1709787374.90 total 640.88 last 44.73) - open/close 15381 (time 1709787384.90 total 650.89 last 43.69) - open/close 15782 (time 1709787394.92 total 660.90 last 40.04) - open/close 16166 (time 1709787404.94 total 670.92 last 38.33) - open/close 16548 (time 1709787414.94 total 680.92 last 38.19) - open/close 16961 (time 1709787424.94 total 690.93 last 41.29) - open/close 17369 (time 1709787434.96 total 700.94 last 40.73) - open/close 17765 (time 1709787444.97 total 710.95 last 39.56) - open/close 18123 (time 1709787455.00 total 720.98 last 35.70) - open/close 18500 (time 1709787465.02 total 731.00 last 37.62) - open/close 18836 (time 1709787475.03 total 741.01 last 33.57) - open/close 19165 (time 1709787485.04 total 751.03 last 32.84) - open/close 19573 (time 1709787495.05 total 761.04 last 40.76) - open/close 19813 (time 1709787512.71 total 778.69 last 13.59) - open/close 20000 (time 1709787518.22 total 784.20 last 33.96) - open/close 20382 (time 1709787528.26 total 794.24 last 38.05) - open/close 20730 (time 1709787538.27 total 804.26 last 34.74) - open/close 21049 (time 1709787548.31 total 814.29 last 31.78) - open/close 21371 (time 1709787558.31 total 824.30 last 32.19) - open/close 21674 (time 1709787568.32 total 834.30 last 30.28) - open/close 22013 (time 1709787578.33 total 844.31 last 33.87) - open/close 22331 (time 1709787588.35 total 854.34 last 31.72) - open/close 22675 (time 1709787598.36 total 864.35 last 34.37) - open/close 22843 (time 1709787614.69 total 880.67 last 10.29) - open/close 22848 (time 1709787625.90 total 891.88 last 0.45) - open/close 23171 (time 1709787635.93 total 901.92 last 32.19) - open/close 23506 (time 1709787645.98 total 911.97 last 33.33) - open/close 23814 (time 1709787656.01 total 921.99 last 30.73) - open/close 24101 (time 1709787666.02 total 932.00 last 28.67) - open/close 24385 (time 1709787676.04 total 942.03 last 28.33) - open/close 24691 (time 1709787686.08 total 952.07 last 30.48) - open/close 25002 (time 1709787696.09 total 962.08 last 31.07) - open/close 25042 (time 1709787708.22 total 974.20 last 3.30) - open/close 25361 (time 1709787718.23 total 984.22 last 31.84) - open/close 25707 (time 1709787728.28 total 994.26 last 34.44) - open/close 26027 (time 1709787738.29 total 1004.27 last 31.98) - open/close 26324 (time 1709787748.32 total 1014.30 last 29.60) - open/close 26641 (time 1709787758.33 total 1024.32 last 31.66) - open/close 26941 (time 1709787768.34 total 1034.33 last 29.97) - open/close 27245 (time 1709787778.35 total 1044.34 last 30.38) - open/close 27529 (time 1709787788.36 total 1054.34 last 28.39) - open/close 27812 (time 1709787798.36 total 1064.34 last 28.29) - open/close 28111 (time 1709787808.37 total 1074.36 last 29.85) - open/close 28418 (time 1709787818.40 total 1084.38 last 30.62) - open/close 28704 (time 1709787828.41 total 1094.40 last 28.56) - open/close 28971 (time 1709787838.47 total 1104.45 last 26.56) - open/close 29232 (time 1709787848.56 total 1114.54 last 25.87) - open/close 29492 (time 1709787858.59 total 1124.57 last 25.91) - open/close 29775 (time 1709787868.62 total 1134.60 last 28.21) - open/close 30000 (time 1709787877.64 total 1143.63 last 24.93) - open/close 30252 (time 1709787887.65 total 1153.64 last 25.18) - open/close 30519 (time 1709787897.65 total 1163.64 last 26.70) - open/close 30783 (time 1709787907.69 total 1173.68 last 26.29) - open/close 31056 (time 1709787917.74 total 1183.72 last 27.19) - open/close 31288 (time 1709787927.79 total 1193.77 last 23.08) - open/close 31549 (time 1709787937.80 total 1203.79 last 26.06) - open/close 31737 (time 1709787955.73 total 1221.72 last 10.49) - open/close 32017 (time 1709787965.76 total 1231.74 last 27.94) - open/close 32291 (time 1709787975.77 total 1241.75 last 27.36) - open/close 32586 (time 1709787985.78 total 1251.76 last 29.47) - open/close 32870 (time 1709787995.80 total 1261.78 last 28.34) - open/close 33132 (time 1709788005.84 total 1271.82 last 26.11) - open/close 33411 (time 1709788015.86 total 1281.85 last 27.83) - open/close 33667 (time 1709788025.87 total 1291.86 last 25.57) - open/close 33934 (time 1709788035.90 total 1301.89 last 26.62) - open/close 34203 (time 1709788045.93 total 1311.91 last 26.84) - open/close 34472 (time 1709788055.95 total 1321.93 last 26.84) - open/close 34730 (time 1709788065.97 total 1331.95 last 25.74) - open/close 34994 (time 1709788075.98 total 1341.97 last 26.37) - open/close 35278 (time 1709788086.01 total 1351.99 last 28.33) - open/close 35549 (time 1709788096.04 total 1362.02 last 27.02) - open/close 35801 (time 1709788106.07 total 1372.05 last 25.12) - open/close 36047 (time 1709788116.10 total 1382.09 last 24.51) - open/close 36300 (time 1709788126.15 total 1392.14 last 25.18) - open/close 36559 (time 1709788136.16 total 1402.14 last 25.89) - open/close 36801 (time 1709788146.18 total 1412.17 last 24.14) - open/close 37039 (time 1709788156.21 total 1422.20 last 23.73) - open/close 37282 (time 1709788166.28 total 1432.27 last 24.13) - open/close 37535 (time 1709788176.32 total 1442.31 last 25.21) - open/close 37787 (time 1709788186.32 total 1452.31 last 25.19) - open/close 38040 (time 1709788196.33 total 1462.32 last 25.28) - open/close 38328 (time 1709788206.35 total 1472.34 last 28.74) - open/close 38607 (time 1709788216.38 total 1482.36 last 27.83) - open/close 38861 (time 1709788226.38 total 1492.37 last 25.39) - open/close 39130 (time 1709788236.41 total 1502.40 last 26.82) - open/close 39369 (time 1709788246.44 total 1512.43 last 23.83) - open/close 39595 (time 1709788256.48 total 1522.46 last 22.53) - open/close 39830 (time 1709788266.49 total 1532.48 last 23.45) - open/close 40000 (time 1709788273.77 total 1539.76 last 23.36) - open/close 40245 (time 1709788283.80 total 1549.78 last 24.44) - open/close 40503 (time 1709788293.80 total 1559.79 last 25.79) - open/close 40756 (time 1709788303.83 total 1569.82 last 25.22) - open/close 41005 (time 1709788313.86 total 1579.85 last 24.83) - open/close 41252 (time 1709788323.89 total 1589.88 last 24.63) - open/close 41484 (time 1709788333.94 total 1599.92 last 23.10) - open/close 41708 (time 1709788343.96 total 1609.95 last 22.35) - open/close 41947 (time 1709788353.99 total 1619.98 last 23.83) - open/close 42197 (time 1709788363.99 total 1629.98 last 25.00) - open/close 42437 (time 1709788374.01 total 1639.99 last 23.97) - open/close 42678 (time 1709788384.03 total 1650.01 last 24.06) - open/close 42916 (time 1709788394.04 total 1660.02 last 23.77) - open/close 43148 (time 1709788404.06 total 1670.04 last 23.15) - open/close 43388 (time 1709788414.08 total 1680.07 last 23.95) - open/close 43602 (time 1709788424.11 total 1690.09 last 21.35) - open/close 43821 (time 1709788434.15 total 1700.13 last 21.81) - open/close 44038 (time 1709788444.17 total 1710.15 last 21.65) - open/close 44247 (time 1709788454.19 total 1720.17 last 20.86) - open/close 44453 (time 1709788464.20 total 1730.18 last 20.58) - open/close 44673 (time 1709788474.21 total 1740.20 last 21.97) - open/close 44896 (time 1709788484.24 total 1750.22 last 22.25) - open/close 45131 (time 1709788494.28 total 1760.27 last 23.39) - open/close 45363 (time 1709788504.31 total 1770.30 last 23.13) - open/close 45588 (time 1709788514.32 total 1780.31 last 22.48) - open/close 45818 (time 1709788524.34 total 1790.32 last 22.96) - open/close 46047 (time 1709788534.35 total 1800.33 last 22.88) - open/close 46287 (time 1709788544.39 total 1810.38 last 23.89) - open/close 46518 (time 1709788554.43 total 1820.41 last 23.02) - open/close 46753 (time 1709788564.44 total 1830.43 last 23.47) - open/close 46982 (time 1709788574.48 total 1840.47 last 22.80) - open/close 47195 (time 1709788584.52 total 1850.51 last 21.21) - open/close 47419 (time 1709788594.55 total 1860.53 last 22.35) - open/close 47640 (time 1709788604.58 total 1870.56 last 22.04) - open/close 47867 (time 1709788614.59 total 1880.58 last 22.66) - open/close 48099 (time 1709788624.63 total 1890.61 last 23.12) - open/close 48325 (time 1709788634.66 total 1900.65 last 22.53) - open/close 48539 (time 1709788644.70 total 1910.69 last 21.31) - open/close 48753 (time 1709788654.72 total 1920.70 last 21.37) - open/close 48966 (time 1709788664.73 total 1930.72 last 21.27) - open/close 49177 (time 1709788674.75 total 1940.73 last 21.06) - open/close 49373 (time 1709788684.76 total 1950.74 last 19.59) - open/close 49574 (time 1709788694.78 total 1960.77 last 20.04) - open/close 49780 (time 1709788704.78 total 1970.77 last 20.60) - open/close 49996 (time 1709788714.82 total 1980.81 last 21.52) total: 50000 open/close in 1980.95 seconds: 25.24 ops/second