== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:40 (1709786920) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg116-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1709786968.98 total 10.72 last 12.59) - open/close 342 (time 1709786979.27 total 21.01 last 20.11) - open/close 584 (time 1709786989.31 total 31.05 last 24.11) - open/close 804 (time 1709786999.55 total 41.30 last 21.48) - open/close 964 (time 1709787009.63 total 51.38 last 15.87) - open/close 1138 (time 1709787020.32 total 62.06 last 16.28) - open/close 1399 (time 1709787030.61 total 72.36 last 25.35) - open/close 1633 (time 1709787040.67 total 82.42 last 23.27) - open/close 1922 (time 1709787050.68 total 92.42 last 28.87) - open/close 2104 (time 1709787061.08 total 102.82 last 17.50) - open/close 2196 (time 1709787078.49 total 120.24 last 5.28) - open/close 2467 (time 1709787089.32 total 131.06 last 25.02) - open/close 2645 (time 1709787099.66 total 141.40 last 17.22) - open/close 2848 (time 1709787109.67 total 151.41 last 20.29) - open/close 3034 (time 1709787120.65 total 162.39 last 16.94) - open/close 3271 (time 1709787131.06 total 172.80 last 22.77) - open/close 3411 (time 1709787141.08 total 182.83 last 13.96) - open/close 3624 (time 1709787151.09 total 192.84 last 21.28) - open/close 3734 (time 1709787161.11 total 202.85 last 10.98) - open/close 3857 (time 1709787171.35 total 213.09 last 12.01) - open/close 3969 (time 1709787182.45 total 224.20 last 10.09) - open/close 4157 (time 1709787192.87 total 234.61 last 18.05) - open/close 4315 (time 1709787203.40 total 245.15 last 15.00) - open/close 4448 (time 1709787213.41 total 255.16 last 13.29) - open/close 4601 (time 1709787223.62 total 265.37 last 14.99) - open/close 4771 (time 1709787234.14 total 275.88 last 16.17) - open/close 4936 (time 1709787244.17 total 285.91 last 16.44) - open/close 5155 (time 1709787254.33 total 296.08 last 21.55) - open/close 5265 (time 1709787264.34 total 306.08 last 10.99) - open/close 5351 (time 1709787275.05 total 316.79 last 8.03) - open/close 5434 (time 1709787288.71 total 330.46 last 6.07) - open/close 5630 (time 1709787301.21 total 342.96 last 15.68) - open/close 5896 (time 1709787311.73 total 353.47 last 25.31) - open/close 6059 (time 1709787322.04 total 363.78 last 15.81) - open/close 6193 (time 1709787332.04 total 373.79 last 13.39) - open/close 6342 (time 1709787388.67 total 430.42 last 2.63) - open/close 6500 (time 1709787398.68 total 440.43 last 15.78) - open/close 6644 (time 1709787409.37 total 451.12 last 13.47) - open/close 6830 (time 1709787419.39 total 461.13 last 18.57) - open/close 6990 (time 1709787429.83 total 471.58 last 15.32) - open/close 7189 (time 1709787440.26 total 482.00 last 19.09) - open/close 7411 (time 1709787450.58 total 492.32 last 21.51) - open/close 7603 (time 1709787460.76 total 502.50 last 18.86) - open/close 7799 (time 1709787470.77 total 512.51 last 19.58) - open/close 7978 (time 1709787480.79 total 522.53 last 17.87) - open/close 8104 (time 1709787491.17 total 532.92 last 12.13) - open/close 8232 (time 1709787501.56 total 543.31 last 12.32) - open/close 8453 (time 1709787511.94 total 553.68 last 21.31) - open/close 8729 (time 1709787522.59 total 564.34 last 25.90) - open/close 8845 (time 1709787532.61 total 574.35 last 11.58) - open/close 9154 (time 1709787542.62 total 584.36 last 30.87) - open/close 9558 (time 1709787552.62 total 594.37 last 40.38) - open/close 9871 (time 1709787570.80 total 612.54 last 17.22) - open/close 10000 (time 1709787573.71 total 615.45 last 44.30) - open/close 10438 (time 1709787583.73 total 625.47 last 43.71) - open/close 10898 (time 1709787593.78 total 635.52 last 45.78) - open/close 11305 (time 1709787603.78 total 645.52 last 40.69) - open/close 11738 (time 1709787613.81 total 655.55 last 43.18) - open/close 12174 (time 1709787623.81 total 665.56 last 43.57) - open/close 12584 (time 1709787633.82 total 675.56 last 40.99) - open/close 13045 (time 1709787643.83 total 685.58 last 46.02) - open/close 13466 (time 1709787653.89 total 695.63 last 41.87) - open/close 13837 (time 1709787663.89 total 705.63 last 37.10) - open/close 14235 (time 1709787673.89 total 715.64 last 39.79) - open/close 14604 (time 1709787683.91 total 725.65 last 36.84) - open/close 14978 (time 1709787693.92 total 735.67 last 37.34) - open/close 15359 (time 1709787703.94 total 745.69 last 38.03) - open/close 15757 (time 1709787713.97 total 755.71 last 39.70) - open/close 16156 (time 1709787723.98 total 765.73 last 39.85) - open/close 16280 (time 1709787738.41 total 780.16 last 8.59) - open/close 16626 (time 1709787748.43 total 790.17 last 34.54) - open/close 16971 (time 1709787758.44 total 800.18 last 34.46) - open/close 17302 (time 1709787768.44 total 810.18 last 33.10) - open/close 17657 (time 1709787778.45 total 820.19 last 35.47) - open/close 17977 (time 1709787788.46 total 830.20 last 31.97) - open/close 18300 (time 1709787798.47 total 840.22 last 32.25) - open/close 18641 (time 1709787808.49 total 850.23 last 34.06) - open/close 18998 (time 1709787818.51 total 860.25 last 35.62) - open/close 19323 (time 1709787828.53 total 870.27 last 32.44) - open/close 19624 (time 1709787838.54 total 880.28 last 30.06) - open/close 19901 (time 1709787848.54 total 890.29 last 27.69) - open/close 20000 (time 1709787852.22 total 893.97 last 26.91) - open/close 20303 (time 1709787862.22 total 903.97 last 30.29) - open/close 20588 (time 1709787872.23 total 913.98 last 28.47) - open/close 20893 (time 1709787882.25 total 924.00 last 30.45) - open/close 21192 (time 1709787892.25 total 934.00 last 29.89) - open/close 21455 (time 1709787902.28 total 944.02 last 26.24) - open/close 21726 (time 1709787912.31 total 954.06 last 27.00) - open/close 21991 (time 1709787922.34 total 964.09 last 26.41) - open/close 22269 (time 1709787932.39 total 974.14 last 27.67) - open/close 22538 (time 1709787942.43 total 984.18 last 26.79) - open/close 22826 (time 1709787952.43 total 994.18 last 28.80) - open/close 23139 (time 1709787962.46 total 1004.21 last 31.21) - open/close 23412 (time 1709787972.47 total 1014.21 last 27.28) - open/close 23741 (time 1709787982.49 total 1024.24 last 32.82) - open/close 24044 (time 1709787992.52 total 1034.26 last 30.23) - open/close 24339 (time 1709788002.54 total 1044.28 last 29.44) - open/close 24411 (time 1709788016.17 total 1057.91 last 5.28) - open/close 24709 (time 1709788026.17 total 1067.92 last 29.79) - open/close 25013 (time 1709788036.20 total 1077.94 last 30.33) - open/close 25316 (time 1709788046.23 total 1087.97 last 30.21) - open/close 25633 (time 1709788056.25 total 1097.99 last 31.63) - open/close 25920 (time 1709788066.28 total 1108.02 last 28.62) - open/close 26225 (time 1709788076.31 total 1118.06 last 30.39) - open/close 26525 (time 1709788086.35 total 1128.09 last 29.90) - open/close 26813 (time 1709788096.38 total 1138.12 last 28.71) - open/close 27080 (time 1709788106.41 total 1148.16 last 26.61) - open/close 27351 (time 1709788116.43 total 1158.17 last 27.05) - open/close 27624 (time 1709788126.46 total 1168.20 last 27.22) - open/close 27913 (time 1709788136.48 total 1178.22 last 28.84) - open/close 28170 (time 1709788146.50 total 1188.24 last 25.65) - open/close 28432 (time 1709788156.53 total 1198.28 last 26.11) - open/close 28691 (time 1709788166.54 total 1208.29 last 25.87) - open/close 28961 (time 1709788176.57 total 1218.32 last 26.92) - open/close 29216 (time 1709788186.60 total 1228.34 last 25.44) - open/close 29468 (time 1709788196.61 total 1238.35 last 25.17) - open/close 29765 (time 1709788206.63 total 1248.38 last 29.62) - open/close 30000 (time 1709788214.74 total 1256.48 last 29.00) - open/close 30278 (time 1709788224.74 total 1266.49 last 27.78) - open/close 30589 (time 1709788234.76 total 1276.51 last 31.04) - open/close 30864 (time 1709788244.77 total 1286.51 last 27.49) - open/close 31103 (time 1709788254.80 total 1296.55 last 23.82) - open/close 31347 (time 1709788264.84 total 1306.59 last 24.30) - open/close 31597 (time 1709788274.86 total 1316.61 last 24.95) - open/close 31860 (time 1709788284.90 total 1326.64 last 26.21) - open/close 32123 (time 1709788294.90 total 1336.65 last 26.29) - open/close 32394 (time 1709788304.90 total 1346.65 last 27.09) - open/close 32660 (time 1709788314.92 total 1356.67 last 26.55) - open/close 32936 (time 1709788324.92 total 1366.67 last 27.60) - open/close 33197 (time 1709788334.95 total 1376.69 last 26.04) - open/close 33450 (time 1709788344.95 total 1386.69 last 25.30) - open/close 33712 (time 1709788354.98 total 1396.72 last 26.13) - open/close 33980 (time 1709788364.98 total 1406.72 last 26.79) - open/close 34231 (time 1709788374.99 total 1416.73 last 25.08) - open/close 34496 (time 1709788385.02 total 1426.76 last 26.42) - open/close 34749 (time 1709788395.04 total 1436.78 last 25.24) - open/close 35005 (time 1709788405.05 total 1446.80 last 25.56) - open/close 35272 (time 1709788415.08 total 1456.83 last 26.62) - open/close 35500 (time 1709788425.09 total 1466.84 last 22.78) - open/close 35740 (time 1709788435.10 total 1476.85 last 23.98) - open/close 35972 (time 1709788445.12 total 1486.86 last 23.17) - open/close 36205 (time 1709788455.15 total 1496.90 last 23.21) - open/close 36424 (time 1709788465.19 total 1506.94 last 21.81) - open/close 36660 (time 1709788475.21 total 1516.95 last 23.57) - open/close 36879 (time 1709788485.25 total 1526.99 last 21.82) - open/close 37129 (time 1709788495.26 total 1537.01 last 24.97) - open/close 37360 (time 1709788505.31 total 1547.05 last 23.00) - open/close 37614 (time 1709788515.34 total 1557.08 last 25.31) - open/close 37851 (time 1709788525.38 total 1567.13 last 23.59) - open/close 38091 (time 1709788535.42 total 1577.16 last 23.93) - open/close 38318 (time 1709788545.45 total 1587.19 last 22.63) - open/close 38547 (time 1709788555.47 total 1597.22 last 22.84) - open/close 38791 (time 1709788565.50 total 1607.25 last 24.32) - open/close 39004 (time 1709788575.53 total 1617.27 last 21.25) - open/close 39227 (time 1709788585.57 total 1627.31 last 22.22) - open/close 39449 (time 1709788595.60 total 1637.34 last 22.13) - open/close 39674 (time 1709788605.63 total 1647.37 last 22.44) - open/close 39918 (time 1709788615.66 total 1657.40 last 24.32) - open/close 40000 (time 1709788618.83 total 1660.57 last 25.87) - open/close 40265 (time 1709788628.83 total 1670.58 last 26.49) - open/close 40503 (time 1709788638.87 total 1680.61 last 23.71) - open/close 40735 (time 1709788648.91 total 1690.65 last 23.11) - open/close 40960 (time 1709788658.94 total 1700.69 last 22.42) - open/close 41184 (time 1709788668.98 total 1710.72 last 22.32) - open/close 41390 (time 1709788679.01 total 1720.75 last 20.54) - open/close 41586 (time 1709788689.02 total 1730.76 last 19.58) - open/close 41784 (time 1709788699.02 total 1740.77 last 19.78) - open/close 42010 (time 1709788709.04 total 1750.78 last 22.56) - open/close 42228 (time 1709788719.04 total 1760.79 last 21.79) - open/close 42454 (time 1709788729.07 total 1770.82 last 22.54) - open/close 42659 (time 1709788739.09 total 1780.84 last 20.46) - open/close 42852 (time 1709788749.12 total 1790.86 last 19.25) - open/close 43052 (time 1709788759.16 total 1800.90 last 19.92) - open/close 43239 (time 1709788769.16 total 1810.91 last 18.69) - open/close 43435 (time 1709788779.18 total 1820.93 last 19.56) - open/close 43652 (time 1709788789.22 total 1830.96 last 21.62) - open/close 43866 (time 1709788799.27 total 1841.01 last 21.29) - open/close 44078 (time 1709788809.28 total 1851.03 last 21.16) - open/close 44298 (time 1709788819.30 total 1861.05 last 21.96) - open/close 44511 (time 1709788829.34 total 1871.08 last 21.23) - open/close 44726 (time 1709788839.37 total 1881.11 last 21.43) - open/close 44943 (time 1709788849.42 total 1891.16 last 21.60) - open/close 45151 (time 1709788859.45 total 1901.20 last 20.72) - open/close 45352 (time 1709788869.46 total 1911.20 last 20.08) - open/close 45563 (time 1709788879.50 total 1921.24 last 21.02) - open/close 45760 (time 1709788889.50 total 1931.25 last 19.69) - open/close 45975 (time 1709788899.50 total 1941.25 last 21.50) - open/close 46175 (time 1709788909.55 total 1951.29 last 19.91) - open/close 46377 (time 1709788919.57 total 1961.31 last 20.16) - open/close 46574 (time 1709788929.58 total 1971.32 last 19.68) - open/close 46778 (time 1709788939.62 total 1981.36 last 20.32) - open/close 46973 (time 1709788949.65 total 1991.40 last 19.43) - open/close 47164 (time 1709788959.65 total 2001.40 last 19.10) - open/close 47354 (time 1709788969.70 total 2011.44 last 18.92) - open/close 47555 (time 1709788979.70 total 2021.44 last 20.09) - open/close 47745 (time 1709788989.74 total 2031.48 last 18.92) - open/close 47939 (time 1709788999.79 total 2041.54 last 19.29) - open/close 48130 (time 1709789009.83 total 2051.58 last 19.02) - open/close 48327 (time 1709789019.85 total 2061.59 last 19.68) - open/close 48531 (time 1709789029.86 total 2071.60 last 20.38) - open/close 48728 (time 1709789039.91 total 2081.65 last 19.60) - open/close 48916 (time 1709789049.94 total 2091.68 last 18.74) - open/close 49097 (time 1709789059.96 total 2101.70 last 18.06) - open/close 49276 (time 1709789069.96 total 2111.71 last 17.89) - open/close 49448 (time 1709789080.02 total 2121.76 last 17.11) - open/close 49614 (time 1709789090.02 total 2131.76 last 16.60) - open/close 49798 (time 1709789100.05 total 2141.79 last 18.34) - open/close 49976 (time 1709789110.05 total 2151.80 last 17.79) total: 50000 open/close in 2153.06 seconds: 23.22 ops/second