== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:44:52 (1709786692)
start mds service on oleg142-server
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all
pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg142-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all
pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1
Started lustre-OST0000
oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg142-client.virtnet:  -o user_xattr,flock oleg142-server@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
 - open/close 229 (time 1709786723.98 total 10.40 last 22.02)
 - open/close 470 (time 1709786734.84 total 21.26 last 22.19)
 - open/close 705 (time 1709786746.10 total 32.51 last 20.88)
 - open/close 901 (time 1709786756.45 total 42.86 last 18.94)
 - open/close 1049 (time 1709786767.37 total 53.78 last 13.55)
 - open/close 1239 (time 1709786778.16 total 64.58 last 17.60)
 - open/close 1498 (time 1709786788.97 total 75.39 last 23.97)
 - open/close 1725 (time 1709786799.93 total 86.35 last 20.71)
 - open/close 1956 (time 1709786810.89 total 97.31 last 21.07)
 - open/close 2167 (time 1709786821.50 total 107.92 last 19.89)
 - open/close 2438 (time 1709786834.27 total 120.69 last 21.22)
 - open/close 2604 (time 1709786845.00 total 131.42 last 15.47)
 - open/close 2814 (time 1709786855.01 total 141.42 last 20.99)
 - open/close 3016 (time 1709786865.28 total 151.70 last 19.65)
 - open/close 3223 (time 1709786876.61 total 163.02 last 18.28)
 - open/close 3384 (time 1709786886.62 total 173.03 last 16.08)
 - open/close 3558 (time 1709786897.20 total 183.61 last 16.45)
 - open/close 3719 (time 1709786907.21 total 193.63 last 16.08)
 - open/close 3959 (time 1709786917.22 total 203.64 last 23.97)
 - open/close 4247 (time 1709786928.32 total 214.73 last 25.96)
 - open/close 4544 (time 1709786938.58 total 224.99 last 28.94)
 - open/close 4664 (time 1709786948.59 total 235.01 last 11.98)
 - open/close 4885 (time 1709786958.60 total 245.01 last 22.09)
 - open/close 5127 (time 1709786968.60 total 255.02 last 24.19)
 - open/close 5358 (time 1709786979.34 total 265.76 last 21.50)
 - open/close 5540 (time 1709786990.26 total 276.68 last 16.67)
 - open/close 5717 (time 1709787000.75 total 287.17 last 16.87)
 - open/close 5965 (time 1709787010.75 total 297.17 last 24.79)
 - open/close 6105 (time 1709787020.76 total 307.17 last 14.00)
 - open/close 6239 (time 1709787033.42 total 319.83 last 10.59)
 - open/close 6408 (time 1709787044.28 total 330.69 last 15.55)
 - open/close 6576 (time 1709787054.28 total 340.70 last 16.79)
 - open/close 6754 (time 1709787064.29 total 350.71 last 17.79)
 - open/close 7011 (time 1709787074.74 total 361.15 last 24.61)
 - open/close 7127 (time 1709787084.97 total 371.39 last 11.33)
 - open/close 7219 (time 1709787094.99 total 381.40 last 9.19)
 - open/close 7419 (time 1709787105.02 total 391.44 last 19.93)
 - open/close 7678 (time 1709787115.03 total 401.44 last 25.88)
 - open/close 7836 (time 1709787126.07 total 412.48 last 14.32)
 - open/close 7953 (time 1709787136.60 total 423.01 last 11.11)
 - open/close 8243 (time 1709787146.62 total 433.04 last 28.93)
 - open/close 8373 (time 1709787156.96 total 443.37 last 12.58)
 - open/close 8472 (time 1709787167.16 total 453.58 last 9.70)
 - open/close 8589 (time 1709787177.90 total 464.31 last 10.90)
 - open/close 8706 (time 1709787187.90 total 474.32 last 11.69)
 - open/close 8837 (time 1709787198.26 total 484.67 last 12.65)
 - open/close 9008 (time 1709787208.67 total 495.08 last 16.43)
 - open/close 9153 (time 1709787219.63 total 506.05 last 13.22)
 - open/close 9423 (time 1709787229.65 total 516.07 last 26.95)
 - open/close 9837 (time 1709787239.65 total 526.07 last 41.40)
 - open/close 10000 (time 1709787243.11 total 529.52 last 47.16)
 - open/close 10506 (time 1709787253.13 total 539.54 last 50.52)
 - open/close 10734 (time 1709787268.54 total 554.95 last 14.80)
 - open/close 11215 (time 1709787278.54 total 564.96 last 48.08)
 - open/close 11708 (time 1709787288.57 total 574.99 last 49.15)
 - open/close 12193 (time 1709787298.61 total 585.02 last 48.33)
 - open/close 12608 (time 1709787308.63 total 595.05 last 41.39)
 - open/close 13023 (time 1709787318.64 total 605.06 last 41.47)
 - open/close 13458 (time 1709787328.65 total 615.07 last 43.44)
 - open/close 13867 (time 1709787338.66 total 625.07 last 40.88)
 - open/close 14225 (time 1709787348.67 total 635.08 last 35.77)
 - open/close 14627 (time 1709787358.68 total 645.10 last 40.14)
 - open/close 14958 (time 1709787368.70 total 655.11 last 33.05)
 - open/close 15342 (time 1709787378.71 total 665.13 last 38.34)
 - open/close 15666 (time 1709787388.71 total 675.13 last 32.40)
 - open/close 15993 (time 1709787398.72 total 685.13 last 32.69)
 - open/close 16321 (time 1709787408.74 total 695.15 last 32.73)
 - open/close 16665 (time 1709787418.76 total 705.17 last 34.33)
 - open/close 16903 (time 1709787436.78 total 723.20 last 13.21)
 - open/close 17239 (time 1709787446.79 total 733.21 last 33.56)
 - open/close 17572 (time 1709787456.80 total 743.21 last 33.28)
 - open/close 17911 (time 1709787466.82 total 753.23 last 33.83)
 - open/close 18213 (time 1709787476.83 total 763.24 last 30.16)
 - open/close 18504 (time 1709787486.86 total 773.28 last 29.00)
 - open/close 18853 (time 1709787496.89 total 783.30 last 34.82)
 - open/close 19155 (time 1709787506.90 total 793.31 last 30.16)
 - open/close 19490 (time 1709787516.91 total 803.33 last 33.45)
 - open/close 19821 (time 1709787526.95 total 813.36 last 32.98)
 - open/close 20000 (time 1709787532.86 total 819.27 last 30.28)
 - open/close 20288 (time 1709787542.90 total 829.31 last 28.69)
 - open/close 20569 (time 1709787552.94 total 839.36 last 27.97)
 - open/close 20840 (time 1709787562.98 total 849.40 last 27.00)
 - open/close 21114 (time 1709787573.02 total 859.43 last 27.31)
 - open/close 21390 (time 1709787583.02 total 869.44 last 27.58)
 - open/close 21704 (time 1709787593.04 total 879.46 last 31.33)
 - open/close 21982 (time 1709787603.05 total 889.47 last 27.77)
 - open/close 22256 (time 1709787613.06 total 899.48 last 27.37)
 - open/close 22541 (time 1709787623.06 total 909.48 last 28.50)
 - open/close 22829 (time 1709787633.08 total 919.49 last 28.76)
 - open/close 23123 (time 1709787643.08 total 929.50 last 29.39)
 - open/close 23396 (time 1709787653.13 total 939.55 last 27.17)
 - open/close 23665 (time 1709787663.16 total 949.58 last 26.81)
 - open/close 23925 (time 1709787673.17 total 959.59 last 25.98)
 - open/close 24192 (time 1709787683.19 total 969.61 last 26.65)
 - open/close 24460 (time 1709787693.20 total 979.61 last 26.78)
 - open/close 24736 (time 1709787703.20 total 989.62 last 27.59)
 - open/close 25025 (time 1709787713.22 total 999.63 last 28.85)
 - open/close 25323 (time 1709787723.25 total 1009.66 last 29.71)
 - open/close 25632 (time 1709787733.26 total 1019.68 last 30.85)
 - open/close 25887 (time 1709787743.32 total 1029.74 last 25.35)
 - open/close 26153 (time 1709787753.33 total 1039.75 last 26.58)
 - open/close 26397 (time 1709787763.34 total 1049.75 last 24.39)
 - open/close 26662 (time 1709787773.34 total 1059.76 last 26.49)
 - open/close 26905 (time 1709787783.34 total 1069.76 last 24.30)
 - open/close 27124 (time 1709787793.38 total 1079.80 last 21.81)
 - open/close 27367 (time 1709787803.39 total 1089.80 last 24.29)
 - open/close 27638 (time 1709787813.41 total 1099.82 last 27.05)
 - open/close 27886 (time 1709787823.42 total 1109.84 last 24.75)
 - open/close 28111 (time 1709787833.45 total 1119.86 last 22.45)
 - open/close 28334 (time 1709787843.46 total 1129.88 last 22.27)
 - open/close 28556 (time 1709787853.48 total 1139.90 last 22.15)
 - open/close 28793 (time 1709787863.49 total 1149.91 last 23.67)
 - open/close 29028 (time 1709787873.51 total 1159.93 last 23.45)
 - open/close 29247 (time 1709787883.53 total 1169.94 last 21.87)
 - open/close 29477 (time 1709787893.54 total 1179.96 last 22.97)
 - open/close 29706 (time 1709787903.63 total 1190.04 last 22.71)
 - open/close 29922 (time 1709787913.63 total 1200.04 last 21.60)
 - open/close 30000 (time 1709787917.29 total 1203.70 last 21.31)
 - open/close 30210 (time 1709787927.29 total 1213.71 last 20.98)
 - open/close 30417 (time 1709787937.31 total 1223.73 last 20.66)
 - open/close 30636 (time 1709787947.36 total 1233.77 last 21.80)
 - open/close 30650 (time 1709787959.14 total 1245.55 last 1.19)
 - open/close 30900 (time 1709787969.15 total 1255.57 last 24.96)
 - open/close 31139 (time 1709787979.18 total 1265.59 last 23.84)
 - open/close 31399 (time 1709787989.20 total 1275.61 last 25.95)
 - open/close 31655 (time 1709787999.20 total 1285.62 last 25.59)
 - open/close 31900 (time 1709788009.21 total 1295.62 last 24.48)
 - open/close 32148 (time 1709788019.23 total 1305.64 last 24.75)
 - open/close 32382 (time 1709788029.26 total 1315.67 last 23.33)
 - open/close 32633 (time 1709788039.28 total 1325.70 last 25.03)
 - open/close 32878 (time 1709788049.29 total 1335.71 last 24.48)
 - open/close 33118 (time 1709788059.30 total 1345.72 last 23.98)
 - open/close 33363 (time 1709788069.33 total 1355.74 last 24.44)
 - open/close 33630 (time 1709788079.35 total 1365.76 last 26.64)
 - open/close 33879 (time 1709788089.38 total 1375.80 last 24.82)
 - open/close 34118 (time 1709788099.41 total 1385.83 last 23.83)
 - open/close 34351 (time 1709788109.45 total 1395.87 last 23.21)
 - open/close 34581 (time 1709788119.47 total 1405.89 last 22.95)
 - open/close 34811 (time 1709788129.50 total 1415.91 last 22.94)
 - open/close 35072 (time 1709788139.50 total 1425.92 last 26.08)
 - open/close 35296 (time 1709788149.53 total 1435.95 last 22.34)
 - open/close 35514 (time 1709788159.56 total 1445.97 last 21.74)
 - open/close 35735 (time 1709788169.58 total 1456.00 last 22.05)
 - open/close 35972 (time 1709788179.58 total 1466.00 last 23.70)
 - open/close 36202 (time 1709788189.60 total 1476.02 last 22.96)
 - open/close 36424 (time 1709788199.63 total 1486.04 last 22.14)
 - open/close 36685 (time 1709788209.63 total 1496.04 last 26.10)
 - open/close 36939 (time 1709788219.65 total 1506.07 last 25.34)
 - open/close 37186 (time 1709788229.67 total 1516.08 last 24.67)
 - open/close 37433 (time 1709788239.69 total 1526.10 last 24.66)
 - open/close 37678 (time 1709788249.71 total 1536.12 last 24.45)
 - open/close 37881 (time 1709788259.73 total 1546.15 last 20.25)
 - open/close 38101 (time 1709788269.74 total 1556.16 last 21.98)
 - open/close 38326 (time 1709788279.78 total 1566.20 last 22.41)
 - open/close 38565 (time 1709788289.82 total 1576.24 last 23.81)
 - open/close 38797 (time 1709788299.83 total 1586.25 last 23.18)
 - open/close 39035 (time 1709788309.86 total 1596.27 last 23.75)
 - open/close 39253 (time 1709788319.88 total 1606.30 last 21.74)
 - open/close 39488 (time 1709788329.92 total 1616.33 last 23.42)
 - open/close 39716 (time 1709788339.93 total 1626.35 last 22.77)
 - open/close 39935 (time 1709788349.94 total 1636.36 last 21.87)
 - open/close 40000 (time 1709788352.74 total 1639.16 last 23.23)
 - open/close 40236 (time 1709788362.76 total 1649.18 last 23.55)
 - open/close 40461 (time 1709788372.77 total 1659.18 last 22.49)
 - open/close 40693 (time 1709788382.77 total 1669.18 last 23.19)
 - open/close 40920 (time 1709788392.82 total 1679.23 last 22.59)
 - open/close 41140 (time 1709788402.85 total 1689.26 last 21.94)
 - open/close 41371 (time 1709788412.85 total 1699.27 last 23.08)
 - open/close 41584 (time 1709788422.89 total 1709.30 last 21.23)
 - open/close 41800 (time 1709788432.90 total 1719.31 last 21.58)
 - open/close 42000 (time 1709788442.92 total 1729.34 last 19.95)
 - open/close 42199 (time 1709788452.96 total 1739.37 last 19.83)
 - open/close 42412 (time 1709788462.97 total 1749.38 last 21.27)
 - open/close 42612 (time 1709788472.98 total 1759.40 last 19.98)
 - open/close 42817 (time 1709788483.02 total 1769.43 last 20.43)
 - open/close 43026 (time 1709788493.05 total 1779.47 last 20.82)
 - open/close 43239 (time 1709788503.10 total 1789.52 last 21.19)
 - open/close 43455 (time 1709788513.11 total 1799.53 last 21.58)
 - open/close 43666 (time 1709788523.15 total 1809.56 last 21.02)
 - open/close 43881 (time 1709788533.16 total 1819.57 last 21.48)
 - open/close 44110 (time 1709788543.18 total 1829.60 last 22.84)
 - open/close 44330 (time 1709788553.23 total 1839.64 last 21.91)
 - open/close 44553 (time 1709788563.24 total 1849.65 last 22.27)
 - open/close 44775 (time 1709788573.25 total 1859.66 last 22.18)
 - open/close 44979 (time 1709788583.27 total 1869.69 last 20.35)
 - open/close 45197 (time 1709788593.30 total 1879.71 last 21.75)
 - open/close 45413 (time 1709788603.30 total 1889.72 last 21.59)
 - open/close 45632 (time 1709788613.30 total 1899.72 last 21.90)
 - open/close 45852 (time 1709788623.33 total 1909.75 last 21.93)
 - open/close 46075 (time 1709788633.35 total 1919.76 last 22.27)
 - open/close 46278 (time 1709788643.37 total 1929.79 last 20.25)
 - open/close 46480 (time 1709788653.40 total 1939.82 last 20.14)
 - open/close 46673 (time 1709788663.41 total 1949.82 last 19.29)
 - open/close 46868 (time 1709788673.42 total 1959.84 last 19.46)
 - open/close 47042 (time 1709788683.45 total 1969.87 last 17.35)
 - open/close 47116 (time 1709788698.56 total 1984.98 last 4.90)
 - open/close 47321 (time 1709788708.58 total 1994.99 last 20.47)
 - open/close 47525 (time 1709788718.59 total 2005.00 last 20.37)
 - open/close 47726 (time 1709788728.60 total 2015.02 last 20.07)
 - open/close 47918 (time 1709788738.62 total 2025.03 last 19.17)
 - open/close 48096 (time 1709788748.66 total 2035.07 last 17.74)
 - open/close 48274 (time 1709788758.68 total 2045.10 last 17.75)
 - open/close 48441 (time 1709788768.71 total 2055.12 last 16.66)
 - open/close 48614 (time 1709788778.71 total 2065.12 last 17.30)
 - open/close 48798 (time 1709788788.75 total 2075.17 last 18.32)
 - open/close 48997 (time 1709788798.77 total 2085.19 last 19.86)
 - open/close 49196 (time 1709788808.80 total 2095.22 last 19.84)
 - open/close 49399 (time 1709788818.82 total 2105.23 last 20.27)
 - open/close 49593 (time 1709788828.85 total 2115.26 last 19.35)
 - open/close 49793 (time 1709788838.89 total 2125.31 last 19.91)
 - open/close 49993 (time 1709788848.91 total 2135.33 last 19.97)
total: 50000 open/close in 2135.61 seconds: 23.41 ops/second