== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:42:26 (1709786546) start mds service on oleg158-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg158-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock oleg158-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1709786575.43 total 10.04 last 18.12) - open/close 390 (time 1709786587.65 total 22.26 last 17.03) - open/close 599 (time 1709786597.65 total 32.26 last 20.89) - open/close 883 (time 1709786607.66 total 42.27 last 28.39) - open/close 1118 (time 1709786617.96 total 52.56 last 22.82) - open/close 1506 (time 1709786627.96 total 62.57 last 38.78) - open/close 1726 (time 1709786637.97 total 72.58 last 21.99) - open/close 1993 (time 1709786648.67 total 83.28 last 24.94) - open/close 2186 (time 1709786658.68 total 93.29 last 19.29) - open/close 2451 (time 1709786668.87 total 103.48 last 26.00) - open/close 2734 (time 1709786679.35 total 113.96 last 27.01) - open/close 2969 (time 1709786689.48 total 124.08 last 23.21) - open/close 3210 (time 1709786699.62 total 134.23 last 23.76) - open/close 3459 (time 1709786710.10 total 144.71 last 23.75) - open/close 3680 (time 1709786720.16 total 154.77 last 21.97) - open/close 4018 (time 1709786730.19 total 164.80 last 33.71) - open/close 4290 (time 1709786741.09 total 175.70 last 24.95) - open/close 4485 (time 1709786751.93 total 186.54 last 17.98) - open/close 4726 (time 1709786762.02 total 196.63 last 23.89) - open/close 4838 (time 1709786772.03 total 206.64 last 11.19) - open/close 5084 (time 1709786782.81 total 217.42 last 22.81) - open/close 5351 (time 1709786792.82 total 227.43 last 26.69) - open/close 5478 (time 1709786802.82 total 237.43 last 12.70) - open/close 5620 (time 1709786813.17 total 247.78 last 13.72) - open/close 5828 (time 1709786823.19 total 257.80 last 20.75) - open/close 6021 (time 1709786836.17 total 270.78 last 14.87) - open/close 6283 (time 1709786846.18 total 280.79 last 26.19) - open/close 6464 (time 1709786856.43 total 291.03 last 17.66) - open/close 6662 (time 1709786866.43 total 301.03 last 19.80) - open/close 6771 (time 1709786876.64 total 311.25 last 10.67) - open/close 6940 (time 1709786887.41 total 322.02 last 15.69) - open/close 7098 (time 1709786898.13 total 332.74 last 14.74) - open/close 7215 (time 1709786908.44 total 343.05 last 11.35) - open/close 7470 (time 1709786918.45 total 353.06 last 25.48) - open/close 7663 (time 1709786929.26 total 363.87 last 17.85) - open/close 7822 (time 1709786939.28 total 373.89 last 15.87) - open/close 7931 (time 1709786949.41 total 384.02 last 10.76) - open/close 8183 (time 1709786959.44 total 394.05 last 25.13) - open/close 8358 (time 1709786970.20 total 404.81 last 16.26) - open/close 8590 (time 1709786980.21 total 414.81 last 23.19) - open/close 8837 (time 1709786990.77 total 425.38 last 23.39) - open/close 9321 (time 1709787000.78 total 435.39 last 48.32) - open/close 9412 (time 1709787013.46 total 448.07 last 7.18) - open/close 9954 (time 1709787023.47 total 458.08 last 54.17) - open/close 10521 (time 1709787033.48 total 468.09 last 56.62) - open/close 11083 (time 1709787043.49 total 478.10 last 56.16) - open/close 11627 (time 1709787053.50 total 488.11 last 54.33) - open/close 12109 (time 1709787063.51 total 498.12 last 48.16) - open/close 12540 (time 1709787073.52 total 508.13 last 43.05) - open/close 13027 (time 1709787083.52 total 518.13 last 48.68) - open/close 13478 (time 1709787093.55 total 528.16 last 44.98) - open/close 13914 (time 1709787114.47 total 549.08 last 20.84) - open/close 14312 (time 1709787124.47 total 559.08 last 39.77) - open/close 14730 (time 1709787134.50 total 569.11 last 41.69) - open/close 14973 (time 1709787152.74 total 587.35 last 13.32) - open/close 15356 (time 1709787162.77 total 597.37 last 38.21) - open/close 15716 (time 1709787172.79 total 607.40 last 35.90) - open/close 16050 (time 1709787182.81 total 617.42 last 33.35) - open/close 16373 (time 1709787192.83 total 627.44 last 32.23) - open/close 16700 (time 1709787202.86 total 637.47 last 32.60) - open/close 17012 (time 1709787212.88 total 647.48 last 31.14) - open/close 17328 (time 1709787222.91 total 657.52 last 31.48) - open/close 17657 (time 1709787232.96 total 667.57 last 32.74) - open/close 17969 (time 1709787242.98 total 677.59 last 31.15) - open/close 18327 (time 1709787253.00 total 687.61 last 35.73) - open/close 18683 (time 1709787263.04 total 697.65 last 35.44) - open/close 19040 (time 1709787273.06 total 707.67 last 35.63) - open/close 19365 (time 1709787283.07 total 717.68 last 32.46) - open/close 19684 (time 1709787293.09 total 727.70 last 31.85) - open/close 19790 (time 1709787307.27 total 741.88 last 7.47) - open/close 20000 (time 1709787314.98 total 749.59 last 27.25) - open/close 20276 (time 1709787325.02 total 759.63 last 27.48) - open/close 20550 (time 1709787335.05 total 769.66 last 27.32) - open/close 20786 (time 1709787354.87 total 789.48 last 11.91) - open/close 21056 (time 1709787364.90 total 799.51 last 26.93) - open/close 21337 (time 1709787374.92 total 809.53 last 28.05) - open/close 21591 (time 1709787384.94 total 819.55 last 25.34) - open/close 21851 (time 1709787394.96 total 829.57 last 25.95) - open/close 22121 (time 1709787404.96 total 839.57 last 26.99) - open/close 22400 (time 1709787414.97 total 849.58 last 27.88) - open/close 22698 (time 1709787425.00 total 859.61 last 29.73) - open/close 22918 (time 1709787435.03 total 869.64 last 21.92) - open/close 23161 (time 1709787445.06 total 879.67 last 24.22) - open/close 23402 (time 1709787455.09 total 889.70 last 24.04) - open/close 23638 (time 1709787465.09 total 899.70 last 23.59) - open/close 23872 (time 1709787475.14 total 909.75 last 23.29) - open/close 24100 (time 1709787485.19 total 919.80 last 22.69) - open/close 24332 (time 1709787495.20 total 929.81 last 23.17) - open/close 24550 (time 1709787505.22 total 939.83 last 21.75) - open/close 24802 (time 1709787515.25 total 949.86 last 25.12) - open/close 25053 (time 1709787525.30 total 959.91 last 24.98) - open/close 25306 (time 1709787535.31 total 969.92 last 25.28) - open/close 25547 (time 1709787545.31 total 979.92 last 24.08) - open/close 25776 (time 1709787555.33 total 989.94 last 22.87) - open/close 25991 (time 1709787565.35 total 999.96 last 21.46) - open/close 26231 (time 1709787575.37 total 1009.98 last 23.95) - open/close 26464 (time 1709787585.40 total 1020.00 last 23.24) - open/close 26708 (time 1709787595.40 total 1030.00 last 24.40) - open/close 26940 (time 1709787605.41 total 1040.02 last 23.16) - open/close 27181 (time 1709787615.44 total 1050.05 last 24.02) - open/close 27415 (time 1709787625.45 total 1060.05 last 23.39) - open/close 27556 (time 1709787642.74 total 1077.35 last 8.15) - open/close 27789 (time 1709787652.75 total 1087.36 last 23.28) - open/close 28012 (time 1709787662.82 total 1097.43 last 22.15) - open/close 28225 (time 1709787672.83 total 1107.44 last 21.28) - open/close 28467 (time 1709787682.85 total 1117.46 last 24.14) - open/close 28697 (time 1709787692.86 total 1127.47 last 22.99) - open/close 28923 (time 1709787702.88 total 1137.49 last 22.55) - open/close 29170 (time 1709787712.91 total 1147.52 last 24.63) - open/close 29414 (time 1709787722.93 total 1157.54 last 24.35) - open/close 29664 (time 1709787732.98 total 1167.59 last 24.88) - open/close 29887 (time 1709787742.99 total 1177.60 last 22.27) - open/close 30000 (time 1709787747.70 total 1182.31 last 24.03) - open/close 30229 (time 1709787757.73 total 1192.34 last 22.83) - open/close 30290 (time 1709787771.75 total 1206.35 last 4.35) - open/close 30526 (time 1709787781.82 total 1216.43 last 23.43) - open/close 30739 (time 1709787791.83 total 1226.44 last 21.27) - open/close 30941 (time 1709787801.87 total 1236.48 last 20.12) - open/close 31175 (time 1709787811.87 total 1246.48 last 23.40) - open/close 31414 (time 1709787821.89 total 1256.50 last 23.86) - open/close 31636 (time 1709787831.90 total 1266.50 last 22.19) - open/close 31835 (time 1709787841.94 total 1276.55 last 19.81) - open/close 32034 (time 1709787851.97 total 1286.58 last 19.85) - open/close 32245 (time 1709787862.00 total 1296.60 last 21.04) - open/close 32438 (time 1709787882.12 total 1316.73 last 9.59) - open/close 32652 (time 1709787892.15 total 1326.76 last 21.34) - open/close 32860 (time 1709787902.18 total 1336.79 last 20.73) - open/close 33060 (time 1709787912.21 total 1346.82 last 19.94) - open/close 33249 (time 1709787922.22 total 1356.83 last 18.89) - open/close 33450 (time 1709787932.25 total 1366.86 last 20.03) - open/close 33651 (time 1709787942.30 total 1376.90 last 20.02) - open/close 33869 (time 1709787952.30 total 1386.91 last 21.78) - open/close 34104 (time 1709787962.32 total 1396.93 last 23.46) - open/close 34316 (time 1709787972.35 total 1406.95 last 21.15) - open/close 34544 (time 1709787982.40 total 1417.00 last 22.69) - open/close 34783 (time 1709787992.43 total 1427.04 last 23.81) - open/close 35010 (time 1709788002.43 total 1437.04 last 22.70) - open/close 35235 (time 1709788012.45 total 1447.06 last 22.47) - open/close 35445 (time 1709788022.49 total 1457.10 last 20.91) - open/close 35677 (time 1709788032.51 total 1467.12 last 23.16) - open/close 35914 (time 1709788042.55 total 1477.16 last 23.61) - open/close 36136 (time 1709788052.57 total 1487.18 last 22.14) - open/close 36367 (time 1709788062.62 total 1497.22 last 23.01) - open/close 36605 (time 1709788072.64 total 1507.25 last 23.74) - open/close 36840 (time 1709788082.67 total 1517.28 last 23.43) - open/close 37063 (time 1709788092.67 total 1527.28 last 22.30) - open/close 37294 (time 1709788102.68 total 1537.29 last 23.08) - open/close 37505 (time 1709788112.69 total 1547.30 last 21.07) - open/close 37714 (time 1709788122.69 total 1557.30 last 20.89) - open/close 37932 (time 1709788132.72 total 1567.33 last 21.74) - open/close 38160 (time 1709788142.73 total 1577.33 last 22.78) - open/close 38366 (time 1709788152.73 total 1587.34 last 20.59) - open/close 38579 (time 1709788162.73 total 1597.34 last 21.29) - open/close 38786 (time 1709788172.76 total 1607.37 last 20.65) - open/close 38997 (time 1709788182.78 total 1617.39 last 21.06) - open/close 39203 (time 1709788192.81 total 1627.42 last 20.53) - open/close 39419 (time 1709788202.81 total 1637.42 last 21.60) - open/close 39663 (time 1709788212.82 total 1647.43 last 24.38) - open/close 39890 (time 1709788222.84 total 1657.45 last 22.65) - open/close 40000 (time 1709788227.85 total 1662.46 last 21.96) - open/close 40230 (time 1709788237.85 total 1672.46 last 23.00) - open/close 40442 (time 1709788247.87 total 1682.48 last 21.17) - open/close 40623 (time 1709788257.89 total 1692.50 last 18.06) - open/close 40829 (time 1709788267.92 total 1702.53 last 20.54) - open/close 41039 (time 1709788277.94 total 1712.55 last 20.95) - open/close 41251 (time 1709788287.97 total 1722.58 last 21.14) - open/close 41474 (time 1709788298.00 total 1732.61 last 22.22) - open/close 41690 (time 1709788308.05 total 1742.66 last 21.50) - open/close 41894 (time 1709788318.09 total 1752.70 last 20.32) - open/close 42124 (time 1709788328.13 total 1762.74 last 22.92) - open/close 42343 (time 1709788338.16 total 1772.76 last 21.84) - open/close 42546 (time 1709788348.19 total 1782.80 last 20.23) - open/close 42765 (time 1709788358.24 total 1792.85 last 21.80) - open/close 42978 (time 1709788368.27 total 1802.88 last 21.23) - open/close 43188 (time 1709788378.27 total 1812.88 last 20.99) - open/close 43402 (time 1709788388.29 total 1822.90 last 21.36) - open/close 43605 (time 1709788398.31 total 1832.92 last 20.27) - open/close 43817 (time 1709788408.34 total 1842.95 last 21.13) - open/close 44024 (time 1709788418.36 total 1852.97 last 20.66) - open/close 44220 (time 1709788428.39 total 1863.00 last 19.55) - open/close 44421 (time 1709788438.43 total 1873.04 last 20.02) - open/close 44623 (time 1709788448.44 total 1883.05 last 20.17) - open/close 44844 (time 1709788458.47 total 1893.08 last 22.03) - open/close 45038 (time 1709788468.48 total 1903.09 last 19.38) - open/close 45230 (time 1709788478.49 total 1913.10 last 19.18) - open/close 45426 (time 1709788488.54 total 1923.15 last 19.51) - open/close 45633 (time 1709788498.55 total 1933.16 last 20.68) - open/close 45837 (time 1709788508.55 total 1943.16 last 20.39) - open/close 46055 (time 1709788518.57 total 1953.18 last 21.75) - open/close 46262 (time 1709788528.58 total 1963.18 last 20.69) - open/close 46475 (time 1709788538.62 total 1973.23 last 21.20) - open/close 46677 (time 1709788548.65 total 1983.26 last 20.14) - open/close 46883 (time 1709788558.67 total 1993.28 last 20.56) - open/close 47100 (time 1709788568.68 total 2003.29 last 21.69) - open/close 47301 (time 1709788578.68 total 2013.29 last 20.10) - open/close 47496 (time 1709788588.68 total 2023.29 last 19.50) - open/close 47706 (time 1709788598.70 total 2033.31 last 20.96) - open/close 47914 (time 1709788608.70 total 2043.31 last 20.79) - open/close 48129 (time 1709788618.75 total 2053.36 last 21.40) - open/close 48362 (time 1709788628.79 total 2063.39 last 23.21) - open/close 48576 (time 1709788638.81 total 2073.42 last 21.35) - open/close 48782 (time 1709788648.82 total 2083.43 last 20.58) - open/close 48986 (time 1709788658.82 total 2093.43 last 20.40) - open/close 49187 (time 1709788668.83 total 2103.44 last 20.07) - open/close 49378 (time 1709788678.84 total 2113.45 last 19.09) - open/close 49563 (time 1709788688.86 total 2123.46 last 18.47) - open/close 49748 (time 1709788698.86 total 2133.47 last 18.48) - open/close 49945 (time 1709788708.89 total 2143.50 last 19.65) total: 50000 open/close in 2146.25 seconds: 23.30 ops/second