== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:09 (1709786889) start mds service on oleg130-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg130-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg130-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg130-client.virtnet: -o user_xattr,flock oleg130-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1709786936.05 total 10.48 last 19.55) - open/close 362 (time 1709786946.08 total 20.51 last 15.66) - open/close 517 (time 1709786956.89 total 31.32 last 14.34) - open/close 737 (time 1709786966.95 total 41.38 last 21.87) - open/close 945 (time 1709786976.96 total 51.38 last 20.79) - open/close 1231 (time 1709786986.96 total 61.39 last 28.58) - open/close 1409 (time 1709786997.27 total 71.70 last 17.27) - open/close 1693 (time 1709787007.67 total 82.09 last 27.32) - open/close 1937 (time 1709787017.81 total 92.24 last 24.05) - open/close 2192 (time 1709787028.61 total 103.04 last 23.61) - open/close 2424 (time 1709787039.58 total 114.01 last 21.14) - open/close 2616 (time 1709787050.38 total 124.81 last 17.78) - open/close 2873 (time 1709787060.38 total 134.81 last 25.70) - open/close 3069 (time 1709787071.10 total 145.53 last 18.29) - open/close 3322 (time 1709787081.89 total 156.32 last 23.45) - open/close 3568 (time 1709787092.79 total 167.21 last 22.58) - open/close 3742 (time 1709787103.13 total 177.56 last 16.82) - open/close 3861 (time 1709787113.16 total 187.58 last 11.87) - open/close 4021 (time 1709787123.82 total 198.25 last 15.01) - open/close 4193 (time 1709787133.82 total 208.25 last 17.20) - open/close 4435 (time 1709787143.85 total 218.28 last 24.13) - open/close 4555 (time 1709787154.18 total 228.61 last 11.61) - open/close 4727 (time 1709787164.20 total 238.63 last 17.17) - open/close 4995 (time 1709787174.38 total 248.81 last 26.32) - open/close 5152 (time 1709787184.40 total 258.83 last 15.68) - open/close 5325 (time 1709787194.41 total 268.84 last 17.28) - open/close 5521 (time 1709787205.07 total 279.50 last 18.38) - open/close 5675 (time 1709787215.53 total 289.96 last 14.72) - open/close 5862 (time 1709787225.85 total 300.28 last 18.12) - open/close 6088 (time 1709787235.86 total 310.29 last 22.57) - open/close 6340 (time 1709787246.89 total 321.32 last 22.86) - open/close 6440 (time 1709787257.04 total 331.47 last 9.85) - open/close 6590 (time 1709787267.47 total 341.90 last 14.37) - open/close 6742 (time 1709787277.98 total 352.41 last 14.46) - open/close 6925 (time 1709787287.98 total 362.41 last 18.30) - open/close 7105 (time 1709787298.77 total 373.20 last 16.69) - open/close 7262 (time 1709787309.50 total 383.93 last 14.63) - open/close 7378 (time 1709787320.20 total 394.63 last 10.84) - open/close 7596 (time 1709787330.70 total 405.13 last 20.76) - open/close 7725 (time 1709787340.86 total 415.29 last 12.70) - open/close 7889 (time 1709787350.86 total 425.29 last 16.40) - open/close 8045 (time 1709787360.86 total 435.29 last 15.59) - open/close 8158 (time 1709787370.87 total 445.30 last 11.29) - open/close 8211 (time 1709787381.04 total 455.47 last 5.21) - open/close 8376 (time 1709787391.08 total 465.51 last 16.43) - open/close 8554 (time 1709787401.87 total 476.30 last 16.49) - open/close 8699 (time 1709787412.82 total 487.25 last 13.24) - open/close 8802 (time 1709787423.25 total 497.67 last 9.88) - open/close 8829 (time 1709787437.08 total 511.51 last 1.95) - open/close 8980 (time 1709787447.14 total 521.57 last 15.01) - open/close 9207 (time 1709787457.93 total 532.36 last 21.05) - open/close 9336 (time 1709787468.57 total 543.00 last 12.12) - open/close 9439 (time 1709787478.57 total 553.00 last 10.30) - open/close 9632 (time 1709787488.85 total 563.28 last 18.78) - open/close 10000 (time 1709787497.70 total 572.13 last 41.58) - open/close 10419 (time 1709787507.71 total 582.14 last 41.84) - open/close 10809 (time 1709787517.71 total 592.14 last 39.00) - open/close 11244 (time 1709787527.74 total 602.17 last 43.38) - open/close 11707 (time 1709787537.75 total 612.18 last 46.27) - open/close 11965 (time 1709787555.47 total 629.90 last 14.56) - open/close 12137 (time 1709787571.45 total 645.88 last 10.77) - open/close 12544 (time 1709787581.46 total 655.89 last 40.66) - open/close 12911 (time 1709787591.48 total 665.91 last 36.61) - open/close 13264 (time 1709787601.51 total 675.94 last 35.18) - open/close 13659 (time 1709787611.52 total 685.95 last 39.49) - open/close 14042 (time 1709787621.55 total 695.98 last 38.16) - open/close 14390 (time 1709787631.57 total 706.00 last 34.75) - open/close 14786 (time 1709787641.61 total 716.03 last 39.46) - open/close 15162 (time 1709787651.61 total 726.04 last 37.57) - open/close 15498 (time 1709787661.64 total 736.07 last 33.51) - open/close 15814 (time 1709787671.66 total 746.09 last 31.55) - open/close 16189 (time 1709787681.68 total 756.11 last 37.42) - open/close 16492 (time 1709787691.71 total 766.14 last 30.20) - open/close 16821 (time 1709787701.72 total 776.15 last 32.87) - open/close 16941 (time 1709787716.02 total 790.45 last 8.39) - open/close 17274 (time 1709787726.03 total 800.46 last 33.27) - open/close 17623 (time 1709787736.03 total 810.46 last 34.87) - open/close 17924 (time 1709787746.05 total 820.47 last 30.07) - open/close 18210 (time 1709787756.06 total 830.49 last 28.55) - open/close 18518 (time 1709787766.08 total 840.51 last 30.75) - open/close 18826 (time 1709787776.08 total 850.51 last 30.78) - open/close 19114 (time 1709787786.11 total 860.54 last 28.71) - open/close 19392 (time 1709787796.14 total 870.57 last 27.73) - open/close 19710 (time 1709787806.16 total 880.59 last 31.74) - open/close 20000 (time 1709787814.99 total 889.42 last 32.84) - open/close 20310 (time 1709787824.99 total 899.42 last 30.99) - open/close 20583 (time 1709787835.01 total 909.44 last 27.24) - open/close 20835 (time 1709787845.02 total 919.45 last 25.19) - open/close 21117 (time 1709787855.04 total 929.47 last 28.13) - open/close 21393 (time 1709787865.07 total 939.50 last 27.53) - open/close 21684 (time 1709787875.09 total 949.52 last 29.04) - open/close 21951 (time 1709787885.12 total 959.55 last 26.62) - open/close 22233 (time 1709787895.15 total 969.57 last 28.13) - open/close 22506 (time 1709787905.15 total 979.58 last 27.29) - open/close 22785 (time 1709787915.17 total 989.60 last 27.85) - open/close 23030 (time 1709787925.22 total 999.65 last 24.37) - open/close 23282 (time 1709787935.23 total 1009.66 last 25.17) - open/close 23547 (time 1709787945.26 total 1019.69 last 26.41) - open/close 23820 (time 1709787955.27 total 1029.70 last 27.29) - open/close 24100 (time 1709787965.27 total 1039.70 last 27.99) - open/close 24387 (time 1709787975.29 total 1049.72 last 28.63) - open/close 24687 (time 1709787985.31 total 1059.74 last 29.94) - open/close 24989 (time 1709787995.34 total 1069.77 last 30.11) - open/close 25273 (time 1709788005.35 total 1079.78 last 28.39) - open/close 25553 (time 1709788015.37 total 1089.80 last 27.94) - open/close 25832 (time 1709788025.38 total 1099.81 last 27.86) - open/close 26115 (time 1709788035.40 total 1109.82 last 28.27) - open/close 26397 (time 1709788045.42 total 1119.85 last 28.13) - open/close 26679 (time 1709788055.43 total 1129.86 last 28.18) - open/close 26946 (time 1709788065.46 total 1139.89 last 26.61) - open/close 27229 (time 1709788075.47 total 1149.90 last 28.28) - open/close 27530 (time 1709788085.49 total 1159.92 last 30.04) - open/close 27806 (time 1709788095.50 total 1169.93 last 27.57) - open/close 28080 (time 1709788105.53 total 1179.96 last 27.32) - open/close 28337 (time 1709788115.54 total 1189.97 last 25.66) - open/close 28598 (time 1709788125.56 total 1199.99 last 26.06) - open/close 28888 (time 1709788135.59 total 1210.02 last 28.90) - open/close 29158 (time 1709788145.59 total 1220.02 last 26.99) - open/close 29426 (time 1709788155.60 total 1230.03 last 26.79) - open/close 29674 (time 1709788165.60 total 1240.03 last 24.80) - open/close 29924 (time 1709788175.60 total 1250.03 last 25.00) - open/close 30000 (time 1709788178.48 total 1252.91 last 26.42) - open/close 30247 (time 1709788188.50 total 1262.93 last 24.64) - open/close 30488 (time 1709788198.52 total 1272.94 last 24.07) - open/close 30761 (time 1709788208.54 total 1282.97 last 27.24) - open/close 31023 (time 1709788218.55 total 1292.98 last 26.17) - open/close 31285 (time 1709788228.57 total 1303.00 last 26.15) - open/close 31558 (time 1709788238.60 total 1313.03 last 27.22) - open/close 31804 (time 1709788248.61 total 1323.04 last 24.58) - open/close 32001 (time 1709788258.62 total 1333.05 last 19.68) - open/close 32244 (time 1709788268.66 total 1343.09 last 24.21) - open/close 32474 (time 1709788278.66 total 1353.09 last 22.98) - open/close 32749 (time 1709788288.68 total 1363.11 last 27.46) - open/close 33022 (time 1709788298.69 total 1373.12 last 27.27) - open/close 33273 (time 1709788308.73 total 1383.15 last 25.01) - open/close 33541 (time 1709788318.74 total 1393.17 last 26.75) - open/close 33809 (time 1709788328.76 total 1403.19 last 26.76) - open/close 34067 (time 1709788338.79 total 1413.22 last 25.72) - open/close 34315 (time 1709788348.84 total 1423.27 last 24.68) - open/close 34575 (time 1709788358.86 total 1433.29 last 25.93) - open/close 34833 (time 1709788368.89 total 1443.32 last 25.73) - open/close 35086 (time 1709788378.91 total 1453.34 last 25.25) - open/close 35339 (time 1709788388.93 total 1463.36 last 25.26) - open/close 35577 (time 1709788398.94 total 1473.37 last 23.78) - open/close 35822 (time 1709788408.95 total 1483.38 last 24.48) - open/close 36065 (time 1709788418.97 total 1493.40 last 24.25) - open/close 36277 (time 1709788428.99 total 1503.42 last 21.16) - open/close 36493 (time 1709788439.00 total 1513.43 last 21.58) - open/close 36722 (time 1709788449.06 total 1523.49 last 22.76) - open/close 36950 (time 1709788459.09 total 1533.52 last 22.74) - open/close 37149 (time 1709788469.09 total 1543.52 last 19.90) - open/close 37372 (time 1709788479.10 total 1553.53 last 22.28) - open/close 37594 (time 1709788489.10 total 1563.53 last 22.19) - open/close 37824 (time 1709788499.13 total 1573.56 last 22.93) - open/close 38060 (time 1709788509.17 total 1583.59 last 23.52) - open/close 38311 (time 1709788519.17 total 1593.60 last 25.08) - open/close 38547 (time 1709788529.18 total 1603.61 last 23.58) - open/close 38790 (time 1709788539.20 total 1613.63 last 24.26) - open/close 39030 (time 1709788549.21 total 1623.64 last 23.98) - open/close 39260 (time 1709788559.24 total 1633.67 last 22.93) - open/close 39502 (time 1709788569.28 total 1643.71 last 24.10) - open/close 39717 (time 1709788579.31 total 1653.74 last 21.43) - open/close 39943 (time 1709788589.33 total 1663.76 last 22.55) - open/close 40000 (time 1709788591.81 total 1666.23 last 23.04) - open/close 40239 (time 1709788601.82 total 1676.25 last 23.85) - open/close 40484 (time 1709788611.83 total 1686.26 last 24.48) - open/close 40739 (time 1709788621.84 total 1696.27 last 25.49) - open/close 40999 (time 1709788631.85 total 1706.28 last 25.96) - open/close 41236 (time 1709788641.88 total 1716.31 last 23.63) - open/close 41472 (time 1709788651.93 total 1726.36 last 23.49) - open/close 41702 (time 1709788661.94 total 1736.37 last 22.98) - open/close 41927 (time 1709788671.96 total 1746.39 last 22.45) - open/close 42134 (time 1709788681.99 total 1756.42 last 20.64) - open/close 42346 (time 1709788692.03 total 1766.46 last 21.13) - open/close 42555 (time 1709788702.06 total 1776.49 last 20.83) - open/close 42778 (time 1709788712.08 total 1786.51 last 22.26) - open/close 43006 (time 1709788722.11 total 1796.54 last 22.72) - open/close 43230 (time 1709788732.12 total 1806.55 last 22.39) - open/close 43444 (time 1709788742.14 total 1816.57 last 21.35) - open/close 43638 (time 1709788752.15 total 1826.58 last 19.39) - open/close 43821 (time 1709788762.20 total 1836.63 last 18.20) - open/close 44010 (time 1709788772.26 total 1846.69 last 18.79) - open/close 44204 (time 1709788782.28 total 1856.71 last 19.37) - open/close 44418 (time 1709788792.32 total 1866.75 last 21.31) - open/close 44634 (time 1709788802.34 total 1876.77 last 21.55) - open/close 44858 (time 1709788812.36 total 1886.79 last 22.36) - open/close 45075 (time 1709788822.36 total 1896.79 last 21.69) - open/close 45289 (time 1709788832.36 total 1906.79 last 21.39) - open/close 45506 (time 1709788842.38 total 1916.81 last 21.66) - open/close 45729 (time 1709788852.38 total 1926.81 last 22.29) - open/close 45940 (time 1709788862.40 total 1936.83 last 21.07) - open/close 46159 (time 1709788872.44 total 1946.87 last 21.80) - open/close 46379 (time 1709788882.50 total 1956.92 last 21.88) - open/close 46591 (time 1709788892.50 total 1966.93 last 21.19) - open/close 46807 (time 1709788902.54 total 1976.97 last 21.51) - open/close 47013 (time 1709788912.55 total 1986.98 last 20.58) - open/close 47215 (time 1709788922.59 total 1997.02 last 20.12) - open/close 47418 (time 1709788932.61 total 2007.04 last 20.27) - open/close 47615 (time 1709788942.61 total 2017.04 last 19.69) - open/close 47806 (time 1709788952.63 total 2027.06 last 19.07) - open/close 48001 (time 1709788962.68 total 2037.11 last 19.41) - open/close 48195 (time 1709788972.70 total 2047.13 last 19.36) - open/close 48397 (time 1709788982.74 total 2057.17 last 20.11) - open/close 48594 (time 1709788992.76 total 2067.19 last 19.66) - open/close 48788 (time 1709789002.79 total 2077.22 last 19.35) - open/close 48979 (time 1709789012.85 total 2087.28 last 18.98) - open/close 49186 (time 1709789022.87 total 2097.30 last 20.67) - open/close 49384 (time 1709789032.87 total 2107.30 last 19.80) - open/close 49573 (time 1709789042.90 total 2117.33 last 18.83) - open/close 49760 (time 1709789052.93 total 2127.36 last 18.64) - open/close 49948 (time 1709789062.95 total 2137.38 last 18.77) total: 50000 open/close in 2140.11 seconds: 23.36 ops/second