== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:39 (1709786859) start mds service on oleg302-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg302-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock oleg302-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1709786889.62 total 10.05 last 14.33) - open/close 347 (time 1709786900.55 total 20.98 last 18.57) - open/close 707 (time 1709786910.79 total 31.21 last 35.16) - open/close 1051 (time 1709786921.59 total 42.02 last 31.85) - open/close 1260 (time 1709786931.60 total 52.02 last 20.88) - open/close 1473 (time 1709786941.73 total 62.16 last 21.01) - open/close 1586 (time 1709786952.49 total 72.91 last 10.51) - open/close 1774 (time 1709786963.47 total 83.90 last 17.11) - open/close 2064 (time 1709786973.47 total 93.90 last 29.00) - open/close 2229 (time 1709786984.12 total 104.54 last 15.50) - open/close 2447 (time 1709786994.77 total 115.20 last 20.46) - open/close 2639 (time 1709787004.78 total 125.21 last 19.18) - open/close 2861 (time 1709787015.69 total 136.11 last 20.36) - open/close 3090 (time 1709787026.20 total 146.63 last 21.78) - open/close 3355 (time 1709787036.21 total 156.64 last 26.48) - open/close 3539 (time 1709787047.20 total 167.63 last 16.73) - open/close 3760 (time 1709787057.65 total 178.08 last 21.16) - open/close 3955 (time 1709787067.65 total 188.08 last 19.50) - open/close 4106 (time 1709787078.25 total 198.67 last 14.25) - open/close 4248 (time 1709787088.66 total 209.09 last 13.63) - open/close 4472 (time 1709787098.69 total 219.12 last 22.34) - open/close 4716 (time 1709787108.82 total 229.24 last 24.10) - open/close 4863 (time 1709787118.83 total 239.26 last 14.68) - open/close 4962 (time 1709787129.49 total 249.92 last 9.28) - open/close 5121 (time 1709787139.52 total 259.94 last 15.86) - open/close 5231 (time 1709787149.52 total 269.95 last 11.00) - open/close 5440 (time 1709787159.54 total 279.97 last 20.86) - open/close 5517 (time 1709787170.22 total 290.64 last 7.21) - open/close 5637 (time 1709787180.91 total 301.34 last 11.22) - open/close 5843 (time 1709787191.17 total 311.60 last 20.08) - open/close 5970 (time 1709787201.94 total 322.37 last 11.79) - open/close 6173 (time 1709787212.04 total 332.47 last 20.10) - open/close 6305 (time 1709787222.06 total 342.49 last 13.18) - open/close 6536 (time 1709787232.59 total 353.02 last 21.94) - open/close 6686 (time 1709787242.90 total 363.33 last 14.55) - open/close 6838 (time 1709787253.05 total 373.47 last 14.98) - open/close 7009 (time 1709787263.82 total 384.25 last 15.87) - open/close 7146 (time 1709787274.04 total 394.47 last 13.41) - open/close 7292 (time 1709787284.05 total 404.47 last 14.59) - open/close 7529 (time 1709787297.01 total 417.44 last 18.28) - open/close 7679 (time 1709787308.00 total 428.43 last 13.65) - open/close 7786 (time 1709787318.51 total 438.94 last 10.18) - open/close 7930 (time 1709787328.88 total 449.30 last 13.89) - open/close 8146 (time 1709787339.65 total 460.08 last 20.05) - open/close 8326 (time 1709787349.68 total 470.10 last 17.95) - open/close 8470 (time 1709787359.68 total 480.11 last 14.39) - open/close 8721 (time 1709787370.52 total 490.95 last 23.16) - open/close 8839 (time 1709787381.07 total 501.50 last 11.18) - open/close 9040 (time 1709787391.14 total 511.57 last 19.97) - open/close 9194 (time 1709787401.15 total 521.58 last 15.38) - open/close 9378 (time 1709787411.30 total 531.72 last 18.13) - open/close 9499 (time 1709787421.70 total 542.12 last 11.63) - open/close 9617 (time 1709787432.41 total 552.83 last 11.02) - open/close 9950 (time 1709787442.41 total 562.84 last 33.29) - open/close 10444 (time 1709787452.43 total 572.85 last 49.31) - open/close 10896 (time 1709787462.44 total 582.86 last 45.16) - open/close 11268 (time 1709787472.45 total 592.88 last 37.14) - open/close 11680 (time 1709787482.47 total 602.90 last 41.10) - open/close 12096 (time 1709787492.48 total 612.90 last 41.59) - open/close 12465 (time 1709787502.48 total 622.91 last 36.88) - open/close 12884 (time 1709787512.50 total 632.92 last 41.84) - open/close 13274 (time 1709787522.53 total 642.95 last 38.89) - open/close 13640 (time 1709787532.53 total 652.96 last 36.57) - open/close 14023 (time 1709787542.54 total 662.96 last 38.29) - open/close 14395 (time 1709787552.54 total 672.96 last 37.19) - open/close 14730 (time 1709787562.54 total 682.97 last 33.49) - open/close 15042 (time 1709787572.57 total 693.00 last 31.10) - open/close 15354 (time 1709787582.58 total 703.00 last 31.19) - open/close 15698 (time 1709787592.59 total 713.01 last 34.36) - open/close 15973 (time 1709787602.62 total 723.05 last 27.41) - open/close 16282 (time 1709787612.65 total 733.08 last 30.82) - open/close 16610 (time 1709787622.66 total 743.08 last 32.78) - open/close 16919 (time 1709787632.69 total 753.11 last 30.80) - open/close 17238 (time 1709787642.69 total 763.11 last 31.89) - open/close 17554 (time 1709787652.70 total 773.12 last 31.57) - open/close 17864 (time 1709787662.74 total 783.17 last 30.86) - open/close 18156 (time 1709787672.75 total 793.17 last 29.19) - open/close 18416 (time 1709787682.76 total 803.19 last 25.97) - open/close 18690 (time 1709787692.77 total 813.19 last 27.38) - open/close 18957 (time 1709787702.79 total 823.22 last 26.63) - open/close 19229 (time 1709787712.80 total 833.23 last 27.18) - open/close 19514 (time 1709787722.81 total 843.24 last 28.47) - open/close 19805 (time 1709787732.84 total 853.27 last 29.02) - open/close 20000 (time 1709787740.23 total 860.66 last 26.39) - open/close 20262 (time 1709787750.26 total 870.69 last 26.12) - open/close 20522 (time 1709787760.28 total 880.70 last 25.97) - open/close 20793 (time 1709787770.31 total 890.73 last 27.01) - open/close 21064 (time 1709787780.36 total 900.78 last 26.97) - open/close 21330 (time 1709787790.37 total 910.79 last 26.57) - open/close 21629 (time 1709787800.39 total 920.81 last 29.84) - open/close 21930 (time 1709787810.40 total 930.82 last 30.07) - open/close 22209 (time 1709787820.40 total 940.83 last 27.88) - open/close 22502 (time 1709787830.43 total 950.86 last 29.22) - open/close 22809 (time 1709787840.47 total 960.89 last 30.60) - open/close 23092 (time 1709787850.47 total 970.90 last 28.28) - open/close 23366 (time 1709787860.51 total 980.94 last 27.30) - open/close 23647 (time 1709787870.51 total 990.94 last 28.10) - open/close 23924 (time 1709787880.52 total 1000.95 last 27.66) - open/close 24199 (time 1709787890.54 total 1010.96 last 27.46) - open/close 24475 (time 1709787900.57 total 1021.00 last 27.51) - open/close 24771 (time 1709787910.59 total 1031.02 last 29.54) - open/close 25038 (time 1709787920.60 total 1041.02 last 26.68) - open/close 25306 (time 1709787930.63 total 1051.06 last 26.71) - open/close 25578 (time 1709787940.64 total 1061.07 last 27.18) - open/close 25829 (time 1709787950.64 total 1071.07 last 25.09) - open/close 26107 (time 1709787960.65 total 1081.07 last 27.79) - open/close 26372 (time 1709787970.65 total 1091.07 last 26.50) - open/close 26638 (time 1709787980.65 total 1101.08 last 26.59) - open/close 26919 (time 1709787990.66 total 1111.08 last 28.09) - open/close 27207 (time 1709788000.66 total 1121.09 last 28.79) - open/close 27467 (time 1709788010.68 total 1131.11 last 25.95) - open/close 27742 (time 1709788020.71 total 1141.14 last 27.41) - open/close 27976 (time 1709788030.73 total 1151.15 last 23.36) - open/close 28226 (time 1709788040.74 total 1161.17 last 24.97) - open/close 28468 (time 1709788050.78 total 1171.20 last 24.11) - open/close 28721 (time 1709788060.84 total 1181.27 last 25.14) - open/close 28975 (time 1709788070.87 total 1191.29 last 25.34) - open/close 29276 (time 1709788080.88 total 1201.31 last 30.05) - open/close 29584 (time 1709788090.91 total 1211.33 last 30.72) - open/close 29899 (time 1709788100.91 total 1221.34 last 31.48) - open/close 30000 (time 1709788104.13 total 1224.55 last 31.43) - open/close 30299 (time 1709788114.13 total 1234.56 last 29.89) - open/close 30621 (time 1709788124.13 total 1244.56 last 32.20) - open/close 30935 (time 1709788134.16 total 1254.59 last 31.30) - open/close 31263 (time 1709788144.17 total 1264.60 last 32.78) - open/close 31545 (time 1709788154.18 total 1274.61 last 28.17) - open/close 31854 (time 1709788164.19 total 1284.62 last 30.87) - open/close 32158 (time 1709788174.21 total 1294.63 last 30.36) - open/close 32439 (time 1709788184.23 total 1304.66 last 28.03) - open/close 32729 (time 1709788194.26 total 1314.69 last 28.92) - open/close 33010 (time 1709788204.28 total 1324.71 last 28.05) - open/close 33279 (time 1709788214.30 total 1334.73 last 26.84) - open/close 33575 (time 1709788224.31 total 1344.74 last 29.57) - open/close 33869 (time 1709788234.34 total 1354.77 last 29.30) - open/close 34138 (time 1709788244.35 total 1364.78 last 26.88) - open/close 34387 (time 1709788254.39 total 1374.82 last 24.80) - open/close 34666 (time 1709788264.42 total 1384.85 last 27.82) - open/close 34921 (time 1709788274.44 total 1394.86 last 25.46) - open/close 35174 (time 1709788284.44 total 1404.87 last 25.29) - open/close 35412 (time 1709788294.47 total 1414.90 last 23.73) - open/close 35668 (time 1709788304.50 total 1424.92 last 25.53) - open/close 35922 (time 1709788314.52 total 1434.94 last 25.35) - open/close 36167 (time 1709788324.53 total 1444.96 last 24.47) - open/close 36441 (time 1709788334.54 total 1454.97 last 27.36) - open/close 36713 (time 1709788344.57 total 1465.00 last 27.12) - open/close 36980 (time 1709788354.59 total 1475.01 last 26.66) - open/close 37233 (time 1709788364.63 total 1485.06 last 25.19) - open/close 37467 (time 1709788374.65 total 1495.07 last 23.36) - open/close 37709 (time 1709788384.67 total 1505.10 last 24.14) - open/close 37952 (time 1709788394.70 total 1515.13 last 24.22) - open/close 38216 (time 1709788404.71 total 1525.13 last 26.39) - open/close 38458 (time 1709788414.73 total 1535.16 last 24.14) - open/close 38730 (time 1709788424.74 total 1545.17 last 27.18) - open/close 38977 (time 1709788434.74 total 1555.17 last 24.69) - open/close 39242 (time 1709788444.74 total 1565.17 last 26.50) - open/close 39492 (time 1709788454.75 total 1575.17 last 24.99) - open/close 39743 (time 1709788464.78 total 1585.21 last 25.02) - open/close 39992 (time 1709788474.79 total 1595.21 last 24.88) - open/close 40246 (time 1709788484.80 total 1605.23 last 25.37) - open/close 40474 (time 1709788494.83 total 1615.25 last 22.74) - open/close 40723 (time 1709788504.83 total 1625.26 last 24.89) - open/close 40988 (time 1709788514.85 total 1635.27 last 26.46) - open/close 41242 (time 1709788524.85 total 1645.28 last 25.39) - open/close 41493 (time 1709788534.88 total 1655.30 last 25.03) - open/close 41753 (time 1709788544.90 total 1665.32 last 25.95) - open/close 42005 (time 1709788554.93 total 1675.36 last 25.12) - open/close 42242 (time 1709788564.95 total 1685.38 last 23.64) - open/close 42498 (time 1709788574.96 total 1695.39 last 25.57) - open/close 42755 (time 1709788584.98 total 1705.40 last 25.67) - open/close 43001 (time 1709788594.98 total 1715.40 last 24.60) - open/close 43248 (time 1709788604.98 total 1725.41 last 24.68) - open/close 43486 (time 1709788615.02 total 1735.45 last 23.71) - open/close 43733 (time 1709788625.05 total 1745.47 last 24.64) - open/close 43974 (time 1709788635.05 total 1755.48 last 24.09) - open/close 44186 (time 1709788645.06 total 1765.49 last 21.18) - open/close 44376 (time 1709788655.10 total 1775.53 last 18.92) - open/close 44567 (time 1709788665.11 total 1785.54 last 19.08) - open/close 44785 (time 1709788675.12 total 1795.55 last 21.78) - open/close 45006 (time 1709788685.13 total 1805.55 last 22.10) - open/close 45228 (time 1709788695.15 total 1815.57 last 22.15) - open/close 45451 (time 1709788705.15 total 1825.57 last 22.30) - open/close 45670 (time 1709788715.19 total 1835.62 last 21.80) - open/close 45892 (time 1709788725.23 total 1845.66 last 22.11) - open/close 46138 (time 1709788735.25 total 1855.67 last 24.57) - open/close 46385 (time 1709788745.27 total 1865.69 last 24.65) - open/close 46618 (time 1709788755.29 total 1875.72 last 23.24) - open/close 46834 (time 1709788765.30 total 1885.73 last 21.58) - open/close 47036 (time 1709788775.31 total 1895.73 last 20.19) - open/close 47241 (time 1709788785.36 total 1905.78 last 20.40) - open/close 47449 (time 1709788795.36 total 1915.79 last 20.79) - open/close 47647 (time 1709788805.40 total 1925.82 last 19.73) - open/close 47847 (time 1709788815.45 total 1935.88 last 19.90) - open/close 48044 (time 1709788825.47 total 1945.89 last 19.67) - open/close 48255 (time 1709788835.51 total 1955.93 last 21.02) - open/close 48467 (time 1709788845.52 total 1965.94 last 21.18) - open/close 48657 (time 1709788855.55 total 1975.98 last 18.93) - open/close 48845 (time 1709788865.61 total 1986.03 last 18.69) - open/close 49050 (time 1709788875.63 total 1996.05 last 20.46) - open/close 49259 (time 1709788885.63 total 2006.06 last 20.89) - open/close 49490 (time 1709788895.66 total 2016.08 last 23.04) - open/close 49705 (time 1709788905.68 total 2026.10 last 21.46) - open/close 49931 (time 1709788915.69 total 2036.11 last 22.58) total: 50000 open/close in 2039.24 seconds: 24.52 ops/second - unlinked 0 (time 1709788920 ; total 0 ; last 0) - unlinked 10000 (time 1709789454 ; total 534 ; last 534) - unlinked 20000 (time 1709789893 ; total 973 ; last 439) - unlinked 30000 (time 1709790176 ; total 1256 ; last 283) - unlinked 40000 (time 1709790305 ; total 1385 ; last 129) total: 50000 unlinks in 1436 seconds: 34.818943 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server unloading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2