== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:30 (1709787210) start mds service on oleg439-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg439-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock oleg439-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 161 (time 1709787244.60 total 10.93 last 14.74) - open/close 300 (time 1709787254.62 total 20.94 last 13.88) - open/close 401 (time 1709787264.96 total 31.28 last 9.76) - open/close 490 (time 1709787275.19 total 41.51 last 8.70) - open/close 822 (time 1709787286.09 total 52.41 last 30.45) - open/close 938 (time 1709787296.13 total 62.45 last 11.56) - open/close 1178 (time 1709787306.52 total 72.84 last 23.11) - open/close 1439 (time 1709787316.58 total 82.91 last 25.92) - open/close 1641 (time 1709787326.61 total 92.93 last 20.15) - open/close 1768 (time 1709787337.57 total 103.89 last 11.59) - open/close 1947 (time 1709787348.20 total 114.52 last 16.83) - open/close 2158 (time 1709787358.37 total 124.70 last 20.74) - open/close 2296 (time 1709787368.81 total 135.13 last 13.23) - open/close 2485 (time 1709787379.14 total 145.47 last 18.28) - open/close 2701 (time 1709787389.41 total 155.73 last 21.05) - open/close 2852 (time 1709787400.01 total 166.33 last 14.24) - open/close 2973 (time 1709787410.11 total 176.43 last 11.98) - open/close 3084 (time 1709787420.25 total 186.57 last 10.94) - open/close 3372 (time 1709787430.54 total 196.87 last 27.98) - open/close 3600 (time 1709787440.62 total 206.95 last 22.62) - open/close 3806 (time 1709787451.56 total 217.88 last 18.83) - open/close 3964 (time 1709787462.40 total 228.72 last 14.58) - open/close 4023 (time 1709787472.77 total 239.09 last 5.69) - open/close 4257 (time 1709787483.68 total 250.00 last 21.45) - open/close 4302 (time 1709787493.74 total 260.07 last 4.47) - open/close 4417 (time 1709787504.12 total 270.44 last 11.09) - open/close 4624 (time 1709787514.12 total 280.45 last 20.68) - open/close 4779 (time 1709787524.16 total 290.48 last 15.44) - open/close 4955 (time 1709787534.19 total 300.51 last 17.54) - open/close 5187 (time 1709787544.76 total 311.08 last 21.95) - open/close 5414 (time 1709787554.98 total 321.31 last 22.21) - open/close 5569 (time 1709787565.63 total 331.95 last 14.56) - open/close 5801 (time 1709787576.57 total 342.89 last 21.20) - open/close 6062 (time 1709787586.57 total 352.89 last 26.10) - open/close 6326 (time 1709787596.92 total 363.25 last 25.50) - open/close 6581 (time 1709787607.64 total 373.96 last 23.80) - open/close 6820 (time 1709787618.45 total 384.77 last 22.11) - open/close 7008 (time 1709787628.45 total 394.77 last 18.80) - open/close 7290 (time 1709787638.46 total 404.79 last 28.16) - open/close 7468 (time 1709787648.47 total 414.79 last 17.79) - open/close 7794 (time 1709787658.48 total 424.81 last 32.55) - open/close 8055 (time 1709787668.70 total 435.02 last 25.56) - open/close 8220 (time 1709787679.16 total 445.48 last 15.77) - open/close 8407 (time 1709787689.93 total 456.25 last 17.36) - open/close 8706 (time 1709787700.57 total 466.89 last 28.11) - open/close 8908 (time 1709787710.58 total 476.90 last 20.17) - open/close 8980 (time 1709787723.23 total 489.55 last 5.69) - open/close 9213 (time 1709787733.26 total 499.58 last 23.22) - open/close 9475 (time 1709787744.26 total 510.58 last 23.82) - open/close 9749 (time 1709787754.93 total 521.25 last 25.69) - open/close 10000 (time 1709787761.51 total 527.84 last 38.09) - open/close 10534 (time 1709787771.52 total 537.84 last 53.39) - open/close 11067 (time 1709787781.53 total 547.85 last 53.23) - open/close 11548 (time 1709787791.55 total 557.88 last 47.99) - open/close 12054 (time 1709787801.56 total 567.88 last 50.59) - open/close 12619 (time 1709787811.57 total 577.89 last 56.43) - open/close 13105 (time 1709787821.58 total 587.90 last 48.55) - open/close 13547 (time 1709787831.59 total 597.91 last 44.17) - open/close 14054 (time 1709787841.60 total 607.92 last 50.63) - open/close 14540 (time 1709787851.61 total 617.94 last 48.52) - open/close 15076 (time 1709787861.62 total 627.94 last 53.59) - open/close 15609 (time 1709787871.62 total 637.94 last 53.27) - open/close 16152 (time 1709787881.64 total 647.96 last 54.19) - open/close 16661 (time 1709787891.66 total 657.98 last 50.83) - open/close 17148 (time 1709787901.67 total 667.99 last 48.64) - open/close 17600 (time 1709787911.68 total 678.00 last 45.16) - open/close 18055 (time 1709787921.68 total 688.00 last 45.49) - open/close 18486 (time 1709787931.68 total 698.00 last 43.10) - open/close 18929 (time 1709787941.70 total 708.02 last 44.22) - open/close 19353 (time 1709787951.72 total 718.04 last 42.32) - open/close 19798 (time 1709787961.73 total 728.05 last 44.44) - open/close 20000 (time 1709787966.55 total 732.87 last 41.95) - open/close 20400 (time 1709787976.56 total 742.88 last 39.95) - open/close 20803 (time 1709787986.58 total 752.90 last 40.22) - open/close 21208 (time 1709787996.59 total 762.91 last 40.47) - open/close 21629 (time 1709788006.60 total 772.92 last 42.06) - open/close 22051 (time 1709788016.63 total 782.95 last 42.06) - open/close 22429 (time 1709788026.63 total 792.95 last 37.79) - open/close 22825 (time 1709788036.64 total 802.96 last 39.57) - open/close 23194 (time 1709788046.64 total 812.96 last 36.89) - open/close 23538 (time 1709788056.65 total 822.97 last 34.38) - open/close 23900 (time 1709788066.66 total 832.99 last 36.14) - open/close 24280 (time 1709788076.67 total 843.00 last 37.97) - open/close 24640 (time 1709788086.69 total 853.01 last 35.93) - open/close 25019 (time 1709788096.71 total 863.03 last 37.84) - open/close 25367 (time 1709788106.73 total 873.05 last 34.73) - open/close 25728 (time 1709788116.74 total 883.06 last 36.05) - open/close 26097 (time 1709788126.75 total 893.08 last 36.86) - open/close 26427 (time 1709788136.76 total 903.09 last 32.97) - open/close 26776 (time 1709788146.79 total 913.11 last 34.82) - open/close 27099 (time 1709788156.81 total 923.13 last 32.23) - open/close 27421 (time 1709788166.82 total 933.14 last 32.18) - open/close 27752 (time 1709788176.84 total 943.16 last 33.03) - open/close 28095 (time 1709788186.84 total 953.17 last 34.28) - open/close 28424 (time 1709788196.86 total 963.18 last 32.85) - open/close 28730 (time 1709788206.88 total 973.20 last 30.55) - open/close 29049 (time 1709788216.89 total 983.21 last 31.86) - open/close 29353 (time 1709788226.89 total 993.21 last 30.39) - open/close 29649 (time 1709788236.90 total 1003.22 last 29.59) - open/close 29927 (time 1709788246.91 total 1013.24 last 27.75) - open/close 30000 (time 1709788249.22 total 1015.54 last 31.63) - open/close 30325 (time 1709788259.25 total 1025.57 last 32.42) - open/close 30650 (time 1709788269.25 total 1035.57 last 32.49) - open/close 30943 (time 1709788279.28 total 1045.60 last 29.21) - open/close 31252 (time 1709788289.30 total 1055.62 last 30.83) - open/close 31566 (time 1709788299.33 total 1065.65 last 31.32) - open/close 31871 (time 1709788309.36 total 1075.68 last 30.40) - open/close 32163 (time 1709788319.38 total 1085.70 last 29.15) - open/close 32475 (time 1709788329.39 total 1095.72 last 31.15) - open/close 32770 (time 1709788339.41 total 1105.73 last 29.46) - open/close 33038 (time 1709788349.41 total 1115.74 last 26.78) - open/close 33306 (time 1709788359.42 total 1125.74 last 26.79) - open/close 33578 (time 1709788369.44 total 1135.77 last 27.13) - open/close 33862 (time 1709788379.45 total 1145.78 last 28.38) - open/close 34142 (time 1709788389.48 total 1155.80 last 27.93) - open/close 34427 (time 1709788399.48 total 1165.80 last 28.50) - open/close 34709 (time 1709788409.49 total 1175.81 last 28.18) - open/close 34984 (time 1709788419.50 total 1185.82 last 27.47) - open/close 35265 (time 1709788429.50 total 1195.82 last 28.09) - open/close 35532 (time 1709788439.54 total 1205.86 last 26.59) - open/close 35813 (time 1709788449.56 total 1215.88 last 28.05) - open/close 36090 (time 1709788459.59 total 1225.91 last 27.62) - open/close 36366 (time 1709788469.59 total 1235.91 last 27.59) - open/close 36638 (time 1709788479.60 total 1245.92 last 27.18) - open/close 36902 (time 1709788489.63 total 1255.95 last 26.31) - open/close 37169 (time 1709788499.65 total 1265.97 last 26.66) - open/close 37469 (time 1709788509.66 total 1275.98 last 29.95) - open/close 37743 (time 1709788519.69 total 1286.01 last 27.33) - open/close 37988 (time 1709788529.73 total 1296.05 last 24.41) - open/close 38257 (time 1709788539.76 total 1306.09 last 26.80) - open/close 38527 (time 1709788549.80 total 1316.12 last 26.90) - open/close 38795 (time 1709788559.84 total 1326.16 last 26.70) - open/close 39064 (time 1709788569.87 total 1336.19 last 26.81) - open/close 39331 (time 1709788579.91 total 1346.23 last 26.60) - open/close 39604 (time 1709788589.94 total 1356.27 last 27.21) - open/close 39883 (time 1709788599.97 total 1366.30 last 27.81) - open/close 40000 (time 1709788604.28 total 1370.60 last 27.18) - open/close 40265 (time 1709788614.29 total 1380.61 last 26.48) - open/close 40540 (time 1709788624.32 total 1390.64 last 27.42) - open/close 40822 (time 1709788634.35 total 1400.67 last 28.11) - open/close 41103 (time 1709788644.35 total 1410.67 last 28.09) - open/close 41376 (time 1709788654.37 total 1420.69 last 27.25) - open/close 41642 (time 1709788664.38 total 1430.70 last 26.58) - open/close 41905 (time 1709788674.40 total 1440.72 last 26.25) - open/close 42172 (time 1709788684.42 total 1450.74 last 26.64) - open/close 42448 (time 1709788694.44 total 1460.76 last 27.54) - open/close 42717 (time 1709788704.46 total 1470.78 last 26.86) - open/close 42971 (time 1709788714.46 total 1480.78 last 25.39) - open/close 43216 (time 1709788724.48 total 1490.81 last 24.44) - open/close 43465 (time 1709788734.52 total 1500.85 last 24.80) - open/close 43704 (time 1709788755.17 total 1521.49 last 11.58) - open/close 43955 (time 1709788765.17 total 1531.50 last 25.08) - open/close 44209 (time 1709788775.18 total 1541.50 last 25.40) - open/close 44441 (time 1709788785.19 total 1551.51 last 23.16) - open/close 44681 (time 1709788795.21 total 1561.53 last 23.96) - open/close 44911 (time 1709788805.23 total 1571.55 last 22.95) - open/close 45147 (time 1709788815.27 total 1581.59 last 23.52) - open/close 45383 (time 1709788825.28 total 1591.60 last 23.57) - open/close 45623 (time 1709788835.29 total 1601.61 last 23.99) - open/close 45852 (time 1709788845.29 total 1611.61 last 22.90) - open/close 46080 (time 1709788855.33 total 1621.65 last 22.71) - open/close 46315 (time 1709788865.36 total 1631.68 last 23.42) - open/close 46555 (time 1709788875.39 total 1641.71 last 23.92) - open/close 46786 (time 1709788885.41 total 1651.74 last 23.05) - open/close 47010 (time 1709788895.42 total 1661.74 last 22.39) - open/close 47237 (time 1709788905.43 total 1671.75 last 22.68) - open/close 47468 (time 1709788915.46 total 1681.78 last 23.04) - open/close 47691 (time 1709788925.46 total 1691.79 last 22.28) - open/close 47915 (time 1709788935.48 total 1701.80 last 22.36) - open/close 48143 (time 1709788945.50 total 1711.82 last 22.75) - open/close 48365 (time 1709788955.51 total 1721.83 last 22.18) - open/close 48590 (time 1709788965.54 total 1731.87 last 22.43) - open/close 48810 (time 1709788975.55 total 1741.87 last 21.99) - open/close 49025 (time 1709788985.57 total 1751.89 last 21.47) - open/close 49242 (time 1709788995.57 total 1761.90 last 21.68) - open/close 49465 (time 1709789005.61 total 1771.93 last 22.23) - open/close 49683 (time 1709789015.63 total 1781.95 last 21.74) - open/close 49904 (time 1709789025.64 total 1791.96 last 22.08) total: 50000 open/close in 1796.21 seconds: 27.84 ops/second - unlinked 0 (time 1709789031 ; total 0 ; last 0) - unlinked 10000 (time 1709789546 ; total 515 ; last 515) - unlinked 20000 (time 1709789962 ; total 931 ; last 416) - unlinked 30000 (time 1709790215 ; total 1184 ; last 253) - unlinked 40000 (time 1709790301 ; total 1270 ; last 86) total: 50000 unlinks in 1297 seconds: 38.550503 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server unloading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2