== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:47 (1709787167) start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 194 (time 1709787202.70 total 10.73 last 18.08) - open/close 331 (time 1709787213.09 total 21.13 last 13.18) - open/close 468 (time 1709787223.29 total 31.33 last 13.43) - open/close 627 (time 1709787233.92 total 41.95 last 14.96) - open/close 666 (time 1709787247.57 total 55.61 last 2.86) - open/close 888 (time 1709787258.15 total 66.18 last 20.99) - open/close 1081 (time 1709787268.97 total 77.01 last 17.83) - open/close 1193 (time 1709787279.60 total 87.64 last 10.53) - open/close 1367 (time 1709787290.39 total 98.42 last 16.13) - open/close 1658 (time 1709787300.40 total 108.43 last 29.06) - open/close 1834 (time 1709787310.92 total 118.96 last 16.73) - open/close 2022 (time 1709787320.93 total 128.96 last 18.79) - open/close 2227 (time 1709787331.26 total 139.30 last 19.83) - open/close 2396 (time 1709787341.35 total 149.39 last 16.75) - open/close 2530 (time 1709787351.88 total 159.91 last 12.73) - open/close 2735 (time 1709787361.88 total 169.92 last 20.49) - open/close 2884 (time 1709787371.89 total 179.93 last 14.88) - open/close 3046 (time 1709787381.89 total 189.93 last 16.20) - open/close 3144 (time 1709787392.14 total 200.18 last 9.56) - open/close 3287 (time 1709787403.02 total 211.06 last 13.14) - open/close 3387 (time 1709787413.04 total 221.07 last 9.99) - open/close 3563 (time 1709787423.04 total 231.08 last 17.59) - open/close 3765 (time 1709787433.96 total 241.99 last 18.50) - open/close 3901 (time 1709787444.61 total 252.64 last 12.77) - open/close 4074 (time 1709787454.92 total 262.95 last 16.78) - open/close 4222 (time 1709787464.93 total 272.96 last 14.79) - open/close 4414 (time 1709787475.59 total 283.63 last 18.01) - open/close 4586 (time 1709787486.49 total 294.52 last 15.78) - open/close 4791 (time 1709787496.51 total 304.54 last 20.46) - open/close 4931 (time 1709787506.53 total 314.56 last 13.97) - open/close 5151 (time 1709787516.88 total 324.92 last 21.25) - open/close 5338 (time 1709787526.90 total 334.94 last 18.66) - open/close 5525 (time 1709787537.43 total 345.47 last 17.75) - open/close 5647 (time 1709787548.03 total 356.07 last 11.51) - open/close 5777 (time 1709787558.34 total 366.38 last 12.61) - open/close 5933 (time 1709787569.28 total 377.32 last 14.26) - open/close 6099 (time 1709787579.30 total 387.33 last 16.58) - open/close 6233 (time 1709787589.31 total 397.34 last 13.38) - open/close 6330 (time 1709787599.50 total 407.54 last 9.52) - open/close 6626 (time 1709787610.10 total 418.14 last 27.92) - open/close 6777 (time 1709787620.49 total 428.52 last 14.53) - open/close 6967 (time 1709787630.65 total 438.68 last 18.71) - open/close 7141 (time 1709787640.66 total 448.70 last 17.37) - open/close 7356 (time 1709787652.86 total 460.89 last 17.63) - open/close 7484 (time 1709787663.43 total 471.47 last 12.10) - open/close 7577 (time 1709787673.43 total 481.47 last 9.30) - open/close 7739 (time 1709787683.45 total 491.49 last 16.17) - open/close 7860 (time 1709787693.46 total 501.49 last 12.09) - open/close 8012 (time 1709787703.79 total 511.83 last 14.71) - open/close 8267 (time 1709787713.80 total 521.83 last 25.48) - open/close 8432 (time 1709787723.82 total 531.86 last 16.46) - open/close 8646 (time 1709787734.16 total 542.19 last 20.70) - open/close 8810 (time 1709787745.72 total 553.75 last 14.19) - open/close 9109 (time 1709787756.71 total 564.74 last 27.20) - open/close 9280 (time 1709787766.72 total 574.75 last 17.08) - open/close 9622 (time 1709787776.72 total 584.76 last 34.19) - open/close 10000 (time 1709787784.27 total 592.31 last 50.05) - open/close 10435 (time 1709787794.29 total 602.32 last 43.43) - open/close 10889 (time 1709787804.30 total 612.33 last 45.36) - open/close 11380 (time 1709787814.30 total 622.34 last 49.06) - open/close 11855 (time 1709787824.33 total 632.36 last 47.40) - open/close 12235 (time 1709787834.34 total 642.38 last 37.94) - open/close 12607 (time 1709787844.36 total 652.40 last 37.12) - open/close 13024 (time 1709787854.37 total 662.40 last 41.69) - open/close 13451 (time 1709787864.38 total 672.42 last 42.63) - open/close 13867 (time 1709787874.40 total 682.43 last 41.53) - open/close 14223 (time 1709787884.45 total 692.48 last 35.44) - open/close 14604 (time 1709787894.46 total 702.50 last 38.04) - open/close 14974 (time 1709787904.48 total 712.51 last 36.95) - open/close 15338 (time 1709787914.48 total 722.52 last 36.37) - open/close 15676 (time 1709787924.51 total 732.55 last 33.70) - open/close 16025 (time 1709787934.53 total 742.56 last 34.85) - open/close 16374 (time 1709787944.54 total 752.57 last 34.85) - open/close 16753 (time 1709787954.54 total 762.58 last 37.90) - open/close 17130 (time 1709787964.56 total 772.59 last 37.63) - open/close 17490 (time 1709787974.56 total 782.60 last 35.98) - open/close 17887 (time 1709787984.56 total 792.60 last 39.70) - open/close 18271 (time 1709787994.58 total 802.61 last 38.35) - open/close 18621 (time 1709788004.60 total 812.63 last 34.93) - open/close 18979 (time 1709788014.60 total 822.63 last 35.79) - open/close 19318 (time 1709788024.61 total 832.64 last 33.86) - open/close 19675 (time 1709788034.63 total 842.67 last 35.62) - open/close 20000 (time 1709788043.63 total 851.66 last 36.13) - open/close 20342 (time 1709788053.64 total 861.68 last 34.14) - open/close 20681 (time 1709788063.65 total 871.68 last 33.88) - open/close 21014 (time 1709788073.66 total 881.70 last 33.26) - open/close 21354 (time 1709788083.67 total 891.71 last 33.96) - open/close 21690 (time 1709788093.70 total 901.73 last 33.52) - open/close 22017 (time 1709788103.71 total 911.75 last 32.65) - open/close 22316 (time 1709788113.73 total 921.76 last 29.85) - open/close 22636 (time 1709788123.75 total 931.79 last 31.93) - open/close 22931 (time 1709788133.77 total 941.80 last 29.46) - open/close 23244 (time 1709788143.77 total 951.80 last 31.29) - open/close 23542 (time 1709788153.78 total 961.82 last 29.76) - open/close 23830 (time 1709788163.81 total 971.84 last 28.73) - open/close 24124 (time 1709788173.84 total 981.87 last 29.30) - open/close 24411 (time 1709788183.87 total 991.91 last 28.61) - open/close 24702 (time 1709788193.89 total 1001.93 last 29.04) - open/close 25011 (time 1709788203.91 total 1011.94 last 30.86) - open/close 25347 (time 1709788213.93 total 1021.97 last 33.51) - open/close 25664 (time 1709788223.95 total 1031.98 last 31.66) - open/close 26000 (time 1709788233.95 total 1041.99 last 33.58) - open/close 26305 (time 1709788243.96 total 1051.99 last 30.48) - open/close 26581 (time 1709788254.00 total 1062.04 last 27.48) - open/close 26837 (time 1709788264.01 total 1072.05 last 25.58) - open/close 27114 (time 1709788274.04 total 1082.08 last 27.61) - open/close 27395 (time 1709788284.08 total 1092.11 last 28.01) - open/close 27706 (time 1709788294.10 total 1102.13 last 31.04) - open/close 27993 (time 1709788304.11 total 1112.14 last 28.66) - open/close 28272 (time 1709788314.13 total 1122.17 last 27.84) - open/close 28563 (time 1709788324.15 total 1132.18 last 29.05) - open/close 28856 (time 1709788334.17 total 1142.21 last 29.24) - open/close 29135 (time 1709788344.19 total 1152.23 last 27.83) - open/close 29420 (time 1709788354.20 total 1162.24 last 28.48) - open/close 29716 (time 1709788364.20 total 1172.24 last 29.60) - open/close 29988 (time 1709788374.22 total 1182.25 last 27.16) - open/close 30280 (time 1709788384.24 total 1192.27 last 29.14) - open/close 30560 (time 1709788394.26 total 1202.29 last 27.95) - open/close 30835 (time 1709788404.30 total 1212.33 last 27.40) - open/close 31121 (time 1709788414.32 total 1222.36 last 28.52) - open/close 31373 (time 1709788424.34 total 1232.37 last 25.16) - open/close 31614 (time 1709788434.36 total 1242.40 last 24.03) - open/close 31862 (time 1709788444.38 total 1252.41 last 24.77) - open/close 32115 (time 1709788454.39 total 1262.43 last 25.26) - open/close 32361 (time 1709788464.42 total 1272.46 last 24.54) - open/close 32601 (time 1709788474.45 total 1282.48 last 23.94) - open/close 32852 (time 1709788484.46 total 1292.49 last 25.07) - open/close 33103 (time 1709788494.47 total 1302.50 last 25.07) - open/close 33348 (time 1709788504.52 total 1312.56 last 24.37) - open/close 33615 (time 1709788514.53 total 1322.56 last 26.68) - open/close 33875 (time 1709788524.56 total 1332.59 last 25.92) - open/close 34138 (time 1709788534.57 total 1342.60 last 26.28) - open/close 34404 (time 1709788544.57 total 1352.60 last 26.60) - open/close 34660 (time 1709788554.60 total 1362.64 last 25.51) - open/close 34926 (time 1709788564.64 total 1372.67 last 26.51) - open/close 35187 (time 1709788574.67 total 1382.70 last 26.01) - open/close 35415 (time 1709788584.69 total 1392.72 last 22.77) - open/close 35672 (time 1709788594.70 total 1402.74 last 25.66) - open/close 35925 (time 1709788604.73 total 1412.77 last 25.23) - open/close 36183 (time 1709788614.75 total 1422.79 last 25.74) - open/close 36456 (time 1709788624.76 total 1432.80 last 27.27) - open/close 36725 (time 1709788634.77 total 1442.80 last 26.89) - open/close 36970 (time 1709788644.78 total 1452.82 last 24.46) - open/close 37216 (time 1709788654.82 total 1462.85 last 24.52) - open/close 37460 (time 1709788664.83 total 1472.86 last 24.38) - open/close 37690 (time 1709788674.84 total 1482.88 last 22.96) - open/close 37909 (time 1709788684.88 total 1492.91 last 21.83) - open/close 38128 (time 1709788694.93 total 1502.96 last 21.78) - open/close 38362 (time 1709788704.96 total 1512.99 last 23.34) - open/close 38600 (time 1709788714.97 total 1523.00 last 23.76) - open/close 38844 (time 1709788725.01 total 1533.05 last 24.30) - open/close 39078 (time 1709788735.04 total 1543.07 last 23.34) - open/close 39302 (time 1709788745.07 total 1553.11 last 22.33) - open/close 39501 (time 1709788755.11 total 1563.14 last 19.83) - open/close 39702 (time 1709788765.13 total 1573.16 last 20.05) - open/close 39905 (time 1709788775.16 total 1583.20 last 20.24) - open/close 40000 (time 1709788779.35 total 1587.39 last 22.65) - open/close 40221 (time 1709788789.38 total 1597.42 last 22.03) - open/close 40448 (time 1709788799.43 total 1607.47 last 22.59) - open/close 40675 (time 1709788809.44 total 1617.47 last 22.69) - open/close 40907 (time 1709788819.45 total 1627.48 last 23.18) - open/close 41128 (time 1709788829.49 total 1637.53 last 22.00) - open/close 41354 (time 1709788839.50 total 1647.53 last 22.59) - open/close 41586 (time 1709788849.51 total 1657.54 last 23.18) - open/close 41810 (time 1709788859.54 total 1667.57 last 22.33) - open/close 42024 (time 1709788869.56 total 1677.60 last 21.35) - open/close 42249 (time 1709788879.58 total 1687.61 last 22.47) - open/close 42460 (time 1709788889.59 total 1697.63 last 21.06) - open/close 42688 (time 1709788899.61 total 1707.64 last 22.77) - open/close 42900 (time 1709788909.64 total 1717.68 last 21.12) - open/close 43110 (time 1709788919.67 total 1727.70 last 20.95) - open/close 43317 (time 1709788929.68 total 1737.71 last 20.67) - open/close 43534 (time 1709788939.68 total 1747.72 last 21.70) - open/close 43738 (time 1709788949.71 total 1757.75 last 20.33) - open/close 43939 (time 1709788959.72 total 1767.76 last 20.08) - open/close 44139 (time 1709788969.76 total 1777.80 last 19.92) - open/close 44349 (time 1709788979.77 total 1787.80 last 21.00) - open/close 44550 (time 1709788989.78 total 1797.81 last 20.08) - open/close 44753 (time 1709788999.81 total 1807.85 last 20.23) - open/close 44952 (time 1709789009.81 total 1817.85 last 19.90) - open/close 45161 (time 1709789019.85 total 1827.88 last 20.82) - open/close 45373 (time 1709789029.89 total 1837.92 last 21.11) - open/close 45578 (time 1709789039.90 total 1847.94 last 20.48) - open/close 45771 (time 1709789049.94 total 1857.98 last 19.22) - open/close 45959 (time 1709789059.96 total 1867.99 last 18.78) - open/close 46147 (time 1709789069.96 total 1878.00 last 18.78) - open/close 46330 (time 1709789080.01 total 1888.05 last 18.21) - open/close 46502 (time 1709789090.04 total 1898.08 last 17.15) - open/close 46689 (time 1709789100.09 total 1908.12 last 18.62) - open/close 46873 (time 1709789110.09 total 1918.12 last 18.40) - open/close 47052 (time 1709789120.09 total 1928.13 last 17.89) - open/close 47225 (time 1709789130.11 total 1938.14 last 17.27) - open/close 47399 (time 1709789140.16 total 1948.20 last 17.31) - open/close 47576 (time 1709789150.21 total 1958.25 last 17.61) - open/close 47758 (time 1709789160.25 total 1968.29 last 18.13) - open/close 47935 (time 1709789170.30 total 1978.33 last 17.62) - open/close 48102 (time 1709789180.34 total 1988.38 last 16.63) - open/close 48272 (time 1709789190.36 total 1998.40 last 16.96) - open/close 48454 (time 1709789200.37 total 2008.41 last 18.18) - open/close 48627 (time 1709789210.38 total 2018.42 last 17.29) - open/close 48807 (time 1709789220.43 total 2028.47 last 17.91) - open/close 48993 (time 1709789230.43 total 2038.47 last 18.60) - open/close 49173 (time 1709789240.49 total 2048.52 last 17.90) - open/close 49350 (time 1709789250.53 total 2058.57 last 17.62) - open/close 49527 (time 1709789260.57 total 2068.60 last 17.64) - open/close 49702 (time 1709789270.62 total 2078.65 last 17.42) - open/close 49872 (time 1709789280.64 total 2088.68 last 16.96) total: 50000 open/close in 2096.84 seconds: 23.85 ops/second - unlinked 0 (time 1709789290 ; total 0 ; last 0) - unlinked 10000 (time 1709789935 ; total 645 ; last 645) - unlinked 20000 (time 1709790386 ; total 1096 ; last 451) - unlinked 30000 (time 1709790614 ; total 1324 ; last 228) - unlinked 40000 (time 1709790708 ; total 1418 ; last 94) total: 50000 unlinks in 1465 seconds: 34.129692 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server unloading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2