== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:08 (1709787248) start mds service on oleg155-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock oleg155-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 162 (time 1709787281.76 total 10.06 last 16.10) - open/close 356 (time 1709787292.20 total 20.50 last 18.59) - open/close 586 (time 1709787302.20 total 30.50 last 22.99) - open/close 737 (time 1709787312.69 total 40.99 last 14.39) - open/close 955 (time 1709787322.81 total 51.10 last 21.56) - open/close 1226 (time 1709787332.81 total 61.11 last 27.09) - open/close 1397 (time 1709787342.81 total 71.11 last 17.09) - open/close 1549 (time 1709787353.60 total 81.90 last 14.09) - open/close 1778 (time 1709787364.39 total 92.69 last 21.23) - open/close 1936 (time 1709787374.95 total 103.24 last 14.97) - open/close 2200 (time 1709787385.33 total 113.62 last 25.43) - open/close 2354 (time 1709787396.20 total 124.50 last 14.16) - open/close 2567 (time 1709787406.21 total 134.51 last 21.28) - open/close 2682 (time 1709787416.24 total 144.53 last 11.47) - open/close 2848 (time 1709787426.98 total 155.27 last 15.46) - open/close 3058 (time 1709787437.15 total 165.44 last 20.65) - open/close 3207 (time 1709787447.15 total 175.45 last 14.89) - open/close 3416 (time 1709787457.17 total 185.46 last 20.87) - open/close 3518 (time 1709787467.24 total 195.54 last 10.12) - open/close 3734 (time 1709787478.04 total 206.34 last 20.00) - open/close 3940 (time 1709787489.03 total 217.32 last 18.75) - open/close 4054 (time 1709787499.58 total 227.88 last 10.80) - open/close 4208 (time 1709787509.67 total 237.97 last 15.27) - open/close 4394 (time 1709787520.16 total 248.46 last 17.73) - open/close 4554 (time 1709787530.19 total 258.48 last 15.96) - open/close 4847 (time 1709787540.54 total 268.83 last 28.31) - open/close 4957 (time 1709787551.07 total 279.37 last 10.44) - open/close 5196 (time 1709787561.17 total 289.46 last 23.68) - open/close 5376 (time 1709787571.18 total 299.47 last 17.98) - open/close 5523 (time 1709787581.32 total 309.62 last 14.49) - open/close 5665 (time 1709787591.33 total 319.63 last 14.19) - open/close 5924 (time 1709787602.19 total 330.49 last 23.85) - open/close 6099 (time 1709787615.03 total 343.32 last 13.63) - open/close 6308 (time 1709787625.04 total 353.34 last 20.86) - open/close 6599 (time 1709787635.06 total 363.36 last 29.04) - open/close 6757 (time 1709787645.07 total 373.37 last 15.79) - open/close 6904 (time 1709787655.07 total 383.37 last 14.70) - open/close 7014 (time 1709787665.46 total 393.76 last 10.59) - open/close 7185 (time 1709787675.99 total 404.28 last 16.24) - open/close 7417 (time 1709787686.00 total 414.30 last 23.17) - open/close 7597 (time 1709787697.00 total 425.29 last 16.37) - open/close 7776 (time 1709787707.82 total 436.12 last 16.53) - open/close 7961 (time 1709787718.31 total 446.61 last 17.64) - open/close 8162 (time 1709787728.73 total 457.02 last 19.30) - open/close 8336 (time 1709787738.93 total 467.22 last 17.06) - open/close 8577 (time 1709787748.98 total 477.27 last 23.98) - open/close 8815 (time 1709787758.99 total 487.29 last 23.76) - open/close 9026 (time 1709787769.37 total 497.66 last 20.33) - open/close 9189 (time 1709787779.96 total 508.26 last 15.38) - open/close 9309 (time 1709787790.04 total 518.34 last 11.91) - open/close 9459 (time 1709787800.45 total 528.75 last 14.41) - open/close 9769 (time 1709787810.46 total 538.76 last 30.97) - open/close 9947 (time 1709787820.79 total 549.09 last 17.22) - open/close 10000 (time 1709787829.07 total 557.36 last 6.41) - open/close 10276 (time 1709787839.08 total 567.37 last 27.57) - open/close 10706 (time 1709787849.09 total 577.38 last 42.96) - open/close 11172 (time 1709787859.09 total 587.39 last 46.58) - open/close 11613 (time 1709787869.10 total 597.40 last 44.05) - open/close 12066 (time 1709787879.12 total 607.41 last 45.23) - open/close 12472 (time 1709787889.12 total 617.42 last 40.58) - open/close 12888 (time 1709787899.13 total 627.43 last 41.55) - open/close 13283 (time 1709787909.14 total 637.43 last 39.49) - open/close 13678 (time 1709787919.15 total 647.44 last 39.47) - open/close 13944 (time 1709787937.28 total 665.58 last 14.67) - open/close 14319 (time 1709787947.28 total 675.58 last 37.50) - open/close 14804 (time 1709787957.30 total 685.60 last 48.40) - open/close 15202 (time 1709787967.32 total 695.61 last 39.75) - open/close 15610 (time 1709787977.32 total 705.62 last 40.77) - open/close 16065 (time 1709787987.35 total 715.64 last 45.39) - open/close 16499 (time 1709787997.35 total 725.65 last 43.38) - open/close 16895 (time 1709788007.36 total 735.66 last 39.56) - open/close 17292 (time 1709788017.38 total 745.68 last 39.62) - open/close 17652 (time 1709788027.39 total 755.69 last 35.97) - open/close 18056 (time 1709788037.39 total 765.69 last 40.40) - open/close 18439 (time 1709788047.42 total 775.71 last 38.21) - open/close 18830 (time 1709788057.42 total 785.71 last 39.10) - open/close 19193 (time 1709788067.43 total 795.73 last 36.24) - open/close 19584 (time 1709788077.45 total 805.74 last 39.04) - open/close 19979 (time 1709788087.45 total 815.74 last 39.50) - open/close 20350 (time 1709788097.47 total 825.76 last 37.02) - open/close 20708 (time 1709788107.48 total 835.77 last 35.77) - open/close 21048 (time 1709788117.55 total 845.85 last 33.74) - open/close 21378 (time 1709788127.57 total 855.87 last 32.95) - open/close 21751 (time 1709788137.59 total 865.89 last 37.22) - open/close 22059 (time 1709788147.61 total 875.90 last 30.76) - open/close 22373 (time 1709788157.61 total 885.90 last 31.40) - open/close 22693 (time 1709788167.62 total 895.91 last 31.97) - open/close 23008 (time 1709788177.62 total 905.92 last 31.48) - open/close 23302 (time 1709788187.68 total 915.97 last 29.24) - open/close 23594 (time 1709788197.69 total 925.99 last 29.15) - open/close 23937 (time 1709788207.72 total 936.01 last 34.22) - open/close 24265 (time 1709788217.72 total 946.02 last 32.78) - open/close 24589 (time 1709788227.72 total 956.02 last 32.40) - open/close 24945 (time 1709788237.73 total 966.03 last 35.56) - open/close 25278 (time 1709788247.75 total 976.04 last 33.26) - open/close 25586 (time 1709788257.76 total 986.05 last 30.77) - open/close 25896 (time 1709788267.79 total 996.08 last 30.91) - open/close 26196 (time 1709788277.81 total 1006.11 last 29.93) - open/close 26528 (time 1709788287.83 total 1016.12 last 33.14) - open/close 26849 (time 1709788297.85 total 1026.15 last 32.03) - open/close 27151 (time 1709788307.87 total 1036.16 last 30.15) - open/close 27448 (time 1709788317.90 total 1046.19 last 29.61) - open/close 27756 (time 1709788327.90 total 1056.20 last 30.79) - open/close 28060 (time 1709788337.92 total 1066.21 last 30.35) - open/close 28347 (time 1709788347.95 total 1076.24 last 28.62) - open/close 28659 (time 1709788357.98 total 1086.28 last 31.09) - open/close 28957 (time 1709788367.99 total 1096.28 last 29.79) - open/close 29256 (time 1709788377.99 total 1106.28 last 29.89) - open/close 29544 (time 1709788387.99 total 1116.29 last 28.79) - open/close 29827 (time 1709788398.02 total 1126.32 last 28.22) - open/close 30000 (time 1709788403.99 total 1132.29 last 28.97) - open/close 30293 (time 1709788414.02 total 1142.31 last 29.22) - open/close 30568 (time 1709788424.04 total 1152.33 last 27.46) - open/close 30845 (time 1709788434.06 total 1162.36 last 27.62) - open/close 31108 (time 1709788444.09 total 1172.39 last 26.23) - open/close 31375 (time 1709788454.11 total 1182.41 last 26.63) - open/close 31644 (time 1709788464.13 total 1192.43 last 26.85) - open/close 31895 (time 1709788474.16 total 1202.46 last 25.03) - open/close 32147 (time 1709788484.17 total 1212.47 last 25.17) - open/close 32420 (time 1709788494.21 total 1222.50 last 27.21) - open/close 32688 (time 1709788504.22 total 1232.52 last 26.75) - open/close 32964 (time 1709788514.23 total 1242.53 last 27.58) - open/close 33228 (time 1709788524.24 total 1252.54 last 26.38) - open/close 33500 (time 1709788534.27 total 1262.56 last 27.13) - open/close 33764 (time 1709788544.29 total 1272.58 last 26.35) - open/close 34027 (time 1709788554.30 total 1282.59 last 26.27) - open/close 34293 (time 1709788564.31 total 1292.61 last 26.56) - open/close 34557 (time 1709788574.33 total 1302.62 last 26.37) - open/close 34793 (time 1709788584.35 total 1312.65 last 23.53) - open/close 35053 (time 1709788594.38 total 1322.68 last 25.93) - open/close 35311 (time 1709788604.39 total 1332.69 last 25.76) - open/close 35574 (time 1709788614.43 total 1342.73 last 26.20) - open/close 35862 (time 1709788624.45 total 1352.74 last 28.75) - open/close 36141 (time 1709788634.46 total 1362.76 last 27.85) - open/close 36397 (time 1709788644.48 total 1372.78 last 25.55) - open/close 36653 (time 1709788654.52 total 1382.82 last 25.51) - open/close 36900 (time 1709788664.53 total 1392.82 last 24.69) - open/close 37136 (time 1709788674.54 total 1402.83 last 23.57) - open/close 37366 (time 1709788684.55 total 1412.84 last 22.98) - open/close 37603 (time 1709788694.56 total 1422.86 last 23.67) - open/close 37846 (time 1709788704.59 total 1432.89 last 24.23) - open/close 38095 (time 1709788714.61 total 1442.90 last 24.87) - open/close 38338 (time 1709788724.63 total 1452.93 last 24.23) - open/close 38571 (time 1709788734.67 total 1462.97 last 23.21) - open/close 38795 (time 1709788744.69 total 1472.99 last 22.35) - open/close 39008 (time 1709788754.72 total 1483.01 last 21.24) - open/close 39214 (time 1709788764.73 total 1493.02 last 20.58) - open/close 39418 (time 1709788774.73 total 1503.02 last 20.40) - open/close 39636 (time 1709788784.75 total 1513.04 last 21.76) - open/close 39875 (time 1709788794.76 total 1523.05 last 23.87) - open/close 40000 (time 1709788800.50 total 1528.79 last 21.78) - open/close 40230 (time 1709788810.51 total 1538.81 last 22.96) - open/close 40468 (time 1709788820.54 total 1548.84 last 23.74) - open/close 40699 (time 1709788830.55 total 1558.84 last 23.09) - open/close 40931 (time 1709788840.56 total 1568.85 last 23.18) - open/close 41163 (time 1709788850.57 total 1578.87 last 23.17) - open/close 41384 (time 1709788860.61 total 1588.90 last 22.02) - open/close 41594 (time 1709788870.63 total 1598.92 last 20.96) - open/close 41813 (time 1709788880.63 total 1608.93 last 21.89) - open/close 42028 (time 1709788890.66 total 1618.96 last 21.43) - open/close 42257 (time 1709788900.70 total 1629.00 last 22.80) - open/close 42479 (time 1709788910.73 total 1639.03 last 22.14) - open/close 42694 (time 1709788920.75 total 1649.05 last 21.46) - open/close 42914 (time 1709788930.78 total 1659.07 last 21.94) - open/close 43134 (time 1709788940.82 total 1669.12 last 21.91) - open/close 43341 (time 1709788950.85 total 1679.14 last 20.65) - open/close 43550 (time 1709788960.88 total 1689.17 last 20.83) - open/close 43756 (time 1709788970.90 total 1699.20 last 20.55) - open/close 43972 (time 1709788980.95 total 1709.24 last 21.50) - open/close 44178 (time 1709788990.97 total 1719.26 last 20.56) - open/close 44385 (time 1709789000.97 total 1729.27 last 20.69) - open/close 44587 (time 1709789011.00 total 1739.29 last 20.15) - open/close 44803 (time 1709789021.01 total 1749.30 last 21.58) - open/close 45024 (time 1709789031.05 total 1759.34 last 22.01) - open/close 45237 (time 1709789041.06 total 1769.36 last 21.27) - open/close 45440 (time 1709789051.09 total 1779.39 last 20.24) - open/close 45642 (time 1709789061.12 total 1789.41 last 20.14) - open/close 45842 (time 1709789071.14 total 1799.44 last 19.95) - open/close 46033 (time 1709789081.18 total 1809.47 last 19.04) - open/close 46221 (time 1709789091.22 total 1819.52 last 18.71) - open/close 46418 (time 1709789101.26 total 1829.56 last 19.62) - open/close 46611 (time 1709789111.28 total 1839.58 last 19.27) - open/close 46799 (time 1709789121.33 total 1849.62 last 18.71) - open/close 46983 (time 1709789131.35 total 1859.65 last 18.35) - open/close 47162 (time 1709789141.38 total 1869.68 last 17.85) - open/close 47349 (time 1709789151.43 total 1879.72 last 18.62) - open/close 47534 (time 1709789161.44 total 1889.73 last 18.48) - open/close 47722 (time 1709789171.45 total 1899.75 last 18.78) - open/close 47903 (time 1709789181.47 total 1909.77 last 18.06) - open/close 48091 (time 1709789191.52 total 1919.82 last 18.71) - open/close 48283 (time 1709789201.53 total 1929.83 last 19.18) - open/close 48462 (time 1709789211.58 total 1939.88 last 17.81) - open/close 48646 (time 1709789221.62 total 1949.92 last 18.33) - open/close 48839 (time 1709789231.65 total 1959.95 last 19.24) - open/close 49023 (time 1709789241.70 total 1969.99 last 18.32) - open/close 49209 (time 1709789251.71 total 1980.00 last 18.58) - open/close 49398 (time 1709789261.72 total 1990.02 last 18.88) - open/close 49586 (time 1709789271.77 total 2000.07 last 18.70) - open/close 49768 (time 1709789281.78 total 2010.07 last 18.19) - open/close 49944 (time 1709789291.82 total 2020.12 last 17.52) total: 50000 open/close in 2023.23 seconds: 24.71 ops/second - unlinked 0 (time 1709789296 ; total 0 ; last 0) - unlinked 10000 (time 1709789900 ; total 604 ; last 604) - unlinked 20000 (time 1709790347 ; total 1051 ; last 447) - unlinked 30000 (time 1709790583 ; total 1287 ; last 236) - unlinked 40000 (time 1709790676 ; total 1380 ; last 93) total: 50000 unlinks in 1425 seconds: 35.087719 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2