== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:55:19 (1709787319) start mds service on oleg140-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 149 (time 1709787374.37 total 10.14 last 14.69) - open/close 348 (time 1709787385.33 total 21.10 last 18.16) - open/close 483 (time 1709787395.35 total 31.12 last 13.48) - open/close 720 (time 1709787406.30 total 42.08 last 21.63) - open/close 958 (time 1709787416.96 total 52.74 last 22.33) - open/close 1101 (time 1709787426.96 total 62.74 last 14.29) - open/close 1275 (time 1709787437.76 total 73.53 last 16.13) - open/close 1519 (time 1709787447.96 total 83.74 last 23.91) - open/close 1661 (time 1709787458.87 total 94.65 last 13.02) - open/close 1792 (time 1709787469.38 total 105.16 last 12.46) - open/close 1999 (time 1709787479.47 total 115.24 last 20.52) - open/close 2161 (time 1709787490.37 total 126.15 last 14.86) - open/close 2305 (time 1709787501.16 total 136.94 last 13.35) - open/close 2453 (time 1709787512.00 total 147.77 last 13.66) - open/close 2612 (time 1709787522.00 total 157.77 last 15.90) - open/close 2736 (time 1709787532.00 total 167.78 last 12.40) - open/close 2894 (time 1709787542.97 total 178.75 last 14.40) - open/close 3034 (time 1709787553.91 total 189.69 last 12.80) - open/close 3179 (time 1709787564.12 total 199.90 last 14.20) - open/close 3326 (time 1709787574.44 total 210.22 last 14.25) - open/close 3448 (time 1709787584.92 total 220.70 last 11.64) - open/close 3770 (time 1709787595.59 total 231.37 last 30.18) - open/close 3917 (time 1709787605.59 total 241.37 last 14.70) - open/close 4066 (time 1709787615.73 total 251.51 last 14.70) - open/close 4130 (time 1709787625.91 total 261.69 last 6.29) - open/close 4285 (time 1709787636.56 total 272.34 last 14.55) - open/close 4469 (time 1709787647.01 total 282.79 last 17.61) - open/close 4677 (time 1709787657.02 total 292.79 last 20.79) - open/close 4901 (time 1709787667.05 total 302.83 last 22.31) - open/close 5034 (time 1709787677.68 total 313.46 last 12.51) - open/close 5243 (time 1709787688.39 total 324.17 last 19.52) - open/close 5434 (time 1709787698.93 total 334.71 last 18.11) - open/close 5659 (time 1709787708.95 total 344.72 last 22.47) - open/close 5900 (time 1709787718.96 total 354.73 last 24.08) - open/close 6135 (time 1709787729.62 total 365.39 last 22.04) - open/close 6376 (time 1709787739.94 total 375.71 last 23.35) - open/close 6468 (time 1709787749.95 total 385.72 last 9.19) - open/close 6632 (time 1709787760.00 total 395.78 last 16.31) - open/close 6761 (time 1709787772.49 total 408.27 last 10.33) - open/close 6955 (time 1709787782.50 total 418.28 last 19.38) - open/close 7126 (time 1709787793.03 total 428.81 last 16.24) - open/close 7317 (time 1709787803.04 total 438.82 last 19.08) - open/close 7604 (time 1709787813.15 total 448.93 last 28.38) - open/close 7773 (time 1709787823.41 total 459.18 last 16.48) - open/close 8054 (time 1709787833.42 total 469.20 last 28.06) - open/close 8237 (time 1709787843.60 total 479.38 last 17.98) - open/close 8402 (time 1709787854.52 total 490.30 last 15.11) - open/close 8606 (time 1709787866.47 total 502.25 last 17.07) - open/close 8853 (time 1709787876.69 total 512.47 last 24.16) - open/close 9046 (time 1709787888.89 total 524.67 last 15.82) - open/close 9356 (time 1709787899.85 total 535.62 last 28.30) - open/close 9491 (time 1709787909.99 total 545.76 last 13.31) - open/close 9652 (time 1709787921.64 total 557.42 last 13.81) - open/close 9969 (time 1709787931.66 total 567.44 last 31.64) - open/close 10457 (time 1709787941.67 total 577.45 last 48.75) - open/close 10692 (time 1709787957.53 total 593.31 last 14.82) - open/close 11196 (time 1709787967.53 total 603.31 last 50.37) - open/close 11692 (time 1709787977.54 total 613.32 last 49.56) - open/close 12211 (time 1709787987.55 total 623.32 last 51.87) - open/close 12703 (time 1709787997.56 total 633.33 last 49.16) - open/close 13123 (time 1709788007.58 total 643.35 last 41.91) - open/close 13579 (time 1709788017.60 total 653.37 last 45.51) - open/close 14012 (time 1709788027.61 total 663.38 last 43.25) - open/close 14453 (time 1709788037.61 total 673.39 last 44.09) - open/close 14893 (time 1709788047.62 total 683.40 last 43.96) - open/close 15322 (time 1709788057.63 total 693.41 last 42.84) - open/close 15736 (time 1709788067.65 total 703.43 last 41.32) - open/close 16147 (time 1709788077.66 total 713.44 last 41.08) - open/close 16578 (time 1709788087.67 total 723.45 last 43.04) - open/close 16969 (time 1709788097.69 total 733.46 last 39.04) - open/close 17332 (time 1709788107.70 total 743.48 last 36.26) - open/close 17709 (time 1709788117.71 total 753.49 last 37.65) - open/close 18074 (time 1709788127.73 total 763.51 last 36.42) - open/close 18469 (time 1709788137.74 total 773.51 last 39.49) - open/close 18791 (time 1709788147.75 total 783.53 last 32.16) - open/close 19146 (time 1709788157.76 total 793.54 last 35.46) - open/close 19491 (time 1709788167.77 total 803.54 last 34.48) - open/close 19833 (time 1709788177.80 total 813.58 last 34.09) - open/close 20000 (time 1709788182.74 total 818.52 last 33.80) - open/close 20332 (time 1709788192.75 total 828.53 last 33.16) - open/close 20692 (time 1709788202.76 total 838.54 last 35.96) - open/close 21080 (time 1709788212.79 total 848.57 last 38.70) - open/close 21438 (time 1709788222.80 total 858.58 last 35.76) - open/close 21835 (time 1709788232.82 total 868.60 last 39.62) - open/close 22172 (time 1709788242.85 total 878.62 last 33.62) - open/close 22484 (time 1709788252.86 total 888.64 last 31.15) - open/close 22774 (time 1709788262.87 total 898.65 last 28.98) - open/close 23090 (time 1709788272.89 total 908.66 last 31.55) - open/close 23400 (time 1709788282.89 total 918.67 last 30.99) - open/close 23743 (time 1709788292.89 total 928.67 last 34.29) - open/close 24072 (time 1709788302.90 total 938.68 last 32.89) - open/close 24403 (time 1709788312.92 total 948.70 last 33.02) - open/close 24738 (time 1709788322.93 total 958.70 last 33.49) - open/close 25064 (time 1709788332.94 total 968.72 last 32.57) - open/close 25378 (time 1709788342.99 total 978.77 last 31.24) - open/close 25684 (time 1709788353.00 total 988.77 last 30.58) - open/close 25990 (time 1709788363.01 total 998.79 last 30.55) - open/close 26305 (time 1709788373.01 total 1008.79 last 31.50) - open/close 26630 (time 1709788383.02 total 1018.80 last 32.47) - open/close 26939 (time 1709788393.04 total 1028.81 last 30.86) - open/close 27241 (time 1709788403.07 total 1038.85 last 30.09) - open/close 27558 (time 1709788413.09 total 1048.86 last 31.65) - open/close 27836 (time 1709788423.10 total 1058.87 last 27.77) - open/close 28122 (time 1709788433.11 total 1068.89 last 28.55) - open/close 28393 (time 1709788443.13 total 1078.91 last 27.05) - open/close 28664 (time 1709788453.14 total 1088.91 last 27.09) - open/close 28931 (time 1709788463.17 total 1098.95 last 26.61) - open/close 29195 (time 1709788473.19 total 1108.97 last 26.35) - open/close 29469 (time 1709788483.22 total 1118.99 last 27.33) - open/close 29751 (time 1709788493.24 total 1129.01 last 28.15) - open/close 30000 (time 1709788501.93 total 1137.70 last 28.65) - open/close 30289 (time 1709788511.96 total 1147.73 last 28.81) - open/close 30581 (time 1709788521.96 total 1157.74 last 29.19) - open/close 30870 (time 1709788531.98 total 1167.76 last 28.84) - open/close 31164 (time 1709788542.01 total 1177.79 last 29.32) - open/close 31445 (time 1709788552.02 total 1187.80 last 28.07) - open/close 31734 (time 1709788562.06 total 1197.83 last 28.80) - open/close 31998 (time 1709788572.11 total 1207.89 last 26.26) - open/close 32238 (time 1709788582.11 total 1217.89 last 24.00) - open/close 32500 (time 1709788592.12 total 1227.89 last 26.18) - open/close 32765 (time 1709788602.12 total 1237.90 last 26.48) - open/close 33033 (time 1709788612.15 total 1247.93 last 26.73) - open/close 33312 (time 1709788622.15 total 1257.93 last 27.90) - open/close 33608 (time 1709788632.17 total 1267.94 last 29.55) - open/close 33865 (time 1709788642.18 total 1277.95 last 25.68) - open/close 34117 (time 1709788652.19 total 1287.97 last 25.17) - open/close 34378 (time 1709788662.21 total 1297.99 last 26.03) - open/close 34640 (time 1709788672.24 total 1308.01 last 26.14) - open/close 34868 (time 1709788682.25 total 1318.03 last 22.77) - open/close 35118 (time 1709788692.28 total 1328.05 last 24.94) - open/close 35360 (time 1709788702.28 total 1338.06 last 24.18) - open/close 35619 (time 1709788712.29 total 1348.06 last 25.89) - open/close 35876 (time 1709788722.29 total 1358.06 last 25.70) - open/close 36129 (time 1709788732.31 total 1368.09 last 25.24) - open/close 36377 (time 1709788742.31 total 1378.09 last 24.79) - open/close 36598 (time 1709788752.36 total 1388.13 last 22.01) - open/close 36810 (time 1709788762.39 total 1398.17 last 21.13) - open/close 37031 (time 1709788772.39 total 1408.17 last 22.10) - open/close 37257 (time 1709788782.39 total 1418.17 last 22.60) - open/close 37481 (time 1709788792.44 total 1428.21 last 22.30) - open/close 37711 (time 1709788802.46 total 1438.24 last 22.94) - open/close 37951 (time 1709788812.47 total 1448.24 last 23.99) - open/close 38192 (time 1709788822.49 total 1458.26 last 24.05) - open/close 38438 (time 1709788832.49 total 1468.27 last 24.60) - open/close 38678 (time 1709788842.49 total 1478.27 last 24.00) - open/close 38921 (time 1709788852.52 total 1488.29 last 24.24) - open/close 39145 (time 1709788862.57 total 1498.34 last 22.29) - open/close 39376 (time 1709788872.60 total 1508.38 last 23.01) - open/close 39603 (time 1709788882.64 total 1518.42 last 22.62) - open/close 39822 (time 1709788892.65 total 1528.42 last 21.88) - open/close 40000 (time 1709788899.96 total 1535.74 last 24.34) - open/close 40228 (time 1709788910.00 total 1545.77 last 22.72) - open/close 40452 (time 1709788920.02 total 1555.79 last 22.36) - open/close 40678 (time 1709788930.05 total 1565.83 last 22.52) - open/close 40906 (time 1709788940.08 total 1575.86 last 22.72) - open/close 41133 (time 1709788950.12 total 1585.89 last 22.63) - open/close 41351 (time 1709788960.14 total 1595.92 last 21.75) - open/close 41563 (time 1709788970.15 total 1605.93 last 21.18) - open/close 41783 (time 1709788980.17 total 1615.94 last 21.96) - open/close 41996 (time 1709788990.21 total 1625.98 last 21.22) - open/close 42210 (time 1709789000.21 total 1635.99 last 21.39) - open/close 42423 (time 1709789010.22 total 1645.99 last 21.29) - open/close 42643 (time 1709789020.23 total 1656.01 last 21.97) - open/close 42871 (time 1709789030.26 total 1666.03 last 22.74) - open/close 43093 (time 1709789040.29 total 1676.07 last 22.12) - open/close 43306 (time 1709789050.32 total 1686.09 last 21.25) - open/close 43512 (time 1709789060.33 total 1696.11 last 20.57) - open/close 43719 (time 1709789070.37 total 1706.15 last 20.62) - open/close 43918 (time 1709789080.40 total 1716.18 last 19.83) - open/close 44110 (time 1709789090.42 total 1726.20 last 19.16) - open/close 44314 (time 1709789100.46 total 1736.23 last 20.33) - open/close 44513 (time 1709789110.48 total 1746.26 last 19.84) - open/close 44711 (time 1709789120.52 total 1756.30 last 19.73) - open/close 44895 (time 1709789130.52 total 1766.30 last 18.39) - open/close 45081 (time 1709789140.55 total 1776.33 last 18.55) - open/close 45270 (time 1709789150.58 total 1786.36 last 18.84) - open/close 45459 (time 1709789160.60 total 1796.38 last 18.86) - open/close 45649 (time 1709789170.61 total 1806.38 last 18.99) - open/close 45830 (time 1709789180.62 total 1816.39 last 18.08) - open/close 46022 (time 1709789190.66 total 1826.44 last 19.12) - open/close 46216 (time 1709789200.70 total 1836.48 last 19.32) - open/close 46401 (time 1709789210.72 total 1846.50 last 18.47) - open/close 46587 (time 1709789220.72 total 1856.50 last 18.60) - open/close 46779 (time 1709789230.76 total 1866.53 last 19.13) - open/close 46963 (time 1709789240.80 total 1876.58 last 18.31) - open/close 47152 (time 1709789250.83 total 1886.60 last 18.86) - open/close 47345 (time 1709789260.83 total 1896.61 last 19.29) - open/close 47534 (time 1709789270.84 total 1906.62 last 18.89) - open/close 47722 (time 1709789280.86 total 1916.64 last 18.77) - open/close 47903 (time 1709789290.90 total 1926.67 last 18.03) - open/close 48082 (time 1709789300.93 total 1936.71 last 17.83) - open/close 48260 (time 1709789310.99 total 1946.77 last 17.69) - open/close 48431 (time 1709789321.01 total 1956.79 last 17.07) - open/close 48597 (time 1709789331.02 total 1966.80 last 16.58) - open/close 48767 (time 1709789341.05 total 1976.83 last 16.94) - open/close 48950 (time 1709789351.06 total 1986.84 last 18.29) - open/close 49141 (time 1709789361.07 total 1996.84 last 19.08) - open/close 49325 (time 1709789371.07 total 2006.85 last 18.39) - open/close 49508 (time 1709789381.08 total 2016.85 last 18.30) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 2021.73 seconds: 24.53 ops/second - unlinked 0 (time 1709789387 ; total 0 ; last 0) - unlinked 10000 (time 1709789988 ; total 601 ; last 601) - unlinked 20000 (time 1709790406 ; total 1019 ; last 418) - unlinked 30000 (time 1709790619 ; total 1232 ; last 213) - unlinked 40000 (time 1709790709 ; total 1322 ; last 90) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1366 seconds: 36.310394 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2