== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:42 (1709787162) start mds service on oleg342-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg342-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock oleg342-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1709787197.00 total 10.95 last 17.35) - open/close 451 (time 1709787207.73 total 21.68 last 24.32) - open/close 660 (time 1709787218.16 total 32.11 last 20.04) - open/close 817 (time 1709787228.23 total 42.18 last 15.60) - open/close 993 (time 1709787238.83 total 52.78 last 16.59) - open/close 1139 (time 1709787248.95 total 62.90 last 14.43) - open/close 1259 (time 1709787258.96 total 72.91 last 11.99) - open/close 1473 (time 1709787269.75 total 83.69 last 19.84) - open/close 1615 (time 1709787280.62 total 94.57 last 13.06) - open/close 1761 (time 1709787291.21 total 105.16 last 13.79) - open/close 1876 (time 1709787301.26 total 115.21 last 11.44) - open/close 2094 (time 1709787311.26 total 125.21 last 21.80) - open/close 2288 (time 1709787321.42 total 135.37 last 19.09) - open/close 2497 (time 1709787331.80 total 145.74 last 20.15) - open/close 2645 (time 1709787342.26 total 156.21 last 14.15) - open/close 2775 (time 1709787352.33 total 166.27 last 12.91) - open/close 2897 (time 1709787362.68 total 176.63 last 11.79) - open/close 2995 (time 1709787373.57 total 187.52 last 9.00) - open/close 3141 (time 1709787384.51 total 198.46 last 13.35) - open/close 3273 (time 1709787394.52 total 208.47 last 13.19) - open/close 3381 (time 1709787404.91 total 218.86 last 10.39) - open/close 3601 (time 1709787415.54 total 229.49 last 20.70) - open/close 3784 (time 1709787426.55 total 240.49 last 16.63) - open/close 3976 (time 1709787437.40 total 251.35 last 17.69) - open/close 4198 (time 1709787447.54 total 261.48 last 21.91) - open/close 4385 (time 1709787457.54 total 271.49 last 18.68) - open/close 4597 (time 1709787467.55 total 281.50 last 21.18) - open/close 4751 (time 1709787478.43 total 292.38 last 14.16) - open/close 4835 (time 1709787489.39 total 303.34 last 7.66) - open/close 5013 (time 1709787499.42 total 313.37 last 17.76) - open/close 5151 (time 1709787511.37 total 325.32 last 11.55) - open/close 5302 (time 1709787522.38 total 336.33 last 13.71) - open/close 5424 (time 1709787532.39 total 346.33 last 12.20) - open/close 5554 (time 1709787542.47 total 356.42 last 12.89) - open/close 5786 (time 1709787553.00 total 366.95 last 22.03) - open/close 5963 (time 1709787563.02 total 376.97 last 17.66) - open/close 6079 (time 1709787573.05 total 387.00 last 11.57) - open/close 6191 (time 1709787583.19 total 397.14 last 11.04) - open/close 6346 (time 1709787593.23 total 407.17 last 15.45) - open/close 6488 (time 1709787603.76 total 417.71 last 13.48) - open/close 6643 (time 1709787613.86 total 427.80 last 15.36) - open/close 6804 (time 1709787623.87 total 437.81 last 16.08) - open/close 6913 (time 1709787634.41 total 448.36 last 10.34) - open/close 7018 (time 1709787644.43 total 458.37 last 10.48) - open/close 7133 (time 1709787654.65 total 468.60 last 11.24) - open/close 7217 (time 1709787664.74 total 478.69 last 8.33) - open/close 7376 (time 1709787674.74 total 488.69 last 15.89) - open/close 7466 (time 1709787685.04 total 498.99 last 8.74) - open/close 7679 (time 1709787695.04 total 508.99 last 21.29) - open/close 7811 (time 1709787705.24 total 519.18 last 12.95) - open/close 8005 (time 1709787715.25 total 529.20 last 19.37) - open/close 8143 (time 1709787725.70 total 539.64 last 13.21) - open/close 8282 (time 1709787735.91 total 549.86 last 13.60) - open/close 8462 (time 1709787745.92 total 559.86 last 17.99) - open/close 8578 (time 1709787756.31 total 570.26 last 11.16) - open/close 8723 (time 1709787766.32 total 580.26 last 14.50) - open/close 8920 (time 1709787776.32 total 590.27 last 19.69) - open/close 9055 (time 1709787787.14 total 601.08 last 12.48) - open/close 9259 (time 1709787797.15 total 611.10 last 20.37) - open/close 9446 (time 1709787807.58 total 621.53 last 17.94) - open/close 9740 (time 1709787817.95 total 631.90 last 28.34) - open/close 10000 (time 1709787827.78 total 641.73 last 26.45) - open/close 10558 (time 1709787837.80 total 651.74 last 55.72) - open/close 10891 (time 1709787855.94 total 669.89 last 18.35) - open/close 11372 (time 1709787865.95 total 679.90 last 48.05) - open/close 11831 (time 1709787875.96 total 689.91 last 45.85) - open/close 12286 (time 1709787885.98 total 699.93 last 45.40) - open/close 12633 (time 1709787905.00 total 718.95 last 18.25) - open/close 13072 (time 1709787915.00 total 728.95 last 43.89) - open/close 13504 (time 1709787925.01 total 738.96 last 43.14) - open/close 13913 (time 1709787935.03 total 748.98 last 40.82) - open/close 14331 (time 1709787945.06 total 759.01 last 41.70) - open/close 14747 (time 1709787955.07 total 769.02 last 41.56) - open/close 15115 (time 1709787965.10 total 779.05 last 36.67) - open/close 15530 (time 1709787975.13 total 789.08 last 41.38) - open/close 15957 (time 1709787985.15 total 799.10 last 42.63) - open/close 16388 (time 1709787995.16 total 809.11 last 43.04) - open/close 16793 (time 1709788005.18 total 819.12 last 40.45) - open/close 17178 (time 1709788015.18 total 829.13 last 38.48) - open/close 17543 (time 1709788025.19 total 839.14 last 36.47) - open/close 17909 (time 1709788035.26 total 849.21 last 36.36) - open/close 18247 (time 1709788045.26 total 859.21 last 33.80) - open/close 18589 (time 1709788055.26 total 869.21 last 34.19) - open/close 18940 (time 1709788065.27 total 879.22 last 35.07) - open/close 19290 (time 1709788075.28 total 889.23 last 34.96) - open/close 19735 (time 1709788085.28 total 899.23 last 44.50) - open/close 20000 (time 1709788091.58 total 905.52 last 42.11) - open/close 20421 (time 1709788101.59 total 915.54 last 42.04) - open/close 20860 (time 1709788111.60 total 925.55 last 43.86) - open/close 21285 (time 1709788121.62 total 935.57 last 42.40) - open/close 21721 (time 1709788131.63 total 945.58 last 43.57) - open/close 22156 (time 1709788141.64 total 955.59 last 43.46) - open/close 22548 (time 1709788151.68 total 965.63 last 39.06) - open/close 22949 (time 1709788161.70 total 975.65 last 40.01) - open/close 23344 (time 1709788171.70 total 985.65 last 39.50) - open/close 23726 (time 1709788181.72 total 995.66 last 38.15) - open/close 24117 (time 1709788191.73 total 1005.68 last 39.05) - open/close 24477 (time 1709788201.74 total 1015.68 last 35.98) - open/close 24825 (time 1709788211.74 total 1025.69 last 34.78) - open/close 25195 (time 1709788221.76 total 1035.70 last 36.94) - open/close 25585 (time 1709788231.78 total 1045.73 last 38.89) - open/close 25942 (time 1709788241.80 total 1055.75 last 35.64) - open/close 26277 (time 1709788251.82 total 1065.77 last 33.43) - open/close 26614 (time 1709788261.84 total 1075.78 last 33.64) - open/close 26946 (time 1709788271.86 total 1085.81 last 33.12) - open/close 27269 (time 1709788281.87 total 1095.82 last 32.27) - open/close 27578 (time 1709788291.89 total 1105.83 last 30.85) - open/close 27894 (time 1709788301.91 total 1115.86 last 31.53) - open/close 28187 (time 1709788311.91 total 1125.86 last 29.29) - open/close 28496 (time 1709788321.94 total 1135.89 last 30.81) - open/close 28838 (time 1709788331.95 total 1145.89 last 34.19) - open/close 29168 (time 1709788341.95 total 1155.90 last 32.97) - open/close 29494 (time 1709788351.97 total 1165.92 last 32.56) - open/close 29809 (time 1709788362.01 total 1175.96 last 31.37) - open/close 30000 (time 1709788368.78 total 1182.73 last 28.22) - open/close 30297 (time 1709788378.79 total 1192.74 last 29.67) - open/close 30609 (time 1709788388.79 total 1202.74 last 31.19) - open/close 30921 (time 1709788398.80 total 1212.75 last 31.18) - open/close 31213 (time 1709788408.82 total 1222.77 last 29.14) - open/close 31534 (time 1709788418.84 total 1232.79 last 32.03) - open/close 31843 (time 1709788428.86 total 1242.81 last 30.83) - open/close 32146 (time 1709788438.88 total 1252.82 last 30.26) - open/close 32440 (time 1709788448.89 total 1262.83 last 29.38) - open/close 32739 (time 1709788458.91 total 1272.85 last 29.84) - open/close 33021 (time 1709788468.92 total 1282.87 last 28.15) - open/close 33337 (time 1709788478.93 total 1292.88 last 31.57) - open/close 33609 (time 1709788488.94 total 1302.89 last 27.18) - open/close 33869 (time 1709788498.94 total 1312.89 last 25.99) - open/close 34167 (time 1709788508.95 total 1322.90 last 29.78) - open/close 34469 (time 1709788518.97 total 1332.92 last 30.15) - open/close 34763 (time 1709788529.01 total 1342.96 last 29.27) - open/close 35059 (time 1709788539.02 total 1352.97 last 29.58) - open/close 35358 (time 1709788549.06 total 1363.00 last 29.79) - open/close 35641 (time 1709788559.07 total 1373.02 last 28.26) - open/close 35934 (time 1709788569.07 total 1383.02 last 29.29) - open/close 36242 (time 1709788579.10 total 1393.05 last 30.71) - open/close 36546 (time 1709788589.11 total 1403.06 last 30.39) - open/close 36842 (time 1709788599.13 total 1413.08 last 29.53) - open/close 37139 (time 1709788609.15 total 1423.10 last 29.64) - open/close 37437 (time 1709788619.17 total 1433.12 last 29.75) - open/close 37741 (time 1709788629.19 total 1443.14 last 30.33) - open/close 38020 (time 1709788639.21 total 1453.16 last 27.86) - open/close 38285 (time 1709788649.24 total 1463.18 last 26.42) - open/close 38525 (time 1709788659.25 total 1473.19 last 23.98) - open/close 38766 (time 1709788669.25 total 1483.20 last 24.09) - open/close 39027 (time 1709788679.27 total 1493.22 last 26.05) - open/close 39273 (time 1709788689.28 total 1503.23 last 24.58) - open/close 39523 (time 1709788699.29 total 1513.24 last 24.96) - open/close 39773 (time 1709788709.31 total 1523.26 last 24.95) - open/close 40000 (time 1709788718.80 total 1532.75 last 23.94) - open/close 40277 (time 1709788728.82 total 1542.77 last 27.65) - open/close 40556 (time 1709788738.85 total 1552.80 last 27.81) - open/close 40831 (time 1709788748.86 total 1562.81 last 27.47) - open/close 41089 (time 1709788758.88 total 1572.83 last 25.76) - open/close 41331 (time 1709788768.92 total 1582.86 last 24.11) - open/close 41548 (time 1709788778.93 total 1592.88 last 21.68) - open/close 41778 (time 1709788788.94 total 1602.89 last 22.98) - open/close 42002 (time 1709788798.98 total 1612.92 last 22.32) - open/close 42233 (time 1709788808.98 total 1622.93 last 23.09) - open/close 42474 (time 1709788818.99 total 1632.94 last 24.07) - open/close 42698 (time 1709788829.02 total 1642.97 last 22.33) - open/close 42940 (time 1709788839.04 total 1652.98 last 24.17) - open/close 43168 (time 1709788849.06 total 1663.01 last 22.74) - open/close 43385 (time 1709788859.07 total 1673.02 last 21.67) - open/close 43606 (time 1709788869.08 total 1683.03 last 22.08) - open/close 43825 (time 1709788879.09 total 1693.03 last 21.89) - open/close 44064 (time 1709788889.11 total 1703.06 last 23.85) - open/close 44316 (time 1709788899.13 total 1713.08 last 25.14) - open/close 44552 (time 1709788909.13 total 1723.08 last 23.60) - open/close 44796 (time 1709788919.16 total 1733.10 last 24.34) - open/close 45041 (time 1709788929.19 total 1743.14 last 24.41) - open/close 45276 (time 1709788939.22 total 1753.17 last 23.44) - open/close 45498 (time 1709788949.24 total 1763.18 last 22.16) - open/close 45722 (time 1709788959.24 total 1773.19 last 22.39) - open/close 45958 (time 1709788969.27 total 1783.21 last 23.55) - open/close 46208 (time 1709788979.29 total 1793.23 last 24.95) - open/close 46445 (time 1709788989.29 total 1803.24 last 23.68) - open/close 46669 (time 1709788999.32 total 1813.27 last 22.33) - open/close 46768 (time 1709789014.69 total 1828.64 last 6.44) - open/close 46994 (time 1709789024.72 total 1838.67 last 22.53) - open/close 47212 (time 1709789034.74 total 1848.69 last 21.76) - open/close 47424 (time 1709789044.74 total 1858.69 last 21.19) - open/close 47656 (time 1709789054.75 total 1868.70 last 23.18) - open/close 47888 (time 1709789064.79 total 1878.74 last 23.12) - open/close 48100 (time 1709789074.84 total 1888.79 last 21.10) - open/close 48298 (time 1709789084.87 total 1898.82 last 19.73) - open/close 48502 (time 1709789094.89 total 1908.84 last 20.36) - open/close 48722 (time 1709789104.91 total 1918.86 last 21.95) - open/close 48944 (time 1709789114.92 total 1928.87 last 22.18) - open/close 49158 (time 1709789124.94 total 1938.88 last 21.37) - open/close 49366 (time 1709789134.99 total 1948.94 last 20.69) - open/close 49571 (time 1709789145.03 total 1958.98 last 20.42) - open/close 49786 (time 1709789155.03 total 1968.98 last 21.48) total: 50000 open/close in 1978.55 seconds: 25.27 ops/second - unlinked 0 (time 1709789165 ; total 0 ; last 0) - unlinked 10000 (time 1709789705 ; total 540 ; last 540) - unlinked 20000 (time 1709790145 ; total 980 ; last 440) - unlinked 30000 (time 1709790398 ; total 1233 ; last 253) - unlinked 40000 (time 1709790481 ; total 1316 ; last 83) total: 50000 unlinks in 1353 seconds: 36.954914 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server unloading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2