== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:57:04 (1709787424) start mds service on oleg117-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg117-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock oleg117-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 174 (time 1709787479.79 total 10.91 last 15.95) - open/close 349 (time 1709787489.79 total 20.91 last 17.50) - open/close 531 (time 1709787500.84 total 31.96 last 16.48) - open/close 746 (time 1709787511.47 total 42.59 last 20.21) - open/close 961 (time 1709787521.66 total 52.78 last 21.10) - open/close 1098 (time 1709787532.00 total 63.13 last 13.25) - open/close 1357 (time 1709787542.71 total 73.83 last 24.19) - open/close 1580 (time 1709787552.72 total 83.84 last 22.28) - open/close 1699 (time 1709787562.72 total 93.84 last 11.90) - open/close 1848 (time 1709787572.75 total 103.87 last 14.86) - open/close 2048 (time 1709787583.53 total 114.65 last 18.56) - open/close 2215 (time 1709787594.52 total 125.64 last 15.19) - open/close 2335 (time 1709787604.98 total 136.10 last 11.48) - open/close 2525 (time 1709787615.82 total 146.94 last 17.52) - open/close 2723 (time 1709787626.28 total 157.40 last 18.93) - open/close 2875 (time 1709787637.05 total 168.17 last 14.12) - open/close 3066 (time 1709787647.20 total 178.32 last 18.81) - open/close 3167 (time 1709787657.83 total 188.96 last 9.50) - open/close 3325 (time 1709787668.40 total 199.52 last 14.95) - open/close 3508 (time 1709787678.41 total 209.53 last 18.28) - open/close 3625 (time 1709787688.68 total 219.80 last 11.40) - open/close 3775 (time 1709787699.36 total 230.48 last 14.04) - open/close 3957 (time 1709787710.25 total 241.37 last 16.71) - open/close 4129 (time 1709787720.91 total 252.03 last 16.14) - open/close 4328 (time 1709787730.91 total 262.03 last 19.90) - open/close 4502 (time 1709787740.92 total 272.04 last 17.38) - open/close 4584 (time 1709787751.55 total 282.67 last 7.71) - open/close 4744 (time 1709787761.56 total 292.69 last 15.98) - open/close 4945 (time 1709787771.87 total 302.99 last 19.50) - open/close 5054 (time 1709787781.89 total 313.01 last 10.88) - open/close 5188 (time 1709787791.90 total 323.02 last 13.39) - open/close 5380 (time 1709787802.87 total 333.99 last 17.49) - open/close 5681 (time 1709787813.57 total 344.69 last 28.15) - open/close 5959 (time 1709787824.11 total 355.23 last 26.36) - open/close 6100 (time 1709787834.11 total 365.23 last 14.10) - open/close 6296 (time 1709787844.28 total 375.40 last 19.28) - open/close 6472 (time 1709787855.13 total 386.25 last 16.21) - open/close 6641 (time 1709787865.59 total 396.71 last 16.16) - open/close 6776 (time 1709787876.19 total 407.31 last 12.73) - open/close 6929 (time 1709787886.20 total 417.32 last 15.28) - open/close 7068 (time 1709787898.02 total 429.14 last 11.76) - open/close 7331 (time 1709787908.04 total 439.16 last 26.23) - open/close 7575 (time 1709787918.06 total 449.18 last 24.36) - open/close 7728 (time 1709787928.59 total 459.71 last 14.52) - open/close 7826 (time 1709787941.10 total 472.22 last 7.83) - open/close 8044 (time 1709787951.22 total 482.34 last 21.55) - open/close 8239 (time 1709787961.39 total 492.51 last 19.17) - open/close 8522 (time 1709787971.66 total 502.79 last 27.54) - open/close 8690 (time 1709787981.68 total 512.80 last 16.78) - open/close 8871 (time 1709787992.69 total 523.81 last 16.43) - open/close 9099 (time 1709788002.72 total 533.84 last 22.74) - open/close 9678 (time 1709788012.74 total 543.86 last 57.79) - open/close 10000 (time 1709788018.75 total 549.87 last 53.53) - open/close 10508 (time 1709788028.76 total 559.88 last 50.79) - open/close 11060 (time 1709788038.77 total 569.89 last 55.11) - open/close 11596 (time 1709788048.78 total 579.90 last 53.56) - open/close 12102 (time 1709788058.80 total 589.92 last 50.52) - open/close 12610 (time 1709788068.81 total 599.93 last 50.75) - open/close 13115 (time 1709788078.82 total 609.94 last 50.45) - open/close 13597 (time 1709788088.83 total 619.95 last 48.12) - open/close 14068 (time 1709788098.84 total 629.96 last 47.06) - open/close 14511 (time 1709788108.86 total 639.98 last 44.22) - open/close 14941 (time 1709788118.86 total 649.98 last 43.00) - open/close 15342 (time 1709788128.88 total 660.00 last 40.04) - open/close 15795 (time 1709788138.88 total 670.00 last 45.30) - open/close 16184 (time 1709788148.90 total 680.02 last 38.81) - open/close 16584 (time 1709788158.92 total 690.04 last 39.91) - open/close 16956 (time 1709788168.93 total 700.05 last 37.18) - open/close 17350 (time 1709788178.94 total 710.06 last 39.36) - open/close 17735 (time 1709788188.95 total 720.08 last 38.43) - open/close 17937 (time 1709788205.55 total 736.68 last 12.17) - open/close 18356 (time 1709788215.56 total 746.68 last 41.87) - open/close 18749 (time 1709788225.56 total 756.68 last 39.30) - open/close 19186 (time 1709788235.58 total 766.70 last 43.63) - open/close 19577 (time 1709788245.59 total 776.71 last 39.04) - open/close 19911 (time 1709788255.60 total 786.72 last 33.38) - open/close 20000 (time 1709788258.54 total 789.66 last 30.30) - open/close 20354 (time 1709788268.56 total 799.68 last 35.31) - open/close 20712 (time 1709788278.57 total 809.69 last 35.77) - open/close 21092 (time 1709788288.58 total 819.70 last 37.97) - open/close 21459 (time 1709788298.59 total 829.71 last 36.65) - open/close 21811 (time 1709788308.60 total 839.72 last 35.18) - open/close 22152 (time 1709788318.61 total 849.73 last 34.08) - open/close 22514 (time 1709788328.62 total 859.74 last 36.16) - open/close 22858 (time 1709788338.65 total 869.77 last 34.30) - open/close 23196 (time 1709788348.67 total 879.79 last 33.72) - open/close 23535 (time 1709788358.70 total 889.82 last 33.81) - open/close 23869 (time 1709788368.71 total 899.83 last 33.37) - open/close 24203 (time 1709788378.72 total 909.84 last 33.35) - open/close 24528 (time 1709788388.74 total 919.86 last 32.43) - open/close 24855 (time 1709788398.74 total 929.86 last 32.69) - open/close 25192 (time 1709788408.76 total 939.89 last 33.63) - open/close 25512 (time 1709788418.78 total 949.90 last 31.96) - open/close 25811 (time 1709788428.79 total 959.92 last 29.85) - open/close 26114 (time 1709788438.82 total 969.94 last 30.23) - open/close 26411 (time 1709788448.84 total 979.96 last 29.62) - open/close 26726 (time 1709788458.89 total 990.01 last 31.36) - open/close 27005 (time 1709788468.91 total 1000.03 last 27.84) - open/close 27310 (time 1709788478.92 total 1010.04 last 30.47) - open/close 27600 (time 1709788488.93 total 1020.05 last 28.97) - open/close 27901 (time 1709788498.95 total 1030.07 last 30.04) - open/close 28203 (time 1709788508.97 total 1040.09 last 30.13) - open/close 28515 (time 1709788518.97 total 1050.09 last 31.19) - open/close 28810 (time 1709788528.99 total 1060.11 last 29.44) - open/close 29115 (time 1709788538.99 total 1070.11 last 30.49) - open/close 29417 (time 1709788549.01 total 1080.13 last 30.15) - open/close 29706 (time 1709788559.02 total 1090.14 last 28.86) - open/close 30000 (time 1709788568.90 total 1100.02 last 29.76) - open/close 30275 (time 1709788578.91 total 1110.03 last 27.48) - open/close 30557 (time 1709788588.92 total 1120.04 last 28.17) - open/close 30853 (time 1709788598.95 total 1130.07 last 29.51) - open/close 31148 (time 1709788608.97 total 1140.09 last 29.43) - open/close 31449 (time 1709788618.99 total 1150.11 last 30.03) - open/close 31771 (time 1709788629.02 total 1160.14 last 32.12) - open/close 32063 (time 1709788639.04 total 1170.16 last 29.14) - open/close 32338 (time 1709788649.07 total 1180.19 last 27.41) - open/close 32611 (time 1709788659.10 total 1190.22 last 27.22) - open/close 32875 (time 1709788669.10 total 1200.22 last 26.40) - open/close 33121 (time 1709788679.11 total 1210.23 last 24.57) - open/close 33367 (time 1709788689.13 total 1220.25 last 24.56) - open/close 33616 (time 1709788699.15 total 1230.28 last 24.84) - open/close 33884 (time 1709788709.18 total 1240.31 last 26.72) - open/close 34137 (time 1709788719.20 total 1250.32 last 25.26) - open/close 34397 (time 1709788729.20 total 1260.32 last 26.00) - open/close 34652 (time 1709788739.22 total 1270.34 last 25.45) - open/close 34894 (time 1709788749.22 total 1280.34 last 24.20) - open/close 35135 (time 1709788759.22 total 1290.35 last 24.09) - open/close 35378 (time 1709788769.23 total 1300.35 last 24.30) - open/close 35613 (time 1709788779.24 total 1310.36 last 23.47) - open/close 35620 (time 1709788790.60 total 1321.73 last 0.62) - open/close 35857 (time 1709788800.64 total 1331.76 last 23.61) - open/close 36102 (time 1709788810.68 total 1341.80 last 24.41) - open/close 36355 (time 1709788820.69 total 1351.82 last 25.26) - open/close 36595 (time 1709788830.71 total 1361.83 last 23.97) - open/close 36839 (time 1709788840.72 total 1371.84 last 24.38) - open/close 37083 (time 1709788850.74 total 1381.86 last 24.34) - open/close 37321 (time 1709788860.79 total 1391.92 last 23.67) - open/close 37555 (time 1709788870.80 total 1401.92 last 23.39) - open/close 37799 (time 1709788880.84 total 1411.96 last 24.31) - open/close 38027 (time 1709788890.85 total 1421.97 last 22.77) - open/close 38266 (time 1709788900.87 total 1431.99 last 23.85) - open/close 38492 (time 1709788910.89 total 1442.01 last 22.55) - open/close 38718 (time 1709788920.93 total 1452.05 last 22.52) - open/close 38942 (time 1709788930.94 total 1462.06 last 22.37) - open/close 39168 (time 1709788940.98 total 1472.10 last 22.52) - open/close 39386 (time 1709788950.98 total 1482.10 last 21.79) - open/close 39613 (time 1709788961.00 total 1492.12 last 22.65) - open/close 39838 (time 1709788971.04 total 1502.16 last 22.42) - open/close 40000 (time 1709788977.94 total 1509.07 last 23.45) - open/close 40220 (time 1709788987.96 total 1519.08 last 21.96) - open/close 40441 (time 1709788997.98 total 1529.10 last 22.06) - open/close 40656 (time 1709789008.03 total 1539.15 last 21.39) - open/close 40884 (time 1709789018.05 total 1549.17 last 22.75) - open/close 41125 (time 1709789028.09 total 1559.21 last 24.02) - open/close 41359 (time 1709789038.11 total 1569.23 last 23.35) - open/close 41574 (time 1709789048.14 total 1579.26 last 21.43) - open/close 41788 (time 1709789058.18 total 1589.30 last 21.32) - open/close 41999 (time 1709789068.21 total 1599.33 last 21.04) - open/close 42198 (time 1709789078.25 total 1609.37 last 19.82) - open/close 42393 (time 1709789088.29 total 1619.42 last 19.41) - open/close 42600 (time 1709789098.32 total 1629.44 last 20.64) - open/close 42797 (time 1709789108.34 total 1639.46 last 19.66) - open/close 43000 (time 1709789118.40 total 1649.52 last 20.18) - open/close 43186 (time 1709789128.40 total 1659.52 last 18.60) - open/close 43379 (time 1709789138.43 total 1669.55 last 19.25) - open/close 43575 (time 1709789148.47 total 1679.59 last 19.53) - open/close 43773 (time 1709789158.49 total 1689.61 last 19.75) - open/close 43970 (time 1709789168.51 total 1699.64 last 19.66) - open/close 44162 (time 1709789178.55 total 1709.67 last 19.14) - open/close 44355 (time 1709789188.56 total 1719.68 last 19.27) - open/close 44557 (time 1709789198.58 total 1729.70 last 20.16) - open/close 44754 (time 1709789208.63 total 1739.75 last 19.61) - open/close 44947 (time 1709789218.66 total 1749.78 last 19.23) - open/close 45148 (time 1709789228.68 total 1759.80 last 20.07) - open/close 45344 (time 1709789238.72 total 1769.84 last 19.52) - open/close 45540 (time 1709789248.73 total 1779.85 last 19.58) - open/close 45735 (time 1709789258.75 total 1789.87 last 19.46) - open/close 45935 (time 1709789268.77 total 1799.89 last 19.96) - open/close 46134 (time 1709789278.78 total 1809.90 last 19.89) - open/close 46320 (time 1709789288.79 total 1819.91 last 18.57) - open/close 46510 (time 1709789298.84 total 1829.96 last 18.90) - open/close 46699 (time 1709789308.89 total 1840.01 last 18.81) - open/close 46877 (time 1709789318.93 total 1850.06 last 17.72) - open/close 47055 (time 1709789328.95 total 1860.07 last 17.77) - open/close 47233 (time 1709789338.97 total 1870.09 last 17.76) - open/close 47426 (time 1709789349.05 total 1880.17 last 19.16) - open/close 47624 (time 1709789359.06 total 1890.18 last 19.78) - open/close 47818 (time 1709789369.10 total 1900.22 last 19.32) - open/close 48006 (time 1709789379.14 total 1910.27 last 18.71) - open/close 48196 (time 1709789389.20 total 1920.32 last 18.90) - open/close 48375 (time 1709789399.22 total 1930.34 last 17.87) - open/close 48563 (time 1709789409.28 total 1940.40 last 18.68) - open/close 48750 (time 1709789419.29 total 1950.41 last 18.69) - open/close 48935 (time 1709789429.31 total 1960.43 last 18.46) - open/close 49113 (time 1709789439.35 total 1970.47 last 17.72) - open/close 49298 (time 1709789449.39 total 1980.51 last 18.43) - open/close 49479 (time 1709789459.42 total 1990.55 last 18.04) - open/close 49657 (time 1709789469.47 total 2000.59 last 17.73) - open/close 49829 (time 1709789479.50 total 2010.62 last 17.14) total: 50000 open/close in 2020.17 seconds: 24.75 ops/second - unlinked 0 (time 1709789490 ; total 0 ; last 0) - unlinked 10000 (time 1709790080 ; total 590 ; last 590) - unlinked 20000 (time 1709790473 ; total 983 ; last 393) - unlinked 30000 (time 1709790673 ; total 1183 ; last 200) - unlinked 40000 (time 1709790770 ; total 1280 ; last 97) total: 50000 unlinks in 1323 seconds: 37.792896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2