== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:58:13 (1709787493) start mds service on oleg434-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock oleg434-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1709787545.10 total 10.00 last 20.20) - open/close 290 (time 1709787555.30 total 20.20 last 8.63) - open/close 643 (time 1709787565.30 total 30.20 last 35.28) - open/close 904 (time 1709787575.31 total 40.21 last 26.09) - open/close 1153 (time 1709787585.31 total 50.21 last 24.89) - open/close 1393 (time 1709787595.32 total 60.22 last 23.99) - open/close 1667 (time 1709787605.32 total 70.22 last 27.38) - open/close 1878 (time 1709787615.62 total 80.52 last 20.48) - open/close 2151 (time 1709787625.77 total 90.67 last 26.91) - open/close 2416 (time 1709787636.18 total 101.08 last 25.45) - open/close 2708 (time 1709787647.00 total 111.90 last 27.00) - open/close 2907 (time 1709787657.21 total 122.11 last 19.49) - open/close 3172 (time 1709787667.42 total 132.32 last 25.95) - open/close 3439 (time 1709787678.42 total 143.32 last 24.28) - open/close 3768 (time 1709787688.42 total 153.32 last 32.88) - open/close 3980 (time 1709787698.47 total 163.37 last 21.10) - open/close 4170 (time 1709787708.48 total 173.38 last 18.99) - open/close 4406 (time 1709787719.34 total 184.24 last 21.73) - open/close 4673 (time 1709787730.06 total 194.96 last 24.90) - open/close 4913 (time 1709787740.23 total 205.13 last 23.60) - open/close 5263 (time 1709787750.53 total 215.43 last 33.98) - open/close 5393 (time 1709787761.36 total 226.26 last 12.00) - open/close 5536 (time 1709787771.78 total 236.68 last 13.72) - open/close 5765 (time 1709787781.91 total 246.81 last 22.60) - open/close 6000 (time 1709787792.93 total 257.83 last 21.33) - open/close 6170 (time 1709787803.06 total 267.96 last 16.77) - open/close 6444 (time 1709787813.08 total 277.98 last 27.36) - open/close 6573 (time 1709787823.09 total 287.99 last 12.89) - open/close 6815 (time 1709787833.10 total 298.00 last 24.18) - open/close 6955 (time 1709787843.12 total 308.02 last 13.96) - open/close 7114 (time 1709787853.45 total 318.35 last 15.40) - open/close 7271 (time 1709787863.45 total 328.35 last 15.69) - open/close 7525 (time 1709787875.74 total 340.64 last 20.67) - open/close 7866 (time 1709787885.96 total 350.86 last 33.35) - open/close 8082 (time 1709787896.72 total 361.63 last 20.07) - open/close 8433 (time 1709787906.73 total 371.63 last 35.08) - open/close 8788 (time 1709787916.85 total 381.75 last 35.08) - open/close 8925 (time 1709787926.98 total 391.89 last 13.52) - open/close 9095 (time 1709787937.56 total 402.46 last 16.07) - open/close 9441 (time 1709787948.57 total 413.47 last 31.44) - open/close 9567 (time 1709787959.39 total 424.29 last 11.64) - open/close 9781 (time 1709787969.61 total 434.51 last 20.94) - open/close 10000 (time 1709787975.86 total 440.76 last 35.02) - open/close 10616 (time 1709787985.87 total 450.77 last 61.55) - open/close 11303 (time 1709787995.88 total 460.78 last 68.62) - open/close 11995 (time 1709788005.88 total 470.78 last 69.19) - open/close 12646 (time 1709788015.90 total 480.80 last 65.02) - open/close 13269 (time 1709788025.91 total 490.81 last 62.22) - open/close 13865 (time 1709788035.91 total 500.81 last 59.56) - open/close 14420 (time 1709788045.91 total 510.81 last 55.50) - open/close 14960 (time 1709788055.92 total 520.83 last 53.94) - open/close 15501 (time 1709788065.94 total 530.84 last 54.02) - open/close 16040 (time 1709788075.95 total 540.85 last 53.87) - open/close 16550 (time 1709788085.96 total 550.86 last 50.96) - open/close 17078 (time 1709788095.96 total 560.86 last 52.78) - open/close 17591 (time 1709788105.97 total 570.87 last 51.22) - open/close 18093 (time 1709788115.99 total 580.89 last 50.13) - open/close 18590 (time 1709788126.00 total 590.90 last 49.62) - open/close 19020 (time 1709788136.03 total 600.93 last 42.88) - open/close 19482 (time 1709788146.05 total 610.95 last 46.10) - open/close 19912 (time 1709788156.06 total 620.96 last 42.97) - open/close 20000 (time 1709788158.09 total 623.00 last 43.21) - open/close 20392 (time 1709788168.11 total 633.01 last 39.14) - open/close 20830 (time 1709788178.13 total 643.03 last 43.71) - open/close 21265 (time 1709788188.15 total 653.05 last 43.43) - open/close 21679 (time 1709788198.16 total 663.06 last 41.33) - open/close 22083 (time 1709788208.17 total 673.07 last 40.38) - open/close 22490 (time 1709788218.19 total 683.09 last 40.61) - open/close 22887 (time 1709788228.21 total 693.11 last 39.63) - open/close 23272 (time 1709788238.24 total 703.14 last 38.39) - open/close 23646 (time 1709788248.24 total 713.14 last 37.39) - open/close 24053 (time 1709788258.25 total 723.15 last 40.66) - open/close 24451 (time 1709788268.27 total 733.17 last 39.73) - open/close 24829 (time 1709788278.28 total 743.18 last 37.74) - open/close 25229 (time 1709788288.30 total 753.20 last 39.93) - open/close 25608 (time 1709788298.30 total 763.20 last 37.89) - open/close 25979 (time 1709788308.32 total 773.22 last 37.04) - open/close 26332 (time 1709788318.32 total 783.23 last 35.29) - open/close 26622 (time 1709788337.26 total 802.16 last 15.31) - open/close 26961 (time 1709788347.28 total 812.18 last 33.83) - open/close 27298 (time 1709788357.29 total 822.19 last 33.67) - open/close 27635 (time 1709788367.32 total 832.22 last 33.60) - open/close 27971 (time 1709788377.34 total 842.24 last 33.52) - open/close 28310 (time 1709788387.35 total 852.25 last 33.89) - open/close 28646 (time 1709788397.36 total 862.26 last 33.58) - open/close 28980 (time 1709788407.38 total 872.28 last 33.33) - open/close 29309 (time 1709788417.42 total 882.32 last 32.77) - open/close 29636 (time 1709788427.43 total 892.33 last 32.66) - open/close 29968 (time 1709788437.45 total 902.35 last 33.11) - open/close 30293 (time 1709788447.48 total 912.38 last 32.41) - open/close 30612 (time 1709788457.49 total 922.39 last 31.88) - open/close 30936 (time 1709788467.49 total 932.39 last 32.40) - open/close 31262 (time 1709788477.52 total 942.42 last 32.51) - open/close 31583 (time 1709788487.54 total 952.44 last 32.02) - open/close 31905 (time 1709788497.55 total 962.45 last 32.18) - open/close 32246 (time 1709788507.56 total 972.46 last 34.04) - open/close 32563 (time 1709788517.57 total 982.47 last 31.68) - open/close 32857 (time 1709788527.60 total 992.50 last 29.30) - open/close 33167 (time 1709788537.63 total 1002.53 last 30.90) - open/close 33475 (time 1709788547.64 total 1012.54 last 30.76) - open/close 33786 (time 1709788557.65 total 1022.55 last 31.09) - open/close 34091 (time 1709788567.67 total 1032.57 last 30.44) - open/close 34398 (time 1709788577.68 total 1042.58 last 30.66) - open/close 34701 (time 1709788587.68 total 1052.58 last 30.30) - open/close 35017 (time 1709788597.69 total 1062.59 last 31.57) - open/close 35327 (time 1709788607.71 total 1072.62 last 30.93) - open/close 35631 (time 1709788617.72 total 1082.62 last 30.39) - open/close 35948 (time 1709788627.73 total 1092.63 last 31.67) - open/close 36266 (time 1709788637.74 total 1102.64 last 31.77) - open/close 36581 (time 1709788647.75 total 1112.65 last 31.45) - open/close 36891 (time 1709788657.78 total 1122.68 last 30.90) - open/close 37195 (time 1709788667.78 total 1132.68 last 30.40) - open/close 37499 (time 1709788677.81 total 1142.71 last 30.32) - open/close 37813 (time 1709788687.84 total 1152.74 last 31.32) - open/close 38123 (time 1709788697.85 total 1162.75 last 30.96) - open/close 38422 (time 1709788707.87 total 1172.77 last 29.83) - open/close 38698 (time 1709788717.88 total 1182.78 last 27.57) - open/close 38977 (time 1709788727.92 total 1192.82 last 27.80) - open/close 39258 (time 1709788737.93 total 1202.83 last 28.07) - open/close 39541 (time 1709788747.93 total 1212.83 last 28.30) - open/close 39830 (time 1709788757.95 total 1222.86 last 28.82) - open/close 40000 (time 1709788764.10 total 1229.00 last 27.67) - open/close 40288 (time 1709788774.13 total 1239.03 last 28.71) - open/close 40551 (time 1709788784.15 total 1249.06 last 26.23) - open/close 40820 (time 1709788794.16 total 1259.06 last 26.88) - open/close 41079 (time 1709788804.19 total 1269.09 last 25.82) - open/close 41336 (time 1709788814.24 total 1279.14 last 25.59) - open/close 41592 (time 1709788824.26 total 1289.16 last 25.53) - open/close 41856 (time 1709788834.29 total 1299.19 last 26.32) - open/close 42109 (time 1709788844.32 total 1309.22 last 25.23) - open/close 42364 (time 1709788854.35 total 1319.25 last 25.43) - open/close 42625 (time 1709788864.39 total 1329.29 last 25.99) - open/close 42887 (time 1709788874.43 total 1339.33 last 26.10) - open/close 43141 (time 1709788884.45 total 1349.35 last 25.34) - open/close 43387 (time 1709788894.49 total 1359.39 last 24.51) - open/close 43635 (time 1709788904.51 total 1369.41 last 24.74) - open/close 43889 (time 1709788914.55 total 1379.45 last 25.30) - open/close 44127 (time 1709788924.57 total 1389.47 last 23.76) - open/close 44368 (time 1709788934.59 total 1399.49 last 24.04) - open/close 44608 (time 1709788944.61 total 1409.51 last 23.96) - open/close 44847 (time 1709788954.61 total 1419.51 last 23.88) - open/close 45085 (time 1709788964.65 total 1429.55 last 23.73) - open/close 45317 (time 1709788974.66 total 1439.56 last 23.17) - open/close 45549 (time 1709788984.68 total 1449.58 last 23.15) - open/close 45780 (time 1709788994.71 total 1459.61 last 23.04) - open/close 46015 (time 1709789004.75 total 1469.65 last 23.41) - open/close 46241 (time 1709789014.78 total 1479.68 last 22.53) - open/close 46468 (time 1709789024.80 total 1489.70 last 22.65) - open/close 46696 (time 1709789034.82 total 1499.72 last 22.77) - open/close 46937 (time 1709789044.84 total 1509.74 last 24.05) - open/close 47184 (time 1709789054.86 total 1519.76 last 24.64) - open/close 47427 (time 1709789064.88 total 1529.78 last 24.24) - open/close 47674 (time 1709789074.91 total 1539.81 last 24.62) - open/close 47919 (time 1709789084.92 total 1549.82 last 24.48) - open/close 48143 (time 1709789094.95 total 1559.85 last 22.35) - open/close 48366 (time 1709789104.95 total 1569.85 last 22.29) - open/close 48593 (time 1709789114.99 total 1579.89 last 22.61) - open/close 48832 (time 1709789125.03 total 1589.93 last 23.80) - open/close 49071 (time 1709789135.05 total 1599.95 last 23.86) - open/close 49300 (time 1709789145.08 total 1609.98 last 22.83) - open/close 49538 (time 1709789155.09 total 1619.99 last 23.78) - open/close 49777 (time 1709789165.12 total 1630.02 last 23.83) total: 50000 open/close in 1639.13 seconds: 30.50 ops/second - unlinked 0 (time 1709789175 ; total 0 ; last 0) - unlinked 10000 (time 1709789690 ; total 515 ; last 515) - unlinked 20000 (time 1709790077 ; total 902 ; last 387) - unlinked 30000 (time 1709790278 ; total 1103 ; last 201) - unlinked 40000 (time 1709790339 ; total 1164 ; last 61) total: 50000 unlinks in 1185 seconds: 42.194092 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2