== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:37:01 (1785393421) start mds service on oleg345-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg345-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 63 (time 1785393549.89 total 10.17 last 6.19) - open/close 125 (time 1785393560.10 total 20.37 last 6.08) - open/close 217 (time 1785393571.12 total 31.39 last 8.35) - open/close 322 (time 1785393581.93 total 42.21 last 9.71) - open/close 417 (time 1785393592.87 total 53.15 last 8.69) - open/close 610 (time 1785393603.31 total 63.59 last 18.48) - open/close 715 (time 1785393614.20 total 74.48 last 9.64) - open/close 867 (time 1785393624.21 total 84.49 last 15.19) - open/close 967 (time 1785393635.23 total 95.51 last 9.07) - open/close 1096 (time 1785393645.24 total 105.52 last 12.89) - open/close 1203 (time 1785393655.76 total 116.04 last 10.17) - open/close 1331 (time 1785393666.67 total 126.94 last 11.74) - open/close 1480 (time 1785393677.46 total 137.74 last 13.80) - open/close 1597 (time 1785393688.09 total 148.37 last 11.00) - open/close 1721 (time 1785393699.12 total 159.40 last 11.25) - open/close 1819 (time 1785393709.14 total 169.42 last 9.78) - open/close 1903 (time 1785393719.83 total 180.10 last 7.86) - open/close 2059 (time 1785393729.85 total 190.13 last 15.56) - open/close 2168 (time 1785393739.88 total 200.16 last 10.87) - open/close 2264 (time 1785393750.96 total 211.23 last 8.67) - open/close 2311 (time 1785393761.33 total 221.61 last 4.53) - open/close 2410 (time 1785393772.23 total 232.50 last 9.09) - open/close 2460 (time 1785393783.14 total 243.41 last 4.58) - open/close 2625 (time 1785393793.88 total 254.16 last 15.36) - open/close 2787 (time 1785393803.89 total 264.17 last 16.18) - open/close 2866 (time 1785393814.89 total 275.17 last 7.18) - open/close 3018 (time 1785393825.41 total 285.69 last 14.45) - open/close 3119 (time 1785393835.42 total 295.70 last 10.09) - open/close 3274 (time 1785393845.97 total 306.25 last 14.69) - open/close 3446 (time 1785393856.66 total 316.94 last 16.10) - open/close 3523 (time 1785393867.19 total 327.47 last 7.31) - open/close 3703 (time 1785393877.20 total 337.48 last 17.98) - open/close 3828 (time 1785393887.47 total 347.75 last 12.17) - open/close 3906 (time 1785393898.07 total 358.35 last 7.36) - open/close 3956 (time 1785393908.10 total 368.38 last 4.98) - open/close 4069 (time 1785393918.30 total 378.58 last 11.08) - open/close 4188 (time 1785393928.54 total 388.82 last 11.62) - open/close 4268 (time 1785393939.17 total 399.45 last 7.53) - open/close 4330 (time 1785393949.34 total 409.62 last 6.10) - open/close 4465 (time 1785393960.37 total 420.64 last 12.25) - open/close 4576 (time 1785393970.97 total 431.25 last 10.47) - open/close 4670 (time 1785393981.27 total 441.55 last 9.12) - open/close 4772 (time 1785393991.53 total 451.81 last 9.95) - open/close 4844 (time 1785394002.22 total 462.50 last 6.73) - open/close 4965 (time 1785394012.28 total 472.55 last 12.03) - open/close 5084 (time 1785394022.31 total 482.58 last 11.86) - open/close 5180 (time 1785394032.86 total 493.14 last 9.10) - open/close 5286 (time 1785394043.16 total 503.44 last 10.29) - open/close 5387 (time 1785394054.18 total 514.46 last 9.17) - open/close 5549 (time 1785394064.20 total 524.48 last 16.16) - open/close 5648 (time 1785394075.03 total 535.31 last 9.14) - open/close 5769 (time 1785394085.93 total 546.20 last 11.11) - open/close 5870 (time 1785394096.30 total 556.58 last 9.73) - open/close 5962 (time 1785394106.61 total 566.89 last 8.92) - open/close 6060 (time 1785394117.35 total 577.63 last 9.13) - open/close 6122 (time 1785394127.65 total 587.93 last 6.02) - open/close 6257 (time 1785394138.37 total 598.64 last 12.60) - open/close 6348 (time 1785394148.57 total 608.85 last 8.91) - open/close 6436 (time 1785394159.40 total 619.68 last 8.13) - open/close 6546 (time 1785394169.41 total 629.69 last 10.99) - open/close 6638 (time 1785394180.07 total 640.35 last 8.63) - open/close 6755 (time 1785394190.18 total 650.45 last 11.58) - open/close 6850 (time 1785394201.09 total 661.37 last 8.70) - open/close 6909 (time 1785394211.89 total 672.17 last 5.46) - open/close 6996 (time 1785394222.74 total 683.02 last 8.02) - open/close 7105 (time 1785394232.77 total 693.05 last 10.87) - open/close 7232 (time 1785394243.58 total 703.86 last 11.74) - open/close 7384 (time 1785394253.63 total 713.91 last 15.13) - open/close 7497 (time 1785394264.26 total 724.54 last 10.63) - open/close 7613 (time 1785394274.29 total 734.57 last 11.56) - open/close 7737 (time 1785394284.29 total 744.57 last 12.40) - open/close 7827 (time 1785394294.93 total 755.21 last 8.46) - open/close 7888 (time 1785394305.52 total 765.79 last 5.76) - open/close 8025 (time 1785394316.17 total 776.45 last 12.86) - open/close 8166 (time 1785394326.22 total 786.50 last 14.03) - open/close 8308 (time 1785394337.27 total 797.55 last 12.85) - open/close 8440 (time 1785394347.27 total 807.55 last 13.20) - open/close 8553 (time 1785394357.75 total 818.03 last 10.79) - open/close 8662 (time 1785394368.51 total 828.79 last 10.13) - open/close 8753 (time 1785394378.93 total 839.21 last 8.73) - open/close 8896 (time 1785394388.93 total 849.21 last 14.30) - open/close 9006 (time 1785394399.10 total 859.38 last 10.82) - open/close 9150 (time 1785394409.84 total 870.12 last 13.41) - open/close 9269 (time 1785394420.46 total 880.73 last 11.21) - open/close 9361 (time 1785394431.40 total 891.68 last 8.40) - open/close 9537 (time 1785394441.44 total 901.72 last 17.53) - open/close 9600 (time 1785394451.46 total 911.74 last 6.29) - open/close 9659 (time 1785394461.51 total 921.79 last 5.87) - open/close 9812 (time 1785394471.58 total 931.86 last 15.20) - open/close 9916 (time 1785394482.49 total 942.76 last 9.54) - open/close 10000 (time 1785394489.91 total 950.19 last 11.31) - open/close 10111 (time 1785394500.27 total 960.55 last 10.71) - open/close 10247 (time 1785394510.31 total 970.59 last 13.55) - open/close 10364 (time 1785394520.55 total 980.82 last 11.43) - open/close 10488 (time 1785394530.75 total 991.02 last 12.16) - open/close 10585 (time 1785394541.22 total 1001.50 last 9.26) - open/close 10724 (time 1785394551.26 total 1011.54 last 13.85) - open/close 10821 (time 1785394562.10 total 1022.38 last 8.95) - open/close 10957 (time 1785394572.10 total 1032.38 last 13.60) - open/close 11070 (time 1785394582.13 total 1042.41 last 11.27) - open/close 11182 (time 1785394592.94 total 1053.21 last 10.36) - open/close 11275 (time 1785394602.97 total 1063.25 last 9.27) - open/close 11370 (time 1785394613.99 total 1074.27 last 8.62) - open/close 11465 (time 1785394624.00 total 1084.28 last 9.49) - open/close 11580 (time 1785394634.01 total 1094.29 last 11.49) - open/close 11711 (time 1785394644.04 total 1104.32 last 13.06) - open/close 11861 (time 1785394654.82 total 1115.10 last 13.92) - open/close 11965 (time 1785394665.01 total 1125.29 last 10.20) - open/close 12064 (time 1785394675.30 total 1135.57 last 9.63) - open/close 12191 (time 1785394686.12 total 1146.39 last 11.74) - open/close 12323 (time 1785394696.17 total 1156.45 last 13.13) - open/close 12457 (time 1785394706.59 total 1166.87 last 12.86) - open/close 12550 (time 1785394717.08 total 1177.36 last 8.87) - open/close 12654 (time 1785394727.21 total 1187.48 last 10.27) - open/close 12827 (time 1785394738.00 total 1198.27 last 16.03) - open/close 12926 (time 1785394749.42 total 1209.70 last 8.67) - open/close 13075 (time 1785394760.55 total 1220.83 last 13.38) - open/close 13238 (time 1785394771.04 total 1231.32 last 15.54) - open/close 13375 (time 1785394781.30 total 1241.58 last 13.35) - open/close 13477 (time 1785394791.72 total 1252.00 last 9.79) - open/close 13637 (time 1785394804.44 total 1264.72 last 12.58) - open/close 13693 (time 1785394814.63 total 1274.90 last 5.50) - open/close 13839 (time 1785394824.64 total 1284.92 last 14.58) - open/close 13940 (time 1785394834.91 total 1295.19 last 9.84) - open/close 13998 (time 1785394845.76 total 1306.03 last 5.35) - open/close 14084 (time 1785394856.10 total 1316.38 last 8.31) - open/close 14273 (time 1785394866.12 total 1326.40 last 18.86) - open/close 14378 (time 1785394876.50 total 1336.78 last 10.11) - open/close 14438 (time 1785394886.51 total 1346.79 last 6.00) - open/close 14489 (time 1785394897.33 total 1357.61 last 4.71) - open/close 14607 (time 1785394907.35 total 1367.63 last 11.78) - open/close 14691 (time 1785394918.18 total 1378.45 last 7.76) - open/close 15092 (time 1785394928.18 total 1388.46 last 40.09) - open/close 15554 (time 1785394938.21 total 1398.49 last 46.05) - open/close 16031 (time 1785394948.22 total 1408.50 last 47.64) - open/close 16465 (time 1785394958.23 total 1418.51 last 43.38) - open/close 16868 (time 1785394968.24 total 1428.51 last 40.27) - open/close 17248 (time 1785394978.24 total 1438.52 last 38.00) - open/close 17827 (time 1785394988.24 total 1448.52 last 57.89) - open/close 18392 (time 1785394998.24 total 1458.52 last 56.47) - open/close 18837 (time 1785395008.25 total 1468.53 last 44.48) - open/close 19397 (time 1785395018.25 total 1478.53 last 55.98) - open/close 19928 (time 1785395028.27 total 1488.55 last 53.00) - open/close 20399 (time 1785395038.27 total 1498.55 last 47.09) - open/close 20805 (time 1785395048.31 total 1508.59 last 40.44) - open/close 21234 (time 1785395058.31 total 1518.59 last 42.89) - open/close 21777 (time 1785395068.32 total 1528.60 last 54.28) - open/close 22346 (time 1785395078.32 total 1538.60 last 56.87) - open/close 22838 (time 1785395088.33 total 1548.61 last 49.17) - open/close 23388 (time 1785395098.34 total 1558.62 last 54.95) - open/close 23770 (time 1785395108.35 total 1568.63 last 38.15) - open/close 24285 (time 1785395118.35 total 1578.63 last 51.48) - open/close 24918 (time 1785395128.38 total 1588.65 last 63.15) - open/close 25475 (time 1785395138.40 total 1598.68 last 55.54) - open/close 26032 (time 1785395148.42 total 1608.70 last 55.61) - open/close 26539 (time 1785395158.45 total 1618.73 last 50.57) - open/close 27045 (time 1785395168.45 total 1628.73 last 50.57) - open/close 27502 (time 1785395178.47 total 1638.75 last 45.61) - open/close 28004 (time 1785395188.47 total 1648.75 last 50.19) - open/close 28426 (time 1785395198.51 total 1658.79 last 42.04) - open/close 28910 (time 1785395208.52 total 1668.80 last 48.37) - open/close 29482 (time 1785395218.54 total 1678.81 last 57.10) - open/close 29986 (time 1785395228.55 total 1688.83 last 50.30) - open/close 30363 (time 1785395238.58 total 1698.86 last 37.62) - open/close 30791 (time 1785395248.61 total 1708.88 last 42.68) - open/close 31210 (time 1785395258.61 total 1718.89 last 41.87) - open/close 31689 (time 1785395268.61 total 1728.89 last 47.89) - open/close 32131 (time 1785395278.63 total 1738.91 last 44.13) - open/close 32627 (time 1785395288.64 total 1748.91 last 49.57) - open/close 33241 (time 1785395298.66 total 1758.94 last 61.24) - open/close 33879 (time 1785395308.67 total 1768.95 last 63.73) - open/close 34408 (time 1785395318.67 total 1778.95 last 52.89) - open/close 35080 (time 1785395328.69 total 1788.97 last 67.09) - open/close 35525 (time 1785395338.69 total 1798.97 last 44.49) - open/close 35996 (time 1785395348.70 total 1808.98 last 47.05) - open/close 36440 (time 1785395358.71 total 1818.98 last 44.39) - open/close 36818 (time 1785395368.73 total 1829.01 last 37.71) - open/close 37185 (time 1785395378.74 total 1839.02 last 36.66) - open/close 37719 (time 1785395388.77 total 1849.05 last 53.24) - open/close 38106 (time 1785395398.77 total 1859.05 last 38.70) - open/close 38519 (time 1785395408.77 total 1869.05 last 41.29) - open/close 38919 (time 1785395418.79 total 1879.07 last 39.93) - open/close 39259 (time 1785395428.81 total 1889.08 last 33.96) - open/close 39645 (time 1785395438.81 total 1899.08 last 38.60) - open/close 40000 (time 1785395444.58 total 1904.86 last 61.45) - open/close 40469 (time 1785395454.58 total 1914.86 last 46.90) - open/close 40902 (time 1785395464.58 total 1924.86 last 43.30) - open/close 41451 (time 1785395474.60 total 1934.88 last 54.79) - open/close 41902 (time 1785395484.62 total 1944.89 last 45.04) - open/close 42387 (time 1785395494.63 total 1954.91 last 48.43) - open/close 42902 (time 1785395504.65 total 1964.93 last 51.40) - open/close 43309 (time 1785395514.66 total 1974.94 last 40.65) - open/close 43807 (time 1785395524.66 total 1984.94 last 49.80) - open/close 44321 (time 1785395534.68 total 1994.96 last 51.34) - open/close 44869 (time 1785395544.69 total 2004.97 last 54.72) - open/close 45301 (time 1785395554.72 total 2015.00 last 43.09) - open/close 45677 (time 1785395564.72 total 2025.00 last 37.58) - open/close 46149 (time 1785395574.73 total 2035.01 last 47.18) - open/close 46554 (time 1785395584.73 total 2045.01 last 40.49) - open/close 46920 (time 1785395594.74 total 2055.02 last 36.57) - open/close 47340 (time 1785395604.77 total 2065.04 last 41.88) - open/close 47712 (time 1785395614.78 total 2075.06 last 37.16) - open/close 48136 (time 1785395624.78 total 2085.06 last 42.39) - open/close 48600 (time 1785395634.80 total 2095.08 last 46.28) - open/close 48969 (time 1785395644.82 total 2105.10 last 36.84) - open/close 49427 (time 1785395654.85 total 2115.12 last 45.68) - open/close 49825 (time 1785395664.86 total 2125.14 last 39.74) total: 50000 open/close in 2126.99 seconds: 23.51 ops/second - unlinked 0 (time 1785395673 ; total 0 ; last 0) - unlinked 10000 (time 1785395886 ; total 213 ; last 213) - unlinked 20000 (time 1785396103 ; total 430 ; last 217) - unlinked 30000 (time 1785396315 ; total 642 ; last 212) - unlinked 40000 (time 1785396537 ; total 864 ; last 222) total: 50000 unlinks in 1076 seconds: 46.468403 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2