== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:50:43 (1786006243) start mds service on oleg427-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 90 (time 1786006344.86 total 10.06 last 8.95) - open/close 257 (time 1786006355.88 total 21.07 last 15.16) - open/close 424 (time 1786006368.83 total 34.03 last 12.89) - open/close 580 (time 1786006379.01 total 44.21 last 15.32) - open/close 665 (time 1786006389.76 total 54.96 last 7.91) - open/close 831 (time 1786006400.08 total 65.28 last 16.08) - open/close 879 (time 1786006410.69 total 75.89 last 4.52) - open/close 982 (time 1786006420.74 total 85.94 last 10.25) - open/close 1096 (time 1786006431.43 total 96.62 last 10.67) - open/close 1240 (time 1786006441.44 total 106.64 last 14.38) - open/close 1369 (time 1786006451.78 total 116.98 last 12.48) - open/close 1448 (time 1786006462.55 total 127.75 last 7.33) - open/close 1500 (time 1786006473.06 total 138.26 last 4.95) - open/close 1705 (time 1786006483.72 total 148.91 last 19.24) - open/close 1912 (time 1786006493.72 total 158.92 last 20.69) - open/close 2029 (time 1786006504.07 total 169.26 last 11.31) - open/close 2148 (time 1786006514.33 total 179.53 last 11.60) - open/close 2292 (time 1786006524.36 total 189.56 last 14.35) - open/close 2374 (time 1786006535.04 total 200.23 last 7.68) - open/close 2511 (time 1786006545.04 total 210.23 last 13.70) - open/close 2640 (time 1786006555.09 total 220.29 last 12.83) - open/close 2730 (time 1786006565.11 total 230.31 last 8.98) - open/close 2846 (time 1786006575.37 total 240.56 last 11.31) - open/close 2927 (time 1786006585.37 total 250.57 last 8.10) - open/close 3024 (time 1786006596.36 total 261.56 last 8.83) - open/close 3128 (time 1786006606.99 total 272.19 last 9.78) - open/close 3191 (time 1786006617.05 total 282.25 last 6.26) - open/close 3300 (time 1786006628.04 total 293.24 last 9.92) - open/close 3405 (time 1786006638.48 total 303.68 last 10.06) - open/close 3487 (time 1786006649.30 total 314.50 last 7.58) - open/close 3575 (time 1786006659.32 total 324.52 last 8.78) - open/close 3686 (time 1786006670.04 total 335.24 last 10.35) - open/close 3843 (time 1786006680.05 total 345.25 last 15.69) - open/close 3956 (time 1786006690.53 total 355.73 last 10.78) - open/close 4055 (time 1786006700.96 total 366.16 last 9.49) - open/close 4198 (time 1786006710.99 total 376.19 last 14.26) - open/close 4272 (time 1786006721.12 total 386.31 last 7.31) - open/close 4367 (time 1786006731.99 total 397.19 last 8.73) - open/close 4467 (time 1786006742.21 total 407.41 last 9.78) - open/close 4562 (time 1786006752.22 total 417.42 last 9.49) - open/close 4713 (time 1786006762.32 total 427.52 last 14.94) - open/close 4839 (time 1786006772.35 total 437.55 last 12.56) - open/close 4901 (time 1786006784.64 total 449.84 last 5.04) - open/close 5000 (time 1786006794.66 total 459.85 last 9.89) - open/close 5146 (time 1786006805.26 total 470.46 last 13.77) - open/close 5279 (time 1786006816.28 total 481.47 last 12.07) - open/close 5372 (time 1786006826.40 total 491.60 last 9.18) - open/close 5477 (time 1786006837.21 total 502.41 last 9.71) - open/close 5586 (time 1786006847.24 total 512.44 last 10.86) - open/close 5701 (time 1786006858.21 total 523.41 last 10.49) - open/close 5836 (time 1786006869.02 total 534.22 last 12.49) - open/close 5944 (time 1786006879.03 total 544.23 last 10.78) - open/close 6112 (time 1786006889.04 total 554.24 last 16.79) - open/close 6214 (time 1786006899.84 total 565.04 last 9.45) - open/close 6275 (time 1786006910.81 total 576.00 last 5.56) - open/close 6379 (time 1786006921.92 total 587.12 last 9.36) - open/close 6539 (time 1786006932.00 total 597.20 last 15.86) - open/close 6644 (time 1786006942.01 total 607.21 last 10.50) - open/close 6798 (time 1786006952.53 total 617.73 last 14.64) - open/close 6867 (time 1786006962.97 total 628.17 last 6.61) - open/close 6980 (time 1786006973.01 total 638.20 last 11.26) - open/close 7147 (time 1786006983.63 total 648.83 last 15.72) - open/close 7276 (time 1786006994.09 total 659.29 last 12.33) - open/close 7438 (time 1786007004.10 total 669.30 last 16.19) - open/close 7517 (time 1786007014.22 total 679.42 last 7.81) - open/close 7658 (time 1786007025.13 total 690.33 last 12.92) - open/close 7775 (time 1786007035.41 total 700.60 last 11.39) - open/close 7940 (time 1786007046.18 total 711.38 last 15.32) - open/close 8080 (time 1786007056.22 total 721.42 last 13.94) - open/close 8197 (time 1786007066.28 total 731.48 last 11.63) - open/close 8288 (time 1786007077.32 total 742.52 last 8.24) - open/close 8383 (time 1786007087.71 total 752.91 last 9.15) - open/close 8541 (time 1786007098.59 total 763.79 last 14.52) - open/close 8688 (time 1786007108.60 total 773.80 last 14.69) - open/close 8809 (time 1786007119.67 total 784.87 last 10.93) - open/close 8897 (time 1786007130.26 total 795.46 last 8.31) - open/close 8996 (time 1786007140.36 total 805.56 last 9.80) - open/close 9168 (time 1786007150.89 total 816.08 last 16.34) - open/close 9285 (time 1786007160.91 total 826.11 last 11.67) - open/close 9378 (time 1786007170.91 total 836.11 last 9.30) - open/close 9497 (time 1786007181.36 total 846.56 last 11.39) - open/close 9602 (time 1786007191.65 total 856.85 last 10.21) - open/close 9664 (time 1786007201.66 total 866.86 last 6.19) - open/close 9811 (time 1786007212.68 total 877.88 last 13.34) - open/close 9939 (time 1786007222.70 total 887.89 last 12.78) - open/close 10000 (time 1786007225.46 total 890.66 last 22.06) - open/close 10131 (time 1786007235.47 total 900.67 last 13.09) - open/close 10300 (time 1786007246.29 total 911.49 last 15.62) - open/close 10402 (time 1786007256.42 total 921.61 last 10.08) - open/close 10503 (time 1786007267.16 total 932.35 last 9.40) - open/close 10674 (time 1786007279.90 total 945.09 last 13.42) - open/close 10784 (time 1786007289.97 total 955.17 last 10.92) - open/close 10910 (time 1786007301.01 total 966.20 last 11.41) - open/close 11018 (time 1786007311.02 total 976.21 last 10.79) - open/close 11110 (time 1786007321.83 total 987.03 last 8.51) - open/close 11305 (time 1786007332.56 total 997.75 last 18.18) - open/close 11413 (time 1786007342.88 total 1008.08 last 10.46) - open/close 11535 (time 1786007353.18 total 1018.38 last 11.84) - open/close 11651 (time 1786007363.20 total 1028.40 last 11.57) - open/close 11757 (time 1786007374.05 total 1039.25 last 9.77) - open/close 11862 (time 1786007384.07 total 1049.27 last 10.48) - open/close 11994 (time 1786007394.10 total 1059.29 last 13.16) - open/close 12136 (time 1786007404.13 total 1069.33 last 14.15) - open/close 12233 (time 1786007414.20 total 1079.39 last 9.64) - open/close 12412 (time 1786007424.77 total 1089.97 last 16.92) - open/close 12534 (time 1786007434.83 total 1100.03 last 12.13) - open/close 12628 (time 1786007444.88 total 1110.08 last 9.35) - open/close 12730 (time 1786007454.89 total 1120.09 last 10.19) - open/close 12847 (time 1786007465.57 total 1130.77 last 10.96) - open/close 12932 (time 1786007475.64 total 1140.84 last 8.44) - open/close 13062 (time 1786007485.65 total 1150.85 last 12.98) - open/close 13152 (time 1786007495.69 total 1160.89 last 8.97) - open/close 13285 (time 1786007505.73 total 1170.93 last 13.25) - open/close 13373 (time 1786007515.75 total 1180.95 last 8.78) - open/close 13488 (time 1786007526.33 total 1191.53 last 10.87) - open/close 13538 (time 1786007536.98 total 1202.18 last 4.70) - open/close 13735 (time 1786007547.85 total 1213.05 last 18.13) - open/close 13818 (time 1786007558.48 total 1223.68 last 7.81) - open/close 13941 (time 1786007569.24 total 1234.44 last 11.43) - open/close 14019 (time 1786007579.99 total 1245.18 last 7.26) - open/close 14074 (time 1786007590.02 total 1255.22 last 5.48) - open/close 14188 (time 1786007600.09 total 1265.28 last 11.32) - open/close 14344 (time 1786007610.47 total 1275.67 last 15.03) - open/close 14439 (time 1786007620.48 total 1285.68 last 9.49) - open/close 14587 (time 1786007631.30 total 1296.50 last 13.68) - open/close 14686 (time 1786007642.26 total 1307.45 last 9.03) - open/close 14785 (time 1786007652.26 total 1317.46 last 9.90) - open/close 14889 (time 1786007662.30 total 1327.49 last 10.36) - open/close 15159 (time 1786007672.34 total 1337.54 last 26.87) - open/close 15607 (time 1786007682.35 total 1347.54 last 44.78) - open/close 16065 (time 1786007692.39 total 1357.59 last 45.60) - open/close 16531 (time 1786007702.41 total 1367.60 last 46.53) - open/close 17017 (time 1786007712.42 total 1377.61 last 48.55) - open/close 17497 (time 1786007722.43 total 1387.62 last 47.95) - open/close 18200 (time 1786007732.43 total 1397.63 last 70.24) - open/close 18744 (time 1786007742.46 total 1407.65 last 54.28) - open/close 19180 (time 1786007752.48 total 1417.68 last 43.51) - open/close 19651 (time 1786007762.49 total 1427.68 last 47.06) - open/close 20000 (time 1786007770.92 total 1436.12 last 41.39) - open/close 20447 (time 1786007780.92 total 1446.12 last 44.69) - open/close 21027 (time 1786007790.98 total 1456.17 last 57.68) - open/close 21509 (time 1786007800.99 total 1466.19 last 48.14) - open/close 22045 (time 1786007811.01 total 1476.21 last 53.47) - open/close 22717 (time 1786007821.02 total 1486.22 last 67.12) - open/close 23185 (time 1786007831.04 total 1496.24 last 46.71) - open/close 23641 (time 1786007841.05 total 1506.25 last 45.56) - open/close 24115 (time 1786007851.05 total 1516.25 last 47.39) - open/close 24647 (time 1786007861.08 total 1526.27 last 53.09) - open/close 25306 (time 1786007871.08 total 1536.28 last 65.87) - open/close 26204 (time 1786007881.08 total 1546.28 last 89.78) - open/close 27135 (time 1786007891.11 total 1556.30 last 92.90) - open/close 27723 (time 1786007901.13 total 1566.33 last 58.66) - open/close 28256 (time 1786007911.14 total 1576.34 last 53.25) - open/close 28758 (time 1786007921.15 total 1586.35 last 50.14) - open/close 29332 (time 1786007931.17 total 1596.37 last 57.28) - open/close 29816 (time 1786007941.19 total 1606.39 last 48.31) - open/close 30000 (time 1786007945.34 total 1610.53 last 44.41) - open/close 30574 (time 1786007955.35 total 1620.54 last 57.34) - open/close 31171 (time 1786007965.35 total 1630.55 last 59.67) - open/close 31733 (time 1786007975.35 total 1640.55 last 56.20) - open/close 32701 (time 1786007985.35 total 1650.55 last 96.77) - open/close 33220 (time 1786007995.36 total 1660.56 last 51.85) - open/close 33764 (time 1786008005.37 total 1670.57 last 54.37) - open/close 34186 (time 1786008015.39 total 1680.59 last 42.12) - open/close 34726 (time 1786008025.40 total 1690.60 last 53.93) - open/close 35300 (time 1786008035.41 total 1700.61 last 57.35) - open/close 35817 (time 1786008045.42 total 1710.62 last 51.66) - open/close 36525 (time 1786008055.43 total 1720.63 last 70.73) - open/close 36983 (time 1786008065.46 total 1730.66 last 45.66) - open/close 37395 (time 1786008075.46 total 1740.66 last 41.18) - open/close 37977 (time 1786008085.48 total 1750.68 last 58.08) - open/close 38429 (time 1786008095.49 total 1760.69 last 45.16) - open/close 39013 (time 1786008105.52 total 1770.71 last 58.27) - open/close 39483 (time 1786008115.52 total 1780.71 last 47.00) - open/close 40000 (time 1786008124.64 total 1789.84 last 56.67) - open/close 40429 (time 1786008134.64 total 1799.84 last 42.89) - open/close 40893 (time 1786008144.66 total 1809.85 last 46.33) - open/close 41243 (time 1786008154.69 total 1819.88 last 34.90) - open/close 41704 (time 1786008164.69 total 1829.89 last 46.07) - open/close 42106 (time 1786008174.71 total 1839.91 last 40.13) - open/close 42610 (time 1786008184.73 total 1849.93 last 50.30) - open/close 43088 (time 1786008194.74 total 1859.94 last 47.75) - open/close 43624 (time 1786008204.76 total 1869.95 last 53.51) - open/close 44315 (time 1786008214.77 total 1879.97 last 68.99) - open/close 44859 (time 1786008224.78 total 1889.98 last 54.36) - open/close 45389 (time 1786008234.79 total 1899.99 last 52.96) - open/close 46253 (time 1786008244.80 total 1910.00 last 86.29) - open/close 46709 (time 1786008254.81 total 1920.00 last 45.58) - open/close 47309 (time 1786008264.83 total 1930.03 last 59.85) - open/close 47765 (time 1786008274.83 total 1940.03 last 45.60) - open/close 48162 (time 1786008284.86 total 1950.06 last 39.58) - open/close 48556 (time 1786008294.87 total 1960.07 last 39.37) - open/close 49008 (time 1786008304.89 total 1970.09 last 45.09) - open/close 49436 (time 1786008314.91 total 1980.10 last 42.75) - open/close 49851 (time 1786008324.91 total 1990.11 last 41.47) total: 50000 open/close in 1993.14 seconds: 25.09 ops/second - unlinked 0 (time 1786008334 ; total 0 ; last 0) - unlinked 10000 (time 1786008523 ; total 189 ; last 189) - unlinked 20000 (time 1786008715 ; total 381 ; last 192) - unlinked 30000 (time 1786008923 ; total 589 ; last 208) - unlinked 40000 (time 1786009126 ; total 792 ; last 203) total: 50000 unlinks in 995 seconds: 50.251255 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2