== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:31:01 (1773610261) start mds service on oleg458-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions oleg458-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg458-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg458-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg458-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1773610365.25 total 10.02 last 13.87) - open/close 254 (time 1773610375.72 total 20.49 last 10.98) - open/close 368 (time 1773610386.71 total 31.48 last 10.38) - open/close 466 (time 1773610397.41 total 42.18 last 9.16) - open/close 687 (time 1773610408.34 total 53.11 last 20.23) - open/close 870 (time 1773610418.38 total 63.15 last 18.22) - open/close 915 (time 1773610428.90 total 73.66 last 4.28) - open/close 1023 (time 1773610439.38 total 84.15 last 10.30) - open/close 1074 (time 1773610449.75 total 94.51 last 4.92) - open/close 1190 (time 1773610459.81 total 104.57 last 11.53) - open/close 1346 (time 1773610469.81 total 114.58 last 15.59) - open/close 1597 (time 1773610480.60 total 125.37 last 23.28) - open/close 1653 (time 1773610490.61 total 135.38 last 5.60) - open/close 1701 (time 1773610503.26 total 148.03 last 3.79) - open/close 1869 (time 1773610514.33 total 159.10 last 15.17) - open/close 2006 (time 1773610524.35 total 169.12 last 13.68) - open/close 2104 (time 1773610534.43 total 179.20 last 9.72) - open/close 2188 (time 1773610544.43 total 189.20 last 8.40) - open/close 2240 (time 1773610555.49 total 200.26 last 4.70) - open/close 2442 (time 1773610565.50 total 210.27 last 20.17) - open/close 2549 (time 1773610576.63 total 221.39 last 9.62) - open/close 2617 (time 1773610586.78 total 231.55 last 6.70) - open/close 2766 (time 1773610597.04 total 241.81 last 14.52) - open/close 2846 (time 1773610607.59 total 252.36 last 7.58) - open/close 2973 (time 1773610617.84 total 262.61 last 12.39) - open/close 3064 (time 1773610628.45 total 273.22 last 8.58) - open/close 3291 (time 1773610639.13 total 283.90 last 21.24) - open/close 3393 (time 1773610649.89 total 294.66 last 9.48) - open/close 3595 (time 1773610660.30 total 305.07 last 19.40) - open/close 3666 (time 1773610670.57 total 315.34 last 6.92) - open/close 3715 (time 1773610680.59 total 325.36 last 4.89) - open/close 3801 (time 1773610691.22 total 335.99 last 8.09) - open/close 3904 (time 1773610702.23 total 346.99 last 9.36) - open/close 4057 (time 1773610714.19 total 358.96 last 12.79) - open/close 4140 (time 1773610724.77 total 369.54 last 7.85) - open/close 4236 (time 1773610734.93 total 379.70 last 9.45) - open/close 4290 (time 1773610744.95 total 389.72 last 5.39) - open/close 4350 (time 1773610755.10 total 399.87 last 5.91) - open/close 4452 (time 1773610765.11 total 409.88 last 10.19) - open/close 4509 (time 1773610776.18 total 420.95 last 5.15) - open/close 4612 (time 1773610786.18 total 430.95 last 10.29) - open/close 4752 (time 1773610796.66 total 441.42 last 13.37) - open/close 4847 (time 1773610806.76 total 451.52 last 9.41) - open/close 4911 (time 1773610817.11 total 461.88 last 6.18) - open/close 5019 (time 1773610827.66 total 472.43 last 10.23) - open/close 5145 (time 1773610837.67 total 482.44 last 12.59) - open/close 5211 (time 1773610847.69 total 492.45 last 6.59) - open/close 5363 (time 1773610858.53 total 503.29 last 14.02) - open/close 5419 (time 1773610868.83 total 513.60 last 5.43) - open/close 5543 (time 1773610879.45 total 524.22 last 11.67) - open/close 5698 (time 1773610889.46 total 534.23 last 15.48) - open/close 5816 (time 1773610899.94 total 544.70 last 11.27) - open/close 5945 (time 1773610912.86 total 557.63 last 9.98) - open/close 6097 (time 1773610923.66 total 568.43 last 14.07) - open/close 6168 (time 1773610934.25 total 579.02 last 6.71) - open/close 6250 (time 1773610944.75 total 589.52 last 7.81) - open/close 6358 (time 1773610954.76 total 599.53 last 10.79) - open/close 6434 (time 1773610965.48 total 610.25 last 7.09) - open/close 6536 (time 1773610975.72 total 620.49 last 9.95) - open/close 6647 (time 1773610986.10 total 630.87 last 10.69) - open/close 6728 (time 1773610997.68 total 642.45 last 6.99) - open/close 6828 (time 1773611007.79 total 652.56 last 9.89) - open/close 6917 (time 1773611018.73 total 663.50 last 8.13) - open/close 6956 (time 1773611030.34 total 675.11 last 3.36) - open/close 7045 (time 1773611040.35 total 685.12 last 8.89) - open/close 7141 (time 1773611051.17 total 695.94 last 8.88) - open/close 7193 (time 1773611061.19 total 705.96 last 5.19) - open/close 7314 (time 1773611071.79 total 716.56 last 11.42) - open/close 7411 (time 1773611081.80 total 726.57 last 9.69) - open/close 7510 (time 1773611092.13 total 736.90 last 9.59) - open/close 7615 (time 1773611102.43 total 747.20 last 10.19) - open/close 7715 (time 1773611112.64 total 757.41 last 9.80) - open/close 7796 (time 1773611123.59 total 768.35 last 7.40) - open/close 7893 (time 1773611133.59 total 778.36 last 9.69) - open/close 8006 (time 1773611143.90 total 788.67 last 10.96) - open/close 8122 (time 1773611154.74 total 799.51 last 10.70) - open/close 8236 (time 1773611164.90 total 809.67 last 11.22) - open/close 8392 (time 1773611176.86 total 821.63 last 13.04) - open/close 8534 (time 1773611186.87 total 831.64 last 14.19) - open/close 8744 (time 1773611197.09 total 841.86 last 20.55) - open/close 8855 (time 1773611207.12 total 851.89 last 11.07) - open/close 9044 (time 1773611217.63 total 862.40 last 17.99) - open/close 9191 (time 1773611229.04 total 873.81 last 12.88) - open/close 9293 (time 1773611239.05 total 883.82 last 10.19) - open/close 9394 (time 1773611249.89 total 894.66 last 9.31) - open/close 9515 (time 1773611260.72 total 905.49 last 11.18) - open/close 9742 (time 1773611270.73 total 915.50 last 22.67) - open/close 9935 (time 1773611280.74 total 925.51 last 19.28) - open/close 10000 (time 1773611285.56 total 930.33 last 13.49) - open/close 10191 (time 1773611296.43 total 941.20 last 17.57) - open/close 10369 (time 1773611306.44 total 951.21 last 17.78) - open/close 10437 (time 1773611316.47 total 961.24 last 6.78) - open/close 10550 (time 1773611329.00 total 973.77 last 9.02) - open/close 10660 (time 1773611339.36 total 984.13 last 10.62) - open/close 10814 (time 1773611350.16 total 994.93 last 14.26) - open/close 10966 (time 1773611360.17 total 1004.94 last 15.18) - open/close 11075 (time 1773611370.18 total 1014.95 last 10.89) - open/close 11179 (time 1773611380.96 total 1025.73 last 9.65) - open/close 11253 (time 1773611391.99 total 1036.76 last 6.71) - open/close 11341 (time 1773611402.47 total 1047.24 last 8.40) - open/close 11445 (time 1773611412.47 total 1057.24 last 10.40) - open/close 11631 (time 1773611423.00 total 1067.76 last 17.67) - open/close 11762 (time 1773611433.24 total 1078.00 last 12.79) - open/close 11959 (time 1773611443.87 total 1088.64 last 18.52) - open/close 12030 (time 1773611454.04 total 1098.81 last 6.99) - open/close 12140 (time 1773611464.72 total 1109.49 last 10.30) - open/close 12218 (time 1773611475.18 total 1119.94 last 7.46) - open/close 12292 (time 1773611485.67 total 1130.44 last 7.05) - open/close 12446 (time 1773611496.77 total 1141.54 last 13.87) - open/close 12525 (time 1773611507.18 total 1151.95 last 7.59) - open/close 12658 (time 1773611520.19 total 1164.96 last 10.23) - open/close 12847 (time 1773611536.04 total 1180.81 last 11.92) - open/close 13034 (time 1773611546.06 total 1190.82 last 18.67) - open/close 13501 (time 1773611556.07 total 1200.84 last 46.63) - open/close 14015 (time 1773611566.07 total 1210.84 last 51.38) - open/close 14641 (time 1773611576.08 total 1220.85 last 62.58) - open/close 15216 (time 1773611586.09 total 1230.86 last 57.40) - open/close 15829 (time 1773611596.10 total 1240.87 last 61.26) - open/close 16422 (time 1773611606.12 total 1250.89 last 59.19) - open/close 16983 (time 1773611616.12 total 1260.89 last 56.08) - open/close 17616 (time 1773611626.14 total 1270.91 last 63.20) - open/close 18179 (time 1773611636.14 total 1280.91 last 56.28) - open/close 18788 (time 1773611646.15 total 1290.92 last 60.84) - open/close 19449 (time 1773611656.18 total 1300.95 last 65.94) - open/close 20000 (time 1773611665.88 total 1310.65 last 56.79) - open/close 20731 (time 1773611675.89 total 1320.66 last 73.00) - open/close 21328 (time 1773611685.91 total 1330.68 last 59.60) - open/close 21860 (time 1773611695.95 total 1340.72 last 52.97) - open/close 22441 (time 1773611706.00 total 1350.77 last 57.85) - open/close 22962 (time 1773611716.00 total 1360.77 last 52.09) - open/close 23435 (time 1773611726.00 total 1370.77 last 47.29) - open/close 24037 (time 1773611736.00 total 1380.77 last 60.18) - open/close 24659 (time 1773611746.01 total 1390.78 last 62.17) - open/close 25255 (time 1773611756.02 total 1400.78 last 59.57) - open/close 25857 (time 1773611766.03 total 1410.80 last 60.12) - open/close 26427 (time 1773611776.03 total 1420.80 last 56.99) - open/close 26882 (time 1773611786.04 total 1430.80 last 45.48) - open/close 27488 (time 1773611796.04 total 1440.81 last 60.58) - open/close 28107 (time 1773611806.04 total 1450.81 last 61.89) - open/close 28697 (time 1773611816.05 total 1460.82 last 58.92) - open/close 29321 (time 1773611826.06 total 1470.83 last 62.34) - open/close 30000 (time 1773611835.81 total 1480.58 last 69.68) - open/close 30677 (time 1773611845.81 total 1490.58 last 67.70) - open/close 31254 (time 1773611855.85 total 1500.62 last 57.42) - open/close 31815 (time 1773611865.86 total 1510.63 last 56.06) - open/close 32477 (time 1773611875.88 total 1520.65 last 66.07) - open/close 33178 (time 1773611885.89 total 1530.65 last 70.06) - open/close 33745 (time 1773611895.90 total 1540.67 last 56.64) - open/close 34392 (time 1773611905.92 total 1550.69 last 64.57) - open/close 35073 (time 1773611915.94 total 1560.71 last 67.96) - open/close 35621 (time 1773611925.95 total 1570.72 last 54.74) - open/close 36217 (time 1773611935.95 total 1580.72 last 59.58) - open/close 36882 (time 1773611945.95 total 1590.72 last 66.49) - open/close 37469 (time 1773611955.96 total 1600.73 last 58.66) - open/close 38090 (time 1773611965.96 total 1610.73 last 62.07) - open/close 38756 (time 1773611975.97 total 1620.73 last 66.59) - open/close 39415 (time 1773611985.97 total 1630.74 last 65.86) - open/close 40000 (time 1773611994.67 total 1639.44 last 67.29) - open/close 40666 (time 1773612004.68 total 1649.45 last 66.50) - open/close 41208 (time 1773612014.69 total 1659.46 last 54.16) - open/close 41757 (time 1773612024.71 total 1669.48 last 54.80) - open/close 42352 (time 1773612034.72 total 1679.48 last 59.45) - open/close 43000 (time 1773612044.72 total 1689.49 last 64.77) - open/close 43729 (time 1773612054.73 total 1699.50 last 72.81) - open/close 44336 (time 1773612064.75 total 1709.52 last 60.62) - open/close 45055 (time 1773612074.75 total 1719.52 last 71.88) - open/close 45721 (time 1773612084.76 total 1729.53 last 66.53) - open/close 46229 (time 1773612094.77 total 1739.54 last 50.74) - open/close 46760 (time 1773612104.80 total 1749.56 last 52.97) - open/close 47292 (time 1773612114.80 total 1759.57 last 53.18) - open/close 47908 (time 1773612124.80 total 1769.57 last 61.59) - open/close 48578 (time 1773612134.82 total 1779.59 last 66.88) - open/close 49165 (time 1773612144.83 total 1789.60 last 58.65) - open/close 49754 (time 1773612154.85 total 1799.62 last 58.78) total: 50000 open/close in 1803.87 seconds: 27.72 ops/second - unlinked 0 (time 1773612164 ; total 0 ; last 0) - unlinked 10000 (time 1773612340 ; total 176 ; last 176) - unlinked 20000 (time 1773612505 ; total 341 ; last 165) - unlinked 30000 (time 1773612646 ; total 482 ; last 141) - unlinked 40000 (time 1773612784 ; total 620 ; last 138) total: 50000 unlinks in 756 seconds: 66.137566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg458-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server unloading modules via unload_modules_local on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing unload_modules_local oleg458-server: modules unloaded. pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2