== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:18:46 (1787145526) start mds service on oleg446-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing load_modules_local oleg446-server: Loading modules from /home/green/git/lustre-release/lustre oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg446-server: detected 4 online CPUs by sysfs oleg446-server: MODOPTS_LIBCFS= oleg446-server: Force libcfs to create 2 CPU partitions oleg446-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg446-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg446-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1787145609.39 total 10.01 last 18.98) - open/close 418 (time 1787145619.96 total 20.59 last 21.56) - open/close 698 (time 1787145630.53 total 31.16 last 26.48) - open/close 919 (time 1787145640.56 total 41.18 last 22.05) - open/close 1125 (time 1787145650.94 total 51.57 last 19.83) - open/close 1316 (time 1787145660.94 total 61.57 last 19.10) - open/close 1531 (time 1787145670.95 total 71.57 last 21.50) - open/close 1784 (time 1787145680.96 total 81.59 last 25.26) - open/close 1999 (time 1787145691.92 total 92.55 last 19.61) - open/close 2177 (time 1787145702.98 total 103.60 last 16.11) - open/close 2353 (time 1787145713.71 total 114.34 last 16.40) - open/close 2572 (time 1787145723.73 total 124.35 last 21.86) - open/close 2677 (time 1787145734.32 total 134.95 last 9.91) - open/close 2832 (time 1787145745.09 total 145.72 last 14.39) - open/close 2887 (time 1787145755.09 total 155.72 last 5.50) - open/close 3068 (time 1787145765.10 total 165.72 last 18.09) - open/close 3256 (time 1787145775.50 total 176.13 last 18.07) - open/close 3434 (time 1787145785.50 total 186.13 last 17.80) - open/close 3681 (time 1787145795.50 total 196.13 last 24.69) - open/close 3834 (time 1787145807.95 total 208.57 last 12.30) - open/close 4030 (time 1787145818.47 total 219.09 last 18.63) - open/close 4155 (time 1787145828.47 total 229.10 last 12.49) - open/close 4352 (time 1787145839.57 total 240.20 last 17.76) - open/close 4546 (time 1787145849.58 total 250.21 last 19.38) - open/close 4679 (time 1787145859.59 total 260.22 last 13.28) - open/close 4821 (time 1787145871.83 total 272.45 last 11.61) - open/close 4989 (time 1787145883.32 total 283.95 last 14.61) - open/close 5189 (time 1787145893.76 total 294.39 last 19.16) - open/close 5247 (time 1787145903.77 total 304.39 last 5.80) - open/close 5404 (time 1787145914.74 total 315.37 last 14.30) - open/close 5657 (time 1787145925.69 total 326.32 last 23.10) - open/close 5831 (time 1787145936.63 total 337.26 last 15.90) - open/close 6004 (time 1787145947.26 total 347.89 last 16.28) - open/close 6096 (time 1787145957.31 total 357.94 last 9.16) - open/close 6311 (time 1787145967.32 total 367.95 last 21.47) - open/close 6547 (time 1787145977.33 total 377.95 last 23.59) - open/close 6673 (time 1787145987.33 total 387.96 last 12.59) - open/close 6880 (time 1787145999.23 total 399.86 last 17.40) - open/close 7024 (time 1787146009.35 total 409.98 last 14.22) - open/close 7178 (time 1787146019.85 total 420.47 last 14.68) - open/close 7282 (time 1787146030.26 total 430.89 last 9.99) - open/close 7568 (time 1787146041.11 total 441.73 last 26.37) - open/close 7816 (time 1787146051.98 total 452.60 last 22.82) - open/close 8056 (time 1787146061.99 total 462.62 last 23.96) - open/close 8222 (time 1787146072.83 total 473.46 last 15.32) - open/close 8298 (time 1787146082.88 total 483.51 last 7.56) - open/close 8531 (time 1787146092.88 total 493.51 last 23.29) - open/close 8738 (time 1787146103.36 total 503.98 last 19.76) - open/close 8981 (time 1787146114.10 total 514.73 last 22.61) - open/close 9244 (time 1787146125.11 total 525.74 last 23.89) - open/close 9474 (time 1787146135.12 total 535.75 last 22.97) - open/close 9686 (time 1787146145.40 total 546.03 last 20.63) - open/close 9922 (time 1787146156.87 total 557.50 last 20.57) - open/close 10000 (time 1787146160.35 total 560.98 last 22.44) - open/close 10227 (time 1787146170.49 total 571.12 last 22.39) - open/close 10443 (time 1787146180.49 total 581.12 last 21.59) - open/close 10645 (time 1787146190.50 total 591.13 last 20.19) - open/close 10869 (time 1787146200.50 total 601.13 last 22.39) - open/close 11055 (time 1787146210.54 total 611.17 last 18.53) - open/close 11270 (time 1787146220.56 total 621.19 last 21.46) - open/close 11441 (time 1787146231.30 total 631.93 last 15.92) - open/close 11573 (time 1787146241.49 total 642.12 last 12.96) - open/close 11709 (time 1787146251.49 total 652.12 last 13.60) - open/close 11916 (time 1787146261.57 total 662.20 last 20.54) - open/close 12094 (time 1787146271.58 total 672.21 last 17.77) - open/close 12336 (time 1787146281.60 total 682.23 last 24.16) - open/close 12515 (time 1787146291.79 total 692.42 last 17.57) - open/close 12728 (time 1787146301.80 total 702.43 last 21.28) - open/close 12995 (time 1787146312.43 total 713.06 last 25.12) - open/close 13237 (time 1787146322.45 total 723.08 last 24.16) - open/close 13443 (time 1787146332.54 total 733.17 last 20.41) - open/close 13674 (time 1787146342.91 total 743.53 last 22.29) - open/close 13901 (time 1787146353.91 total 754.54 last 20.62) - open/close 14154 (time 1787146364.21 total 764.84 last 24.58) - open/close 14360 (time 1787146374.32 total 774.95 last 20.36) - open/close 14584 (time 1787146384.33 total 784.96 last 22.39) - open/close 14815 (time 1787146394.36 total 794.99 last 23.03) - open/close 14900 (time 1787146404.37 total 804.99 last 8.49) - open/close 15104 (time 1787146414.59 total 815.21 last 19.96) - open/close 15302 (time 1787146425.47 total 826.10 last 18.19) - open/close 15536 (time 1787146435.48 total 836.11 last 23.37) - open/close 15760 (time 1787146445.51 total 846.14 last 22.35) - open/close 16019 (time 1787146455.52 total 856.15 last 25.85) - open/close 16218 (time 1787146465.53 total 866.16 last 19.88) - open/close 16459 (time 1787146476.41 total 877.04 last 22.16) - open/close 16657 (time 1787146486.42 total 887.05 last 19.77) - open/close 16822 (time 1787146497.08 total 897.71 last 15.48) - open/close 17015 (time 1787146507.48 total 908.11 last 18.55) - open/close 17263 (time 1787146517.49 total 918.12 last 24.78) - open/close 17496 (time 1787146528.25 total 928.88 last 21.65) - open/close 17716 (time 1787146538.26 total 938.88 last 21.99) - open/close 17938 (time 1787146548.48 total 949.11 last 21.70) - open/close 18121 (time 1787146558.49 total 959.11 last 18.30) - open/close 18591 (time 1787146568.49 total 969.12 last 46.97) - open/close 19178 (time 1787146578.50 total 979.13 last 58.65) - open/close 19789 (time 1787146588.50 total 989.13 last 61.07) - open/close 20000 (time 1787146591.79 total 992.42 last 64.13) - open/close 20571 (time 1787146601.81 total 1002.43 last 57.03) - open/close 21195 (time 1787146611.81 total 1012.44 last 62.38) - open/close 21789 (time 1787146621.82 total 1022.45 last 59.32) - open/close 22401 (time 1787146631.83 total 1032.46 last 61.15) - open/close 23057 (time 1787146641.84 total 1042.47 last 65.52) - open/close 23719 (time 1787146651.85 total 1052.48 last 66.16) - open/close 24290 (time 1787146661.86 total 1062.49 last 57.04) - open/close 24824 (time 1787146671.87 total 1072.50 last 53.34) - open/close 25198 (time 1787146681.88 total 1082.51 last 37.36) - open/close 25563 (time 1787146691.89 total 1092.52 last 36.47) - open/close 26092 (time 1787146701.91 total 1102.54 last 52.79) - open/close 26675 (time 1787146711.94 total 1112.57 last 58.11) - open/close 27141 (time 1787146721.98 total 1122.60 last 46.44) - open/close 27554 (time 1787146731.98 total 1132.61 last 41.30) - open/close 28027 (time 1787146741.99 total 1142.62 last 47.22) - open/close 28609 (time 1787146752.00 total 1152.63 last 58.17) - open/close 29259 (time 1787146762.01 total 1162.63 last 64.95) - open/close 29901 (time 1787146772.01 total 1172.64 last 64.18) - open/close 30484 (time 1787146782.02 total 1182.65 last 58.21) - open/close 31075 (time 1787146792.03 total 1192.66 last 59.05) - open/close 31693 (time 1787146802.04 total 1202.67 last 61.75) - open/close 32278 (time 1787146812.04 total 1212.67 last 58.49) - open/close 32859 (time 1787146822.05 total 1222.67 last 58.07) - open/close 33404 (time 1787146832.06 total 1232.69 last 54.39) - open/close 33989 (time 1787146842.10 total 1242.73 last 58.30) - open/close 34606 (time 1787146852.10 total 1252.73 last 61.70) - open/close 35233 (time 1787146862.10 total 1262.73 last 62.67) - open/close 35748 (time 1787146872.12 total 1272.74 last 51.44) - open/close 36332 (time 1787146882.14 total 1282.77 last 58.25) - open/close 36952 (time 1787146892.15 total 1292.77 last 61.98) - open/close 37561 (time 1787146902.16 total 1302.79 last 60.79) - open/close 38160 (time 1787146912.18 total 1312.81 last 59.81) - open/close 38765 (time 1787146922.19 total 1322.82 last 60.44) - open/close 39364 (time 1787146932.19 total 1332.82 last 59.89) - open/close 39964 (time 1787146942.19 total 1342.82 last 59.97) - open/close 40590 (time 1787146952.22 total 1352.85 last 62.42) - open/close 41202 (time 1787146962.23 total 1362.86 last 61.17) - open/close 41854 (time 1787146972.25 total 1372.88 last 65.07) - open/close 42506 (time 1787146982.25 total 1382.88 last 65.18) - open/close 43193 (time 1787146992.25 total 1392.88 last 68.70) - open/close 43810 (time 1787147002.26 total 1402.89 last 61.65) - open/close 44400 (time 1787147012.27 total 1412.90 last 58.95) - open/close 44999 (time 1787147022.27 total 1422.90 last 59.87) - open/close 45587 (time 1787147032.29 total 1432.92 last 58.70) - open/close 46175 (time 1787147042.30 total 1442.93 last 58.74) - open/close 46733 (time 1787147052.33 total 1452.95 last 55.67) - open/close 47334 (time 1787147062.33 total 1462.96 last 60.09) - open/close 47974 (time 1787147072.33 total 1472.96 last 63.99) - open/close 48646 (time 1787147082.34 total 1482.97 last 67.10) - open/close 49272 (time 1787147092.35 total 1492.98 last 62.55) - open/close 49896 (time 1787147102.36 total 1502.99 last 62.36) total: 50000 open/close in 1504.45 seconds: 33.23 ops/second - unlinked 0 (time 1787147108 ; total 0 ; last 0) - unlinked 10000 (time 1787147277 ; total 169 ; last 169) - unlinked 20000 (time 1787147449 ; total 341 ; last 172) - unlinked 30000 (time 1787147613 ; total 505 ; last 164) - unlinked 40000 (time 1787147782 ; total 674 ; last 169) total: 50000 unlinks in 843 seconds: 59.311981 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server unloading modules via unload_modules_local on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing unload_modules_local oleg446-server: modules unloaded. pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2