== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:07:05 (1786039625) start mds service on oleg359-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg359-server: detected 4 online CPUs by sysfs oleg359-server: MODOPTS_LIBCFS= oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg359-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg359-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 113 (time 1786039740.82 total 10.01 last 11.28) - open/close 256 (time 1786039750.84 total 20.03 last 14.27) - open/close 444 (time 1786039761.81 total 31.01 last 17.13) - open/close 598 (time 1786039772.50 total 41.69 last 14.41) - open/close 699 (time 1786039782.60 total 51.80 last 9.99) - open/close 817 (time 1786039792.72 total 61.92 last 11.67) - open/close 975 (time 1786039802.73 total 71.92 last 15.79) - open/close 1094 (time 1786039813.65 total 82.84 last 10.90) - open/close 1181 (time 1786039825.80 total 95.00 last 7.16) - open/close 1333 (time 1786039835.84 total 105.03 last 15.15) - open/close 1437 (time 1786039846.28 total 115.47 last 9.96) - open/close 1484 (time 1786039856.98 total 126.18 last 4.39) - open/close 1577 (time 1786039867.72 total 136.92 last 8.66) - open/close 1616 (time 1786039878.16 total 147.35 last 3.74) - open/close 1686 (time 1786039888.16 total 157.36 last 7.00) - open/close 1793 (time 1786039898.96 total 168.16 last 9.91) - open/close 1914 (time 1786039908.97 total 178.16 last 12.09) - open/close 2002 (time 1786039919.13 total 188.33 last 8.66) - open/close 2145 (time 1786039929.14 total 198.34 last 14.28) - open/close 2301 (time 1786039939.50 total 208.70 last 15.06) - open/close 2434 (time 1786039950.17 total 219.37 last 12.47) - open/close 2595 (time 1786039960.17 total 229.37 last 16.10) - open/close 2774 (time 1786039970.19 total 239.38 last 17.87) - open/close 2964 (time 1786039980.19 total 249.39 last 18.99) - open/close 3079 (time 1786039990.21 total 259.41 last 11.47) - open/close 3206 (time 1786040000.64 total 269.84 last 12.18) - open/close 3377 (time 1786040011.49 total 280.69 last 15.76) - open/close 3468 (time 1786040021.49 total 290.69 last 9.10) - open/close 3571 (time 1786040031.90 total 301.10 last 9.89) - open/close 3718 (time 1786040042.76 total 311.95 last 13.54) - open/close 3816 (time 1786040053.77 total 322.96 last 8.90) - open/close 4040 (time 1786040065.28 total 334.48 last 19.46) - open/close 4157 (time 1786040075.29 total 344.48 last 11.69) - open/close 4327 (time 1786040085.29 total 354.49 last 17.00) - open/close 4424 (time 1786040095.30 total 364.50 last 9.69) - open/close 4534 (time 1786040105.58 total 374.78 last 10.70) - open/close 4651 (time 1786040115.59 total 384.79 last 11.69) - open/close 4753 (time 1786040127.71 total 396.90 last 8.42) - open/close 4870 (time 1786040138.40 total 407.59 last 10.94) - open/close 4935 (time 1786040148.42 total 417.62 last 6.49) - open/close 5033 (time 1786040158.42 total 427.62 last 9.80) - open/close 5132 (time 1786040169.22 total 438.41 last 9.17) - open/close 5223 (time 1786040180.11 total 449.30 last 8.35) - open/close 5344 (time 1786040191.05 total 460.24 last 11.06) - open/close 5484 (time 1786040201.06 total 470.26 last 13.98) - open/close 5621 (time 1786040211.87 total 481.07 last 12.67) - open/close 5757 (time 1786040221.92 total 491.12 last 13.53) - open/close 5894 (time 1786040232.45 total 501.64 last 13.02) - open/close 6023 (time 1786040242.83 total 512.03 last 12.43) - open/close 6113 (time 1786040253.52 total 522.72 last 8.42) - open/close 6226 (time 1786040263.97 total 533.17 last 10.81) - open/close 6343 (time 1786040274.63 total 543.83 last 10.97) - open/close 6438 (time 1786040284.64 total 553.84 last 9.49) - open/close 6582 (time 1786040294.66 total 563.85 last 14.38) - open/close 6701 (time 1786040304.72 total 573.92 last 11.82) - open/close 6814 (time 1786040315.48 total 584.68 last 10.50) - open/close 6911 (time 1786040325.83 total 595.03 last 9.38) - open/close 7048 (time 1786040336.94 total 606.14 last 12.33) - open/close 7149 (time 1786040346.95 total 616.14 last 10.10) - open/close 7250 (time 1786040357.72 total 626.92 last 9.37) - open/close 7413 (time 1786040368.27 total 637.47 last 15.46) - open/close 7465 (time 1786040379.23 total 648.42 last 4.75) - open/close 7504 (time 1786040389.53 total 658.73 last 3.78) - open/close 7666 (time 1786040400.34 total 669.54 last 14.99) - open/close 7827 (time 1786040410.35 total 679.55 last 16.09) - open/close 7951 (time 1786040420.76 total 689.96 last 11.91) - open/close 8076 (time 1786040431.19 total 700.39 last 11.98) - open/close 8177 (time 1786040441.62 total 710.82 last 9.68) - open/close 8314 (time 1786040452.63 total 721.83 last 12.45) - open/close 8433 (time 1786040462.86 total 732.06 last 11.63) - open/close 8477 (time 1786040473.41 total 742.61 last 4.17) - open/close 8618 (time 1786040483.42 total 752.62 last 14.09) - open/close 8744 (time 1786040493.74 total 762.94 last 12.21) - open/close 8847 (time 1786040504.21 total 773.40 last 9.84) - open/close 8992 (time 1786040515.18 total 784.38 last 13.22) - open/close 9199 (time 1786040525.27 total 794.46 last 20.52) - open/close 9251 (time 1786040536.14 total 805.34 last 4.78) - open/close 9302 (time 1786040547.09 total 816.29 last 4.66) - open/close 9377 (time 1786040557.80 total 827.00 last 7.00) - open/close 9494 (time 1786040567.81 total 837.01 last 11.69) - open/close 9582 (time 1786040577.82 total 847.01 last 8.80) - open/close 9681 (time 1786040588.84 total 858.04 last 8.98) - open/close 9783 (time 1786040599.76 total 868.96 last 9.34) - open/close 9910 (time 1786040610.72 total 879.92 last 11.58) - open/close 10000 (time 1786040620.37 total 889.57 last 9.33) - open/close 10050 (time 1786040631.26 total 900.45 last 4.59) - open/close 10138 (time 1786040642.13 total 911.33 last 8.09) - open/close 10292 (time 1786040652.80 total 922.00 last 14.44) - open/close 10401 (time 1786040662.81 total 932.01 last 10.89) - open/close 10550 (time 1786040672.84 total 942.04 last 14.85) - open/close 10719 (time 1786040682.89 total 952.09 last 16.82) - open/close 10788 (time 1786040693.21 total 962.40 last 6.69) - open/close 10937 (time 1786040703.45 total 972.65 last 14.54) - open/close 11006 (time 1786040713.88 total 983.08 last 6.62) - open/close 11099 (time 1786040724.18 total 993.37 last 9.03) - open/close 11162 (time 1786040734.47 total 1003.67 last 6.12) - open/close 11172 (time 1786040744.81 total 1014.01 last 0.97) - open/close 11267 (time 1786040755.14 total 1024.34 last 9.20) - open/close 11341 (time 1786040766.59 total 1035.78 last 6.47) - open/close 11435 (time 1786040776.60 total 1045.80 last 9.38) - open/close 11550 (time 1786040787.15 total 1056.35 last 10.90) - open/close 11666 (time 1786040797.33 total 1066.52 last 11.40) - open/close 11772 (time 1786040808.21 total 1077.41 last 9.74) - open/close 11875 (time 1786040818.35 total 1087.55 last 10.16) - open/close 12016 (time 1786040828.35 total 1097.55 last 14.09) - open/close 12137 (time 1786040838.37 total 1107.57 last 12.08) - open/close 12277 (time 1786040848.43 total 1117.62 last 13.92) - open/close 12419 (time 1786040858.90 total 1128.09 last 13.56) - open/close 12506 (time 1786040869.46 total 1138.66 last 8.24) - open/close 12606 (time 1786040879.47 total 1148.67 last 9.99) - open/close 12699 (time 1786040889.58 total 1158.78 last 9.20) - open/close 12828 (time 1786040899.58 total 1168.78 last 12.90) - open/close 12919 (time 1786040911.70 total 1180.89 last 7.51) - open/close 13020 (time 1786040921.88 total 1191.08 last 9.92) - open/close 13125 (time 1786040932.82 total 1202.02 last 9.60) - open/close 13245 (time 1786040942.84 total 1212.04 last 11.98) - open/close 13372 (time 1786040952.98 total 1222.17 last 12.53) - open/close 13507 (time 1786040963.01 total 1232.21 last 13.45) - open/close 13619 (time 1786040973.08 total 1242.28 last 11.12) - open/close 13746 (time 1786040984.45 total 1253.65 last 11.17) - open/close 13893 (time 1786040994.47 total 1263.66 last 14.68) - open/close 14019 (time 1786041005.45 total 1274.65 last 11.47) - open/close 14122 (time 1786041015.76 total 1284.96 last 9.99) - open/close 14173 (time 1786041026.59 total 1295.79 last 4.71) - open/close 14261 (time 1786041036.77 total 1305.97 last 8.64) - open/close 14461 (time 1786041046.78 total 1315.98 last 19.98) - open/close 14894 (time 1786041056.79 total 1325.99 last 43.24) - open/close 15274 (time 1786041066.82 total 1336.02 last 37.89) - open/close 15732 (time 1786041076.85 total 1346.05 last 45.67) - open/close 16107 (time 1786041086.86 total 1356.06 last 37.46) - open/close 16510 (time 1786041096.87 total 1366.06 last 40.28) - open/close 17046 (time 1786041106.87 total 1376.07 last 53.57) - open/close 17548 (time 1786041116.88 total 1386.08 last 50.16) - open/close 18000 (time 1786041126.88 total 1396.08 last 45.19) - open/close 18494 (time 1786041136.91 total 1406.10 last 49.29) - open/close 18984 (time 1786041146.93 total 1416.12 last 48.90) - open/close 19395 (time 1786041156.93 total 1426.12 last 41.09) - open/close 19888 (time 1786041166.95 total 1436.14 last 49.20) - open/close 20000 (time 1786041169.25 total 1438.45 last 48.68) - open/close 20715 (time 1786041179.26 total 1448.46 last 71.41) - open/close 21228 (time 1786041189.28 total 1458.47 last 51.21) - open/close 21680 (time 1786041199.29 total 1468.49 last 45.15) - open/close 22069 (time 1786041209.29 total 1478.49 last 38.90) - open/close 22653 (time 1786041219.29 total 1488.49 last 58.40) - open/close 23270 (time 1786041229.31 total 1498.50 last 61.60) - open/close 23787 (time 1786041239.31 total 1508.51 last 51.67) - open/close 24263 (time 1786041249.33 total 1518.53 last 47.50) - open/close 24757 (time 1786041259.35 total 1528.55 last 49.31) - open/close 25240 (time 1786041269.35 total 1538.55 last 48.29) - open/close 25643 (time 1786041279.39 total 1548.59 last 40.16) - open/close 26049 (time 1786041289.42 total 1558.61 last 40.50) - open/close 26534 (time 1786041299.42 total 1568.62 last 48.48) - open/close 26942 (time 1786041309.42 total 1578.62 last 40.80) - open/close 27339 (time 1786041319.43 total 1588.63 last 39.65) - open/close 27841 (time 1786041329.46 total 1598.66 last 50.08) - open/close 28316 (time 1786041339.48 total 1608.68 last 47.39) - open/close 28738 (time 1786041349.49 total 1618.69 last 42.17) - open/close 29235 (time 1786041359.50 total 1628.70 last 49.65) - open/close 29701 (time 1786041369.51 total 1638.70 last 46.57) - open/close 30000 (time 1786041375.74 total 1644.94 last 47.98) - open/close 30522 (time 1786041385.76 total 1654.95 last 52.11) - open/close 30964 (time 1786041395.78 total 1664.98 last 44.08) - open/close 31682 (time 1786041405.79 total 1674.99 last 71.76) - open/close 32285 (time 1786041415.79 total 1684.99 last 60.28) - open/close 32742 (time 1786041425.80 total 1694.99 last 45.69) - open/close 33225 (time 1786041435.80 total 1705.00 last 48.26) - open/close 33720 (time 1786041445.80 total 1715.00 last 49.49) - open/close 34210 (time 1786041455.83 total 1725.02 last 48.89) - open/close 34722 (time 1786041465.84 total 1735.04 last 51.12) - open/close 35236 (time 1786041475.87 total 1745.07 last 51.27) - open/close 35696 (time 1786041485.87 total 1755.07 last 46.00) - open/close 36159 (time 1786041495.87 total 1765.07 last 46.30) - open/close 36616 (time 1786041505.87 total 1775.07 last 45.69) - open/close 37102 (time 1786041515.88 total 1785.08 last 48.55) - open/close 37574 (time 1786041525.90 total 1795.10 last 47.09) - open/close 38018 (time 1786041535.94 total 1805.13 last 44.26) - open/close 38500 (time 1786041545.95 total 1815.15 last 48.13) - open/close 38988 (time 1786041555.97 total 1825.17 last 48.69) - open/close 39344 (time 1786041566.06 total 1835.26 last 35.28) - open/close 39605 (time 1786041576.07 total 1845.27 last 26.07) - open/close 39876 (time 1786041586.09 total 1855.29 last 27.05) - open/close 40000 (time 1786041589.16 total 1858.35 last 40.42) - open/close 40350 (time 1786041599.16 total 1868.36 last 34.99) - open/close 40741 (time 1786041609.16 total 1878.36 last 39.08) - open/close 41298 (time 1786041619.19 total 1888.39 last 55.54) - open/close 41737 (time 1786041629.21 total 1898.41 last 43.83) - open/close 42131 (time 1786041639.23 total 1908.43 last 39.30) - open/close 42586 (time 1786041649.24 total 1918.44 last 45.45) - open/close 43023 (time 1786041659.25 total 1928.45 last 43.68) - open/close 43579 (time 1786041669.25 total 1938.45 last 55.59) - open/close 44069 (time 1786041679.26 total 1948.45 last 48.98) - open/close 44525 (time 1786041689.27 total 1958.46 last 45.56) - open/close 45042 (time 1786041699.29 total 1968.48 last 51.59) - open/close 45461 (time 1786041709.30 total 1978.50 last 41.85) - open/close 45907 (time 1786041719.31 total 1988.51 last 44.53) - open/close 46448 (time 1786041729.33 total 1998.52 last 54.04) - open/close 46926 (time 1786041739.34 total 2008.53 last 47.75) - open/close 47578 (time 1786041749.35 total 2018.55 last 65.07) - open/close 48043 (time 1786041759.36 total 2028.56 last 46.46) - open/close 48461 (time 1786041769.36 total 2038.56 last 41.79) - open/close 48944 (time 1786041779.38 total 2048.58 last 48.22) - open/close 49463 (time 1786041789.39 total 2058.58 last 51.86) - open/close 49848 (time 1786041799.39 total 2068.59 last 38.48) total: 50000 open/close in 2071.33 seconds: 24.14 ops/second - unlinked 0 (time 1786041807 ; total 0 ; last 0) - unlinked 10000 (time 1786042016 ; total 209 ; last 209) - unlinked 20000 (time 1786042220 ; total 413 ; last 204) - unlinked 30000 (time 1786042424 ; total 617 ; last 204) - unlinked 40000 (time 1786042645 ; total 838 ; last 221) total: 50000 unlinks in 1051 seconds: 47.573738 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules via unload_modules_local on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local oleg359-server: modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2