== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:20:41 (1786018841) start mds service on oleg441-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg441-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg441-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 162 (time 1786018954.43 total 10.55 last 15.35) - open/close 273 (time 1786018965.30 total 21.43 last 10.21) - open/close 386 (time 1786018975.76 total 31.88 last 10.81) - open/close 442 (time 1786018985.77 total 41.90 last 5.59) - open/close 568 (time 1786018995.85 total 51.97 last 12.50) - open/close 714 (time 1786019005.86 total 61.98 last 14.59) - open/close 833 (time 1786019016.74 total 72.86 last 10.94) - open/close 946 (time 1786019027.33 total 83.46 last 10.66) - open/close 1071 (time 1786019038.20 total 94.32 last 11.51) - open/close 1264 (time 1786019048.97 total 105.10 last 17.91) - open/close 1382 (time 1786019059.87 total 115.99 last 10.83) - open/close 1478 (time 1786019070.15 total 126.28 last 9.33) - open/close 1580 (time 1786019080.44 total 136.57 last 9.91) - open/close 1687 (time 1786019090.94 total 147.07 last 10.19) - open/close 1751 (time 1786019101.10 total 157.23 last 6.30) - open/close 1838 (time 1786019112.03 total 168.15 last 7.97) - open/close 1923 (time 1786019122.62 total 178.74 last 8.03) - open/close 1983 (time 1786019132.64 total 188.76 last 5.99) - open/close 2061 (time 1786019142.64 total 198.77 last 7.80) - open/close 2144 (time 1786019152.98 total 209.10 last 8.03) - open/close 2239 (time 1786019163.00 total 219.12 last 9.48) - open/close 2372 (time 1786019173.91 total 230.03 last 12.19) - open/close 2503 (time 1786019183.92 total 240.05 last 13.08) - open/close 2628 (time 1786019194.61 total 250.74 last 11.69) - open/close 2743 (time 1786019205.44 total 261.56 last 10.63) - open/close 2848 (time 1786019215.47 total 271.60 last 10.46) - open/close 2937 (time 1786019225.97 total 282.09 last 8.48) - open/close 3128 (time 1786019235.97 total 292.09 last 19.09) - open/close 3288 (time 1786019246.69 total 302.81 last 14.93) - open/close 3415 (time 1786019257.28 total 313.40 last 11.99) - open/close 3568 (time 1786019268.26 total 324.38 last 13.94) - open/close 3733 (time 1786019278.87 total 335.00 last 15.55) - open/close 3905 (time 1786019289.92 total 346.04 last 15.57) - open/close 4050 (time 1786019299.93 total 356.05 last 14.48) - open/close 4193 (time 1786019310.95 total 367.08 last 12.97) - open/close 4347 (time 1786019320.98 total 377.10 last 15.36) - open/close 4487 (time 1786019330.99 total 387.11 last 13.99) - open/close 4604 (time 1786019341.98 total 398.10 last 10.64) - open/close 4682 (time 1786019352.57 total 408.69 last 7.36) - open/close 4735 (time 1786019362.58 total 418.70 last 5.30) - open/close 4793 (time 1786019372.67 total 428.79 last 5.75) - open/close 4862 (time 1786019382.69 total 438.81 last 6.88) - open/close 5001 (time 1786019392.71 total 448.83 last 13.87) - open/close 5111 (time 1786019402.71 total 458.83 last 11.00) - open/close 5325 (time 1786019413.63 total 469.75 last 19.60) - open/close 5553 (time 1786019423.66 total 479.79 last 22.72) - open/close 5688 (time 1786019434.70 total 490.82 last 12.23) - open/close 5769 (time 1786019444.70 total 500.83 last 8.10) - open/close 5907 (time 1786019454.70 total 510.83 last 13.80) - open/close 5965 (time 1786019464.71 total 520.83 last 5.80) - open/close 6056 (time 1786019475.25 total 531.38 last 8.63) - open/close 6138 (time 1786019485.85 total 541.97 last 7.74) - open/close 6194 (time 1786019496.62 total 552.74 last 5.20) - open/close 6245 (time 1786019507.57 total 563.70 last 4.65) - open/close 6367 (time 1786019518.03 total 574.15 last 11.67) - open/close 6472 (time 1786019528.05 total 584.18 last 10.47) - open/close 6620 (time 1786019538.60 total 594.73 last 14.03) - open/close 6786 (time 1786019548.62 total 604.75 last 16.57) - open/close 6880 (time 1786019558.64 total 614.76 last 9.39) - open/close 6977 (time 1786019569.70 total 625.83 last 8.77) - open/close 7107 (time 1786019579.72 total 635.84 last 12.98) - open/close 7204 (time 1786019590.60 total 646.72 last 8.92) - open/close 7257 (time 1786019600.65 total 656.77 last 5.27) - open/close 7339 (time 1786019611.14 total 667.26 last 7.82) - open/close 7402 (time 1786019621.31 total 677.43 last 6.19) - open/close 7503 (time 1786019631.36 total 687.49 last 10.04) - open/close 7591 (time 1786019641.36 total 697.49 last 8.80) - open/close 7723 (time 1786019651.42 total 707.55 last 13.12) - open/close 7783 (time 1786019661.42 total 717.55 last 6.00) - open/close 7848 (time 1786019671.56 total 727.68 last 6.42) - open/close 7906 (time 1786019681.74 total 737.86 last 5.70) - open/close 8005 (time 1786019691.76 total 747.89 last 9.87) - open/close 8075 (time 1786019703.67 total 759.79 last 5.88) - open/close 8181 (time 1786019714.54 total 770.67 last 9.75) - open/close 8283 (time 1786019724.67 total 780.80 last 10.07) - open/close 8340 (time 1786019734.70 total 790.82 last 5.68) - open/close 8414 (time 1786019744.70 total 800.83 last 7.40) - open/close 8528 (time 1786019754.71 total 810.84 last 11.39) - open/close 8687 (time 1786019765.14 total 821.26 last 15.25) - open/close 8750 (time 1786019776.01 total 832.13 last 5.80) - open/close 8856 (time 1786019786.36 total 842.49 last 10.23) - open/close 9021 (time 1786019796.38 total 852.50 last 16.47) - open/close 9140 (time 1786019807.00 total 863.12 last 11.21) - open/close 9292 (time 1786019817.11 total 873.23 last 15.03) - open/close 9432 (time 1786019827.21 total 883.33 last 13.86) - open/close 9550 (time 1786019837.57 total 893.70 last 11.39) - open/close 9678 (time 1786019848.47 total 904.60 last 11.74) - open/close 9776 (time 1786019858.88 total 915.00 last 9.42) - open/close 9925 (time 1786019869.19 total 925.31 last 14.45) - open/close 10000 (time 1786019874.61 total 930.73 last 13.84) - open/close 10095 (time 1786019884.65 total 940.77 last 9.46) - open/close 10243 (time 1786019895.37 total 951.50 last 13.80) - open/close 10308 (time 1786019905.38 total 961.50 last 6.49) - open/close 10397 (time 1786019916.33 total 972.45 last 8.13) - open/close 10499 (time 1786019926.61 total 982.74 last 9.91) - open/close 10637 (time 1786019937.26 total 993.39 last 12.96) - open/close 10745 (time 1786019947.66 total 1003.78 last 10.39) - open/close 10883 (time 1786019957.68 total 1013.81 last 13.76) - open/close 11028 (time 1786019967.72 total 1023.84 last 14.45) - open/close 11148 (time 1786019978.30 total 1034.42 last 11.34) - open/close 11267 (time 1786019988.30 total 1044.42 last 11.90) - open/close 11321 (time 1786019998.31 total 1054.43 last 5.40) - open/close 11454 (time 1786020008.33 total 1064.46 last 13.26) - open/close 11522 (time 1786020018.71 total 1074.84 last 6.55) - open/close 11629 (time 1786020029.39 total 1085.52 last 10.02) - open/close 11724 (time 1786020039.45 total 1095.58 last 9.44) - open/close 11869 (time 1786020049.49 total 1105.62 last 14.44) - open/close 11965 (time 1786020059.53 total 1115.65 last 9.56) - open/close 12085 (time 1786020069.55 total 1125.67 last 11.98) - open/close 12216 (time 1786020080.60 total 1136.72 last 11.86) - open/close 12371 (time 1786020091.65 total 1147.77 last 14.03) - open/close 12596 (time 1786020101.67 total 1157.80 last 22.44) - open/close 12734 (time 1786020111.68 total 1167.80 last 13.79) - open/close 12867 (time 1786020122.11 total 1178.23 last 12.75) - open/close 12970 (time 1786020132.78 total 1188.91 last 9.65) - open/close 13111 (time 1786020143.30 total 1199.43 last 13.40) - open/close 13260 (time 1786020153.49 total 1209.62 last 14.62) - open/close 13366 (time 1786020163.63 total 1219.76 last 10.45) - open/close 13475 (time 1786020173.64 total 1229.76 last 10.90) - open/close 13554 (time 1786020184.33 total 1240.45 last 7.39) - open/close 13624 (time 1786020194.78 total 1250.91 last 6.70) - open/close 13743 (time 1786020204.79 total 1260.91 last 11.89) - open/close 13840 (time 1786020214.85 total 1270.98 last 9.64) - open/close 13942 (time 1786020225.05 total 1281.17 last 10.01) - open/close 14037 (time 1786020235.74 total 1291.86 last 8.89) - open/close 14235 (time 1786020246.02 total 1302.14 last 19.26) - open/close 14351 (time 1786020256.39 total 1312.51 last 11.19) - open/close 14406 (time 1786020267.26 total 1323.38 last 5.06) - open/close 14518 (time 1786020277.51 total 1333.63 last 10.93) - open/close 14598 (time 1786020287.56 total 1343.69 last 7.96) - open/close 14718 (time 1786020298.26 total 1354.38 last 11.22) - open/close 14849 (time 1786020309.43 total 1365.55 last 11.72) - open/close 14988 (time 1786020320.54 total 1376.66 last 12.51) - open/close 15081 (time 1786020330.88 total 1387.00 last 8.99) - open/close 15709 (time 1786020340.89 total 1397.01 last 62.76) - open/close 16271 (time 1786020350.90 total 1407.02 last 56.15) - open/close 16746 (time 1786020360.93 total 1417.05 last 47.34) - open/close 17210 (time 1786020370.94 total 1427.06 last 46.34) - open/close 17780 (time 1786020380.95 total 1437.08 last 56.94) - open/close 18114 (time 1786020390.97 total 1447.09 last 33.36) - open/close 18418 (time 1786020400.97 total 1457.09 last 30.39) - open/close 18739 (time 1786020410.98 total 1467.10 last 32.07) - open/close 19285 (time 1786020420.98 total 1477.11 last 54.57) - open/close 19834 (time 1786020430.98 total 1487.11 last 54.89) - open/close 20000 (time 1786020434.42 total 1490.54 last 48.37) - open/close 20499 (time 1786020444.43 total 1500.56 last 49.81) - open/close 20977 (time 1786020454.44 total 1510.56 last 47.79) - open/close 21423 (time 1786020464.44 total 1520.56 last 44.60) - open/close 21861 (time 1786020474.52 total 1530.64 last 43.44) - open/close 22296 (time 1786020484.53 total 1540.65 last 43.47) - open/close 22906 (time 1786020494.53 total 1550.65 last 60.97) - open/close 23420 (time 1786020504.54 total 1560.67 last 51.33) - open/close 23925 (time 1786020514.55 total 1570.67 last 50.50) - open/close 24528 (time 1786020524.55 total 1580.68 last 60.25) - open/close 25099 (time 1786020534.56 total 1590.68 last 57.08) - open/close 25591 (time 1786020544.59 total 1600.71 last 49.06) - open/close 25981 (time 1786020554.61 total 1610.73 last 38.91) - open/close 26432 (time 1786020564.61 total 1620.74 last 45.08) - open/close 26865 (time 1786020574.63 total 1630.76 last 43.22) - open/close 27298 (time 1786020584.64 total 1640.76 last 43.28) - open/close 27738 (time 1786020594.64 total 1650.76 last 43.99) - open/close 28172 (time 1786020604.65 total 1660.78 last 43.35) - open/close 28614 (time 1786020614.66 total 1670.78 last 44.16) - open/close 29129 (time 1786020624.67 total 1680.79 last 51.45) - open/close 29480 (time 1786020634.69 total 1690.81 last 35.03) - open/close 29762 (time 1786020644.71 total 1700.83 last 28.15) - open/close 30000 (time 1786020653.02 total 1709.14 last 28.64) - open/close 30456 (time 1786020663.03 total 1719.15 last 45.54) - open/close 30852 (time 1786020673.04 total 1729.17 last 39.54) - open/close 31183 (time 1786020683.07 total 1739.20 last 33.00) - open/close 31635 (time 1786020693.08 total 1749.21 last 45.15) - open/close 31964 (time 1786020703.09 total 1759.22 last 32.87) - open/close 32466 (time 1786020713.10 total 1769.23 last 50.16) - open/close 32846 (time 1786020723.12 total 1779.24 last 37.94) - open/close 33248 (time 1786020733.13 total 1789.26 last 40.13) - open/close 33704 (time 1786020743.15 total 1799.28 last 45.52) - open/close 34122 (time 1786020753.17 total 1809.29 last 41.72) - open/close 34520 (time 1786020763.18 total 1819.30 last 39.76) - open/close 34869 (time 1786020773.18 total 1829.31 last 34.89) - open/close 35223 (time 1786020783.19 total 1839.31 last 35.39) - open/close 35592 (time 1786020793.21 total 1849.34 last 36.80) - open/close 36126 (time 1786020803.21 total 1859.34 last 53.40) - open/close 36778 (time 1786020813.25 total 1869.37 last 64.98) - open/close 37222 (time 1786020823.25 total 1879.38 last 44.38) - open/close 37650 (time 1786020833.26 total 1889.38 last 42.79) - open/close 38019 (time 1786020843.31 total 1899.43 last 36.70) - open/close 38446 (time 1786020853.34 total 1909.46 last 42.57) - open/close 38889 (time 1786020863.34 total 1919.47 last 44.29) - open/close 39478 (time 1786020873.35 total 1929.47 last 58.87) - open/close 39862 (time 1786020883.38 total 1939.51 last 38.26) - open/close 40000 (time 1786020886.46 total 1942.59 last 44.81) - open/close 40406 (time 1786020896.48 total 1952.60 last 40.54) - open/close 41026 (time 1786020906.50 total 1962.62 last 61.87) - open/close 41491 (time 1786020916.52 total 1972.64 last 46.41) - open/close 42005 (time 1786020926.52 total 1982.65 last 51.38) - open/close 42461 (time 1786020936.54 total 1992.67 last 45.52) - open/close 42909 (time 1786020946.54 total 2002.67 last 44.80) - open/close 43301 (time 1786020956.55 total 2012.68 last 39.16) - open/close 43752 (time 1786020966.58 total 2022.70 last 44.99) - open/close 44163 (time 1786020976.59 total 2032.71 last 41.06) - open/close 44584 (time 1786020986.61 total 2042.73 last 42.02) - open/close 45042 (time 1786020996.66 total 2052.79 last 45.54) - open/close 45344 (time 1786021006.68 total 2062.80 last 30.17) - open/close 45780 (time 1786021016.68 total 2072.81 last 43.57) - open/close 46152 (time 1786021026.69 total 2082.81 last 37.19) - open/close 46589 (time 1786021036.69 total 2092.82 last 43.67) - open/close 46944 (time 1786021046.69 total 2102.82 last 35.49) - open/close 47307 (time 1786021056.70 total 2112.82 last 36.27) - open/close 47758 (time 1786021066.71 total 2122.83 last 45.07) - open/close 48108 (time 1786021076.72 total 2132.84 last 34.97) - open/close 48419 (time 1786021086.75 total 2142.88 last 30.99) - open/close 48768 (time 1786021096.76 total 2152.89 last 34.86) - open/close 49190 (time 1786021106.81 total 2162.93 last 42.00) - open/close 49619 (time 1786021116.83 total 2172.95 last 42.81) total: 50000 open/close in 2181.60 seconds: 22.92 ops/second - unlinked 0 (time 1786021131 ; total 0 ; last 0) - unlinked 10000 (time 1786021351 ; total 220 ; last 220) - unlinked 20000 (time 1786021583 ; total 452 ; last 232) - unlinked 30000 (time 1786021786 ; total 655 ; last 203) - unlinked 40000 (time 1786021998 ; total 867 ; last 212) total: 50000 unlinks in 1076 seconds: 46.468403 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server unloading modules via unload_modules_local on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local oleg441-server: modules unloaded. pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2