== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:11:37 (1784189497) start mds service on oleg348-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg348-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1784189579.37 total 10.68 last 17.14) - open/close 408 (time 1784189589.37 total 20.68 last 22.50) - open/close 533 (time 1784189599.38 total 30.69 last 12.48) - open/close 708 (time 1784189610.28 total 41.58 last 16.07) - open/close 878 (time 1784189620.31 total 51.61 last 16.95) - open/close 1018 (time 1784189630.47 total 61.78 last 13.78) - open/close 1201 (time 1784189640.48 total 71.79 last 18.28) - open/close 1462 (time 1784189650.97 total 82.28 last 24.86) - open/close 1639 (time 1784189661.04 total 92.35 last 17.59) - open/close 1852 (time 1784189671.86 total 103.16 last 19.69) - open/close 1986 (time 1784189682.85 total 114.16 last 12.19) - open/close 2128 (time 1784189692.91 total 124.21 last 14.12) - open/close 2368 (time 1784189702.92 total 134.23 last 23.98) - open/close 2547 (time 1784189713.77 total 145.08 last 16.49) - open/close 2813 (time 1784189724.88 total 156.19 last 23.95) - open/close 3019 (time 1784189736.27 total 167.58 last 18.08) - open/close 3270 (time 1784189746.86 total 178.17 last 23.70) - open/close 3513 (time 1784189757.48 total 188.79 last 22.89) - open/close 3769 (time 1784189768.22 total 199.53 last 23.82) - open/close 3974 (time 1784189778.59 total 209.90 last 19.77) - open/close 4166 (time 1784189789.46 total 220.77 last 17.66) - open/close 4256 (time 1784189799.99 total 231.30 last 8.54) - open/close 4464 (time 1784189810.74 total 242.05 last 19.35) - open/close 4575 (time 1784189823.57 total 254.88 last 8.65) - open/close 4708 (time 1784189834.24 total 265.55 last 12.47) - open/close 4830 (time 1784189844.25 total 275.56 last 12.18) - open/close 4983 (time 1784189855.03 total 286.34 last 14.20) - open/close 5193 (time 1784189865.77 total 297.08 last 19.55) - open/close 5295 (time 1784189875.79 total 307.10 last 10.18) - open/close 5499 (time 1784189885.80 total 317.11 last 20.38) - open/close 5625 (time 1784189895.81 total 327.12 last 12.60) - open/close 5772 (time 1784189906.86 total 338.17 last 13.30) - open/close 5997 (time 1784189916.87 total 348.18 last 22.47) - open/close 6259 (time 1784189926.88 total 358.19 last 26.17) - open/close 6556 (time 1784189937.77 total 369.08 last 27.28) - open/close 6727 (time 1784189948.47 total 379.78 last 15.98) - open/close 6982 (time 1784189958.48 total 389.79 last 25.47) - open/close 7217 (time 1784189968.48 total 399.79 last 23.49) - open/close 7379 (time 1784189979.19 total 410.50 last 15.14) - open/close 7567 (time 1784189989.19 total 420.50 last 18.79) - open/close 7795 (time 1784189999.78 total 431.09 last 21.54) - open/close 7982 (time 1784190010.67 total 441.98 last 17.16) - open/close 8245 (time 1784190020.98 total 452.29 last 25.52) - open/close 8357 (time 1784190031.78 total 463.09 last 10.37) - open/close 8465 (time 1784190042.22 total 473.53 last 10.35) - open/close 8679 (time 1784190052.83 total 484.14 last 20.16) - open/close 8927 (time 1784190062.85 total 494.16 last 24.76) - open/close 9127 (time 1784190072.86 total 504.17 last 19.98) - open/close 9346 (time 1784190082.87 total 514.18 last 21.88) - open/close 9500 (time 1784190093.49 total 524.80 last 14.50) - open/close 9678 (time 1784190103.54 total 534.85 last 17.72) - open/close 9805 (time 1784190113.88 total 545.19 last 12.27) - open/close 9944 (time 1784190124.02 total 555.33 last 13.71) - open/close 10000 (time 1784190127.31 total 558.61 last 17.05) - open/close 10203 (time 1784190137.32 total 568.63 last 20.27) - open/close 10401 (time 1784190147.57 total 578.88 last 19.32) - open/close 10640 (time 1784190157.59 total 588.90 last 23.85) - open/close 10887 (time 1784190168.10 total 599.41 last 23.49) - open/close 11087 (time 1784190178.11 total 609.42 last 19.99) - open/close 11340 (time 1784190188.13 total 619.44 last 25.26) - open/close 11587 (time 1784190198.19 total 629.50 last 24.55) - open/close 11836 (time 1784190208.19 total 639.50 last 24.88) - open/close 11971 (time 1784190218.42 total 649.73 last 13.20) - open/close 12212 (time 1784190229.25 total 660.56 last 22.26) - open/close 12452 (time 1784190239.84 total 671.15 last 22.66) - open/close 12654 (time 1784190250.99 total 682.30 last 18.11) - open/close 12934 (time 1784190261.02 total 692.33 last 27.93) - open/close 13132 (time 1784190271.72 total 703.03 last 18.49) - open/close 13364 (time 1784190281.73 total 713.04 last 23.18) - open/close 13600 (time 1784190291.88 total 723.19 last 23.27) - open/close 13785 (time 1784190301.89 total 733.20 last 18.47) - open/close 13987 (time 1784190311.91 total 743.22 last 20.17) - open/close 14207 (time 1784190321.91 total 753.22 last 21.99) - open/close 14438 (time 1784190332.08 total 763.39 last 22.71) - open/close 14655 (time 1784190342.19 total 773.50 last 21.47) - open/close 14826 (time 1784190352.88 total 784.19 last 15.99) - open/close 15017 (time 1784190362.89 total 794.20 last 19.09) - open/close 15241 (time 1784190373.87 total 805.18 last 20.39) - open/close 15476 (time 1784190384.50 total 815.81 last 22.10) - open/close 15728 (time 1784190394.95 total 826.26 last 24.13) - open/close 15949 (time 1784190405.82 total 837.12 last 20.34) - open/close 16145 (time 1784190415.82 total 847.13 last 19.59) - open/close 16314 (time 1784190426.35 total 857.66 last 16.05) - open/close 16521 (time 1784190437.31 total 868.62 last 18.89) - open/close 16767 (time 1784190448.18 total 879.49 last 22.64) - open/close 17020 (time 1784190458.20 total 889.51 last 25.24) - open/close 17245 (time 1784190469.17 total 900.48 last 20.52) - open/close 17491 (time 1784190479.17 total 910.48 last 24.58) - open/close 17684 (time 1784190489.19 total 920.50 last 19.27) - open/close 17851 (time 1784190499.20 total 930.51 last 16.68) - open/close 18025 (time 1784190509.21 total 940.52 last 17.38) - open/close 18218 (time 1784190519.42 total 950.73 last 18.90) - open/close 18325 (time 1784190530.04 total 961.34 last 10.08) - open/close 18750 (time 1784190540.04 total 971.35 last 42.47) - open/close 19343 (time 1784190550.10 total 981.41 last 58.96) - open/close 19942 (time 1784190560.13 total 991.43 last 59.75) - open/close 20574 (time 1784190570.13 total 1001.44 last 63.19) - open/close 21238 (time 1784190580.13 total 1011.44 last 66.38) - open/close 21903 (time 1784190590.13 total 1021.44 last 66.49) - open/close 22413 (time 1784190600.17 total 1031.48 last 50.82) - open/close 22973 (time 1784190610.18 total 1041.49 last 55.94) - open/close 23607 (time 1784190620.18 total 1051.49 last 63.39) - open/close 24213 (time 1784190630.18 total 1061.49 last 60.58) - open/close 24822 (time 1784190640.19 total 1071.50 last 60.88) - open/close 25392 (time 1784190650.20 total 1081.51 last 56.90) - open/close 26091 (time 1784190660.21 total 1091.52 last 69.87) - open/close 26746 (time 1784190670.21 total 1101.52 last 65.46) - open/close 27375 (time 1784190680.22 total 1111.53 last 62.84) - open/close 28075 (time 1784190690.22 total 1121.53 last 69.99) - open/close 28774 (time 1784190700.23 total 1131.54 last 69.83) - open/close 29437 (time 1784190710.24 total 1141.55 last 66.29) - open/close 30000 (time 1784190719.88 total 1151.19 last 58.39) - open/close 30625 (time 1784190729.88 total 1161.19 last 62.49) - open/close 31280 (time 1784190739.89 total 1171.20 last 65.44) - open/close 31913 (time 1784190749.90 total 1181.21 last 63.20) - open/close 32543 (time 1784190759.91 total 1191.22 last 62.99) - open/close 33162 (time 1784190769.91 total 1201.22 last 61.89) - open/close 33835 (time 1784190779.91 total 1211.22 last 67.30) - open/close 34470 (time 1784190789.91 total 1221.22 last 63.49) - open/close 35127 (time 1784190799.93 total 1231.24 last 65.55) - open/close 35741 (time 1784190809.94 total 1241.25 last 61.34) - open/close 36371 (time 1784190819.95 total 1251.26 last 62.96) - open/close 37047 (time 1784190829.96 total 1261.27 last 67.54) - open/close 37680 (time 1784190839.97 total 1271.28 last 63.22) - open/close 38302 (time 1784190849.97 total 1281.28 last 62.20) - open/close 38882 (time 1784190859.98 total 1291.29 last 57.93) - open/close 39502 (time 1784190869.99 total 1301.30 last 61.97) - open/close 40000 (time 1784190878.47 total 1309.78 last 58.72) - open/close 40564 (time 1784190888.48 total 1319.79 last 56.34) - open/close 41253 (time 1784190898.48 total 1329.79 last 68.88) - open/close 41946 (time 1784190908.50 total 1339.81 last 69.16) - open/close 42621 (time 1784190918.51 total 1349.82 last 67.42) - open/close 43220 (time 1784190928.52 total 1359.83 last 59.84) - open/close 43876 (time 1784190938.54 total 1369.85 last 65.51) - open/close 44537 (time 1784190948.55 total 1379.85 last 66.05) - open/close 45212 (time 1784190958.56 total 1389.87 last 67.40) - open/close 45899 (time 1784190968.56 total 1399.87 last 68.67) - open/close 46500 (time 1784190978.57 total 1409.88 last 60.06) - open/close 47088 (time 1784190988.59 total 1419.90 last 58.70) - open/close 47712 (time 1784190998.60 total 1429.91 last 62.34) - open/close 48328 (time 1784191008.60 total 1439.91 last 61.57) - open/close 48980 (time 1784191018.60 total 1449.91 last 65.19) - open/close 49620 (time 1784191028.60 total 1459.91 last 64.00) total: 50000 open/close in 1465.40 seconds: 34.12 ops/second - unlinked 0 (time 1784191038 ; total 0 ; last 0) - unlinked 10000 (time 1784191196 ; total 158 ; last 158) - unlinked 20000 (time 1784191357 ; total 319 ; last 161) - unlinked 30000 (time 1784191522 ; total 484 ; last 165) - unlinked 40000 (time 1784191678 ; total 640 ; last 156) total: 50000 unlinks in 794 seconds: 62.972294 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: modules unloaded. pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2