== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:56:49 (1782223009) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1782223089.50 total 10.02 last 18.17) - open/close 425 (time 1782223099.52 total 20.04 last 24.25) - open/close 667 (time 1782223109.62 total 30.14 last 23.96) - open/close 831 (time 1782223120.12 total 40.64 last 15.62) - open/close 1022 (time 1782223130.68 total 51.20 last 18.08) - open/close 1191 (time 1782223141.04 total 61.56 last 16.32) - open/close 1263 (time 1782223151.07 total 71.59 last 7.18) - open/close 1340 (time 1782223161.09 total 81.61 last 7.68) - open/close 1421 (time 1782223173.45 total 93.97 last 6.55) - open/close 1632 (time 1782223183.45 total 103.97 last 21.08) - open/close 1744 (time 1782223193.47 total 113.99 last 11.18) - open/close 1989 (time 1782223203.48 total 124.00 last 24.46) - open/close 2120 (time 1782223213.49 total 134.01 last 13.09) - open/close 2222 (time 1782223224.05 total 144.57 last 9.66) - open/close 2414 (time 1782223234.30 total 154.82 last 18.73) - open/close 2618 (time 1782223247.23 total 167.75 last 15.78) - open/close 2842 (time 1782223259.78 total 180.30 last 17.84) - open/close 3072 (time 1782223269.79 total 190.31 last 23.00) - open/close 3237 (time 1782223280.05 total 200.57 last 16.07) - open/close 3383 (time 1782223290.08 total 210.60 last 14.56) - open/close 3459 (time 1782223301.37 total 221.89 last 6.73) - open/close 3610 (time 1782223313.14 total 233.66 last 12.84) - open/close 3714 (time 1782223323.97 total 244.49 last 9.60) - open/close 3903 (time 1782223334.33 total 254.85 last 18.24) - open/close 4009 (time 1782223345.07 total 265.59 last 9.87) - open/close 4216 (time 1782223355.08 total 275.60 last 20.68) - open/close 4377 (time 1782223365.89 total 286.41 last 14.89) - open/close 4484 (time 1782223375.90 total 296.42 last 10.69) - open/close 4684 (time 1782223386.42 total 306.94 last 19.00) - open/close 4872 (time 1782223396.92 total 317.44 last 17.91) - open/close 5090 (time 1782223407.64 total 328.16 last 20.34) - open/close 5271 (time 1782223417.65 total 338.17 last 18.08) - open/close 5486 (time 1782223427.65 total 348.17 last 21.48) - open/close 5698 (time 1782223438.42 total 358.94 last 19.70) - open/close 5960 (time 1782223448.43 total 368.95 last 26.16) - open/close 6120 (time 1782223458.61 total 379.13 last 15.73) - open/close 6321 (time 1782223468.61 total 389.13 last 20.09) - open/close 6522 (time 1782223478.66 total 399.18 last 20.00) - open/close 6725 (time 1782223489.09 total 409.61 last 19.47) - open/close 6920 (time 1782223499.11 total 419.63 last 19.45) - open/close 7177 (time 1782223509.24 total 429.76 last 25.37) - open/close 7414 (time 1782223520.06 total 440.58 last 21.90) - open/close 7627 (time 1782223530.30 total 450.82 last 20.82) - open/close 7896 (time 1782223540.31 total 460.83 last 26.87) - open/close 8111 (time 1782223550.60 total 471.12 last 20.89) - open/close 8357 (time 1782223561.27 total 481.79 last 23.06) - open/close 8476 (time 1782223571.27 total 491.79 last 11.89) - open/close 8603 (time 1782223581.30 total 501.82 last 12.67) - open/close 8802 (time 1782223591.81 total 512.33 last 18.93) - open/close 8973 (time 1782223602.56 total 523.08 last 15.90) - open/close 9201 (time 1782223613.19 total 533.71 last 21.45) - open/close 9447 (time 1782223624.25 total 544.77 last 22.24) - open/close 9673 (time 1782223634.26 total 554.78 last 22.58) - open/close 9870 (time 1782223645.18 total 565.70 last 18.04) - open/close 10000 (time 1782223650.76 total 571.28 last 23.29) - open/close 10211 (time 1782223661.57 total 582.09 last 19.51) - open/close 10378 (time 1782223671.59 total 592.11 last 16.67) - open/close 10495 (time 1782223683.34 total 603.86 last 9.96) - open/close 10687 (time 1782223694.39 total 614.91 last 17.38) - open/close 10844 (time 1782223705.27 total 625.79 last 14.43) - open/close 11016 (time 1782223716.28 total 636.80 last 15.61) - open/close 11149 (time 1782223726.64 total 647.16 last 12.84) - open/close 11325 (time 1782223736.65 total 657.17 last 17.59) - open/close 11460 (time 1782223746.67 total 667.19 last 13.48) - open/close 11689 (time 1782223757.57 total 678.09 last 21.01) - open/close 11926 (time 1782223767.59 total 688.11 last 23.66) - open/close 12112 (time 1782223777.98 total 698.50 last 17.90) - open/close 12355 (time 1782223788.48 total 709.00 last 23.14) - open/close 12573 (time 1782223798.49 total 719.01 last 21.77) - open/close 12784 (time 1782223808.63 total 729.15 last 20.81) - open/close 12995 (time 1782223818.64 total 739.16 last 21.08) - open/close 13224 (time 1782223828.65 total 749.17 last 22.87) - open/close 13448 (time 1782223838.67 total 759.19 last 22.36) - open/close 13658 (time 1782223848.68 total 769.20 last 20.98) - open/close 13900 (time 1782223858.68 total 779.20 last 24.20) - open/close 14112 (time 1782223869.49 total 790.01 last 19.62) - open/close 14347 (time 1782223879.49 total 800.01 last 23.49) - open/close 14547 (time 1782223889.50 total 810.02 last 19.98) - open/close 14682 (time 1782223900.34 total 820.86 last 12.45) - open/close 14940 (time 1782223910.35 total 830.87 last 25.79) - open/close 15147 (time 1782223920.35 total 840.87 last 20.69) - open/close 15328 (time 1782223930.88 total 851.40 last 17.20) - open/close 15574 (time 1782223940.89 total 861.41 last 24.58) - open/close 15795 (time 1782223950.89 total 871.41 last 22.09) - open/close 15923 (time 1782223961.23 total 881.75 last 12.38) - open/close 16047 (time 1782223971.28 total 891.80 last 12.35) - open/close 16255 (time 1782223981.90 total 902.42 last 19.59) - open/close 16462 (time 1782223992.12 total 912.64 last 20.25) - open/close 16703 (time 1782224002.29 total 922.81 last 23.68) - open/close 16858 (time 1782224012.30 total 932.82 last 15.49) - open/close 17094 (time 1782224022.65 total 943.17 last 22.80) - open/close 17326 (time 1782224032.66 total 953.18 last 23.20) - open/close 17931 (time 1782224042.66 total 963.18 last 60.44) - open/close 18614 (time 1782224052.67 total 973.19 last 68.25) - open/close 19273 (time 1782224062.68 total 983.20 last 65.86) - open/close 19873 (time 1782224072.69 total 993.21 last 59.94) - open/close 20507 (time 1782224082.70 total 1003.22 last 63.30) - open/close 21080 (time 1782224092.72 total 1013.24 last 57.20) - open/close 21613 (time 1782224102.74 total 1023.26 last 53.21) - open/close 22277 (time 1782224112.74 total 1033.26 last 66.38) - open/close 22960 (time 1782224122.75 total 1043.27 last 68.22) - open/close 23572 (time 1782224132.76 total 1053.28 last 61.16) - open/close 24209 (time 1782224142.77 total 1063.29 last 63.65) - open/close 24877 (time 1782224152.80 total 1073.32 last 66.58) - open/close 25528 (time 1782224162.80 total 1083.32 last 65.09) - open/close 26151 (time 1782224172.81 total 1093.33 last 62.26) - open/close 26829 (time 1782224182.82 total 1103.34 last 67.72) - open/close 27479 (time 1782224192.82 total 1113.34 last 64.97) - open/close 28166 (time 1782224202.83 total 1123.35 last 68.66) - open/close 28801 (time 1782224212.83 total 1133.35 last 63.50) - open/close 29487 (time 1782224222.84 total 1143.36 last 68.53) - open/close 30000 (time 1782224230.84 total 1151.36 last 64.15) - open/close 30656 (time 1782224240.85 total 1161.37 last 65.49) - open/close 31231 (time 1782224250.86 total 1171.38 last 57.45) - open/close 31839 (time 1782224260.87 total 1181.39 last 60.72) - open/close 32501 (time 1782224270.89 total 1191.41 last 66.12) - open/close 33051 (time 1782224280.89 total 1201.41 last 54.96) - open/close 33713 (time 1782224290.90 total 1211.42 last 66.16) - open/close 34288 (time 1782224300.91 total 1221.43 last 57.43) - open/close 34871 (time 1782224310.92 total 1231.44 last 58.23) - open/close 35506 (time 1782224320.93 total 1241.45 last 63.46) - open/close 36135 (time 1782224330.94 total 1251.46 last 62.85) - open/close 36745 (time 1782224340.95 total 1261.47 last 60.93) - open/close 37321 (time 1782224350.96 total 1271.48 last 57.53) - open/close 37933 (time 1782224360.96 total 1281.48 last 61.18) - open/close 38557 (time 1782224370.98 total 1291.50 last 62.30) - open/close 39220 (time 1782224380.98 total 1301.50 last 66.30) - open/close 39889 (time 1782224390.98 total 1311.50 last 66.89) - open/close 40536 (time 1782224401.00 total 1321.52 last 64.57) - open/close 41173 (time 1782224411.01 total 1331.53 last 63.67) - open/close 41786 (time 1782224421.02 total 1341.54 last 61.24) - open/close 42368 (time 1782224431.02 total 1351.54 last 58.17) - open/close 42996 (time 1782224441.02 total 1361.54 last 62.80) - open/close 43686 (time 1782224451.03 total 1371.55 last 68.93) - open/close 44254 (time 1782224461.04 total 1381.56 last 56.78) - open/close 44924 (time 1782224471.04 total 1391.56 last 66.96) - open/close 45580 (time 1782224481.06 total 1401.58 last 65.50) - open/close 46216 (time 1782224491.14 total 1411.66 last 63.11) - open/close 46848 (time 1782224501.14 total 1421.66 last 63.15) - open/close 47466 (time 1782224511.15 total 1431.67 last 61.74) - open/close 48083 (time 1782224521.15 total 1441.67 last 61.69) - open/close 48712 (time 1782224531.15 total 1451.67 last 62.90) - open/close 49373 (time 1782224541.17 total 1461.69 last 66.01) total: 50000 open/close in 1471.57 seconds: 33.98 ops/second - unlinked 0 (time 1782224555 ; total 0 ; last 0) - unlinked 10000 (time 1782224714 ; total 159 ; last 159) - unlinked 20000 (time 1782224888 ; total 333 ; last 174) - unlinked 30000 (time 1782225068 ; total 513 ; last 180) - unlinked 40000 (time 1782225250 ; total 695 ; last 182) total: 50000 unlinks in 869 seconds: 57.537399 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2