== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:51:24 (1788443484) start mds service on oleg215-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 191 (time 1788443566.39 total 10.23 last 18.66) - open/close 267 (time 1788443576.78 total 20.62 last 7.31) - open/close 329 (time 1788443586.79 total 30.63 last 6.20) - open/close 531 (time 1788443597.84 total 41.68 last 18.27) - open/close 772 (time 1788443608.52 total 52.36 last 22.57) - open/close 1042 (time 1788443619.20 total 63.04 last 25.26) - open/close 1172 (time 1788443629.20 total 73.04 last 13.00) - open/close 1377 (time 1788443640.09 total 83.93 last 18.83) - open/close 1607 (time 1788443650.09 total 93.93 last 23.00) - open/close 1878 (time 1788443660.63 total 104.47 last 25.73) - open/close 2161 (time 1788443670.65 total 114.49 last 28.24) - open/close 2391 (time 1788443680.66 total 124.50 last 22.98) - open/close 2645 (time 1788443690.66 total 134.50 last 25.39) - open/close 2879 (time 1788443700.67 total 144.51 last 23.39) - open/close 3114 (time 1788443710.97 total 154.82 last 22.80) - open/close 3306 (time 1788443720.98 total 164.82 last 19.19) - open/close 3416 (time 1788443731.82 total 175.66 last 10.15) - open/close 3588 (time 1788443742.86 total 186.70 last 15.58) - open/close 3721 (time 1788443752.89 total 196.74 last 13.25) - open/close 3865 (time 1788443763.47 total 207.31 last 13.62) - open/close 4031 (time 1788443773.47 total 217.31 last 16.59) - open/close 4253 (time 1788443783.50 total 227.34 last 22.15) - open/close 4420 (time 1788443793.69 total 237.53 last 16.39) - open/close 4635 (time 1788443805.90 total 249.74 last 17.61) - open/close 4887 (time 1788443815.91 total 259.75 last 25.18) - open/close 5043 (time 1788443826.46 total 270.30 last 14.78) - open/close 5234 (time 1788443836.47 total 280.31 last 19.08) - open/close 5392 (time 1788443847.13 total 290.97 last 14.83) - open/close 5572 (time 1788443858.06 total 301.90 last 16.47) - open/close 5733 (time 1788443868.29 total 312.13 last 15.74) - open/close 5946 (time 1788443879.11 total 322.95 last 19.69) - open/close 6215 (time 1788443890.06 total 333.90 last 24.56) - open/close 6412 (time 1788443900.86 total 344.70 last 18.25) - open/close 6551 (time 1788443911.34 total 355.18 last 13.26) - open/close 6719 (time 1788443921.34 total 365.18 last 16.80) - open/close 6754 (time 1788443931.50 total 375.34 last 3.45) - open/close 6887 (time 1788443941.85 total 385.70 last 12.85) - open/close 6989 (time 1788443952.33 total 396.17 last 9.74) - open/close 7192 (time 1788443962.83 total 406.67 last 19.33) - open/close 7498 (time 1788443972.83 total 416.67 last 30.58) - open/close 7726 (time 1788443982.85 total 426.70 last 22.75) - open/close 7930 (time 1788443992.97 total 436.81 last 20.18) - open/close 8154 (time 1788444003.48 total 447.32 last 21.31) - open/close 8373 (time 1788444013.70 total 457.54 last 21.41) - open/close 8631 (time 1788444023.72 total 467.56 last 25.77) - open/close 8895 (time 1788444033.87 total 477.71 last 26.00) - open/close 9087 (time 1788444044.05 total 487.89 last 18.86) - open/close 9339 (time 1788444054.68 total 498.52 last 23.71) - open/close 9430 (time 1788444065.31 total 509.15 last 8.56) - open/close 9580 (time 1788444075.99 total 519.83 last 14.04) - open/close 9756 (time 1788444086.00 total 529.84 last 17.59) - open/close 9846 (time 1788444096.97 total 540.81 last 8.20) - open/close 9978 (time 1788444106.98 total 550.82 last 13.20) - open/close 10000 (time 1788444109.70 total 553.54 last 8.09) - open/close 10141 (time 1788444120.52 total 564.36 last 13.03) - open/close 10309 (time 1788444132.20 total 576.04 last 14.39) - open/close 10499 (time 1788444142.21 total 586.05 last 18.98) - open/close 10736 (time 1788444152.21 total 596.05 last 23.69) - open/close 11063 (time 1788444162.22 total 606.06 last 32.67) - open/close 11295 (time 1788444172.88 total 616.72 last 21.76) - open/close 11477 (time 1788444182.99 total 626.83 last 18.01) - open/close 11625 (time 1788444195.10 total 638.94 last 12.22) - open/close 11821 (time 1788444206.16 total 650.00 last 17.71) - open/close 12051 (time 1788444216.18 total 660.02 last 22.96) - open/close 12273 (time 1788444226.57 total 670.41 last 21.37) - open/close 12492 (time 1788444236.59 total 680.43 last 21.85) - open/close 12612 (time 1788444246.78 total 690.62 last 11.77) - open/close 12781 (time 1788444257.45 total 701.29 last 15.84) - open/close 12975 (time 1788444268.22 total 712.06 last 18.01) - open/close 13175 (time 1788444278.67 total 722.51 last 19.14) - open/close 13311 (time 1788444288.84 total 732.68 last 13.37) - open/close 13502 (time 1788444298.89 total 742.73 last 19.00) - open/close 13757 (time 1788444308.90 total 752.74 last 25.49) - open/close 13971 (time 1788444318.91 total 762.75 last 21.37) - open/close 14182 (time 1788444328.92 total 772.76 last 21.08) - open/close 14292 (time 1788444339.98 total 783.82 last 9.95) - open/close 14558 (time 1788444350.16 total 794.00 last 26.13) - open/close 14787 (time 1788444360.39 total 804.23 last 22.38) - open/close 14980 (time 1788444370.39 total 814.24 last 19.29) - open/close 15103 (time 1788444380.82 total 824.66 last 11.80) - open/close 15195 (time 1788444391.19 total 835.03 last 8.87) - open/close 15362 (time 1788444401.20 total 845.05 last 16.67) - open/close 15586 (time 1788444411.75 total 855.59 last 21.25) - open/close 15837 (time 1788444421.77 total 865.61 last 25.03) - open/close 16063 (time 1788444431.81 total 875.65 last 22.52) - open/close 16348 (time 1788444441.81 total 885.65 last 28.48) - open/close 16490 (time 1788444451.82 total 895.66 last 14.19) - open/close 16715 (time 1788444462.38 total 906.22 last 21.31) - open/close 16838 (time 1788444472.55 total 916.39 last 12.09) - open/close 17006 (time 1788444483.66 total 927.50 last 15.12) - open/close 17215 (time 1788444493.71 total 937.55 last 20.81) - open/close 17440 (time 1788444503.72 total 947.56 last 22.48) - open/close 17622 (time 1788444514.32 total 958.16 last 17.16) - open/close 17796 (time 1788444524.33 total 968.17 last 17.39) - open/close 18035 (time 1788444535.13 total 978.97 last 22.12) - open/close 18337 (time 1788444545.14 total 988.98 last 30.18) - open/close 19059 (time 1788444555.15 total 998.99 last 72.14) - open/close 19852 (time 1788444565.15 total 1008.99 last 79.26) - open/close 20000 (time 1788444567.37 total 1011.21 last 66.66) - open/close 20747 (time 1788444577.40 total 1021.24 last 74.48) - open/close 21449 (time 1788444587.41 total 1031.25 last 70.13) - open/close 22199 (time 1788444597.41 total 1041.25 last 74.97) - open/close 23012 (time 1788444607.42 total 1051.26 last 81.22) - open/close 23732 (time 1788444617.43 total 1061.27 last 71.94) - open/close 24501 (time 1788444627.44 total 1071.28 last 76.85) - open/close 25318 (time 1788444637.44 total 1081.28 last 81.69) - open/close 26067 (time 1788444647.45 total 1091.29 last 74.83) - open/close 26866 (time 1788444657.46 total 1101.30 last 79.86) - open/close 27636 (time 1788444667.47 total 1111.31 last 76.91) - open/close 28428 (time 1788444677.47 total 1121.31 last 79.16) - open/close 29222 (time 1788444687.47 total 1131.31 last 79.39) - open/close 30000 (time 1788444696.22 total 1140.07 last 88.90) - open/close 30641 (time 1788444706.24 total 1150.08 last 64.01) - open/close 31548 (time 1788444716.25 total 1160.09 last 90.64) - open/close 32539 (time 1788444726.25 total 1170.09 last 99.06) - open/close 33417 (time 1788444736.26 total 1180.10 last 87.69) - open/close 34184 (time 1788444746.27 total 1190.11 last 76.66) - open/close 35031 (time 1788444756.27 total 1200.11 last 84.64) - open/close 35768 (time 1788444766.28 total 1210.12 last 73.66) - open/close 36580 (time 1788444776.30 total 1220.14 last 81.06) - open/close 37211 (time 1788444786.30 total 1230.14 last 63.10) - open/close 37836 (time 1788444796.30 total 1240.14 last 62.47) - open/close 38601 (time 1788444806.31 total 1250.16 last 76.39) - open/close 39368 (time 1788444816.33 total 1260.17 last 76.60) - open/close 40000 (time 1788444823.93 total 1267.77 last 83.10) - open/close 40760 (time 1788444833.94 total 1277.78 last 75.96) - open/close 41475 (time 1788444843.94 total 1287.78 last 71.49) - open/close 42277 (time 1788444853.95 total 1297.79 last 80.15) - open/close 43111 (time 1788444863.95 total 1307.79 last 83.34) - open/close 43905 (time 1788444873.96 total 1317.80 last 79.37) - open/close 44758 (time 1788444883.96 total 1327.80 last 85.29) - open/close 45438 (time 1788444893.97 total 1337.82 last 67.89) - open/close 46229 (time 1788444903.98 total 1347.82 last 79.04) - open/close 47096 (time 1788444913.99 total 1357.83 last 86.63) - open/close 47775 (time 1788444924.00 total 1367.84 last 67.84) - open/close 48460 (time 1788444934.03 total 1377.87 last 68.32) - open/close 48970 (time 1788444944.06 total 1387.90 last 50.81) - open/close 49452 (time 1788444954.07 total 1397.91 last 48.15) total: 50000 open/close in 1404.66 seconds: 35.60 ops/second - unlinked 0 (time 1788444965 ; total 0 ; last 0) - unlinked 10000 (time 1788445096 ; total 131 ; last 131) - unlinked 20000 (time 1788445240 ; total 275 ; last 144) - unlinked 30000 (time 1788445387 ; total 422 ; last 147) - unlinked 40000 (time 1788445545 ; total 580 ; last 158) total: 50000 unlinks in 733 seconds: 68.212822 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2