== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:59:24 (1781240364) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 74 (time 1781240444.89 total 10.78 last 6.86) - open/close 222 (time 1781240455.33 total 21.22 last 14.18) - open/close 398 (time 1781240465.54 total 31.43 last 17.24) - open/close 653 (time 1781240475.68 total 41.58 last 25.13) - open/close 859 (time 1781240485.69 total 51.58 last 20.60) - open/close 1136 (time 1781240495.69 total 61.58 last 27.69) - open/close 1390 (time 1781240506.28 total 72.17 last 23.98) - open/close 1662 (time 1781240516.89 total 82.78 last 25.64) - open/close 1924 (time 1781240527.85 total 93.74 last 23.91) - open/close 2190 (time 1781240537.86 total 103.75 last 26.58) - open/close 2401 (time 1781240548.90 total 114.79 last 19.11) - open/close 2650 (time 1781240558.91 total 124.80 last 24.87) - open/close 2811 (time 1781240569.66 total 135.55 last 14.98) - open/close 3032 (time 1781240580.66 total 146.55 last 20.09) - open/close 3257 (time 1781240590.70 total 156.60 last 22.40) - open/close 3481 (time 1781240600.71 total 166.61 last 22.38) - open/close 3553 (time 1781240610.76 total 176.65 last 7.17) - open/close 3749 (time 1781240621.47 total 187.36 last 18.30) - open/close 3911 (time 1781240631.98 total 197.88 last 15.41) - open/close 4062 (time 1781240642.86 total 208.75 last 13.88) - open/close 4286 (time 1781240653.67 total 219.56 last 20.72) - open/close 4480 (time 1781240663.68 total 229.57 last 19.38) - open/close 4619 (time 1781240673.76 total 239.65 last 13.79) - open/close 4838 (time 1781240684.12 total 250.01 last 21.14) - open/close 5063 (time 1781240694.69 total 260.58 last 21.28) - open/close 5274 (time 1781240705.44 total 271.33 last 19.63) - open/close 5473 (time 1781240715.84 total 281.73 last 19.13) - open/close 5748 (time 1781240726.36 total 292.25 last 26.14) - open/close 6005 (time 1781240736.37 total 302.26 last 25.69) - open/close 6274 (time 1781240747.38 total 313.28 last 24.42) - open/close 6487 (time 1781240757.86 total 323.75 last 20.34) - open/close 6706 (time 1781240768.36 total 334.25 last 20.86) - open/close 6785 (time 1781240778.66 total 344.56 last 7.66) - open/close 6901 (time 1781240788.68 total 354.57 last 11.59) - open/close 7144 (time 1781240798.68 total 364.57 last 24.28) - open/close 7364 (time 1781240810.03 total 375.92 last 19.39) - open/close 7495 (time 1781240820.07 total 385.96 last 13.04) - open/close 7691 (time 1781240830.63 total 396.52 last 18.56) - open/close 7872 (time 1781240840.65 total 406.54 last 18.07) - open/close 8038 (time 1781240850.99 total 416.88 last 16.06) - open/close 8209 (time 1781240861.79 total 427.68 last 15.83) - open/close 8433 (time 1781240872.28 total 438.17 last 21.36) - open/close 8554 (time 1781240882.28 total 448.17 last 12.10) - open/close 8768 (time 1781240892.84 total 458.73 last 20.26) - open/close 9030 (time 1781240902.85 total 468.74 last 26.17) - open/close 9154 (time 1781240913.29 total 479.18 last 11.88) - open/close 9275 (time 1781240923.29 total 489.18 last 12.10) - open/close 9515 (time 1781240936.17 total 502.06 last 18.64) - open/close 9611 (time 1781240946.60 total 512.49 last 9.21) - open/close 9747 (time 1781240957.62 total 523.51 last 12.33) - open/close 9957 (time 1781240967.64 total 533.53 last 20.97) - open/close 10000 (time 1781240969.66 total 535.55 last 21.21) - open/close 10255 (time 1781240980.52 total 546.41 last 23.49) - open/close 10516 (time 1781240990.53 total 556.42 last 26.06) - open/close 10715 (time 1781241000.54 total 566.43 last 19.88) - open/close 10944 (time 1781241010.67 total 576.56 last 22.61) - open/close 11137 (time 1781241020.98 total 586.87 last 18.72) - open/close 11338 (time 1781241031.32 total 597.21 last 19.43) - open/close 11467 (time 1781241043.37 total 609.26 last 10.71) - open/close 11686 (time 1781241053.68 total 619.57 last 21.23) - open/close 11877 (time 1781241064.62 total 630.51 last 17.47) - open/close 12133 (time 1781241075.68 total 641.57 last 23.14) - open/close 12313 (time 1781241086.10 total 651.99 last 17.28) - open/close 12462 (time 1781241096.41 total 662.30 last 14.45) - open/close 12700 (time 1781241106.41 total 672.30 last 23.79) - open/close 12928 (time 1781241119.28 total 685.17 last 17.72) - open/close 13173 (time 1781241129.28 total 695.17 last 24.49) - open/close 13421 (time 1781241139.29 total 705.18 last 24.77) - open/close 13591 (time 1781241149.99 total 715.88 last 15.90) - open/close 13829 (time 1781241160.56 total 726.45 last 22.51) - open/close 13958 (time 1781241171.02 total 736.91 last 12.34) - open/close 14053 (time 1781241181.56 total 747.45 last 9.01) - open/close 14230 (time 1781241191.57 total 757.46 last 17.68) - open/close 14449 (time 1781241201.59 total 767.48 last 21.87) - open/close 14712 (time 1781241211.84 total 777.73 last 25.66) - open/close 14989 (time 1781241222.26 total 788.15 last 26.57) - open/close 15308 (time 1781241232.55 total 798.44 last 30.99) - open/close 15518 (time 1781241246.83 total 812.72 last 14.71) - open/close 15754 (time 1781241257.51 total 823.40 last 22.10) - open/close 16015 (time 1781241268.00 total 833.89 last 24.87) - open/close 16282 (time 1781241278.86 total 844.75 last 24.59) - open/close 16560 (time 1781241289.14 total 855.03 last 27.04) - open/close 16811 (time 1781241299.14 total 865.03 last 25.09) - open/close 16995 (time 1781241309.15 total 875.04 last 18.39) - open/close 17203 (time 1781241319.78 total 885.67 last 19.56) - open/close 17429 (time 1781241330.67 total 896.56 last 20.75) - open/close 17696 (time 1781241340.68 total 906.57 last 26.68) - open/close 17837 (time 1781241350.95 total 916.84 last 13.72) - open/close 18064 (time 1781241361.97 total 927.86 last 20.60) - open/close 18190 (time 1781241372.79 total 938.68 last 11.65) - open/close 18486 (time 1781241383.79 total 949.68 last 26.91) - open/close 18835 (time 1781241393.80 total 959.69 last 34.86) - open/close 19687 (time 1781241403.81 total 969.70 last 85.14) - open/close 20000 (time 1781241407.95 total 973.85 last 75.46) - open/close 20815 (time 1781241417.96 total 983.85 last 81.44) - open/close 21628 (time 1781241427.97 total 993.86 last 81.26) - open/close 22470 (time 1781241437.98 total 1003.87 last 84.09) - open/close 23269 (time 1781241447.99 total 1013.88 last 79.85) - open/close 23982 (time 1781241458.00 total 1023.89 last 71.22) - open/close 24692 (time 1781241468.00 total 1033.89 last 70.98) - open/close 25535 (time 1781241478.01 total 1043.90 last 84.24) - open/close 26316 (time 1781241488.01 total 1053.90 last 78.09) - open/close 27113 (time 1781241498.02 total 1063.91 last 79.63) - open/close 27906 (time 1781241508.03 total 1073.92 last 79.24) - open/close 28695 (time 1781241518.03 total 1083.92 last 78.87) - open/close 29429 (time 1781241528.03 total 1093.92 last 73.38) - open/close 30000 (time 1781241536.54 total 1102.43 last 67.12) - open/close 30897 (time 1781241546.54 total 1112.43 last 89.69) - open/close 31685 (time 1781241556.55 total 1122.44 last 78.73) - open/close 32413 (time 1781241566.55 total 1132.44 last 72.77) - open/close 33199 (time 1781241576.58 total 1142.47 last 78.39) - open/close 34000 (time 1781241586.59 total 1152.49 last 79.98) - open/close 34814 (time 1781241596.60 total 1162.49 last 81.38) - open/close 35695 (time 1781241606.61 total 1172.50 last 88.02) - open/close 36415 (time 1781241616.61 total 1182.50 last 71.97) - open/close 37201 (time 1781241626.64 total 1192.53 last 78.39) - open/close 37900 (time 1781241636.64 total 1202.53 last 69.87) - open/close 38559 (time 1781241646.64 total 1212.54 last 65.87) - open/close 39318 (time 1781241656.65 total 1222.54 last 75.89) - open/close 40000 (time 1781241665.60 total 1231.49 last 76.17) - open/close 40733 (time 1781241675.67 total 1241.56 last 72.76) - open/close 41391 (time 1781241685.69 total 1251.58 last 65.71) - open/close 41991 (time 1781241695.69 total 1261.59 last 59.96) - open/close 42637 (time 1781241705.70 total 1271.60 last 64.53) - open/close 43473 (time 1781241715.71 total 1281.60 last 83.58) - open/close 44184 (time 1781241725.71 total 1291.60 last 71.06) - open/close 44990 (time 1781241735.72 total 1301.61 last 80.57) - open/close 45782 (time 1781241745.73 total 1311.62 last 79.08) - open/close 46652 (time 1781241755.74 total 1321.63 last 86.95) - open/close 47535 (time 1781241765.74 total 1331.64 last 88.23) - open/close 48359 (time 1781241775.76 total 1341.65 last 82.30) - open/close 49163 (time 1781241785.76 total 1351.65 last 80.38) - open/close 49926 (time 1781241795.76 total 1361.66 last 76.26) total: 50000 open/close in 1362.86 seconds: 36.69 ops/second - unlinked 0 (time 1781241800 ; total 0 ; last 0) - unlinked 10000 (time 1781241934 ; total 134 ; last 134) - unlinked 20000 (time 1781242070 ; total 270 ; last 136) - unlinked 30000 (time 1781242203 ; total 403 ; last 133) - unlinked 40000 (time 1781242337 ; total 537 ; last 134) total: 50000 unlinks in 691 seconds: 72.358902 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2