== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:07:59 (1781586479) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1781586556.06 total 10.03 last 23.92) - open/close 482 (time 1781586566.07 total 20.05 last 24.17) - open/close 715 (time 1781586576.08 total 30.05 last 23.29) - open/close 819 (time 1781586586.65 total 40.62 last 9.84) - open/close 1075 (time 1781586598.72 total 52.69 last 21.21) - open/close 1330 (time 1781586609.57 total 63.55 last 23.48) - open/close 1553 (time 1781586620.26 total 74.24 last 20.86) - open/close 1802 (time 1781586630.26 total 84.24 last 24.90) - open/close 2036 (time 1781586641.27 total 95.24 last 21.26) - open/close 2302 (time 1781586651.32 total 105.29 last 26.47) - open/close 2558 (time 1781586661.33 total 115.30 last 25.57) - open/close 2762 (time 1781586672.25 total 126.22 last 18.69) - open/close 2936 (time 1781586682.25 total 136.23 last 17.39) - open/close 3198 (time 1781586693.04 total 147.02 last 24.28) - open/close 3424 (time 1781586703.29 total 157.26 last 22.06) - open/close 3644 (time 1781586713.29 total 167.26 last 22.00) - open/close 3869 (time 1781586723.29 total 177.27 last 22.49) - open/close 4115 (time 1781586733.57 total 187.54 last 23.95) - open/close 4347 (time 1781586743.79 total 197.76 last 22.69) - open/close 4570 (time 1781586755.83 total 209.80 last 18.52) - open/close 4770 (time 1781586765.83 total 219.80 last 20.00) - open/close 4971 (time 1781586776.19 total 230.16 last 19.41) - open/close 5196 (time 1781586786.35 total 240.33 last 22.13) - open/close 5402 (time 1781586796.65 total 250.63 last 20.00) - open/close 5661 (time 1781586807.02 total 260.99 last 24.99) - open/close 5894 (time 1781586817.66 total 271.63 last 21.90) - open/close 6181 (time 1781586827.66 total 281.64 last 28.68) - open/close 6419 (time 1781586837.69 total 291.66 last 23.75) - open/close 6655 (time 1781586847.69 total 301.66 last 23.60) - open/close 6881 (time 1781586857.70 total 311.67 last 22.57) - open/close 7104 (time 1781586867.71 total 321.68 last 22.28) - open/close 7351 (time 1781586878.19 total 332.17 last 23.57) - open/close 7590 (time 1781586889.21 total 343.18 last 21.70) - open/close 7828 (time 1781586899.58 total 353.55 last 22.94) - open/close 8111 (time 1781586909.95 total 363.92 last 27.29) - open/close 8338 (time 1781586919.96 total 373.93 last 22.68) - open/close 8615 (time 1781586930.74 total 384.71 last 25.70) - open/close 8861 (time 1781586941.24 total 395.21 last 23.42) - open/close 9126 (time 1781586951.93 total 405.90 last 24.79) - open/close 9352 (time 1781586961.95 total 415.92 last 22.56) - open/close 9562 (time 1781586971.95 total 425.93 last 20.99) - open/close 9741 (time 1781586982.72 total 436.70 last 16.62) - open/close 9885 (time 1781586992.82 total 446.80 last 14.26) - open/close 10000 (time 1781586999.17 total 453.14 last 18.11) - open/close 10183 (time 1781587010.05 total 464.02 last 16.82) - open/close 10436 (time 1781587020.59 total 474.57 last 23.99) - open/close 10686 (time 1781587030.89 total 484.86 last 24.29) - open/close 10964 (time 1781587041.82 total 495.80 last 25.42) - open/close 11235 (time 1781587054.08 total 508.05 last 22.12) - open/close 11486 (time 1781587064.08 total 518.05 last 25.09) - open/close 11690 (time 1781587074.12 total 528.09 last 20.32) - open/close 11969 (time 1781587084.42 total 538.39 last 27.09) - open/close 12205 (time 1781587095.25 total 549.23 last 21.78) - open/close 12425 (time 1781587105.26 total 559.23 last 21.99) - open/close 12612 (time 1781587116.03 total 570.00 last 17.37) - open/close 12868 (time 1781587126.72 total 580.69 last 23.94) - open/close 13092 (time 1781587136.73 total 590.70 last 22.38) - open/close 13238 (time 1781587146.97 total 600.94 last 14.26) - open/close 13353 (time 1781587157.01 total 610.98 last 11.46) - open/close 13476 (time 1781587168.79 total 622.77 last 10.44) - open/close 13668 (time 1781587179.77 total 633.75 last 17.49) - open/close 13905 (time 1781587190.45 total 644.42 last 22.20) - open/close 14176 (time 1781587200.46 total 654.44 last 27.06) - open/close 14266 (time 1781587210.48 total 664.46 last 8.98) - open/close 14419 (time 1781587221.20 total 675.18 last 14.27) - open/close 14530 (time 1781587231.90 total 685.87 last 10.38) - open/close 14691 (time 1781587241.91 total 695.88 last 16.08) - open/close 14894 (time 1781587252.12 total 706.09 last 19.89) - open/close 15141 (time 1781587262.86 total 716.83 last 22.99) - open/close 15373 (time 1781587273.28 total 727.25 last 22.27) - open/close 15588 (time 1781587284.03 total 738.01 last 20.00) - open/close 15850 (time 1781587294.58 total 748.55 last 24.85) - open/close 16101 (time 1781587305.36 total 759.33 last 23.28) - open/close 16282 (time 1781587316.11 total 770.08 last 16.84) - open/close 16540 (time 1781587326.67 total 780.64 last 24.43) - open/close 16706 (time 1781587336.75 total 790.73 last 16.46) - open/close 16964 (time 1781587347.26 total 801.23 last 24.55) - open/close 17216 (time 1781587357.88 total 811.85 last 23.74) - open/close 17451 (time 1781587368.11 total 822.09 last 22.95) - open/close 17673 (time 1781587378.74 total 832.71 last 20.89) - open/close 17915 (time 1781587389.57 total 843.54 last 22.35) - open/close 18143 (time 1781587399.58 total 853.56 last 22.76) - open/close 18427 (time 1781587410.56 total 864.53 last 25.88) - open/close 18668 (time 1781587420.78 total 874.76 last 23.57) - open/close 18921 (time 1781587431.62 total 885.60 last 23.35) - open/close 19553 (time 1781587441.62 total 895.60 last 63.19) - open/close 20000 (time 1781587448.20 total 902.18 last 67.95) - open/close 20683 (time 1781587458.21 total 912.18 last 68.25) - open/close 21437 (time 1781587468.21 total 922.19 last 75.36) - open/close 22227 (time 1781587478.21 total 932.19 last 78.99) - open/close 23035 (time 1781587488.26 total 942.23 last 80.44) - open/close 23734 (time 1781587498.27 total 952.24 last 69.86) - open/close 24523 (time 1781587508.27 total 962.25 last 78.84) - open/close 25269 (time 1781587518.29 total 972.27 last 74.46) - open/close 26002 (time 1781587528.29 total 982.27 last 73.29) - open/close 26820 (time 1781587538.30 total 992.27 last 81.78) - open/close 27608 (time 1781587548.30 total 1002.27 last 78.79) - open/close 28430 (time 1781587558.30 total 1012.28 last 82.15) - open/close 29218 (time 1781587568.31 total 1022.29 last 78.71) - open/close 29999 (time 1781587578.32 total 1032.30 last 78.04) - open/close 30800 (time 1781587588.33 total 1042.30 last 80.04) - open/close 31625 (time 1781587598.34 total 1052.31 last 82.44) - open/close 32493 (time 1781587608.35 total 1062.32 last 86.70) - open/close 33364 (time 1781587618.36 total 1072.33 last 87.01) - open/close 34148 (time 1781587628.36 total 1082.34 last 78.35) - open/close 34983 (time 1781587638.38 total 1092.35 last 83.39) - open/close 35772 (time 1781587648.38 total 1102.36 last 78.86) - open/close 36528 (time 1781587658.40 total 1112.37 last 75.50) - open/close 37191 (time 1781587668.42 total 1122.40 last 66.13) - open/close 37794 (time 1781587678.43 total 1132.41 last 60.23) - open/close 38551 (time 1781587688.44 total 1142.41 last 75.65) - open/close 39267 (time 1781587698.44 total 1152.42 last 71.57) - open/close 40000 (time 1781587707.67 total 1161.65 last 79.42) - open/close 40746 (time 1781587717.68 total 1171.65 last 74.59) - open/close 41316 (time 1781587727.68 total 1181.65 last 56.97) - open/close 41956 (time 1781587737.68 total 1191.65 last 64.00) - open/close 42681 (time 1781587747.69 total 1201.66 last 72.45) - open/close 43428 (time 1781587757.70 total 1211.67 last 74.64) - open/close 44131 (time 1781587767.70 total 1221.68 last 70.26) - open/close 44818 (time 1781587777.71 total 1231.68 last 68.65) - open/close 45546 (time 1781587787.72 total 1241.69 last 72.74) - open/close 46289 (time 1781587797.72 total 1251.70 last 74.24) - open/close 47087 (time 1781587807.74 total 1261.71 last 79.70) - open/close 47883 (time 1781587817.74 total 1271.72 last 79.57) - open/close 48687 (time 1781587827.75 total 1281.72 last 80.35) - open/close 49488 (time 1781587837.76 total 1291.73 last 80.01) total: 50000 open/close in 1298.51 seconds: 38.51 ops/second - unlinked 0 (time 1781587848 ; total 0 ; last 0) - unlinked 10000 (time 1781587986 ; total 138 ; last 138) - unlinked 20000 (time 1781588131 ; total 283 ; last 145) - unlinked 30000 (time 1781588272 ; total 424 ; last 141) - unlinked 40000 (time 1781588412 ; total 564 ; last 140) total: 50000 unlinks in 703 seconds: 71.123756 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2