== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:13:54 (1780341234) start mds service on oleg341-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1780341319.01 total 10.71 last 20.18) - open/close 440 (time 1780341329.02 total 20.72 last 22.37) - open/close 659 (time 1780341339.27 total 30.96 last 21.38) - open/close 879 (time 1780341349.57 total 41.26 last 21.35) - open/close 1028 (time 1780341359.58 total 51.27 last 14.88) - open/close 1243 (time 1780341371.14 total 62.83 last 18.60) - open/close 1482 (time 1780341381.15 total 72.84 last 23.88) - open/close 1706 (time 1780341391.74 total 83.44 last 21.14) - open/close 1896 (time 1780341401.76 total 93.45 last 18.98) - open/close 2038 (time 1780341411.84 total 103.54 last 14.07) - open/close 2241 (time 1780341421.86 total 113.56 last 20.26) - open/close 2465 (time 1780341432.08 total 123.77 last 21.94) - open/close 2684 (time 1780341442.08 total 133.78 last 21.88) - open/close 2865 (time 1780341452.68 total 144.37 last 17.09) - open/close 3082 (time 1780341463.04 total 154.74 last 20.93) - open/close 3332 (time 1780341473.04 total 164.74 last 25.00) - open/close 3594 (time 1780341483.04 total 174.74 last 26.20) - open/close 3802 (time 1780341493.05 total 184.74 last 20.79) - open/close 4024 (time 1780341504.04 total 195.73 last 20.19) - open/close 4118 (time 1780341514.70 total 206.39 last 8.82) - open/close 4271 (time 1780341525.73 total 217.42 last 13.88) - open/close 4504 (time 1780341535.96 total 227.65 last 22.77) - open/close 4706 (time 1780341545.97 total 237.66 last 20.17) - open/close 4893 (time 1780341556.24 total 247.94 last 18.21) - open/close 5077 (time 1780341566.26 total 257.95 last 18.38) - open/close 5234 (time 1780341576.59 total 268.28 last 15.20) - open/close 5467 (time 1780341586.59 total 278.28 last 23.29) - open/close 5706 (time 1780341597.19 total 288.88 last 22.55) - open/close 5965 (time 1780341608.06 total 299.75 last 23.82) - open/close 6191 (time 1780341618.08 total 309.77 last 22.56) - open/close 6417 (time 1780341628.10 total 319.79 last 22.56) - open/close 6682 (time 1780341638.73 total 330.42 last 24.93) - open/close 6928 (time 1780341649.25 total 340.94 last 23.38) - open/close 7140 (time 1780341659.34 total 351.03 last 21.01) - open/close 7376 (time 1780341669.34 total 361.03 last 23.59) - open/close 7604 (time 1780341679.36 total 371.05 last 22.77) - open/close 7845 (time 1780341690.07 total 381.76 last 22.50) - open/close 8073 (time 1780341700.32 total 392.01 last 22.25) - open/close 8312 (time 1780341710.65 total 402.34 last 23.13) - open/close 8524 (time 1780341720.83 total 412.53 last 20.82) - open/close 8713 (time 1780341731.03 total 422.72 last 18.53) - open/close 8935 (time 1780341741.04 total 432.73 last 22.18) - open/close 9155 (time 1780341751.88 total 443.57 last 20.30) - open/close 9371 (time 1780341762.43 total 454.13 last 20.46) - open/close 9648 (time 1780341772.44 total 464.13 last 27.68) - open/close 9866 (time 1780341782.45 total 474.14 last 21.79) - open/close 10000 (time 1780341788.71 total 480.40 last 21.40) - open/close 10222 (time 1780341798.72 total 490.42 last 22.17) - open/close 10481 (time 1780341808.74 total 500.43 last 25.87) - open/close 10680 (time 1780341818.74 total 510.43 last 19.89) - open/close 10944 (time 1780341828.97 total 520.66 last 25.81) - open/close 11136 (time 1780341840.16 total 531.86 last 17.15) - open/close 11386 (time 1780341850.75 total 542.45 last 23.61) - open/close 11607 (time 1780341861.21 total 552.91 last 21.13) - open/close 11845 (time 1780341872.21 total 563.90 last 21.64) - open/close 12057 (time 1780341882.37 total 574.06 last 20.88) - open/close 12252 (time 1780341893.12 total 584.81 last 18.14) - open/close 12489 (time 1780341903.38 total 595.07 last 23.10) - open/close 12724 (time 1780341913.38 total 605.08 last 23.48) - open/close 12913 (time 1780341923.42 total 615.11 last 18.84) - open/close 13135 (time 1780341935.94 total 627.63 last 17.73) - open/close 13303 (time 1780341946.37 total 638.07 last 16.10) - open/close 13531 (time 1780341956.88 total 648.57 last 21.71) - open/close 13762 (time 1780341967.06 total 658.76 last 22.67) - open/close 13979 (time 1780341977.07 total 668.77 last 21.68) - open/close 14197 (time 1780341987.78 total 679.47 last 20.37) - open/close 14441 (time 1780341997.89 total 689.58 last 24.12) - open/close 14666 (time 1780342007.90 total 699.59 last 22.49) - open/close 14895 (time 1780342017.91 total 709.60 last 22.88) - open/close 15115 (time 1780342028.42 total 720.11 last 20.92) - open/close 15305 (time 1780342038.43 total 730.13 last 18.98) - open/close 15528 (time 1780342049.15 total 740.85 last 20.80) - open/close 15721 (time 1780342059.69 total 751.38 last 18.31) - open/close 15942 (time 1780342069.70 total 761.39 last 22.08) - open/close 16151 (time 1780342079.71 total 771.40 last 20.89) - open/close 16471 (time 1780342089.99 total 781.68 last 31.13) - open/close 16702 (time 1780342100.00 total 791.69 last 23.07) - open/close 16892 (time 1780342110.92 total 802.61 last 17.41) - open/close 17079 (time 1780342120.99 total 812.68 last 18.56) - open/close 17316 (time 1780342131.79 total 823.48 last 21.95) - open/close 17533 (time 1780342141.80 total 833.49 last 21.68) - open/close 17755 (time 1780342151.80 total 843.49 last 22.19) - open/close 17997 (time 1780342161.81 total 853.51 last 24.17) - open/close 18415 (time 1780342171.83 total 863.52 last 41.75) - open/close 19124 (time 1780342181.84 total 873.53 last 70.83) - open/close 19893 (time 1780342191.84 total 883.53 last 76.88) - open/close 20620 (time 1780342201.84 total 893.53 last 72.68) - open/close 21399 (time 1780342211.85 total 903.54 last 77.87) - open/close 22110 (time 1780342221.85 total 913.54 last 71.09) - open/close 22839 (time 1780342231.85 total 923.55 last 72.84) - open/close 23577 (time 1780342241.86 total 933.56 last 73.73) - open/close 24409 (time 1780342251.87 total 943.56 last 83.19) - open/close 25138 (time 1780342261.87 total 953.56 last 72.89) - open/close 25901 (time 1780342271.88 total 963.57 last 76.23) - open/close 26672 (time 1780342281.88 total 973.57 last 77.09) - open/close 27383 (time 1780342291.89 total 983.58 last 71.04) - open/close 28071 (time 1780342301.90 total 993.60 last 68.68) - open/close 28769 (time 1780342311.90 total 1003.60 last 69.79) - open/close 29548 (time 1780342321.91 total 1013.60 last 77.87) - open/close 30000 (time 1780342327.87 total 1019.56 last 75.88) - open/close 30800 (time 1780342337.87 total 1029.57 last 79.94) - open/close 31541 (time 1780342347.88 total 1039.57 last 74.07) - open/close 32297 (time 1780342357.90 total 1049.59 last 75.43) - open/close 33045 (time 1780342367.90 total 1059.59 last 74.78) - open/close 33763 (time 1780342377.90 total 1069.60 last 71.78) - open/close 34486 (time 1780342387.91 total 1079.60 last 72.27) - open/close 35236 (time 1780342397.91 total 1089.60 last 74.98) - open/close 36026 (time 1780342407.93 total 1099.63 last 78.83) - open/close 36755 (time 1780342417.94 total 1109.63 last 72.84) - open/close 37497 (time 1780342427.95 total 1119.64 last 74.16) - open/close 38207 (time 1780342437.96 total 1129.65 last 70.92) - open/close 38976 (time 1780342447.97 total 1139.66 last 76.82) - open/close 39816 (time 1780342457.97 total 1149.66 last 84.00) - open/close 40000 (time 1780342459.99 total 1151.68 last 91.06) - open/close 40915 (time 1780342469.99 total 1161.68 last 91.49) - open/close 41683 (time 1780342479.99 total 1171.68 last 76.77) - open/close 42533 (time 1780342490.00 total 1181.69 last 84.93) - open/close 43355 (time 1780342500.02 total 1191.71 last 82.08) - open/close 44138 (time 1780342510.03 total 1201.72 last 78.20) - open/close 44970 (time 1780342520.03 total 1211.72 last 83.18) - open/close 45808 (time 1780342530.05 total 1221.74 last 83.66) - open/close 46590 (time 1780342540.05 total 1231.74 last 78.20) - open/close 47434 (time 1780342550.06 total 1241.75 last 84.28) - open/close 48207 (time 1780342560.07 total 1251.76 last 77.23) - open/close 48941 (time 1780342570.07 total 1261.76 last 73.39) - open/close 49663 (time 1780342580.07 total 1271.76 last 72.20) total: 50000 open/close in 1276.35 seconds: 39.17 ops/second - unlinked 0 (time 1780342588 ; total 0 ; last 0) - unlinked 10000 (time 1780342719 ; total 131 ; last 131) - unlinked 20000 (time 1780342831 ; total 243 ; last 112) - unlinked 30000 (time 1780343022 ; total 434 ; last 191) - unlinked 40000 (time 1780343188 ; total 600 ; last 166) total: 50000 unlinks in 759 seconds: 65.876152 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2