== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:31:24 (1789637484) start mds service on oleg351-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 04:27 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: kmemleak disabled oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg351-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg351-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1789637533.28 total 10.87 last 21.25) - open/close 523 (time 1789637543.84 total 21.43 last 27.65) - open/close 736 (time 1789637554.78 total 32.37 last 19.47) - open/close 972 (time 1789637564.83 total 42.43 last 23.48) - open/close 1035 (time 1789637574.83 total 52.43 last 6.30) - open/close 1313 (time 1789637584.84 total 62.44 last 27.78) - open/close 1574 (time 1789637595.80 total 73.40 last 23.81) - open/close 1815 (time 1789637605.80 total 83.40 last 24.10) - open/close 2131 (time 1789637616.80 total 94.40 last 28.73) - open/close 2365 (time 1789637626.90 total 104.50 last 23.16) - open/close 2616 (time 1789637637.72 total 115.32 last 23.20) - open/close 2845 (time 1789637647.73 total 125.33 last 22.88) - open/close 3095 (time 1789637657.88 total 135.48 last 24.63) - open/close 3336 (time 1789637668.72 total 146.31 last 22.25) - open/close 3567 (time 1789637679.63 total 157.22 last 21.17) - open/close 3719 (time 1789637690.62 total 168.21 last 13.83) - open/close 3933 (time 1789637701.23 total 178.82 last 20.17) - open/close 4049 (time 1789637711.97 total 189.56 last 10.80) - open/close 4163 (time 1789637722.66 total 200.26 last 10.66) - open/close 4255 (time 1789637732.67 total 210.27 last 9.20) - open/close 4356 (time 1789637743.09 total 220.68 last 9.70) - open/close 4477 (time 1789637754.13 total 231.72 last 10.96) - open/close 4575 (time 1789637764.15 total 241.74 last 9.78) - open/close 4707 (time 1789637774.18 total 251.77 last 13.16) - open/close 4832 (time 1789637784.19 total 261.79 last 12.48) - open/close 4965 (time 1789637795.00 total 272.59 last 12.31) - open/close 5214 (time 1789637805.94 total 283.54 last 22.75) - open/close 5352 (time 1789637817.00 total 294.59 last 12.48) - open/close 5553 (time 1789637827.39 total 304.99 last 19.33) - open/close 5703 (time 1789637837.92 total 315.52 last 14.25) - open/close 5828 (time 1789637847.93 total 325.52 last 12.50) - open/close 6024 (time 1789637857.95 total 335.54 last 19.56) - open/close 6260 (time 1789637867.96 total 345.55 last 23.58) - open/close 6511 (time 1789637877.99 total 355.59 last 25.01) - open/close 6645 (time 1789637889.02 total 366.61 last 12.15) - open/close 6829 (time 1789637899.98 total 377.57 last 16.79) - open/close 7066 (time 1789637910.63 total 388.23 last 22.24) - open/close 7258 (time 1789637920.87 total 398.46 last 18.76) - open/close 7414 (time 1789637931.43 total 409.03 last 14.77) - open/close 7541 (time 1789637943.27 total 420.86 last 10.73) - open/close 7747 (time 1789637955.75 total 433.35 last 16.50) - open/close 8023 (time 1789637965.76 total 443.36 last 27.58) - open/close 8150 (time 1789637976.80 total 454.40 last 11.50) - open/close 8246 (time 1789637987.26 total 464.85 last 9.18) - open/close 8528 (time 1789637997.27 total 474.86 last 28.17) - open/close 8585 (time 1789638008.04 total 485.64 last 5.29) - open/close 8667 (time 1789638018.23 total 495.83 last 8.04) - open/close 8891 (time 1789638029.18 total 506.78 last 20.46) - open/close 9122 (time 1789638039.29 total 516.89 last 22.84) - open/close 9295 (time 1789638050.02 total 527.61 last 16.13) - open/close 9485 (time 1789638060.02 total 537.62 last 18.99) - open/close 9641 (time 1789638070.03 total 547.62 last 15.59) - open/close 9873 (time 1789638080.87 total 558.46 last 21.40) - open/close 9962 (time 1789638092.90 total 570.50 last 7.40) - open/close 10199 (time 1789638103.92 total 581.51 last 21.51) - open/close 10341 (time 1789638113.99 total 591.58 last 14.10) - open/close 10433 (time 1789638124.03 total 601.62 last 9.16) - open/close 10537 (time 1789638134.03 total 611.63 last 10.40) - open/close 10630 (time 1789638144.76 total 622.36 last 8.66) - open/close 10692 (time 1789638155.68 total 633.28 last 5.68) - open/close 10906 (time 1789638166.72 total 644.32 last 19.39) - open/close 11036 (time 1789638176.88 total 654.48 last 12.80) - open/close 11219 (time 1789638186.88 total 664.48 last 18.30) - open/close 11370 (time 1789638198.05 total 675.65 last 13.52) - open/close 11490 (time 1789638208.07 total 685.66 last 11.98) - open/close 11613 (time 1789638218.56 total 696.16 last 11.72) - open/close 11746 (time 1789638228.57 total 706.16 last 13.30) - open/close 11890 (time 1789638240.33 total 717.92 last 12.24) - open/close 11986 (time 1789638250.53 total 728.12 last 9.41) - open/close 12104 (time 1789638260.59 total 738.19 last 11.73) - open/close 12326 (time 1789638271.53 total 749.12 last 20.30) - open/close 12519 (time 1789638282.29 total 759.89 last 17.92) - open/close 12673 (time 1789638292.79 total 770.38 last 14.68) - open/close 12818 (time 1789638302.80 total 780.39 last 14.48) - open/close 12975 (time 1789638313.29 total 790.89 last 14.97) - open/close 13214 (time 1789638323.56 total 801.15 last 23.28) - open/close 13431 (time 1789638334.46 total 812.06 last 19.90) - open/close 13626 (time 1789638344.48 total 822.07 last 19.47) - open/close 13857 (time 1789638354.48 total 832.07 last 23.10) - open/close 13956 (time 1789638364.49 total 842.08 last 9.89) - open/close 14144 (time 1789638374.51 total 852.10 last 18.76) - open/close 14361 (time 1789638385.06 total 862.66 last 20.56) - open/close 14607 (time 1789638395.24 total 872.84 last 24.17) - open/close 14791 (time 1789638405.60 total 883.20 last 17.76) - open/close 15369 (time 1789638415.60 total 893.20 last 57.78) - open/close 16069 (time 1789638425.61 total 903.21 last 69.94) - open/close 16708 (time 1789638435.63 total 913.23 last 63.79) - open/close 17332 (time 1789638445.64 total 923.24 last 62.32) - open/close 17984 (time 1789638455.65 total 933.25 last 65.16) - open/close 18669 (time 1789638465.70 total 943.30 last 68.13) - open/close 19262 (time 1789638475.72 total 953.31 last 59.21) - open/close 19948 (time 1789638485.73 total 963.32 last 68.56) - open/close 20616 (time 1789638495.74 total 973.34 last 66.71) - open/close 21246 (time 1789638505.74 total 983.34 last 62.97) - open/close 21894 (time 1789638515.75 total 993.34 last 64.79) - open/close 22551 (time 1789638525.75 total 1003.34 last 65.69) - open/close 23185 (time 1789638535.76 total 1013.36 last 63.33) - open/close 23873 (time 1789638545.77 total 1023.36 last 68.76) - open/close 24544 (time 1789638555.78 total 1033.38 last 66.99) - open/close 25203 (time 1789638565.81 total 1043.41 last 65.71) - open/close 25814 (time 1789638575.83 total 1053.42 last 61.00) - open/close 26411 (time 1789638585.84 total 1063.44 last 59.60) - open/close 27010 (time 1789638595.85 total 1073.45 last 59.84) - open/close 27716 (time 1789638605.86 total 1083.46 last 70.54) - open/close 28406 (time 1789638615.88 total 1093.47 last 68.89) - open/close 29056 (time 1789638625.89 total 1103.48 last 64.95) - open/close 29655 (time 1789638635.90 total 1113.50 last 59.82) - open/close 30000 (time 1789638640.95 total 1118.54 last 68.36) - open/close 30658 (time 1789638650.95 total 1128.54 last 65.79) - open/close 31338 (time 1789638660.95 total 1138.55 last 67.98) - open/close 31956 (time 1789638670.96 total 1148.56 last 61.73) - open/close 32665 (time 1789638680.97 total 1158.56 last 70.87) - open/close 33354 (time 1789638691.04 total 1168.64 last 68.39) - open/close 34041 (time 1789638701.04 total 1178.64 last 68.68) - open/close 34712 (time 1789638711.05 total 1188.64 last 67.09) - open/close 35448 (time 1789638721.05 total 1198.65 last 73.57) - open/close 36060 (time 1789638731.06 total 1208.65 last 61.15) - open/close 36631 (time 1789638741.06 total 1218.66 last 57.09) - open/close 37159 (time 1789638751.06 total 1228.66 last 52.78) - open/close 37768 (time 1789638761.07 total 1238.66 last 60.88) - open/close 38402 (time 1789638771.07 total 1248.67 last 63.37) - open/close 38949 (time 1789638781.10 total 1258.70 last 54.54) - open/close 39459 (time 1789638791.11 total 1268.71 last 50.94) - open/close 39994 (time 1789638801.12 total 1278.72 last 53.45) - open/close 40629 (time 1789638811.13 total 1288.72 last 63.46) - open/close 41317 (time 1789638821.14 total 1298.73 last 68.72) - open/close 41987 (time 1789638831.14 total 1308.74 last 66.97) - open/close 42635 (time 1789638841.14 total 1318.74 last 64.79) - open/close 43243 (time 1789638851.15 total 1328.75 last 60.75) - open/close 43880 (time 1789638861.17 total 1338.77 last 63.57) - open/close 44524 (time 1789638871.18 total 1348.77 last 64.37) - open/close 45181 (time 1789638881.19 total 1358.78 last 65.65) - open/close 45873 (time 1789638891.19 total 1368.78 last 69.19) - open/close 46420 (time 1789638901.21 total 1378.80 last 54.59) - open/close 47001 (time 1789638911.27 total 1388.86 last 57.75) - open/close 47641 (time 1789638921.27 total 1398.87 last 63.98) - open/close 48237 (time 1789638931.28 total 1408.87 last 59.55) - open/close 48811 (time 1789638941.28 total 1418.88 last 57.39) - open/close 49285 (time 1789638951.31 total 1428.90 last 47.26) - open/close 49671 (time 1789638961.33 total 1438.93 last 38.51) total: 50000 open/close in 1447.64 seconds: 34.54 ops/second - unlinked 0 (time 1789638976 ; total 0 ; last 0) - unlinked 10000 (time 1789639119 ; total 143 ; last 143) - unlinked 20000 (time 1789639250 ; total 274 ; last 131) - unlinked 30000 (time 1789639430 ; total 454 ; last 180) - unlinked 40000 (time 1789639610 ; total 634 ; last 180) total: 50000 unlinks in 814 seconds: 61.425060 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: kmemleak disabled modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2