== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:30:01 (1788215401) start mds service on oleg438-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1755536 Aug 31 10:08 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: kmemleak disabled oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg438-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg438-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg438-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 172 (time 1788215499.28 total 11.69 last 14.71) - open/close 345 (time 1788215510.09 total 22.50 last 16.01) - open/close 569 (time 1788215521.09 total 33.51 last 20.35) - open/close 760 (time 1788215533.38 total 45.80 last 15.54) - open/close 856 (time 1788215543.41 total 55.83 last 9.58) - open/close 1035 (time 1788215554.88 total 67.30 last 15.60) - open/close 1123 (time 1788215565.02 total 77.43 last 8.68) - open/close 1320 (time 1788215575.03 total 87.45 last 19.68) - open/close 1572 (time 1788215585.04 total 97.46 last 25.16) - open/close 1753 (time 1788215596.04 total 108.46 last 16.45) - open/close 1972 (time 1788215609.17 total 121.59 last 16.68) - open/close 2126 (time 1788215619.58 total 132.00 last 14.79) - open/close 2412 (time 1788215630.20 total 142.62 last 26.94) - open/close 2584 (time 1788215641.21 total 153.63 last 15.62) - open/close 2825 (time 1788215651.22 total 163.63 last 24.10) - open/close 3024 (time 1788215664.19 total 176.61 last 15.33) - open/close 3212 (time 1788215674.94 total 187.35 last 17.50) - open/close 3446 (time 1788215685.29 total 197.70 last 22.61) - open/close 3725 (time 1788215695.57 total 207.99 last 27.13) - open/close 3993 (time 1788215705.92 total 218.34 last 25.88) - open/close 4241 (time 1788215716.24 total 228.65 last 24.04) - open/close 4501 (time 1788215727.18 total 239.59 last 23.77) - open/close 4709 (time 1788215738.51 total 250.93 last 18.35) - open/close 4863 (time 1788215749.39 total 261.80 last 14.16) - open/close 5112 (time 1788215759.42 total 271.84 last 24.81) - open/close 5287 (time 1788215771.96 total 284.38 last 13.96) - open/close 5483 (time 1788215781.96 total 294.38 last 19.59) - open/close 5652 (time 1788215792.16 total 304.57 last 16.58) - open/close 5922 (time 1788215803.14 total 315.56 last 24.59) - open/close 6177 (time 1788215813.43 total 325.85 last 24.77) - open/close 6408 (time 1788215824.13 total 336.55 last 21.60) - open/close 6661 (time 1788215835.13 total 347.54 last 23.00) - open/close 6950 (time 1788215845.40 total 357.82 last 28.12) - open/close 7154 (time 1788215855.93 total 368.35 last 19.37) - open/close 7453 (time 1788215867.01 total 379.43 last 26.99) - open/close 7707 (time 1788215877.63 total 390.05 last 23.92) - open/close 7952 (time 1788215887.71 total 400.13 last 24.31) - open/close 8153 (time 1788215898.76 total 411.18 last 18.19) - open/close 8341 (time 1788215909.65 total 422.07 last 17.27) - open/close 8583 (time 1788215920.59 total 433.01 last 22.11) - open/close 8725 (time 1788215930.60 total 443.02 last 14.19) - open/close 8935 (time 1788215941.64 total 454.06 last 19.02) - open/close 9166 (time 1788215952.10 total 464.51 last 22.10) - open/close 9462 (time 1788215962.11 total 474.53 last 29.56) - open/close 9704 (time 1788215972.43 total 484.85 last 23.45) - open/close 9954 (time 1788215982.98 total 495.39 last 23.71) - open/close 10199 (time 1788215993.48 total 505.89 last 23.33) - open/close 10469 (time 1788216003.49 total 515.90 last 26.97) - open/close 10698 (time 1788216013.83 total 526.25 last 22.13) - open/close 10954 (time 1788216024.77 total 537.19 last 23.41) - open/close 11155 (time 1788216034.78 total 547.20 last 20.08) - open/close 11400 (time 1788216044.79 total 557.21 last 24.48) - open/close 11494 (time 1788216054.80 total 567.22 last 9.39) - open/close 11731 (time 1788216065.11 total 577.53 last 22.98) - open/close 11961 (time 1788216076.12 total 588.54 last 20.90) - open/close 12203 (time 1788216086.86 total 599.27 last 22.54) - open/close 12439 (time 1788216097.49 total 609.91 last 22.19) - open/close 12645 (time 1788216107.50 total 619.91 last 20.59) - open/close 12902 (time 1788216118.37 total 630.79 last 23.63) - open/close 13162 (time 1788216128.83 total 641.25 last 24.86) - open/close 13390 (time 1788216139.08 total 651.50 last 22.24) - open/close 13611 (time 1788216149.97 total 662.39 last 20.29) - open/close 13900 (time 1788216160.76 total 673.18 last 26.79) - open/close 14044 (time 1788216171.03 total 683.45 last 14.02) - open/close 14300 (time 1788216181.05 total 693.47 last 25.55) - open/close 14531 (time 1788216191.24 total 703.66 last 22.66) - open/close 14826 (time 1788216201.25 total 713.67 last 29.47) - open/close 15064 (time 1788216211.71 total 724.13 last 22.76) - open/close 15309 (time 1788216221.89 total 734.31 last 24.06) - open/close 15542 (time 1788216232.04 total 744.45 last 22.97) - open/close 15810 (time 1788216242.37 total 754.79 last 25.92) - open/close 16005 (time 1788216252.79 total 765.21 last 18.71) - open/close 16235 (time 1788216262.80 total 775.21 last 22.99) - open/close 16424 (time 1788216272.96 total 785.38 last 18.60) - open/close 16643 (time 1788216283.85 total 796.27 last 20.10) - open/close 16834 (time 1788216293.95 total 806.37 last 18.91) - open/close 17067 (time 1788216304.26 total 816.67 last 22.62) - open/close 17330 (time 1788216314.26 total 826.68 last 26.28) - open/close 17457 (time 1788216324.43 total 836.85 last 12.50) - open/close 17631 (time 1788216334.44 total 846.86 last 17.38) - open/close 17852 (time 1788216344.44 total 856.86 last 22.10) - open/close 18141 (time 1788216355.33 total 867.75 last 26.54) - open/close 18425 (time 1788216365.34 total 877.76 last 28.37) - open/close 18666 (time 1788216375.35 total 887.77 last 24.07) - open/close 18912 (time 1788216385.37 total 897.79 last 24.56) - open/close 19060 (time 1788216395.37 total 907.79 last 14.80) - open/close 19197 (time 1788216405.64 total 918.06 last 13.34) - open/close 19645 (time 1788216415.65 total 928.06 last 44.78) - open/close 20000 (time 1788216420.25 total 932.67 last 77.08) - open/close 20790 (time 1788216430.26 total 942.68 last 78.93) - open/close 21540 (time 1788216440.27 total 952.69 last 74.92) - open/close 22224 (time 1788216450.29 total 962.71 last 68.29) - open/close 22988 (time 1788216460.30 total 972.71 last 76.33) - open/close 23721 (time 1788216470.32 total 982.74 last 73.14) - open/close 24464 (time 1788216480.32 total 992.74 last 74.29) - open/close 25248 (time 1788216490.32 total 1002.74 last 78.38) - open/close 26020 (time 1788216500.32 total 1012.74 last 77.19) - open/close 26753 (time 1788216510.33 total 1022.75 last 73.24) - open/close 27544 (time 1788216520.33 total 1032.75 last 79.08) - open/close 28335 (time 1788216530.34 total 1042.76 last 79.05) - open/close 29103 (time 1788216540.35 total 1052.77 last 76.70) - open/close 29855 (time 1788216550.36 total 1062.78 last 75.13) - open/close 30642 (time 1788216560.37 total 1072.79 last 78.65) - open/close 31439 (time 1788216570.37 total 1082.79 last 79.66) - open/close 32216 (time 1788216580.38 total 1092.80 last 77.66) - open/close 32911 (time 1788216590.38 total 1102.80 last 69.48) - open/close 33659 (time 1788216600.38 total 1112.80 last 74.79) - open/close 34410 (time 1788216610.40 total 1122.82 last 74.96) - open/close 35204 (time 1788216620.41 total 1132.83 last 79.33) - open/close 35922 (time 1788216630.42 total 1142.84 last 71.75) - open/close 36673 (time 1788216640.43 total 1152.85 last 74.98) - open/close 37386 (time 1788216650.44 total 1162.86 last 71.24) - open/close 38142 (time 1788216660.45 total 1172.87 last 75.53) - open/close 38949 (time 1788216670.45 total 1182.87 last 80.68) - open/close 39536 (time 1788216680.46 total 1192.88 last 58.69) - open/close 40000 (time 1788216687.97 total 1200.39 last 61.75) - open/close 40549 (time 1788216697.98 total 1210.39 last 54.87) - open/close 41196 (time 1788216707.98 total 1220.40 last 64.65) - open/close 41865 (time 1788216717.99 total 1230.41 last 66.84) - open/close 42461 (time 1788216728.01 total 1240.42 last 59.52) - open/close 43143 (time 1788216738.02 total 1250.44 last 68.12) - open/close 43777 (time 1788216748.03 total 1260.45 last 63.30) - open/close 44525 (time 1788216758.05 total 1270.47 last 74.66) - open/close 45263 (time 1788216768.06 total 1280.48 last 73.71) - open/close 46018 (time 1788216778.07 total 1290.49 last 75.46) - open/close 46695 (time 1788216788.07 total 1300.49 last 67.70) - open/close 47442 (time 1788216798.08 total 1310.49 last 74.65) - open/close 48117 (time 1788216808.08 total 1320.50 last 67.45) - open/close 48870 (time 1788216818.09 total 1330.51 last 75.25) - open/close 49616 (time 1788216828.09 total 1340.51 last 74.58) total: 50000 open/close in 1345.76 seconds: 37.15 ops/second - unlinked 0 (time 1788216838 ; total 0 ; last 0) - unlinked 10000 (time 1788216977 ; total 139 ; last 139) - unlinked 20000 (time 1788217120 ; total 282 ; last 143) - unlinked 30000 (time 1788217256 ; total 418 ; last 136) - unlinked 40000 (time 1788217398 ; total 560 ; last 142) total: 50000 unlinks in 701 seconds: 71.326675 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: kmemleak disabled modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2