== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:26:40 (1789482400) start mds service on oleg301-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 15 02:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: kmemleak disabled oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg301-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg301-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 90 (time 1789482484.51 total 10.45 last 8.62) - open/close 313 (time 1789482495.19 total 21.13 last 20.87) - open/close 483 (time 1789482506.12 total 32.06 last 15.56) - open/close 723 (time 1789482516.70 total 42.63 last 22.70) - open/close 971 (time 1789482527.37 total 53.31 last 23.23) - open/close 1203 (time 1789482537.39 total 63.32 last 23.16) - open/close 1452 (time 1789482548.20 total 74.14 last 23.02) - open/close 1718 (time 1789482558.21 total 84.14 last 26.59) - open/close 1963 (time 1789482568.79 total 94.72 last 23.15) - open/close 2170 (time 1789482579.79 total 105.73 last 18.81) - open/close 2422 (time 1789482589.80 total 115.74 last 25.18) - open/close 2648 (time 1789482599.83 total 125.77 last 22.54) - open/close 2941 (time 1789482609.84 total 135.78 last 29.26) - open/close 3164 (time 1789482620.29 total 146.23 last 21.34) - open/close 3363 (time 1789482630.30 total 156.23 last 19.89) - open/close 3553 (time 1789482640.65 total 166.59 last 18.35) - open/close 3830 (time 1789482650.66 total 176.60 last 27.68) - open/close 4007 (time 1789482660.69 total 186.62 last 17.65) - open/close 4282 (time 1789482670.71 total 196.65 last 27.43) - open/close 4433 (time 1789482680.74 total 206.67 last 15.07) - open/close 4522 (time 1789482692.93 total 218.87 last 7.30) - open/close 4651 (time 1789482704.67 total 230.61 last 10.99) - open/close 4753 (time 1789482715.33 total 241.26 last 9.58) - open/close 4981 (time 1789482725.71 total 251.64 last 21.97) - open/close 5139 (time 1789482735.88 total 261.81 last 15.54) - open/close 5268 (time 1789482745.93 total 271.87 last 12.83) - open/close 5568 (time 1789482756.80 total 282.74 last 27.60) - open/close 5834 (time 1789482766.88 total 292.81 last 26.41) - open/close 6071 (time 1789482777.47 total 303.41 last 22.37) - open/close 6303 (time 1789482788.16 total 314.10 last 21.70) - open/close 6542 (time 1789482798.91 total 324.85 last 22.24) - open/close 6802 (time 1789482808.92 total 334.85 last 25.99) - open/close 7018 (time 1789482821.51 total 347.45 last 17.14) - open/close 7268 (time 1789482832.58 total 358.52 last 22.58) - open/close 7573 (time 1789482843.26 total 369.20 last 28.57) - open/close 7750 (time 1789482855.31 total 381.25 last 14.68) - open/close 7916 (time 1789482866.31 total 392.24 last 15.10) - open/close 7994 (time 1789482876.43 total 402.36 last 7.71) - open/close 8204 (time 1789482886.99 total 412.93 last 19.88) - open/close 8314 (time 1789482897.50 total 423.43 last 10.47) - open/close 8426 (time 1789482908.55 total 434.49 last 10.13) - open/close 8554 (time 1789482919.12 total 445.06 last 12.11) - open/close 8658 (time 1789482929.12 total 455.06 last 10.40) - open/close 8796 (time 1789482939.47 total 465.41 last 13.33) - open/close 8942 (time 1789482949.76 total 475.69 last 14.20) - open/close 9185 (time 1789482960.45 total 486.38 last 22.73) - open/close 9430 (time 1789482971.01 total 496.94 last 23.20) - open/close 9634 (time 1789482981.58 total 507.51 last 19.30) - open/close 9920 (time 1789482992.25 total 518.19 last 26.79) - open/close 10000 (time 1789482994.47 total 520.41 last 36.03) - open/close 10217 (time 1789483004.48 total 530.42 last 21.68) - open/close 10464 (time 1789483014.78 total 540.72 last 23.98) - open/close 10692 (time 1789483025.07 total 551.01 last 22.16) - open/close 10924 (time 1789483035.46 total 561.40 last 22.33) - open/close 11180 (time 1789483046.28 total 572.21 last 23.67) - open/close 11290 (time 1789483057.03 total 582.97 last 10.23) - open/close 11518 (time 1789483067.04 total 592.98 last 22.77) - open/close 11751 (time 1789483077.18 total 603.12 last 22.97) - open/close 11949 (time 1789483087.19 total 613.13 last 19.78) - open/close 12207 (time 1789483098.13 total 624.07 last 23.58) - open/close 12338 (time 1789483108.80 total 634.73 last 12.28) - open/close 12602 (time 1789483121.54 total 647.48 last 20.72) - open/close 12845 (time 1789483131.56 total 657.50 last 24.25) - open/close 13071 (time 1789483141.57 total 667.51 last 22.58) - open/close 13341 (time 1789483151.57 total 677.51 last 26.99) - open/close 13571 (time 1789483161.57 total 687.51 last 23.00) - open/close 13786 (time 1789483171.57 total 697.51 last 21.50) - open/close 14041 (time 1789483181.59 total 707.53 last 25.46) - open/close 14233 (time 1789483191.61 total 717.54 last 19.17) - open/close 14447 (time 1789483202.45 total 728.38 last 19.74) - open/close 14552 (time 1789483212.49 total 738.42 last 10.45) - open/close 14699 (time 1789483223.12 total 749.06 last 13.82) - open/close 14824 (time 1789483233.13 total 759.07 last 12.49) - open/close 14968 (time 1789483243.15 total 769.09 last 14.38) - open/close 15230 (time 1789483254.15 total 780.09 last 23.81) - open/close 15396 (time 1789483264.32 total 790.26 last 16.32) - open/close 15658 (time 1789483275.13 total 801.06 last 24.25) - open/close 15807 (time 1789483285.37 total 811.30 last 14.55) - open/close 16046 (time 1789483295.37 total 821.31 last 23.89) - open/close 16305 (time 1789483305.39 total 831.33 last 25.85) - open/close 16543 (time 1789483315.40 total 841.34 last 23.78) - open/close 16766 (time 1789483325.41 total 851.34 last 22.28) - open/close 16996 (time 1789483335.43 total 861.37 last 22.95) - open/close 17234 (time 1789483345.44 total 871.37 last 23.79) - open/close 17467 (time 1789483355.45 total 881.39 last 23.27) - open/close 17767 (time 1789483366.13 total 892.07 last 28.09) - open/close 17977 (time 1789483376.15 total 902.09 last 20.96) - open/close 18227 (time 1789483386.90 total 912.84 last 23.25) - open/close 18471 (time 1789483397.31 total 923.24 last 23.45) - open/close 18707 (time 1789483407.74 total 933.68 last 22.61) - open/close 18962 (time 1789483417.75 total 943.68 last 25.49) - open/close 19188 (time 1789483428.12 total 954.05 last 21.80) - open/close 19792 (time 1789483438.12 total 964.06 last 60.36) - open/close 20000 (time 1789483441.19 total 967.13 last 67.75) - open/close 20758 (time 1789483451.21 total 977.14 last 75.70) - open/close 21591 (time 1789483461.21 total 987.15 last 83.27) - open/close 22451 (time 1789483471.21 total 997.15 last 85.97) - open/close 23283 (time 1789483481.22 total 1007.16 last 83.12) - open/close 24017 (time 1789483491.24 total 1017.18 last 73.28) - open/close 24876 (time 1789483501.24 total 1027.18 last 85.89) - open/close 25711 (time 1789483511.25 total 1037.18 last 83.45) - open/close 26535 (time 1789483521.26 total 1047.19 last 82.34) - open/close 27373 (time 1789483531.26 total 1057.20 last 83.76) - open/close 28220 (time 1789483541.27 total 1067.20 last 84.65) - open/close 29040 (time 1789483551.27 total 1077.21 last 81.93) - open/close 29870 (time 1789483561.28 total 1087.22 last 82.95) - open/close 30673 (time 1789483571.29 total 1097.22 last 80.27) - open/close 31500 (time 1789483581.29 total 1107.23 last 82.63) - open/close 32255 (time 1789483591.29 total 1117.23 last 75.50) - open/close 33066 (time 1789483601.30 total 1127.23 last 81.07) - open/close 33912 (time 1789483611.31 total 1137.24 last 84.52) - open/close 34666 (time 1789483621.32 total 1147.26 last 75.31) - open/close 35391 (time 1789483631.33 total 1157.26 last 72.45) - open/close 36033 (time 1789483641.34 total 1167.27 last 64.14) - open/close 36735 (time 1789483651.36 total 1177.30 last 70.02) - open/close 37470 (time 1789483661.38 total 1187.31 last 73.39) - open/close 38260 (time 1789483671.39 total 1197.32 last 78.93) - open/close 39057 (time 1789483681.39 total 1207.32 last 79.68) - open/close 39916 (time 1789483691.39 total 1217.33 last 85.84) - open/close 40666 (time 1789483701.40 total 1227.34 last 74.96) - open/close 41476 (time 1789483711.42 total 1237.35 last 80.84) - open/close 42315 (time 1789483721.43 total 1247.36 last 83.84) - open/close 43129 (time 1789483731.44 total 1257.37 last 81.30) - open/close 43898 (time 1789483741.44 total 1267.38 last 76.87) - open/close 44730 (time 1789483751.45 total 1277.38 last 83.16) - open/close 45557 (time 1789483761.45 total 1287.38 last 82.70) - open/close 46361 (time 1789483771.47 total 1297.40 last 80.26) - open/close 47168 (time 1789483781.47 total 1307.40 last 80.68) - open/close 47955 (time 1789483791.48 total 1317.42 last 78.59) - open/close 48781 (time 1789483801.48 total 1327.42 last 82.59) - open/close 49557 (time 1789483811.48 total 1337.42 last 77.59) total: 50000 open/close in 1343.41 seconds: 37.22 ops/second - unlinked 0 (time 1789483821 ; total 0 ; last 0) - unlinked 10000 (time 1789483947 ; total 126 ; last 126) - unlinked 20000 (time 1789484078 ; total 257 ; last 131) - unlinked 30000 (time 1789484208 ; total 387 ; last 130) - unlinked 40000 (time 1789484341 ; total 520 ; last 133) total: 50000 unlinks in 653 seconds: 76.569679 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: kmemleak disabled modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2