== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:36:15 (1788798975) start mds service on oleg620-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 7 10:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing load_modules_local oleg620-server: kmemleak disabled oleg620-server: Loading modules from /home/green/git/lustre-release/lustre oleg620-server: detected 4 online CPUs by sysfs oleg620-server: MODOPTS_LIBCFS= oleg620-server: Force libcfs to create 2 CPU partitions oleg620-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg620-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg620-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg620-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg620-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Started lustre-OST0000 oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 226 (time 1788799057.99 total 10.27 last 22.01) - open/close 522 (time 1788799068.82 total 21.10 last 27.32) - open/close 724 (time 1788799079.25 total 31.53 last 19.38) - open/close 854 (time 1788799089.25 total 41.53 last 12.99) - open/close 966 (time 1788799099.98 total 52.26 last 10.44) - open/close 1210 (time 1788799110.80 total 63.07 last 22.56) - open/close 1487 (time 1788799120.81 total 73.09 last 27.65) - open/close 1701 (time 1788799130.84 total 83.12 last 21.34) - open/close 1932 (time 1788799141.31 total 93.59 last 22.06) - open/close 2087 (time 1788799151.34 total 103.61 last 15.47) - open/close 2339 (time 1788799161.87 total 114.15 last 23.92) - open/close 2589 (time 1788799171.87 total 124.15 last 24.99) - open/close 2823 (time 1788799181.96 total 134.23 last 23.21) - open/close 3079 (time 1788799192.00 total 144.28 last 25.49) - open/close 3337 (time 1788799202.96 total 155.24 last 23.55) - open/close 3552 (time 1788799212.97 total 165.24 last 21.48) - open/close 3807 (time 1788799223.05 total 175.33 last 25.28) - open/close 4036 (time 1788799233.74 total 186.02 last 21.43) - open/close 4221 (time 1788799243.74 total 196.02 last 18.49) - open/close 4350 (time 1788799253.75 total 206.03 last 12.89) - open/close 4581 (time 1788799263.75 total 216.03 last 23.10) - open/close 4772 (time 1788799274.88 total 227.16 last 17.16) - open/close 5005 (time 1788799284.89 total 237.17 last 23.27) - open/close 5124 (time 1788799295.06 total 247.34 last 11.71) - open/close 5251 (time 1788799305.07 total 257.35 last 12.68) - open/close 5481 (time 1788799315.10 total 267.38 last 22.93) - open/close 5713 (time 1788799325.26 total 277.54 last 22.84) - open/close 5989 (time 1788799335.35 total 287.63 last 27.34) - open/close 6260 (time 1788799345.39 total 297.67 last 27.01) - open/close 6433 (time 1788799355.66 total 307.93 last 16.85) - open/close 6723 (time 1788799366.62 total 318.90 last 26.46) - open/close 6963 (time 1788799377.56 total 329.84 last 21.93) - open/close 7168 (time 1788799388.61 total 340.89 last 18.56) - open/close 7319 (time 1788799398.68 total 350.96 last 14.99) - open/close 7479 (time 1788799409.50 total 361.78 last 14.79) - open/close 7711 (time 1788799421.39 total 373.67 last 19.51) - open/close 8011 (time 1788799433.95 total 386.22 last 23.89) - open/close 8278 (time 1788799444.62 total 396.90 last 25.00) - open/close 8474 (time 1788799454.63 total 406.91 last 19.59) - open/close 8683 (time 1788799464.63 total 416.91 last 20.90) - open/close 8854 (time 1788799475.42 total 427.70 last 15.84) - open/close 9067 (time 1788799486.17 total 438.45 last 19.81) - open/close 9263 (time 1788799496.19 total 448.47 last 19.58) - open/close 9508 (time 1788799507.23 total 459.51 last 22.19) - open/close 9758 (time 1788799518.23 total 470.51 last 22.72) - open/close 9955 (time 1788799528.24 total 480.52 last 19.68) - open/close 10198 (time 1788799538.58 total 490.86 last 23.51) - open/close 10379 (time 1788799549.44 total 501.72 last 16.66) - open/close 10642 (time 1788799559.44 total 511.72 last 26.29) - open/close 10908 (time 1788799569.46 total 521.74 last 26.55) - open/close 11116 (time 1788799579.77 total 532.05 last 20.18) - open/close 11406 (time 1788799590.60 total 542.88 last 26.79) - open/close 11677 (time 1788799601.42 total 553.70 last 25.04) - open/close 11959 (time 1788799612.26 total 564.54 last 26.02) - open/close 12235 (time 1788799622.26 total 574.54 last 27.60) - open/close 12483 (time 1788799632.98 total 585.26 last 23.13) - open/close 12796 (time 1788799643.01 total 595.29 last 31.21) - open/close 13046 (time 1788799653.02 total 605.30 last 24.97) - open/close 13318 (time 1788799663.02 total 615.30 last 27.20) - open/close 13522 (time 1788799673.05 total 625.33 last 20.34) - open/close 13728 (time 1788799683.53 total 635.81 last 19.64) - open/close 13993 (time 1788799694.23 total 646.51 last 24.78) - open/close 14183 (time 1788799704.25 total 656.53 last 18.96) - open/close 14367 (time 1788799714.26 total 666.54 last 18.38) - open/close 14577 (time 1788799724.27 total 676.55 last 20.98) - open/close 14826 (time 1788799734.27 total 686.55 last 24.89) - open/close 15047 (time 1788799744.97 total 697.25 last 20.66) - open/close 15297 (time 1788799755.33 total 707.61 last 24.12) - open/close 15551 (time 1788799765.80 total 718.08 last 24.27) - open/close 15797 (time 1788799775.80 total 728.08 last 24.60) - open/close 16022 (time 1788799786.84 total 739.12 last 20.39) - open/close 16226 (time 1788799796.87 total 749.15 last 20.34) - open/close 16483 (time 1788799807.48 total 759.76 last 24.21) - open/close 16745 (time 1788799817.82 total 770.10 last 25.34) - open/close 17008 (time 1788799828.46 total 780.74 last 24.73) - open/close 17183 (time 1788799838.89 total 791.17 last 16.78) - open/close 17461 (time 1788799849.68 total 801.96 last 25.76) - open/close 17735 (time 1788799860.30 total 812.58 last 25.80) - open/close 17961 (time 1788799871.15 total 823.42 last 20.84) - open/close 18215 (time 1788799881.81 total 834.09 last 23.81) - open/close 18435 (time 1788799891.83 total 844.11 last 21.97) - open/close 18735 (time 1788799902.47 total 854.75 last 28.18) - open/close 19003 (time 1788799913.36 total 865.64 last 24.63) - open/close 19268 (time 1788799923.36 total 875.64 last 26.50) - open/close 19508 (time 1788799934.04 total 886.32 last 22.46) - open/close 19821 (time 1788799944.98 total 897.26 last 28.62) - open/close 20000 (time 1788799951.03 total 903.31 last 29.59) - open/close 20238 (time 1788799961.92 total 914.20 last 21.84) - open/close 20369 (time 1788799971.93 total 924.21 last 13.09) - open/close 20486 (time 1788799981.96 total 934.24 last 11.67) - open/close 21289 (time 1788799991.96 total 944.24 last 80.25) - open/close 22141 (time 1788800001.98 total 954.26 last 85.07) - open/close 22898 (time 1788800011.99 total 964.27 last 75.62) - open/close 23698 (time 1788800022.03 total 974.31 last 79.66) - open/close 24489 (time 1788800032.04 total 984.32 last 79.05) - open/close 25112 (time 1788800042.04 total 994.32 last 62.27) - open/close 25850 (time 1788800052.05 total 1004.32 last 73.77) - open/close 26609 (time 1788800062.05 total 1014.33 last 75.84) - open/close 27387 (time 1788800072.09 total 1024.37 last 77.52) - open/close 28181 (time 1788800082.09 total 1034.37 last 79.38) - open/close 28955 (time 1788800092.10 total 1044.38 last 77.35) - open/close 29828 (time 1788800102.10 total 1054.38 last 87.29) - open/close 30000 (time 1788800104.16 total 1056.44 last 83.32) - open/close 30801 (time 1788800114.18 total 1066.46 last 79.96) - open/close 31612 (time 1788800124.19 total 1076.47 last 81.05) - open/close 32437 (time 1788800134.20 total 1086.48 last 82.36) - open/close 33275 (time 1788800144.22 total 1096.50 last 83.64) - open/close 34064 (time 1788800154.23 total 1106.51 last 78.87) - open/close 34913 (time 1788800164.23 total 1116.51 last 84.84) - open/close 35755 (time 1788800174.24 total 1126.52 last 84.17) - open/close 36546 (time 1788800184.25 total 1136.52 last 79.04) - open/close 37395 (time 1788800194.25 total 1146.53 last 84.87) - open/close 38276 (time 1788800204.26 total 1156.54 last 87.97) - open/close 39125 (time 1788800214.27 total 1166.55 last 84.83) - open/close 39869 (time 1788800224.28 total 1176.56 last 74.36) - open/close 40731 (time 1788800234.29 total 1186.56 last 86.13) - open/close 41541 (time 1788800244.29 total 1196.57 last 80.96) - open/close 42394 (time 1788800254.31 total 1206.58 last 85.17) - open/close 43274 (time 1788800264.31 total 1216.59 last 87.96) - open/close 44173 (time 1788800274.32 total 1226.59 last 89.85) - open/close 44987 (time 1788800284.32 total 1236.60 last 81.40) - open/close 45863 (time 1788800294.33 total 1246.60 last 87.52) - open/close 46635 (time 1788800304.33 total 1256.61 last 77.17) - open/close 47467 (time 1788800314.34 total 1266.61 last 83.15) - open/close 48319 (time 1788800324.34 total 1276.62 last 85.17) - open/close 49121 (time 1788800334.34 total 1286.62 last 80.20) - open/close 49965 (time 1788800344.35 total 1296.63 last 84.32) total: 50000 open/close in 1297.08 seconds: 38.55 ops/second - unlinked 0 (time 1788800348 ; total 0 ; last 0) - unlinked 10000 (time 1788800473 ; total 125 ; last 125) - unlinked 20000 (time 1788800599 ; total 251 ; last 126) - unlinked 30000 (time 1788800732 ; total 384 ; last 133) - unlinked 40000 (time 1788800861 ; total 513 ; last 129) total: 50000 unlinks in 646 seconds: 77.399384 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg620-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server stop mds service on oleg620-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg620-server unloading modules via unload_modules_local on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing unload_modules_local oleg620-server: kmemleak disabled modules unloaded. pdsh@oleg620-client: oleg620-client: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-server: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-server: ssh exited with exit code 2 pdsh@oleg620-client: oleg620-client: ssh exited with exit code 2