== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:04:45 (1789178685) start mds service on oleg327-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 11 09:35 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: kmemleak disabled oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg327-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg327-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 16 (time 1789178771.91 total 10.76 last 1.49) - open/close 246 (time 1789178781.91 total 20.76 last 23.00) - open/close 466 (time 1789178792.62 total 31.47 last 20.54) - open/close 666 (time 1789178802.65 total 41.50 last 19.95) - open/close 860 (time 1789178813.15 total 52.00 last 18.46) - open/close 1091 (time 1789178825.25 total 64.10 last 19.10) - open/close 1180 (time 1789178836.25 total 75.10 last 8.09) - open/close 1286 (time 1789178847.91 total 86.76 last 9.09) - open/close 1531 (time 1789178857.93 total 96.78 last 24.46) - open/close 1782 (time 1789178868.95 total 107.81 last 22.76) - open/close 1924 (time 1789178879.33 total 118.18 last 13.69) - open/close 2136 (time 1789178889.78 total 128.63 last 20.30) - open/close 2349 (time 1789178902.32 total 141.17 last 16.97) - open/close 2479 (time 1789178912.69 total 151.54 last 12.54) - open/close 2682 (time 1789178923.31 total 162.16 last 19.12) - open/close 2869 (time 1789178935.42 total 174.27 last 15.44) - open/close 3057 (time 1789178945.57 total 184.42 last 18.51) - open/close 3188 (time 1789178957.89 total 196.74 last 10.64) - open/close 3319 (time 1789178968.51 total 207.36 last 12.33) - open/close 3490 (time 1789178978.52 total 217.37 last 17.08) - open/close 3606 (time 1789178989.35 total 228.20 last 10.71) - open/close 3684 (time 1789178999.87 total 238.73 last 7.41) - open/close 3736 (time 1789179010.54 total 249.39 last 4.88) - open/close 3858 (time 1789179020.55 total 259.40 last 12.18) - open/close 4026 (time 1789179030.83 total 269.68 last 16.34) - open/close 4159 (time 1789179040.99 total 279.84 last 13.10) - open/close 4397 (time 1789179052.00 total 290.85 last 21.61) - open/close 4645 (time 1789179062.02 total 300.87 last 24.76) - open/close 4902 (time 1789179072.16 total 311.01 last 25.33) - open/close 5046 (time 1789179082.79 total 321.64 last 13.55) - open/close 5212 (time 1789179093.35 total 332.20 last 15.73) - open/close 5451 (time 1789179104.02 total 342.87 last 22.39) - open/close 5551 (time 1789179114.03 total 352.88 last 9.99) - open/close 5657 (time 1789179125.10 total 363.95 last 9.58) - open/close 5813 (time 1789179135.10 total 373.96 last 15.59) - open/close 5991 (time 1789179147.29 total 386.14 last 14.61) - open/close 6189 (time 1789179157.59 total 396.44 last 19.21) - open/close 6314 (time 1789179167.60 total 406.45 last 12.49) - open/close 6512 (time 1789179177.83 total 416.68 last 19.36) - open/close 6716 (time 1789179188.18 total 427.03 last 19.71) - open/close 6933 (time 1789179198.19 total 437.04 last 21.69) - open/close 7109 (time 1789179208.20 total 447.05 last 17.58) - open/close 7198 (time 1789179218.59 total 457.44 last 8.56) - open/close 7364 (time 1789179229.55 total 468.40 last 15.15) - open/close 7595 (time 1789179239.91 total 478.76 last 22.31) - open/close 7768 (time 1789179249.92 total 488.77 last 17.28) - open/close 7948 (time 1789179260.71 total 499.56 last 16.68) - open/close 8195 (time 1789179271.73 total 510.58 last 22.41) - open/close 8336 (time 1789179283.56 total 522.41 last 11.92) - open/close 8587 (time 1789179294.43 total 533.28 last 23.09) - open/close 8730 (time 1789179304.45 total 543.30 last 14.28) - open/close 8948 (time 1789179314.98 total 553.83 last 20.71) - open/close 9191 (time 1789179325.01 total 563.86 last 24.23) - open/close 9364 (time 1789179336.41 total 575.26 last 15.17) - open/close 9488 (time 1789179347.61 total 586.46 last 11.07) - open/close 9653 (time 1789179357.61 total 596.46 last 16.49) - open/close 9862 (time 1789179368.30 total 607.15 last 19.55) - open/close 10000 (time 1789179374.21 total 613.06 last 23.38) - open/close 10195 (time 1789179385.11 total 623.96 last 17.88) - open/close 10447 (time 1789179395.12 total 633.97 last 25.17) - open/close 10702 (time 1789179406.05 total 644.90 last 23.33) - open/close 10959 (time 1789179416.08 total 654.93 last 25.62) - open/close 11082 (time 1789179427.80 total 666.65 last 10.50) - open/close 11301 (time 1789179438.44 total 677.29 last 20.59) - open/close 11545 (time 1789179448.46 total 687.31 last 24.34) - open/close 11750 (time 1789179458.47 total 697.32 last 20.47) - open/close 11912 (time 1789179468.49 total 707.34 last 16.18) - open/close 12157 (time 1789179478.87 total 717.72 last 23.58) - open/close 12405 (time 1789179489.77 total 728.62 last 22.77) - open/close 12609 (time 1789179499.87 total 738.72 last 20.19) - open/close 12865 (time 1789179509.89 total 748.74 last 25.55) - open/close 13048 (time 1789179520.07 total 758.92 last 17.98) - open/close 13126 (time 1789179530.81 total 769.66 last 7.27) - open/close 13364 (time 1789179541.53 total 780.38 last 22.19) - open/close 13537 (time 1789179551.75 total 790.60 last 16.93) - open/close 13785 (time 1789179562.71 total 801.56 last 22.63) - open/close 14084 (time 1789179573.67 total 812.52 last 27.29) - open/close 14340 (time 1789179584.74 total 823.59 last 23.13) - open/close 14567 (time 1789179594.76 total 833.61 last 22.65) - open/close 14762 (time 1789179604.79 total 843.64 last 19.44) - open/close 15012 (time 1789179615.86 total 854.71 last 22.58) - open/close 15244 (time 1789179626.47 total 865.32 last 21.87) - open/close 15513 (time 1789179636.47 total 875.32 last 26.89) - open/close 15731 (time 1789179647.47 total 886.32 last 19.82) - open/close 15910 (time 1789179657.50 total 896.35 last 17.85) - open/close 16059 (time 1789179667.53 total 906.38 last 14.86) - open/close 16172 (time 1789179677.53 total 916.38 last 11.29) - open/close 16272 (time 1789179687.54 total 926.39 last 10.00) - open/close 16387 (time 1789179697.55 total 936.40 last 11.49) - open/close 16601 (time 1789179708.46 total 947.31 last 19.60) - open/close 16849 (time 1789179718.49 total 957.34 last 24.73) - open/close 17094 (time 1789179728.80 total 967.65 last 23.77) - open/close 17319 (time 1789179738.91 total 977.76 last 22.25) - open/close 17554 (time 1789179749.13 total 987.98 last 23.00) - open/close 18219 (time 1789179759.13 total 997.98 last 66.44) - open/close 18943 (time 1789179769.14 total 1007.99 last 72.37) - open/close 19586 (time 1789179779.14 total 1017.99 last 64.29) - open/close 20000 (time 1789179784.68 total 1023.53 last 74.73) - open/close 20683 (time 1789179794.69 total 1033.54 last 68.25) - open/close 21360 (time 1789179804.69 total 1043.54 last 67.67) - open/close 22060 (time 1789179814.70 total 1053.55 last 69.97) - open/close 22693 (time 1789179824.70 total 1063.55 last 63.29) - open/close 23366 (time 1789179834.71 total 1073.56 last 67.21) - open/close 24027 (time 1789179844.72 total 1083.57 last 66.06) - open/close 24635 (time 1789179854.72 total 1093.57 last 60.77) - open/close 25194 (time 1789179864.73 total 1103.58 last 55.85) - open/close 25688 (time 1789179874.74 total 1113.59 last 49.35) - open/close 26277 (time 1789179884.76 total 1123.61 last 58.77) - open/close 26892 (time 1789179894.77 total 1133.62 last 61.46) - open/close 27524 (time 1789179904.78 total 1143.63 last 63.13) - open/close 28154 (time 1789179914.79 total 1153.64 last 62.90) - open/close 28829 (time 1789179924.81 total 1163.66 last 67.42) - open/close 29550 (time 1789179934.81 total 1173.66 last 72.09) - open/close 30000 (time 1789179942.13 total 1180.98 last 61.46) - open/close 30640 (time 1789179952.14 total 1190.99 last 63.93) - open/close 31289 (time 1789179962.15 total 1201.00 last 64.85) - open/close 31930 (time 1789179972.15 total 1211.00 last 64.08) - open/close 32573 (time 1789179982.15 total 1221.00 last 64.29) - open/close 33260 (time 1789179992.16 total 1231.02 last 68.61) - open/close 33963 (time 1789180002.17 total 1241.02 last 70.29) - open/close 34568 (time 1789180012.18 total 1251.03 last 60.41) - open/close 35067 (time 1789180022.18 total 1261.03 last 49.89) - open/close 35681 (time 1789180032.19 total 1271.04 last 61.35) - open/close 36302 (time 1789180042.19 total 1281.05 last 62.08) - open/close 36918 (time 1789180052.20 total 1291.05 last 61.58) - open/close 37555 (time 1789180062.21 total 1301.06 last 63.63) - open/close 38172 (time 1789180072.21 total 1311.06 last 61.68) - open/close 38787 (time 1789180082.22 total 1321.07 last 61.46) - open/close 39452 (time 1789180092.24 total 1331.09 last 66.39) - open/close 40000 (time 1789180100.48 total 1339.33 last 66.46) - open/close 40650 (time 1789180110.50 total 1349.35 last 64.90) - open/close 41317 (time 1789180120.50 total 1359.35 last 66.70) - open/close 41951 (time 1789180130.50 total 1369.35 last 63.38) - open/close 42586 (time 1789180140.56 total 1379.41 last 63.16) - open/close 43246 (time 1789180150.56 total 1389.41 last 65.99) - open/close 43927 (time 1789180160.57 total 1399.42 last 68.03) - open/close 44591 (time 1789180170.58 total 1409.43 last 66.35) - open/close 45240 (time 1789180180.58 total 1419.43 last 64.87) - open/close 45893 (time 1789180190.58 total 1429.43 last 65.28) - open/close 46588 (time 1789180200.59 total 1439.44 last 69.45) - open/close 47246 (time 1789180210.60 total 1449.45 last 65.75) - open/close 47984 (time 1789180220.61 total 1459.46 last 73.74) - open/close 48655 (time 1789180230.61 total 1469.46 last 67.04) - open/close 49317 (time 1789180240.62 total 1479.47 last 66.17) - open/close 49849 (time 1789180250.62 total 1489.47 last 53.20) total: 50000 open/close in 1491.90 seconds: 33.51 ops/second - unlinked 0 (time 1789180257 ; total 0 ; last 0) - unlinked 10000 (time 1789180418 ; total 161 ; last 161) - unlinked 20000 (time 1789180577 ; total 320 ; last 159) - unlinked 30000 (time 1789180739 ; total 482 ; last 162) - unlinked 40000 (time 1789180900 ; total 643 ; last 161) total: 50000 unlinks in 808 seconds: 61.881187 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: kmemleak disabled modules unloaded. pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2