== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:39:25 (1788655165) start mds service on oleg144-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921680 Sep 5 00:29 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: kmemleak disabled oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg144-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg144-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg144-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1788655256.26 total 10.01 last 23.37) - open/close 391 (time 1788655266.65 total 20.40 last 15.12) - open/close 628 (time 1788655277.64 total 31.39 last 21.56) - open/close 859 (time 1788655288.47 total 42.22 last 21.33) - open/close 1083 (time 1788655298.48 total 52.23 last 22.37) - open/close 1320 (time 1788655309.03 total 62.77 last 22.48) - open/close 1521 (time 1788655320.14 total 73.89 last 18.09) - open/close 1742 (time 1788655330.52 total 84.27 last 21.29) - open/close 1858 (time 1788655341.48 total 95.23 last 10.58) - open/close 1996 (time 1788655352.39 total 106.14 last 12.64) - open/close 2203 (time 1788655362.43 total 116.18 last 20.62) - open/close 2304 (time 1788655374.24 total 127.99 last 8.55) - open/close 2485 (time 1788655384.24 total 137.99 last 18.10) - open/close 2663 (time 1788655394.25 total 148.00 last 17.78) - open/close 2895 (time 1788655404.27 total 158.02 last 23.16) - open/close 3093 (time 1788655414.29 total 168.04 last 19.75) - open/close 3251 (time 1788655424.55 total 178.30 last 15.40) - open/close 3505 (time 1788655434.56 total 188.31 last 25.38) - open/close 3650 (time 1788655444.94 total 198.69 last 13.96) - open/close 3821 (time 1788655456.53 total 210.27 last 14.76) - open/close 4023 (time 1788655466.78 total 220.53 last 19.70) - open/close 4211 (time 1788655477.14 total 230.89 last 18.15) - open/close 4414 (time 1788655487.81 total 241.56 last 19.02) - open/close 4539 (time 1788655497.82 total 251.57 last 12.49) - open/close 4710 (time 1788655508.50 total 262.25 last 16.01) - open/close 4972 (time 1788655518.51 total 272.26 last 26.17) - open/close 5115 (time 1788655528.87 total 282.61 last 13.81) - open/close 5186 (time 1788655539.16 total 292.90 last 6.90) - open/close 5373 (time 1788655549.84 total 303.59 last 17.50) - open/close 5587 (time 1788655561.43 total 315.18 last 18.47) - open/close 5812 (time 1788655571.61 total 325.35 last 22.11) - open/close 6011 (time 1788655581.61 total 335.36 last 19.90) - open/close 6103 (time 1788655592.07 total 345.82 last 8.79) - open/close 6350 (time 1788655602.96 total 356.71 last 22.68) - open/close 6580 (time 1788655612.97 total 366.72 last 22.98) - open/close 6677 (time 1788655623.32 total 377.07 last 9.37) - open/close 6801 (time 1788655633.36 total 387.11 last 12.35) - open/close 7043 (time 1788655644.33 total 398.07 last 22.07) - open/close 7169 (time 1788655655.07 total 408.82 last 11.73) - open/close 7263 (time 1788655666.09 total 419.84 last 8.53) - open/close 7515 (time 1788655676.10 total 429.85 last 25.18) - open/close 7741 (time 1788655686.11 total 439.86 last 22.58) - open/close 7918 (time 1788655696.11 total 449.86 last 17.70) - open/close 8112 (time 1788655706.13 total 459.88 last 19.36) - open/close 8247 (time 1788655717.14 total 470.89 last 12.27) - open/close 8451 (time 1788655727.79 total 481.54 last 19.15) - open/close 8700 (time 1788655737.82 total 491.57 last 24.82) - open/close 8956 (time 1788655747.84 total 501.59 last 25.56) - open/close 9179 (time 1788655758.31 total 512.06 last 21.29) - open/close 9402 (time 1788655768.32 total 522.07 last 22.27) - open/close 9618 (time 1788655778.77 total 532.52 last 20.67) - open/close 9841 (time 1788655788.78 total 542.53 last 22.27) - open/close 9892 (time 1788655799.12 total 552.87 last 4.93) - open/close 10000 (time 1788655803.56 total 557.30 last 24.36) - open/close 10150 (time 1788655813.56 total 567.31 last 14.99) - open/close 10307 (time 1788655824.43 total 578.18 last 14.45) - open/close 10498 (time 1788655835.14 total 588.89 last 17.83) - open/close 10686 (time 1788655845.15 total 598.90 last 18.79) - open/close 10845 (time 1788655855.31 total 609.06 last 15.65) - open/close 11016 (time 1788655866.34 total 620.09 last 15.50) - open/close 11126 (time 1788655876.84 total 630.59 last 10.48) - open/close 11222 (time 1788655887.53 total 641.28 last 8.98) - open/close 11351 (time 1788655897.55 total 651.30 last 12.87) - open/close 11454 (time 1788655908.06 total 661.81 last 9.80) - open/close 11572 (time 1788655918.61 total 672.36 last 11.19) - open/close 11660 (time 1788655928.62 total 682.37 last 8.79) - open/close 11741 (time 1788655938.63 total 692.38 last 8.09) - open/close 11868 (time 1788655948.63 total 702.38 last 12.70) - open/close 12042 (time 1788655958.63 total 712.38 last 17.40) - open/close 12209 (time 1788655969.02 total 722.76 last 16.08) - open/close 12415 (time 1788655980.31 total 734.06 last 18.24) - open/close 12684 (time 1788655990.32 total 744.07 last 26.88) - open/close 12877 (time 1788656001.32 total 755.07 last 17.54) - open/close 13092 (time 1788656011.69 total 765.44 last 20.73) - open/close 13356 (time 1788656021.70 total 775.45 last 26.38) - open/close 13587 (time 1788656032.72 total 786.47 last 20.96) - open/close 13792 (time 1788656043.03 total 796.77 last 19.89) - open/close 14031 (time 1788656053.04 total 806.78 last 23.88) - open/close 14273 (time 1788656064.02 total 817.77 last 22.04) - open/close 14350 (time 1788656074.04 total 827.79 last 7.68) - open/close 14522 (time 1788656084.79 total 838.54 last 15.99) - open/close 14776 (time 1788656094.80 total 848.55 last 25.38) - open/close 14875 (time 1788656104.86 total 858.61 last 9.85) - open/close 15012 (time 1788656115.49 total 869.23 last 12.89) - open/close 15168 (time 1788656126.86 total 880.61 last 13.72) - open/close 15355 (time 1788656136.86 total 890.61 last 18.69) - open/close 15452 (time 1788656147.04 total 900.79 last 9.53) - open/close 15525 (time 1788656157.73 total 911.48 last 6.83) - open/close 15722 (time 1788656167.88 total 921.63 last 19.40) - open/close 15895 (time 1788656177.89 total 931.64 last 17.29) - open/close 15951 (time 1788656188.89 total 942.64 last 5.09) - open/close 15994 (time 1788656200.48 total 954.23 last 3.71) - open/close 16061 (time 1788656211.48 total 965.23 last 6.09) - open/close 16249 (time 1788656221.51 total 975.26 last 18.75) - open/close 16389 (time 1788656231.90 total 985.65 last 13.47) - open/close 16612 (time 1788656242.36 total 996.11 last 21.32) - open/close 16794 (time 1788656252.77 total 1006.52 last 17.48) - open/close 16961 (time 1788656264.18 total 1017.93 last 14.64) - open/close 17683 (time 1788656274.19 total 1027.94 last 72.13) - open/close 18319 (time 1788656284.20 total 1037.95 last 63.54) - open/close 19090 (time 1788656294.21 total 1047.96 last 77.01) - open/close 19832 (time 1788656304.23 total 1057.98 last 74.08) - open/close 20000 (time 1788656306.62 total 1060.37 last 70.13) - open/close 20669 (time 1788656316.63 total 1070.38 last 66.84) - open/close 21410 (time 1788656326.66 total 1080.41 last 73.89) - open/close 22112 (time 1788656336.68 total 1090.43 last 70.06) - open/close 22824 (time 1788656346.68 total 1100.43 last 71.19) - open/close 23532 (time 1788656356.68 total 1110.43 last 70.79) - open/close 24354 (time 1788656366.69 total 1120.43 last 82.17) - open/close 25081 (time 1788656376.69 total 1130.44 last 72.66) - open/close 25759 (time 1788656386.71 total 1140.46 last 67.66) - open/close 26420 (time 1788656396.73 total 1150.48 last 65.96) - open/close 27094 (time 1788656406.74 total 1160.49 last 67.34) - open/close 27802 (time 1788656416.75 total 1170.49 last 70.78) - open/close 28480 (time 1788656426.75 total 1180.50 last 67.79) - open/close 29175 (time 1788656436.75 total 1190.50 last 69.47) - open/close 29712 (time 1788656446.78 total 1200.53 last 53.54) - open/close 30000 (time 1788656451.71 total 1205.46 last 58.42) - open/close 30616 (time 1788656461.73 total 1215.48 last 61.46) - open/close 31090 (time 1788656471.74 total 1225.49 last 47.37) - open/close 31829 (time 1788656481.74 total 1235.49 last 73.89) - open/close 32524 (time 1788656491.74 total 1245.49 last 69.49) - open/close 33261 (time 1788656501.76 total 1255.50 last 73.61) - open/close 33956 (time 1788656511.76 total 1265.51 last 69.45) - open/close 34654 (time 1788656521.77 total 1275.52 last 69.75) - open/close 35251 (time 1788656531.79 total 1285.54 last 59.60) - open/close 35907 (time 1788656541.80 total 1295.54 last 65.55) - open/close 36575 (time 1788656551.80 total 1305.54 last 66.79) - open/close 37308 (time 1788656561.80 total 1315.55 last 73.24) - open/close 37977 (time 1788656571.81 total 1325.56 last 66.86) - open/close 38555 (time 1788656581.84 total 1335.59 last 57.64) - open/close 39249 (time 1788656591.84 total 1345.59 last 69.39) - open/close 40000 (time 1788656601.07 total 1354.82 last 81.34) - open/close 40697 (time 1788656611.09 total 1364.84 last 69.57) - open/close 41342 (time 1788656621.09 total 1374.84 last 64.47) - open/close 42001 (time 1788656631.10 total 1384.85 last 65.85) - open/close 42584 (time 1788656641.12 total 1394.87 last 58.19) - open/close 43303 (time 1788656651.13 total 1404.88 last 71.85) - open/close 44016 (time 1788656661.14 total 1414.89 last 71.20) - open/close 44713 (time 1788656671.16 total 1424.90 last 69.61) - open/close 45469 (time 1788656681.16 total 1434.91 last 75.59) - open/close 46132 (time 1788656691.17 total 1444.92 last 66.23) - open/close 46779 (time 1788656701.18 total 1454.93 last 64.61) - open/close 47507 (time 1788656711.19 total 1464.94 last 72.74) - open/close 48298 (time 1788656721.21 total 1474.96 last 78.96) - open/close 49051 (time 1788656731.22 total 1484.97 last 75.22) - open/close 49780 (time 1788656741.23 total 1494.97 last 72.85) total: 50000 open/close in 1497.90 seconds: 33.38 ops/second - unlinked 0 (time 1788656748 ; total 0 ; last 0) - unlinked 10000 (time 1788656896 ; total 148 ; last 148) - unlinked 20000 (time 1788657042 ; total 294 ; last 146) - unlinked 30000 (time 1788657183 ; total 435 ; last 141) - unlinked 40000 (time 1788657325 ; total 577 ; last 142) total: 50000 unlinks in 718 seconds: 69.637886 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: kmemleak disabled modules unloaded. pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2