== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:02:45 (1788451365) start mds service on oleg444-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 3 02:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: kmemleak disabled oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg444-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1788451459.61 total 10.01 last 12.99) - open/close 198 (time 1788451469.61 total 20.01 last 6.80) - open/close 424 (time 1788451479.63 total 30.03 last 22.56) - open/close 603 (time 1788451489.67 total 40.07 last 17.83) - open/close 820 (time 1788451500.53 total 50.93 last 19.97) - open/close 997 (time 1788451510.63 total 61.03 last 17.53) - open/close 1144 (time 1788451521.06 total 71.46 last 14.09) - open/close 1354 (time 1788451531.76 total 82.16 last 19.63) - open/close 1429 (time 1788451541.77 total 92.17 last 7.50) - open/close 1550 (time 1788451551.77 total 102.17 last 12.10) - open/close 1685 (time 1788451562.24 total 112.64 last 12.90) - open/close 1807 (time 1788451572.30 total 122.70 last 12.13) - open/close 1945 (time 1788451582.78 total 133.18 last 13.16) - open/close 2030 (time 1788451592.79 total 143.19 last 8.49) - open/close 2269 (time 1788451602.81 total 153.20 last 23.87) - open/close 2520 (time 1788451612.82 total 163.22 last 25.07) - open/close 2612 (time 1788451623.86 total 174.25 last 8.34) - open/close 2840 (time 1788451633.86 total 184.26 last 22.79) - open/close 3059 (time 1788451643.90 total 194.29 last 21.82) - open/close 3209 (time 1788451655.47 total 205.87 last 12.95) - open/close 3298 (time 1788451665.97 total 216.37 last 8.48) - open/close 3369 (time 1788451676.20 total 226.60 last 6.94) - open/close 3532 (time 1788451686.33 total 236.73 last 16.09) - open/close 3669 (time 1788451696.72 total 247.12 last 13.19) - open/close 3822 (time 1788451707.51 total 257.91 last 14.18) - open/close 3923 (time 1788451718.00 total 268.40 last 9.63) - open/close 4082 (time 1788451728.02 total 278.41 last 15.87) - open/close 4261 (time 1788451738.28 total 288.68 last 17.44) - open/close 4413 (time 1788451748.58 total 298.98 last 14.75) - open/close 4561 (time 1788451758.72 total 309.12 last 14.60) - open/close 4650 (time 1788451768.76 total 319.15 last 8.87) - open/close 4697 (time 1788451779.51 total 329.91 last 4.37) - open/close 4844 (time 1788451790.18 total 340.58 last 13.78) - open/close 4956 (time 1788451800.19 total 350.59 last 11.19) - open/close 5143 (time 1788451810.44 total 360.84 last 18.24) - open/close 5322 (time 1788451820.44 total 370.84 last 17.89) - open/close 5432 (time 1788451830.45 total 380.85 last 10.99) - open/close 5584 (time 1788451840.88 total 391.28 last 14.58) - open/close 5716 (time 1788451850.89 total 401.29 last 13.18) - open/close 5873 (time 1788451861.82 total 412.22 last 14.36) - open/close 6048 (time 1788451872.85 total 423.25 last 15.87) - open/close 6231 (time 1788451885.85 total 436.25 last 14.07) - open/close 6452 (time 1788451895.86 total 446.26 last 22.09) - open/close 6677 (time 1788451906.66 total 457.06 last 20.83) - open/close 6872 (time 1788451916.66 total 467.06 last 19.50) - open/close 7011 (time 1788451926.89 total 477.29 last 13.59) - open/close 7182 (time 1788451936.91 total 487.30 last 17.07) - open/close 7301 (time 1788451946.91 total 497.31 last 11.89) - open/close 7430 (time 1788451957.37 total 507.76 last 12.34) - open/close 7509 (time 1788451967.39 total 517.78 last 7.88) - open/close 7659 (time 1788451977.93 total 528.33 last 14.22) - open/close 7802 (time 1788451988.78 total 539.18 last 13.18) - open/close 7911 (time 1788451999.68 total 550.08 last 10.00) - open/close 8042 (time 1788452009.72 total 560.12 last 13.06) - open/close 8199 (time 1788452020.32 total 570.72 last 14.81) - open/close 8361 (time 1788452030.33 total 580.73 last 16.18) - open/close 8564 (time 1788452040.36 total 590.76 last 20.25) - open/close 8785 (time 1788452050.38 total 600.78 last 22.05) - open/close 8965 (time 1788452061.41 total 611.81 last 16.32) - open/close 9110 (time 1788452074.54 total 624.94 last 11.04) - open/close 9197 (time 1788452085.55 total 635.95 last 7.90) - open/close 9345 (time 1788452096.24 total 646.64 last 13.84) - open/close 9471 (time 1788452106.26 total 656.66 last 12.57) - open/close 9673 (time 1788452117.02 total 667.42 last 18.78) - open/close 9920 (time 1788452127.14 total 677.54 last 24.40) - open/close 10000 (time 1788452129.78 total 680.18 last 30.32) - open/close 10131 (time 1788452139.80 total 690.20 last 13.07) - open/close 10201 (time 1788452150.60 total 701.00 last 6.48) - open/close 10303 (time 1788452160.64 total 711.04 last 10.16) - open/close 10394 (time 1788452170.72 total 721.12 last 9.03) - open/close 10573 (time 1788452181.43 total 731.83 last 16.71) - open/close 10657 (time 1788452191.67 total 742.07 last 8.21) - open/close 10814 (time 1788452202.28 total 752.68 last 14.79) - open/close 10870 (time 1788452213.02 total 763.42 last 5.21) - open/close 11010 (time 1788452224.23 total 774.63 last 12.49) - open/close 11230 (time 1788452234.99 total 785.39 last 20.45) - open/close 11469 (time 1788452244.99 total 795.39 last 23.90) - open/close 11648 (time 1788452255.18 total 805.58 last 17.57) - open/close 11898 (time 1788452265.19 total 815.59 last 24.96) - open/close 12132 (time 1788452276.03 total 826.42 last 21.60) - open/close 12357 (time 1788452286.85 total 837.25 last 20.78) - open/close 12586 (time 1788452297.77 total 848.17 last 20.98) - open/close 12740 (time 1788452308.12 total 858.51 last 14.88) - open/close 12831 (time 1788452318.13 total 868.53 last 9.09) - open/close 13002 (time 1788452329.04 total 879.44 last 15.66) - open/close 13221 (time 1788452339.05 total 889.44 last 21.90) - open/close 13382 (time 1788452349.41 total 899.81 last 15.53) - open/close 13601 (time 1788452359.42 total 909.82 last 21.89) - open/close 13722 (time 1788452369.60 total 920.00 last 11.88) - open/close 13825 (time 1788452379.97 total 930.37 last 9.94) - open/close 14068 (time 1788452389.97 total 940.37 last 24.30) - open/close 14273 (time 1788452400.87 total 951.27 last 18.81) - open/close 14508 (time 1788452410.88 total 961.27 last 23.48) - open/close 14690 (time 1788452421.89 total 972.28 last 16.53) - open/close 14891 (time 1788452432.24 total 982.64 last 19.41) - open/close 15161 (time 1788452442.27 total 992.67 last 26.92) - open/close 15345 (time 1788452456.55 total 1006.95 last 12.89) - open/close 15487 (time 1788452467.23 total 1017.63 last 13.29) - open/close 15623 (time 1788452478.09 total 1028.49 last 12.53) - open/close 15821 (time 1788452489.00 total 1039.40 last 18.15) - open/close 15929 (time 1788452499.05 total 1049.45 last 10.74) - open/close 16168 (time 1788452510.01 total 1060.41 last 21.81) - open/close 16303 (time 1788452520.94 total 1071.34 last 12.35) - open/close 16672 (time 1788452530.95 total 1081.35 last 36.88) - open/close 17322 (time 1788452540.95 total 1091.35 last 64.99) - open/close 17959 (time 1788452550.96 total 1101.36 last 63.66) - open/close 18542 (time 1788452560.97 total 1111.37 last 58.23) - open/close 19115 (time 1788452570.97 total 1121.37 last 57.27) - open/close 19686 (time 1788452580.98 total 1131.38 last 57.07) - open/close 20000 (time 1788452586.19 total 1136.59 last 60.25) - open/close 20544 (time 1788452596.20 total 1146.59 last 54.38) - open/close 21199 (time 1788452606.21 total 1156.61 last 65.40) - open/close 21748 (time 1788452616.22 total 1166.62 last 54.86) - open/close 22314 (time 1788452626.23 total 1176.63 last 56.55) - open/close 22830 (time 1788452636.24 total 1186.64 last 51.56) - open/close 23378 (time 1788452646.25 total 1196.65 last 54.74) - open/close 23956 (time 1788452656.26 total 1206.66 last 57.71) - open/close 24401 (time 1788452666.28 total 1216.68 last 44.44) - open/close 24884 (time 1788452676.33 total 1226.73 last 48.04) - open/close 25412 (time 1788452686.33 total 1236.73 last 52.79) - open/close 26020 (time 1788452696.34 total 1246.74 last 60.75) - open/close 26614 (time 1788452706.35 total 1256.75 last 59.36) - open/close 27204 (time 1788452716.36 total 1266.76 last 58.96) - open/close 27805 (time 1788452726.37 total 1276.77 last 60.01) - open/close 28372 (time 1788452736.38 total 1286.78 last 56.64) - open/close 28961 (time 1788452746.40 total 1296.80 last 58.82) - open/close 29584 (time 1788452756.41 total 1306.80 last 62.25) - open/close 30000 (time 1788452762.30 total 1312.70 last 70.56) - open/close 30646 (time 1788452772.31 total 1322.71 last 64.51) - open/close 31312 (time 1788452782.33 total 1332.73 last 66.49) - open/close 31901 (time 1788452792.37 total 1342.77 last 58.65) - open/close 32473 (time 1788452802.38 total 1352.78 last 57.18) - open/close 33147 (time 1788452812.38 total 1362.78 last 67.36) - open/close 33734 (time 1788452822.38 total 1372.78 last 58.69) - open/close 34219 (time 1788452832.43 total 1382.83 last 48.27) - open/close 34744 (time 1788452842.44 total 1392.84 last 52.45) - open/close 35308 (time 1788452852.46 total 1402.85 last 56.32) - open/close 35897 (time 1788452862.47 total 1412.87 last 58.81) - open/close 36499 (time 1788452872.49 total 1422.89 last 60.08) - open/close 37037 (time 1788452882.50 total 1432.90 last 53.76) - open/close 37514 (time 1788452892.50 total 1442.90 last 47.68) - open/close 37962 (time 1788452902.51 total 1452.91 last 44.78) - open/close 38534 (time 1788452912.51 total 1462.91 last 57.20) - open/close 39104 (time 1788452922.52 total 1472.92 last 56.95) - open/close 39725 (time 1788452932.53 total 1482.93 last 62.04) - open/close 40000 (time 1788452936.45 total 1486.85 last 70.17) - open/close 40581 (time 1788452946.47 total 1496.87 last 57.98) - open/close 41208 (time 1788452956.47 total 1506.87 last 62.67) - open/close 41870 (time 1788452966.48 total 1516.88 last 66.16) - open/close 42544 (time 1788452976.48 total 1526.88 last 67.37) - open/close 43193 (time 1788452986.48 total 1536.88 last 64.90) - open/close 43799 (time 1788452996.50 total 1546.89 last 60.53) - open/close 44447 (time 1788453006.50 total 1556.90 last 64.77) - open/close 45103 (time 1788453016.50 total 1566.90 last 65.59) - open/close 45791 (time 1788453026.50 total 1576.90 last 68.78) - open/close 46461 (time 1788453036.51 total 1586.91 last 66.96) - open/close 47170 (time 1788453046.52 total 1596.91 last 70.86) - open/close 47749 (time 1788453056.52 total 1606.92 last 57.90) - open/close 48176 (time 1788453066.54 total 1616.93 last 42.62) - open/close 48594 (time 1788453076.55 total 1626.95 last 41.74) - open/close 49072 (time 1788453086.56 total 1636.96 last 47.76) - open/close 49545 (time 1788453096.58 total 1646.97 last 47.22) total: 50000 open/close in 1655.47 seconds: 30.20 ops/second - unlinked 0 (time 1788453110 ; total 0 ; last 0) - unlinked 10000 (time 1788453291 ; total 181 ; last 181) - unlinked 20000 (time 1788453461 ; total 351 ; last 170) - unlinked 30000 (time 1788453628 ; total 518 ; last 167) - unlinked 40000 (time 1788453791 ; total 681 ; last 163) total: 50000 unlinks in 844 seconds: 59.241707 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: kmemleak disabled modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2