== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:59:27 (1788425967) start mds service on oleg216-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921464 Sep 3 03:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: kmemleak disabled oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg216-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg216-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg216-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 156 (time 1788426049.08 total 10.35 last 15.07) - open/close 341 (time 1788426059.09 total 20.36 last 18.48) - open/close 513 (time 1788426071.97 total 33.24 last 13.35) - open/close 723 (time 1788426082.06 total 43.33 last 20.82) - open/close 860 (time 1788426092.09 total 53.36 last 13.66) - open/close 973 (time 1788426102.87 total 64.14 last 10.48) - open/close 1079 (time 1788426113.26 total 74.53 last 10.21) - open/close 1300 (time 1788426123.28 total 84.54 last 22.06) - open/close 1560 (time 1788426134.24 total 95.51 last 23.72) - open/close 1751 (time 1788426144.30 total 105.57 last 18.99) - open/close 1980 (time 1788426155.09 total 116.36 last 21.22) - open/close 2244 (time 1788426165.11 total 126.38 last 26.34) - open/close 2488 (time 1788426175.13 total 136.40 last 24.35) - open/close 2698 (time 1788426185.14 total 146.41 last 20.99) - open/close 2860 (time 1788426195.92 total 157.19 last 15.02) - open/close 3072 (time 1788426208.93 total 170.20 last 16.29) - open/close 3215 (time 1788426218.96 total 180.23 last 14.27) - open/close 3406 (time 1788426229.14 total 190.41 last 18.75) - open/close 3633 (time 1788426239.16 total 200.43 last 22.66) - open/close 3766 (time 1788426251.11 total 212.37 last 11.14) - open/close 3914 (time 1788426261.11 total 222.38 last 14.80) - open/close 4108 (time 1788426271.57 total 232.84 last 18.54) - open/close 4351 (time 1788426281.59 total 242.86 last 24.26) - open/close 4574 (time 1788426291.60 total 252.87 last 22.28) - open/close 4715 (time 1788426301.94 total 263.21 last 13.63) - open/close 4823 (time 1788426312.91 total 274.18 last 9.84) - open/close 5054 (time 1788426323.97 total 285.24 last 20.88) - open/close 5176 (time 1788426334.18 total 295.45 last 11.96) - open/close 5367 (time 1788426344.18 total 305.45 last 19.10) - open/close 5493 (time 1788426354.19 total 315.46 last 12.59) - open/close 5610 (time 1788426366.15 total 327.42 last 9.78) - open/close 5842 (time 1788426376.17 total 337.44 last 23.17) - open/close 6001 (time 1788426386.18 total 347.45 last 15.87) - open/close 6129 (time 1788426397.17 total 358.43 last 11.66) - open/close 6300 (time 1788426408.22 total 369.49 last 15.47) - open/close 6475 (time 1788426418.23 total 379.50 last 17.48) - open/close 6683 (time 1788426428.25 total 389.51 last 20.78) - open/close 6778 (time 1788426439.27 total 400.54 last 8.62) - open/close 6854 (time 1788426449.69 total 410.96 last 7.29) - open/close 6986 (time 1788426459.73 total 421.00 last 13.15) - open/close 7210 (time 1788426469.73 total 431.00 last 22.40) - open/close 7431 (time 1788426479.74 total 441.01 last 22.07) - open/close 7622 (time 1788426489.74 total 451.01 last 19.10) - open/close 7796 (time 1788426499.75 total 461.02 last 17.40) - open/close 7967 (time 1788426509.75 total 471.02 last 17.10) - open/close 8200 (time 1788426525.88 total 487.15 last 14.45) - open/close 8426 (time 1788426535.90 total 497.17 last 22.54) - open/close 8595 (time 1788426545.92 total 507.19 last 16.87) - open/close 8770 (time 1788426556.67 total 517.94 last 16.27) - open/close 8936 (time 1788426567.38 total 528.65 last 15.50) - open/close 9063 (time 1788426577.39 total 538.66 last 12.69) - open/close 9152 (time 1788426589.09 total 550.36 last 7.60) - open/close 9425 (time 1788426599.10 total 560.37 last 27.29) - open/close 9578 (time 1788426609.89 total 571.16 last 14.18) - open/close 9690 (time 1788426620.96 total 582.23 last 10.12) - open/close 9850 (time 1788426630.97 total 592.24 last 15.99) - open/close 9994 (time 1788426640.99 total 602.26 last 14.37) - open/close 10159 (time 1788426651.15 total 612.42 last 16.23) - open/close 10280 (time 1788426661.94 total 623.21 last 11.22) - open/close 10508 (time 1788426671.95 total 633.21 last 22.78) - open/close 10608 (time 1788426681.96 total 643.22 last 9.99) - open/close 10762 (time 1788426692.81 total 654.08 last 14.19) - open/close 10997 (time 1788426702.84 total 664.11 last 23.43) - open/close 11243 (time 1788426712.84 total 674.11 last 24.60) - open/close 11328 (time 1788426723.48 total 684.74 last 7.99) - open/close 11523 (time 1788426733.79 total 695.06 last 18.90) - open/close 11730 (time 1788426743.81 total 705.08 last 20.65) - open/close 11825 (time 1788426754.04 total 715.31 last 9.29) - open/close 11950 (time 1788426764.06 total 725.33 last 12.47) - open/close 12122 (time 1788426774.68 total 735.95 last 16.19) - open/close 12278 (time 1788426785.58 total 746.85 last 14.32) - open/close 12482 (time 1788426795.76 total 757.03 last 20.03) - open/close 12636 (time 1788426808.71 total 769.98 last 11.89) - open/close 12755 (time 1788426819.74 total 781.01 last 10.78) - open/close 12850 (time 1788426830.71 total 791.97 last 8.67) - open/close 12957 (time 1788426841.07 total 802.33 last 10.33) - open/close 13192 (time 1788426851.10 total 812.36 last 23.43) - open/close 13375 (time 1788426861.41 total 822.68 last 17.75) - open/close 13576 (time 1788426872.29 total 833.55 last 18.48) - open/close 13720 (time 1788426882.29 total 843.56 last 14.40) - open/close 13804 (time 1788426892.97 total 854.24 last 7.86) - open/close 13886 (time 1788426905.96 total 867.23 last 6.31) - open/close 14062 (time 1788426916.25 total 877.52 last 17.09) - open/close 14175 (time 1788426926.56 total 887.83 last 10.96) - open/close 14277 (time 1788426936.62 total 897.89 last 10.14) - open/close 14473 (time 1788426946.94 total 908.21 last 19.00) - open/close 14596 (time 1788426957.55 total 918.82 last 11.59) - open/close 14743 (time 1788426967.57 total 928.84 last 14.68) - open/close 14968 (time 1788426978.40 total 939.67 last 20.76) - open/close 15198 (time 1788426989.41 total 950.68 last 20.89) - open/close 15322 (time 1788426999.87 total 961.14 last 11.86) - open/close 15476 (time 1788427009.88 total 971.15 last 15.38) - open/close 15588 (time 1788427020.70 total 981.97 last 10.35) - open/close 15643 (time 1788427031.44 total 992.71 last 5.12) - open/close 15716 (time 1788427042.54 total 1003.81 last 6.58) - open/close 15812 (time 1788427052.55 total 1013.82 last 9.60) - open/close 15901 (time 1788427062.77 total 1024.04 last 8.70) - open/close 16073 (time 1788427073.82 total 1035.09 last 15.58) - open/close 16157 (time 1788427083.82 total 1045.09 last 8.39) - open/close 16262 (time 1788427094.37 total 1055.63 last 9.96) - open/close 16367 (time 1788427104.47 total 1065.74 last 10.39) - open/close 16505 (time 1788427115.18 total 1076.45 last 12.89) - open/close 16854 (time 1788427125.19 total 1086.46 last 34.85) - open/close 17329 (time 1788427135.20 total 1096.47 last 47.46) - open/close 17850 (time 1788427145.21 total 1106.47 last 52.07) - open/close 18441 (time 1788427155.21 total 1116.48 last 59.04) - open/close 19078 (time 1788427165.23 total 1126.50 last 63.61) - open/close 19670 (time 1788427175.24 total 1136.50 last 59.16) - open/close 20000 (time 1788427180.22 total 1141.49 last 66.20) - open/close 20593 (time 1788427190.23 total 1151.50 last 59.25) - open/close 21210 (time 1788427200.24 total 1161.51 last 61.63) - open/close 21784 (time 1788427210.24 total 1171.51 last 57.37) - open/close 22335 (time 1788427220.25 total 1181.51 last 55.10) - open/close 22997 (time 1788427230.27 total 1191.54 last 66.05) - open/close 23717 (time 1788427240.27 total 1201.54 last 71.99) - open/close 24186 (time 1788427250.32 total 1211.59 last 46.65) - open/close 24670 (time 1788427260.33 total 1221.60 last 48.37) - open/close 25185 (time 1788427270.34 total 1231.61 last 51.43) - open/close 25721 (time 1788427280.34 total 1241.61 last 53.59) - open/close 26191 (time 1788427290.37 total 1251.64 last 46.87) - open/close 26627 (time 1788427300.40 total 1261.67 last 43.47) - open/close 27126 (time 1788427310.41 total 1271.68 last 49.87) - open/close 27548 (time 1788427320.41 total 1281.68 last 42.18) - open/close 28103 (time 1788427330.42 total 1291.69 last 55.47) - open/close 28755 (time 1788427340.43 total 1301.70 last 65.14) - open/close 29359 (time 1788427350.43 total 1311.70 last 60.40) - open/close 30000 (time 1788427359.89 total 1321.16 last 67.72) - open/close 30681 (time 1788427369.90 total 1331.17 last 68.06) - open/close 31420 (time 1788427379.90 total 1341.17 last 73.86) - open/close 32120 (time 1788427389.91 total 1351.18 last 69.95) - open/close 32769 (time 1788427399.92 total 1361.19 last 64.84) - open/close 33355 (time 1788427409.93 total 1371.20 last 58.53) - open/close 33935 (time 1788427419.94 total 1381.20 last 57.98) - open/close 34525 (time 1788427429.94 total 1391.21 last 58.95) - open/close 35142 (time 1788427439.95 total 1401.21 last 61.69) - open/close 35630 (time 1788427450.03 total 1411.30 last 48.39) - open/close 36275 (time 1788427460.04 total 1421.31 last 64.43) - open/close 36921 (time 1788427470.05 total 1431.32 last 64.56) - open/close 37493 (time 1788427480.06 total 1441.33 last 57.12) - open/close 38138 (time 1788427490.07 total 1451.34 last 64.43) - open/close 38840 (time 1788427500.09 total 1461.36 last 70.09) - open/close 39523 (time 1788427510.09 total 1471.36 last 68.29) - open/close 40000 (time 1788427517.59 total 1478.86 last 63.56) - open/close 40611 (time 1788427527.60 total 1488.87 last 61.09) - open/close 41230 (time 1788427537.60 total 1498.87 last 61.89) - open/close 41949 (time 1788427547.61 total 1508.88 last 71.79) - open/close 42559 (time 1788427557.65 total 1518.92 last 60.80) - open/close 43173 (time 1788427567.67 total 1528.94 last 61.26) - open/close 43932 (time 1788427577.68 total 1538.95 last 75.80) - open/close 44506 (time 1788427587.69 total 1548.96 last 57.34) - open/close 45139 (time 1788427597.69 total 1558.96 last 63.29) - open/close 45812 (time 1788427607.70 total 1568.97 last 67.27) - open/close 46423 (time 1788427617.71 total 1578.97 last 61.05) - open/close 47088 (time 1788427627.71 total 1588.98 last 66.47) - open/close 47793 (time 1788427637.72 total 1598.99 last 70.43) - open/close 48441 (time 1788427647.72 total 1608.99 last 64.77) - open/close 49120 (time 1788427657.73 total 1618.99 last 67.89) - open/close 49671 (time 1788427667.73 total 1629.00 last 55.05) total: 50000 open/close in 1635.08 seconds: 30.58 ops/second - unlinked 0 (time 1788427680 ; total 0 ; last 0) - unlinked 10000 (time 1788427851 ; total 171 ; last 171) - unlinked 20000 (time 1788428020 ; total 340 ; last 169) - unlinked 30000 (time 1788428192 ; total 512 ; last 172) - unlinked 40000 (time 1788428353 ; total 673 ; last 161) total: 50000 unlinks in 831 seconds: 60.168472 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server unloading modules via unload_modules_local on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: kmemleak disabled modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2