== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:51:43 (1790974303) start mds service on oleg430-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 2 13:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: kmemleak disabled oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg430-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg430-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1790974377.82 total 10.05 last 15.92) - open/close 414 (time 1790974387.82 total 20.05 last 25.40) - open/close 634 (time 1790974398.08 total 30.30 last 21.46) - open/close 854 (time 1790974408.62 total 40.85 last 20.87) - open/close 1012 (time 1790974420.65 total 52.88 last 13.13) - open/close 1279 (time 1790974430.66 total 62.89 last 26.66) - open/close 1438 (time 1790974440.89 total 73.12 last 15.55) - open/close 1595 (time 1790974451.55 total 83.78 last 14.72) - open/close 1843 (time 1790974461.56 total 93.79 last 24.79) - open/close 2045 (time 1790974471.57 total 103.80 last 20.18) - open/close 2331 (time 1790974481.58 total 113.81 last 28.58) - open/close 2547 (time 1790974491.59 total 123.82 last 21.56) - open/close 2739 (time 1790974502.37 total 134.60 last 17.82) - open/close 2985 (time 1790974512.93 total 145.16 last 23.29) - open/close 3242 (time 1790974523.70 total 155.93 last 23.87) - open/close 3445 (time 1790974536.57 total 168.80 last 15.77) - open/close 3675 (time 1790974546.62 total 178.85 last 22.89) - open/close 3833 (time 1790974556.62 total 188.85 last 15.79) - open/close 3974 (time 1790974566.64 total 198.87 last 14.08) - open/close 4240 (time 1790974577.52 total 209.75 last 24.45) - open/close 4460 (time 1790974587.52 total 219.75 last 21.99) - open/close 4702 (time 1790974597.94 total 230.17 last 23.24) - open/close 4880 (time 1790974607.94 total 240.17 last 17.80) - open/close 5116 (time 1790974618.92 total 251.15 last 21.48) - open/close 5294 (time 1790974629.09 total 261.32 last 17.51) - open/close 5529 (time 1790974639.84 total 272.07 last 21.86) - open/close 5818 (time 1790974649.85 total 282.08 last 28.86) - open/close 6015 (time 1790974659.86 total 292.09 last 19.69) - open/close 6301 (time 1790974669.86 total 302.09 last 28.58) - open/close 6457 (time 1790974680.04 total 312.27 last 15.33) - open/close 6728 (time 1790974691.05 total 323.27 last 24.63) - open/close 6930 (time 1790974701.92 total 334.15 last 18.57) - open/close 7121 (time 1790974711.93 total 344.16 last 19.10) - open/close 7369 (time 1790974721.96 total 354.19 last 24.71) - open/close 7511 (time 1790974732.78 total 365.01 last 13.13) - open/close 7779 (time 1790974742.79 total 375.02 last 26.77) - open/close 8037 (time 1790974752.80 total 385.03 last 25.76) - open/close 8284 (time 1790974762.86 total 395.09 last 24.57) - open/close 8499 (time 1790974773.86 total 406.09 last 19.55) - open/close 8641 (time 1790974783.87 total 416.10 last 14.18) - open/close 8846 (time 1790974794.62 total 426.85 last 19.07) - open/close 9017 (time 1790974805.15 total 437.38 last 16.25) - open/close 9143 (time 1790974815.60 total 447.83 last 12.06) - open/close 9228 (time 1790974826.15 total 458.38 last 8.05) - open/close 9345 (time 1790974836.33 total 468.56 last 11.49) - open/close 9544 (time 1790974847.26 total 479.49 last 18.21) - open/close 9819 (time 1790974857.77 total 490.00 last 26.18) - open/close 10000 (time 1790974865.77 total 498.00 last 22.63) - open/close 10245 (time 1790974875.78 total 508.01 last 24.48) - open/close 10510 (time 1790974886.24 total 518.47 last 25.33) - open/close 10709 (time 1790974896.24 total 528.47 last 19.89) - open/close 10907 (time 1790974906.28 total 538.51 last 19.73) - open/close 11113 (time 1790974916.64 total 548.87 last 19.87) - open/close 11265 (time 1790974926.81 total 559.04 last 14.95) - open/close 11505 (time 1790974937.86 total 570.09 last 21.73) - open/close 11699 (time 1790974950.50 total 582.73 last 15.35) - open/close 11864 (time 1790974961.02 total 593.25 last 15.68) - open/close 12064 (time 1790974971.02 total 603.25 last 20.00) - open/close 12275 (time 1790974981.03 total 613.26 last 21.08) - open/close 12483 (time 1790974991.89 total 624.12 last 19.16) - open/close 12700 (time 1790975003.37 total 635.60 last 18.89) - open/close 12883 (time 1790975015.54 total 647.77 last 15.04) - open/close 13115 (time 1790975026.40 total 658.63 last 21.36) - open/close 13392 (time 1790975037.42 total 669.65 last 25.15) - open/close 13518 (time 1790975047.67 total 679.90 last 12.29) - open/close 13675 (time 1790975058.05 total 690.28 last 15.13) - open/close 13904 (time 1790975068.59 total 700.82 last 21.73) - open/close 14140 (time 1790975080.43 total 712.66 last 19.93) - open/close 14340 (time 1790975090.44 total 722.67 last 19.97) - open/close 14492 (time 1790975100.46 total 732.69 last 15.18) - open/close 14738 (time 1790975110.69 total 742.92 last 24.04) - open/close 14908 (time 1790975120.93 total 753.16 last 16.61) - open/close 15035 (time 1790975131.52 total 763.75 last 11.98) - open/close 15262 (time 1790975142.37 total 774.59 last 20.94) - open/close 15496 (time 1790975152.39 total 784.62 last 23.34) - open/close 15739 (time 1790975162.41 total 794.64 last 24.25) - open/close 15933 (time 1790975172.97 total 805.20 last 18.38) - open/close 16153 (time 1790975183.97 total 816.19 last 20.00) - open/close 16300 (time 1790975195.30 total 827.53 last 12.96) - open/close 16507 (time 1790975205.42 total 837.65 last 20.46) - open/close 16750 (time 1790975215.43 total 847.66 last 24.29) - open/close 16871 (time 1790975226.02 total 858.25 last 11.42) - open/close 17051 (time 1790975236.64 total 868.87 last 16.95) - open/close 17172 (time 1790975246.65 total 878.88 last 12.09) - open/close 17245 (time 1790975258.66 total 890.89 last 6.08) - open/close 17379 (time 1790975268.85 total 901.07 last 13.16) - open/close 17500 (time 1790975279.22 total 911.45 last 11.66) - open/close 17671 (time 1790975289.28 total 921.51 last 17.00) - open/close 17770 (time 1790975299.65 total 931.88 last 9.54) - open/close 17951 (time 1790975310.14 total 942.37 last 17.25) - open/close 18029 (time 1790975320.24 total 952.47 last 7.72) - open/close 18096 (time 1790975330.80 total 963.03 last 6.35) - open/close 18339 (time 1790975343.18 total 975.41 last 19.63) - open/close 18925 (time 1790975353.19 total 985.42 last 58.50) - open/close 19658 (time 1790975363.20 total 995.43 last 73.24) - open/close 20000 (time 1790975367.59 total 999.82 last 77.90) - open/close 20615 (time 1790975377.59 total 1009.82 last 61.50) - open/close 21139 (time 1790975387.61 total 1019.84 last 52.30) - open/close 21759 (time 1790975397.62 total 1029.85 last 61.92) - open/close 22313 (time 1790975407.64 total 1039.87 last 55.32) - open/close 22982 (time 1790975417.64 total 1049.87 last 66.86) - open/close 23639 (time 1790975427.65 total 1059.88 last 65.65) - open/close 24375 (time 1790975437.66 total 1069.89 last 73.51) - open/close 25061 (time 1790975447.67 total 1079.90 last 68.53) - open/close 25736 (time 1790975457.68 total 1089.91 last 67.43) - open/close 26475 (time 1790975467.70 total 1099.93 last 73.80) - open/close 27197 (time 1790975477.70 total 1109.93 last 72.16) - open/close 27949 (time 1790975487.72 total 1119.95 last 75.07) - open/close 28689 (time 1790975497.73 total 1129.96 last 73.95) - open/close 29469 (time 1790975507.74 total 1139.97 last 77.93) - open/close 30000 (time 1790975515.19 total 1147.42 last 71.20) - open/close 30684 (time 1790975525.20 total 1157.43 last 68.34) - open/close 31288 (time 1790975535.21 total 1167.44 last 60.34) - open/close 31774 (time 1790975545.22 total 1177.45 last 48.55) - open/close 32305 (time 1790975555.23 total 1187.46 last 53.09) - open/close 32984 (time 1790975565.23 total 1197.46 last 67.89) - open/close 33695 (time 1790975575.23 total 1207.46 last 71.08) - open/close 34372 (time 1790975585.24 total 1217.47 last 67.67) - open/close 35027 (time 1790975595.28 total 1227.51 last 65.24) - open/close 35750 (time 1790975605.28 total 1237.51 last 72.26) - open/close 36456 (time 1790975615.29 total 1247.52 last 70.54) - open/close 37204 (time 1790975625.30 total 1257.53 last 74.71) - open/close 37901 (time 1790975635.31 total 1267.54 last 69.66) - open/close 38580 (time 1790975645.32 total 1277.55 last 67.79) - open/close 39335 (time 1790975655.33 total 1287.56 last 75.43) - open/close 40000 (time 1790975665.24 total 1297.47 last 67.13) - open/close 40684 (time 1790975675.25 total 1307.48 last 68.36) - open/close 41366 (time 1790975685.26 total 1317.49 last 68.13) - open/close 42034 (time 1790975695.27 total 1327.49 last 66.75) - open/close 42729 (time 1790975705.27 total 1337.50 last 69.49) - open/close 43420 (time 1790975715.27 total 1347.50 last 69.06) - open/close 44147 (time 1790975725.27 total 1357.50 last 72.68) - open/close 44830 (time 1790975735.28 total 1367.51 last 68.27) - open/close 45551 (time 1790975745.28 total 1377.51 last 72.08) - open/close 46245 (time 1790975755.28 total 1387.51 last 69.37) - open/close 46961 (time 1790975765.30 total 1397.53 last 71.51) - open/close 47560 (time 1790975775.31 total 1407.54 last 59.84) - open/close 48245 (time 1790975785.31 total 1417.54 last 68.45) - open/close 48941 (time 1790975795.33 total 1427.56 last 69.50) - open/close 49678 (time 1790975805.35 total 1437.58 last 73.54) total: 50000 open/close in 1443.85 seconds: 34.63 ops/second - unlinked 0 (time 1790975817 ; total 0 ; last 0) - unlinked 10000 (time 1790975965 ; total 148 ; last 148) - unlinked 20000 (time 1790976108 ; total 291 ; last 143) - unlinked 30000 (time 1790976255 ; total 438 ; last 147) - unlinked 40000 (time 1790976398 ; total 581 ; last 143) total: 50000 unlinks in 734 seconds: 68.119888 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local oleg430-server: kmemleak disabled modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2