== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:57:27 (1788440247) start mds service on oleg407-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: kmemleak disabled oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg407-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1788440339.46 total 10.00 last 13.39) - open/close 267 (time 1788440349.47 total 20.01 last 13.29) - open/close 447 (time 1788440359.72 total 30.26 last 17.57) - open/close 600 (time 1788440369.74 total 40.27 last 15.27) - open/close 698 (time 1788440380.41 total 50.95 last 9.18) - open/close 817 (time 1788440390.78 total 61.32 last 11.47) - open/close 972 (time 1788440400.79 total 71.33 last 15.48) - open/close 1152 (time 1788440411.04 total 81.58 last 17.57) - open/close 1208 (time 1788440422.05 total 92.59 last 5.08) - open/close 1266 (time 1788440432.06 total 102.60 last 5.79) - open/close 1475 (time 1788440443.03 total 113.57 last 19.05) - open/close 1697 (time 1788440454.06 total 124.60 last 20.14) - open/close 1923 (time 1788440466.88 total 137.42 last 17.63) - open/close 2084 (time 1788440476.89 total 147.43 last 16.08) - open/close 2197 (time 1788440487.64 total 158.18 last 10.51) - open/close 2377 (time 1788440497.64 total 168.18 last 18.00) - open/close 2601 (time 1788440507.66 total 178.20 last 22.37) - open/close 2764 (time 1788440517.68 total 188.22 last 16.27) - open/close 3010 (time 1788440527.72 total 198.26 last 24.51) - open/close 3168 (time 1788440537.73 total 208.27 last 15.77) - open/close 3337 (time 1788440547.74 total 218.27 last 16.90) - open/close 3515 (time 1788440558.18 total 228.72 last 17.04) - open/close 3551 (time 1788440568.38 total 238.92 last 3.53) - open/close 3720 (time 1788440579.12 total 249.66 last 15.73) - open/close 3789 (time 1788440590.12 total 260.66 last 6.27) - open/close 3978 (time 1788440600.13 total 270.67 last 18.88) - open/close 4208 (time 1788440610.14 total 280.68 last 22.97) - open/close 4329 (time 1788440621.15 total 291.69 last 10.99) - open/close 4451 (time 1788440636.69 total 307.23 last 7.85) - open/close 4550 (time 1788440646.72 total 317.26 last 9.87) - open/close 4743 (time 1788440657.30 total 327.84 last 18.24) - open/close 4782 (time 1788440667.35 total 337.89 last 3.88) - open/close 4942 (time 1788440677.37 total 347.91 last 15.96) - open/close 5108 (time 1788440687.40 total 357.94 last 16.56) - open/close 5293 (time 1788440697.64 total 368.18 last 18.06) - open/close 5547 (time 1788440708.01 total 378.55 last 24.50) - open/close 5733 (time 1788440718.55 total 389.09 last 17.65) - open/close 5858 (time 1788440729.38 total 399.92 last 11.54) - open/close 5999 (time 1788440739.41 total 409.95 last 14.06) - open/close 6176 (time 1788440752.37 total 422.90 last 13.66) - open/close 6293 (time 1788440763.42 total 433.96 last 10.58) - open/close 6400 (time 1788440774.11 total 444.65 last 10.01) - open/close 6502 (time 1788440787.03 total 457.57 last 7.89) - open/close 6694 (time 1788440797.04 total 467.58 last 19.19) - open/close 6826 (time 1788440807.08 total 477.62 last 13.15) - open/close 6975 (time 1788440817.59 total 488.13 last 14.18) - open/close 7053 (time 1788440827.99 total 498.53 last 7.50) - open/close 7236 (time 1788440838.05 total 508.59 last 18.21) - open/close 7317 (time 1788440848.73 total 519.27 last 7.58) - open/close 7543 (time 1788440859.80 total 530.34 last 20.42) - open/close 7766 (time 1788440869.80 total 540.34 last 22.30) - open/close 7936 (time 1788440880.66 total 551.20 last 15.65) - open/close 8060 (time 1788440892.57 total 563.11 last 10.42) - open/close 8265 (time 1788440902.85 total 573.39 last 19.93) - open/close 8452 (time 1788440912.86 total 583.40 last 18.69) - open/close 8748 (time 1788440923.68 total 594.22 last 27.35) - open/close 8888 (time 1788440933.68 total 604.22 last 14.00) - open/close 9030 (time 1788440943.85 total 614.39 last 13.97) - open/close 9186 (time 1788440954.47 total 625.01 last 14.68) - open/close 9373 (time 1788440964.49 total 635.03 last 18.66) - open/close 9518 (time 1788440975.39 total 645.93 last 13.31) - open/close 9673 (time 1788440985.72 total 656.26 last 15.01) - open/close 9760 (time 1788440996.50 total 667.04 last 8.07) - open/close 9888 (time 1788441006.87 total 677.41 last 12.34) - open/close 10000 (time 1788441012.91 total 683.45 last 18.56) - open/close 10151 (time 1788441022.91 total 693.45 last 15.09) - open/close 10314 (time 1788441032.93 total 703.47 last 16.27) - open/close 10400 (time 1788441043.70 total 714.24 last 7.99) - open/close 10525 (time 1788441054.58 total 725.12 last 11.49) - open/close 10719 (time 1788441065.25 total 735.79 last 18.18) - open/close 10982 (time 1788441076.07 total 746.61 last 24.32) - open/close 11098 (time 1788441086.08 total 756.62 last 11.58) - open/close 11296 (time 1788441097.16 total 767.70 last 17.86) - open/close 11481 (time 1788441108.19 total 778.73 last 16.77) - open/close 11600 (time 1788441119.14 total 789.68 last 10.87) - open/close 11809 (time 1788441129.14 total 799.68 last 20.89) - open/close 11983 (time 1788441139.29 total 809.83 last 17.15) - open/close 12097 (time 1788441149.31 total 819.85 last 11.38) - open/close 12289 (time 1788441159.60 total 830.14 last 18.64) - open/close 12416 (time 1788441169.61 total 840.15 last 12.69) - open/close 12638 (time 1788441179.61 total 850.15 last 22.19) - open/close 12886 (time 1788441189.63 total 860.17 last 24.76) - open/close 13064 (time 1788441199.63 total 870.17 last 17.79) - open/close 13261 (time 1788441210.56 total 881.10 last 18.03) - open/close 13310 (time 1788441221.39 total 891.93 last 4.52) - open/close 13417 (time 1788441231.57 total 902.11 last 10.51) - open/close 13462 (time 1788441242.21 total 912.75 last 4.23) - open/close 13560 (time 1788441253.22 total 923.76 last 8.90) - open/close 13660 (time 1788441263.24 total 933.78 last 9.98) - open/close 13791 (time 1788441273.24 total 943.78 last 13.10) - open/close 13918 (time 1788441283.25 total 953.79 last 12.69) - open/close 13975 (time 1788441293.40 total 963.94 last 5.62) - open/close 14082 (time 1788441303.41 total 973.95 last 10.69) - open/close 14240 (time 1788441314.03 total 984.57 last 14.88) - open/close 14358 (time 1788441325.09 total 995.63 last 10.68) - open/close 14447 (time 1788441335.39 total 1005.93 last 8.64) - open/close 14510 (time 1788441346.42 total 1016.96 last 5.72) - open/close 14675 (time 1788441356.43 total 1026.97 last 16.48) - open/close 14799 (time 1788441366.60 total 1037.14 last 12.18) - open/close 14922 (time 1788441376.97 total 1047.51 last 11.86) - open/close 15042 (time 1788441387.84 total 1058.38 last 11.05) - open/close 15244 (time 1788441397.84 total 1068.38 last 20.20) - open/close 15363 (time 1788441408.80 total 1079.34 last 10.85) - open/close 15431 (time 1788441419.52 total 1090.06 last 6.35) - open/close 15558 (time 1788441429.81 total 1100.35 last 12.34) - open/close 16211 (time 1788441439.82 total 1110.36 last 65.24) - open/close 16923 (time 1788441449.84 total 1120.38 last 71.09) - open/close 17529 (time 1788441459.84 total 1130.38 last 60.58) - open/close 18188 (time 1788441469.85 total 1140.39 last 65.86) - open/close 18816 (time 1788441479.86 total 1150.40 last 62.73) - open/close 19460 (time 1788441489.86 total 1160.40 last 64.37) - open/close 20000 (time 1788441498.36 total 1168.90 last 63.54) - open/close 20688 (time 1788441508.39 total 1178.93 last 68.61) - open/close 21370 (time 1788441518.39 total 1188.93 last 68.19) - open/close 22034 (time 1788441528.41 total 1198.95 last 66.27) - open/close 22662 (time 1788441538.43 total 1208.97 last 62.65) - open/close 23291 (time 1788441548.46 total 1219.00 last 62.73) - open/close 23896 (time 1788441558.46 total 1229.00 last 60.49) - open/close 24572 (time 1788441568.47 total 1239.01 last 67.57) - open/close 25172 (time 1788441578.48 total 1249.02 last 59.90) - open/close 25733 (time 1788441588.49 total 1259.02 last 56.08) - open/close 26350 (time 1788441598.51 total 1269.05 last 61.52) - open/close 26937 (time 1788441608.52 total 1279.06 last 58.65) - open/close 27559 (time 1788441618.53 total 1289.07 last 62.15) - open/close 28187 (time 1788441628.54 total 1299.08 last 62.76) - open/close 28809 (time 1788441638.57 total 1309.11 last 61.99) - open/close 29436 (time 1788441648.57 total 1319.11 last 62.68) - open/close 30000 (time 1788441657.39 total 1327.93 last 64.00) - open/close 30567 (time 1788441667.41 total 1337.95 last 56.56) - open/close 31202 (time 1788441677.42 total 1347.96 last 63.44) - open/close 31901 (time 1788441687.42 total 1357.96 last 69.89) - open/close 32466 (time 1788441697.43 total 1367.97 last 56.43) - open/close 33096 (time 1788441707.45 total 1377.99 last 62.89) - open/close 33658 (time 1788441717.46 total 1388.00 last 56.15) - open/close 34211 (time 1788441727.47 total 1398.01 last 55.26) - open/close 34832 (time 1788441737.48 total 1408.02 last 62.01) - open/close 35461 (time 1788441747.50 total 1418.04 last 62.81) - open/close 36114 (time 1788441757.50 total 1428.04 last 65.28) - open/close 36614 (time 1788441767.51 total 1438.05 last 49.94) - open/close 37148 (time 1788441777.52 total 1448.06 last 53.37) - open/close 37711 (time 1788441787.53 total 1458.07 last 56.23) - open/close 38340 (time 1788441797.56 total 1468.10 last 62.67) - open/close 38978 (time 1788441807.56 total 1478.10 last 63.80) - open/close 39650 (time 1788441817.57 total 1488.11 last 67.17) - open/close 40000 (time 1788441823.07 total 1493.61 last 63.67) - open/close 40628 (time 1788441833.07 total 1503.61 last 62.74) - open/close 41231 (time 1788441843.09 total 1513.63 last 60.20) - open/close 41830 (time 1788441853.09 total 1523.63 last 59.89) - open/close 42464 (time 1788441863.11 total 1533.65 last 63.28) - open/close 43116 (time 1788441873.12 total 1543.66 last 65.14) - open/close 43751 (time 1788441883.15 total 1553.68 last 63.36) - open/close 44389 (time 1788441893.16 total 1563.70 last 63.68) - open/close 44981 (time 1788441903.17 total 1573.71 last 59.14) - open/close 45579 (time 1788441913.20 total 1583.74 last 59.67) - open/close 46202 (time 1788441923.20 total 1593.74 last 62.26) - open/close 46807 (time 1788441933.23 total 1603.77 last 60.35) - open/close 47413 (time 1788441943.23 total 1613.77 last 60.57) - open/close 47897 (time 1788441953.24 total 1623.78 last 48.38) - open/close 48405 (time 1788441963.26 total 1633.80 last 50.69) - open/close 48939 (time 1788441973.27 total 1643.81 last 53.34) - open/close 49457 (time 1788441983.29 total 1653.83 last 51.71) total: 50000 open/close in 1663.41 seconds: 30.06 ops/second - unlinked 0 (time 1788441997 ; total 0 ; last 0) - unlinked 10000 (time 1788442167 ; total 170 ; last 170) - unlinked 20000 (time 1788442335 ; total 338 ; last 168) - unlinked 30000 (time 1788442511 ; total 514 ; last 176) - unlinked 40000 (time 1788442674 ; total 677 ; last 163) total: 50000 unlinks in 846 seconds: 59.101654 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: kmemleak disabled modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2