== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:39:07 (1788442747) start mds service on oleg208-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: kmemleak disabled oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg208-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg208-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 185 (time 1788442839.46 total 10.01 last 18.49) - open/close 304 (time 1788442849.47 total 20.01 last 11.89) - open/close 514 (time 1788442860.40 total 30.94 last 19.21) - open/close 648 (time 1788442870.41 total 40.96 last 13.38) - open/close 807 (time 1788442883.40 total 53.95 last 12.24) - open/close 1053 (time 1788442893.42 total 63.96 last 24.56) - open/close 1266 (time 1788442903.93 total 74.48 last 20.26) - open/close 1493 (time 1788442913.93 total 84.48 last 22.69) - open/close 1680 (time 1788442924.05 total 94.59 last 18.49) - open/close 1851 (time 1788442934.33 total 104.88 last 16.62) - open/close 2141 (time 1788442945.25 total 115.79 last 26.58) - open/close 2328 (time 1788442955.25 total 125.80 last 18.70) - open/close 2558 (time 1788442965.70 total 136.25 last 22.00) - open/close 2769 (time 1788442975.71 total 146.26 last 21.08) - open/close 2970 (time 1788442986.44 total 156.98 last 18.74) - open/close 3113 (time 1788442996.57 total 167.12 last 14.11) - open/close 3354 (time 1788443006.59 total 177.14 last 24.05) - open/close 3522 (time 1788443016.62 total 187.17 last 16.75) - open/close 3727 (time 1788443027.28 total 197.83 last 19.24) - open/close 3840 (time 1788443037.30 total 207.84 last 11.28) - open/close 3905 (time 1788443047.32 total 217.86 last 6.49) - open/close 4097 (time 1788443057.87 total 228.42 last 18.19) - open/close 4232 (time 1788443068.90 total 239.45 last 12.24) - open/close 4405 (time 1788443078.91 total 249.45 last 17.29) - open/close 4504 (time 1788443088.94 total 259.49 last 9.86) - open/close 4666 (time 1788443099.59 total 270.14 last 15.22) - open/close 4879 (time 1788443110.29 total 280.84 last 19.91) - open/close 5108 (time 1788443120.31 total 290.86 last 22.85) - open/close 5232 (time 1788443130.56 total 301.11 last 12.10) - open/close 5371 (time 1788443142.65 total 313.19 last 11.50) - open/close 5557 (time 1788443152.65 total 323.19 last 18.60) - open/close 5757 (time 1788443162.65 total 333.20 last 19.99) - open/close 5856 (time 1788443175.61 total 346.16 last 7.64) - open/close 5976 (time 1788443186.00 total 356.54 last 11.55) - open/close 6126 (time 1788443196.04 total 366.59 last 14.93) - open/close 6235 (time 1788443206.35 total 376.89 last 10.58) - open/close 6345 (time 1788443216.35 total 386.90 last 10.99) - open/close 6512 (time 1788443227.15 total 397.70 last 15.47) - open/close 6670 (time 1788443237.71 total 408.26 last 14.95) - open/close 6912 (time 1788443247.98 total 418.53 last 23.57) - open/close 7031 (time 1788443259.08 total 429.63 last 10.72) - open/close 7230 (time 1788443270.07 total 440.62 last 18.10) - open/close 7476 (time 1788443280.09 total 450.64 last 24.56) - open/close 7703 (time 1788443290.36 total 460.91 last 22.10) - open/close 7931 (time 1788443300.48 total 471.02 last 22.54) - open/close 8148 (time 1788443310.50 total 481.05 last 21.65) - open/close 8229 (time 1788443321.04 total 491.58 last 7.69) - open/close 8415 (time 1788443331.26 total 501.80 last 18.20) - open/close 8619 (time 1788443341.27 total 511.82 last 20.37) - open/close 8800 (time 1788443352.13 total 522.68 last 16.66) - open/close 8958 (time 1788443362.49 total 533.03 last 15.26) - open/close 9125 (time 1788443373.37 total 543.92 last 15.34) - open/close 9305 (time 1788443384.26 total 554.81 last 16.53) - open/close 9425 (time 1788443394.28 total 564.83 last 11.98) - open/close 9542 (time 1788443405.24 total 575.78 last 10.68) - open/close 9695 (time 1788443416.08 total 586.62 last 14.12) - open/close 9769 (time 1788443426.24 total 596.79 last 7.28) - open/close 9992 (time 1788443436.74 total 607.28 last 21.25) - open/close 10000 (time 1788443440.09 total 610.64 last 2.39) - open/close 10078 (time 1788443450.16 total 620.71 last 7.74) - open/close 10222 (time 1788443463.16 total 633.71 last 11.08) - open/close 10395 (time 1788443473.18 total 643.73 last 17.26) - open/close 10567 (time 1788443483.21 total 653.76 last 17.15) - open/close 10746 (time 1788443493.44 total 663.98 last 17.50) - open/close 10960 (time 1788443503.45 total 674.00 last 21.37) - open/close 11102 (time 1788443513.45 total 684.00 last 14.20) - open/close 11322 (time 1788443523.46 total 694.00 last 21.99) - open/close 11540 (time 1788443533.46 total 704.01 last 21.79) - open/close 11734 (time 1788443544.00 total 714.54 last 18.41) - open/close 11955 (time 1788443554.00 total 724.55 last 22.08) - open/close 12154 (time 1788443564.17 total 734.71 last 19.58) - open/close 12314 (time 1788443574.17 total 744.72 last 16.00) - open/close 12399 (time 1788443586.69 total 757.24 last 6.79) - open/close 12591 (time 1788443596.70 total 767.25 last 19.18) - open/close 12782 (time 1788443606.98 total 777.53 last 18.59) - open/close 13054 (time 1788443617.27 total 787.81 last 26.44) - open/close 13240 (time 1788443627.45 total 798.00 last 18.26) - open/close 13402 (time 1788443638.01 total 808.56 last 15.34) - open/close 13623 (time 1788443648.19 total 818.74 last 21.71) - open/close 13745 (time 1788443658.20 total 828.74 last 12.19) - open/close 13978 (time 1788443668.20 total 838.75 last 23.29) - open/close 14168 (time 1788443678.21 total 848.75 last 18.99) - open/close 14406 (time 1788443688.22 total 858.77 last 23.77) - open/close 14629 (time 1788443698.62 total 869.17 last 21.45) - open/close 14889 (time 1788443708.62 total 879.17 last 25.99) - open/close 15072 (time 1788443719.02 total 889.57 last 17.60) - open/close 15133 (time 1788443729.34 total 899.89 last 5.91) - open/close 15260 (time 1788443739.84 total 910.38 last 12.10) - open/close 15407 (time 1788443749.84 total 920.39 last 14.69) - open/close 15548 (time 1788443760.60 total 931.15 last 13.10) - open/close 15647 (time 1788443771.39 total 941.94 last 9.17) - open/close 15892 (time 1788443781.40 total 951.95 last 24.48) - open/close 16069 (time 1788443791.40 total 961.95 last 17.69) - open/close 16258 (time 1788443801.94 total 972.49 last 17.93) - open/close 16495 (time 1788443812.93 total 983.47 last 21.58) - open/close 16749 (time 1788443822.94 total 993.49 last 25.37) - open/close 16852 (time 1788443833.75 total 1004.30 last 9.53) - open/close 16923 (time 1788443843.96 total 1014.50 last 6.96) - open/close 17064 (time 1788443854.53 total 1025.08 last 13.33) - open/close 17157 (time 1788443865.16 total 1035.71 last 8.75) - open/close 17315 (time 1788443876.31 total 1046.85 last 14.17) - open/close 18098 (time 1788443886.32 total 1056.87 last 78.21) - open/close 18822 (time 1788443896.32 total 1066.87 last 72.39) - open/close 19543 (time 1788443906.35 total 1076.89 last 71.92) - open/close 20000 (time 1788443913.41 total 1083.96 last 64.66) - open/close 20599 (time 1788443923.42 total 1093.97 last 59.85) - open/close 21121 (time 1788443933.43 total 1103.98 last 52.13) - open/close 21705 (time 1788443943.45 total 1113.99 last 58.32) - open/close 22450 (time 1788443953.45 total 1124.00 last 74.48) - open/close 23143 (time 1788443963.45 total 1134.00 last 69.28) - open/close 23935 (time 1788443973.48 total 1144.03 last 79.00) - open/close 24589 (time 1788443983.48 total 1154.03 last 65.40) - open/close 25316 (time 1788443993.49 total 1164.04 last 72.60) - open/close 25948 (time 1788444003.50 total 1174.04 last 63.18) - open/close 26563 (time 1788444013.50 total 1184.05 last 61.45) - open/close 27137 (time 1788444023.53 total 1194.07 last 57.26) - open/close 27785 (time 1788444033.53 total 1204.08 last 64.76) - open/close 28383 (time 1788444043.54 total 1214.09 last 59.75) - open/close 28934 (time 1788444053.55 total 1224.09 last 55.07) - open/close 29489 (time 1788444063.56 total 1234.10 last 55.44) - open/close 30000 (time 1788444071.30 total 1241.85 last 65.97) - open/close 30565 (time 1788444081.33 total 1251.87 last 56.37) - open/close 31111 (time 1788444091.37 total 1261.91 last 54.39) - open/close 31692 (time 1788444101.37 total 1271.92 last 58.06) - open/close 32294 (time 1788444111.38 total 1281.92 last 60.17) - open/close 32935 (time 1788444121.38 total 1291.93 last 64.08) - open/close 33596 (time 1788444131.39 total 1301.94 last 66.04) - open/close 34304 (time 1788444141.40 total 1311.94 last 70.75) - open/close 34997 (time 1788444151.40 total 1321.95 last 69.26) - open/close 35685 (time 1788444161.41 total 1331.96 last 68.73) - open/close 36313 (time 1788444171.41 total 1341.96 last 62.79) - open/close 36992 (time 1788444181.42 total 1351.97 last 67.86) - open/close 37614 (time 1788444191.43 total 1361.98 last 62.15) - open/close 38194 (time 1788444201.44 total 1371.99 last 57.94) - open/close 38898 (time 1788444211.46 total 1382.00 last 70.28) - open/close 39486 (time 1788444221.47 total 1392.01 last 58.75) - open/close 40000 (time 1788444228.11 total 1398.66 last 77.37) - open/close 40713 (time 1788444238.12 total 1408.67 last 71.22) - open/close 41438 (time 1788444248.12 total 1418.67 last 72.49) - open/close 42130 (time 1788444258.13 total 1428.68 last 69.15) - open/close 42801 (time 1788444268.14 total 1438.69 last 67.03) - open/close 43388 (time 1788444278.14 total 1448.69 last 58.68) - open/close 44036 (time 1788444288.15 total 1458.70 last 64.73) - open/close 44583 (time 1788444298.17 total 1468.72 last 54.60) - open/close 45234 (time 1788444308.20 total 1478.75 last 64.94) - open/close 45908 (time 1788444318.20 total 1488.75 last 67.38) - open/close 46616 (time 1788444328.21 total 1498.76 last 70.74) - open/close 47176 (time 1788444338.23 total 1508.78 last 55.88) - open/close 47906 (time 1788444348.24 total 1518.79 last 72.93) - open/close 48534 (time 1788444358.25 total 1528.79 last 62.76) - open/close 49201 (time 1788444368.25 total 1538.80 last 66.66) - open/close 49676 (time 1788444378.26 total 1548.81 last 47.47) total: 50000 open/close in 1554.76 seconds: 32.16 ops/second - unlinked 0 (time 1788444390 ; total 1 ; last 1) - unlinked 10000 (time 1788444546 ; total 157 ; last 156) - unlinked 20000 (time 1788444703 ; total 314 ; last 157) - unlinked 30000 (time 1788444862 ; total 473 ; last 159) - unlinked 40000 (time 1788445035 ; total 646 ; last 173) total: 50000 unlinks in 794 seconds: 62.972294 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: kmemleak disabled modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2