== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:21:36 (1789780896) start mds service on oleg313-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: kmemleak disabled oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg313-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg313-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg313-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1789780980.46 total 10.51 last 20.35) - open/close 450 (time 1789780990.46 total 20.52 last 23.58) - open/close 627 (time 1789781001.11 total 31.17 last 16.62) - open/close 895 (time 1789781011.52 total 41.58 last 25.75) - open/close 1134 (time 1789781021.54 total 51.60 last 23.86) - open/close 1364 (time 1789781032.23 total 62.29 last 21.51) - open/close 1583 (time 1789781042.25 total 72.31 last 21.84) - open/close 1835 (time 1789781055.17 total 85.23 last 19.51) - open/close 1963 (time 1789781067.04 total 97.10 last 10.78) - open/close 2208 (time 1789781077.06 total 107.12 last 24.45) - open/close 2454 (time 1789781087.64 total 117.69 last 23.26) - open/close 2715 (time 1789781098.40 total 128.46 last 24.24) - open/close 2826 (time 1789781108.64 total 138.69 last 10.85) - open/close 2983 (time 1789781119.34 total 149.40 last 14.66) - open/close 3050 (time 1789781131.67 total 161.73 last 5.43) - open/close 3291 (time 1789781147.46 total 177.52 last 15.27) - open/close 3389 (time 1789781157.87 total 187.93 last 9.41) - open/close 3484 (time 1789781168.77 total 198.83 last 8.72) - open/close 3622 (time 1789781178.79 total 208.85 last 13.77) - open/close 3817 (time 1789781189.74 total 219.79 last 17.81) - open/close 4033 (time 1789781200.04 total 230.10 last 20.96) - open/close 4301 (time 1789781210.05 total 240.11 last 26.78) - open/close 4552 (time 1789781220.07 total 250.13 last 25.04) - open/close 4750 (time 1789781231.04 total 261.10 last 18.06) - open/close 4911 (time 1789781241.05 total 271.11 last 16.08) - open/close 5062 (time 1789781251.06 total 281.12 last 15.08) - open/close 5280 (time 1789781261.65 total 291.71 last 20.58) - open/close 5522 (time 1789781271.65 total 301.71 last 24.20) - open/close 5726 (time 1789781281.96 total 312.02 last 19.79) - open/close 5957 (time 1789781291.97 total 322.03 last 23.07) - open/close 6069 (time 1789781302.33 total 332.39 last 10.81) - open/close 6177 (time 1789781313.07 total 343.12 last 10.06) - open/close 6424 (time 1789781324.09 total 354.15 last 22.41) - open/close 6661 (time 1789781334.10 total 364.15 last 23.68) - open/close 6836 (time 1789781344.10 total 374.16 last 17.49) - open/close 7107 (time 1789781354.13 total 384.19 last 27.02) - open/close 7325 (time 1789781365.52 total 395.58 last 19.14) - open/close 7479 (time 1789781375.97 total 406.03 last 14.73) - open/close 7649 (time 1789781386.66 total 416.72 last 15.90) - open/close 7830 (time 1789781398.82 total 428.88 last 14.89) - open/close 8005 (time 1789781408.86 total 438.92 last 17.42) - open/close 8227 (time 1789781418.87 total 448.93 last 22.18) - open/close 8426 (time 1789781428.90 total 458.96 last 19.84) - open/close 8609 (time 1789781438.90 total 468.96 last 18.30) - open/close 8833 (time 1789781449.65 total 479.71 last 20.83) - open/close 8949 (time 1789781459.66 total 489.71 last 11.60) - open/close 9076 (time 1789781470.58 total 500.64 last 11.62) - open/close 9287 (time 1789781481.17 total 511.23 last 19.93) - open/close 9489 (time 1789781491.17 total 521.23 last 20.20) - open/close 9581 (time 1789781501.41 total 531.47 last 8.99) - open/close 9734 (time 1789781513.32 total 543.37 last 12.85) - open/close 9861 (time 1789781526.04 total 556.10 last 9.98) - open/close 9953 (time 1789781537.66 total 567.72 last 7.92) - open/close 10086 (time 1789781547.92 total 577.98 last 12.96) - open/close 10171 (time 1789781558.51 total 588.57 last 8.03) - open/close 10301 (time 1789781569.05 total 599.11 last 12.33) - open/close 10427 (time 1789781579.87 total 609.93 last 11.65) - open/close 10575 (time 1789781589.87 total 619.93 last 14.80) - open/close 10745 (time 1789781599.91 total 629.97 last 16.93) - open/close 10948 (time 1789781610.48 total 640.54 last 19.20) - open/close 11185 (time 1789781620.48 total 650.54 last 23.69) - open/close 11405 (time 1789781630.83 total 660.89 last 21.26) - open/close 11611 (time 1789781641.19 total 671.25 last 19.90) - open/close 11858 (time 1789781651.62 total 681.68 last 23.67) - open/close 12084 (time 1789781661.65 total 691.71 last 22.54) - open/close 12206 (time 1789781673.91 total 703.97 last 9.95) - open/close 12260 (time 1789781684.79 total 714.85 last 4.96) - open/close 12512 (time 1789781695.84 total 725.90 last 22.81) - open/close 12743 (time 1789781706.29 total 736.35 last 22.10) - open/close 12909 (time 1789781717.04 total 747.09 last 15.45) - open/close 13114 (time 1789781727.04 total 757.10 last 20.50) - open/close 13353 (time 1789781738.00 total 768.05 last 21.81) - open/close 13512 (time 1789781748.90 total 778.96 last 14.58) - open/close 13766 (time 1789781759.72 total 789.78 last 23.49) - open/close 13979 (time 1789781770.16 total 800.22 last 20.40) - open/close 14207 (time 1789781780.34 total 810.40 last 22.40) - open/close 14389 (time 1789781790.34 total 820.40 last 18.20) - open/close 14620 (time 1789781801.18 total 831.24 last 21.30) - open/close 14752 (time 1789781811.21 total 841.27 last 13.16) - open/close 14896 (time 1789781821.48 total 851.54 last 14.03) - open/close 15134 (time 1789781832.28 total 862.34 last 22.04) - open/close 15381 (time 1789781842.28 total 872.34 last 24.69) - open/close 15631 (time 1789781852.29 total 882.35 last 24.98) - open/close 15830 (time 1789781863.27 total 893.33 last 18.13) - open/close 16086 (time 1789781874.00 total 904.06 last 23.84) - open/close 16328 (time 1789781884.64 total 914.70 last 22.74) - open/close 16568 (time 1789781895.71 total 925.77 last 21.68) - open/close 16792 (time 1789781905.95 total 936.01 last 21.87) - open/close 16889 (time 1789781916.08 total 946.14 last 9.58) - open/close 17013 (time 1789781926.09 total 956.15 last 12.39) - open/close 17176 (time 1789781938.67 total 968.73 last 12.96) - open/close 17349 (time 1789781948.70 total 978.76 last 17.25) - open/close 17563 (time 1789781958.71 total 988.77 last 21.38) - open/close 17753 (time 1789781968.72 total 998.78 last 18.99) - open/close 18343 (time 1789781978.72 total 1008.78 last 58.99) - open/close 19003 (time 1789781988.73 total 1018.79 last 65.93) - open/close 19676 (time 1789781998.74 total 1028.80 last 67.25) - open/close 20000 (time 1789782004.43 total 1034.49 last 56.91) - open/close 20695 (time 1789782014.43 total 1044.49 last 69.48) - open/close 21351 (time 1789782024.44 total 1054.50 last 65.55) - open/close 21908 (time 1789782034.45 total 1064.51 last 55.65) - open/close 22587 (time 1789782044.45 total 1074.51 last 67.86) - open/close 23211 (time 1789782054.45 total 1084.51 last 62.40) - open/close 23760 (time 1789782064.46 total 1094.52 last 54.89) - open/close 24303 (time 1789782074.47 total 1104.53 last 54.24) - open/close 24856 (time 1789782084.48 total 1114.54 last 55.24) - open/close 25483 (time 1789782094.50 total 1124.56 last 62.58) - open/close 26088 (time 1789782104.50 total 1134.56 last 60.48) - open/close 26757 (time 1789782114.52 total 1144.57 last 66.81) - open/close 27393 (time 1789782124.52 total 1154.58 last 63.55) - open/close 28001 (time 1789782134.53 total 1164.59 last 60.75) - open/close 28642 (time 1789782144.54 total 1174.60 last 64.03) - open/close 29290 (time 1789782154.55 total 1184.61 last 64.72) - open/close 29927 (time 1789782164.56 total 1194.62 last 63.68) - open/close 30536 (time 1789782174.56 total 1204.62 last 60.89) - open/close 31182 (time 1789782184.56 total 1214.62 last 64.55) - open/close 31731 (time 1789782194.58 total 1224.64 last 54.80) - open/close 32307 (time 1789782204.61 total 1234.67 last 57.44) - open/close 32786 (time 1789782214.62 total 1244.68 last 47.85) - open/close 33423 (time 1789782224.64 total 1254.69 last 63.61) - open/close 34059 (time 1789782234.64 total 1264.70 last 63.55) - open/close 34685 (time 1789782244.64 total 1274.70 last 62.60) - open/close 35253 (time 1789782254.66 total 1284.72 last 56.73) - open/close 35847 (time 1789782264.67 total 1294.73 last 59.33) - open/close 36448 (time 1789782274.68 total 1304.74 last 60.00) - open/close 37152 (time 1789782284.70 total 1314.76 last 70.27) - open/close 37816 (time 1789782294.70 total 1324.76 last 66.40) - open/close 38400 (time 1789782304.72 total 1334.78 last 58.29) - open/close 38965 (time 1789782314.73 total 1344.79 last 56.45) - open/close 39568 (time 1789782324.75 total 1354.81 last 60.18) - open/close 40000 (time 1789782332.70 total 1362.76 last 54.37) - open/close 40552 (time 1789782342.72 total 1372.78 last 55.06) - open/close 41165 (time 1789782352.73 total 1382.79 last 61.25) - open/close 41764 (time 1789782362.74 total 1392.80 last 59.87) - open/close 42349 (time 1789782372.76 total 1402.82 last 58.37) - open/close 43004 (time 1789782382.77 total 1412.83 last 65.45) - open/close 43659 (time 1789782392.77 total 1422.83 last 65.45) - open/close 44230 (time 1789782402.78 total 1432.84 last 57.04) - open/close 44803 (time 1789782412.79 total 1442.85 last 57.27) - open/close 45324 (time 1789782422.80 total 1452.86 last 52.05) - open/close 45952 (time 1789782432.80 total 1462.86 last 62.77) - open/close 46465 (time 1789782442.84 total 1472.90 last 51.12) - open/close 46943 (time 1789782452.87 total 1482.93 last 47.63) - open/close 47460 (time 1789782462.89 total 1492.95 last 51.62) - open/close 48059 (time 1789782472.90 total 1502.96 last 59.85) - open/close 48624 (time 1789782482.90 total 1512.96 last 56.46) - open/close 49119 (time 1789782492.91 total 1522.96 last 49.49) - open/close 49705 (time 1789782502.92 total 1532.97 last 58.54) total: 50000 open/close in 1538.14 seconds: 32.51 ops/second - unlinked 0 (time 1789782515 ; total 0 ; last 0) - unlinked 10000 (time 1789782688 ; total 173 ; last 173) - unlinked 20000 (time 1789782863 ; total 348 ; last 175) - unlinked 30000 (time 1789783026 ; total 511 ; last 163) - unlinked 40000 (time 1789783192 ; total 677 ; last 166) total: 50000 unlinks in 834 seconds: 59.952038 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules via unload_modules_local on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local oleg313-server: kmemleak disabled modules unloaded. pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2