== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:53:40 (1789440820) start mds service on oleg234-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 12:06 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: kmemleak disabled oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg234-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 253 (time 1789440905.36 total 10.01 last 25.28) - open/close 514 (time 1789440915.37 total 20.01 last 26.08) - open/close 752 (time 1789440925.88 total 30.53 last 22.63) - open/close 953 (time 1789440936.23 total 40.88 last 19.43) - open/close 1163 (time 1789440946.85 total 51.50 last 19.77) - open/close 1302 (time 1789440957.77 total 62.42 last 12.73) - open/close 1516 (time 1789440968.32 total 72.97 last 20.28) - open/close 1720 (time 1789440979.14 total 83.79 last 18.85) - open/close 1958 (time 1789440990.02 total 94.67 last 21.88) - open/close 2183 (time 1789441000.03 total 104.67 last 22.48) - open/close 2510 (time 1789441010.26 total 114.91 last 31.94) - open/close 2773 (time 1789441020.57 total 125.22 last 25.51) - open/close 2959 (time 1789441030.59 total 135.24 last 18.56) - open/close 3211 (time 1789441041.35 total 146.00 last 23.42) - open/close 3443 (time 1789441051.80 total 156.44 last 22.21) - open/close 3677 (time 1789441062.43 total 167.08 last 22.00) - open/close 3797 (time 1789441072.43 total 177.08 last 12.00) - open/close 3896 (time 1789441082.76 total 187.41 last 9.59) - open/close 4136 (time 1789441092.78 total 197.43 last 23.95) - open/close 4281 (time 1789441102.79 total 207.44 last 14.49) - open/close 4494 (time 1789441112.82 total 217.46 last 21.25) - open/close 4637 (time 1789441123.26 total 227.91 last 13.70) - open/close 4811 (time 1789441133.57 total 238.22 last 16.87) - open/close 5023 (time 1789441143.59 total 248.23 last 21.17) - open/close 5215 (time 1789441154.64 total 259.28 last 17.37) - open/close 5463 (time 1789441164.65 total 269.30 last 24.77) - open/close 5675 (time 1789441174.92 total 279.56 last 20.65) - open/close 5828 (time 1789441188.00 total 292.65 last 11.69) - open/close 6041 (time 1789441198.01 total 302.66 last 21.28) - open/close 6310 (time 1789441208.02 total 312.67 last 26.86) - open/close 6556 (time 1789441218.60 total 323.25 last 23.26) - open/close 6787 (time 1789441228.61 total 333.26 last 23.07) - open/close 7044 (time 1789441238.61 total 343.26 last 25.69) - open/close 7228 (time 1789441249.33 total 353.98 last 17.18) - open/close 7465 (time 1789441259.44 total 364.09 last 23.44) - open/close 7628 (time 1789441270.28 total 374.93 last 15.03) - open/close 7816 (time 1789441281.00 total 385.65 last 17.54) - open/close 7991 (time 1789441291.66 total 396.31 last 16.42) - open/close 8200 (time 1789441301.66 total 406.31 last 20.90) - open/close 8426 (time 1789441312.29 total 416.94 last 21.26) - open/close 8616 (time 1789441322.92 total 427.57 last 17.87) - open/close 8891 (time 1789441332.94 total 437.59 last 27.45) - open/close 9009 (time 1789441343.49 total 448.14 last 11.19) - open/close 9166 (time 1789441354.05 total 458.70 last 14.86) - open/close 9292 (time 1789441364.07 total 468.71 last 12.58) - open/close 9416 (time 1789441374.17 total 478.82 last 12.27) - open/close 9522 (time 1789441384.17 total 488.82 last 10.60) - open/close 9645 (time 1789441394.19 total 498.83 last 12.28) - open/close 9810 (time 1789441405.72 total 510.37 last 14.31) - open/close 9928 (time 1789441415.73 total 520.37 last 11.79) - open/close 10000 (time 1789441418.37 total 523.01 last 27.27) - open/close 10178 (time 1789441428.46 total 533.11 last 17.63) - open/close 10272 (time 1789441438.86 total 543.51 last 9.04) - open/close 10501 (time 1789441449.57 total 554.22 last 21.39) - open/close 10615 (time 1789441460.27 total 564.92 last 10.65) - open/close 10790 (time 1789441470.60 total 575.25 last 16.94) - open/close 11020 (time 1789441481.31 total 585.96 last 21.49) - open/close 11252 (time 1789441491.32 total 595.97 last 23.17) - open/close 11440 (time 1789441501.55 total 606.20 last 18.38) - open/close 11597 (time 1789441513.80 total 618.45 last 12.81) - open/close 11707 (time 1789441523.84 total 628.49 last 10.95) - open/close 11861 (time 1789441533.85 total 638.50 last 15.38) - open/close 11959 (time 1789441543.93 total 648.58 last 9.73) - open/close 12168 (time 1789441554.89 total 659.54 last 19.07) - open/close 12463 (time 1789441564.89 total 669.54 last 29.48) - open/close 12622 (time 1789441575.97 total 680.62 last 14.36) - open/close 12881 (time 1789441586.00 total 690.65 last 25.82) - open/close 13049 (time 1789441596.01 total 700.66 last 16.79) - open/close 13193 (time 1789441606.01 total 710.66 last 14.40) - open/close 13423 (time 1789441617.04 total 721.68 last 20.86) - open/close 13656 (time 1789441627.07 total 731.71 last 23.23) - open/close 13913 (time 1789441637.11 total 741.75 last 25.60) - open/close 14144 (time 1789441648.46 total 753.11 last 20.34) - open/close 14344 (time 1789441659.34 total 763.99 last 18.39) - open/close 14495 (time 1789441670.10 total 774.74 last 14.04) - open/close 14741 (time 1789441680.12 total 784.77 last 24.53) - open/close 14914 (time 1789441690.15 total 794.80 last 17.25) - open/close 15089 (time 1789441700.44 total 805.08 last 17.02) - open/close 15253 (time 1789441710.45 total 815.10 last 16.37) - open/close 15448 (time 1789441723.52 total 828.17 last 14.92) - open/close 15664 (time 1789441733.54 total 838.18 last 21.57) - open/close 15922 (time 1789441744.47 total 849.12 last 23.60) - open/close 16042 (time 1789441754.48 total 859.13 last 11.99) - open/close 16165 (time 1789441764.48 total 869.13 last 12.30) - open/close 16340 (time 1789441775.05 total 879.70 last 16.56) - open/close 16478 (time 1789441785.82 total 890.47 last 12.81) - open/close 16593 (time 1789441795.82 total 900.47 last 11.50) - open/close 16729 (time 1789441806.37 total 911.02 last 12.89) - open/close 16972 (time 1789441816.38 total 921.03 last 24.28) - open/close 17148 (time 1789441827.30 total 931.95 last 16.12) - open/close 17370 (time 1789441837.59 total 942.23 last 21.59) - open/close 17609 (time 1789441848.60 total 953.25 last 21.70) - open/close 17784 (time 1789441858.61 total 963.26 last 17.49) - open/close 17923 (time 1789441868.77 total 973.41 last 13.68) - open/close 18115 (time 1789441878.79 total 983.43 last 19.16) - open/close 18357 (time 1789441889.86 total 994.51 last 21.85) - open/close 18577 (time 1789441899.87 total 1004.52 last 21.98) - open/close 19088 (time 1789441909.89 total 1014.53 last 51.04) - open/close 19902 (time 1789441919.91 total 1024.55 last 81.24) - open/close 20719 (time 1789441929.91 total 1034.55 last 81.69) - open/close 21537 (time 1789441939.91 total 1044.56 last 81.73) - open/close 22441 (time 1789441949.92 total 1054.57 last 90.35) - open/close 23311 (time 1789441959.93 total 1064.57 last 86.94) - open/close 24124 (time 1789441969.94 total 1074.58 last 81.23) - open/close 24874 (time 1789441979.94 total 1084.58 last 74.99) - open/close 25464 (time 1789441989.95 total 1094.60 last 58.92) - open/close 26098 (time 1789441999.96 total 1104.60 last 63.36) - open/close 27042 (time 1789442009.96 total 1114.61 last 94.38) - open/close 27912 (time 1789442019.97 total 1124.61 last 86.93) - open/close 28691 (time 1789442029.97 total 1134.62 last 77.85) - open/close 29419 (time 1789442039.99 total 1144.63 last 72.70) - open/close 30000 (time 1789442046.71 total 1151.36 last 86.37) - open/close 30670 (time 1789442056.72 total 1161.36 last 66.98) - open/close 31639 (time 1789442066.73 total 1171.38 last 96.74) - open/close 32314 (time 1789442076.74 total 1181.39 last 67.43) - open/close 33124 (time 1789442086.75 total 1191.39 last 80.97) - open/close 33907 (time 1789442096.75 total 1201.40 last 78.25) - open/close 34752 (time 1789442106.77 total 1211.42 last 84.36) - open/close 35409 (time 1789442116.80 total 1221.44 last 65.53) - open/close 36145 (time 1789442126.80 total 1231.45 last 73.56) - open/close 36898 (time 1789442136.81 total 1241.45 last 75.27) - open/close 37581 (time 1789442146.81 total 1251.46 last 68.25) - open/close 38420 (time 1789442156.82 total 1261.47 last 83.84) - open/close 39164 (time 1789442166.83 total 1271.48 last 74.30) - open/close 39884 (time 1789442176.84 total 1281.49 last 71.98) - open/close 40663 (time 1789442186.85 total 1291.50 last 77.81) - open/close 41417 (time 1789442196.85 total 1301.50 last 75.40) - open/close 42188 (time 1789442206.87 total 1311.52 last 76.96) - open/close 42831 (time 1789442216.89 total 1321.53 last 64.18) - open/close 43529 (time 1789442226.89 total 1331.54 last 69.75) - open/close 44397 (time 1789442236.90 total 1341.54 last 86.79) - open/close 45175 (time 1789442246.91 total 1351.55 last 77.73) - open/close 45969 (time 1789442256.91 total 1361.55 last 79.39) - open/close 46796 (time 1789442266.91 total 1371.56 last 82.69) - open/close 47512 (time 1789442276.91 total 1381.56 last 71.59) - open/close 48309 (time 1789442286.92 total 1391.56 last 79.63) - open/close 49083 (time 1789442296.92 total 1401.57 last 77.34) - open/close 49767 (time 1789442306.93 total 1411.57 last 68.38) total: 50000 open/close in 1414.39 seconds: 35.35 ops/second - unlinked 0 (time 1789442314 ; total 0 ; last 0) - unlinked 10000 (time 1789442451 ; total 137 ; last 137) - unlinked 20000 (time 1789442595 ; total 281 ; last 144) - unlinked 30000 (time 1789442735 ; total 421 ; last 140) - unlinked 40000 (time 1789442867 ; total 553 ; last 132) total: 50000 unlinks in 680 seconds: 73.529411 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: kmemleak disabled modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2