== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:06:57 (1789603617) start mds service on oleg454-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 16 11:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: kmemleak disabled oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg454-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg454-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 222 (time 1789603699.82 total 10.85 last 20.47) - open/close 328 (time 1789603710.19 total 21.22 last 10.22) - open/close 458 (time 1789603720.20 total 31.23 last 12.98) - open/close 618 (time 1789603730.22 total 41.25 last 15.97) - open/close 741 (time 1789603742.93 total 53.96 last 9.68) - open/close 933 (time 1789603753.12 total 64.15 last 18.84) - open/close 1164 (time 1789603763.64 total 74.67 last 21.96) - open/close 1407 (time 1789603774.57 total 85.60 last 22.22) - open/close 1688 (time 1789603785.07 total 96.10 last 26.77) - open/close 1930 (time 1789603795.18 total 106.21 last 23.93) - open/close 2158 (time 1789603805.18 total 116.21 last 22.80) - open/close 2383 (time 1789603815.27 total 126.30 last 22.30) - open/close 2604 (time 1789603825.28 total 136.31 last 22.08) - open/close 2822 (time 1789603835.49 total 146.52 last 21.34) - open/close 3068 (time 1789603846.39 total 157.42 last 22.57) - open/close 3263 (time 1789603860.93 total 171.96 last 13.42) - open/close 3491 (time 1789603871.26 total 182.29 last 22.06) - open/close 3734 (time 1789603881.98 total 193.01 last 22.68) - open/close 3966 (time 1789603892.79 total 203.82 last 21.46) - open/close 4217 (time 1789603903.79 total 214.82 last 22.82) - open/close 4459 (time 1789603914.42 total 225.45 last 22.77) - open/close 4702 (time 1789603924.97 total 236.00 last 23.02) - open/close 4964 (time 1789603934.98 total 246.01 last 26.18) - open/close 5172 (time 1789603945.44 total 256.47 last 19.88) - open/close 5402 (time 1789603955.87 total 266.90 last 22.05) - open/close 5595 (time 1789603965.93 total 276.96 last 19.20) - open/close 5776 (time 1789603976.16 total 287.19 last 17.68) - open/close 5937 (time 1789603986.22 total 297.25 last 16.01) - open/close 6038 (time 1789604001.27 total 312.30 last 6.71) - open/close 6227 (time 1789604011.28 total 322.30 last 18.89) - open/close 6445 (time 1789604024.16 total 335.19 last 16.91) - open/close 6672 (time 1789604034.93 total 345.96 last 21.09) - open/close 6924 (time 1789604045.81 total 356.84 last 23.15) - open/close 7101 (time 1789604056.81 total 367.84 last 16.09) - open/close 7333 (time 1789604067.38 total 378.41 last 21.96) - open/close 7573 (time 1789604077.38 total 388.41 last 24.00) - open/close 7765 (time 1789604088.27 total 399.30 last 17.64) - open/close 7979 (time 1789604099.24 total 410.27 last 19.50) - open/close 8238 (time 1789604109.24 total 420.27 last 25.89) - open/close 8437 (time 1789604119.33 total 430.36 last 19.73) - open/close 8678 (time 1789604130.02 total 441.05 last 22.54) - open/close 8899 (time 1789604140.33 total 451.36 last 21.44) - open/close 9154 (time 1789604150.34 total 461.37 last 25.47) - open/close 9427 (time 1789604160.35 total 471.38 last 27.28) - open/close 9644 (time 1789604171.33 total 482.36 last 19.75) - open/close 9721 (time 1789604183.49 total 494.52 last 6.34) - open/close 9843 (time 1789604193.52 total 504.55 last 12.16) - open/close 9991 (time 1789604203.54 total 514.57 last 14.78) - open/close 10169 (time 1789604214.45 total 525.48 last 16.32) - open/close 10475 (time 1789604224.46 total 535.49 last 30.56) - open/close 10635 (time 1789604234.49 total 545.52 last 15.94) - open/close 10826 (time 1789604245.42 total 556.44 last 17.49) - open/close 11044 (time 1789604255.43 total 566.46 last 21.78) - open/close 11224 (time 1789604265.44 total 576.47 last 17.98) - open/close 11401 (time 1789604275.45 total 586.48 last 17.67) - open/close 11633 (time 1789604286.05 total 597.08 last 21.90) - open/close 11849 (time 1789604296.88 total 607.91 last 19.94) - open/close 12076 (time 1789604307.51 total 618.53 last 21.36) - open/close 12286 (time 1789604318.16 total 629.18 last 19.72) - open/close 12402 (time 1789604329.02 total 640.04 last 10.68) - open/close 12458 (time 1789604339.03 total 650.06 last 5.59) - open/close 12667 (time 1789604349.29 total 660.32 last 20.37) - open/close 12783 (time 1789604359.48 total 670.51 last 11.39) - open/close 12936 (time 1789604370.15 total 681.18 last 14.33) - open/close 12968 (time 1789604384.35 total 695.38 last 2.25) - open/close 13169 (time 1789604395.06 total 706.09 last 18.76) - open/close 13331 (time 1789604405.81 total 716.84 last 15.07) - open/close 13421 (time 1789604416.60 total 727.63 last 8.34) - open/close 13577 (time 1789604427.20 total 738.23 last 14.72) - open/close 13806 (time 1789604437.21 total 748.24 last 22.87) - open/close 14039 (time 1789604447.22 total 758.25 last 23.28) - open/close 14229 (time 1789604458.10 total 769.13 last 17.46) - open/close 14421 (time 1789604468.13 total 779.15 last 19.15) - open/close 14539 (time 1789604478.15 total 789.18 last 11.77) - open/close 14639 (time 1789604488.32 total 799.35 last 9.83) - open/close 14797 (time 1789604498.92 total 809.95 last 14.90) - open/close 14982 (time 1789604509.94 total 820.97 last 16.79) - open/close 15167 (time 1789604520.12 total 831.15 last 18.17) - open/close 15304 (time 1789604530.13 total 841.16 last 13.69) - open/close 15403 (time 1789604540.16 total 851.19 last 9.88) - open/close 15493 (time 1789604550.70 total 861.73 last 8.54) - open/close 15659 (time 1789604561.53 total 872.56 last 15.33) - open/close 15807 (time 1789604572.61 total 883.64 last 13.35) - open/close 15944 (time 1789604583.15 total 894.18 last 13.00) - open/close 16176 (time 1789604593.21 total 904.24 last 23.07) - open/close 16253 (time 1789604603.22 total 914.25 last 7.69) - open/close 16489 (time 1789604613.24 total 924.27 last 23.55) - open/close 16675 (time 1789604623.57 total 934.60 last 18.00) - open/close 16863 (time 1789604633.58 total 944.60 last 18.79) - open/close 16979 (time 1789604643.61 total 954.63 last 11.56) - open/close 17066 (time 1789604655.15 total 966.18 last 7.54) - open/close 17189 (time 1789604666.18 total 977.20 last 11.15) - open/close 17387 (time 1789604677.88 total 988.91 last 16.92) - open/close 17501 (time 1789604687.91 total 998.94 last 11.37) - open/close 18058 (time 1789604697.92 total 1008.95 last 55.64) - open/close 18637 (time 1789604707.93 total 1018.96 last 57.86) - open/close 19244 (time 1789604717.93 total 1028.96 last 60.67) - open/close 19849 (time 1789604727.94 total 1038.97 last 60.47) - open/close 20000 (time 1789604730.77 total 1041.80 last 53.33) - open/close 20563 (time 1789604740.79 total 1051.82 last 56.19) - open/close 21089 (time 1789604750.79 total 1061.82 last 52.58) - open/close 21644 (time 1789604760.80 total 1071.83 last 55.43) - open/close 22243 (time 1789604770.81 total 1081.83 last 59.89) - open/close 22838 (time 1789604780.81 total 1091.84 last 59.48) - open/close 23414 (time 1789604790.82 total 1101.84 last 57.56) - open/close 24021 (time 1789604800.82 total 1111.84 last 60.70) - open/close 24552 (time 1789604810.82 total 1121.85 last 53.07) - open/close 25135 (time 1789604820.83 total 1131.86 last 58.23) - open/close 25738 (time 1789604830.84 total 1141.87 last 60.24) - open/close 26306 (time 1789604840.85 total 1151.88 last 56.75) - open/close 26917 (time 1789604850.86 total 1161.89 last 61.07) - open/close 27517 (time 1789604860.86 total 1171.89 last 59.98) - open/close 28169 (time 1789604870.87 total 1181.90 last 65.16) - open/close 28698 (time 1789604880.88 total 1191.91 last 52.84) - open/close 29196 (time 1789604890.89 total 1201.92 last 49.76) - open/close 29777 (time 1789604900.91 total 1211.94 last 57.96) - open/close 30000 (time 1789604904.99 total 1216.02 last 54.72) - open/close 30605 (time 1789604915.00 total 1226.03 last 60.43) - open/close 31164 (time 1789604925.02 total 1236.05 last 55.77) - open/close 31745 (time 1789604935.03 total 1246.06 last 58.05) - open/close 32317 (time 1789604945.03 total 1256.06 last 57.19) - open/close 32912 (time 1789604955.04 total 1266.07 last 59.43) - open/close 33520 (time 1789604965.06 total 1276.08 last 60.73) - open/close 34135 (time 1789604975.06 total 1286.09 last 61.45) - open/close 34750 (time 1789604985.06 total 1296.09 last 61.49) - open/close 35353 (time 1789604995.08 total 1306.11 last 60.22) - open/close 35943 (time 1789605005.10 total 1316.13 last 58.90) - open/close 36517 (time 1789605015.10 total 1326.13 last 57.36) - open/close 37121 (time 1789605025.11 total 1336.14 last 60.37) - open/close 37727 (time 1789605035.11 total 1346.14 last 60.57) - open/close 38304 (time 1789605045.12 total 1356.15 last 57.67) - open/close 38901 (time 1789605055.12 total 1366.15 last 59.69) - open/close 39480 (time 1789605065.13 total 1376.16 last 57.84) - open/close 40000 (time 1789605074.76 total 1385.78 last 54.02) - open/close 40557 (time 1789605084.76 total 1395.79 last 55.69) - open/close 41129 (time 1789605094.76 total 1405.79 last 57.16) - open/close 41736 (time 1789605104.77 total 1415.80 last 60.69) - open/close 42348 (time 1789605114.78 total 1425.81 last 61.12) - open/close 42942 (time 1789605124.80 total 1435.83 last 59.27) - open/close 43542 (time 1789605134.81 total 1445.83 last 59.97) - open/close 44142 (time 1789605144.83 total 1455.86 last 59.86) - open/close 44761 (time 1789605154.85 total 1465.88 last 61.77) - open/close 45320 (time 1789605164.85 total 1475.88 last 55.89) - open/close 45955 (time 1789605174.86 total 1485.89 last 63.42) - open/close 46532 (time 1789605184.88 total 1495.91 last 57.63) - open/close 47150 (time 1789605194.89 total 1505.92 last 61.69) - open/close 47685 (time 1789605204.90 total 1515.92 last 53.49) - open/close 48237 (time 1789605214.90 total 1525.93 last 55.19) - open/close 48787 (time 1789605224.90 total 1535.93 last 55.00) - open/close 49396 (time 1789605234.91 total 1545.94 last 60.82) - open/close 49922 (time 1789605244.93 total 1555.96 last 52.50) total: 50000 open/close in 1557.05 seconds: 32.11 ops/second - unlinked 0 (time 1789605251 ; total 0 ; last 0) - unlinked 10000 (time 1789605432 ; total 181 ; last 181) - unlinked 20000 (time 1789605610 ; total 359 ; last 178) - unlinked 30000 (time 1789605791 ; total 540 ; last 181) - unlinked 40000 (time 1789605962 ; total 711 ; last 171) total: 50000 unlinks in 883 seconds: 56.625141 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: kmemleak disabled modules unloaded. pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2