== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:33:40 (1788755620) start mds service on oleg122-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: kmemleak disabled oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg122-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg122-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1788755714.78 total 10.01 last 19.88) - open/close 412 (time 1788755725.81 total 21.04 last 19.31) - open/close 592 (time 1788755736.18 total 31.41 last 17.36) - open/close 823 (time 1788755746.37 total 41.59 last 22.67) - open/close 1108 (time 1788755756.39 total 51.61 last 28.44) - open/close 1331 (time 1788755766.52 total 61.74 last 22.01) - open/close 1577 (time 1788755777.10 total 72.32 last 23.25) - open/close 1838 (time 1788755787.83 total 83.06 last 24.31) - open/close 2088 (time 1788755798.01 total 93.24 last 24.58) - open/close 2335 (time 1788755808.90 total 104.13 last 22.68) - open/close 2568 (time 1788755819.64 total 114.87 last 21.69) - open/close 2792 (time 1788755829.64 total 124.87 last 22.40) - open/close 3049 (time 1788755839.65 total 134.87 last 25.70) - open/close 3283 (time 1788755849.66 total 144.89 last 23.37) - open/close 3555 (time 1788755860.34 total 155.56 last 25.47) - open/close 3761 (time 1788755870.91 total 166.14 last 19.48) - open/close 3940 (time 1788755883.52 total 178.75 last 14.20) - open/close 4071 (time 1788755894.15 total 189.38 last 12.32) - open/close 4279 (time 1788755904.98 total 200.21 last 19.21) - open/close 4392 (time 1788755915.79 total 211.01 last 10.46) - open/close 4522 (time 1788755926.29 total 221.51 last 12.38) - open/close 4656 (time 1788755936.68 total 231.91 last 12.89) - open/close 4755 (time 1788755947.10 total 242.33 last 9.51) - open/close 5015 (time 1788755957.11 total 252.33 last 25.98) - open/close 5233 (time 1788755968.05 total 263.28 last 19.92) - open/close 5406 (time 1788755978.05 total 273.28 last 17.30) - open/close 5579 (time 1788755988.10 total 283.32 last 17.22) - open/close 5804 (time 1788755998.42 total 293.64 last 21.80) - open/close 6064 (time 1788756009.16 total 304.39 last 24.20) - open/close 6284 (time 1788756019.91 total 315.14 last 20.45) - open/close 6448 (time 1788756030.49 total 325.71 last 15.52) - open/close 6520 (time 1788756043.26 total 338.49 last 5.64) - open/close 6753 (time 1788756054.16 total 349.39 last 21.37) - open/close 6959 (time 1788756064.16 total 359.39 last 20.60) - open/close 7167 (time 1788756074.43 total 369.66 last 20.25) - open/close 7321 (time 1788756084.54 total 379.77 last 15.23) - open/close 7526 (time 1788756094.81 total 390.04 last 19.97) - open/close 7773 (time 1788756105.73 total 400.96 last 22.62) - open/close 7892 (time 1788756117.50 total 412.73 last 10.11) - open/close 8083 (time 1788756127.51 total 422.74 last 19.08) - open/close 8344 (time 1788756137.68 total 432.91 last 25.66) - open/close 8516 (time 1788756148.49 total 443.71 last 15.92) - open/close 8670 (time 1788756159.10 total 454.33 last 14.51) - open/close 8812 (time 1788756169.10 total 464.33 last 14.20) - open/close 9024 (time 1788756179.52 total 474.75 last 20.35) - open/close 9248 (time 1788756189.52 total 484.75 last 22.39) - open/close 9465 (time 1788756199.96 total 495.19 last 20.80) - open/close 9551 (time 1788756210.12 total 505.35 last 8.46) - open/close 9720 (time 1788756220.89 total 516.11 last 15.70) - open/close 9881 (time 1788756230.99 total 526.21 last 15.94) - open/close 9931 (time 1788756242.77 total 538.00 last 4.24) - open/close 10000 (time 1788756247.52 total 542.75 last 14.52) - open/close 10101 (time 1788756257.54 total 552.77 last 10.08) - open/close 10357 (time 1788756267.90 total 563.13 last 24.72) - open/close 10584 (time 1788756277.93 total 573.16 last 22.62) - open/close 10832 (time 1788756287.94 total 583.17 last 24.77) - open/close 11038 (time 1788756297.95 total 593.17 last 20.60) - open/close 11315 (time 1788756308.21 total 603.44 last 26.98) - open/close 11602 (time 1788756318.22 total 613.45 last 28.68) - open/close 11816 (time 1788756328.24 total 623.46 last 21.36) - open/close 12015 (time 1788756339.07 total 634.29 last 18.38) - open/close 12272 (time 1788756349.09 total 644.32 last 25.64) - open/close 12512 (time 1788756359.16 total 654.39 last 23.82) - open/close 12651 (time 1788756369.17 total 664.40 last 13.89) - open/close 12859 (time 1788756379.19 total 674.42 last 20.76) - open/close 13035 (time 1788756390.17 total 685.40 last 16.02) - open/close 13120 (time 1788756400.25 total 695.48 last 8.43) - open/close 13278 (time 1788756411.21 total 706.44 last 14.41) - open/close 13429 (time 1788756421.84 total 717.06 last 14.22) - open/close 13492 (time 1788756432.52 total 727.75 last 5.90) - open/close 13773 (time 1788756442.53 total 737.76 last 28.07) - open/close 14035 (time 1788756452.55 total 747.78 last 26.15) - open/close 14244 (time 1788756463.18 total 758.40 last 19.66) - open/close 14482 (time 1788756473.61 total 768.83 last 22.82) - open/close 14728 (time 1788756483.62 total 778.85 last 24.56) - open/close 14976 (time 1788756493.63 total 788.86 last 24.79) - open/close 15172 (time 1788756504.06 total 799.28 last 18.79) - open/close 15407 (time 1788756514.78 total 810.01 last 21.92) - open/close 15615 (time 1788756524.79 total 820.02 last 20.78) - open/close 15867 (time 1788756535.29 total 830.52 last 24.01) - open/close 16097 (time 1788756545.30 total 840.53 last 22.96) - open/close 16313 (time 1788756555.38 total 850.61 last 21.43) - open/close 16517 (time 1788756565.50 total 860.73 last 20.17) - open/close 16746 (time 1788756575.50 total 870.73 last 22.89) - open/close 16953 (time 1788756585.92 total 881.15 last 19.87) - open/close 17215 (time 1788756596.86 total 892.09 last 23.95) - open/close 17462 (time 1788756608.72 total 903.95 last 20.83) - open/close 17611 (time 1788756619.23 total 914.46 last 14.17) - open/close 17844 (time 1788756630.14 total 925.37 last 21.36) - open/close 18097 (time 1788756640.14 total 935.37 last 25.29) - open/close 18349 (time 1788756650.16 total 945.39 last 25.16) - open/close 18588 (time 1788756660.29 total 955.52 last 23.59) - open/close 18829 (time 1788756670.69 total 965.92 last 23.17) - open/close 19111 (time 1788756680.70 total 975.92 last 28.20) - open/close 19796 (time 1788756690.71 total 985.94 last 68.38) - open/close 20000 (time 1788756693.69 total 988.91 last 68.62) - open/close 20781 (time 1788756703.70 total 998.93 last 77.98) - open/close 21552 (time 1788756713.71 total 1008.94 last 77.04) - open/close 22254 (time 1788756723.71 total 1018.94 last 70.17) - open/close 22967 (time 1788756733.71 total 1028.94 last 71.28) - open/close 23748 (time 1788756743.72 total 1038.94 last 78.09) - open/close 24566 (time 1788756753.72 total 1048.95 last 81.77) - open/close 25285 (time 1788756763.72 total 1058.95 last 71.88) - open/close 26098 (time 1788756773.72 total 1068.95 last 81.29) - open/close 26856 (time 1788756783.73 total 1078.96 last 75.72) - open/close 27579 (time 1788756793.74 total 1088.97 last 72.26) - open/close 28355 (time 1788756803.74 total 1098.97 last 77.59) - open/close 29206 (time 1788756813.77 total 1109.00 last 84.85) - open/close 29949 (time 1788756823.77 total 1119.00 last 74.27) - open/close 30742 (time 1788756833.77 total 1129.00 last 79.28) - open/close 31638 (time 1788756843.78 total 1139.00 last 89.59) - open/close 32510 (time 1788756853.78 total 1149.00 last 87.19) - open/close 33303 (time 1788756863.78 total 1159.01 last 79.25) - open/close 34240 (time 1788756873.78 total 1169.01 last 93.69) - open/close 35101 (time 1788756883.79 total 1179.02 last 86.04) - open/close 35877 (time 1788756893.79 total 1189.02 last 77.58) - open/close 36715 (time 1788756903.80 total 1199.03 last 83.73) - open/close 37457 (time 1788756913.84 total 1209.07 last 73.91) - open/close 38225 (time 1788756923.85 total 1219.08 last 76.72) - open/close 39069 (time 1788756933.86 total 1229.09 last 84.31) - open/close 39940 (time 1788756943.87 total 1239.10 last 87.07) - open/close 40725 (time 1788756953.87 total 1249.10 last 78.47) - open/close 41610 (time 1788756963.87 total 1259.10 last 88.48) - open/close 42320 (time 1788756973.92 total 1269.15 last 70.68) - open/close 42992 (time 1788756983.92 total 1279.15 last 67.18) - open/close 43763 (time 1788756993.93 total 1289.16 last 77.06) - open/close 44601 (time 1788757003.93 total 1299.16 last 83.76) - open/close 45456 (time 1788757013.94 total 1309.17 last 85.41) - open/close 46240 (time 1788757023.94 total 1319.17 last 78.40) - open/close 47042 (time 1788757033.95 total 1329.18 last 80.16) - open/close 47831 (time 1788757043.96 total 1339.19 last 78.81) - open/close 48578 (time 1788757053.97 total 1349.20 last 74.63) - open/close 49290 (time 1788757063.98 total 1359.20 last 71.15) - open/close 49947 (time 1788757073.98 total 1369.21 last 65.69) total: 50000 open/close in 1369.97 seconds: 36.50 ops/second - unlinked 0 (time 1788757078 ; total 0 ; last 0) - unlinked 10000 (time 1788757217 ; total 139 ; last 139) - unlinked 20000 (time 1788757358 ; total 280 ; last 141) - unlinked 30000 (time 1788757505 ; total 427 ; last 147) - unlinked 40000 (time 1788757650 ; total 572 ; last 145) total: 50000 unlinks in 708 seconds: 70.621468 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: kmemleak disabled modules unloaded. pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2