== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:41:01 (1788471661) start mds service on oleg142-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: kmemleak disabled oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: detected 4 online CPUs by sysfs oleg142-server: MODOPTS_LIBCFS= oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg142-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg142-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg142-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg142-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 180 (time 1788471755.56 total 11.23 last 16.02) - open/close 449 (time 1788471765.58 total 21.25 last 26.86) - open/close 700 (time 1788471776.10 total 31.77 last 23.85) - open/close 938 (time 1788471786.98 total 42.65 last 21.87) - open/close 1126 (time 1788471797.41 total 53.08 last 18.02) - open/close 1359 (time 1788471808.03 total 63.70 last 21.94) - open/close 1596 (time 1788471818.49 total 74.16 last 22.66) - open/close 1879 (time 1788471828.54 total 84.21 last 28.17) - open/close 2022 (time 1788471838.98 total 94.65 last 13.70) - open/close 2273 (time 1788471848.98 total 104.65 last 25.10) - open/close 2503 (time 1788471861.73 total 117.40 last 18.03) - open/close 2655 (time 1788471871.97 total 127.64 last 14.85) - open/close 2805 (time 1788471882.56 total 138.23 last 14.17) - open/close 2922 (time 1788471893.16 total 148.83 last 11.03) - open/close 3023 (time 1788471905.77 total 161.44 last 8.01) - open/close 3260 (time 1788471916.63 total 172.30 last 21.81) - open/close 3505 (time 1788471926.65 total 182.32 last 24.46) - open/close 3719 (time 1788471936.94 total 192.61 last 20.80) - open/close 3969 (time 1788471946.96 total 202.63 last 24.94) - open/close 4166 (time 1788471956.99 total 212.66 last 19.65) - open/close 4282 (time 1788471967.22 total 222.89 last 11.33) - open/close 4477 (time 1788471977.23 total 232.90 last 19.48) - open/close 4665 (time 1788471988.03 total 243.70 last 17.41) - open/close 4887 (time 1788471998.69 total 254.36 last 20.82) - open/close 5100 (time 1788472008.70 total 264.37 last 21.29) - open/close 5302 (time 1788472019.23 total 274.90 last 19.18) - open/close 5511 (time 1788472029.41 total 285.08 last 20.53) - open/close 5787 (time 1788472039.42 total 295.08 last 27.59) - open/close 5997 (time 1788472051.23 total 306.90 last 17.77) - open/close 6116 (time 1788472061.26 total 316.93 last 11.87) - open/close 6321 (time 1788472071.28 total 326.95 last 20.45) - open/close 6582 (time 1788472081.83 total 337.50 last 24.75) - open/close 6826 (time 1788472092.46 total 348.13 last 22.96) - open/close 7108 (time 1788472103.47 total 359.14 last 25.60) - open/close 7388 (time 1788472114.53 total 370.20 last 25.31) - open/close 7605 (time 1788472124.54 total 380.21 last 21.68) - open/close 7742 (time 1788472134.91 total 390.58 last 13.22) - open/close 7869 (time 1788472144.92 total 400.59 last 12.68) - open/close 8023 (time 1788472155.05 total 410.71 last 15.21) - open/close 8184 (time 1788472167.27 total 422.94 last 13.17) - open/close 8373 (time 1788472177.28 total 432.95 last 18.89) - open/close 8557 (time 1788472188.22 total 443.88 last 16.82) - open/close 8842 (time 1788472198.23 total 453.90 last 28.46) - open/close 8958 (time 1788472208.23 total 463.90 last 11.60) - open/close 9149 (time 1788472218.23 total 473.90 last 19.10) - open/close 9320 (time 1788472229.98 total 485.65 last 14.55) - open/close 9488 (time 1788472241.08 total 496.75 last 15.14) - open/close 9624 (time 1788472251.14 total 506.81 last 13.52) - open/close 9869 (time 1788472262.12 total 517.79 last 22.32) - open/close 9964 (time 1788472272.12 total 527.79 last 9.50) - open/close 10000 (time 1788472274.13 total 529.80 last 17.93) - open/close 10052 (time 1788472285.11 total 540.78 last 4.74) - open/close 10180 (time 1788472295.11 total 550.78 last 12.80) - open/close 10442 (time 1788472305.12 total 560.79 last 26.17) - open/close 10702 (time 1788472315.12 total 570.79 last 25.99) - open/close 10857 (time 1788472325.16 total 580.83 last 15.45) - open/close 11011 (time 1788472335.86 total 591.52 last 14.40) - open/close 11102 (time 1788472346.29 total 601.96 last 8.72) - open/close 11379 (time 1788472356.31 total 611.98 last 27.66) - open/close 11611 (time 1788472366.92 total 622.59 last 21.86) - open/close 11893 (time 1788472376.94 total 632.61 last 28.15) - open/close 12072 (time 1788472389.37 total 645.04 last 14.40) - open/close 12269 (time 1788472400.40 total 656.07 last 17.86) - open/close 12447 (time 1788472410.40 total 666.07 last 17.80) - open/close 12549 (time 1788472420.59 total 676.26 last 10.01) - open/close 12636 (time 1788472430.61 total 686.28 last 8.68) - open/close 12813 (time 1788472442.67 total 698.34 last 14.67) - open/close 12920 (time 1788472453.30 total 708.97 last 10.07) - open/close 13133 (time 1788472464.12 total 719.79 last 19.68) - open/close 13340 (time 1788472474.14 total 729.81 last 20.66) - open/close 13450 (time 1788472486.33 total 741.99 last 9.03) - open/close 13754 (time 1788472496.34 total 752.01 last 30.35) - open/close 13881 (time 1788472507.25 total 762.92 last 11.64) - open/close 14067 (time 1788472517.32 total 772.99 last 18.47) - open/close 14260 (time 1788472527.32 total 782.99 last 19.30) - open/close 14514 (time 1788472537.34 total 793.01 last 25.36) - open/close 14745 (time 1788472547.89 total 803.56 last 21.90) - open/close 14860 (time 1788472557.91 total 813.58 last 11.48) - open/close 15097 (time 1788472567.91 total 823.58 last 23.69) - open/close 15301 (time 1788472578.03 total 833.70 last 20.15) - open/close 15517 (time 1788472588.59 total 844.26 last 20.45) - open/close 15789 (time 1788472599.69 total 855.35 last 24.52) - open/close 15982 (time 1788472609.82 total 865.49 last 19.04) - open/close 16266 (time 1788472619.83 total 875.50 last 28.37) - open/close 16474 (time 1788472630.96 total 886.63 last 18.70) - open/close 16743 (time 1788472640.97 total 896.64 last 26.87) - open/close 16987 (time 1788472650.98 total 906.65 last 24.37) - open/close 17173 (time 1788472662.09 total 917.76 last 16.73) - open/close 17330 (time 1788472672.31 total 927.98 last 15.37) - open/close 17440 (time 1788472682.95 total 938.62 last 10.34) - open/close 17612 (time 1788472692.95 total 948.62 last 17.19) - open/close 17826 (time 1788472703.07 total 958.74 last 21.15) - open/close 18001 (time 1788472713.14 total 968.81 last 17.37) - open/close 18158 (time 1788472723.53 total 979.20 last 15.11) - open/close 18402 (time 1788472734.37 total 990.04 last 22.52) - open/close 18791 (time 1788472744.38 total 1000.05 last 38.86) - open/close 19622 (time 1788472754.38 total 1010.05 last 83.05) - open/close 20000 (time 1788472759.30 total 1014.97 last 76.82) - open/close 20841 (time 1788472769.31 total 1024.98 last 84.04) - open/close 21704 (time 1788472779.31 total 1034.98 last 86.29) - open/close 22604 (time 1788472789.31 total 1044.98 last 89.99) - open/close 23571 (time 1788472799.32 total 1054.99 last 96.64) - open/close 24400 (time 1788472809.32 total 1064.99 last 82.90) - open/close 25336 (time 1788472819.32 total 1074.99 last 93.57) - open/close 26209 (time 1788472829.33 total 1085.00 last 87.27) - open/close 27119 (time 1788472839.33 total 1095.00 last 90.96) - open/close 27888 (time 1788472849.33 total 1105.00 last 76.89) - open/close 28765 (time 1788472859.35 total 1115.02 last 87.57) - open/close 29532 (time 1788472869.35 total 1125.02 last 76.68) - open/close 30000 (time 1788472875.42 total 1131.09 last 77.02) - open/close 30888 (time 1788472885.43 total 1141.10 last 88.72) - open/close 31694 (time 1788472895.45 total 1151.12 last 80.49) - open/close 32440 (time 1788472905.45 total 1161.12 last 74.55) - open/close 33262 (time 1788472915.46 total 1171.13 last 82.13) - open/close 34015 (time 1788472925.47 total 1181.14 last 75.26) - open/close 34818 (time 1788472935.48 total 1191.15 last 80.25) - open/close 35546 (time 1788472945.48 total 1201.15 last 72.78) - open/close 36262 (time 1788472955.48 total 1211.15 last 71.58) - open/close 37193 (time 1788472965.51 total 1221.18 last 92.87) - open/close 37939 (time 1788472975.51 total 1231.18 last 74.53) - open/close 38614 (time 1788472985.53 total 1241.20 last 67.40) - open/close 39520 (time 1788472995.54 total 1251.20 last 90.54) - open/close 40000 (time 1788473000.94 total 1256.61 last 88.77) - open/close 40635 (time 1788473010.95 total 1266.62 last 63.44) - open/close 41398 (time 1788473020.98 total 1276.65 last 76.12) - open/close 42238 (time 1788473030.99 total 1286.66 last 83.89) - open/close 43030 (time 1788473041.00 total 1296.67 last 79.13) - open/close 43662 (time 1788473051.01 total 1306.68 last 63.12) - open/close 44368 (time 1788473061.03 total 1316.70 last 70.49) - open/close 45166 (time 1788473071.03 total 1326.70 last 79.74) - open/close 45933 (time 1788473081.05 total 1336.72 last 76.58) - open/close 46699 (time 1788473091.06 total 1346.72 last 76.56) - open/close 47296 (time 1788473101.11 total 1356.78 last 59.39) - open/close 47980 (time 1788473111.11 total 1366.78 last 68.37) - open/close 48559 (time 1788473121.12 total 1376.79 last 57.85) - open/close 49333 (time 1788473131.13 total 1386.80 last 77.34) total: 50000 open/close in 1396.14 seconds: 35.81 ops/second - unlinked 0 (time 1788473145 ; total 0 ; last 0) - unlinked 10000 (time 1788473285 ; total 140 ; last 140) - unlinked 20000 (time 1788473417 ; total 272 ; last 132) - unlinked 30000 (time 1788473552 ; total 407 ; last 135) - unlinked 40000 (time 1788473686 ; total 541 ; last 134) total: 50000 unlinks in 681 seconds: 73.421440 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server unloading modules via unload_modules_local on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local oleg142-server: kmemleak disabled modules unloaded. pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2