== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:58:03 (1788847083) start mds service on oleg332-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:48 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: kmemleak disabled oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg332-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg332-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1788847170.41 total 10.01 last 13.08) - open/close 290 (time 1788847181.28 total 20.89 last 14.62) - open/close 468 (time 1788847191.39 total 31.00 last 17.60) - open/close 643 (time 1788847201.40 total 41.01 last 17.49) - open/close 913 (time 1788847211.42 total 51.02 last 26.95) - open/close 1155 (time 1788847222.11 total 61.71 last 22.64) - open/close 1383 (time 1788847232.12 total 71.73 last 22.77) - open/close 1612 (time 1788847242.78 total 82.39 last 21.48) - open/close 1787 (time 1788847253.22 total 92.82 last 16.77) - open/close 1946 (time 1788847265.07 total 104.67 last 13.42) - open/close 2179 (time 1788847275.07 total 114.68 last 23.29) - open/close 2444 (time 1788847286.05 total 125.65 last 24.14) - open/close 2679 (time 1788847296.79 total 136.39 last 21.88) - open/close 2926 (time 1788847306.80 total 146.41 last 24.66) - open/close 3218 (time 1788847317.75 total 157.36 last 26.67) - open/close 3393 (time 1788847328.56 total 168.16 last 16.20) - open/close 3581 (time 1788847338.56 total 178.17 last 18.79) - open/close 3820 (time 1788847349.09 total 188.69 last 22.71) - open/close 4015 (time 1788847359.96 total 199.56 last 17.94) - open/close 4234 (time 1788847370.20 total 209.80 last 21.39) - open/close 4363 (time 1788847380.21 total 219.82 last 12.88) - open/close 4587 (time 1788847390.75 total 230.35 last 21.26) - open/close 4730 (time 1788847400.86 total 240.47 last 14.14) - open/close 4965 (time 1788847411.92 total 251.53 last 21.25) - open/close 5225 (time 1788847422.91 total 262.51 last 23.66) - open/close 5381 (time 1788847433.18 total 272.79 last 15.19) - open/close 5593 (time 1788847443.51 total 283.12 last 20.52) - open/close 5889 (time 1788847453.53 total 293.13 last 29.55) - open/close 6127 (time 1788847464.55 total 304.16 last 21.59) - open/close 6383 (time 1788847474.56 total 314.17 last 25.57) - open/close 6553 (time 1788847484.99 total 324.59 last 16.31) - open/close 6827 (time 1788847495.00 total 334.60 last 27.37) - open/close 7017 (time 1788847505.64 total 345.24 last 17.86) - open/close 7230 (time 1788847516.57 total 356.18 last 19.48) - open/close 7352 (time 1788847526.60 total 366.20 last 12.17) - open/close 7479 (time 1788847536.60 total 376.20 last 12.69) - open/close 7674 (time 1788847546.61 total 386.22 last 19.48) - open/close 7865 (time 1788847557.86 total 397.47 last 16.98) - open/close 8084 (time 1788847567.89 total 407.50 last 21.83) - open/close 8353 (time 1788847578.27 total 417.88 last 25.92) - open/close 8568 (time 1788847589.73 total 429.33 last 18.77) - open/close 8809 (time 1788847599.75 total 439.36 last 24.04) - open/close 9037 (time 1788847611.80 total 451.40 last 18.93) - open/close 9230 (time 1788847621.83 total 461.43 last 19.24) - open/close 9464 (time 1788847632.75 total 472.35 last 21.43) - open/close 9598 (time 1788847645.80 total 485.40 last 10.27) - open/close 9855 (time 1788847655.81 total 495.41 last 25.67) - open/close 10000 (time 1788847662.68 total 502.29 last 21.10) - open/close 10222 (time 1788847672.90 total 512.50 last 21.73) - open/close 10437 (time 1788847683.07 total 522.68 last 21.13) - open/close 10683 (time 1788847693.93 total 533.53 last 22.66) - open/close 10867 (time 1788847703.93 total 543.53 last 18.40) - open/close 11043 (time 1788847714.72 total 554.33 last 16.30) - open/close 11181 (time 1788847725.62 total 565.23 last 12.66) - open/close 11317 (time 1788847735.63 total 575.24 last 13.59) - open/close 11564 (time 1788847745.64 total 585.25 last 24.68) - open/close 11796 (time 1788847756.31 total 595.92 last 21.74) - open/close 12009 (time 1788847766.51 total 606.11 last 20.89) - open/close 12043 (time 1788847776.61 total 616.21 last 3.37) - open/close 12207 (time 1788847786.64 total 626.24 last 16.35) - open/close 12360 (time 1788847796.65 total 636.25 last 15.29) - open/close 12498 (time 1788847808.79 total 648.39 last 11.37) - open/close 12713 (time 1788847818.87 total 658.48 last 21.31) - open/close 12796 (time 1788847829.29 total 668.90 last 7.97) - open/close 12965 (time 1788847839.86 total 679.47 last 15.98) - open/close 13185 (time 1788847849.89 total 689.49 last 21.95) - open/close 13416 (time 1788847860.83 total 700.44 last 21.10) - open/close 13548 (time 1788847870.90 total 710.51 last 13.11) - open/close 13735 (time 1788847880.93 total 720.54 last 18.64) - open/close 13882 (time 1788847891.73 total 731.34 last 13.61) - open/close 14039 (time 1788847901.85 total 741.46 last 15.51) - open/close 14186 (time 1788847911.92 total 751.52 last 14.60) - open/close 14416 (time 1788847922.86 total 762.46 last 21.03) - open/close 14542 (time 1788847932.87 total 772.47 last 12.59) - open/close 14726 (time 1788847942.97 total 782.57 last 18.22) - open/close 14888 (time 1788847952.98 total 792.58 last 16.18) - open/close 14961 (time 1788847963.65 total 803.25 last 6.85) - open/close 15082 (time 1788847973.66 total 813.27 last 12.08) - open/close 15265 (time 1788847983.69 total 823.29 last 18.26) - open/close 15422 (time 1788847995.27 total 834.87 last 13.55) - open/close 15551 (time 1788848005.28 total 844.88 last 12.89) - open/close 15654 (time 1788848016.04 total 855.64 last 9.57) - open/close 15797 (time 1788848027.10 total 866.71 last 12.92) - open/close 15953 (time 1788848037.11 total 876.72 last 15.58) - open/close 16123 (time 1788848047.12 total 886.73 last 16.99) - open/close 16291 (time 1788848060.44 total 900.05 last 12.61) - open/close 16438 (time 1788848070.68 total 910.29 last 14.35) - open/close 16694 (time 1788848081.69 total 921.29 last 23.27) - open/close 16863 (time 1788848092.44 total 932.05 last 15.72) - open/close 17074 (time 1788848103.71 total 943.32 last 18.72) - open/close 17299 (time 1788848116.81 total 956.42 last 17.18) - open/close 17495 (time 1788848127.74 total 967.34 last 17.94) - open/close 17575 (time 1788848137.76 total 977.36 last 7.99) - open/close 17702 (time 1788848148.13 total 987.74 last 12.24) - open/close 17971 (time 1788848158.16 total 997.77 last 26.81) - open/close 18592 (time 1788848168.17 total 1007.78 last 62.04) - open/close 19210 (time 1788848178.18 total 1017.79 last 61.76) - open/close 19817 (time 1788848188.21 total 1027.81 last 60.52) - open/close 20000 (time 1788848191.21 total 1030.82 last 60.91) - open/close 20583 (time 1788848201.22 total 1040.82 last 58.29) - open/close 21125 (time 1788848211.22 total 1050.82 last 54.19) - open/close 21805 (time 1788848221.23 total 1060.83 last 67.96) - open/close 22470 (time 1788848231.23 total 1070.83 last 66.47) - open/close 23160 (time 1788848241.24 total 1080.85 last 68.92) - open/close 23825 (time 1788848251.25 total 1090.85 last 66.44) - open/close 24430 (time 1788848261.25 total 1100.86 last 60.48) - open/close 25040 (time 1788848271.26 total 1110.86 last 60.98) - open/close 25637 (time 1788848281.26 total 1120.87 last 59.67) - open/close 26221 (time 1788848291.27 total 1130.87 last 58.37) - open/close 26757 (time 1788848301.27 total 1140.87 last 53.59) - open/close 27468 (time 1788848311.30 total 1150.90 last 70.91) - open/close 28009 (time 1788848321.30 total 1160.90 last 54.08) - open/close 28651 (time 1788848331.30 total 1170.90 last 64.19) - open/close 29246 (time 1788848341.31 total 1180.91 last 59.44) - open/close 29833 (time 1788848351.33 total 1190.93 last 58.60) - open/close 30000 (time 1788848353.54 total 1193.15 last 75.33) - open/close 30623 (time 1788848363.55 total 1203.15 last 62.28) - open/close 31198 (time 1788848373.56 total 1213.16 last 57.43) - open/close 31783 (time 1788848383.56 total 1223.16 last 58.49) - open/close 32389 (time 1788848393.56 total 1233.17 last 60.57) - open/close 32956 (time 1788848403.57 total 1243.17 last 56.67) - open/close 33545 (time 1788848413.58 total 1253.18 last 58.86) - open/close 34167 (time 1788848423.60 total 1263.20 last 62.08) - open/close 34772 (time 1788848433.61 total 1273.21 last 60.44) - open/close 35409 (time 1788848443.61 total 1283.21 last 63.70) - open/close 35957 (time 1788848453.61 total 1293.21 last 54.78) - open/close 36587 (time 1788848463.62 total 1303.22 last 62.95) - open/close 37186 (time 1788848473.62 total 1313.23 last 59.87) - open/close 37781 (time 1788848483.65 total 1323.25 last 59.34) - open/close 38320 (time 1788848493.65 total 1333.26 last 53.88) - open/close 38880 (time 1788848503.67 total 1343.28 last 55.90) - open/close 39456 (time 1788848513.67 total 1353.28 last 57.59) - open/close 40000 (time 1788848523.14 total 1362.74 last 57.48) - open/close 40596 (time 1788848533.15 total 1372.76 last 59.50) - open/close 41153 (time 1788848543.17 total 1382.77 last 55.63) - open/close 41793 (time 1788848553.18 total 1392.78 last 63.93) - open/close 42325 (time 1788848563.19 total 1402.79 last 53.14) - open/close 42818 (time 1788848573.20 total 1412.80 last 49.26) - open/close 43353 (time 1788848583.20 total 1422.81 last 53.48) - open/close 43890 (time 1788848593.20 total 1432.81 last 53.70) - open/close 44498 (time 1788848603.22 total 1442.82 last 60.72) - open/close 45029 (time 1788848613.22 total 1452.83 last 53.05) - open/close 45536 (time 1788848623.23 total 1462.83 last 50.68) - open/close 46096 (time 1788848633.23 total 1472.84 last 55.96) - open/close 46708 (time 1788848643.24 total 1482.85 last 61.16) - open/close 47293 (time 1788848653.26 total 1492.87 last 58.36) - open/close 47863 (time 1788848663.29 total 1502.89 last 56.88) - open/close 48458 (time 1788848673.30 total 1512.90 last 59.43) - open/close 49040 (time 1788848683.31 total 1522.92 last 58.11) - open/close 49627 (time 1788848693.33 total 1532.93 last 58.60) total: 50000 open/close in 1538.65 seconds: 32.50 ops/second - unlinked 0 (time 1788848702 ; total 0 ; last 0) - unlinked 10000 (time 1788848879 ; total 177 ; last 177) - unlinked 20000 (time 1788849053 ; total 351 ; last 174) - unlinked 30000 (time 1788849220 ; total 518 ; last 167) - unlinked 40000 (time 1788849380 ; total 678 ; last 160) total: 50000 unlinks in 833 seconds: 60.024010 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: kmemleak disabled modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2