== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:54:23 (1790895263) start mds service on oleg258-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Oct 1 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing load_modules_local oleg258-server: kmemleak disabled oleg258-server: Loading modules from /home/green/git/lustre-release/lustre oleg258-server: detected 4 online CPUs by sysfs oleg258-server: MODOPTS_LIBCFS= oleg258-server: Force libcfs to create 2 CPU partitions oleg258-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg258-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg258-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg258-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 249 (time 1790895361.50 total 10.73 last 23.20) - open/close 472 (time 1790895371.51 total 20.74 last 22.29) - open/close 608 (time 1790895381.51 total 30.74 last 13.60) - open/close 837 (time 1790895391.52 total 40.75 last 22.88) - open/close 906 (time 1790895402.41 total 51.64 last 6.34) - open/close 1090 (time 1790895412.42 total 61.65 last 18.39) - open/close 1210 (time 1790895422.42 total 71.66 last 11.99) - open/close 1411 (time 1790895432.43 total 81.66 last 20.10) - open/close 1550 (time 1790895443.38 total 92.61 last 12.69) - open/close 1672 (time 1790895453.92 total 103.15 last 11.57) - open/close 1750 (time 1790895464.04 total 113.27 last 7.71) - open/close 1866 (time 1790895474.33 total 123.56 last 11.28) - open/close 2050 (time 1790895485.10 total 134.33 last 17.08) - open/close 2101 (time 1790895495.88 total 145.11 last 4.73) - open/close 2244 (time 1790895505.92 total 155.15 last 14.24) - open/close 2321 (time 1790895516.28 total 165.51 last 7.43) - open/close 2455 (time 1790895526.70 total 175.94 last 12.86) - open/close 2597 (time 1790895536.73 total 185.96 last 14.17) - open/close 2756 (time 1790895546.73 total 195.97 last 15.89) - open/close 2912 (time 1790895557.43 total 206.66 last 14.59) - open/close 3072 (time 1790895567.43 total 216.67 last 15.99) - open/close 3215 (time 1790895577.77 total 227.00 last 13.84) - open/close 3294 (time 1790895587.78 total 237.01 last 7.89) - open/close 3464 (time 1790895597.78 total 247.02 last 16.99) - open/close 3661 (time 1790895607.90 total 257.13 last 19.48) - open/close 3942 (time 1790895617.91 total 267.14 last 28.07) - open/close 4119 (time 1790895628.34 total 277.57 last 16.98) - open/close 4310 (time 1790895638.35 total 287.58 last 19.07) - open/close 4419 (time 1790895648.36 total 297.59 last 10.89) - open/close 4506 (time 1790895658.55 total 307.78 last 8.54) - open/close 4633 (time 1790895668.58 total 317.81 last 12.67) - open/close 4730 (time 1790895679.07 total 328.30 last 9.24) - open/close 4855 (time 1790895689.32 total 338.55 last 12.20) - open/close 4907 (time 1790895699.32 total 348.55 last 5.20) - open/close 5047 (time 1790895709.88 total 359.11 last 13.26) - open/close 5120 (time 1790895719.88 total 369.11 last 7.30) - open/close 5184 (time 1790895730.72 total 379.95 last 5.90) - open/close 5333 (time 1790895742.59 total 391.82 last 12.56) - open/close 5413 (time 1790895754.03 total 403.26 last 6.99) - open/close 5601 (time 1790895767.02 total 416.25 last 14.47) - open/close 5831 (time 1790895777.95 total 427.18 last 21.05) - open/close 6079 (time 1790895787.96 total 437.19 last 24.77) - open/close 6248 (time 1790895798.31 total 447.55 last 16.32) - open/close 6424 (time 1790895808.61 total 457.84 last 17.10) - open/close 6494 (time 1790895818.61 total 467.84 last 7.00) - open/close 6715 (time 1790895828.62 total 477.85 last 22.09) - open/close 6862 (time 1790895839.15 total 488.38 last 13.96) - open/close 7023 (time 1790895850.05 total 499.28 last 14.77) - open/close 7124 (time 1790895860.40 total 509.64 last 9.75) - open/close 7311 (time 1790895871.23 total 520.46 last 17.28) - open/close 7588 (time 1790895881.24 total 530.47 last 27.67) - open/close 7705 (time 1790895891.24 total 540.47 last 11.70) - open/close 7898 (time 1790895901.54 total 550.78 last 18.73) - open/close 8034 (time 1790895911.57 total 560.80 last 13.57) - open/close 8111 (time 1790895921.84 total 571.08 last 7.49) - open/close 8245 (time 1790895932.47 total 581.70 last 12.61) - open/close 8345 (time 1790895943.07 total 592.30 last 9.44) - open/close 8468 (time 1790895954.10 total 603.33 last 11.15) - open/close 8533 (time 1790895967.22 total 616.45 last 4.95) - open/close 8624 (time 1790895979.65 total 628.88 last 7.32) - open/close 8758 (time 1790895990.13 total 639.36 last 12.79) - open/close 8968 (time 1790896001.00 total 650.23 last 19.32) - open/close 9006 (time 1790896012.03 total 661.26 last 3.45) - open/close 9124 (time 1790896022.04 total 671.27 last 11.78) - open/close 9284 (time 1790896033.09 total 682.32 last 14.48) - open/close 9471 (time 1790896043.09 total 692.32 last 18.69) - open/close 9645 (time 1790896053.10 total 702.33 last 17.39) - open/close 9819 (time 1790896063.78 total 713.01 last 16.28) - open/close 9932 (time 1790896074.28 total 723.51 last 10.77) - open/close 10000 (time 1790896081.38 total 730.61 last 9.57) - open/close 10043 (time 1790896091.97 total 741.20 last 4.06) - open/close 10101 (time 1790896103.02 total 752.26 last 5.25) - open/close 10212 (time 1790896113.84 total 763.07 last 10.27) - open/close 10357 (time 1790896123.96 total 773.19 last 14.33) - open/close 10471 (time 1790896133.99 total 783.22 last 11.36) - open/close 10525 (time 1790896144.23 total 793.46 last 5.27) - open/close 10680 (time 1790896154.66 total 803.89 last 14.86) - open/close 10749 (time 1790896164.91 total 814.14 last 6.74) - open/close 10828 (time 1790896175.33 total 824.56 last 7.58) - open/close 10919 (time 1790896186.29 total 835.52 last 8.30) - open/close 10959 (time 1790896199.02 total 848.26 last 3.14) - open/close 11073 (time 1790896210.02 total 859.25 last 10.37) - open/close 11272 (time 1790896220.21 total 869.44 last 19.53) - open/close 11393 (time 1790896230.22 total 879.45 last 12.08) - open/close 11478 (time 1790896240.65 total 889.88 last 8.15) - open/close 11536 (time 1790896251.57 total 900.80 last 5.31) - open/close 11736 (time 1790896262.25 total 911.48 last 18.73) - open/close 11885 (time 1790896273.32 total 922.55 last 13.46) - open/close 12016 (time 1790896283.59 total 932.82 last 12.75) - open/close 12075 (time 1790896294.51 total 943.74 last 5.41) - open/close 12168 (time 1790896305.58 total 954.81 last 8.40) - open/close 12315 (time 1790896315.58 total 964.82 last 14.70) - open/close 12414 (time 1790896326.24 total 975.47 last 9.29) - open/close 12567 (time 1790896336.50 total 985.73 last 14.91) - open/close 12707 (time 1790896346.81 total 996.05 last 13.57) - open/close 12863 (time 1790896356.81 total 1006.05 last 15.60) - open/close 12990 (time 1790896366.89 total 1016.12 last 12.60) - open/close 13076 (time 1790896376.90 total 1026.13 last 8.59) - open/close 13176 (time 1790896387.04 total 1036.27 last 9.86) - open/close 13285 (time 1790896397.05 total 1046.28 last 10.90) - open/close 13404 (time 1790896407.05 total 1056.28 last 11.89) - open/close 13494 (time 1790896417.58 total 1066.82 last 8.55) - open/close 13643 (time 1790896428.58 total 1077.81 last 13.56) - open/close 13771 (time 1790896438.68 total 1087.91 last 12.67) - open/close 13938 (time 1790896448.69 total 1097.92 last 16.68) - open/close 14206 (time 1790896459.39 total 1108.62 last 25.05) - open/close 14814 (time 1790896469.42 total 1118.65 last 60.59) - open/close 15448 (time 1790896479.42 total 1128.65 last 63.38) - open/close 16056 (time 1790896489.43 total 1138.67 last 60.73) - open/close 16590 (time 1790896499.45 total 1148.69 last 53.29) - open/close 17105 (time 1790896509.46 total 1158.69 last 51.47) - open/close 17822 (time 1790896519.48 total 1168.71 last 71.59) - open/close 18531 (time 1790896529.48 total 1178.71 last 70.87) - open/close 19149 (time 1790896539.49 total 1188.72 last 61.72) - open/close 19846 (time 1790896549.51 total 1198.74 last 69.60) - open/close 20561 (time 1790896559.52 total 1208.75 last 71.41) - open/close 21227 (time 1790896569.53 total 1218.76 last 66.57) - open/close 21908 (time 1790896579.53 total 1228.77 last 68.04) - open/close 22523 (time 1790896589.55 total 1238.78 last 61.40) - open/close 23230 (time 1790896599.56 total 1248.79 last 70.66) - open/close 23942 (time 1790896609.56 total 1258.79 last 71.20) - open/close 24770 (time 1790896619.56 total 1268.79 last 82.78) - open/close 25480 (time 1790896629.57 total 1278.80 last 70.91) - open/close 26114 (time 1790896639.58 total 1288.81 last 63.34) - open/close 26820 (time 1790896649.59 total 1298.82 last 70.54) - open/close 27427 (time 1790896659.59 total 1308.82 last 60.70) - open/close 28094 (time 1790896669.60 total 1318.83 last 66.63) - open/close 28717 (time 1790896679.62 total 1328.85 last 62.19) - open/close 29408 (time 1790896689.63 total 1338.86 last 69.02) - open/close 30000 (time 1790896699.04 total 1348.27 last 62.92) - open/close 30720 (time 1790896709.04 total 1358.27 last 71.97) - open/close 31448 (time 1790896719.05 total 1368.28 last 72.77) - open/close 32073 (time 1790896729.06 total 1378.29 last 62.43) - open/close 32732 (time 1790896739.06 total 1388.29 last 65.86) - open/close 33466 (time 1790896749.07 total 1398.30 last 73.36) - open/close 34110 (time 1790896759.08 total 1408.31 last 64.36) - open/close 34754 (time 1790896769.10 total 1418.33 last 64.27) - open/close 35460 (time 1790896779.13 total 1428.36 last 70.36) - open/close 36134 (time 1790896789.13 total 1438.37 last 67.36) - open/close 36746 (time 1790896799.14 total 1448.37 last 61.14) - open/close 37500 (time 1790896809.15 total 1458.38 last 75.35) - open/close 38201 (time 1790896819.15 total 1468.38 last 70.10) - open/close 38741 (time 1790896829.16 total 1478.39 last 53.97) - open/close 39343 (time 1790896839.16 total 1488.39 last 60.17) - open/close 40000 (time 1790896847.53 total 1496.76 last 78.49) - open/close 40637 (time 1790896857.55 total 1506.78 last 63.56) - open/close 41315 (time 1790896867.56 total 1516.79 last 67.74) - open/close 41929 (time 1790896877.56 total 1526.79 last 61.39) - open/close 42564 (time 1790896887.57 total 1536.80 last 63.49) - open/close 43124 (time 1790896897.57 total 1546.81 last 55.95) - open/close 43799 (time 1790896907.58 total 1556.81 last 67.49) - open/close 44450 (time 1790896917.59 total 1566.82 last 65.04) - open/close 45064 (time 1790896927.60 total 1576.83 last 61.33) - open/close 45683 (time 1790896937.62 total 1586.85 last 61.79) - open/close 46363 (time 1790896947.64 total 1596.87 last 67.86) - open/close 46959 (time 1790896957.65 total 1606.88 last 59.51) - open/close 47625 (time 1790896967.66 total 1616.89 last 66.52) - open/close 48189 (time 1790896977.67 total 1626.90 last 56.34) - open/close 48789 (time 1790896987.69 total 1636.92 last 59.88) - open/close 49331 (time 1790896997.70 total 1646.93 last 54.17) total: 50000 open/close in 1656.35 seconds: 30.19 ops/second - unlinked 0 (time 1790897011 ; total 0 ; last 0) - unlinked 10000 (time 1790897159 ; total 148 ; last 148) - unlinked 20000 (time 1790897303 ; total 292 ; last 144) - unlinked 30000 (time 1790897459 ; total 448 ; last 156) - unlinked 40000 (time 1790897617 ; total 606 ; last 158) total: 50000 unlinks in 770 seconds: 64.935066 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server stop mds service on oleg258-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server unloading modules via unload_modules_local on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing unload_modules_local oleg258-server: kmemleak disabled modules unloaded. pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2