== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:33:01 (1790789581) start mds service on oleg215-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 30 05:20 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: kmemleak disabled oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg215-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg215-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 108 (time 1790789708.31 total 10.17 last 10.62) - open/close 401 (time 1790789718.31 total 20.18 last 29.29) - open/close 592 (time 1790789728.43 total 30.30 last 18.88) - open/close 800 (time 1790789738.99 total 40.85 last 19.71) - open/close 980 (time 1790789749.98 total 51.84 last 16.38) - open/close 1143 (time 1790789760.68 total 62.54 last 15.23) - open/close 1262 (time 1790789770.69 total 72.55 last 11.89) - open/close 1422 (time 1790789781.46 total 83.32 last 14.86) - open/close 1521 (time 1790789792.21 total 94.07 last 9.21) - open/close 1731 (time 1790789802.41 total 104.27 last 20.59) - open/close 1972 (time 1790789813.33 total 115.19 last 22.07) - open/close 2121 (time 1790789823.60 total 125.47 last 14.50) - open/close 2290 (time 1790789835.07 total 136.93 last 14.74) - open/close 2417 (time 1790789845.16 total 147.03 last 12.59) - open/close 2631 (time 1790789855.61 total 157.48 last 20.48) - open/close 2825 (time 1790789866.29 total 168.15 last 18.17) - open/close 3036 (time 1790789876.31 total 178.17 last 21.05) - open/close 3216 (time 1790789886.77 total 188.64 last 17.20) - open/close 3417 (time 1790789896.99 total 198.85 last 19.67) - open/close 3622 (time 1790789907.72 total 209.58 last 19.11) - open/close 3836 (time 1790789918.02 total 219.89 last 20.77) - open/close 4073 (time 1790789928.04 total 229.90 last 23.67) - open/close 4291 (time 1790789938.93 total 240.80 last 20.00) - open/close 4388 (time 1790789949.58 total 251.44 last 9.11) - open/close 4500 (time 1790789960.45 total 262.31 last 10.30) - open/close 4643 (time 1790789971.39 total 273.25 last 13.07) - open/close 4804 (time 1790789981.41 total 283.28 last 16.06) - open/close 4930 (time 1790789993.47 total 295.33 last 10.45) - open/close 5096 (time 1790790003.48 total 305.34 last 16.58) - open/close 5199 (time 1790790014.32 total 316.18 last 9.51) - open/close 5404 (time 1790790024.32 total 326.19 last 20.49) - open/close 5522 (time 1790790034.43 total 336.29 last 11.68) - open/close 5689 (time 1790790046.64 total 348.50 last 13.68) - open/close 5805 (time 1790790056.65 total 358.51 last 11.59) - open/close 5898 (time 1790790066.68 total 368.54 last 9.27) - open/close 6007 (time 1790790076.74 total 378.60 last 10.84) - open/close 6226 (time 1790790088.88 total 390.74 last 18.04) - open/close 6352 (time 1790790099.37 total 401.24 last 12.01) - open/close 6576 (time 1790790109.38 total 411.24 last 22.39) - open/close 6831 (time 1790790119.38 total 421.24 last 25.49) - open/close 7064 (time 1790790129.83 total 431.70 last 22.29) - open/close 7257 (time 1790790139.84 total 441.71 last 19.28) - open/close 7405 (time 1790790150.65 total 452.52 last 13.69) - open/close 7628 (time 1790790160.66 total 462.52 last 22.29) - open/close 7794 (time 1790790171.51 total 473.37 last 15.30) - open/close 8038 (time 1790790181.51 total 483.38 last 24.38) - open/close 8189 (time 1790790191.98 total 493.84 last 14.43) - open/close 8269 (time 1790790202.35 total 504.21 last 7.72) - open/close 8362 (time 1790790214.38 total 516.24 last 7.73) - open/close 8461 (time 1790790224.39 total 526.26 last 9.89) - open/close 8583 (time 1790790235.39 total 537.25 last 11.10) - open/close 8780 (time 1790790246.20 total 548.06 last 18.22) - open/close 8838 (time 1790790256.24 total 558.11 last 5.78) - open/close 8986 (time 1790790266.88 total 568.75 last 13.91) - open/close 9163 (time 1790790277.10 total 578.96 last 17.32) - open/close 9354 (time 1790790287.10 total 588.97 last 19.09) - open/close 9527 (time 1790790298.12 total 599.98 last 15.71) - open/close 9663 (time 1790790308.12 total 609.99 last 13.60) - open/close 9862 (time 1790790318.81 total 620.68 last 18.61) - open/close 10000 (time 1790790325.84 total 627.71 last 19.63) - open/close 10138 (time 1790790336.25 total 638.11 last 13.27) - open/close 10273 (time 1790790346.62 total 648.48 last 13.02) - open/close 10368 (time 1790790356.64 total 658.50 last 9.48) - open/close 10580 (time 1790790367.67 total 669.53 last 19.22) - open/close 10837 (time 1790790380.03 total 681.89 last 20.80) - open/close 10993 (time 1790790390.32 total 692.19 last 15.15) - open/close 11235 (time 1790790400.76 total 702.63 last 23.17) - open/close 11491 (time 1790790410.78 total 712.64 last 25.57) - open/close 11715 (time 1790790423.87 total 725.73 last 17.11) - open/close 11921 (time 1790790434.71 total 736.57 last 19.01) - open/close 12158 (time 1790790444.71 total 746.57 last 23.69) - open/close 12323 (time 1790790454.73 total 756.60 last 16.46) - open/close 12442 (time 1790790465.41 total 767.28 last 11.14) - open/close 12527 (time 1790790476.62 total 778.49 last 7.58) - open/close 12633 (time 1790790486.64 total 788.50 last 10.58) - open/close 12824 (time 1790790496.67 total 798.53 last 19.04) - open/close 12977 (time 1790790507.51 total 809.38 last 14.11) - open/close 13136 (time 1790790517.52 total 819.38 last 15.89) - open/close 13270 (time 1790790528.33 total 830.20 last 12.39) - open/close 13291 (time 1790790538.34 total 840.20 last 2.10) - open/close 13407 (time 1790790548.40 total 850.27 last 11.53) - open/close 13501 (time 1790790559.24 total 861.10 last 8.68) - open/close 13564 (time 1790790569.24 total 871.10 last 6.30) - open/close 13751 (time 1790790579.64 total 881.50 last 17.98) - open/close 13856 (time 1790790590.65 total 892.51 last 9.54) - open/close 13975 (time 1790790600.67 total 902.54 last 11.87) - open/close 14092 (time 1790790610.69 total 912.56 last 11.68) - open/close 14221 (time 1790790621.33 total 923.19 last 12.13) - open/close 14370 (time 1790790631.81 total 933.67 last 14.22) - open/close 14557 (time 1790790641.82 total 943.68 last 18.68) - open/close 14720 (time 1790790651.82 total 953.68 last 16.29) - open/close 14945 (time 1790790661.85 total 963.71 last 22.44) - open/close 15170 (time 1790790671.85 total 973.71 last 22.50) - open/close 15410 (time 1790790681.86 total 983.72 last 23.98) - open/close 15562 (time 1790790692.49 total 994.35 last 14.30) - open/close 15634 (time 1790790702.61 total 1004.47 last 7.11) - open/close 15785 (time 1790790713.25 total 1015.11 last 14.19) - open/close 15898 (time 1790790723.53 total 1025.40 last 10.99) - open/close 16018 (time 1790790733.54 total 1035.40 last 11.99) - open/close 16107 (time 1790790743.76 total 1045.62 last 8.71) - open/close 16329 (time 1790790753.77 total 1055.63 last 22.18) - open/close 16408 (time 1790790763.78 total 1065.64 last 7.89) - open/close 16517 (time 1790790773.78 total 1075.64 last 10.90) - open/close 16608 (time 1790790784.54 total 1086.40 last 8.45) - open/close 16728 (time 1790790795.00 total 1096.86 last 11.47) - open/close 17031 (time 1790790805.00 total 1106.86 last 30.29) - open/close 17658 (time 1790790815.01 total 1116.87 last 62.65) - open/close 18278 (time 1790790825.02 total 1126.88 last 61.94) - open/close 18923 (time 1790790835.06 total 1136.93 last 64.22) - open/close 19618 (time 1790790845.07 total 1146.93 last 69.46) - open/close 20000 (time 1790790849.71 total 1151.57 last 82.33) - open/close 20713 (time 1790790859.72 total 1161.58 last 71.24) - open/close 21382 (time 1790790869.73 total 1171.59 last 66.82) - open/close 21925 (time 1790790879.74 total 1181.60 last 54.26) - open/close 22503 (time 1790790889.76 total 1191.63 last 57.64) - open/close 23158 (time 1790790899.77 total 1201.64 last 65.44) - open/close 23896 (time 1790790909.78 total 1211.64 last 73.76) - open/close 24527 (time 1790790919.79 total 1221.65 last 63.04) - open/close 25214 (time 1790790929.79 total 1231.65 last 68.68) - open/close 25929 (time 1790790939.79 total 1241.65 last 71.49) - open/close 26506 (time 1790790949.81 total 1251.67 last 57.60) - open/close 27001 (time 1790790959.81 total 1261.67 last 49.49) - open/close 27538 (time 1790790969.82 total 1271.68 last 53.67) - open/close 28127 (time 1790790979.82 total 1281.68 last 58.90) - open/close 28714 (time 1790790989.83 total 1291.70 last 58.60) - open/close 29309 (time 1790790999.89 total 1301.75 last 59.19) - open/close 29752 (time 1790791009.89 total 1311.75 last 44.28) - open/close 30000 (time 1790791013.92 total 1315.78 last 61.58) - open/close 30788 (time 1790791023.92 total 1325.78 last 78.78) - open/close 31595 (time 1790791033.93 total 1335.79 last 80.63) - open/close 32179 (time 1790791043.94 total 1345.80 last 58.36) - open/close 32859 (time 1790791053.94 total 1355.80 last 67.98) - open/close 33546 (time 1790791063.98 total 1365.84 last 68.46) - open/close 34342 (time 1790791073.98 total 1375.85 last 79.53) - open/close 35094 (time 1790791083.99 total 1385.85 last 75.19) - open/close 35812 (time 1790791093.99 total 1395.85 last 71.77) - open/close 36502 (time 1790791103.99 total 1405.86 last 68.98) - open/close 37176 (time 1790791114.00 total 1415.86 last 67.36) - open/close 37801 (time 1790791124.00 total 1425.87 last 62.47) - open/close 38436 (time 1790791134.01 total 1435.87 last 63.46) - open/close 38903 (time 1790791144.02 total 1445.88 last 46.64) - open/close 39411 (time 1790791154.03 total 1455.89 last 50.77) - open/close 39939 (time 1790791164.04 total 1465.90 last 52.73) - open/close 40526 (time 1790791174.05 total 1475.92 last 58.62) - open/close 41073 (time 1790791184.06 total 1485.92 last 54.66) - open/close 41641 (time 1790791194.07 total 1495.93 last 56.77) - open/close 42119 (time 1790791204.09 total 1505.96 last 47.67) - open/close 42600 (time 1790791214.11 total 1515.97 last 48.04) - open/close 43265 (time 1790791224.12 total 1525.99 last 66.39) - open/close 43926 (time 1790791234.14 total 1536.00 last 66.01) - open/close 44576 (time 1790791244.15 total 1546.01 last 64.94) - open/close 45269 (time 1790791254.15 total 1556.02 last 69.25) - open/close 45991 (time 1790791264.16 total 1566.02 last 72.17) - open/close 46531 (time 1790791274.19 total 1576.05 last 53.84) - open/close 46983 (time 1790791284.21 total 1586.08 last 45.09) - open/close 47448 (time 1790791294.22 total 1596.08 last 46.49) - open/close 47944 (time 1790791304.22 total 1606.09 last 49.57) - open/close 48503 (time 1790791314.23 total 1616.09 last 55.85) - open/close 49094 (time 1790791324.24 total 1626.10 last 59.04) - open/close 49702 (time 1790791334.27 total 1636.13 last 60.64) total: 50000 open/close in 1641.94 seconds: 30.45 ops/second - unlinked 0 (time 1790791345 ; total 0 ; last 0) - unlinked 10000 (time 1790791502 ; total 157 ; last 157) - unlinked 20000 (time 1790791662 ; total 317 ; last 160) - unlinked 30000 (time 1790791818 ; total 473 ; last 156) - unlinked 40000 (time 1790791986 ; total 641 ; last 168) total: 50000 unlinks in 812 seconds: 61.576355 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: kmemleak disabled modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2