== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:24:17 (1788441857) start mds service on oleg624-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: kmemleak disabled oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg624-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg624-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg624-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg624-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 261 (time 1788441943.29 total 10.82 last 24.13) - open/close 531 (time 1788441953.61 total 21.14 last 26.15) - open/close 708 (time 1788441964.28 total 31.81 last 16.60) - open/close 960 (time 1788441974.90 total 42.43 last 23.72) - open/close 1173 (time 1788441985.08 total 52.61 last 20.92) - open/close 1395 (time 1788441995.26 total 62.79 last 21.81) - open/close 1630 (time 1788442005.27 total 72.80 last 23.48) - open/close 1858 (time 1788442015.29 total 82.82 last 22.75) - open/close 2083 (time 1788442025.46 total 92.99 last 22.14) - open/close 2346 (time 1788442035.48 total 103.01 last 26.22) - open/close 2572 (time 1788442045.49 total 113.02 last 22.59) - open/close 2791 (time 1788442056.75 total 124.28 last 19.45) - open/close 2847 (time 1788442067.14 total 134.66 last 5.39) - open/close 3072 (time 1788442077.16 total 144.69 last 22.44) - open/close 3338 (time 1788442087.18 total 154.71 last 26.56) - open/close 3594 (time 1788442097.33 total 164.86 last 25.21) - open/close 3842 (time 1788442107.90 total 175.43 last 23.47) - open/close 4089 (time 1788442118.21 total 185.74 last 23.95) - open/close 4357 (time 1788442129.23 total 196.76 last 24.32) - open/close 4544 (time 1788442139.24 total 206.77 last 18.68) - open/close 4774 (time 1788442149.36 total 216.89 last 22.72) - open/close 4983 (time 1788442159.68 total 227.21 last 20.25) - open/close 5202 (time 1788442169.68 total 237.21 last 21.90) - open/close 5406 (time 1788442179.70 total 247.22 last 20.38) - open/close 5619 (time 1788442189.70 total 257.23 last 21.30) - open/close 5826 (time 1788442199.74 total 267.27 last 20.61) - open/close 6078 (time 1788442210.29 total 277.82 last 23.88) - open/close 6348 (time 1788442220.91 total 288.44 last 25.44) - open/close 6575 (time 1788442231.03 total 298.56 last 22.41) - open/close 6809 (time 1788442241.20 total 308.73 last 23.01) - open/close 7035 (time 1788442251.33 total 318.85 last 22.33) - open/close 7285 (time 1788442261.50 total 329.03 last 24.56) - open/close 7561 (time 1788442271.93 total 339.46 last 26.47) - open/close 7789 (time 1788442281.95 total 349.48 last 22.76) - open/close 8060 (time 1788442292.80 total 360.33 last 24.98) - open/close 8322 (time 1788442303.48 total 371.01 last 24.54) - open/close 8607 (time 1788442313.48 total 381.01 last 28.49) - open/close 8790 (time 1788442324.15 total 391.68 last 17.14) - open/close 9091 (time 1788442335.11 total 402.64 last 27.47) - open/close 9346 (time 1788442345.66 total 413.19 last 24.17) - open/close 9575 (time 1788442355.68 total 423.21 last 22.85) - open/close 9751 (time 1788442365.70 total 433.23 last 17.57) - open/close 10000 (time 1788442375.71 total 443.24 last 24.86) - open/close 10214 (time 1788442385.73 total 453.26 last 21.36) - open/close 10460 (time 1788442396.48 total 464.01 last 22.89) - open/close 10675 (time 1788442406.64 total 474.17 last 21.16) - open/close 10897 (time 1788442416.88 total 484.41 last 21.68) - open/close 11164 (time 1788442427.83 total 495.36 last 24.38) - open/close 11428 (time 1788442437.88 total 505.41 last 26.26) - open/close 11709 (time 1788442448.64 total 516.17 last 26.13) - open/close 11943 (time 1788442459.53 total 527.06 last 21.49) - open/close 12206 (time 1788442469.53 total 537.06 last 26.29) - open/close 12400 (time 1788442479.53 total 547.06 last 19.40) - open/close 12633 (time 1788442490.31 total 557.84 last 21.62) - open/close 12878 (time 1788442500.33 total 567.86 last 24.46) - open/close 13090 (time 1788442510.51 total 578.04 last 20.81) - open/close 13332 (time 1788442520.82 total 588.35 last 23.47) - open/close 13551 (time 1788442531.12 total 598.65 last 21.27) - open/close 13780 (time 1788442541.44 total 608.96 last 22.20) - open/close 14025 (time 1788442551.44 total 618.97 last 24.48) - open/close 14243 (time 1788442561.95 total 629.48 last 20.75) - open/close 14503 (time 1788442572.80 total 640.33 last 23.96) - open/close 14731 (time 1788442583.41 total 650.94 last 21.49) - open/close 15008 (time 1788442593.41 total 660.94 last 27.69) - open/close 15239 (time 1788442603.42 total 670.95 last 23.10) - open/close 15483 (time 1788442614.22 total 681.75 last 22.58) - open/close 15751 (time 1788442624.27 total 691.80 last 26.69) - open/close 15977 (time 1788442636.92 total 704.45 last 17.86) - open/close 16188 (time 1788442646.93 total 714.46 last 21.07) - open/close 16380 (time 1788442657.26 total 724.79 last 18.59) - open/close 16604 (time 1788442667.27 total 734.80 last 22.36) - open/close 16812 (time 1788442677.55 total 745.08 last 20.24) - open/close 17037 (time 1788442687.93 total 755.46 last 21.68) - open/close 17239 (time 1788442698.46 total 765.99 last 19.18) - open/close 17346 (time 1788442709.45 total 776.98 last 9.73) - open/close 17606 (time 1788442721.93 total 789.46 last 20.83) - open/close 17818 (time 1788442732.32 total 799.85 last 20.42) - open/close 18004 (time 1788442743.17 total 810.70 last 17.14) - open/close 18139 (time 1788442753.47 total 821.00 last 13.11) - open/close 18396 (time 1788442763.77 total 831.29 last 24.97) - open/close 18644 (time 1788442773.77 total 841.30 last 24.79) - open/close 18811 (time 1788442784.05 total 851.58 last 16.24) - open/close 19079 (time 1788442794.05 total 861.58 last 26.80) - open/close 19310 (time 1788442804.72 total 872.25 last 21.65) - open/close 19565 (time 1788442814.74 total 882.27 last 25.46) - open/close 19793 (time 1788442824.77 total 892.29 last 22.73) - open/close 19977 (time 1788442834.78 total 902.31 last 18.38) - open/close 20000 (time 1788442837.41 total 904.94 last 8.73) - open/close 20201 (time 1788442847.76 total 915.29 last 19.41) - open/close 20863 (time 1788442857.78 total 925.31 last 66.12) - open/close 21512 (time 1788442867.78 total 935.31 last 64.88) - open/close 22190 (time 1788442877.78 total 945.31 last 67.80) - open/close 22792 (time 1788442887.78 total 955.31 last 60.19) - open/close 23492 (time 1788442897.79 total 965.31 last 69.98) - open/close 24158 (time 1788442907.80 total 975.32 last 66.53) - open/close 24831 (time 1788442917.80 total 985.33 last 67.25) - open/close 25522 (time 1788442927.82 total 995.35 last 68.95) - open/close 26191 (time 1788442937.83 total 1005.36 last 66.84) - open/close 26848 (time 1788442947.84 total 1015.36 last 65.68) - open/close 27549 (time 1788442957.84 total 1025.37 last 70.08) - open/close 28224 (time 1788442967.84 total 1035.37 last 67.47) - open/close 28895 (time 1788442977.85 total 1045.38 last 67.02) - open/close 29497 (time 1788442987.86 total 1055.39 last 60.17) - open/close 30000 (time 1788442995.24 total 1062.77 last 68.19) - open/close 30714 (time 1788443005.24 total 1072.77 last 71.35) - open/close 31382 (time 1788443015.28 total 1082.81 last 66.57) - open/close 32051 (time 1788443025.28 total 1092.81 last 66.89) - open/close 32710 (time 1788443035.29 total 1102.82 last 65.82) - open/close 33448 (time 1788443045.30 total 1112.83 last 73.73) - open/close 34144 (time 1788443055.31 total 1122.84 last 69.58) - open/close 34866 (time 1788443065.31 total 1132.84 last 72.15) - open/close 35512 (time 1788443075.32 total 1142.85 last 64.53) - open/close 36206 (time 1788443085.33 total 1152.86 last 69.35) - open/close 36858 (time 1788443095.35 total 1162.88 last 65.10) - open/close 37518 (time 1788443105.38 total 1172.91 last 65.80) - open/close 38168 (time 1788443115.39 total 1182.92 last 64.94) - open/close 38798 (time 1788443125.39 total 1192.92 last 62.97) - open/close 39463 (time 1788443135.39 total 1202.92 last 66.50) - open/close 40000 (time 1788443142.80 total 1210.33 last 72.50) - open/close 40609 (time 1788443152.81 total 1220.34 last 60.84) - open/close 41256 (time 1788443162.82 total 1230.35 last 64.64) - open/close 41888 (time 1788443172.82 total 1240.35 last 63.19) - open/close 42540 (time 1788443182.83 total 1250.35 last 65.16) - open/close 43259 (time 1788443192.84 total 1260.37 last 71.79) - open/close 43896 (time 1788443202.89 total 1270.42 last 63.38) - open/close 44585 (time 1788443212.90 total 1280.43 last 68.85) - open/close 45298 (time 1788443222.91 total 1290.44 last 71.17) - open/close 45987 (time 1788443232.92 total 1300.45 last 68.85) - open/close 46699 (time 1788443242.92 total 1310.45 last 71.19) - open/close 47357 (time 1788443252.93 total 1320.46 last 65.76) - open/close 48067 (time 1788443262.95 total 1330.47 last 70.89) - open/close 48789 (time 1788443272.97 total 1340.50 last 72.00) - open/close 49476 (time 1788443282.98 total 1350.51 last 68.68) total: 50000 open/close in 1357.67 seconds: 36.83 ops/second - unlinked 0 (time 1788443294 ; total 0 ; last 0) - unlinked 10000 (time 1788443448 ; total 154 ; last 154) - unlinked 20000 (time 1788443601 ; total 307 ; last 153) - unlinked 30000 (time 1788443763 ; total 469 ; last 162) - unlinked 40000 (time 1788443914 ; total 620 ; last 151) total: 50000 unlinks in 787 seconds: 63.532402 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg624-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server unloading modules via unload_modules_local on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing unload_modules_local oleg624-server: kmemleak disabled modules unloaded. pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2