== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:05:22 (1789304722) start mds service on oleg327-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 13 00:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: kmemleak disabled oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg327-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg327-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1789304805.64 total 10.26 last 18.52) - open/close 456 (time 1789304816.00 total 20.62 last 25.68) - open/close 697 (time 1789304826.02 total 30.64 last 24.05) - open/close 841 (time 1789304836.29 total 40.91 last 14.02) - open/close 1033 (time 1789304847.16 total 51.78 last 17.65) - open/close 1198 (time 1789304858.43 total 63.05 last 14.65) - open/close 1302 (time 1789304868.54 total 73.16 last 10.29) - open/close 1507 (time 1789304878.56 total 83.18 last 20.45) - open/close 1761 (time 1789304889.59 total 94.21 last 23.03) - open/close 1894 (time 1789304900.49 total 105.10 last 12.21) - open/close 2116 (time 1789304910.75 total 115.37 last 21.63) - open/close 2310 (time 1789304920.79 total 125.41 last 19.32) - open/close 2590 (time 1789304930.80 total 135.41 last 27.99) - open/close 2818 (time 1789304941.13 total 145.74 last 22.07) - open/close 3040 (time 1789304951.58 total 156.20 last 21.24) - open/close 3261 (time 1789304961.60 total 166.22 last 22.04) - open/close 3518 (time 1789304971.62 total 176.24 last 25.66) - open/close 3729 (time 1789304982.15 total 186.77 last 20.03) - open/close 3959 (time 1789304992.15 total 196.77 last 22.99) - open/close 4171 (time 1789305002.15 total 206.77 last 21.20) - open/close 4399 (time 1789305012.87 total 217.49 last 21.28) - open/close 4638 (time 1789305022.90 total 227.51 last 23.84) - open/close 4886 (time 1789305033.54 total 238.15 last 23.31) - open/close 5101 (time 1789305043.79 total 248.41 last 20.96) - open/close 5307 (time 1789305054.85 total 259.47 last 18.62) - open/close 5527 (time 1789305065.02 total 269.64 last 21.64) - open/close 5764 (time 1789305076.09 total 280.71 last 21.41) - open/close 5949 (time 1789305086.29 total 290.91 last 18.14) - open/close 6212 (time 1789305096.29 total 300.91 last 26.29) - open/close 6444 (time 1789305107.26 total 311.87 last 21.15) - open/close 6674 (time 1789305118.15 total 322.76 last 21.12) - open/close 6904 (time 1789305128.31 total 332.93 last 22.64) - open/close 7065 (time 1789305138.32 total 342.94 last 16.08) - open/close 7268 (time 1789305148.79 total 353.41 last 19.39) - open/close 7459 (time 1789305159.05 total 363.67 last 18.61) - open/close 7673 (time 1789305169.67 total 374.29 last 20.16) - open/close 7931 (time 1789305179.68 total 384.29 last 25.78) - open/close 8147 (time 1789305190.21 total 394.82 last 20.51) - open/close 8391 (time 1789305200.33 total 404.94 last 24.11) - open/close 8580 (time 1789305210.33 total 414.95 last 18.89) - open/close 8761 (time 1789305220.36 total 424.98 last 18.05) - open/close 8949 (time 1789305230.52 total 435.13 last 18.52) - open/close 9182 (time 1789305240.53 total 445.15 last 23.27) - open/close 9443 (time 1789305250.68 total 455.30 last 25.71) - open/close 9668 (time 1789305261.37 total 465.99 last 21.04) - open/close 9855 (time 1789305272.00 total 476.61 last 17.60) - open/close 10000 (time 1789305281.42 total 486.04 last 15.39) - open/close 10200 (time 1789305291.59 total 496.21 last 19.66) - open/close 10396 (time 1789305301.96 total 506.57 last 18.91) - open/close 10634 (time 1789305312.95 total 517.57 last 21.64) - open/close 10823 (time 1789305322.96 total 527.58 last 18.89) - open/close 11001 (time 1789305333.95 total 538.56 last 16.20) - open/close 11201 (time 1789305344.39 total 549.01 last 19.14) - open/close 11400 (time 1789305354.50 total 559.12 last 19.69) - open/close 11580 (time 1789305365.31 total 569.93 last 16.64) - open/close 11779 (time 1789305375.32 total 579.93 last 19.90) - open/close 11938 (time 1789305385.92 total 590.54 last 15.00) - open/close 12128 (time 1789305395.92 total 600.54 last 19.00) - open/close 12296 (time 1789305406.47 total 611.09 last 15.92) - open/close 12510 (time 1789305416.48 total 621.10 last 21.37) - open/close 12694 (time 1789305427.40 total 632.02 last 16.86) - open/close 12897 (time 1789305437.75 total 642.37 last 19.61) - open/close 13126 (time 1789305448.38 total 653.00 last 21.55) - open/close 13350 (time 1789305458.57 total 663.19 last 21.99) - open/close 13522 (time 1789305468.73 total 673.35 last 16.93) - open/close 13691 (time 1789305478.73 total 683.35 last 16.89) - open/close 13894 (time 1789305488.73 total 693.35 last 20.30) - open/close 14084 (time 1789305498.74 total 703.36 last 18.99) - open/close 14213 (time 1789305508.80 total 713.41 last 12.83) - open/close 14347 (time 1789305518.80 total 723.42 last 13.39) - open/close 14430 (time 1789305529.21 total 733.83 last 7.98) - open/close 14578 (time 1789305539.22 total 743.84 last 14.79) - open/close 14710 (time 1789305549.22 total 753.84 last 13.19) - open/close 14823 (time 1789305560.00 total 764.61 last 10.49) - open/close 14963 (time 1789305570.00 total 774.62 last 13.99) - open/close 15865 (time 1789305580.01 total 784.63 last 90.09) - open/close 16814 (time 1789305590.02 total 794.64 last 94.79) - open/close 17737 (time 1789305600.03 total 804.65 last 92.21) - open/close 18674 (time 1789305610.04 total 814.65 last 93.68) - open/close 19595 (time 1789305620.04 total 824.66 last 92.03) - open/close 20000 (time 1789305624.72 total 829.33 last 86.68) - open/close 20950 (time 1789305634.72 total 839.34 last 94.97) - open/close 21840 (time 1789305644.72 total 849.34 last 88.97) - open/close 22736 (time 1789305654.72 total 859.34 last 89.59) - open/close 23885 (time 1789305664.73 total 869.35 last 114.84) - open/close 25042 (time 1789305674.73 total 879.35 last 115.68) - open/close 26002 (time 1789305684.73 total 889.35 last 95.98) - open/close 26931 (time 1789305694.75 total 899.36 last 92.78) - open/close 27888 (time 1789305704.76 total 909.38 last 95.54) - open/close 28745 (time 1789305714.77 total 919.39 last 85.67) - open/close 29691 (time 1789305724.77 total 929.39 last 94.55) - open/close 30000 (time 1789305727.77 total 932.39 last 102.97) - open/close 31107 (time 1789305737.78 total 942.40 last 110.66) - open/close 32070 (time 1789305747.78 total 952.40 last 96.24) - open/close 33103 (time 1789305757.79 total 962.40 last 103.28) - open/close 34130 (time 1789305767.79 total 972.41 last 102.62) - open/close 35147 (time 1789305777.80 total 982.42 last 101.63) - open/close 36001 (time 1789305787.81 total 992.43 last 85.33) - open/close 36864 (time 1789305797.81 total 1002.43 last 86.28) - open/close 37843 (time 1789305807.82 total 1012.43 last 97.87) - open/close 38887 (time 1789305817.82 total 1022.44 last 104.32) - open/close 40000 (time 1789305827.15 total 1031.77 last 119.30) - open/close 41245 (time 1789305837.16 total 1041.77 last 124.44) - open/close 42313 (time 1789305847.16 total 1051.77 last 106.79) - open/close 43215 (time 1789305857.16 total 1061.78 last 90.13) - open/close 44230 (time 1789305867.17 total 1071.79 last 101.45) - open/close 45121 (time 1789305877.17 total 1081.79 last 89.06) - open/close 45979 (time 1789305887.18 total 1091.80 last 85.75) - open/close 46935 (time 1789305897.19 total 1101.81 last 95.49) - open/close 47843 (time 1789305907.20 total 1111.82 last 90.71) - open/close 48726 (time 1789305917.21 total 1121.83 last 88.24) - open/close 49546 (time 1789305927.21 total 1131.83 last 81.97) total: 50000 open/close in 1137.12 seconds: 43.97 ops/second - unlinked 0 (time 1789305935 ; total 0 ; last 0) - unlinked 10000 (time 1789306033 ; total 98 ; last 98) - unlinked 20000 (time 1789306105 ; total 170 ; last 72) - unlinked 30000 (time 1789306159 ; total 224 ; last 54) - unlinked 40000 (time 1789306208 ; total 273 ; last 49) total: 50000 unlinks in 321 seconds: 155.763245 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: kmemleak disabled modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2