== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:23:01 (1788751381) start mds service on oleg242-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: kmemleak disabled oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg242-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 201 (time 1788751461.20 total 10.87 last 18.49) - open/close 471 (time 1788751471.44 total 21.11 last 26.36) - open/close 747 (time 1788751482.18 total 31.85 last 25.70) - open/close 943 (time 1788751492.44 total 42.11 last 19.10) - open/close 1200 (time 1788751502.73 total 52.40 last 24.98) - open/close 1443 (time 1788751512.97 total 62.64 last 23.74) - open/close 1626 (time 1788751524.04 total 73.71 last 16.53) - open/close 1808 (time 1788751534.67 total 84.34 last 17.13) - open/close 2036 (time 1788751545.66 total 95.34 last 20.73) - open/close 2316 (time 1788751556.43 total 106.10 last 26.01) - open/close 2546 (time 1788751566.44 total 116.11 last 22.97) - open/close 2787 (time 1788751577.48 total 127.15 last 21.84) - open/close 3002 (time 1788751587.49 total 137.17 last 21.47) - open/close 3234 (time 1788751598.35 total 148.02 last 21.38) - open/close 3432 (time 1788751608.51 total 158.19 last 19.48) - open/close 3653 (time 1788751618.54 total 168.21 last 22.05) - open/close 3852 (time 1788751629.15 total 178.82 last 18.75) - open/close 4079 (time 1788751639.19 total 188.86 last 22.62) - open/close 4313 (time 1788751649.19 total 198.86 last 23.40) - open/close 4550 (time 1788751659.19 total 208.86 last 23.70) - open/close 4774 (time 1788751669.19 total 218.86 last 22.39) - open/close 5010 (time 1788751680.08 total 229.75 last 21.68) - open/close 5255 (time 1788751690.99 total 240.66 last 22.46) - open/close 5459 (time 1788751701.38 total 251.05 last 19.63) - open/close 5702 (time 1788751712.13 total 261.81 last 22.60) - open/close 5911 (time 1788751722.47 total 272.14 last 20.22) - open/close 6089 (time 1788751732.88 total 282.55 last 17.10) - open/close 6276 (time 1788751742.97 total 292.65 last 18.52) - open/close 6501 (time 1788751752.98 total 302.65 last 22.49) - open/close 6672 (time 1788751763.28 total 312.95 last 16.60) - open/close 6890 (time 1788751775.95 total 325.62 last 17.21) - open/close 7101 (time 1788751786.52 total 336.19 last 19.96) - open/close 7338 (time 1788751796.55 total 346.22 last 23.64) - open/close 7584 (time 1788751807.49 total 357.16 last 22.47) - open/close 7816 (time 1788751817.50 total 367.17 last 23.19) - open/close 8021 (time 1788751827.79 total 377.46 last 19.91) - open/close 8258 (time 1788751838.04 total 387.72 last 23.11) - open/close 8483 (time 1788751849.03 total 398.70 last 20.48) - open/close 8729 (time 1788751859.86 total 409.53 last 22.72) - open/close 8971 (time 1788751869.87 total 419.54 last 24.17) - open/close 9195 (time 1788751879.88 total 429.56 last 22.37) - open/close 9397 (time 1788751889.98 total 439.65 last 20.01) - open/close 9626 (time 1788751899.98 total 449.65 last 22.90) - open/close 9828 (time 1788751909.98 total 459.65 last 20.20) - open/close 10000 (time 1788751916.81 total 466.48 last 25.19) - open/close 10182 (time 1788751927.05 total 476.72 last 17.78) - open/close 10356 (time 1788751938.00 total 487.67 last 15.90) - open/close 10579 (time 1788751948.69 total 498.36 last 20.86) - open/close 10809 (time 1788751959.10 total 508.77 last 22.08) - open/close 10990 (time 1788751970.00 total 519.67 last 16.62) - open/close 11262 (time 1788751980.01 total 529.68 last 27.17) - open/close 11465 (time 1788751990.48 total 540.16 last 19.37) - open/close 11629 (time 1788752000.50 total 550.17 last 16.38) - open/close 11833 (time 1788752010.72 total 560.39 last 19.96) - open/close 12047 (time 1788752021.57 total 571.24 last 19.73) - open/close 12286 (time 1788752032.33 total 582.00 last 22.21) - open/close 12505 (time 1788752042.70 total 592.37 last 21.12) - open/close 12681 (time 1788752053.52 total 603.19 last 16.27) - open/close 12865 (time 1788752064.33 total 614.01 last 17.01) - open/close 13044 (time 1788752074.34 total 624.01 last 17.89) - open/close 13216 (time 1788752085.01 total 634.68 last 16.12) - open/close 13399 (time 1788752096.09 total 645.76 last 16.51) - open/close 13627 (time 1788752106.24 total 655.91 last 22.46) - open/close 13772 (time 1788752116.80 total 666.48 last 13.72) - open/close 13970 (time 1788752126.92 total 676.59 last 19.57) - open/close 14199 (time 1788752137.93 total 687.60 last 20.80) - open/close 14412 (time 1788752148.36 total 698.03 last 20.41) - open/close 14538 (time 1788752158.59 total 708.27 last 12.31) - open/close 14762 (time 1788752168.60 total 718.28 last 22.38) - open/close 14967 (time 1788752178.62 total 728.29 last 20.47) - open/close 15143 (time 1788752189.44 total 739.11 last 16.26) - open/close 15392 (time 1788752199.44 total 749.12 last 24.89) - open/close 16297 (time 1788752209.46 total 759.13 last 90.40) - open/close 17239 (time 1788752219.46 total 769.14 last 94.12) - open/close 18239 (time 1788752229.47 total 779.14 last 99.99) - open/close 19179 (time 1788752239.47 total 789.15 last 93.91) - open/close 20000 (time 1788752247.47 total 797.15 last 102.64) - open/close 20959 (time 1788752257.48 total 807.15 last 95.85) - open/close 21986 (time 1788752267.50 total 817.17 last 102.53) - open/close 22996 (time 1788752277.50 total 827.18 last 100.93) - open/close 23980 (time 1788752287.52 total 837.20 last 98.20) - open/close 25023 (time 1788752297.53 total 847.20 last 104.22) - open/close 26003 (time 1788752307.53 total 857.21 last 97.98) - open/close 27056 (time 1788752317.54 total 867.21 last 105.28) - open/close 27991 (time 1788752327.54 total 877.21 last 93.47) - open/close 29037 (time 1788752337.54 total 887.21 last 104.57) - open/close 29978 (time 1788752347.55 total 897.22 last 94.02) - open/close 30964 (time 1788752357.55 total 907.22 last 98.57) - open/close 31934 (time 1788752367.56 total 917.23 last 96.91) - open/close 32995 (time 1788752377.57 total 927.24 last 106.06) - open/close 33910 (time 1788752387.57 total 937.24 last 91.44) - open/close 34854 (time 1788752397.58 total 947.25 last 94.31) - open/close 35913 (time 1788752407.59 total 957.26 last 105.85) - open/close 36862 (time 1788752417.59 total 967.26 last 94.88) - open/close 37718 (time 1788752427.59 total 977.26 last 85.58) - open/close 38542 (time 1788752437.60 total 987.27 last 82.31) - open/close 39437 (time 1788752447.61 total 997.28 last 89.42) - open/close 40000 (time 1788752453.23 total 1002.90 last 100.26) - open/close 40950 (time 1788752463.23 total 1012.90 last 94.98) - open/close 42076 (time 1788752473.23 total 1022.90 last 112.57) - open/close 43103 (time 1788752483.24 total 1032.91 last 102.66) - open/close 44183 (time 1788752493.26 total 1042.93 last 107.79) - open/close 45254 (time 1788752503.26 total 1052.93 last 107.05) - open/close 46335 (time 1788752513.29 total 1062.96 last 107.75) - open/close 47390 (time 1788752523.30 total 1072.97 last 105.43) - open/close 48408 (time 1788752533.30 total 1082.97 last 101.78) - open/close 49468 (time 1788752543.31 total 1092.98 last 105.93) total: 50000 open/close in 1099.24 seconds: 45.49 ops/second - unlinked 0 (time 1788752552 ; total 0 ; last 0) - unlinked 10000 (time 1788752726 ; total 174 ; last 174) - unlinked 20000 (time 1788752835 ; total 283 ; last 109) - unlinked 30000 (time 1788752925 ; total 373 ; last 90) - unlinked 40000 (time 1788753011 ; total 459 ; last 86) total: 50000 unlinks in 548 seconds: 91.240875 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: kmemleak disabled modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2