== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:35:51 (1788802551) start mds service on oleg306-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 7 12:38 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: kmemleak disabled oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg306-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg306-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg306-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1788802635.40 total 10.32 last 22.09) - open/close 491 (time 1788802646.16 total 21.08 last 24.44) - open/close 791 (time 1788802656.18 total 31.10 last 29.95) - open/close 1013 (time 1788802666.20 total 41.12 last 22.15) - open/close 1302 (time 1788802677.19 total 52.11 last 26.30) - open/close 1572 (time 1788802687.50 total 62.42 last 26.19) - open/close 1794 (time 1788802697.51 total 72.43 last 22.19) - open/close 1971 (time 1788802707.83 total 82.75 last 17.14) - open/close 2076 (time 1788802720.34 total 95.26 last 8.39) - open/close 2215 (time 1788802730.35 total 105.27 last 13.89) - open/close 2337 (time 1788802740.37 total 115.29 last 12.18) - open/close 2524 (time 1788802752.21 total 127.13 last 15.79) - open/close 2761 (time 1788802762.21 total 137.13 last 23.69) - open/close 2945 (time 1788802772.48 total 147.40 last 17.92) - open/close 3207 (time 1788802782.49 total 157.41 last 26.18) - open/close 3423 (time 1788802792.61 total 167.53 last 21.35) - open/close 3656 (time 1788802802.62 total 177.54 last 23.27) - open/close 3897 (time 1788802812.62 total 187.54 last 24.10) - open/close 4156 (time 1788802822.63 total 197.55 last 25.89) - open/close 4383 (time 1788802832.73 total 207.65 last 22.46) - open/close 4633 (time 1788802843.42 total 218.33 last 23.41) - open/close 4879 (time 1788802853.53 total 228.45 last 24.32) - open/close 5115 (time 1788802864.15 total 239.07 last 22.23) - open/close 5337 (time 1788802874.84 total 249.76 last 20.76) - open/close 5590 (time 1788802885.85 total 260.77 last 22.98) - open/close 5781 (time 1788802896.47 total 271.39 last 17.97) - open/close 6027 (time 1788802906.70 total 281.62 last 24.05) - open/close 6255 (time 1788802916.82 total 291.74 last 22.53) - open/close 6488 (time 1788802926.85 total 301.77 last 23.22) - open/close 6712 (time 1788802937.80 total 312.72 last 20.46) - open/close 6945 (time 1788802948.57 total 323.49 last 21.64) - open/close 7203 (time 1788802959.06 total 333.97 last 24.60) - open/close 7442 (time 1788802970.00 total 344.92 last 21.84) - open/close 7685 (time 1788802980.32 total 355.24 last 23.54) - open/close 7915 (time 1788802990.33 total 365.25 last 22.97) - open/close 8154 (time 1788803000.34 total 375.26 last 23.90) - open/close 8394 (time 1788803010.35 total 385.27 last 23.97) - open/close 8652 (time 1788803021.09 total 396.01 last 24.03) - open/close 8868 (time 1788803031.31 total 406.23 last 21.13) - open/close 9098 (time 1788803041.69 total 416.61 last 22.15) - open/close 9356 (time 1788803052.31 total 427.23 last 24.29) - open/close 9619 (time 1788803063.29 total 438.21 last 23.96) - open/close 9831 (time 1788803073.62 total 448.54 last 20.53) - open/close 10000 (time 1788803080.49 total 455.41 last 24.60) - open/close 10229 (time 1788803090.59 total 465.51 last 22.67) - open/close 10509 (time 1788803101.61 total 476.53 last 25.41) - open/close 10740 (time 1788803111.63 total 486.55 last 23.05) - open/close 11029 (time 1788803121.64 total 496.56 last 28.85) - open/close 11261 (time 1788803131.73 total 506.65 last 23.00) - open/close 11453 (time 1788803141.75 total 516.67 last 19.15) - open/close 11668 (time 1788803151.78 total 526.70 last 21.45) - open/close 11904 (time 1788803161.94 total 536.86 last 23.23) - open/close 12167 (time 1788803172.95 total 547.87 last 23.88) - open/close 12398 (time 1788803182.96 total 557.88 last 23.07) - open/close 12612 (time 1788803193.94 total 568.86 last 19.50) - open/close 12869 (time 1788803204.69 total 579.61 last 23.90) - open/close 13058 (time 1788803215.13 total 590.05 last 18.11) - open/close 13309 (time 1788803225.81 total 600.73 last 23.49) - open/close 13536 (time 1788803236.04 total 610.96 last 22.18) - open/close 13722 (time 1788803246.99 total 621.91 last 16.99) - open/close 13942 (time 1788803257.61 total 632.53 last 20.72) - open/close 14193 (time 1788803268.38 total 643.30 last 23.30) - open/close 14448 (time 1788803278.38 total 653.30 last 25.49) - open/close 14693 (time 1788803288.39 total 663.31 last 24.47) - open/close 14934 (time 1788803299.22 total 674.14 last 22.27) - open/close 15143 (time 1788803309.22 total 684.14 last 20.88) - open/close 15414 (time 1788803320.27 total 695.19 last 24.53) - open/close 15647 (time 1788803330.27 total 705.19 last 23.30) - open/close 15877 (time 1788803341.28 total 716.20 last 20.89) - open/close 16095 (time 1788803352.14 total 727.06 last 20.07) - open/close 16286 (time 1788803362.25 total 737.17 last 18.90) - open/close 16490 (time 1788803372.64 total 747.56 last 19.65) - open/close 16760 (time 1788803383.63 total 758.55 last 24.57) - open/close 16967 (time 1788803394.34 total 769.25 last 19.33) - open/close 17076 (time 1788803405.26 total 780.18 last 9.97) - open/close 17329 (time 1788803415.83 total 790.75 last 23.95) - open/close 17560 (time 1788803426.39 total 801.31 last 21.87) - open/close 17786 (time 1788803437.02 total 811.94 last 21.26) - open/close 17981 (time 1788803448.09 total 823.01 last 17.62) - open/close 18212 (time 1788803458.38 total 833.30 last 22.45) - open/close 18597 (time 1788803468.39 total 843.31 last 38.49) - open/close 19387 (time 1788803478.39 total 853.31 last 78.96) - open/close 20000 (time 1788803486.62 total 861.54 last 74.49) - open/close 20832 (time 1788803496.63 total 871.55 last 83.11) - open/close 21656 (time 1788803506.64 total 881.56 last 82.36) - open/close 22473 (time 1788803516.64 total 891.56 last 81.68) - open/close 23300 (time 1788803526.65 total 901.57 last 82.63) - open/close 24079 (time 1788803536.65 total 911.57 last 77.88) - open/close 24809 (time 1788803546.65 total 921.57 last 72.97) - open/close 25430 (time 1788803556.66 total 931.58 last 62.05) - open/close 26133 (time 1788803566.68 total 941.60 last 70.19) - open/close 26917 (time 1788803576.68 total 951.60 last 78.38) - open/close 27606 (time 1788803586.68 total 961.60 last 68.88) - open/close 28345 (time 1788803596.70 total 971.62 last 73.80) - open/close 29099 (time 1788803606.71 total 981.63 last 75.31) - open/close 29872 (time 1788803616.71 total 991.63 last 77.27) - open/close 30531 (time 1788803626.72 total 1001.64 last 65.85) - open/close 31318 (time 1788803636.73 total 1011.65 last 78.62) - open/close 32039 (time 1788803646.74 total 1021.66 last 72.02) - open/close 32805 (time 1788803656.75 total 1031.67 last 76.49) - open/close 33479 (time 1788803666.76 total 1041.68 last 67.34) - open/close 34210 (time 1788803676.78 total 1051.70 last 72.95) - open/close 34923 (time 1788803686.79 total 1061.71 last 71.26) - open/close 35745 (time 1788803696.80 total 1071.72 last 82.13) - open/close 36536 (time 1788803706.80 total 1081.72 last 79.04) - open/close 37290 (time 1788803716.81 total 1091.73 last 75.38) - open/close 38046 (time 1788803726.82 total 1101.74 last 75.54) - open/close 38733 (time 1788803736.82 total 1111.74 last 68.66) - open/close 39497 (time 1788803746.84 total 1121.76 last 76.29) - open/close 40000 (time 1788803754.06 total 1128.98 last 69.67) - open/close 40810 (time 1788803764.06 total 1138.98 last 80.93) - open/close 41607 (time 1788803774.08 total 1149.00 last 79.59) - open/close 42405 (time 1788803784.08 total 1159.00 last 79.76) - open/close 43207 (time 1788803794.09 total 1169.01 last 80.17) - open/close 44024 (time 1788803804.09 total 1179.01 last 81.66) - open/close 44833 (time 1788803814.10 total 1189.02 last 80.88) - open/close 45619 (time 1788803824.10 total 1199.02 last 78.59) - open/close 46403 (time 1788803834.10 total 1209.02 last 78.38) - open/close 47192 (time 1788803844.10 total 1219.02 last 78.86) - open/close 48014 (time 1788803854.11 total 1229.03 last 82.12) - open/close 48804 (time 1788803864.12 total 1239.04 last 78.94) - open/close 49625 (time 1788803874.13 total 1249.05 last 82.02) total: 50000 open/close in 1253.97 seconds: 39.87 ops/second - unlinked 0 (time 1788803882 ; total 0 ; last 0) - unlinked 10000 (time 1788804025 ; total 143 ; last 143) - unlinked 20000 (time 1788804158 ; total 276 ; last 133) - unlinked 30000 (time 1788804295 ; total 413 ; last 137) - unlinked 40000 (time 1788804421 ; total 539 ; last 126) total: 50000 unlinks in 675 seconds: 74.074074 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: kmemleak disabled modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2