== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:11:52 (1790788312) start mds service on oleg310-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 30 05:20 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: kmemleak disabled oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg310-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg310-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 188 (time 1790788395.65 total 10.00 last 18.79) - open/close 411 (time 1790788405.67 total 20.02 last 22.26) - open/close 594 (time 1790788416.01 total 30.36 last 17.70) - open/close 849 (time 1790788427.03 total 41.38 last 23.14) - open/close 1101 (time 1790788437.71 total 52.06 last 23.61) - open/close 1352 (time 1790788447.72 total 62.07 last 25.07) - open/close 1566 (time 1790788457.73 total 72.08 last 21.38) - open/close 1761 (time 1790788467.74 total 82.09 last 19.49) - open/close 1944 (time 1790788477.85 total 92.20 last 18.10) - open/close 2129 (time 1790788488.41 total 102.76 last 17.52) - open/close 2389 (time 1790788498.90 total 113.25 last 24.79) - open/close 2649 (time 1790788509.32 total 123.68 last 24.93) - open/close 2935 (time 1790788521.58 total 135.93 last 23.34) - open/close 3171 (time 1790788531.58 total 145.93 last 23.59) - open/close 3306 (time 1790788543.32 total 157.67 last 11.50) - open/close 3513 (time 1790788553.33 total 167.69 last 20.67) - open/close 3778 (time 1790788564.38 total 178.73 last 24.00) - open/close 3844 (time 1790788574.39 total 188.74 last 6.59) - open/close 3993 (time 1790788584.66 total 199.02 last 14.50) - open/close 4212 (time 1790788594.67 total 209.02 last 21.90) - open/close 4419 (time 1790788604.68 total 219.03 last 20.67) - open/close 4703 (time 1790788615.19 total 229.54 last 27.03) - open/close 4960 (time 1790788625.87 total 240.22 last 24.06) - open/close 5198 (time 1790788636.62 total 250.97 last 22.14) - open/close 5388 (time 1790788646.62 total 260.98 last 18.99) - open/close 5627 (time 1790788656.97 total 271.33 last 23.10) - open/close 5858 (time 1790788667.79 total 282.15 last 21.35) - open/close 6005 (time 1790788677.81 total 292.16 last 14.68) - open/close 6085 (time 1790788690.68 total 305.04 last 6.21) - open/close 6169 (time 1790788700.74 total 315.10 last 8.35) - open/close 6420 (time 1790788710.75 total 325.10 last 25.10) - open/close 6632 (time 1790788720.75 total 335.10 last 21.19) - open/close 6896 (time 1790788730.76 total 345.11 last 26.38) - open/close 7124 (time 1790788741.07 total 355.42 last 22.12) - open/close 7369 (time 1790788751.37 total 365.73 last 23.76) - open/close 7604 (time 1790788763.78 total 378.14 last 18.94) - open/close 7767 (time 1790788774.71 total 389.07 last 14.91) - open/close 8038 (time 1790788784.72 total 399.07 last 27.09) - open/close 8238 (time 1790788794.73 total 409.08 last 19.98) - open/close 8484 (time 1790788804.74 total 419.10 last 24.56) - open/close 8636 (time 1790788814.75 total 429.10 last 15.19) - open/close 8831 (time 1790788824.75 total 439.10 last 19.50) - open/close 8956 (time 1790788835.28 total 449.63 last 11.88) - open/close 9133 (time 1790788846.27 total 460.62 last 16.10) - open/close 9410 (time 1790788856.28 total 470.63 last 27.68) - open/close 9646 (time 1790788866.29 total 480.64 last 23.58) - open/close 9876 (time 1790788876.62 total 490.97 last 22.26) - open/close 10000 (time 1790788880.02 total 494.37 last 36.48) - open/close 10220 (time 1790788890.12 total 504.47 last 21.79) - open/close 10451 (time 1790788900.12 total 514.47 last 23.09) - open/close 10671 (time 1790788910.34 total 524.69 last 21.52) - open/close 10936 (time 1790788921.00 total 535.36 last 24.85) - open/close 11191 (time 1790788931.01 total 545.36 last 25.49) - open/close 11456 (time 1790788941.02 total 555.37 last 26.47) - open/close 11722 (time 1790788951.37 total 565.72 last 25.71) - open/close 11983 (time 1790788962.17 total 576.53 last 24.15) - open/close 12180 (time 1790788972.82 total 587.18 last 18.50) - open/close 12383 (time 1790788983.69 total 598.04 last 18.69) - open/close 12597 (time 1790788993.70 total 608.06 last 21.37) - open/close 12798 (time 1790789003.70 total 618.06 last 20.10) - open/close 12945 (time 1790789013.71 total 628.06 last 14.69) - open/close 13148 (time 1790789023.72 total 638.07 last 20.29) - open/close 13355 (time 1790789033.72 total 648.07 last 20.70) - open/close 13572 (time 1790789043.73 total 658.08 last 21.68) - open/close 13829 (time 1790789053.74 total 668.09 last 25.68) - open/close 14054 (time 1790789063.75 total 678.10 last 22.47) - open/close 14267 (time 1790789073.76 total 688.11 last 21.29) - open/close 14513 (time 1790789083.76 total 698.11 last 24.60) - open/close 14740 (time 1790789094.54 total 708.89 last 21.05) - open/close 14973 (time 1790789105.43 total 719.79 last 21.39) - open/close 15208 (time 1790789115.66 total 730.01 last 22.98) - open/close 15481 (time 1790789125.67 total 740.03 last 27.26) - open/close 15734 (time 1790789136.28 total 750.64 last 23.85) - open/close 15991 (time 1790789146.29 total 760.64 last 25.68) - open/close 16267 (time 1790789157.27 total 771.62 last 25.15) - open/close 16534 (time 1790789168.21 total 782.56 last 24.40) - open/close 16794 (time 1790789180.94 total 795.29 last 20.42) - open/close 17027 (time 1790789191.04 total 805.39 last 23.06) - open/close 17263 (time 1790789201.05 total 815.40 last 23.59) - open/close 17500 (time 1790789211.06 total 825.42 last 23.66) - open/close 17764 (time 1790789221.34 total 835.69 last 25.69) - open/close 18019 (time 1790789232.01 total 846.36 last 23.91) - open/close 18248 (time 1790789242.11 total 856.47 last 22.65) - open/close 18481 (time 1790789252.12 total 866.47 last 23.29) - open/close 18694 (time 1790789262.14 total 876.49 last 21.26) - open/close 18952 (time 1790789272.90 total 887.25 last 23.98) - open/close 19190 (time 1790789283.21 total 897.56 last 23.07) - open/close 19454 (time 1790789294.02 total 908.38 last 24.42) - open/close 20000 (time 1790789300.85 total 915.20 last 80.01) - open/close 20824 (time 1790789310.87 total 925.22 last 82.22) - open/close 21510 (time 1790789320.87 total 935.22 last 68.59) - open/close 22190 (time 1790789330.88 total 945.24 last 67.91) - open/close 23030 (time 1790789340.89 total 955.24 last 83.97) - open/close 23822 (time 1790789350.90 total 965.25 last 79.12) - open/close 24581 (time 1790789360.90 total 975.25 last 75.88) - open/close 25330 (time 1790789370.91 total 985.26 last 74.86) - open/close 26118 (time 1790789380.91 total 995.26 last 78.80) - open/close 26952 (time 1790789390.91 total 1005.26 last 83.38) - open/close 27796 (time 1790789400.92 total 1015.27 last 84.33) - open/close 28547 (time 1790789410.92 total 1025.27 last 75.07) - open/close 29335 (time 1790789420.92 total 1035.28 last 78.78) - open/close 30000 (time 1790789429.13 total 1043.48 last 81.09) - open/close 30820 (time 1790789439.14 total 1053.49 last 81.87) - open/close 31611 (time 1790789449.16 total 1063.51 last 78.97) - open/close 32407 (time 1790789459.16 total 1073.51 last 79.58) - open/close 33198 (time 1790789469.17 total 1083.52 last 79.03) - open/close 34001 (time 1790789479.18 total 1093.53 last 80.24) - open/close 34753 (time 1790789489.18 total 1103.53 last 75.20) - open/close 35550 (time 1790789499.19 total 1113.54 last 79.59) - open/close 36303 (time 1790789509.20 total 1123.55 last 75.25) - open/close 37094 (time 1790789519.20 total 1133.55 last 79.08) - open/close 37827 (time 1790789529.20 total 1143.55 last 73.29) - open/close 38577 (time 1790789539.20 total 1153.56 last 74.98) - open/close 39389 (time 1790789549.20 total 1163.56 last 81.20) - open/close 40000 (time 1790789556.24 total 1170.59 last 86.86) - open/close 40809 (time 1790789566.25 total 1180.60 last 80.79) - open/close 41610 (time 1790789576.26 total 1190.61 last 80.06) - open/close 42354 (time 1790789586.29 total 1200.64 last 74.17) - open/close 42978 (time 1790789596.29 total 1210.65 last 62.36) - open/close 43709 (time 1790789606.30 total 1220.65 last 73.08) - open/close 44547 (time 1790789616.31 total 1230.66 last 83.72) - open/close 45452 (time 1790789626.31 total 1240.66 last 90.45) - open/close 46208 (time 1790789636.32 total 1250.67 last 75.53) - open/close 47010 (time 1790789646.32 total 1260.67 last 80.19) - open/close 47793 (time 1790789656.33 total 1270.68 last 78.25) - open/close 48545 (time 1790789666.33 total 1280.69 last 75.17) - open/close 49383 (time 1790789676.33 total 1290.69 last 83.78) total: 50000 open/close in 1298.54 seconds: 38.50 ops/second - unlinked 0 (time 1790789687 ; total 0 ; last 0) - unlinked 10000 (time 1790789814 ; total 127 ; last 127) - unlinked 20000 (time 1790789947 ; total 260 ; last 133) - unlinked 30000 (time 1790790085 ; total 398 ; last 138) - unlinked 40000 (time 1790790225 ; total 538 ; last 140) total: 50000 unlinks in 679 seconds: 73.637703 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: kmemleak disabled modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2