== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:26:05 (1788927965) start mds service on oleg311-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 8 22:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: kmemleak disabled oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg311-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg311-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1788928060.36 total 13.04 last 12.04) - open/close 227 (time 1788928071.30 total 23.98 last 6.40) - open/close 403 (time 1788928081.81 total 34.49 last 16.75) - open/close 506 (time 1788928092.74 total 45.41 last 9.43) - open/close 612 (time 1788928103.62 total 56.30 last 9.74) - open/close 687 (time 1788928113.88 total 66.56 last 7.31) - open/close 715 (time 1788928124.94 total 77.62 last 2.53) - open/close 826 (time 1788928135.94 total 88.61 last 10.10) - open/close 1068 (time 1788928146.32 total 98.99 last 23.32) - open/close 1220 (time 1788928157.19 total 109.86 last 13.98) - open/close 1340 (time 1788928167.19 total 119.87 last 12.00) - open/close 1558 (time 1788928178.02 total 130.69 last 20.14) - open/close 1716 (time 1788928188.55 total 141.22 last 15.00) - open/close 1843 (time 1788928200.73 total 153.41 last 10.42) - open/close 2079 (time 1788928213.07 total 165.75 last 19.12) - open/close 2230 (time 1788928223.58 total 176.25 last 14.38) - open/close 2412 (time 1788928233.59 total 186.26 last 18.18) - open/close 2526 (time 1788928244.36 total 197.04 last 10.58) - open/close 2762 (time 1788928254.99 total 207.67 last 22.20) - open/close 2891 (time 1788928265.60 total 218.28 last 12.16) - open/close 3103 (time 1788928276.23 total 228.91 last 19.94) - open/close 3288 (time 1788928286.53 total 239.20 last 17.97) - open/close 3530 (time 1788928297.05 total 249.72 last 23.00) - open/close 3714 (time 1788928308.16 total 260.84 last 16.55) - open/close 3945 (time 1788928318.17 total 270.84 last 23.09) - open/close 4155 (time 1788928328.17 total 280.85 last 21.00) - open/close 4312 (time 1788928338.19 total 290.86 last 15.68) - open/close 4479 (time 1788928349.05 total 301.72 last 15.37) - open/close 4732 (time 1788928359.55 total 312.22 last 24.10) - open/close 4818 (time 1788928370.04 total 322.71 last 8.20) - open/close 4964 (time 1788928380.35 total 333.02 last 14.16) - open/close 5101 (time 1788928390.91 total 343.59 last 12.97) - open/close 5337 (time 1788928401.66 total 354.34 last 21.96) - open/close 5571 (time 1788928412.50 total 365.17 last 21.60) - open/close 5746 (time 1788928422.50 total 375.17 last 17.49) - open/close 5882 (time 1788928433.01 total 385.69 last 12.94) - open/close 6060 (time 1788928443.03 total 395.70 last 17.77) - open/close 6245 (time 1788928453.03 total 405.70 last 18.50) - open/close 6411 (time 1788928463.81 total 416.48 last 15.40) - open/close 6545 (time 1788928474.96 total 427.64 last 12.01) - open/close 6703 (time 1788928484.97 total 437.65 last 15.78) - open/close 6914 (time 1788928494.98 total 447.65 last 21.09) - open/close 7151 (time 1788928505.99 total 458.66 last 21.53) - open/close 7387 (time 1788928518.04 total 470.71 last 19.59) - open/close 7565 (time 1788928528.99 total 481.66 last 16.25) - open/close 7731 (time 1788928539.79 total 492.46 last 15.38) - open/close 7884 (time 1788928549.82 total 502.49 last 15.25) - open/close 8127 (time 1788928559.83 total 512.50 last 24.28) - open/close 8301 (time 1788928570.64 total 523.32 last 16.09) - open/close 8452 (time 1788928580.65 total 533.32 last 15.09) - open/close 8695 (time 1788928590.65 total 543.33 last 24.30) - open/close 8909 (time 1788928601.71 total 554.39 last 19.35) - open/close 9106 (time 1788928612.04 total 564.71 last 19.08) - open/close 9325 (time 1788928622.05 total 574.72 last 21.88) - open/close 9504 (time 1788928632.06 total 584.73 last 17.89) - open/close 9667 (time 1788928642.07 total 594.74 last 16.27) - open/close 9785 (time 1788928654.07 total 606.74 last 9.84) - open/close 9916 (time 1788928664.51 total 617.18 last 12.55) - open/close 10000 (time 1788928668.89 total 621.56 last 19.16) - open/close 10145 (time 1788928679.78 total 632.45 last 13.32) - open/close 10326 (time 1788928690.89 total 643.56 last 16.29) - open/close 10555 (time 1788928702.61 total 655.28 last 19.54) - open/close 10746 (time 1788928713.03 total 665.70 last 18.33) - open/close 10915 (time 1788928723.98 total 676.66 last 15.43) - open/close 10980 (time 1788928734.02 total 686.69 last 6.48) - open/close 11200 (time 1788928744.85 total 697.52 last 20.32) - open/close 11446 (time 1788928755.78 total 708.46 last 22.50) - open/close 11589 (time 1788928771.32 total 723.99 last 9.21) - open/close 11825 (time 1788928781.32 total 733.99 last 23.59) - open/close 11939 (time 1788928791.32 total 743.99 last 11.40) - open/close 12067 (time 1788928801.33 total 754.00 last 12.79) - open/close 12308 (time 1788928811.99 total 764.66 last 22.60) - open/close 12493 (time 1788928824.66 total 777.34 last 14.60) - open/close 12647 (time 1788928835.68 total 788.35 last 13.98) - open/close 12748 (time 1788928851.73 total 804.41 last 6.29) - open/close 12906 (time 1788928862.23 total 814.90 last 15.06) - open/close 13076 (time 1788928872.70 total 825.38 last 16.23) - open/close 13185 (time 1788928885.37 total 838.04 last 8.61) - open/close 13358 (time 1788928895.41 total 848.08 last 17.23) - open/close 13633 (time 1788928905.41 total 858.08 last 27.50) - open/close 13800 (time 1788928916.84 total 869.52 last 14.60) - open/close 13989 (time 1788928926.92 total 879.59 last 18.76) - open/close 14182 (time 1788928937.03 total 889.71 last 19.08) - open/close 14355 (time 1788928947.99 total 900.66 last 15.79) - open/close 14581 (time 1788928958.91 total 911.58 last 20.69) - open/close 14757 (time 1788928969.74 total 922.41 last 16.25) - open/close 14989 (time 1788928979.76 total 932.43 last 23.16) - open/close 15092 (time 1788928990.37 total 943.04 last 9.71) - open/close 15179 (time 1788929000.92 total 953.60 last 8.24) - open/close 15278 (time 1788929011.84 total 964.52 last 9.07) - open/close 15460 (time 1788929021.97 total 974.65 last 17.96) - open/close 15644 (time 1788929034.18 total 986.86 last 15.07) - open/close 15860 (time 1788929044.80 total 997.47 last 20.34) - open/close 16034 (time 1788929055.33 total 1008.00 last 16.53) - open/close 16145 (time 1788929065.74 total 1018.41 last 10.66) - open/close 16292 (time 1788929077.19 total 1029.87 last 12.83) - open/close 16435 (time 1788929087.74 total 1040.41 last 13.57) - open/close 16769 (time 1788929097.75 total 1050.43 last 33.33) - open/close 17370 (time 1788929107.77 total 1060.45 last 60.00) - open/close 18013 (time 1788929117.78 total 1070.46 last 64.24) - open/close 18679 (time 1788929127.80 total 1080.47 last 66.48) - open/close 19336 (time 1788929137.81 total 1090.48 last 65.66) - open/close 19979 (time 1788929147.82 total 1100.49 last 64.23) - open/close 20645 (time 1788929157.82 total 1110.50 last 66.55) - open/close 21311 (time 1788929167.84 total 1120.51 last 66.50) - open/close 22011 (time 1788929177.84 total 1130.51 last 70.00) - open/close 22729 (time 1788929187.84 total 1140.52 last 71.78) - open/close 23402 (time 1788929197.86 total 1150.53 last 67.18) - open/close 23931 (time 1788929207.87 total 1160.54 last 52.86) - open/close 24395 (time 1788929217.88 total 1170.56 last 46.31) - open/close 24793 (time 1788929227.90 total 1180.57 last 39.74) - open/close 25369 (time 1788929237.92 total 1190.59 last 57.47) - open/close 26003 (time 1788929247.93 total 1200.60 last 63.36) - open/close 26690 (time 1788929257.94 total 1210.61 last 68.65) - open/close 27279 (time 1788929267.94 total 1220.62 last 58.86) - open/close 27968 (time 1788929277.95 total 1230.62 last 68.86) - open/close 28622 (time 1788929287.98 total 1240.65 last 65.18) - open/close 29146 (time 1788929298.00 total 1250.67 last 52.30) - open/close 29713 (time 1788929308.01 total 1260.69 last 56.61) - open/close 30000 (time 1788929313.63 total 1266.31 last 51.09) - open/close 30548 (time 1788929323.64 total 1276.32 last 54.74) - open/close 31125 (time 1788929333.67 total 1286.34 last 57.55) - open/close 31647 (time 1788929343.68 total 1296.35 last 52.16) - open/close 32188 (time 1788929353.69 total 1306.36 last 54.04) - open/close 32675 (time 1788929363.70 total 1316.37 last 48.65) - open/close 33324 (time 1788929373.70 total 1326.37 last 64.90) - open/close 34006 (time 1788929383.71 total 1336.38 last 68.12) - open/close 34550 (time 1788929393.74 total 1346.41 last 54.26) - open/close 35126 (time 1788929403.75 total 1356.42 last 57.54) - open/close 35759 (time 1788929413.75 total 1366.43 last 63.25) - open/close 36253 (time 1788929423.76 total 1376.43 last 49.37) - open/close 36943 (time 1788929433.77 total 1386.44 last 68.95) - open/close 37558 (time 1788929443.77 total 1396.45 last 61.47) - open/close 38183 (time 1788929453.78 total 1406.45 last 62.45) - open/close 38802 (time 1788929463.80 total 1416.47 last 61.77) - open/close 39385 (time 1788929473.81 total 1426.48 last 58.25) - open/close 40000 (time 1788929483.23 total 1435.90 last 65.28) - open/close 40593 (time 1788929493.26 total 1445.93 last 59.14) - open/close 41174 (time 1788929503.27 total 1455.94 last 58.05) - open/close 41704 (time 1788929513.27 total 1465.94 last 52.97) - open/close 42261 (time 1788929523.27 total 1475.95 last 55.68) - open/close 42894 (time 1788929533.29 total 1485.96 last 63.21) - open/close 43505 (time 1788929543.30 total 1495.97 last 61.04) - open/close 44111 (time 1788929553.30 total 1505.97 last 60.59) - open/close 44666 (time 1788929563.32 total 1515.99 last 55.40) - open/close 45337 (time 1788929573.32 total 1526.00 last 67.04) - open/close 45877 (time 1788929583.33 total 1536.01 last 53.96) - open/close 46398 (time 1788929593.34 total 1546.01 last 52.06) - open/close 47021 (time 1788929603.34 total 1556.02 last 62.28) - open/close 47546 (time 1788929613.34 total 1566.02 last 52.50) - open/close 48057 (time 1788929623.35 total 1576.02 last 51.05) - open/close 48629 (time 1788929633.35 total 1586.03 last 57.19) - open/close 49229 (time 1788929643.35 total 1596.03 last 59.99) - open/close 49838 (time 1788929653.37 total 1606.04 last 60.81) total: 50000 open/close in 1608.79 seconds: 31.08 ops/second - unlinked 0 (time 1788929661 ; total 1 ; last 1) - unlinked 10000 (time 1788929855 ; total 195 ; last 194) - unlinked 20000 (time 1788930028 ; total 368 ; last 173) - unlinked 30000 (time 1788930201 ; total 541 ; last 173) - unlinked 40000 (time 1788930386 ; total 726 ; last 185) total: 50000 unlinks in 893 seconds: 55.991043 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: kmemleak disabled modules unloaded. pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2