== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:13:49 (1788833629) start mds service on oleg236-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing load_modules_local oleg236-server: kmemleak disabled oleg236-server: Loading modules from /home/green/git/lustre-release/lustre oleg236-server: detected 4 online CPUs by sysfs oleg236-server: MODOPTS_LIBCFS= oleg236-server: Force libcfs to create 2 CPU partitions oleg236-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg236-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg236-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg236-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg236-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 143 (time 1788833722.44 total 10.06 last 14.21) - open/close 363 (time 1788833732.46 total 20.08 last 21.95) - open/close 538 (time 1788833742.49 total 30.11 last 17.46) - open/close 701 (time 1788833753.08 total 40.70 last 15.40) - open/close 892 (time 1788833763.08 total 50.70 last 19.09) - open/close 1090 (time 1788833773.14 total 60.76 last 19.70) - open/close 1262 (time 1788833783.15 total 70.77 last 17.18) - open/close 1513 (time 1788833793.53 total 81.15 last 24.18) - open/close 1768 (time 1788833804.40 total 92.02 last 23.45) - open/close 1985 (time 1788833814.70 total 102.32 last 21.08) - open/close 2211 (time 1788833825.28 total 112.90 last 21.36) - open/close 2338 (time 1788833836.60 total 124.22 last 11.22) - open/close 2457 (time 1788833846.60 total 134.22 last 11.90) - open/close 2630 (time 1788833856.62 total 144.24 last 17.27) - open/close 2799 (time 1788833866.66 total 154.28 last 16.83) - open/close 2841 (time 1788833876.86 total 164.48 last 4.12) - open/close 2895 (time 1788833887.28 total 174.90 last 5.18) - open/close 3078 (time 1788833897.29 total 184.91 last 18.29) - open/close 3227 (time 1788833907.58 total 195.20 last 14.48) - open/close 3380 (time 1788833918.53 total 206.15 last 13.97) - open/close 3547 (time 1788833929.60 total 217.22 last 15.09) - open/close 3671 (time 1788833939.76 total 227.38 last 12.20) - open/close 3846 (time 1788833949.77 total 237.39 last 17.49) - open/close 3964 (time 1788833960.71 total 248.33 last 10.79) - open/close 4129 (time 1788833970.73 total 258.35 last 16.46) - open/close 4205 (time 1788833982.14 total 269.76 last 6.66) - open/close 4388 (time 1788833992.15 total 279.77 last 18.28) - open/close 4513 (time 1788834002.68 total 290.30 last 11.87) - open/close 4638 (time 1788834012.93 total 300.55 last 12.20) - open/close 4808 (time 1788834023.47 total 311.09 last 16.13) - open/close 4898 (time 1788834034.38 total 322.00 last 8.25) - open/close 4959 (time 1788834044.79 total 332.41 last 5.86) - open/close 5039 (time 1788834055.21 total 342.83 last 7.67) - open/close 5159 (time 1788834065.61 total 353.23 last 11.54) - open/close 5216 (time 1788834075.66 total 363.28 last 5.67) - open/close 5337 (time 1788834085.68 total 373.30 last 12.07) - open/close 5503 (time 1788834095.75 total 383.37 last 16.49) - open/close 5614 (time 1788834105.78 total 393.40 last 11.07) - open/close 5777 (time 1788834115.83 total 403.45 last 16.21) - open/close 5942 (time 1788834126.67 total 414.29 last 15.23) - open/close 6098 (time 1788834136.69 total 424.31 last 15.57) - open/close 6190 (time 1788834146.70 total 434.32 last 9.19) - open/close 6302 (time 1788834157.50 total 445.12 last 10.37) - open/close 6404 (time 1788834167.53 total 455.15 last 10.17) - open/close 6515 (time 1788834177.55 total 465.17 last 11.08) - open/close 6617 (time 1788834188.31 total 475.93 last 9.49) - open/close 6865 (time 1788834198.32 total 485.94 last 24.77) - open/close 7029 (time 1788834208.32 total 495.94 last 16.39) - open/close 7251 (time 1788834218.93 total 506.55 last 20.92) - open/close 7374 (time 1788834228.94 total 516.56 last 12.29) - open/close 7569 (time 1788834239.35 total 526.97 last 18.73) - open/close 7791 (time 1788834252.34 total 539.96 last 17.09) - open/close 7894 (time 1788834262.35 total 549.97 last 10.29) - open/close 7988 (time 1788834272.88 total 560.50 last 8.93) - open/close 8094 (time 1788834282.93 total 570.55 last 10.55) - open/close 8220 (time 1788834292.99 total 580.61 last 12.53) - open/close 8390 (time 1788834303.15 total 590.77 last 16.72) - open/close 8611 (time 1788834313.16 total 600.78 last 22.09) - open/close 8737 (time 1788834323.57 total 611.19 last 12.10) - open/close 8851 (time 1788834334.07 total 621.69 last 10.86) - open/close 8963 (time 1788834344.59 total 632.21 last 10.65) - open/close 9094 (time 1788834354.62 total 642.24 last 13.06) - open/close 9280 (time 1788834365.26 total 652.88 last 17.48) - open/close 9459 (time 1788834375.72 total 663.34 last 17.11) - open/close 9692 (time 1788834385.72 total 673.34 last 23.29) - open/close 9926 (time 1788834395.73 total 683.35 last 23.38) - open/close 10000 (time 1788834399.47 total 687.09 last 19.80) - open/close 10201 (time 1788834410.38 total 698.00 last 18.42) - open/close 10361 (time 1788834420.42 total 708.04 last 15.94) - open/close 10539 (time 1788834430.82 total 718.44 last 17.11) - open/close 10591 (time 1788834441.51 total 729.13 last 4.86) - open/close 10791 (time 1788834452.29 total 739.91 last 18.56) - open/close 10963 (time 1788834462.30 total 749.92 last 17.19) - open/close 11074 (time 1788834472.51 total 760.13 last 10.87) - open/close 11207 (time 1788834483.25 total 770.87 last 12.38) - open/close 11413 (time 1788834494.11 total 781.73 last 18.96) - open/close 11601 (time 1788834504.39 total 792.01 last 18.29) - open/close 11709 (time 1788834514.41 total 802.03 last 10.78) - open/close 11804 (time 1788834526.07 total 813.69 last 8.15) - open/close 11945 (time 1788834536.09 total 823.71 last 14.06) - open/close 12069 (time 1788834546.57 total 834.19 last 11.84) - open/close 12159 (time 1788834557.11 total 844.73 last 8.54) - open/close 12268 (time 1788834567.11 total 854.73 last 10.89) - open/close 12434 (time 1788834577.12 total 864.74 last 16.59) - open/close 12552 (time 1788834587.53 total 875.15 last 11.34) - open/close 12749 (time 1788834597.53 total 885.15 last 19.69) - open/close 12881 (time 1788834610.02 total 897.64 last 10.57) - open/close 13115 (time 1788834620.03 total 907.65 last 23.37) - open/close 13307 (time 1788834630.84 total 918.46 last 17.77) - open/close 13441 (time 1788834641.16 total 928.78 last 12.99) - open/close 13599 (time 1788834651.20 total 938.82 last 15.73) - open/close 13768 (time 1788834661.20 total 948.82 last 16.90) - open/close 13919 (time 1788834672.02 total 959.64 last 13.96) - open/close 14044 (time 1788834685.06 total 972.68 last 9.58) - open/close 14149 (time 1788834695.28 total 982.90 last 10.28) - open/close 14413 (time 1788834705.91 total 993.53 last 24.84) - open/close 14489 (time 1788834716.03 total 1003.65 last 7.51) - open/close 14743 (time 1788834726.14 total 1013.76 last 25.14) - open/close 14952 (time 1788834737.09 total 1024.71 last 19.08) - open/close 15016 (time 1788834748.56 total 1036.18 last 5.58) - open/close 15207 (time 1788834758.59 total 1046.21 last 19.05) - open/close 15267 (time 1788834768.61 total 1056.23 last 5.99) - open/close 15370 (time 1788834779.44 total 1067.06 last 9.51) - open/close 15429 (time 1788834790.26 total 1077.88 last 5.45) - open/close 15672 (time 1788834800.28 total 1087.90 last 24.25) - open/close 16362 (time 1788834810.28 total 1097.90 last 68.96) - open/close 17011 (time 1788834820.29 total 1107.91 last 64.85) - open/close 17677 (time 1788834830.29 total 1117.91 last 66.58) - open/close 18341 (time 1788834840.29 total 1127.91 last 66.40) - open/close 18879 (time 1788834850.30 total 1137.92 last 53.78) - open/close 19429 (time 1788834860.33 total 1147.95 last 54.83) - open/close 19998 (time 1788834870.34 total 1157.96 last 56.82) - open/close 20548 (time 1788834880.35 total 1167.97 last 54.98) - open/close 21112 (time 1788834890.35 total 1177.97 last 56.38) - open/close 21727 (time 1788834900.37 total 1187.99 last 61.39) - open/close 22342 (time 1788834910.37 total 1197.99 last 61.49) - open/close 23055 (time 1788834920.37 total 1207.99 last 71.27) - open/close 23734 (time 1788834930.38 total 1218.00 last 67.89) - open/close 24489 (time 1788834940.38 total 1228.00 last 75.45) - open/close 25156 (time 1788834950.40 total 1238.02 last 66.59) - open/close 25921 (time 1788834960.41 total 1248.03 last 76.43) - open/close 26622 (time 1788834970.41 total 1258.03 last 70.09) - open/close 27341 (time 1788834980.44 total 1268.06 last 71.70) - open/close 28011 (time 1788834990.44 total 1278.06 last 66.98) - open/close 28783 (time 1788835000.44 total 1288.06 last 77.19) - open/close 29406 (time 1788835010.44 total 1298.06 last 62.29) - open/close 30000 (time 1788835019.73 total 1307.35 last 63.98) - open/close 30733 (time 1788835029.74 total 1317.36 last 73.19) - open/close 31407 (time 1788835039.76 total 1327.38 last 67.28) - open/close 31994 (time 1788835049.77 total 1337.39 last 58.66) - open/close 32636 (time 1788835059.77 total 1347.39 last 64.18) - open/close 33328 (time 1788835069.78 total 1357.40 last 69.10) - open/close 33992 (time 1788835079.80 total 1367.42 last 66.31) - open/close 34549 (time 1788835089.81 total 1377.43 last 55.62) - open/close 35145 (time 1788835099.84 total 1387.46 last 59.45) - open/close 35746 (time 1788835109.85 total 1397.47 last 60.03) - open/close 36350 (time 1788835119.87 total 1407.49 last 60.29) - open/close 36930 (time 1788835129.88 total 1417.50 last 57.93) - open/close 37455 (time 1788835139.89 total 1427.51 last 52.44) - open/close 38066 (time 1788835149.90 total 1437.52 last 61.05) - open/close 38667 (time 1788835159.91 total 1447.53 last 60.03) - open/close 39298 (time 1788835169.92 total 1457.54 last 63.06) - open/close 40000 (time 1788835179.39 total 1467.01 last 74.10) - open/close 40726 (time 1788835189.41 total 1477.03 last 72.47) - open/close 41415 (time 1788835199.42 total 1487.04 last 68.84) - open/close 42109 (time 1788835209.42 total 1497.04 last 69.35) - open/close 42788 (time 1788835219.43 total 1507.05 last 67.86) - open/close 43397 (time 1788835229.47 total 1517.09 last 60.64) - open/close 44077 (time 1788835239.48 total 1527.10 last 67.93) - open/close 44713 (time 1788835249.49 total 1537.11 last 63.54) - open/close 45353 (time 1788835259.50 total 1547.12 last 63.95) - open/close 45809 (time 1788835269.52 total 1557.14 last 45.51) - open/close 46353 (time 1788835279.54 total 1567.16 last 54.29) - open/close 46983 (time 1788835289.55 total 1577.17 last 62.96) - open/close 47586 (time 1788835299.56 total 1587.18 last 60.21) - open/close 48201 (time 1788835309.58 total 1597.20 last 61.41) - open/close 48799 (time 1788835319.59 total 1607.21 last 59.69) - open/close 49411 (time 1788835329.60 total 1617.22 last 61.19) total: 50000 open/close in 1626.45 seconds: 30.74 ops/second - unlinked 0 (time 1788835344 ; total 0 ; last 0) - unlinked 10000 (time 1788835504 ; total 160 ; last 160) - unlinked 20000 (time 1788835687 ; total 343 ; last 183) - unlinked 30000 (time 1788835865 ; total 521 ; last 178) - unlinked 40000 (time 1788836022 ; total 678 ; last 157) total: 50000 unlinks in 842 seconds: 59.382423 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg236-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg236-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg236-server stop mds service on oleg236-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg236-server unloading modules via unload_modules_local on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing unload_modules_local oleg236-server: kmemleak disabled modules unloaded. pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2