== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:12:49 (1789585969) start mds service on oleg312-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 16 09:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing load_modules_local oleg312-server: kmemleak disabled oleg312-server: Loading modules from /home/green/git/lustre-release/lustre oleg312-server: detected 4 online CPUs by sysfs oleg312-server: MODOPTS_LIBCFS= oleg312-server: Force libcfs to create 2 CPU partitions oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg312-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg312-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg312-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1789586056.61 total 10.01 last 20.77) - open/close 420 (time 1789586067.67 total 21.07 last 19.18) - open/close 646 (time 1789586077.67 total 31.07 last 22.59) - open/close 803 (time 1789586088.59 total 41.99 last 14.38) - open/close 928 (time 1789586099.03 total 52.43 last 11.98) - open/close 1049 (time 1789586109.96 total 63.36 last 11.07) - open/close 1168 (time 1789586120.97 total 74.37 last 10.81) - open/close 1422 (time 1789586131.92 total 85.32 last 23.20) - open/close 1684 (time 1789586142.87 total 96.27 last 23.93) - open/close 1936 (time 1789586153.81 total 107.21 last 23.03) - open/close 2143 (time 1789586163.82 total 117.22 last 20.67) - open/close 2329 (time 1789586173.91 total 127.31 last 18.45) - open/close 2523 (time 1789586183.92 total 137.32 last 19.37) - open/close 2711 (time 1789586195.98 total 149.39 last 15.59) - open/close 2864 (time 1789586205.99 total 159.39 last 15.29) - open/close 3096 (time 1789586216.01 total 169.41 last 23.16) - open/close 3320 (time 1789586226.63 total 180.03 last 21.10) - open/close 3561 (time 1789586237.34 total 190.74 last 22.50) - open/close 3803 (time 1789586247.35 total 200.75 last 24.17) - open/close 4036 (time 1789586257.35 total 210.75 last 23.29) - open/close 4257 (time 1789586267.60 total 221.00 last 21.56) - open/close 4520 (time 1789586278.17 total 231.57 last 24.89) - open/close 4785 (time 1789586289.09 total 242.49 last 24.26) - open/close 4973 (time 1789586299.10 total 252.50 last 18.78) - open/close 5087 (time 1789586309.97 total 263.37 last 10.49) - open/close 5267 (time 1789586323.87 total 277.27 last 12.95) - open/close 5497 (time 1789586334.80 total 288.20 last 21.03) - open/close 5706 (time 1789586344.84 total 298.24 last 20.82) - open/close 5899 (time 1789586355.81 total 309.21 last 17.59) - open/close 6146 (time 1789586365.82 total 319.22 last 24.69) - open/close 6363 (time 1789586376.08 total 329.48 last 21.15) - open/close 6556 (time 1789586386.08 total 339.48 last 19.29) - open/close 6825 (time 1789586396.09 total 349.49 last 26.88) - open/close 7038 (time 1789586407.18 total 360.58 last 19.21) - open/close 7287 (time 1789586417.18 total 370.58 last 24.90) - open/close 7514 (time 1789586427.68 total 381.08 last 21.61) - open/close 7767 (time 1789586437.69 total 391.09 last 25.28) - open/close 8007 (time 1789586447.70 total 401.10 last 23.98) - open/close 8241 (time 1789586460.75 total 414.15 last 17.93) - open/close 8375 (time 1789586471.64 total 425.04 last 12.31) - open/close 8530 (time 1789586481.65 total 435.05 last 15.47) - open/close 8640 (time 1789586491.66 total 445.06 last 10.99) - open/close 8742 (time 1789586502.29 total 455.69 last 9.60) - open/close 8943 (time 1789586513.43 total 466.83 last 18.04) - open/close 9170 (time 1789586524.16 total 477.56 last 21.15) - open/close 9400 (time 1789586534.68 total 488.08 last 21.86) - open/close 9547 (time 1789586544.72 total 498.13 last 14.64) - open/close 9647 (time 1789586554.74 total 508.14 last 9.99) - open/close 9928 (time 1789586565.45 total 518.86 last 26.22) - open/close 10000 (time 1789586567.60 total 521.00 last 33.60) - open/close 10193 (time 1789586578.83 total 532.23 last 17.18) - open/close 10460 (time 1789586588.84 total 542.25 last 26.67) - open/close 10684 (time 1789586599.02 total 552.42 last 22.02) - open/close 10918 (time 1789586609.49 total 562.89 last 22.34) - open/close 11148 (time 1789586619.50 total 572.90 last 22.99) - open/close 11393 (time 1789586630.28 total 583.68 last 22.73) - open/close 11589 (time 1789586640.97 total 594.37 last 18.33) - open/close 11843 (time 1789586651.10 total 604.50 last 25.08) - open/close 11999 (time 1789586661.12 total 614.52 last 15.56) - open/close 12134 (time 1789586671.43 total 624.83 last 13.09) - open/close 12318 (time 1789586681.45 total 634.85 last 18.37) - open/close 12578 (time 1789586691.92 total 645.32 last 24.82) - open/close 12801 (time 1789586701.93 total 655.33 last 22.28) - open/close 13042 (time 1789586712.52 total 665.92 last 22.77) - open/close 13244 (time 1789586723.53 total 676.93 last 18.34) - open/close 13406 (time 1789586733.71 total 687.11 last 15.92) - open/close 13604 (time 1789586743.84 total 697.24 last 19.54) - open/close 13763 (time 1789586753.85 total 707.25 last 15.88) - open/close 13842 (time 1789586764.18 total 717.58 last 7.65) - open/close 14110 (time 1789586774.20 total 727.60 last 26.75) - open/close 14322 (time 1789586784.85 total 738.25 last 19.91) - open/close 14568 (time 1789586795.81 total 749.21 last 22.43) - open/close 14832 (time 1789586805.83 total 759.23 last 26.35) - open/close 15075 (time 1789586816.53 total 769.93 last 22.72) - open/close 15261 (time 1789586826.53 total 779.93 last 18.60) - open/close 15412 (time 1789586836.88 total 790.28 last 14.59) - open/close 15546 (time 1789586847.64 total 801.04 last 12.46) - open/close 15705 (time 1789586857.67 total 811.07 last 15.85) - open/close 15926 (time 1789586867.93 total 821.33 last 21.53) - open/close 16178 (time 1789586877.95 total 831.35 last 25.17) - open/close 16318 (time 1789586888.40 total 841.80 last 13.39) - open/close 16594 (time 1789586898.45 total 851.85 last 27.46) - open/close 16831 (time 1789586909.21 total 862.61 last 22.04) - open/close 17083 (time 1789586919.23 total 872.63 last 25.14) - open/close 17267 (time 1789586929.62 total 883.02 last 17.71) - open/close 17476 (time 1789586940.39 total 893.79 last 19.41) - open/close 17597 (time 1789586952.98 total 906.38 last 9.61) - open/close 17770 (time 1789586963.60 total 917.00 last 16.28) - open/close 17881 (time 1789586974.16 total 927.56 last 10.51) - open/close 18085 (time 1789586984.65 total 938.05 last 19.44) - open/close 18347 (time 1789586994.66 total 948.06 last 26.19) - open/close 18576 (time 1789587004.71 total 958.11 last 22.77) - open/close 19138 (time 1789587014.74 total 968.14 last 56.06) - open/close 19600 (time 1789587024.76 total 978.16 last 46.10) - open/close 20000 (time 1789587033.65 total 987.05 last 44.97) - open/close 20590 (time 1789587043.68 total 997.08 last 58.85) - open/close 21211 (time 1789587053.68 total 1007.08 last 62.09) - open/close 21920 (time 1789587063.69 total 1017.09 last 70.79) - open/close 22399 (time 1789587073.71 total 1027.11 last 47.83) - open/close 22946 (time 1789587083.73 total 1037.13 last 54.61) - open/close 23431 (time 1789587093.74 total 1047.15 last 48.41) - open/close 24007 (time 1789587103.76 total 1057.16 last 57.53) - open/close 24616 (time 1789587113.76 total 1067.16 last 60.87) - open/close 25239 (time 1789587123.76 total 1077.17 last 62.28) - open/close 25909 (time 1789587133.79 total 1087.19 last 66.82) - open/close 26531 (time 1789587143.82 total 1097.22 last 62.03) - open/close 27208 (time 1789587153.84 total 1107.24 last 67.59) - open/close 27905 (time 1789587163.85 total 1117.25 last 69.62) - open/close 28601 (time 1789587173.86 total 1127.26 last 69.54) - open/close 29344 (time 1789587183.86 total 1137.26 last 74.27) - open/close 30000 (time 1789587193.32 total 1146.72 last 69.34) - open/close 30642 (time 1789587203.33 total 1156.74 last 64.10) - open/close 31184 (time 1789587213.36 total 1166.76 last 54.07) - open/close 31768 (time 1789587223.36 total 1176.76 last 58.37) - open/close 32353 (time 1789587233.37 total 1186.77 last 58.44) - open/close 32923 (time 1789587243.38 total 1196.78 last 56.96) - open/close 33526 (time 1789587253.39 total 1206.79 last 60.23) - open/close 34129 (time 1789587263.40 total 1216.80 last 60.28) - open/close 34880 (time 1789587273.40 total 1226.80 last 75.07) - open/close 35567 (time 1789587283.41 total 1236.81 last 68.63) - open/close 36190 (time 1789587293.42 total 1246.82 last 62.28) - open/close 36826 (time 1789587303.42 total 1256.82 last 63.55) - open/close 37335 (time 1789587313.43 total 1266.83 last 50.85) - open/close 37834 (time 1789587323.44 total 1276.84 last 49.88) - open/close 38307 (time 1789587333.45 total 1286.85 last 47.22) - open/close 38849 (time 1789587343.47 total 1296.87 last 54.12) - open/close 39501 (time 1789587353.47 total 1306.87 last 65.17) - open/close 40000 (time 1789587361.39 total 1314.79 last 63.04) - open/close 40661 (time 1789587371.39 total 1324.79 last 66.07) - open/close 41284 (time 1789587381.44 total 1334.84 last 62.03) - open/close 41898 (time 1789587391.45 total 1344.85 last 61.33) - open/close 42545 (time 1789587401.46 total 1354.86 last 64.62) - open/close 43175 (time 1789587411.48 total 1364.88 last 62.90) - open/close 43807 (time 1789587421.48 total 1374.88 last 63.17) - open/close 44440 (time 1789587431.51 total 1384.91 last 63.11) - open/close 45025 (time 1789587441.53 total 1394.93 last 58.39) - open/close 45613 (time 1789587451.53 total 1404.93 last 58.78) - open/close 46228 (time 1789587461.55 total 1414.95 last 61.39) - open/close 46840 (time 1789587471.55 total 1424.95 last 61.20) - open/close 47517 (time 1789587481.57 total 1434.97 last 67.61) - open/close 48202 (time 1789587491.57 total 1444.97 last 68.48) - open/close 48896 (time 1789587501.58 total 1454.98 last 69.30) - open/close 49561 (time 1789587511.60 total 1465.00 last 66.38) total: 50000 open/close in 1473.01 seconds: 33.94 ops/second - unlinked 0 (time 1789587524 ; total 0 ; last 0) - unlinked 10000 (time 1789587708 ; total 184 ; last 184) - unlinked 20000 (time 1789587874 ; total 350 ; last 166) - unlinked 30000 (time 1789588047 ; total 523 ; last 173) - unlinked 40000 (time 1789588224 ; total 700 ; last 177) total: 50000 unlinks in 881 seconds: 56.753689 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg312-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg312-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server unloading modules via unload_modules_local on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing unload_modules_local oleg312-server: kmemleak disabled modules unloaded. pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2