== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:56:36 (1789300596) start mds service on oleg143-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: kmemleak disabled oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg143-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg143-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1789300685.38 total 10.13 last 22.60) - open/close 439 (time 1789300695.80 total 20.55 last 20.15) - open/close 635 (time 1789300708.07 total 32.83 last 15.97) - open/close 756 (time 1789300719.01 total 43.76 last 11.07) - open/close 897 (time 1789300730.01 total 54.76 last 12.81) - open/close 1017 (time 1789300740.91 total 65.66 last 11.01) - open/close 1176 (time 1789300751.96 total 76.72 last 14.38) - open/close 1369 (time 1789300761.97 total 86.72 last 19.29) - open/close 1517 (time 1789300776.59 total 101.34 last 10.12) - open/close 1702 (time 1789300786.72 total 111.47 last 18.27) - open/close 1980 (time 1789300797.08 total 121.83 last 26.83) - open/close 2161 (time 1789300807.94 total 132.70 last 16.66) - open/close 2412 (time 1789300818.00 total 142.75 last 24.97) - open/close 2595 (time 1789300828.02 total 152.77 last 18.26) - open/close 2781 (time 1789300838.04 total 162.79 last 18.56) - open/close 3051 (time 1789300848.91 total 173.66 last 24.83) - open/close 3318 (time 1789300858.93 total 183.68 last 26.64) - open/close 3515 (time 1789300869.98 total 194.73 last 17.83) - open/close 3766 (time 1789300880.88 total 205.63 last 23.03) - open/close 3900 (time 1789300890.89 total 215.64 last 13.38) - open/close 4052 (time 1789300900.90 total 225.65 last 15.18) - open/close 4171 (time 1789300911.20 total 235.95 last 11.56) - open/close 4339 (time 1789300921.75 total 246.50 last 15.92) - open/close 4570 (time 1789300932.03 total 256.78 last 22.48) - open/close 4835 (time 1789300942.04 total 266.79 last 26.47) - open/close 5069 (time 1789300952.60 total 277.35 last 22.15) - open/close 5307 (time 1789300963.34 total 288.09 last 22.17) - open/close 5332 (time 1789300974.23 total 298.98 last 2.30) - open/close 5549 (time 1789300984.54 total 309.30 last 21.04) - open/close 5884 (time 1789300995.37 total 320.12 last 30.94) - open/close 6143 (time 1789301005.96 total 330.72 last 24.45) - open/close 6318 (time 1789301016.98 total 341.73 last 15.89) - open/close 6523 (time 1789301027.35 total 352.11 last 19.76) - open/close 6771 (time 1789301037.71 total 362.46 last 23.94) - open/close 7010 (time 1789301048.44 total 373.19 last 22.28) - open/close 7272 (time 1789301058.44 total 383.20 last 26.18) - open/close 7436 (time 1789301069.09 total 393.84 last 15.41) - open/close 7639 (time 1789301079.40 total 404.15 last 19.68) - open/close 7820 (time 1789301089.55 total 414.30 last 17.83) - open/close 8004 (time 1789301100.25 total 425.00 last 17.19) - open/close 8196 (time 1789301110.35 total 435.10 last 19.01) - open/close 8232 (time 1789301120.79 total 445.54 last 3.45) - open/close 8396 (time 1789301131.80 total 456.55 last 14.90) - open/close 8558 (time 1789301142.35 total 467.11 last 15.35) - open/close 8772 (time 1789301155.47 total 480.23 last 16.31) - open/close 9006 (time 1789301165.51 total 490.26 last 23.32) - open/close 9204 (time 1789301177.84 total 502.59 last 16.06) - open/close 9376 (time 1789301187.84 total 512.60 last 17.19) - open/close 9599 (time 1789301197.86 total 522.61 last 22.26) - open/close 9822 (time 1789301207.88 total 532.63 last 22.27) - open/close 10000 (time 1789301215.16 total 539.92 last 24.42) - open/close 10220 (time 1789301225.59 total 550.34 last 21.10) - open/close 10365 (time 1789301236.07 total 560.82 last 13.84) - open/close 10553 (time 1789301246.08 total 570.83 last 18.79) - open/close 10674 (time 1789301256.08 total 580.84 last 12.09) - open/close 10748 (time 1789301266.67 total 591.42 last 6.99) - open/close 10865 (time 1789301276.67 total 601.42 last 11.70) - open/close 11039 (time 1789301288.35 total 613.11 last 14.89) - open/close 11237 (time 1789301298.36 total 623.12 last 19.78) - open/close 11494 (time 1789301309.02 total 633.77 last 24.11) - open/close 11550 (time 1789301319.11 total 643.86 last 5.55) - open/close 11719 (time 1789301330.12 total 654.87 last 15.35) - open/close 11780 (time 1789301341.03 total 665.78 last 5.59) - open/close 11946 (time 1789301351.44 total 676.20 last 15.94) - open/close 12179 (time 1789301361.45 total 686.20 last 23.29) - open/close 12431 (time 1789301371.46 total 696.21 last 25.16) - open/close 12673 (time 1789301384.08 total 708.83 last 19.18) - open/close 12833 (time 1789301395.85 total 720.61 last 13.59) - open/close 13062 (time 1789301406.68 total 731.43 last 21.15) - open/close 13196 (time 1789301416.69 total 741.44 last 13.39) - open/close 13420 (time 1789301426.76 total 751.51 last 22.25) - open/close 13618 (time 1789301437.65 total 762.40 last 18.19) - open/close 13735 (time 1789301447.65 total 772.41 last 11.69) - open/close 13881 (time 1789301457.77 total 782.52 last 14.43) - open/close 14058 (time 1789301467.77 total 792.52 last 17.70) - open/close 14323 (time 1789301477.78 total 802.53 last 26.48) - open/close 14542 (time 1789301487.80 total 812.56 last 21.85) - open/close 14757 (time 1789301497.81 total 822.57 last 21.48) - open/close 15033 (time 1789301508.70 total 833.45 last 25.36) - open/close 15218 (time 1789301518.90 total 843.65 last 18.13) - open/close 15465 (time 1789301529.53 total 854.28 last 23.24) - open/close 15754 (time 1789301539.93 total 864.69 last 27.77) - open/close 15896 (time 1789301550.62 total 875.37 last 13.29) - open/close 16132 (time 1789301560.63 total 885.38 last 23.58) - open/close 16371 (time 1789301571.57 total 896.32 last 21.84) - open/close 16508 (time 1789301581.73 total 906.48 last 13.48) - open/close 16739 (time 1789301592.67 total 917.42 last 21.12) - open/close 16877 (time 1789301602.68 total 927.43 last 13.79) - open/close 17090 (time 1789301613.71 total 938.47 last 19.30) - open/close 17292 (time 1789301624.27 total 949.02 last 19.13) - open/close 17430 (time 1789301634.49 total 959.24 last 13.51) - open/close 17704 (time 1789301644.49 total 969.25 last 27.39) - open/close 17947 (time 1789301655.28 total 980.03 last 22.54) - open/close 18187 (time 1789301665.28 total 990.03 last 24.00) - open/close 18413 (time 1789301676.21 total 1000.96 last 20.68) - open/close 18762 (time 1789301686.21 total 1010.96 last 34.90) - open/close 19568 (time 1789301696.22 total 1020.97 last 80.54) - open/close 20000 (time 1789301702.90 total 1027.66 last 64.62) - open/close 20863 (time 1789301712.91 total 1037.66 last 86.27) - open/close 21813 (time 1789301722.91 total 1047.66 last 94.96) - open/close 22627 (time 1789301732.92 total 1057.67 last 81.30) - open/close 23268 (time 1789301742.93 total 1067.68 last 64.07) - open/close 24180 (time 1789301752.94 total 1077.69 last 91.09) - open/close 24936 (time 1789301762.95 total 1087.70 last 75.52) - open/close 25796 (time 1789301772.96 total 1097.71 last 85.93) - open/close 26533 (time 1789301782.96 total 1107.71 last 73.66) - open/close 27171 (time 1789301792.98 total 1117.73 last 63.69) - open/close 27965 (time 1789301802.98 total 1127.73 last 79.38) - open/close 28668 (time 1789301812.99 total 1137.74 last 70.24) - open/close 29419 (time 1789301823.00 total 1147.76 last 75.00) - open/close 30000 (time 1789301831.54 total 1156.29 last 68.09) - open/close 30577 (time 1789301841.55 total 1166.30 last 57.63) - open/close 31298 (time 1789301851.56 total 1176.31 last 72.03) - open/close 31855 (time 1789301861.57 total 1186.32 last 55.63) - open/close 32529 (time 1789301871.61 total 1196.37 last 67.12) - open/close 33131 (time 1789301881.63 total 1206.38 last 60.10) - open/close 33882 (time 1789301891.63 total 1216.39 last 75.07) - open/close 34705 (time 1789301901.63 total 1226.39 last 82.30) - open/close 35520 (time 1789301911.64 total 1236.39 last 81.49) - open/close 36186 (time 1789301921.64 total 1246.39 last 66.58) - open/close 36933 (time 1789301931.65 total 1256.40 last 74.65) - open/close 37738 (time 1789301941.65 total 1266.40 last 80.48) - open/close 38579 (time 1789301951.65 total 1276.40 last 84.09) - open/close 39351 (time 1789301961.67 total 1286.42 last 77.05) - open/close 40000 (time 1789301969.32 total 1294.07 last 84.86) - open/close 40771 (time 1789301979.32 total 1304.07 last 77.06) - open/close 41659 (time 1789301989.35 total 1314.10 last 88.56) - open/close 42481 (time 1789301999.36 total 1324.11 last 82.12) - open/close 43307 (time 1789302009.37 total 1334.12 last 82.55) - open/close 44085 (time 1789302019.38 total 1344.14 last 77.65) - open/close 44925 (time 1789302029.39 total 1354.14 last 83.97) - open/close 45718 (time 1789302039.39 total 1364.15 last 79.26) - open/close 46508 (time 1789302049.40 total 1374.15 last 78.97) - open/close 47298 (time 1789302059.40 total 1384.16 last 78.95) - open/close 48013 (time 1789302069.41 total 1394.16 last 71.45) - open/close 48854 (time 1789302079.41 total 1404.17 last 84.07) - open/close 49499 (time 1789302089.42 total 1414.17 last 64.46) total: 50000 open/close in 1420.88 seconds: 35.19 ops/second - unlinked 0 (time 1789302100 ; total 0 ; last 0) - unlinked 10000 (time 1789302232 ; total 132 ; last 132) - unlinked 20000 (time 1789302369 ; total 269 ; last 137) - unlinked 30000 (time 1789302524 ; total 424 ; last 155) - unlinked 40000 (time 1789302657 ; total 557 ; last 133) total: 50000 unlinks in 684 seconds: 73.099419 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: kmemleak disabled modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2