== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:35:20 (1789587320) start mds service on oleg443-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 16 09:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: kmemleak disabled oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg443-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg443-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg443-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg443-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 215 (time 1789587438.99 total 10.21 last 21.05) - open/close 344 (time 1789587449.04 total 20.26 last 12.84) - open/close 534 (time 1789587460.06 total 31.29 last 17.24) - open/close 739 (time 1789587470.15 total 41.38 last 20.31) - open/close 828 (time 1789587480.99 total 52.21 last 8.21) - open/close 981 (time 1789587493.84 total 65.07 last 11.90) - open/close 1121 (time 1789587504.36 total 75.58 last 13.31) - open/close 1285 (time 1789587514.37 total 85.59 last 16.39) - open/close 1420 (time 1789587524.93 total 96.16 last 12.78) - open/close 1633 (time 1789587534.94 total 106.17 last 21.28) - open/close 1826 (time 1789587544.94 total 116.17 last 19.30) - open/close 2034 (time 1789587555.54 total 126.77 last 19.63) - open/close 2268 (time 1789587566.54 total 137.77 last 21.27) - open/close 2459 (time 1789587576.65 total 147.87 last 18.90) - open/close 2688 (time 1789587587.40 total 158.63 last 21.29) - open/close 2839 (time 1789587598.23 total 169.46 last 13.94) - open/close 3033 (time 1789587609.24 total 180.47 last 17.62) - open/close 3280 (time 1789587619.26 total 190.49 last 24.64) - open/close 3415 (time 1789587631.57 total 202.80 last 10.97) - open/close 3576 (time 1789587642.13 total 213.35 last 15.26) - open/close 3750 (time 1789587652.81 total 224.04 last 16.28) - open/close 3931 (time 1789587663.26 total 234.48 last 17.33) - open/close 4005 (time 1789587673.27 total 244.50 last 7.39) - open/close 4116 (time 1789587683.34 total 254.57 last 11.03) - open/close 4232 (time 1789587693.35 total 264.57 last 11.59) - open/close 4334 (time 1789587703.45 total 274.67 last 10.10) - open/close 4402 (time 1789587715.65 total 286.88 last 5.57) - open/close 4553 (time 1789587726.18 total 297.40 last 14.35) - open/close 4635 (time 1789587736.84 total 308.06 last 7.69) - open/close 4811 (time 1789587746.84 total 318.07 last 17.59) - open/close 4948 (time 1789587757.34 total 328.56 last 13.06) - open/close 5200 (time 1789587767.34 total 338.56 last 25.20) - open/close 5419 (time 1789587777.77 total 349.00 last 20.99) - open/close 5542 (time 1789587787.78 total 359.01 last 12.29) - open/close 5759 (time 1789587798.53 total 369.76 last 20.18) - open/close 5940 (time 1789587809.04 total 380.27 last 17.22) - open/close 6121 (time 1789587819.07 total 390.30 last 18.05) - open/close 6346 (time 1789587829.08 total 400.31 last 22.47) - open/close 6464 (time 1789587840.01 total 411.23 last 10.81) - open/close 6565 (time 1789587850.77 total 422.00 last 9.38) - open/close 6718 (time 1789587861.38 total 432.61 last 14.42) - open/close 6808 (time 1789587871.70 total 442.92 last 8.73) - open/close 6939 (time 1789587881.70 total 452.92 last 13.10) - open/close 7118 (time 1789587892.44 total 463.67 last 16.66) - open/close 7221 (time 1789587902.47 total 473.70 last 10.27) - open/close 7279 (time 1789587912.73 total 483.95 last 5.66) - open/close 7442 (time 1789587922.84 total 494.06 last 16.12) - open/close 7609 (time 1789587932.84 total 504.07 last 16.69) - open/close 7667 (time 1789587943.82 total 515.05 last 5.28) - open/close 7819 (time 1789587956.63 total 527.86 last 11.86) - open/close 7862 (time 1789587967.20 total 538.43 last 4.07) - open/close 8058 (time 1789587977.48 total 548.71 last 19.07) - open/close 8222 (time 1789587987.49 total 558.71 last 16.39) - open/close 8287 (time 1789587998.09 total 569.32 last 6.13) - open/close 8316 (time 1789588008.11 total 579.33 last 2.90) - open/close 8514 (time 1789588018.45 total 589.67 last 19.15) - open/close 8672 (time 1789588028.58 total 599.80 last 15.60) - open/close 8711 (time 1789588039.07 total 610.29 last 3.72) - open/close 8840 (time 1789588049.92 total 621.15 last 11.88) - open/close 8960 (time 1789588059.96 total 631.19 last 11.95) - open/close 9095 (time 1789588069.97 total 641.20 last 13.48) - open/close 9184 (time 1789588079.99 total 651.22 last 8.88) - open/close 9369 (time 1789588090.14 total 661.36 last 18.24) - open/close 9555 (time 1789588100.50 total 671.73 last 17.95) - open/close 9648 (time 1789588111.44 total 682.67 last 8.50) - open/close 9710 (time 1789588121.46 total 692.68 last 6.19) - open/close 9856 (time 1789588132.00 total 703.22 last 13.85) - open/close 10000 (time 1789588140.12 total 711.34 last 17.73) - open/close 10117 (time 1789588151.01 total 722.24 last 10.74) - open/close 10264 (time 1789588162.04 total 733.26 last 13.33) - open/close 10357 (time 1789588172.06 total 743.29 last 9.28) - open/close 10466 (time 1789588182.07 total 753.30 last 10.89) - open/close 10620 (time 1789588192.64 total 763.87 last 14.57) - open/close 10765 (time 1789588202.66 total 773.88 last 14.48) - open/close 10856 (time 1789588213.66 total 784.88 last 8.27) - open/close 11046 (time 1789588224.99 total 796.22 last 16.76) - open/close 11209 (time 1789588235.76 total 806.98 last 15.14) - open/close 11333 (time 1789588246.47 total 817.70 last 11.57) - open/close 11409 (time 1789588256.79 total 828.02 last 7.36) - open/close 11533 (time 1789588271.09 total 842.31 last 8.67) - open/close 11626 (time 1789588281.81 total 853.04 last 8.67) - open/close 11767 (time 1789588294.37 total 865.59 last 11.23) - open/close 11944 (time 1789588305.35 total 876.57 last 16.12) - open/close 12033 (time 1789588316.10 total 887.33 last 8.27) - open/close 12196 (time 1789588326.29 total 897.52 last 16.00) - open/close 12323 (time 1789588336.55 total 907.78 last 12.38) - open/close 12402 (time 1789588347.88 total 919.11 last 6.97) - open/close 12671 (time 1789588358.68 total 929.90 last 24.92) - open/close 12871 (time 1789588369.59 total 940.82 last 18.32) - open/close 13084 (time 1789588382.38 total 953.61 last 16.65) - open/close 13241 (time 1789588392.41 total 963.64 last 15.65) - open/close 13332 (time 1789588403.21 total 974.43 last 8.43) - open/close 13412 (time 1789588414.44 total 985.67 last 7.12) - open/close 13526 (time 1789588424.47 total 995.70 last 11.37) - open/close 13672 (time 1789588434.49 total 1005.72 last 14.57) - open/close 13756 (time 1789588445.22 total 1016.45 last 7.83) - open/close 13828 (time 1789588455.24 total 1026.47 last 7.18) - open/close 13995 (time 1789588467.09 total 1038.32 last 14.09) - open/close 14177 (time 1789588477.10 total 1048.32 last 18.20) - open/close 14415 (time 1789588487.11 total 1058.34 last 23.76) - open/close 14667 (time 1789588497.73 total 1068.96 last 23.72) - open/close 14862 (time 1789588508.81 total 1080.04 last 17.60) - open/close 15052 (time 1789588519.19 total 1090.41 last 18.31) - open/close 15173 (time 1789588529.20 total 1100.42 last 12.09) - open/close 15315 (time 1789588540.18 total 1111.40 last 12.93) - open/close 15617 (time 1789588550.20 total 1121.43 last 30.12) - open/close 16043 (time 1789588560.23 total 1131.45 last 42.50) - open/close 16491 (time 1789588570.23 total 1141.46 last 44.78) - open/close 17122 (time 1789588580.24 total 1151.47 last 63.03) - open/close 17759 (time 1789588590.25 total 1161.48 last 63.62) - open/close 18390 (time 1789588600.26 total 1171.49 last 63.04) - open/close 19016 (time 1789588610.28 total 1181.50 last 62.52) - open/close 19605 (time 1789588620.29 total 1191.52 last 58.80) - open/close 20000 (time 1789588627.58 total 1198.81 last 54.18) - open/close 20563 (time 1789588637.65 total 1208.88 last 55.91) - open/close 21072 (time 1789588647.67 total 1218.89 last 50.82) - open/close 21531 (time 1789588657.70 total 1228.93 last 45.75) - open/close 22062 (time 1789588667.71 total 1238.94 last 53.04) - open/close 22687 (time 1789588677.71 total 1248.94 last 62.50) - open/close 23231 (time 1789588687.72 total 1258.95 last 54.36) - open/close 23853 (time 1789588697.76 total 1268.99 last 61.92) - open/close 24425 (time 1789588707.77 total 1278.99 last 57.18) - open/close 25074 (time 1789588717.77 total 1289.00 last 64.89) - open/close 25712 (time 1789588727.77 total 1299.00 last 63.80) - open/close 26307 (time 1789588737.77 total 1309.00 last 59.48) - open/close 26993 (time 1789588747.78 total 1319.01 last 68.53) - open/close 27621 (time 1789588757.79 total 1329.02 last 62.76) - open/close 28252 (time 1789588767.80 total 1339.02 last 63.07) - open/close 28833 (time 1789588777.80 total 1349.02 last 58.08) - open/close 29361 (time 1789588787.80 total 1359.03 last 52.77) - open/close 29984 (time 1789588797.81 total 1369.04 last 62.24) - open/close 30517 (time 1789588807.83 total 1379.06 last 53.19) - open/close 31101 (time 1789588817.85 total 1389.07 last 58.33) - open/close 31608 (time 1789588827.86 total 1399.09 last 50.62) - open/close 32215 (time 1789588837.88 total 1409.10 last 60.62) - open/close 32687 (time 1789588847.88 total 1419.11 last 47.19) - open/close 33236 (time 1789588857.89 total 1429.12 last 54.84) - open/close 33770 (time 1789588867.89 total 1439.12 last 53.39) - open/close 34307 (time 1789588877.90 total 1449.12 last 53.67) - open/close 34977 (time 1789588887.91 total 1459.14 last 66.89) - open/close 35551 (time 1789588897.93 total 1469.15 last 57.32) - open/close 36184 (time 1789588907.95 total 1479.17 last 63.18) - open/close 36689 (time 1789588917.95 total 1489.17 last 50.49) - open/close 37299 (time 1789588927.95 total 1499.17 last 60.99) - open/close 37895 (time 1789588937.97 total 1509.19 last 59.49) - open/close 38475 (time 1789588948.00 total 1519.22 last 57.83) - open/close 39038 (time 1789588958.00 total 1529.23 last 56.26) - open/close 39593 (time 1789588968.02 total 1539.24 last 55.42) - open/close 40000 (time 1789588974.49 total 1545.72 last 62.87) - open/close 40563 (time 1789588984.51 total 1555.73 last 56.21) - open/close 41273 (time 1789588994.51 total 1565.74 last 70.97) - open/close 41907 (time 1789589004.52 total 1575.74 last 63.37) - open/close 42498 (time 1789589014.55 total 1585.77 last 58.92) - open/close 43025 (time 1789589024.55 total 1595.77 last 52.69) - open/close 43559 (time 1789589034.55 total 1605.78 last 53.38) - open/close 44208 (time 1789589044.57 total 1615.80 last 64.77) - open/close 44711 (time 1789589054.59 total 1625.82 last 50.21) - open/close 45274 (time 1789589064.60 total 1635.83 last 56.25) - open/close 45833 (time 1789589074.61 total 1645.83 last 55.86) - open/close 46341 (time 1789589084.61 total 1655.84 last 50.77) - open/close 46941 (time 1789589094.64 total 1665.87 last 59.82) - open/close 47563 (time 1789589104.68 total 1675.90 last 62.01) - open/close 48179 (time 1789589114.68 total 1685.91 last 61.55) - open/close 48798 (time 1789589124.69 total 1695.91 last 61.87) - open/close 49421 (time 1789589134.69 total 1705.92 last 62.26) total: 50000 open/close in 1714.71 seconds: 29.16 ops/second - unlinked 0 (time 1789589148 ; total 0 ; last 0) - unlinked 10000 (time 1789589327 ; total 179 ; last 179) - unlinked 20000 (time 1789589506 ; total 358 ; last 179) - unlinked 30000 (time 1789589687 ; total 539 ; last 181) - unlinked 40000 (time 1789589860 ; total 712 ; last 173) total: 50000 unlinks in 879 seconds: 56.882820 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local oleg443-server: kmemleak disabled modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2