== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:46:19 (1788515179) start mds service on oleg135-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1933656 Sep 3 13:58 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: kmemleak disabled oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg135-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg135-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1788515297.19 total 10.90 last 22.29) - open/close 484 (time 1788515310.19 total 23.89 last 18.55) - open/close 734 (time 1788515320.87 total 34.58 last 23.40) - open/close 1031 (time 1788515331.64 total 45.34 last 27.58) - open/close 1280 (time 1788515342.26 total 55.96 last 23.44) - open/close 1488 (time 1788515352.95 total 66.66 last 19.45) - open/close 1718 (time 1788515363.33 total 77.04 last 22.15) - open/close 1948 (time 1788515373.77 total 87.48 last 22.04) - open/close 2181 (time 1788515383.77 total 97.48 last 23.29) - open/close 2460 (time 1788515393.78 total 107.48 last 27.89) - open/close 2672 (time 1788515404.65 total 118.36 last 19.49) - open/close 2910 (time 1788515417.33 total 131.04 last 18.78) - open/close 3151 (time 1788515427.62 total 141.32 last 23.42) - open/close 3400 (time 1788515437.63 total 151.33 last 24.88) - open/close 3622 (time 1788515448.23 total 161.94 last 20.93) - open/close 3900 (time 1788515459.04 total 172.75 last 25.71) - open/close 4154 (time 1788515469.42 total 183.12 last 24.48) - open/close 4373 (time 1788515479.59 total 193.30 last 21.52) - open/close 4584 (time 1788515490.87 total 204.58 last 18.71) - open/close 4859 (time 1788515501.80 total 215.51 last 25.16) - open/close 5108 (time 1788515512.55 total 226.26 last 23.16) - open/close 5326 (time 1788515522.59 total 236.29 last 21.72) - open/close 5564 (time 1788515532.60 total 246.31 last 23.77) - open/close 5649 (time 1788515542.90 total 256.61 last 8.25) - open/close 5710 (time 1788515552.95 total 266.66 last 6.07) - open/close 5948 (time 1788515562.96 total 276.67 last 23.77) - open/close 6174 (time 1788515573.60 total 287.30 last 21.26) - open/close 6451 (time 1788515584.37 total 298.07 last 25.72) - open/close 6717 (time 1788515594.37 total 308.08 last 26.59) - open/close 6919 (time 1788515604.38 total 318.08 last 20.19) - open/close 7093 (time 1788515617.19 total 330.90 last 13.58) - open/close 7269 (time 1788515627.21 total 340.92 last 17.56) - open/close 7347 (time 1788515637.31 total 351.01 last 7.73) - open/close 7463 (time 1788515648.36 total 362.07 last 10.49) - open/close 7629 (time 1788515658.77 total 372.47 last 15.96) - open/close 7846 (time 1788515669.49 total 383.20 last 20.24) - open/close 8088 (time 1788515679.50 total 393.20 last 24.19) - open/close 8324 (time 1788515689.51 total 403.22 last 23.57) - open/close 8512 (time 1788515699.53 total 413.24 last 18.75) - open/close 8641 (time 1788515709.54 total 423.25 last 12.89) - open/close 8828 (time 1788515720.32 total 434.03 last 17.35) - open/close 8920 (time 1788515732.24 total 445.95 last 7.72) - open/close 9147 (time 1788515742.25 total 455.96 last 22.67) - open/close 9390 (time 1788515752.26 total 465.96 last 24.29) - open/close 9601 (time 1788515762.30 total 476.01 last 21.00) - open/close 9802 (time 1788515772.87 total 486.58 last 19.02) - open/close 10000 (time 1788515781.67 total 495.38 last 22.50) - open/close 10195 (time 1788515793.12 total 506.82 last 17.04) - open/close 10419 (time 1788515804.08 total 517.78 last 20.44) - open/close 10683 (time 1788515814.89 total 528.59 last 24.42) - open/close 10924 (time 1788515825.77 total 539.47 last 22.15) - open/close 11200 (time 1788515836.62 total 550.33 last 25.43) - open/close 11432 (time 1788515846.64 total 560.34 last 23.16) - open/close 11757 (time 1788515856.65 total 570.35 last 32.46) - open/close 11995 (time 1788515866.65 total 580.36 last 23.79) - open/close 12199 (time 1788515876.66 total 590.36 last 20.39) - open/close 12428 (time 1788515886.66 total 600.37 last 22.89) - open/close 12633 (time 1788515897.64 total 611.34 last 18.68) - open/close 12660 (time 1788515907.95 total 621.65 last 2.62) - open/close 12790 (time 1788515918.57 total 632.27 last 12.24) - open/close 12988 (time 1788515929.46 total 643.17 last 18.17) - open/close 13217 (time 1788515939.47 total 653.18 last 22.88) - open/close 13427 (time 1788515949.48 total 663.18 last 20.99) - open/close 13654 (time 1788515959.98 total 673.69 last 21.61) - open/close 13837 (time 1788515971.00 total 684.71 last 16.60) - open/close 14102 (time 1788515981.01 total 694.72 last 26.48) - open/close 14340 (time 1788515991.08 total 704.79 last 23.63) - open/close 14568 (time 1788516001.87 total 715.57 last 21.14) - open/close 14801 (time 1788516011.87 total 725.57 last 23.30) - open/close 15080 (time 1788516021.89 total 735.59 last 27.85) - open/close 15238 (time 1788516032.88 total 746.58 last 14.37) - open/close 15518 (time 1788516043.65 total 757.36 last 25.99) - open/close 15685 (time 1788516053.88 total 767.59 last 16.32) - open/close 15879 (time 1788516063.90 total 777.61 last 19.36) - open/close 16102 (time 1788516074.28 total 787.99 last 21.48) - open/close 16321 (time 1788516084.29 total 797.99 last 21.90) - open/close 16444 (time 1788516095.39 total 809.09 last 11.08) - open/close 16621 (time 1788516106.16 total 819.86 last 16.43) - open/close 16792 (time 1788516116.16 total 829.86 last 17.10) - open/close 16920 (time 1788516126.86 total 840.57 last 11.96) - open/close 16984 (time 1788516137.14 total 850.85 last 6.23) - open/close 17060 (time 1788516147.86 total 861.56 last 7.09) - open/close 17209 (time 1788516157.90 total 871.61 last 14.83) - open/close 17344 (time 1788516168.77 total 882.48 last 12.42) - open/close 17480 (time 1788516178.87 total 892.58 last 13.47) - open/close 17742 (time 1788516188.87 total 902.58 last 26.20) - open/close 17962 (time 1788516198.88 total 912.58 last 21.98) - open/close 18222 (time 1788516208.91 total 922.61 last 25.92) - open/close 18414 (time 1788516219.81 total 933.52 last 17.60) - open/close 18651 (time 1788516230.69 total 944.40 last 21.78) - open/close 18858 (time 1788516240.96 total 954.66 last 20.17) - open/close 19112 (time 1788516251.93 total 965.63 last 23.15) - open/close 19720 (time 1788516261.93 total 975.64 last 60.76) - open/close 20000 (time 1788516265.83 total 979.54 last 71.79) - open/close 20622 (time 1788516275.85 total 989.56 last 62.08) - open/close 21250 (time 1788516285.85 total 999.56 last 62.80) - open/close 22013 (time 1788516295.86 total 1009.56 last 76.26) - open/close 22666 (time 1788516305.86 total 1019.57 last 65.29) - open/close 23476 (time 1788516315.88 total 1029.59 last 80.80) - open/close 23940 (time 1788516325.90 total 1039.61 last 46.33) - open/close 24464 (time 1788516335.90 total 1049.61 last 52.38) - open/close 25077 (time 1788516345.91 total 1059.62 last 61.27) - open/close 25792 (time 1788516355.92 total 1069.62 last 71.44) - open/close 26558 (time 1788516365.92 total 1079.63 last 76.58) - open/close 27403 (time 1788516375.93 total 1089.64 last 84.43) - open/close 28099 (time 1788516385.93 total 1099.64 last 69.56) - open/close 28815 (time 1788516395.95 total 1109.66 last 71.49) - open/close 29457 (time 1788516405.97 total 1119.68 last 64.04) - open/close 30000 (time 1788516415.95 total 1129.66 last 54.42) - open/close 30407 (time 1788516425.95 total 1139.66 last 40.70) - open/close 31301 (time 1788516435.95 total 1149.66 last 89.40) - open/close 32038 (time 1788516445.95 total 1159.66 last 73.70) - open/close 32720 (time 1788516455.96 total 1169.66 last 68.16) - open/close 33374 (time 1788516465.96 total 1179.67 last 65.38) - open/close 33879 (time 1788516475.97 total 1189.68 last 50.45) - open/close 34388 (time 1788516485.98 total 1199.69 last 50.83) - open/close 34985 (time 1788516495.99 total 1209.70 last 59.66) - open/close 35543 (time 1788516506.01 total 1219.71 last 55.71) - open/close 35993 (time 1788516516.01 total 1229.72 last 44.96) - open/close 36389 (time 1788516526.05 total 1239.76 last 39.44) - open/close 36976 (time 1788516536.06 total 1249.77 last 58.66) - open/close 37635 (time 1788516546.06 total 1259.77 last 65.88) - open/close 38196 (time 1788516556.08 total 1269.78 last 56.02) - open/close 38712 (time 1788516566.11 total 1279.81 last 51.46) - open/close 39097 (time 1788516576.12 total 1289.83 last 38.45) - open/close 39637 (time 1788516586.16 total 1299.86 last 53.80) - open/close 40000 (time 1788516594.91 total 1308.61 last 41.48) - open/close 40567 (time 1788516604.91 total 1318.62 last 56.66) - open/close 41226 (time 1788516614.92 total 1328.62 last 65.89) - open/close 41907 (time 1788516624.93 total 1338.63 last 68.02) - open/close 42425 (time 1788516634.94 total 1348.65 last 51.74) - open/close 42885 (time 1788516644.94 total 1358.65 last 45.99) - open/close 43447 (time 1788516654.95 total 1368.65 last 56.16) - open/close 44114 (time 1788516664.95 total 1378.65 last 66.70) - open/close 44867 (time 1788516674.95 total 1388.66 last 75.29) - open/close 45632 (time 1788516684.95 total 1398.66 last 76.46) - open/close 46330 (time 1788516694.96 total 1408.67 last 69.76) - open/close 47020 (time 1788516704.97 total 1418.67 last 68.96) - open/close 47897 (time 1788516714.97 total 1428.68 last 87.62) - open/close 48769 (time 1788516724.98 total 1438.69 last 87.14) - open/close 49576 (time 1788516734.99 total 1448.70 last 80.61) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1449.60 seconds: 34.24 ops/second - unlinked 0 (time 1788516741 ; total 0 ; last 0) - unlinked 10000 (time 1788516874 ; total 133 ; last 133) - unlinked 20000 (time 1788517004 ; total 263 ; last 130) - unlinked 30000 (time 1788517143 ; total 402 ; last 139) - unlinked 40000 (time 1788517285 ; total 544 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 678 seconds: 73.202065 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local oleg135-server: kmemleak disabled modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2