== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:05:01 (1790744701) start mds service on oleg436-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1947032 Sep 29 13:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: kmemleak disabled oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions oleg436-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg436-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg436-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg436-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg436-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg436-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 184 (time 1790744814.79 total 10.01 last 18.38) - open/close 313 (time 1790744825.73 total 20.94 last 11.80) - open/close 522 (time 1790744836.79 total 32.01 last 18.88) - open/close 726 (time 1790744846.80 total 42.02 last 20.39) - open/close 923 (time 1790744857.07 total 52.29 last 19.18) - open/close 1111 (time 1790744867.75 total 62.97 last 17.60) - open/close 1228 (time 1790744878.63 total 73.85 last 10.75) - open/close 1402 (time 1790744889.59 total 84.80 last 15.89) - open/close 1494 (time 1790744900.29 total 95.50 last 8.60) - open/close 1626 (time 1790744910.29 total 105.51 last 13.20) - open/close 1893 (time 1790744920.44 total 115.66 last 26.30) - open/close 2130 (time 1790744930.89 total 126.11 last 22.68) - open/close 2383 (time 1790744940.91 total 136.12 last 25.26) - open/close 2556 (time 1790744951.62 total 146.83 last 16.15) - open/close 2839 (time 1790744962.66 total 157.88 last 25.62) - open/close 2947 (time 1790744973.43 total 168.65 last 10.03) - open/close 3034 (time 1790744983.60 total 178.82 last 8.55) - open/close 3272 (time 1790744994.20 total 189.42 last 22.45) - open/close 3546 (time 1790745004.22 total 199.44 last 27.34) - open/close 3770 (time 1790745014.86 total 210.08 last 21.06) - open/close 3948 (time 1790745024.86 total 220.08 last 17.80) - open/close 4091 (time 1790745035.13 total 230.34 last 13.93) - open/close 4348 (time 1790745050.88 total 246.09 last 16.32) - open/close 4493 (time 1790745061.61 total 256.82 last 13.51) - open/close 4707 (time 1790745071.92 total 267.14 last 20.74) - open/close 4896 (time 1790745082.98 total 278.20 last 17.09) - open/close 5140 (time 1790745092.99 total 288.21 last 24.38) - open/close 5316 (time 1790745103.28 total 298.50 last 17.10) - open/close 5406 (time 1790745113.90 total 309.12 last 8.48) - open/close 5658 (time 1790745124.40 total 319.61 last 24.00) - open/close 5895 (time 1790745134.57 total 329.79 last 23.29) - open/close 6117 (time 1790745144.94 total 340.16 last 21.41) - open/close 6347 (time 1790745155.51 total 350.73 last 21.77) - open/close 6401 (time 1790745166.38 total 361.60 last 4.97) - open/close 6576 (time 1790745176.43 total 371.65 last 17.41) - open/close 6700 (time 1790745186.46 total 381.67 last 12.37) - open/close 6866 (time 1790745196.47 total 391.68 last 16.58) - open/close 7017 (time 1790745207.03 total 402.25 last 14.29) - open/close 7154 (time 1790745217.05 total 412.27 last 13.67) - open/close 7295 (time 1790745227.06 total 422.27 last 14.09) - open/close 7480 (time 1790745237.07 total 432.29 last 18.47) - open/close 7719 (time 1790745247.81 total 443.03 last 22.26) - open/close 7946 (time 1790745258.10 total 453.32 last 22.05) - open/close 8172 (time 1790745268.71 total 463.93 last 21.30) - open/close 8366 (time 1790745279.41 total 474.63 last 18.13) - open/close 8512 (time 1790745289.42 total 484.63 last 14.59) - open/close 8677 (time 1790745299.95 total 495.16 last 15.67) - open/close 8917 (time 1790745310.00 total 505.22 last 23.87) - open/close 9205 (time 1790745320.38 total 515.59 last 27.77) - open/close 9388 (time 1790745332.98 total 528.20 last 14.51) - open/close 9562 (time 1790745344.54 total 539.76 last 15.05) - open/close 9782 (time 1790745356.14 total 551.36 last 18.97) - open/close 9859 (time 1790745366.89 total 562.11 last 7.16) - open/close 10000 (time 1790745370.47 total 565.69 last 39.37) - open/close 10109 (time 1790745380.87 total 576.09 last 10.48) - open/close 10281 (time 1790745391.34 total 586.56 last 16.43) - open/close 10480 (time 1790745401.35 total 596.57 last 19.88) - open/close 10659 (time 1790745411.38 total 606.60 last 17.85) - open/close 10823 (time 1790745421.70 total 616.92 last 15.89) - open/close 10985 (time 1790745432.32 total 627.54 last 15.25) - open/close 11207 (time 1790745442.33 total 637.55 last 22.19) - open/close 11405 (time 1790745452.86 total 648.08 last 18.79) - open/close 11657 (time 1790745462.89 total 658.10 last 25.14) - open/close 11809 (time 1790745473.60 total 668.82 last 14.19) - open/close 12019 (time 1790745483.72 total 678.94 last 20.75) - open/close 12240 (time 1790745494.08 total 689.30 last 21.33) - open/close 12528 (time 1790745504.93 total 700.15 last 26.54) - open/close 12775 (time 1790745515.76 total 710.98 last 22.81) - open/close 12901 (time 1790745525.77 total 720.98 last 12.59) - open/close 13090 (time 1790745535.77 total 730.99 last 18.90) - open/close 13333 (time 1790745546.41 total 741.63 last 22.83) - open/close 13387 (time 1790745556.99 total 752.20 last 5.11) - open/close 13509 (time 1790745566.99 total 762.21 last 12.20) - open/close 13728 (time 1790745577.20 total 772.41 last 21.46) - open/close 14015 (time 1790745587.70 total 782.92 last 27.31) - open/close 14048 (time 1790745598.84 total 794.06 last 2.96) - open/close 14195 (time 1790745608.87 total 804.09 last 14.65) - open/close 14426 (time 1790745619.58 total 814.79 last 21.59) - open/close 14671 (time 1790745629.59 total 824.81 last 24.46) - open/close 14857 (time 1790745639.61 total 834.83 last 18.57) - open/close 15045 (time 1790745649.96 total 845.18 last 18.16) - open/close 15243 (time 1790745660.74 total 855.96 last 18.36) - open/close 15296 (time 1790745671.28 total 866.50 last 5.03) - open/close 15478 (time 1790745681.31 total 876.52 last 18.16) - open/close 15717 (time 1790745691.31 total 886.52 last 23.90) - open/close 15892 (time 1790745701.33 total 896.55 last 17.46) - open/close 16079 (time 1790745711.35 total 906.56 last 18.67) - open/close 16327 (time 1790745722.10 total 917.32 last 23.06) - open/close 16601 (time 1790745732.62 total 927.84 last 26.06) - open/close 16876 (time 1790745743.61 total 938.83 last 25.02) - open/close 17077 (time 1790745753.62 total 948.83 last 20.09) - open/close 17356 (time 1790745763.64 total 958.86 last 27.83) - open/close 17562 (time 1790745773.99 total 969.21 last 19.90) - open/close 17646 (time 1790745784.42 total 979.64 last 8.05) - open/close 17804 (time 1790745794.44 total 989.66 last 15.78) - open/close 18003 (time 1790745804.59 total 999.81 last 19.60) - open/close 18414 (time 1790745814.59 total 1009.81 last 41.10) - open/close 19123 (time 1790745824.59 total 1019.81 last 70.89) - open/close 19863 (time 1790745834.61 total 1029.82 last 73.91) - open/close 20554 (time 1790745844.61 total 1039.83 last 69.07) - open/close 21214 (time 1790745854.62 total 1049.84 last 65.94) - open/close 21785 (time 1790745864.63 total 1059.85 last 57.05) - open/close 22443 (time 1790745874.64 total 1069.86 last 65.74) - open/close 23168 (time 1790745884.66 total 1079.87 last 72.38) - open/close 23820 (time 1790745894.66 total 1089.88 last 65.16) - open/close 24493 (time 1790745904.67 total 1099.88 last 67.26) - open/close 25152 (time 1790745914.67 total 1109.89 last 65.88) - open/close 25816 (time 1790745924.67 total 1119.89 last 66.40) - open/close 26570 (time 1790745934.68 total 1129.89 last 75.35) - open/close 27293 (time 1790745944.69 total 1139.91 last 72.21) - open/close 28032 (time 1790745954.71 total 1149.92 last 73.76) - open/close 28706 (time 1790745964.72 total 1159.93 last 67.34) - open/close 29407 (time 1790745974.72 total 1169.94 last 70.08) - open/close 30000 (time 1790745983.56 total 1178.78 last 67.06) - open/close 30705 (time 1790745993.57 total 1188.79 last 70.44) - open/close 31473 (time 1790746003.58 total 1198.79 last 76.76) - open/close 32158 (time 1790746013.58 total 1208.80 last 68.44) - open/close 32910 (time 1790746023.59 total 1218.81 last 75.15) - open/close 33623 (time 1790746033.61 total 1228.83 last 71.18) - open/close 34305 (time 1790746043.62 total 1238.83 last 68.16) - open/close 35060 (time 1790746053.62 total 1248.84 last 75.45) - open/close 35713 (time 1790746063.63 total 1258.84 last 65.28) - open/close 36457 (time 1790746073.63 total 1268.85 last 74.34) - open/close 37164 (time 1790746083.64 total 1278.85 last 70.68) - open/close 37875 (time 1790746093.64 total 1288.86 last 71.05) - open/close 38584 (time 1790746103.65 total 1298.87 last 70.85) - open/close 39182 (time 1790746113.66 total 1308.87 last 59.77) - open/close 39890 (time 1790746123.67 total 1318.89 last 70.68) - open/close 40570 (time 1790746133.68 total 1328.89 last 67.97) - open/close 41246 (time 1790746143.68 total 1338.90 last 67.58) - open/close 41909 (time 1790746153.70 total 1348.92 last 66.18) - open/close 42588 (time 1790746163.70 total 1358.92 last 67.88) - open/close 43280 (time 1790746173.71 total 1368.92 last 69.17) - open/close 43966 (time 1790746183.71 total 1378.93 last 68.56) - open/close 44673 (time 1790746193.72 total 1388.93 last 70.68) - open/close 45380 (time 1790746203.72 total 1398.94 last 70.67) - open/close 46052 (time 1790746213.73 total 1408.94 last 67.17) - open/close 46773 (time 1790746223.73 total 1418.95 last 72.06) - open/close 47479 (time 1790746233.74 total 1428.96 last 70.55) - open/close 48181 (time 1790746243.74 total 1438.96 last 70.17) - open/close 48884 (time 1790746253.75 total 1448.97 last 70.23) - open/close 49586 (time 1790746263.76 total 1458.98 last 70.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1459.51 seconds: 34.01 ops/second - unlinked 0 (time 1790746268 ; total 0 ; last 0) - unlinked 10000 (time 1790746412 ; total 144 ; last 144) - unlinked 20000 (time 1790746554 ; total 286 ; last 142) - unlinked 30000 (time 1790746693 ; total 425 ; last 139) - unlinked 40000 (time 1790746833 ; total 565 ; last 140) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 704 seconds: 70.498581 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg436-server unloading modules via unload_modules_local on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local oleg436-server: kmemleak disabled modules unloaded. pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2