== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:51:47 (1789131107) start mds service on oleg435-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1789131199.49 total 12.29 last 10.98) - open/close 295 (time 1789131210.38 total 23.18 last 14.70) - open/close 344 (time 1789131220.54 total 33.34 last 4.82) - open/close 434 (time 1789131230.58 total 43.38 last 8.97) - open/close 551 (time 1789131241.40 total 54.19 last 10.82) - open/close 746 (time 1789131251.76 total 64.56 last 18.81) - open/close 986 (time 1789131261.77 total 74.56 last 23.99) - open/close 1238 (time 1789131271.98 total 84.78 last 24.66) - open/close 1447 (time 1789131282.43 total 95.23 last 20.02) - open/close 1627 (time 1789131294.26 total 107.06 last 15.21) - open/close 1749 (time 1789131304.28 total 117.08 last 12.17) - open/close 1973 (time 1789131315.19 total 127.99 last 20.53) - open/close 2221 (time 1789131326.13 total 138.93 last 22.67) - open/close 2416 (time 1789131336.15 total 148.95 last 19.47) - open/close 2643 (time 1789131346.15 total 158.95 last 22.69) - open/close 2848 (time 1789131356.34 total 169.14 last 20.12) - open/close 3088 (time 1789131367.09 total 179.89 last 22.33) - open/close 3238 (time 1789131377.75 total 190.55 last 14.07) - open/close 3363 (time 1789131387.75 total 200.55 last 12.49) - open/close 3536 (time 1789131397.82 total 210.62 last 17.19) - open/close 3647 (time 1789131408.24 total 221.04 last 10.65) - open/close 3804 (time 1789131419.07 total 231.87 last 14.49) - open/close 3914 (time 1789131429.09 total 241.89 last 10.98) - open/close 4049 (time 1789131440.07 total 252.87 last 12.29) - open/close 4117 (time 1789131450.09 total 262.89 last 6.79) - open/close 4215 (time 1789131460.14 total 272.94 last 9.75) - open/close 4376 (time 1789131470.86 total 283.66 last 15.01) - open/close 4429 (time 1789131483.14 total 295.93 last 4.32) - open/close 4626 (time 1789131493.45 total 306.25 last 19.10) - open/close 4746 (time 1789131503.46 total 316.26 last 11.99) - open/close 4869 (time 1789131516.45 total 329.25 last 9.47) - open/close 5104 (time 1789131526.58 total 339.38 last 23.20) - open/close 5333 (time 1789131537.07 total 349.87 last 21.84) - open/close 5547 (time 1789131549.18 total 361.98 last 17.66) - open/close 5663 (time 1789131559.85 total 372.65 last 10.87) - open/close 5844 (time 1789131569.87 total 382.67 last 18.08) - open/close 6068 (time 1789131579.88 total 392.67 last 22.38) - open/close 6267 (time 1789131589.88 total 402.68 last 19.88) - open/close 6444 (time 1789131600.51 total 413.31 last 16.65) - open/close 6688 (time 1789131610.52 total 423.32 last 24.39) - open/close 6920 (time 1789131621.26 total 434.06 last 21.60) - open/close 7160 (time 1789131631.27 total 444.07 last 23.97) - open/close 7388 (time 1789131641.97 total 454.77 last 21.30) - open/close 7580 (time 1789131651.98 total 464.77 last 19.19) - open/close 7779 (time 1789131662.38 total 475.18 last 19.13) - open/close 7978 (time 1789131672.76 total 485.56 last 19.17) - open/close 8220 (time 1789131683.65 total 496.45 last 22.23) - open/close 8464 (time 1789131693.66 total 506.46 last 24.38) - open/close 8645 (time 1789131703.85 total 516.65 last 17.76) - open/close 8842 (time 1789131714.54 total 527.34 last 18.43) - open/close 9059 (time 1789131724.54 total 537.34 last 21.69) - open/close 9316 (time 1789131735.99 total 548.79 last 22.45) - open/close 9548 (time 1789131746.15 total 558.95 last 22.84) - open/close 9685 (time 1789131756.17 total 568.97 last 13.67) - open/close 9891 (time 1789131766.20 total 579.00 last 20.55) - open/close 10000 (time 1789131770.68 total 583.48 last 24.32) - open/close 10160 (time 1789131781.56 total 594.36 last 14.70) - open/close 10378 (time 1789131791.57 total 604.37 last 21.79) - open/close 10530 (time 1789131801.98 total 614.78 last 14.59) - open/close 10602 (time 1789131812.07 total 624.87 last 7.14) - open/close 10666 (time 1789131822.40 total 635.19 last 6.20) - open/close 10786 (time 1789131832.42 total 645.22 last 11.97) - open/close 10926 (time 1789131843.31 total 656.11 last 12.86) - open/close 11002 (time 1789131853.94 total 666.74 last 7.15) - open/close 11129 (time 1789131864.30 total 677.10 last 12.26) - open/close 11224 (time 1789131874.31 total 687.11 last 9.49) - open/close 11403 (time 1789131884.80 total 697.60 last 17.07) - open/close 11549 (time 1789131894.80 total 707.60 last 14.60) - open/close 11780 (time 1789131905.13 total 717.93 last 22.36) - open/close 12012 (time 1789131915.68 total 728.48 last 21.99) - open/close 12197 (time 1789131926.26 total 739.06 last 17.48) - open/close 12479 (time 1789131936.28 total 749.08 last 28.15) - open/close 12682 (time 1789131946.30 total 759.10 last 20.26) - open/close 12922 (time 1789131956.30 total 769.10 last 24.00) - open/close 13040 (time 1789131966.47 total 779.27 last 11.60) - open/close 13239 (time 1789131976.47 total 789.27 last 19.89) - open/close 13445 (time 1789131986.94 total 799.74 last 19.69) - open/close 13639 (time 1789131998.33 total 811.13 last 17.04) - open/close 13869 (time 1789132009.00 total 821.80 last 21.55) - open/close 14088 (time 1789132020.61 total 833.41 last 18.86) - open/close 14214 (time 1789132030.71 total 843.51 last 12.47) - open/close 14457 (time 1789132041.64 total 854.44 last 22.23) - open/close 14562 (time 1789132052.34 total 865.14 last 9.82) - open/close 14757 (time 1789132062.36 total 875.16 last 19.46) - open/close 14885 (time 1789132072.39 total 885.19 last 12.76) - open/close 15108 (time 1789132082.73 total 895.53 last 21.56) - open/close 15363 (time 1789132092.74 total 905.54 last 25.48) - open/close 15596 (time 1789132102.78 total 915.57 last 23.22) - open/close 15776 (time 1789132113.06 total 925.86 last 17.50) - open/close 15981 (time 1789132123.19 total 935.99 last 20.25) - open/close 16164 (time 1789132134.01 total 946.81 last 16.91) - open/close 16328 (time 1789132144.83 total 957.63 last 15.15) - open/close 16495 (time 1789132155.85 total 968.65 last 15.16) - open/close 16708 (time 1789132166.39 total 979.19 last 20.21) - open/close 16841 (time 1789132176.40 total 989.20 last 13.29) - open/close 17051 (time 1789132187.33 total 1000.13 last 19.21) - open/close 17262 (time 1789132197.34 total 1010.14 last 21.09) - open/close 17499 (time 1789132207.35 total 1020.15 last 23.67) - open/close 18128 (time 1789132217.35 total 1030.15 last 62.89) - open/close 18711 (time 1789132227.36 total 1040.16 last 58.25) - open/close 19336 (time 1789132237.37 total 1050.17 last 62.40) - open/close 19957 (time 1789132247.38 total 1060.18 last 62.05) - open/close 20591 (time 1789132257.39 total 1070.19 last 63.33) - open/close 21207 (time 1789132267.40 total 1080.20 last 61.56) - open/close 21882 (time 1789132277.41 total 1090.21 last 67.43) - open/close 22512 (time 1789132287.42 total 1100.22 last 62.94) - open/close 23119 (time 1789132297.42 total 1110.22 last 60.68) - open/close 23712 (time 1789132307.44 total 1120.24 last 59.16) - open/close 24369 (time 1789132317.46 total 1130.26 last 65.61) - open/close 24960 (time 1789132327.46 total 1140.26 last 59.09) - open/close 25540 (time 1789132337.47 total 1150.27 last 57.93) - open/close 26199 (time 1789132347.48 total 1160.28 last 65.87) - open/close 26850 (time 1789132357.48 total 1170.28 last 65.06) - open/close 27532 (time 1789132367.49 total 1180.29 last 68.13) - open/close 28140 (time 1789132377.50 total 1190.30 last 60.77) - open/close 28782 (time 1789132387.51 total 1200.31 last 64.14) - open/close 29416 (time 1789132397.52 total 1210.32 last 63.30) - open/close 30000 (time 1789132406.21 total 1219.01 last 67.21) - open/close 30618 (time 1789132416.24 total 1229.04 last 61.61) - open/close 31274 (time 1789132426.26 total 1239.06 last 65.47) - open/close 31841 (time 1789132436.26 total 1249.06 last 56.69) - open/close 32456 (time 1789132446.27 total 1259.07 last 61.45) - open/close 33069 (time 1789132456.28 total 1269.08 last 61.26) - open/close 33700 (time 1789132466.28 total 1279.08 last 63.08) - open/close 34355 (time 1789132476.29 total 1289.09 last 65.42) - open/close 34972 (time 1789132486.30 total 1299.10 last 61.68) - open/close 35560 (time 1789132496.31 total 1309.11 last 58.73) - open/close 36199 (time 1789132506.32 total 1319.12 last 63.85) - open/close 36826 (time 1789132516.32 total 1329.12 last 62.66) - open/close 37379 (time 1789132526.33 total 1339.13 last 55.25) - open/close 38015 (time 1789132536.33 total 1349.13 last 63.58) - open/close 38660 (time 1789132546.35 total 1359.15 last 64.41) - open/close 39254 (time 1789132556.36 total 1369.16 last 59.35) - open/close 39961 (time 1789132566.36 total 1379.16 last 70.68) - open/close 40618 (time 1789132576.37 total 1389.17 last 65.63) - open/close 41290 (time 1789132586.38 total 1399.18 last 67.14) - open/close 41893 (time 1789132596.39 total 1409.19 last 60.24) - open/close 42511 (time 1789132606.39 total 1419.19 last 61.79) - open/close 43184 (time 1789132616.41 total 1429.21 last 67.19) - open/close 43792 (time 1789132626.41 total 1439.21 last 60.77) - open/close 44351 (time 1789132636.42 total 1449.22 last 55.83) - open/close 44981 (time 1789132646.44 total 1459.24 last 62.91) - open/close 45600 (time 1789132656.44 total 1469.24 last 61.88) - open/close 46249 (time 1789132666.44 total 1479.24 last 64.87) - open/close 46890 (time 1789132676.45 total 1489.25 last 64.05) - open/close 47540 (time 1789132686.46 total 1499.26 last 64.92) - open/close 48149 (time 1789132696.47 total 1509.27 last 60.85) - open/close 48786 (time 1789132706.49 total 1519.29 last 63.58) - open/close 49437 (time 1789132716.50 total 1529.30 last 65.01) total: 50000 open/close in 1537.68 seconds: 32.52 ops/second - unlinked 0 (time 1789132728 ; total 0 ; last 0) - unlinked 10000 (time 1789132892 ; total 164 ; last 164) - unlinked 20000 (time 1789133047 ; total 319 ; last 155) - unlinked 30000 (time 1789133226 ; total 498 ; last 179) - unlinked 40000 (time 1789133396 ; total 668 ; last 170) total: 50000 unlinks in 839 seconds: 59.594757 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2