== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:43:35 (1781239415) start mds service on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: MODOPTS_LIBCFS= oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg308-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1781239524.52 total 10.52 last 15.78) - open/close 384 (time 1781239534.57 total 20.57 last 21.69) - open/close 606 (time 1781239544.59 total 30.58 last 22.17) - open/close 874 (time 1781239555.53 total 41.53 last 24.48) - open/close 1131 (time 1781239566.16 total 52.16 last 24.19) - open/close 1395 (time 1781239576.85 total 62.85 last 24.69) - open/close 1699 (time 1781239586.87 total 72.87 last 30.34) - open/close 1921 (time 1781239596.88 total 82.87 last 22.19) - open/close 2208 (time 1781239606.88 total 92.88 last 28.69) - open/close 2447 (time 1781239617.29 total 103.29 last 22.96) - open/close 2533 (time 1781239627.36 total 113.36 last 8.54) - open/close 2660 (time 1781239639.35 total 125.34 last 10.59) - open/close 2778 (time 1781239649.52 total 135.51 last 11.60) - open/close 2986 (time 1781239659.91 total 145.91 last 20.01) - open/close 3131 (time 1781239671.97 total 157.96 last 12.03) - open/close 3380 (time 1781239682.14 total 168.13 last 24.48) - open/close 3508 (time 1781239692.82 total 178.81 last 11.98) - open/close 3672 (time 1781239702.83 total 188.82 last 16.39) - open/close 3809 (time 1781239713.95 total 199.94 last 12.32) - open/close 4060 (time 1781239729.89 total 215.89 last 15.75) - open/close 4307 (time 1781239740.34 total 226.34 last 23.63) - open/close 4520 (time 1781239752.66 total 238.66 last 17.29) - open/close 4714 (time 1781239762.68 total 248.67 last 19.37) - open/close 4786 (time 1781239772.69 total 258.69 last 7.19) - open/close 5038 (time 1781239783.59 total 269.58 last 23.13) - open/close 5213 (time 1781239794.46 total 280.45 last 16.10) - open/close 5327 (time 1781239805.27 total 291.26 last 10.55) - open/close 5604 (time 1781239815.27 total 301.27 last 27.68) - open/close 5879 (time 1781239827.62 total 313.61 last 22.28) - open/close 6011 (time 1781239840.22 total 326.21 last 10.47) - open/close 6194 (time 1781239850.25 total 336.24 last 18.24) - open/close 6341 (time 1781239863.52 total 349.51 last 11.08) - open/close 6458 (time 1781239874.07 total 360.07 last 11.09) - open/close 6569 (time 1781239884.52 total 370.52 last 10.62) - open/close 6690 (time 1781239894.55 total 380.54 last 12.07) - open/close 6846 (time 1781239905.24 total 391.23 last 14.59) - open/close 6945 (time 1781239917.33 total 403.33 last 8.19) - open/close 7129 (time 1781239929.14 total 415.13 last 15.59) - open/close 7387 (time 1781239940.38 total 426.37 last 22.95) - open/close 7451 (time 1781239954.02 total 440.02 last 4.69) - open/close 7689 (time 1781239964.75 total 450.75 last 22.19) - open/close 7865 (time 1781239975.64 total 461.64 last 16.16) - open/close 8092 (time 1781239985.96 total 471.95 last 22.01) - open/close 8294 (time 1781239995.96 total 481.96 last 20.18) - open/close 8569 (time 1781240006.62 total 492.61 last 25.82) - open/close 8823 (time 1781240016.63 total 502.62 last 25.37) - open/close 9048 (time 1781240026.63 total 512.63 last 22.49) - open/close 9252 (time 1781240036.63 total 522.63 last 20.39) - open/close 9483 (time 1781240047.29 total 533.28 last 21.68) - open/close 9758 (time 1781240057.64 total 543.63 last 26.57) - open/close 9871 (time 1781240067.94 total 553.93 last 10.97) - open/close 10000 (time 1781240077.56 total 563.56 last 13.40) - open/close 10200 (time 1781240087.57 total 573.57 last 19.98) - open/close 10449 (time 1781240098.21 total 584.21 last 23.41) - open/close 10696 (time 1781240108.63 total 594.62 last 23.72) - open/close 10931 (time 1781240118.65 total 604.65 last 23.43) - open/close 11078 (time 1781240129.30 total 615.29 last 13.81) - open/close 11347 (time 1781240139.62 total 625.61 last 26.06) - open/close 11631 (time 1781240149.98 total 635.98 last 27.40) - open/close 11813 (time 1781240160.01 total 646.00 last 18.15) - open/close 11999 (time 1781240170.58 total 656.57 last 17.60) - open/close 12264 (time 1781240180.58 total 666.57 last 26.50) - open/close 12422 (time 1781240191.83 total 677.83 last 14.04) - open/close 12608 (time 1781240202.83 total 688.83 last 16.91) - open/close 12727 (time 1781240213.60 total 699.60 last 11.05) - open/close 12983 (time 1781240223.61 total 709.60 last 25.59) - open/close 13140 (time 1781240233.90 total 719.89 last 15.26) - open/close 13282 (time 1781240244.89 total 730.89 last 12.91) - open/close 13451 (time 1781240255.72 total 741.72 last 15.60) - open/close 13718 (time 1781240266.78 total 752.78 last 24.14) - open/close 13920 (time 1781240276.95 total 762.94 last 19.88) - open/close 14109 (time 1781240286.95 total 772.95 last 18.89) - open/close 14285 (time 1781240297.63 total 783.63 last 16.48) - open/close 14515 (time 1781240308.38 total 794.38 last 21.40) - open/close 14714 (time 1781240319.70 total 805.70 last 17.58) - open/close 14848 (time 1781240329.71 total 815.71 last 13.39) - open/close 15063 (time 1781240340.20 total 826.19 last 20.51) - open/close 15322 (time 1781240350.20 total 836.20 last 25.88) - open/close 15543 (time 1781240360.61 total 846.61 last 21.23) - open/close 15804 (time 1781240370.62 total 856.62 last 26.07) - open/close 16033 (time 1781240383.60 total 869.59 last 17.65) - open/close 16211 (time 1781240394.17 total 880.16 last 16.84) - open/close 16418 (time 1781240404.17 total 890.17 last 20.69) - open/close 16614 (time 1781240415.20 total 901.20 last 17.77) - open/close 16876 (time 1781240425.22 total 911.21 last 26.16) - open/close 17051 (time 1781240440.54 total 926.53 last 11.43) - open/close 17301 (time 1781240451.30 total 937.29 last 23.23) - open/close 17541 (time 1781240462.12 total 948.11 last 22.18) - open/close 17737 (time 1781240473.14 total 959.13 last 17.78) - open/close 17929 (time 1781240484.09 total 970.09 last 17.53) - open/close 18214 (time 1781240494.48 total 980.48 last 27.43) - open/close 18479 (time 1781240504.48 total 990.48 last 26.49) - open/close 19088 (time 1781240514.52 total 1000.51 last 60.71) - open/close 19727 (time 1781240524.52 total 1010.52 last 63.86) - open/close 20000 (time 1781240528.36 total 1014.36 last 71.13) - open/close 20694 (time 1781240538.36 total 1024.36 last 69.39) - open/close 21352 (time 1781240548.38 total 1034.38 last 65.65) - open/close 22053 (time 1781240558.39 total 1044.38 last 70.10) - open/close 22727 (time 1781240568.39 total 1054.39 last 67.36) - open/close 23570 (time 1781240578.40 total 1064.40 last 84.19) - open/close 24325 (time 1781240588.40 total 1074.40 last 75.50) - open/close 25011 (time 1781240598.41 total 1084.41 last 68.53) - open/close 25682 (time 1781240608.42 total 1094.41 last 67.07) - open/close 26327 (time 1781240618.42 total 1104.42 last 64.48) - open/close 26994 (time 1781240628.44 total 1114.44 last 66.58) - open/close 27737 (time 1781240638.44 total 1124.44 last 74.29) - open/close 28355 (time 1781240648.45 total 1134.44 last 61.76) - open/close 29024 (time 1781240658.47 total 1144.47 last 66.75) - open/close 29583 (time 1781240668.47 total 1154.47 last 55.90) - open/close 30000 (time 1781240675.35 total 1161.34 last 60.66) - open/close 30749 (time 1781240685.37 total 1171.37 last 74.72) - open/close 31613 (time 1781240695.37 total 1181.37 last 86.35) - open/close 32397 (time 1781240705.39 total 1191.39 last 78.28) - open/close 33140 (time 1781240715.39 total 1201.39 last 74.28) - open/close 33865 (time 1781240725.40 total 1211.40 last 72.44) - open/close 34543 (time 1781240735.42 total 1221.42 last 67.67) - open/close 35205 (time 1781240745.42 total 1231.42 last 66.17) - open/close 35878 (time 1781240755.44 total 1241.44 last 67.20) - open/close 36538 (time 1781240765.44 total 1251.44 last 66.00) - open/close 37221 (time 1781240775.44 total 1261.44 last 68.28) - open/close 37909 (time 1781240785.44 total 1271.44 last 68.80) - open/close 38610 (time 1781240795.45 total 1281.45 last 70.05) - open/close 39283 (time 1781240805.46 total 1291.46 last 67.23) - open/close 39982 (time 1781240815.46 total 1301.46 last 69.88) - open/close 40638 (time 1781240825.47 total 1311.47 last 65.54) - open/close 41315 (time 1781240835.48 total 1321.48 last 67.65) - open/close 42021 (time 1781240845.49 total 1331.48 last 70.56) - open/close 42697 (time 1781240855.49 total 1341.48 last 67.60) - open/close 43364 (time 1781240865.49 total 1351.49 last 66.66) - open/close 44078 (time 1781240875.50 total 1361.49 last 71.39) - open/close 44687 (time 1781240885.50 total 1371.50 last 60.85) - open/close 45402 (time 1781240895.51 total 1381.50 last 71.46) - open/close 46047 (time 1781240905.52 total 1391.51 last 64.45) - open/close 46777 (time 1781240915.52 total 1401.52 last 72.96) - open/close 47404 (time 1781240925.52 total 1411.52 last 62.69) - open/close 48068 (time 1781240935.55 total 1421.54 last 66.25) - open/close 48758 (time 1781240945.56 total 1431.55 last 68.93) - open/close 49397 (time 1781240955.57 total 1441.57 last 63.82) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1444.72 seconds: 34.35 ops/second - unlinked 0 (time 1781240963 ; total 0 ; last 0) - unlinked 10000 (time 1781241119 ; total 156 ; last 156) - unlinked 20000 (time 1781241274 ; total 311 ; last 155) - unlinked 30000 (time 1781241428 ; total 465 ; last 154) - unlinked 40000 (time 1781241566 ; total 603 ; last 138) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 727 seconds: 68.268227 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server unloading modules via unload_modules_local on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: modules unloaded. pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2