== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:39:52 (1789133992) start mds service on oleg643-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing load_modules_local oleg643-server: Loading modules from /home/green/git/lustre-release/lustre oleg643-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg643-server: detected 4 online CPUs by sysfs oleg643-server: MODOPTS_LIBCFS= oleg643-server: Force libcfs to create 2 CPU partitions oleg643-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg643-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg643-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg643-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1789134086.15 total 10.00 last 13.10) - open/close 346 (time 1789134097.21 total 21.07 last 19.44) - open/close 557 (time 1789134107.21 total 31.07 last 21.09) - open/close 762 (time 1789134118.08 total 41.94 last 18.87) - open/close 1011 (time 1789134128.09 total 51.95 last 24.88) - open/close 1281 (time 1789134138.70 total 62.55 last 25.45) - open/close 1514 (time 1789134149.76 total 73.62 last 21.06) - open/close 1733 (time 1789134160.56 total 84.42 last 20.27) - open/close 1994 (time 1789134171.32 total 95.18 last 24.26) - open/close 2188 (time 1789134182.03 total 105.89 last 18.12) - open/close 2427 (time 1789134192.91 total 116.76 last 21.97) - open/close 2599 (time 1789134202.91 total 126.77 last 17.20) - open/close 2854 (time 1789134213.70 total 137.56 last 23.63) - open/close 3046 (time 1789134223.70 total 147.56 last 19.20) - open/close 3310 (time 1789134233.72 total 157.58 last 26.34) - open/close 3504 (time 1789134244.44 total 168.30 last 18.10) - open/close 3656 (time 1789134260.60 total 184.46 last 9.41) - open/close 3839 (time 1789134270.63 total 194.48 last 18.25) - open/close 4020 (time 1789134280.71 total 204.57 last 17.95) - open/close 4210 (time 1789134291.27 total 215.12 last 17.99) - open/close 4403 (time 1789134301.28 total 225.14 last 19.27) - open/close 4675 (time 1789134316.60 total 240.45 last 17.76) - open/close 4898 (time 1789134326.60 total 250.46 last 22.29) - open/close 5055 (time 1789134336.90 total 260.75 last 15.25) - open/close 5228 (time 1789134347.51 total 271.37 last 16.30) - open/close 5508 (time 1789134358.52 total 282.38 last 25.43) - open/close 5779 (time 1789134368.90 total 292.76 last 26.11) - open/close 6068 (time 1789134378.91 total 302.77 last 28.86) - open/close 6354 (time 1789134389.76 total 313.62 last 26.36) - open/close 6532 (time 1789134400.00 total 323.85 last 17.39) - open/close 6684 (time 1789134410.00 total 333.86 last 15.20) - open/close 6932 (time 1789134420.15 total 344.01 last 24.42) - open/close 7175 (time 1789134432.52 total 356.38 last 19.65) - open/close 7450 (time 1789134442.52 total 366.38 last 27.50) - open/close 7631 (time 1789134452.87 total 376.73 last 17.49) - open/close 7896 (time 1789134463.49 total 387.34 last 24.95) - open/close 8195 (time 1789134473.51 total 397.37 last 29.83) - open/close 8451 (time 1789134483.65 total 407.51 last 25.24) - open/close 8648 (time 1789134493.66 total 417.52 last 19.69) - open/close 8865 (time 1789134504.45 total 428.30 last 20.12) - open/close 9130 (time 1789134515.43 total 439.29 last 24.13) - open/close 9261 (time 1789134526.03 total 449.88 last 12.36) - open/close 9457 (time 1789134536.05 total 459.90 last 19.56) - open/close 9715 (time 1789134546.20 total 470.06 last 25.41) - open/close 9909 (time 1789134556.22 total 480.08 last 19.36) - open/close 10000 (time 1789134562.75 total 486.61 last 13.93) - open/close 10166 (time 1789134572.76 total 496.62 last 16.59) - open/close 10311 (time 1789134582.76 total 506.62 last 14.50) - open/close 10428 (time 1789134593.01 total 516.87 last 11.42) - open/close 10666 (time 1789134603.42 total 527.28 last 22.87) - open/close 10934 (time 1789134613.44 total 537.30 last 26.74) - open/close 11130 (time 1789134624.26 total 548.12 last 18.11) - open/close 11419 (time 1789134634.27 total 558.13 last 28.88) - open/close 11657 (time 1789134644.86 total 568.72 last 22.46) - open/close 11876 (time 1789134654.92 total 578.77 last 21.78) - open/close 12139 (time 1789134664.92 total 588.78 last 26.30) - open/close 12381 (time 1789134675.52 total 599.38 last 22.82) - open/close 12630 (time 1789134685.75 total 609.61 last 24.34) - open/close 12790 (time 1789134698.73 total 622.59 last 12.33) - open/close 13075 (time 1789134711.16 total 635.02 last 22.92) - open/close 13324 (time 1789134721.93 total 645.79 last 23.13) - open/close 13449 (time 1789134731.94 total 655.79 last 12.49) - open/close 13618 (time 1789134742.92 total 666.77 last 15.39) - open/close 13792 (time 1789134753.52 total 677.38 last 16.40) - open/close 14053 (time 1789134764.48 total 688.34 last 23.82) - open/close 14211 (time 1789134775.10 total 698.96 last 14.88) - open/close 14457 (time 1789134785.12 total 708.97 last 24.56) - open/close 14599 (time 1789134795.13 total 718.99 last 14.17) - open/close 14668 (time 1789134805.17 total 729.02 last 6.88) - open/close 14811 (time 1789134815.18 total 739.04 last 14.28) - open/close 15046 (time 1789134826.01 total 749.86 last 21.71) - open/close 15294 (time 1789134836.18 total 760.04 last 24.38) - open/close 15493 (time 1789134846.18 total 770.04 last 19.89) - open/close 15744 (time 1789134856.74 total 780.59 last 23.79) - open/close 15956 (time 1789134867.59 total 791.45 last 19.53) - open/close 16108 (time 1789134877.75 total 801.61 last 14.96) - open/close 16191 (time 1789134887.75 total 811.61 last 8.30) - open/close 16346 (time 1789134897.91 total 821.77 last 15.26) - open/close 16626 (time 1789134908.41 total 832.27 last 26.66) - open/close 16829 (time 1789134918.49 total 842.35 last 20.14) - open/close 16975 (time 1789134929.08 total 852.94 last 13.78) - open/close 17091 (time 1789134939.65 total 863.51 last 10.98) - open/close 17335 (time 1789134949.77 total 873.63 last 24.12) - open/close 17470 (time 1789134960.72 total 884.58 last 12.33) - open/close 17674 (time 1789134970.75 total 894.60 last 20.35) - open/close 17839 (time 1789134981.45 total 905.30 last 15.42) - open/close 17945 (time 1789134993.08 total 916.94 last 9.11) - open/close 18087 (time 1789135003.13 total 926.99 last 14.13) - open/close 18308 (time 1789135013.39 total 937.25 last 21.54) - open/close 18573 (time 1789135024.32 total 948.18 last 24.24) - open/close 18843 (time 1789135034.97 total 958.83 last 25.35) - open/close 19131 (time 1789135045.91 total 969.77 last 26.33) - open/close 19499 (time 1789135055.94 total 979.79 last 36.71) - open/close 20000 (time 1789135061.99 total 985.85 last 82.77) - open/close 20706 (time 1789135071.99 total 995.85 last 70.60) - open/close 21452 (time 1789135081.99 total 1005.85 last 74.58) - open/close 22115 (time 1789135092.01 total 1015.87 last 66.18) - open/close 22926 (time 1789135102.02 total 1025.87 last 81.04) - open/close 23712 (time 1789135112.02 total 1035.88 last 78.57) - open/close 24564 (time 1789135122.03 total 1045.88 last 85.15) - open/close 25379 (time 1789135132.03 total 1055.89 last 81.48) - open/close 26171 (time 1789135142.04 total 1065.89 last 79.15) - open/close 26990 (time 1789135152.06 total 1075.92 last 81.71) - open/close 27743 (time 1789135162.07 total 1085.93 last 75.21) - open/close 28443 (time 1789135172.08 total 1095.94 last 69.91) - open/close 29008 (time 1789135182.10 total 1105.96 last 56.41) - open/close 29539 (time 1789135192.11 total 1115.97 last 53.02) - open/close 30000 (time 1789135199.46 total 1123.32 last 62.73) - open/close 30802 (time 1789135209.47 total 1133.32 last 80.16) - open/close 31566 (time 1789135219.47 total 1143.33 last 76.38) - open/close 32095 (time 1789135229.48 total 1153.34 last 52.85) - open/close 32616 (time 1789135239.49 total 1163.35 last 52.03) - open/close 33071 (time 1789135249.49 total 1173.35 last 45.50) - open/close 33716 (time 1789135259.49 total 1183.35 last 64.49) - open/close 34495 (time 1789135269.50 total 1193.36 last 77.84) - open/close 35265 (time 1789135279.50 total 1203.36 last 76.97) - open/close 36035 (time 1789135289.50 total 1213.36 last 77.00) - open/close 36687 (time 1789135299.51 total 1223.37 last 65.16) - open/close 37406 (time 1789135309.52 total 1233.38 last 71.84) - open/close 38221 (time 1789135319.55 total 1243.41 last 81.21) - open/close 38832 (time 1789135329.56 total 1253.42 last 61.05) - open/close 39354 (time 1789135339.58 total 1263.43 last 52.13) - open/close 39879 (time 1789135349.58 total 1273.44 last 52.48) - open/close 40000 (time 1789135351.62 total 1275.48 last 59.28) - open/close 40520 (time 1789135361.63 total 1285.48 last 51.97) - open/close 41073 (time 1789135371.64 total 1295.50 last 55.21) - open/close 41648 (time 1789135381.65 total 1305.51 last 57.44) - open/close 42384 (time 1789135391.66 total 1315.52 last 73.53) - open/close 43096 (time 1789135401.68 total 1325.54 last 71.10) - open/close 43723 (time 1789135411.70 total 1335.56 last 62.53) - open/close 44378 (time 1789135421.73 total 1345.58 last 65.36) - open/close 45034 (time 1789135431.74 total 1355.60 last 65.53) - open/close 45652 (time 1789135441.75 total 1365.60 last 61.75) - open/close 46352 (time 1789135451.75 total 1375.61 last 69.98) - open/close 46924 (time 1789135461.76 total 1385.61 last 57.15) - open/close 47429 (time 1789135471.79 total 1395.65 last 50.34) - open/close 47870 (time 1789135481.80 total 1405.66 last 44.03) - open/close 48351 (time 1789135491.82 total 1415.67 last 48.04) - open/close 48900 (time 1789135501.82 total 1425.68 last 54.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1435.51 seconds: 34.57 ops/second - unlinked 0 (time 1789135516 ; total 0 ; last 0) - unlinked 10000 (time 1789135674 ; total 158 ; last 158) - unlinked 20000 (time 1789135829 ; total 313 ; last 155) - unlinked 30000 (time 1789135962 ; total 446 ; last 133) - unlinked 40000 (time 1789136101 ; total 585 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 709 seconds: 70.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg643-server unloading modules via unload_modules_local on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing unload_modules_local oleg643-server: modules unloaded. pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2