== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:37:18 (1781001438) start mds service on oleg105-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg105-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg105-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 288 (time 1781001542.79 total 10.01 last 28.76) - open/close 558 (time 1781001553.18 total 20.40 last 26.00) - open/close 794 (time 1781001563.18 total 30.40 last 23.59) - open/close 1002 (time 1781001574.10 total 41.33 last 19.04) - open/close 1107 (time 1781001586.52 total 53.75 last 8.46) - open/close 1321 (time 1781001597.78 total 65.01 last 19.00) - open/close 1431 (time 1781001608.59 total 75.81 last 10.18) - open/close 1635 (time 1781001618.96 total 86.19 last 19.67) - open/close 1745 (time 1781001629.45 total 96.68 last 10.48) - open/close 1997 (time 1781001639.46 total 106.68 last 25.19) - open/close 2219 (time 1781001649.46 total 116.69 last 22.19) - open/close 2437 (time 1781001660.50 total 127.73 last 19.75) - open/close 2661 (time 1781001670.51 total 137.73 last 22.38) - open/close 2875 (time 1781001680.52 total 147.74 last 21.38) - open/close 3003 (time 1781001690.53 total 157.76 last 12.78) - open/close 3140 (time 1781001701.39 total 168.62 last 12.61) - open/close 3299 (time 1781001714.21 total 181.43 last 12.41) - open/close 3502 (time 1781001725.14 total 192.36 last 18.57) - open/close 3688 (time 1781001735.17 total 202.40 last 18.54) - open/close 3855 (time 1781001745.42 total 212.64 last 16.30) - open/close 3941 (time 1781001756.24 total 223.47 last 7.95) - open/close 4164 (time 1781001766.82 total 234.05 last 21.08) - open/close 4368 (time 1781001776.85 total 244.07 last 20.34) - open/close 4554 (time 1781001786.86 total 254.08 last 18.58) - open/close 4670 (time 1781001797.27 total 264.49 last 11.15) - open/close 4882 (time 1781001807.27 total 274.49 last 21.20) - open/close 5124 (time 1781001817.93 total 285.16 last 22.70) - open/close 5248 (time 1781001828.38 total 295.61 last 11.86) - open/close 5444 (time 1781001838.43 total 305.65 last 19.52) - open/close 5612 (time 1781001848.43 total 315.65 last 16.80) - open/close 5722 (time 1781001858.46 total 325.69 last 10.96) - open/close 5862 (time 1781001869.19 total 336.42 last 13.04) - open/close 6001 (time 1781001879.65 total 346.87 last 13.30) - open/close 6208 (time 1781001889.69 total 356.91 last 20.61) - open/close 6428 (time 1781001899.70 total 366.92 last 21.99) - open/close 6568 (time 1781001909.86 total 377.08 last 13.78) - open/close 6697 (time 1781001920.23 total 387.45 last 12.44) - open/close 6836 (time 1781001933.07 total 400.30 last 10.82) - open/close 6977 (time 1781001943.09 total 410.32 last 14.07) - open/close 7125 (time 1781001955.37 total 422.59 last 12.06) - open/close 7359 (time 1781001965.90 total 433.12 last 22.22) - open/close 7651 (time 1781001975.90 total 443.13 last 29.19) - open/close 7742 (time 1781001985.91 total 453.14 last 9.09) - open/close 7852 (time 1781001996.68 total 463.90 last 10.22) - open/close 8015 (time 1781002006.68 total 473.91 last 16.29) - open/close 8131 (time 1781002018.77 total 485.99 last 9.60) - open/close 8251 (time 1781002029.84 total 497.07 last 10.84) - open/close 8493 (time 1781002042.07 total 509.30 last 19.78) - open/close 8657 (time 1781002052.69 total 519.91 last 15.45) - open/close 8796 (time 1781002063.06 total 530.28 last 13.40) - open/close 9113 (time 1781002079.25 total 546.48 last 19.57) - open/close 9340 (time 1781002089.26 total 556.48 last 22.69) - open/close 9515 (time 1781002099.68 total 566.91 last 16.78) - open/close 9742 (time 1781002110.24 total 577.46 last 21.51) - open/close 9934 (time 1781002121.05 total 588.27 last 17.76) - open/close 10000 (time 1781002127.96 total 595.18 last 9.56) - open/close 10230 (time 1781002137.96 total 605.18 last 22.99) - open/close 10414 (time 1781002150.96 total 618.19 last 14.15) - open/close 10554 (time 1781002161.36 total 628.58 last 13.47) - open/close 10767 (time 1781002171.37 total 638.59 last 21.29) - open/close 10943 (time 1781002181.39 total 648.61 last 17.56) - open/close 11131 (time 1781002191.57 total 658.79 last 18.46) - open/close 11215 (time 1781002201.58 total 668.81 last 8.39) - open/close 11407 (time 1781002212.50 total 679.72 last 17.59) - open/close 11628 (time 1781002222.56 total 689.78 last 21.97) - open/close 11878 (time 1781002233.45 total 700.68 last 22.95) - open/close 12006 (time 1781002243.57 total 710.79 last 12.65) - open/close 12091 (time 1781002254.19 total 721.41 last 8.01) - open/close 12246 (time 1781002264.53 total 731.75 last 14.99) - open/close 12434 (time 1781002274.54 total 741.76 last 18.79) - open/close 12633 (time 1781002284.55 total 751.78 last 19.86) - open/close 12740 (time 1781002295.12 total 762.34 last 10.13) - open/close 12869 (time 1781002305.38 total 772.60 last 12.57) - open/close 13089 (time 1781002316.43 total 783.66 last 19.90) - open/close 13177 (time 1781002326.62 total 793.85 last 8.64) - open/close 13466 (time 1781002336.63 total 803.85 last 28.88) - open/close 13624 (time 1781002347.23 total 814.46 last 14.90) - open/close 13812 (time 1781002358.08 total 825.31 last 17.32) - open/close 13934 (time 1781002368.72 total 835.95 last 11.47) - open/close 14120 (time 1781002378.73 total 845.96 last 18.58) - open/close 14233 (time 1781002390.17 total 857.39 last 9.88) - open/close 14548 (time 1781002402.92 total 870.14 last 24.71) - open/close 14740 (time 1781002412.93 total 880.15 last 19.19) - open/close 15038 (time 1781002423.91 total 891.14 last 27.12) - open/close 15296 (time 1781002433.93 total 901.15 last 25.77) - open/close 15577 (time 1781002443.93 total 911.16 last 28.08) - open/close 15835 (time 1781002453.95 total 921.17 last 25.77) - open/close 16164 (time 1781002464.74 total 931.96 last 30.49) - open/close 16305 (time 1781002475.34 total 942.56 last 13.30) - open/close 16528 (time 1781002485.93 total 953.15 last 21.05) - open/close 16728 (time 1781002496.47 total 963.69 last 18.98) - open/close 16858 (time 1781002507.11 total 974.33 last 12.21) - open/close 16981 (time 1781002517.78 total 985.00 last 11.53) - open/close 17209 (time 1781002527.82 total 995.05 last 22.70) - open/close 17385 (time 1781002538.27 total 1005.49 last 16.85) - open/close 17606 (time 1781002548.76 total 1015.98 last 21.07) - open/close 17695 (time 1781002560.77 total 1028.00 last 7.41) - open/close 17782 (time 1781002570.90 total 1038.13 last 8.59) - open/close 18015 (time 1781002580.91 total 1048.14 last 23.28) - open/close 18566 (time 1781002590.91 total 1058.14 last 55.09) - open/close 19365 (time 1781002600.92 total 1068.14 last 79.88) - open/close 20000 (time 1781002607.56 total 1074.79 last 95.51) - open/close 20892 (time 1781002617.58 total 1084.80 last 89.11) - open/close 21731 (time 1781002627.58 total 1094.80 last 83.86) - open/close 22632 (time 1781002637.59 total 1104.81 last 90.00) - open/close 23400 (time 1781002647.61 total 1114.84 last 76.64) - open/close 23990 (time 1781002657.62 total 1124.84 last 58.95) - open/close 24806 (time 1781002667.63 total 1134.85 last 81.51) - open/close 25632 (time 1781002677.68 total 1144.90 last 82.19) - open/close 26612 (time 1781002687.69 total 1154.92 last 97.86) - open/close 27470 (time 1781002697.71 total 1164.93 last 85.67) - open/close 28243 (time 1781002707.73 total 1174.95 last 77.17) - open/close 29058 (time 1781002717.73 total 1184.96 last 81.43) - open/close 29707 (time 1781002727.74 total 1194.97 last 64.84) - open/close 30000 (time 1781002731.19 total 1198.41 last 85.04) - open/close 30849 (time 1781002741.19 total 1208.41 last 84.87) - open/close 31805 (time 1781002751.19 total 1218.42 last 95.59) - open/close 32689 (time 1781002761.21 total 1228.43 last 88.28) - open/close 33547 (time 1781002771.23 total 1238.45 last 85.63) - open/close 34252 (time 1781002781.24 total 1248.46 last 70.40) - open/close 34971 (time 1781002791.25 total 1258.48 last 71.81) - open/close 35757 (time 1781002801.27 total 1268.50 last 78.45) - open/close 36598 (time 1781002811.29 total 1278.51 last 83.97) - open/close 37431 (time 1781002821.30 total 1288.52 last 83.23) - open/close 38235 (time 1781002831.32 total 1298.55 last 80.20) - open/close 39170 (time 1781002841.34 total 1308.56 last 93.35) - open/close 39864 (time 1781002851.34 total 1318.56 last 69.39) - open/close 40517 (time 1781002861.35 total 1328.57 last 65.23) - open/close 41314 (time 1781002871.37 total 1338.59 last 79.53) - open/close 41969 (time 1781002881.39 total 1348.61 last 65.37) - open/close 42625 (time 1781002891.39 total 1358.62 last 65.56) - open/close 43314 (time 1781002901.40 total 1368.62 last 68.86) - open/close 43790 (time 1781002911.43 total 1378.66 last 47.45) - open/close 44451 (time 1781002921.45 total 1388.67 last 66.01) - open/close 45062 (time 1781002931.46 total 1398.69 last 60.99) - open/close 45692 (time 1781002941.47 total 1408.69 last 62.97) - open/close 46604 (time 1781002951.48 total 1418.70 last 91.11) - open/close 47380 (time 1781002961.49 total 1428.71 last 77.55) - open/close 48223 (time 1781002971.49 total 1438.71 last 84.27) - open/close 49035 (time 1781002981.49 total 1448.71 last 81.20) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1455.51 seconds: 34.10 ops/second - unlinked 0 (time 1781002992 ; total 0 ; last 0) - unlinked 10000 (time 1781003111 ; total 119 ; last 119) - unlinked 20000 (time 1781003234 ; total 242 ; last 123) - unlinked 30000 (time 1781003366 ; total 374 ; last 132) - unlinked 40000 (time 1781003494 ; total 502 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 632 seconds: 78.530060 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: modules unloaded. pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2