== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:12:12 (1788603132) start mds service on oleg232-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg232-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg232-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1788603245.47 total 10.93 last 11.89) - open/close 311 (time 1788603255.49 total 20.95 last 18.07) - open/close 544 (time 1788603268.37 total 33.83 last 18.09) - open/close 786 (time 1788603278.38 total 43.84 last 24.17) - open/close 1030 (time 1788603288.39 total 53.86 last 24.36) - open/close 1148 (time 1788603299.30 total 64.76 last 10.82) - open/close 1227 (time 1788603309.32 total 74.78 last 7.88) - open/close 1461 (time 1788603322.02 total 87.48 last 18.43) - open/close 1679 (time 1788603332.04 total 97.50 last 21.76) - open/close 1814 (time 1788603342.04 total 107.51 last 13.49) - open/close 2028 (time 1788603352.53 total 117.99 last 20.41) - open/close 2259 (time 1788603363.04 total 128.50 last 21.99) - open/close 2486 (time 1788603373.13 total 138.60 last 22.48) - open/close 2677 (time 1788603384.16 total 149.62 last 17.32) - open/close 2856 (time 1788603398.73 total 164.19 last 12.29) - open/close 3044 (time 1788603409.84 total 175.30 last 16.92) - open/close 3272 (time 1788603420.86 total 186.32 last 20.68) - open/close 3515 (time 1788603431.70 total 197.16 last 22.42) - open/close 3746 (time 1788603441.73 total 207.19 last 23.03) - open/close 3958 (time 1788603452.26 total 217.72 last 20.14) - open/close 4232 (time 1788603462.80 total 228.27 last 25.98) - open/close 4495 (time 1788603473.24 total 238.70 last 25.20) - open/close 4722 (time 1788603483.25 total 248.72 last 22.67) - open/close 4957 (time 1788603494.04 total 259.51 last 21.78) - open/close 5186 (time 1788603504.37 total 269.83 last 22.18) - open/close 5415 (time 1788603514.97 total 280.43 last 21.60) - open/close 5578 (time 1788603529.36 total 294.82 last 11.33) - open/close 5672 (time 1788603539.37 total 304.83 last 9.39) - open/close 5871 (time 1788603549.38 total 314.85 last 19.87) - open/close 6173 (time 1788603560.22 total 325.68 last 27.88) - open/close 6441 (time 1788603570.22 total 335.69 last 26.78) - open/close 6545 (time 1788603581.08 total 346.54 last 9.58) - open/close 6709 (time 1788603591.10 total 356.56 last 16.37) - open/close 6876 (time 1788603601.10 total 366.57 last 16.69) - open/close 7065 (time 1788603611.64 total 377.10 last 17.94) - open/close 7273 (time 1788603621.65 total 387.12 last 20.76) - open/close 7348 (time 1788603631.67 total 397.13 last 7.49) - open/close 7504 (time 1788603644.69 total 410.15 last 11.98) - open/close 7653 (time 1788603655.65 total 421.11 last 13.60) - open/close 7876 (time 1788603666.27 total 431.73 last 21.00) - open/close 8087 (time 1788603677.21 total 442.67 last 19.28) - open/close 8357 (time 1788603687.22 total 452.68 last 26.98) - open/close 8635 (time 1788603698.24 total 463.71 last 25.21) - open/close 8833 (time 1788603708.39 total 473.85 last 19.52) - open/close 8998 (time 1788603718.43 total 483.89 last 16.44) - open/close 9261 (time 1788603728.47 total 493.93 last 26.19) - open/close 9455 (time 1788603738.53 total 503.99 last 19.28) - open/close 9711 (time 1788603748.53 total 513.99 last 25.59) - open/close 9899 (time 1788603759.32 total 524.78 last 17.42) - open/close 10055 (time 1788603769.56 total 535.02 last 15.24) - open/close 10118 (time 1788603782.63 total 548.09 last 4.82) - open/close 10252 (time 1788603792.79 total 558.25 last 13.19) - open/close 10290 (time 1788603804.19 total 569.65 last 3.33) - open/close 10453 (time 1788603814.63 total 580.10 last 15.61) - open/close 10602 (time 1788603824.65 total 590.11 last 14.87) - open/close 10806 (time 1788603834.79 total 600.25 last 20.13) - open/close 11033 (time 1788603846.49 total 611.96 last 19.39) - open/close 11161 (time 1788603857.12 total 622.58 last 12.05) - open/close 11366 (time 1788603867.82 total 633.28 last 19.16) - open/close 11555 (time 1788603878.47 total 643.93 last 17.75) - open/close 11824 (time 1788603888.48 total 653.94 last 26.86) - open/close 11993 (time 1788603899.06 total 664.53 last 15.97) - open/close 12198 (time 1788603909.42 total 674.88 last 19.80) - open/close 12467 (time 1788603919.43 total 684.90 last 26.87) - open/close 12726 (time 1788603929.76 total 695.22 last 25.09) - open/close 12963 (time 1788603939.76 total 705.23 last 23.68) - open/close 13232 (time 1788603949.77 total 715.23 last 26.89) - open/close 13427 (time 1788603959.78 total 725.24 last 19.48) - open/close 13624 (time 1788603969.80 total 735.27 last 19.66) - open/close 13906 (time 1788603980.30 total 745.76 last 26.87) - open/close 14170 (time 1788603990.31 total 755.77 last 26.38) - open/close 14353 (time 1788604000.86 total 766.33 last 17.34) - open/close 14430 (time 1788604011.49 total 776.96 last 7.24) - open/close 14531 (time 1788604021.66 total 787.12 last 9.94) - open/close 14652 (time 1788604031.91 total 797.38 last 11.79) - open/close 14732 (time 1788604043.04 total 808.50 last 7.19) - open/close 14957 (time 1788604053.91 total 819.38 last 20.69) - open/close 15038 (time 1788604064.93 total 830.40 last 7.35) - open/close 15277 (time 1788604077.80 total 843.27 last 18.57) - open/close 15441 (time 1788604088.63 total 854.10 last 15.14) - open/close 15661 (time 1788604099.38 total 864.85 last 20.46) - open/close 15802 (time 1788604109.42 total 874.88 last 14.05) - open/close 15958 (time 1788604120.16 total 885.63 last 14.51) - open/close 16177 (time 1788604130.21 total 895.68 last 21.79) - open/close 16389 (time 1788604142.63 total 908.09 last 17.08) - open/close 16594 (time 1788604155.04 total 920.51 last 16.51) - open/close 16770 (time 1788604165.84 total 931.31 last 16.29) - open/close 17032 (time 1788604175.85 total 941.32 last 26.18) - open/close 17281 (time 1788604185.87 total 951.34 last 24.85) - open/close 17470 (time 1788604197.73 total 963.19 last 15.94) - open/close 17704 (time 1788604207.83 total 973.29 last 23.18) - open/close 17960 (time 1788604218.47 total 983.93 last 24.05) - open/close 18187 (time 1788604228.49 total 993.95 last 22.66) - open/close 18438 (time 1788604238.50 total 1003.96 last 25.07) - open/close 18718 (time 1788604249.35 total 1014.81 last 25.81) - open/close 19382 (time 1788604259.36 total 1024.82 last 66.33) - open/close 20000 (time 1788604267.22 total 1032.69 last 78.58) - open/close 20733 (time 1788604277.24 total 1042.70 last 73.17) - open/close 21465 (time 1788604287.24 total 1052.71 last 73.17) - open/close 22167 (time 1788604297.25 total 1062.71 last 70.19) - open/close 22894 (time 1788604307.26 total 1072.72 last 72.61) - open/close 23727 (time 1788604317.26 total 1082.72 last 83.30) - open/close 24618 (time 1788604327.26 total 1092.72 last 89.09) - open/close 25376 (time 1788604337.27 total 1102.73 last 75.75) - open/close 26052 (time 1788604347.27 total 1112.73 last 67.60) - open/close 26690 (time 1788604357.29 total 1122.75 last 63.65) - open/close 27218 (time 1788604367.30 total 1132.76 last 52.77) - open/close 27882 (time 1788604377.30 total 1142.76 last 66.38) - open/close 28616 (time 1788604387.31 total 1152.78 last 73.30) - open/close 29281 (time 1788604397.31 total 1162.78 last 66.49) - open/close 30000 (time 1788604406.78 total 1172.24 last 75.96) - open/close 30824 (time 1788604416.78 total 1182.25 last 82.38) - open/close 31611 (time 1788604426.78 total 1192.25 last 78.68) - open/close 32420 (time 1788604436.79 total 1202.25 last 80.87) - open/close 33161 (time 1788604446.80 total 1212.26 last 74.02) - open/close 34008 (time 1788604456.82 total 1222.29 last 84.50) - open/close 34859 (time 1788604466.83 total 1232.29 last 85.03) - open/close 35597 (time 1788604476.86 total 1242.32 last 73.61) - open/close 36406 (time 1788604486.86 total 1252.32 last 80.89) - open/close 37181 (time 1788604496.87 total 1262.33 last 77.42) - open/close 38013 (time 1788604506.88 total 1272.34 last 83.09) - open/close 38965 (time 1788604516.88 total 1282.35 last 95.19) - open/close 39805 (time 1788604526.89 total 1292.35 last 83.94) - open/close 40000 (time 1788604529.61 total 1295.07 last 71.77) - open/close 40639 (time 1788604539.61 total 1305.08 last 63.86) - open/close 41269 (time 1788604549.61 total 1315.08 last 62.98) - open/close 41895 (time 1788604559.63 total 1325.09 last 62.50) - open/close 42586 (time 1788604569.63 total 1335.10 last 69.07) - open/close 43234 (time 1788604579.64 total 1345.10 last 64.79) - open/close 43897 (time 1788604589.66 total 1355.12 last 66.17) - open/close 44597 (time 1788604599.67 total 1365.13 last 69.94) - open/close 45257 (time 1788604609.67 total 1375.14 last 65.95) - open/close 46104 (time 1788604619.68 total 1385.14 last 84.64) - open/close 46947 (time 1788604629.69 total 1395.15 last 84.21) - open/close 47798 (time 1788604639.70 total 1405.16 last 85.06) - open/close 48617 (time 1788604649.70 total 1415.16 last 81.90) - open/close 49257 (time 1788604659.70 total 1425.17 last 63.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1430.88 seconds: 34.69 ops/second - unlinked 0 (time 1788604670 ; total 0 ; last 0) - unlinked 10000 (time 1788604813 ; total 143 ; last 143) - unlinked 20000 (time 1788604948 ; total 278 ; last 135) - unlinked 30000 (time 1788605089 ; total 419 ; last 141) - unlinked 40000 (time 1788605230 ; total 560 ; last 141) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 692 seconds: 71.722542 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local oleg232-server: modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2