== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:51:56 (1788447116) start mds service on oleg401-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg401-server: detected 4 online CPUs by sysfs oleg401-server: MODOPTS_LIBCFS= oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg401-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg401-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 173 (time 1788447225.65 total 10.84 last 15.97) - open/close 359 (time 1788447236.61 total 21.79 last 16.97) - open/close 603 (time 1788447247.15 total 32.33 last 23.15) - open/close 835 (time 1788447257.17 total 42.35 last 23.15) - open/close 1108 (time 1788447267.21 total 52.39 last 27.21) - open/close 1313 (time 1788447278.13 total 63.31 last 18.76) - open/close 1533 (time 1788447288.18 total 73.36 last 21.90) - open/close 1818 (time 1788447298.18 total 83.37 last 28.49) - open/close 2047 (time 1788447308.34 total 93.52 last 22.56) - open/close 2334 (time 1788447318.76 total 103.94 last 27.53) - open/close 2571 (time 1788447329.01 total 114.19 last 23.13) - open/close 2807 (time 1788447339.40 total 124.59 last 22.70) - open/close 3112 (time 1788447349.41 total 134.59 last 30.48) - open/close 3395 (time 1788447359.43 total 144.61 last 28.26) - open/close 3631 (time 1788447369.43 total 154.61 last 23.59) - open/close 3875 (time 1788447379.69 total 164.87 last 23.78) - open/close 4098 (time 1788447389.69 total 174.88 last 22.29) - open/close 4410 (time 1788447399.72 total 184.90 last 31.12) - open/close 4632 (time 1788447410.73 total 195.92 last 20.15) - open/close 4877 (time 1788447420.90 total 206.08 last 24.10) - open/close 5069 (time 1788447430.91 total 216.09 last 19.18) - open/close 5315 (time 1788447441.02 total 226.20 last 24.33) - open/close 5410 (time 1788447451.05 total 236.23 last 9.47) - open/close 5538 (time 1788447461.42 total 246.60 last 12.34) - open/close 5781 (time 1788447471.43 total 256.61 last 24.28) - open/close 5973 (time 1788447481.67 total 266.85 last 18.75) - open/close 6166 (time 1788447492.41 total 277.59 last 17.97) - open/close 6291 (time 1788447502.42 total 287.60 last 12.49) - open/close 6481 (time 1788447512.90 total 298.09 last 18.12) - open/close 6642 (time 1788447522.99 total 308.18 last 15.96) - open/close 6809 (time 1788447533.00 total 318.18 last 16.69) - open/close 6988 (time 1788447543.34 total 328.53 last 17.31) - open/close 7298 (time 1788447553.35 total 338.53 last 30.99) - open/close 7440 (time 1788447565.50 total 350.68 last 11.68) - open/close 7698 (time 1788447575.51 total 360.69 last 25.78) - open/close 7923 (time 1788447585.52 total 370.70 last 22.49) - open/close 8166 (time 1788447596.28 total 381.46 last 22.57) - open/close 8397 (time 1788447606.38 total 391.56 last 22.88) - open/close 8604 (time 1788447616.39 total 401.57 last 20.68) - open/close 8871 (time 1788447626.40 total 411.58 last 26.68) - open/close 9073 (time 1788447636.60 total 421.78 last 19.80) - open/close 9296 (time 1788447647.43 total 432.61 last 20.59) - open/close 9468 (time 1788447658.16 total 443.34 last 16.03) - open/close 9651 (time 1788447668.16 total 453.34 last 18.29) - open/close 9853 (time 1788447678.53 total 463.72 last 19.48) - open/close 9990 (time 1788447688.82 total 474.00 last 13.32) - open/close 10167 (time 1788447700.02 total 485.20 last 15.81) - open/close 10338 (time 1788447710.14 total 495.32 last 16.90) - open/close 10543 (time 1788447721.08 total 506.26 last 18.74) - open/close 10732 (time 1788447731.79 total 516.97 last 17.64) - open/close 10956 (time 1788447742.77 total 527.95 last 20.40) - open/close 11176 (time 1788447753.26 total 538.44 last 20.97) - open/close 11376 (time 1788447763.27 total 548.45 last 19.98) - open/close 11566 (time 1788447773.88 total 559.06 last 17.91) - open/close 11818 (time 1788447783.89 total 569.07 last 25.17) - open/close 12049 (time 1788447794.73 total 579.91 last 21.31) - open/close 12122 (time 1788447805.14 total 590.32 last 7.01) - open/close 12275 (time 1788447815.54 total 600.72 last 14.71) - open/close 12459 (time 1788447825.55 total 610.73 last 18.38) - open/close 12702 (time 1788447838.35 total 623.53 last 18.99) - open/close 12881 (time 1788447848.71 total 633.90 last 17.26) - open/close 13130 (time 1788447859.12 total 644.31 last 23.92) - open/close 13396 (time 1788447869.68 total 654.86 last 25.19) - open/close 13667 (time 1788447879.69 total 664.87 last 27.07) - open/close 13892 (time 1788447889.71 total 674.90 last 22.45) - open/close 14145 (time 1788447900.27 total 685.45 last 23.97) - open/close 14368 (time 1788447910.29 total 695.47 last 22.26) - open/close 14518 (time 1788447920.45 total 705.63 last 14.76) - open/close 14698 (time 1788447930.48 total 715.67 last 17.93) - open/close 14818 (time 1788447941.38 total 726.56 last 11.02) - open/close 15048 (time 1788447951.39 total 736.57 last 22.96) - open/close 15232 (time 1788447962.03 total 747.21 last 17.30) - open/close 15479 (time 1788447972.05 total 757.23 last 24.65) - open/close 15623 (time 1788447982.14 total 767.32 last 14.27) - open/close 15792 (time 1788447992.64 total 777.83 last 16.08) - open/close 15866 (time 1788448002.65 total 787.83 last 7.40) - open/close 16108 (time 1788448012.73 total 797.91 last 24.01) - open/close 16322 (time 1788448022.74 total 807.92 last 21.38) - open/close 16585 (time 1788448032.75 total 817.93 last 26.27) - open/close 16827 (time 1788448042.99 total 828.17 last 23.64) - open/close 17086 (time 1788448053.62 total 838.81 last 24.35) - open/close 17327 (time 1788448063.63 total 848.81 last 24.08) - open/close 17560 (time 1788448073.64 total 858.82 last 23.29) - open/close 17735 (time 1788448084.66 total 869.84 last 15.88) - open/close 17979 (time 1788448094.67 total 879.85 last 24.38) - open/close 18194 (time 1788448104.95 total 890.13 last 20.90) - open/close 18379 (time 1788448115.71 total 900.89 last 17.20) - open/close 18555 (time 1788448125.73 total 910.91 last 17.56) - open/close 18719 (time 1788448136.56 total 921.74 last 15.14) - open/close 18935 (time 1788448147.32 total 932.51 last 20.07) - open/close 19098 (time 1788448157.35 total 942.54 last 16.25) - open/close 19286 (time 1788448167.38 total 952.56 last 18.75) - open/close 19722 (time 1788448177.38 total 962.57 last 43.59) - open/close 20000 (time 1788448184.54 total 969.73 last 38.82) - open/close 20544 (time 1788448194.57 total 979.75 last 54.25) - open/close 21119 (time 1788448204.58 total 989.77 last 57.43) - open/close 21820 (time 1788448214.59 total 999.77 last 70.06) - open/close 22440 (time 1788448224.60 total 1009.78 last 61.93) - open/close 23068 (time 1788448234.63 total 1019.81 last 62.65) - open/close 23750 (time 1788448244.65 total 1029.83 last 68.01) - open/close 24326 (time 1788448254.66 total 1039.84 last 57.57) - open/close 24986 (time 1788448264.66 total 1049.85 last 65.96) - open/close 25762 (time 1788448274.68 total 1059.86 last 77.50) - open/close 26390 (time 1788448284.68 total 1069.86 last 62.79) - open/close 26983 (time 1788448294.68 total 1079.86 last 59.29) - open/close 27640 (time 1788448304.68 total 1089.87 last 65.67) - open/close 28338 (time 1788448314.70 total 1099.88 last 69.72) - open/close 28987 (time 1788448324.70 total 1109.88 last 64.86) - open/close 29683 (time 1788448334.71 total 1119.90 last 69.52) - open/close 30000 (time 1788448339.47 total 1124.65 last 66.65) - open/close 30632 (time 1788448349.48 total 1134.66 last 63.14) - open/close 31266 (time 1788448359.48 total 1144.66 last 63.37) - open/close 31932 (time 1788448369.49 total 1154.68 last 66.53) - open/close 32591 (time 1788448379.49 total 1164.68 last 65.89) - open/close 33251 (time 1788448389.50 total 1174.68 last 65.98) - open/close 33956 (time 1788448399.50 total 1184.69 last 70.46) - open/close 34628 (time 1788448409.51 total 1194.70 last 67.13) - open/close 35247 (time 1788448419.52 total 1204.70 last 61.87) - open/close 35897 (time 1788448429.52 total 1214.71 last 64.97) - open/close 36543 (time 1788448439.53 total 1224.71 last 64.58) - open/close 37183 (time 1788448449.53 total 1234.72 last 63.95) - open/close 37921 (time 1788448459.56 total 1244.74 last 73.62) - open/close 38612 (time 1788448469.56 total 1254.74 last 69.10) - open/close 39302 (time 1788448479.57 total 1264.75 last 68.95) - open/close 39975 (time 1788448489.58 total 1274.76 last 67.23) - open/close 40622 (time 1788448499.58 total 1284.76 last 64.70) - open/close 41263 (time 1788448509.58 total 1294.77 last 64.06) - open/close 41862 (time 1788448519.60 total 1304.78 last 59.82) - open/close 42447 (time 1788448529.60 total 1314.79 last 58.45) - open/close 43028 (time 1788448539.61 total 1324.79 last 58.07) - open/close 43579 (time 1788448549.64 total 1334.82 last 54.93) - open/close 44182 (time 1788448559.65 total 1344.83 last 60.27) - open/close 44742 (time 1788448569.65 total 1354.83 last 55.98) - open/close 45405 (time 1788448579.66 total 1364.84 last 66.22) - open/close 46026 (time 1788448589.68 total 1374.86 last 61.99) - open/close 46631 (time 1788448599.68 total 1384.86 last 60.50) - open/close 47263 (time 1788448609.68 total 1394.86 last 63.19) - open/close 47875 (time 1788448619.69 total 1404.87 last 61.16) - open/close 48495 (time 1788448629.69 total 1414.87 last 61.99) - open/close 49126 (time 1788448639.72 total 1424.90 last 62.92) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1432.38 seconds: 34.65 ops/second - unlinked 0 (time 1788448652 ; total 0 ; last 0) - unlinked 10000 (time 1788448804 ; total 152 ; last 152) - unlinked 20000 (time 1788448960 ; total 308 ; last 156) - unlinked 30000 (time 1788449115 ; total 463 ; last 155) - unlinked 40000 (time 1788449266 ; total 614 ; last 151) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 763 seconds: 65.047180 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local oleg401-server: modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2