== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:54:23 (1789124063) start mds service on oleg432-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg432-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg432-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 168 (time 1789124182.31 total 10.61 last 15.83) - open/close 349 (time 1789124193.05 total 21.35 last 16.85) - open/close 541 (time 1789124203.07 total 31.37 last 19.16) - open/close 785 (time 1789124213.09 total 41.39 last 24.35) - open/close 982 (time 1789124223.09 total 51.39 last 19.70) - open/close 1198 (time 1789124233.09 total 61.39 last 21.60) - open/close 1369 (time 1789124244.05 total 72.35 last 15.61) - open/close 1562 (time 1789124254.16 total 82.47 last 19.08) - open/close 1776 (time 1789124265.12 total 93.42 last 19.53) - open/close 1998 (time 1789124275.13 total 103.43 last 22.18) - open/close 2196 (time 1789124286.42 total 114.72 last 17.54) - open/close 2449 (time 1789124296.43 total 124.74 last 25.26) - open/close 2674 (time 1789124307.08 total 135.39 last 21.13) - open/close 2966 (time 1789124317.83 total 146.14 last 27.16) - open/close 3164 (time 1789124328.60 total 156.91 last 18.38) - open/close 3407 (time 1789124339.48 total 167.79 last 22.34) - open/close 3560 (time 1789124349.52 total 177.82 last 15.25) - open/close 3813 (time 1789124359.54 total 187.84 last 25.25) - open/close 3973 (time 1789124369.70 total 198.01 last 15.74) - open/close 4118 (time 1789124380.58 total 208.88 last 13.33) - open/close 4237 (time 1789124391.61 total 219.91 last 10.79) - open/close 4371 (time 1789124401.61 total 229.92 last 13.39) - open/close 4515 (time 1789124411.62 total 239.93 last 14.39) - open/close 4587 (time 1789124422.61 total 250.91 last 6.55) - open/close 4723 (time 1789124433.21 total 261.51 last 12.84) - open/close 4863 (time 1789124443.23 total 271.53 last 13.96) - open/close 5012 (time 1789124453.26 total 281.56 last 14.86) - open/close 5185 (time 1789124463.32 total 291.62 last 17.19) - open/close 5348 (time 1789124473.67 total 301.97 last 15.74) - open/close 5506 (time 1789124484.71 total 313.01 last 14.31) - open/close 5714 (time 1789124494.99 total 323.29 last 20.23) - open/close 5919 (time 1789124505.02 total 333.32 last 20.44) - open/close 6144 (time 1789124515.67 total 343.97 last 21.13) - open/close 6376 (time 1789124525.84 total 354.15 last 22.80) - open/close 6547 (time 1789124536.40 total 364.70 last 16.20) - open/close 6757 (time 1789124546.41 total 374.71 last 20.98) - open/close 6956 (time 1789124556.57 total 384.87 last 19.59) - open/close 7148 (time 1789124567.44 total 395.75 last 17.65) - open/close 7396 (time 1789124577.44 total 405.75 last 24.80) - open/close 7586 (time 1789124588.16 total 416.47 last 17.72) - open/close 7652 (time 1789124599.01 total 427.31 last 6.09) - open/close 7781 (time 1789124609.95 total 438.26 last 11.78) - open/close 7955 (time 1789124621.41 total 449.72 last 15.18) - open/close 8048 (time 1789124632.04 total 460.34 last 8.75) - open/close 8231 (time 1789124642.51 total 470.81 last 17.48) - open/close 8415 (time 1789124653.41 total 481.72 last 16.88) - open/close 8654 (time 1789124664.14 total 492.44 last 22.28) - open/close 8879 (time 1789124674.57 total 502.88 last 21.56) - open/close 9179 (time 1789124685.04 total 513.34 last 28.68) - open/close 9385 (time 1789124695.04 total 523.35 last 20.59) - open/close 9591 (time 1789124705.05 total 533.36 last 20.57) - open/close 9828 (time 1789124715.92 total 544.23 last 21.81) - open/close 10000 (time 1789124722.67 total 550.97 last 25.50) - open/close 10237 (time 1789124732.68 total 560.98 last 23.68) - open/close 10361 (time 1789124742.70 total 571.00 last 12.37) - open/close 10532 (time 1789124752.71 total 581.01 last 17.08) - open/close 10690 (time 1789124763.35 total 591.65 last 14.85) - open/close 10953 (time 1789124774.02 total 602.33 last 24.63) - open/close 11158 (time 1789124784.03 total 612.33 last 20.50) - open/close 11420 (time 1789124794.72 total 623.02 last 24.50) - open/close 11631 (time 1789124805.79 total 634.09 last 19.07) - open/close 11804 (time 1789124816.72 total 645.03 last 15.82) - open/close 12035 (time 1789124827.80 total 656.10 last 20.86) - open/close 12214 (time 1789124838.42 total 666.72 last 16.85) - open/close 12280 (time 1789124850.40 total 678.70 last 5.51) - open/close 12373 (time 1789124860.61 total 688.91 last 9.11) - open/close 12449 (time 1789124871.38 total 699.69 last 7.05) - open/close 12583 (time 1789124881.41 total 709.71 last 13.37) - open/close 12811 (time 1789124891.81 total 720.11 last 21.93) - open/close 12947 (time 1789124904.42 total 732.72 last 10.79) - open/close 13058 (time 1789124914.83 total 743.14 last 10.65) - open/close 13280 (time 1789124925.35 total 753.65 last 21.12) - open/close 13583 (time 1789124935.37 total 763.67 last 30.23) - open/close 13631 (time 1789124946.67 total 774.97 last 4.25) - open/close 13831 (time 1789124956.67 total 784.98 last 19.98) - open/close 14010 (time 1789124967.44 total 795.74 last 16.63) - open/close 14223 (time 1789124977.47 total 805.77 last 21.25) - open/close 14379 (time 1789124987.92 total 816.23 last 14.92) - open/close 14498 (time 1789124998.61 total 826.91 last 11.14) - open/close 14663 (time 1789125011.93 total 840.23 last 12.38) - open/close 14852 (time 1789125021.93 total 850.24 last 18.89) - open/close 15062 (time 1789125031.94 total 860.25 last 20.98) - open/close 15231 (time 1789125042.52 total 870.82 last 15.97) - open/close 15400 (time 1789125053.60 total 881.90 last 15.26) - open/close 15637 (time 1789125064.35 total 892.65 last 22.05) - open/close 15886 (time 1789125074.54 total 902.85 last 24.41) - open/close 16106 (time 1789125085.65 total 913.96 last 19.80) - open/close 16351 (time 1789125095.66 total 923.96 last 24.49) - open/close 16527 (time 1789125106.07 total 934.37 last 16.90) - open/close 16758 (time 1789125116.07 total 944.38 last 23.09) - open/close 16898 (time 1789125126.09 total 954.39 last 13.98) - open/close 17036 (time 1789125136.11 total 964.42 last 13.77) - open/close 17234 (time 1789125146.20 total 974.50 last 19.63) - open/close 17405 (time 1789125156.21 total 984.51 last 17.08) - open/close 17637 (time 1789125166.22 total 994.53 last 23.17) - open/close 17850 (time 1789125176.22 total 1004.53 last 21.30) - open/close 17981 (time 1789125189.23 total 1017.53 last 10.07) - open/close 18232 (time 1789125199.59 total 1027.89 last 24.23) - open/close 18513 (time 1789125209.63 total 1037.93 last 27.98) - open/close 19258 (time 1789125219.65 total 1047.95 last 74.36) - open/close 19995 (time 1789125229.65 total 1057.95 last 73.68) - open/close 20733 (time 1789125239.68 total 1067.99 last 73.55) - open/close 21457 (time 1789125249.71 total 1078.01 last 72.24) - open/close 22221 (time 1789125259.71 total 1088.01 last 76.40) - open/close 22981 (time 1789125269.71 total 1098.01 last 75.99) - open/close 23649 (time 1789125279.71 total 1108.02 last 66.77) - open/close 24401 (time 1789125289.73 total 1118.03 last 75.09) - open/close 25190 (time 1789125299.74 total 1128.04 last 78.81) - open/close 25935 (time 1789125309.76 total 1138.06 last 74.35) - open/close 26716 (time 1789125319.78 total 1148.08 last 77.97) - open/close 27347 (time 1789125329.78 total 1158.09 last 63.05) - open/close 28032 (time 1789125339.79 total 1168.10 last 68.46) - open/close 28723 (time 1789125349.81 total 1178.12 last 68.94) - open/close 29514 (time 1789125359.81 total 1188.12 last 79.10) - open/close 30000 (time 1789125366.55 total 1194.85 last 72.21) - open/close 30613 (time 1789125376.57 total 1204.87 last 61.17) - open/close 31158 (time 1789125386.58 total 1214.89 last 54.42) - open/close 31722 (time 1789125396.58 total 1224.89 last 56.39) - open/close 32361 (time 1789125406.59 total 1234.89 last 63.89) - open/close 33051 (time 1789125416.59 total 1244.89 last 68.98) - open/close 33765 (time 1789125426.60 total 1254.90 last 71.34) - open/close 34479 (time 1789125436.61 total 1264.91 last 71.32) - open/close 35237 (time 1789125446.61 total 1274.92 last 75.75) - open/close 36026 (time 1789125456.62 total 1284.93 last 78.85) - open/close 36650 (time 1789125466.63 total 1294.93 last 62.37) - open/close 37264 (time 1789125476.65 total 1304.95 last 61.28) - open/close 38004 (time 1789125486.65 total 1314.96 last 73.96) - open/close 38783 (time 1789125496.66 total 1324.96 last 77.86) - open/close 39475 (time 1789125506.66 total 1334.96 last 69.20) - open/close 40000 (time 1789125513.37 total 1341.68 last 78.14) - open/close 40690 (time 1789125523.38 total 1351.68 last 68.97) - open/close 41406 (time 1789125533.39 total 1361.69 last 71.55) - open/close 42163 (time 1789125543.39 total 1371.69 last 75.67) - open/close 42869 (time 1789125553.39 total 1381.70 last 70.56) - open/close 43658 (time 1789125563.40 total 1391.71 last 78.84) - open/close 44359 (time 1789125573.42 total 1401.72 last 69.98) - open/close 44994 (time 1789125583.44 total 1411.74 last 63.39) - open/close 45539 (time 1789125593.44 total 1421.74 last 54.48) - open/close 46040 (time 1789125603.44 total 1431.75 last 50.07) - open/close 46664 (time 1789125613.45 total 1441.75 last 62.37) - open/close 47312 (time 1789125623.45 total 1451.76 last 64.78) - open/close 47928 (time 1789125633.49 total 1461.79 last 61.38) - open/close 48671 (time 1789125643.49 total 1471.80 last 74.26) - open/close 49377 (time 1789125653.51 total 1481.82 last 70.47) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1485.24 seconds: 33.42 ops/second - unlinked 0 (time 1789125662 ; total 0 ; last 0) - unlinked 10000 (time 1789125814 ; total 152 ; last 152) - unlinked 20000 (time 1789125956 ; total 294 ; last 142) - unlinked 30000 (time 1789126099 ; total 437 ; last 143) - unlinked 40000 (time 1789126243 ; total 581 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 719 seconds: 69.026428 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2