== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:31:54 (1788906714) start mds service on oleg130-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg130-server' oleg130-server: oleg130-server.virtnet: executing load_modules_local oleg130-server: Loading modules from /home/green/git/lustre-release/lustre oleg130-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg130-server: detected 4 online CPUs by sysfs oleg130-server: MODOPTS_LIBCFS= oleg130-server: Force libcfs to create 2 CPU partitions oleg130-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg130-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg130-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg130-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg130-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 200 (time 1788906832.97 total 10.01 last 19.98) - open/close 446 (time 1788906842.98 total 20.01 last 24.59) - open/close 645 (time 1788906852.98 total 30.02 last 19.90) - open/close 902 (time 1788906863.70 total 40.74 last 23.96) - open/close 1157 (time 1788906876.40 total 53.44 last 20.08) - open/close 1378 (time 1788906886.65 total 63.69 last 21.57) - open/close 1593 (time 1788906896.93 total 73.97 last 20.91) - open/close 1842 (time 1788906906.93 total 83.97 last 24.89) - open/close 2120 (time 1788906916.95 total 93.99 last 27.76) - open/close 2337 (time 1788906927.74 total 104.77 last 20.12) - open/close 2558 (time 1788906937.75 total 114.78 last 22.08) - open/close 2802 (time 1788906947.76 total 124.80 last 24.35) - open/close 2980 (time 1788906958.70 total 135.73 last 16.28) - open/close 3255 (time 1788906969.64 total 146.68 last 25.13) - open/close 3458 (time 1788906979.94 total 156.98 last 19.71) - open/close 3709 (time 1788906990.77 total 167.81 last 23.17) - open/close 3928 (time 1788907000.78 total 177.82 last 21.88) - open/close 4210 (time 1788907011.62 total 188.66 last 26.02) - open/close 4483 (time 1788907022.24 total 199.28 last 25.70) - open/close 4736 (time 1788907032.24 total 209.28 last 25.30) - open/close 4973 (time 1788907042.26 total 219.29 last 23.67) - open/close 5176 (time 1788907052.33 total 229.37 last 20.15) - open/close 5364 (time 1788907063.34 total 240.38 last 17.07) - open/close 5605 (time 1788907073.35 total 250.39 last 24.07) - open/close 5822 (time 1788907084.40 total 261.44 last 19.64) - open/close 6064 (time 1788907094.94 total 271.98 last 22.96) - open/close 6363 (time 1788907105.69 total 282.73 last 27.82) - open/close 6653 (time 1788907116.27 total 293.31 last 27.42) - open/close 6858 (time 1788907126.76 total 303.80 last 19.54) - open/close 7179 (time 1788907137.77 total 314.81 last 29.16) - open/close 7411 (time 1788907147.77 total 324.81 last 23.19) - open/close 7629 (time 1788907157.94 total 334.98 last 21.45) - open/close 7853 (time 1788907167.94 total 344.98 last 22.40) - open/close 8071 (time 1788907178.05 total 355.09 last 21.56) - open/close 8249 (time 1788907188.90 total 365.94 last 16.40) - open/close 8495 (time 1788907199.27 total 376.30 last 23.74) - open/close 8725 (time 1788907209.79 total 386.83 last 21.85) - open/close 8941 (time 1788907220.02 total 397.06 last 21.11) - open/close 9159 (time 1788907230.04 total 407.08 last 21.75) - open/close 9423 (time 1788907240.05 total 417.09 last 26.38) - open/close 9664 (time 1788907250.06 total 427.10 last 24.08) - open/close 9883 (time 1788907260.21 total 437.25 last 21.58) - open/close 10000 (time 1788907264.29 total 441.33 last 28.71) - open/close 10202 (time 1788907275.24 total 452.28 last 18.45) - open/close 10445 (time 1788907285.25 total 462.29 last 24.26) - open/close 10585 (time 1788907295.26 total 472.29 last 13.99) - open/close 10708 (time 1788907305.33 total 482.37 last 12.20) - open/close 10828 (time 1788907316.25 total 493.29 last 10.99) - open/close 10966 (time 1788907326.26 total 503.30 last 13.79) - open/close 11037 (time 1788907337.13 total 514.17 last 6.53) - open/close 11228 (time 1788907348.14 total 525.18 last 17.35) - open/close 11420 (time 1788907358.83 total 535.87 last 17.96) - open/close 11618 (time 1788907368.89 total 545.93 last 19.68) - open/close 11809 (time 1788907379.11 total 556.15 last 18.69) - open/close 11998 (time 1788907389.11 total 566.15 last 18.90) - open/close 12186 (time 1788907400.04 total 577.08 last 17.21) - open/close 12470 (time 1788907410.07 total 587.10 last 28.33) - open/close 12615 (time 1788907421.10 total 598.14 last 13.14) - open/close 12809 (time 1788907431.10 total 608.14 last 19.40) - open/close 13064 (time 1788907443.55 total 620.59 last 20.49) - open/close 13319 (time 1788907453.55 total 630.59 last 25.49) - open/close 13551 (time 1788907463.71 total 640.75 last 22.84) - open/close 13667 (time 1788907473.73 total 650.77 last 11.57) - open/close 13810 (time 1788907483.74 total 660.78 last 14.29) - open/close 14015 (time 1788907493.76 total 670.80 last 20.46) - open/close 14222 (time 1788907503.77 total 680.80 last 20.69) - open/close 14404 (time 1788907515.32 total 692.36 last 15.75) - open/close 14579 (time 1788907525.36 total 702.40 last 17.43) - open/close 14741 (time 1788907535.77 total 712.81 last 15.56) - open/close 14860 (time 1788907546.31 total 723.34 last 11.30) - open/close 15142 (time 1788907556.64 total 733.68 last 27.29) - open/close 15389 (time 1788907567.40 total 744.44 last 22.95) - open/close 15543 (time 1788907577.64 total 754.68 last 15.04) - open/close 15706 (time 1788907587.89 total 764.92 last 15.91) - open/close 15990 (time 1788907598.50 total 775.54 last 26.76) - open/close 16228 (time 1788907609.20 total 786.24 last 22.23) - open/close 16440 (time 1788907619.31 total 796.35 last 20.97) - open/close 16684 (time 1788907630.18 total 807.22 last 22.46) - open/close 16873 (time 1788907640.20 total 817.24 last 18.85) - open/close 17099 (time 1788907651.16 total 828.19 last 20.63) - open/close 17356 (time 1788907661.86 total 838.90 last 24.01) - open/close 17584 (time 1788907672.42 total 849.45 last 21.59) - open/close 17862 (time 1788907684.59 total 861.63 last 22.83) - open/close 18080 (time 1788907696.82 total 873.86 last 17.83) - open/close 18267 (time 1788907706.83 total 883.87 last 18.68) - open/close 18517 (time 1788907717.22 total 894.26 last 24.05) - open/close 18796 (time 1788907727.86 total 904.89 last 26.24) - open/close 19101 (time 1788907737.88 total 914.91 last 30.44) - open/close 19321 (time 1788907748.51 total 925.55 last 20.69) - open/close 19933 (time 1788907758.51 total 935.55 last 61.16) - open/close 20753 (time 1788907768.52 total 945.56 last 81.96) - open/close 21601 (time 1788907778.53 total 955.57 last 84.71) - open/close 22443 (time 1788907788.54 total 965.58 last 84.09) - open/close 23310 (time 1788907798.55 total 975.58 last 86.67) - open/close 24187 (time 1788907808.55 total 985.59 last 87.69) - open/close 25043 (time 1788907818.55 total 995.59 last 85.57) - open/close 25888 (time 1788907828.58 total 1005.62 last 84.23) - open/close 26695 (time 1788907838.60 total 1015.63 last 80.60) - open/close 27615 (time 1788907848.60 total 1025.64 last 91.94) - open/close 28527 (time 1788907858.61 total 1035.65 last 91.15) - open/close 29354 (time 1788907868.62 total 1045.65 last 82.64) - open/close 30000 (time 1788907876.33 total 1053.37 last 83.71) - open/close 30766 (time 1788907886.34 total 1063.38 last 76.55) - open/close 31583 (time 1788907896.34 total 1073.38 last 81.68) - open/close 32427 (time 1788907906.36 total 1083.40 last 84.26) - open/close 33293 (time 1788907916.37 total 1093.41 last 86.49) - open/close 34096 (time 1788907926.39 total 1103.43 last 80.16) - open/close 34922 (time 1788907936.40 total 1113.43 last 82.53) - open/close 35731 (time 1788907946.40 total 1123.44 last 80.88) - open/close 36568 (time 1788907956.41 total 1133.45 last 83.58) - open/close 37472 (time 1788907966.42 total 1143.45 last 90.37) - open/close 38356 (time 1788907976.43 total 1153.47 last 88.30) - open/close 39267 (time 1788907986.43 total 1163.47 last 91.10) - open/close 40000 (time 1788907995.48 total 1172.52 last 80.96) - open/close 40861 (time 1788908005.49 total 1182.53 last 86.05) - open/close 41832 (time 1788908015.49 total 1192.53 last 97.10) - open/close 42740 (time 1788908025.49 total 1202.53 last 90.78) - open/close 43691 (time 1788908035.50 total 1212.53 last 95.05) - open/close 44587 (time 1788908045.50 total 1222.54 last 89.56) - open/close 45473 (time 1788908055.51 total 1232.55 last 88.53) - open/close 46353 (time 1788908065.51 total 1242.55 last 87.96) - open/close 47269 (time 1788908075.52 total 1252.56 last 91.53) - open/close 48237 (time 1788908085.52 total 1262.56 last 96.79) - open/close 49140 (time 1788908095.53 total 1272.57 last 90.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1277.44 seconds: 38.85 ops/second - unlinked 0 (time 1788908103 ; total 0 ; last 0) - unlinked 10000 (time 1788908205 ; total 102 ; last 102) - unlinked 20000 (time 1788908300 ; total 197 ; last 95) - unlinked 30000 (time 1788908362 ; total 259 ; last 62) - unlinked 40000 (time 1788908419 ; total 316 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 370 seconds: 134.137833 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg130-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg130-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg130-server stop mds service on oleg130-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg130-server stop mds service on oleg130-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg130-server unloading modules via unload_modules_local on: 'oleg130-server' oleg130-server: oleg130-server.virtnet: executing unload_modules_local oleg130-server: modules unloaded. pdsh@oleg130-client: oleg130-server: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-client: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-client: ssh exited with exit code 2