== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:52:26 (1782229946) start mds service on oleg302-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg302-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg302-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 163 (time 1782230043.75 total 10.01 last 16.28) - open/close 425 (time 1782230054.16 total 20.43 last 25.16) - open/close 679 (time 1782230065.22 total 31.49 last 22.97) - open/close 938 (time 1782230075.40 total 41.66 last 25.45) - open/close 1181 (time 1782230086.21 total 52.48 last 22.47) - open/close 1442 (time 1782230096.22 total 62.48 last 26.08) - open/close 1597 (time 1782230107.16 total 73.43 last 14.16) - open/close 1901 (time 1782230117.76 total 84.03 last 28.68) - open/close 2187 (time 1782230128.46 total 94.73 last 26.73) - open/close 2443 (time 1782230138.47 total 104.73 last 25.59) - open/close 2693 (time 1782230148.47 total 114.73 last 25.00) - open/close 2855 (time 1782230158.49 total 124.76 last 16.15) - open/close 3100 (time 1782230168.51 total 134.77 last 24.46) - open/close 3375 (time 1782230178.57 total 144.83 last 27.34) - open/close 3665 (time 1782230189.32 total 155.59 last 26.97) - open/close 3902 (time 1782230200.01 total 166.27 last 22.18) - open/close 4103 (time 1782230210.32 total 176.59 last 19.48) - open/close 4369 (time 1782230220.33 total 186.59 last 26.59) - open/close 4667 (time 1782230230.35 total 196.61 last 29.74) - open/close 4865 (time 1782230240.36 total 206.63 last 19.77) - open/close 5132 (time 1782230250.64 total 216.90 last 25.99) - open/close 5454 (time 1782230261.40 total 227.66 last 29.93) - open/close 5714 (time 1782230271.50 total 237.77 last 25.73) - open/close 5975 (time 1782230282.52 total 248.79 last 23.68) - open/close 6222 (time 1782230292.56 total 258.82 last 24.62) - open/close 6425 (time 1782230302.80 total 269.06 last 19.81) - open/close 6689 (time 1782230313.29 total 279.56 last 25.17) - open/close 6989 (time 1782230323.29 total 289.56 last 30.00) - open/close 7233 (time 1782230333.65 total 299.92 last 23.55) - open/close 7509 (time 1782230343.82 total 310.09 last 27.14) - open/close 7747 (time 1782230354.82 total 321.09 last 21.64) - open/close 8046 (time 1782230364.83 total 331.09 last 29.88) - open/close 8294 (time 1782230374.87 total 341.13 last 24.71) - open/close 8560 (time 1782230384.87 total 351.13 last 26.60) - open/close 8785 (time 1782230394.88 total 361.14 last 22.48) - open/close 9062 (time 1782230404.88 total 371.15 last 27.69) - open/close 9322 (time 1782230415.11 total 381.38 last 25.41) - open/close 9585 (time 1782230425.64 total 391.91 last 24.98) - open/close 9843 (time 1782230435.97 total 402.24 last 24.98) - open/close 10000 (time 1782230441.44 total 407.71 last 28.69) - open/close 10227 (time 1782230451.46 total 417.73 last 22.65) - open/close 10465 (time 1782230461.59 total 427.85 last 23.51) - open/close 10709 (time 1782230471.82 total 438.09 last 23.85) - open/close 10974 (time 1782230481.83 total 448.09 last 26.49) - open/close 11203 (time 1782230491.92 total 458.18 last 22.69) - open/close 11423 (time 1782230502.03 total 468.30 last 21.75) - open/close 11544 (time 1782230512.59 total 478.86 last 11.46) - open/close 11758 (time 1782230523.48 total 489.75 last 19.65) - open/close 11833 (time 1782230533.54 total 499.80 last 7.46) - open/close 11905 (time 1782230543.56 total 509.82 last 7.19) - open/close 12082 (time 1782230553.68 total 519.94 last 17.49) - open/close 12379 (time 1782230563.69 total 529.96 last 29.65) - open/close 12633 (time 1782230573.71 total 539.97 last 25.37) - open/close 12837 (time 1782230583.81 total 550.08 last 20.19) - open/close 12995 (time 1782230594.00 total 560.27 last 15.50) - open/close 13306 (time 1782230604.01 total 570.27 last 31.08) - open/close 13555 (time 1782230614.73 total 580.99 last 23.24) - open/close 13855 (time 1782230624.74 total 591.00 last 29.96) - open/close 13999 (time 1782230634.76 total 601.03 last 14.36) - open/close 14187 (time 1782230646.37 total 612.63 last 16.21) - open/close 14359 (time 1782230656.39 total 622.66 last 17.16) - open/close 14604 (time 1782230666.52 total 632.78 last 24.20) - open/close 14912 (time 1782230676.52 total 642.79 last 30.78) - open/close 15184 (time 1782230687.12 total 653.39 last 25.66) - open/close 15476 (time 1782230698.00 total 664.26 last 26.85) - open/close 15682 (time 1782230708.06 total 674.32 last 20.48) - open/close 15911 (time 1782230718.09 total 684.35 last 22.83) - open/close 16152 (time 1782230728.42 total 694.68 last 23.33) - open/close 16407 (time 1782230739.14 total 705.41 last 23.78) - open/close 16503 (time 1782230749.52 total 715.78 last 9.25) - open/close 16670 (time 1782230759.82 total 726.08 last 16.21) - open/close 16851 (time 1782230771.09 total 737.35 last 16.06) - open/close 17055 (time 1782230782.35 total 748.61 last 18.12) - open/close 17156 (time 1782230793.93 total 760.19 last 8.72) - open/close 17343 (time 1782230806.82 total 773.08 last 14.51) - open/close 17551 (time 1782230821.40 total 787.66 last 14.27) - open/close 17796 (time 1782230832.40 total 798.67 last 22.26) - open/close 18008 (time 1782230842.41 total 808.68 last 21.18) - open/close 18299 (time 1782230852.51 total 818.77 last 28.83) - open/close 18546 (time 1782230863.55 total 829.82 last 22.36) - open/close 18736 (time 1782230873.57 total 839.83 last 18.97) - open/close 18885 (time 1782230884.32 total 850.58 last 13.86) - open/close 19170 (time 1782230894.33 total 860.60 last 28.47) - open/close 19400 (time 1782230904.99 total 871.25 last 21.58) - open/close 19633 (time 1782230915.33 total 881.59 last 22.53) - open/close 19915 (time 1782230925.33 total 891.60 last 28.19) - open/close 20000 (time 1782230928.84 total 895.10 last 24.27) - open/close 20243 (time 1782230938.85 total 905.11 last 24.28) - open/close 20528 (time 1782230949.65 total 915.91 last 26.38) - open/close 20902 (time 1782230959.66 total 925.92 last 37.37) - open/close 21616 (time 1782230969.66 total 935.92 last 71.38) - open/close 22537 (time 1782230979.67 total 945.93 last 92.05) - open/close 23507 (time 1782230989.67 total 955.93 last 96.95) - open/close 24454 (time 1782230999.67 total 965.94 last 94.70) - open/close 25423 (time 1782231009.67 total 975.94 last 96.86) - open/close 26210 (time 1782231019.69 total 985.95 last 78.59) - open/close 27231 (time 1782231029.70 total 995.96 last 101.99) - open/close 28191 (time 1782231039.70 total 1005.96 last 96.00) - open/close 29100 (time 1782231049.70 total 1015.96 last 90.90) - open/close 30000 (time 1782231059.42 total 1025.68 last 92.58) - open/close 30933 (time 1782231069.42 total 1035.69 last 93.26) - open/close 31759 (time 1782231079.43 total 1045.69 last 82.58) - open/close 32640 (time 1782231089.44 total 1055.70 last 87.99) - open/close 33557 (time 1782231099.47 total 1065.74 last 91.40) - open/close 34395 (time 1782231109.48 total 1075.74 last 83.76) - open/close 35364 (time 1782231119.49 total 1085.75 last 96.81) - open/close 36247 (time 1782231129.49 total 1095.75 last 88.30) - open/close 37154 (time 1782231139.50 total 1105.76 last 90.59) - open/close 38110 (time 1782231149.51 total 1115.77 last 95.50) - open/close 39079 (time 1782231159.51 total 1125.77 last 96.88) - open/close 40000 (time 1782231168.62 total 1134.88 last 101.14) - open/close 40989 (time 1782231178.62 total 1144.89 last 98.82) - open/close 41991 (time 1782231188.63 total 1154.89 last 100.14) - open/close 42897 (time 1782231198.64 total 1164.90 last 90.55) - open/close 43815 (time 1782231208.64 total 1174.91 last 91.75) - open/close 44788 (time 1782231218.64 total 1184.91 last 97.29) - open/close 45773 (time 1782231228.64 total 1194.91 last 98.50) - open/close 46805 (time 1782231238.64 total 1204.91 last 103.18) - open/close 47771 (time 1782231248.65 total 1214.91 last 96.59) - open/close 48746 (time 1782231258.65 total 1224.92 last 97.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1234.18 seconds: 40.21 ops/second - unlinked 0 (time 1782231271 ; total 0 ; last 0) - unlinked 10000 (time 1782231372 ; total 101 ; last 101) - unlinked 20000 (time 1782231464 ; total 193 ; last 92) - unlinked 30000 (time 1782231520 ; total 249 ; last 56) - unlinked 40000 (time 1782231577 ; total 306 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 359 seconds: 138.247910 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2