== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:53:08 (1787565188) start mds service on oleg659-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing load_modules_local oleg659-server: Loading modules from /home/green/git/lustre-release/lustre oleg659-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg659-server: detected 4 online CPUs by sysfs oleg659-server: MODOPTS_LIBCFS= oleg659-server: Force libcfs to create 2 CPU partitions oleg659-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg659-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg659-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg659-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 224 (time 1787565284.92 total 10.76 last 20.83) - open/close 531 (time 1787565294.93 total 20.77 last 30.66) - open/close 787 (time 1787565304.98 total 30.82 last 25.46) - open/close 1003 (time 1787565315.01 total 40.85 last 21.54) - open/close 1257 (time 1787565325.02 total 50.86 last 25.38) - open/close 1514 (time 1787565336.00 total 61.84 last 23.40) - open/close 1699 (time 1787565348.72 total 74.56 last 14.54) - open/close 1967 (time 1787565359.24 total 85.07 last 25.49) - open/close 2226 (time 1787565369.41 total 95.24 last 25.47) - open/close 2500 (time 1787565379.43 total 105.27 last 27.32) - open/close 2777 (time 1787565390.32 total 116.15 last 25.45) - open/close 3044 (time 1787565400.32 total 126.16 last 26.69) - open/close 3357 (time 1787565410.34 total 136.17 last 31.25) - open/close 3521 (time 1787565420.41 total 146.24 last 16.29) - open/close 3749 (time 1787565431.11 total 156.94 last 21.31) - open/close 4040 (time 1787565441.66 total 167.50 last 27.57) - open/close 4324 (time 1787565452.10 total 177.93 last 27.21) - open/close 4642 (time 1787565462.11 total 187.95 last 31.76) - open/close 4909 (time 1787565472.24 total 198.08 last 26.35) - open/close 5187 (time 1787565483.12 total 208.96 last 25.56) - open/close 5289 (time 1787565493.12 total 218.96 last 10.20) - open/close 5439 (time 1787565503.60 total 229.44 last 14.31) - open/close 5504 (time 1787565514.30 total 240.14 last 6.08) - open/close 5632 (time 1787565525.23 total 251.07 last 11.71) - open/close 5799 (time 1787565536.91 total 262.75 last 14.30) - open/close 6083 (time 1787565547.60 total 273.43 last 26.58) - open/close 6368 (time 1787565558.11 total 283.95 last 27.10) - open/close 6556 (time 1787565568.91 total 294.75 last 17.41) - open/close 6814 (time 1787565579.76 total 305.59 last 23.79) - open/close 7072 (time 1787565589.88 total 315.72 last 25.48) - open/close 7355 (time 1787565600.96 total 326.80 last 25.55) - open/close 7577 (time 1787565610.98 total 336.81 last 22.17) - open/close 7865 (time 1787565621.67 total 347.51 last 26.92) - open/close 8135 (time 1787565632.63 total 358.46 last 24.65) - open/close 8389 (time 1787565642.87 total 368.71 last 24.80) - open/close 8609 (time 1787565653.68 total 379.52 last 20.34) - open/close 8841 (time 1787565663.71 total 389.54 last 23.14) - open/close 9112 (time 1787565674.74 total 400.57 last 24.57) - open/close 9305 (time 1787565684.74 total 410.58 last 19.29) - open/close 9530 (time 1787565695.18 total 421.02 last 21.55) - open/close 9809 (time 1787565706.18 total 432.01 last 25.38) - open/close 10000 (time 1787565711.68 total 437.52 last 34.69) - open/close 10253 (time 1787565721.74 total 447.57 last 25.17) - open/close 10500 (time 1787565732.60 total 458.44 last 22.73) - open/close 10769 (time 1787565742.61 total 468.44 last 26.90) - open/close 11007 (time 1787565752.63 total 478.47 last 23.74) - open/close 11222 (time 1787565763.06 total 488.90 last 20.61) - open/close 11532 (time 1787565773.07 total 498.91 last 30.96) - open/close 11719 (time 1787565783.48 total 509.31 last 17.97) - open/close 12037 (time 1787565796.09 total 521.92 last 25.22) - open/close 12278 (time 1787565806.09 total 531.93 last 24.08) - open/close 12577 (time 1787565816.09 total 541.93 last 29.90) - open/close 12826 (time 1787565827.25 total 553.09 last 22.31) - open/close 13123 (time 1787565838.26 total 564.10 last 26.97) - open/close 13402 (time 1787565848.63 total 574.47 last 26.91) - open/close 13682 (time 1787565858.66 total 584.49 last 27.94) - open/close 13919 (time 1787565868.66 total 594.49 last 23.70) - open/close 14199 (time 1787565879.40 total 605.24 last 26.06) - open/close 14487 (time 1787565889.71 total 615.55 last 27.93) - open/close 14814 (time 1787565899.71 total 625.55 last 32.70) - open/close 15095 (time 1787565910.03 total 635.87 last 27.24) - open/close 15343 (time 1787565920.03 total 645.87 last 24.79) - open/close 15620 (time 1787565930.50 total 656.33 last 26.47) - open/close 15902 (time 1787565940.50 total 666.34 last 28.18) - open/close 16183 (time 1787565950.58 total 676.41 last 27.89) - open/close 16448 (time 1787565960.59 total 686.42 last 26.48) - open/close 16697 (time 1787565970.93 total 696.77 last 24.07) - open/close 16940 (time 1787565980.94 total 706.78 last 24.27) - open/close 17177 (time 1787565990.98 total 716.82 last 23.61) - open/close 17436 (time 1787566000.98 total 726.82 last 25.89) - open/close 17729 (time 1787566010.99 total 736.82 last 29.29) - open/close 17932 (time 1787566022.00 total 747.83 last 18.43) - open/close 18239 (time 1787566032.83 total 758.67 last 28.33) - open/close 18444 (time 1787566043.40 total 769.23 last 19.40) - open/close 18695 (time 1787566055.82 total 781.65 last 20.21) - open/close 18995 (time 1787566066.80 total 792.64 last 27.32) - open/close 19252 (time 1787566076.80 total 802.64 last 25.69) - open/close 19555 (time 1787566087.30 total 813.13 last 28.88) - open/close 19724 (time 1787566097.30 total 823.13 last 16.90) - open/close 19982 (time 1787566107.31 total 833.15 last 25.76) - open/close 20147 (time 1787566117.56 total 843.40 last 16.10) - open/close 20340 (time 1787566128.18 total 854.02 last 18.17) - open/close 20495 (time 1787566138.32 total 864.16 last 15.29) - open/close 20758 (time 1787566149.17 total 875.00 last 24.24) - open/close 21031 (time 1787566159.17 total 885.01 last 27.29) - open/close 21302 (time 1787566169.17 total 895.01 last 27.10) - open/close 21885 (time 1787566179.18 total 905.01 last 58.28) - open/close 22844 (time 1787566189.19 total 915.03 last 95.75) - open/close 23788 (time 1787566199.19 total 925.03 last 94.37) - open/close 24761 (time 1787566209.20 total 935.04 last 97.23) - open/close 25681 (time 1787566219.21 total 945.05 last 91.90) - open/close 26624 (time 1787566229.22 total 955.05 last 94.26) - open/close 27509 (time 1787566239.23 total 965.06 last 88.42) - open/close 28513 (time 1787566249.24 total 975.07 last 100.30) - open/close 29520 (time 1787566259.25 total 985.09 last 100.57) - open/close 30000 (time 1787566264.46 total 990.30 last 92.05) - open/close 31042 (time 1787566274.47 total 1000.30 last 104.17) - open/close 31996 (time 1787566284.47 total 1010.31 last 95.33) - open/close 32919 (time 1787566294.47 total 1020.31 last 92.29) - open/close 33842 (time 1787566304.48 total 1030.32 last 92.25) - open/close 34835 (time 1787566314.49 total 1040.33 last 99.21) - open/close 35835 (time 1787566324.49 total 1050.33 last 99.95) - open/close 36706 (time 1787566334.51 total 1060.35 last 86.94) - open/close 37678 (time 1787566344.51 total 1070.35 last 97.17) - open/close 38621 (time 1787566354.52 total 1080.35 last 94.29) - open/close 39690 (time 1787566364.53 total 1090.37 last 106.76) - open/close 40000 (time 1787566368.73 total 1094.57 last 73.78) - open/close 40900 (time 1787566378.75 total 1104.59 last 89.83) - open/close 41898 (time 1787566388.75 total 1114.59 last 99.76) - open/close 42780 (time 1787566398.76 total 1124.59 last 88.18) - open/close 43837 (time 1787566408.76 total 1134.59 last 105.70) - open/close 44712 (time 1787566418.76 total 1144.60 last 87.43) - open/close 45397 (time 1787566428.77 total 1154.61 last 68.44) - open/close 45881 (time 1787566438.77 total 1164.61 last 48.39) - open/close 46666 (time 1787566448.77 total 1174.61 last 78.50) - open/close 47603 (time 1787566458.78 total 1184.62 last 93.63) - open/close 48540 (time 1787566468.79 total 1194.63 last 93.60) - open/close 49407 (time 1787566478.81 total 1204.65 last 86.54) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1207.10 seconds: 41.12 ops/second - unlinked 0 (time 1787566486 ; total 0 ; last 0) - unlinked 10000 (time 1787566593 ; total 107 ; last 107) - unlinked 20000 (time 1787566697 ; total 211 ; last 104) - unlinked 30000 (time 1787566757 ; total 271 ; last 60) - unlinked 40000 (time 1787566821 ; total 335 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 390 seconds: 127.258972 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg659-server unloading modules via unload_modules_local on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing unload_modules_local oleg659-server: modules unloaded. pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2