== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:03:48 (1787709828) start mds service on oleg613-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing load_modules_local oleg613-server: Loading modules from /home/green/git/lustre-release/lustre oleg613-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg613-server: detected 4 online CPUs by sysfs oleg613-server: MODOPTS_LIBCFS= oleg613-server: Force libcfs to create 2 CPU partitions oleg613-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg613-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg613-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 263 (time 1787709901.78 total 10.01 last 26.27) - open/close 461 (time 1787709911.92 total 20.15 last 19.53) - open/close 731 (time 1787709921.93 total 30.16 last 26.99) - open/close 979 (time 1787709931.93 total 40.16 last 24.79) - open/close 1200 (time 1787709942.07 total 50.29 last 21.81) - open/close 1469 (time 1787709952.08 total 60.31 last 26.85) - open/close 1718 (time 1787709962.10 total 70.33 last 24.86) - open/close 1966 (time 1787709972.80 total 81.03 last 23.17) - open/close 2169 (time 1787709982.81 total 91.04 last 20.28) - open/close 2407 (time 1787709993.53 total 101.76 last 22.20) - open/close 2688 (time 1787710003.90 total 112.13 last 27.10) - open/close 2963 (time 1787710013.90 total 122.13 last 27.50) - open/close 3054 (time 1787710024.09 total 132.32 last 8.93) - open/close 3257 (time 1787710034.10 total 142.33 last 20.29) - open/close 3505 (time 1787710045.06 total 153.28 last 22.63) - open/close 3818 (time 1787710055.86 total 164.09 last 28.97) - open/close 4069 (time 1787710066.29 total 174.51 last 24.07) - open/close 4324 (time 1787710076.40 total 184.63 last 25.21) - open/close 4617 (time 1787710086.77 total 195.00 last 28.27) - open/close 4870 (time 1787710096.78 total 205.00 last 25.28) - open/close 5130 (time 1787710107.58 total 215.81 last 24.06) - open/close 5424 (time 1787710118.27 total 226.50 last 27.52) - open/close 5657 (time 1787710128.27 total 236.50 last 23.28) - open/close 5911 (time 1787710138.83 total 247.06 last 24.06) - open/close 6183 (time 1787710148.84 total 257.06 last 27.18) - open/close 6419 (time 1787710158.85 total 267.07 last 23.58) - open/close 6658 (time 1787710168.96 total 277.19 last 23.62) - open/close 6963 (time 1787710178.96 total 287.19 last 30.50) - open/close 7193 (time 1787710188.98 total 297.21 last 22.96) - open/close 7458 (time 1787710199.35 total 307.58 last 25.55) - open/close 7726 (time 1787710210.29 total 318.52 last 24.50) - open/close 7977 (time 1787710220.30 total 328.52 last 25.09) - open/close 8243 (time 1787710230.58 total 338.81 last 25.86) - open/close 8511 (time 1787710240.98 total 349.20 last 25.79) - open/close 8744 (time 1787710251.71 total 359.94 last 21.71) - open/close 8996 (time 1787710261.71 total 369.94 last 25.20) - open/close 9229 (time 1787710271.71 total 379.94 last 23.29) - open/close 9394 (time 1787710281.72 total 389.95 last 16.49) - open/close 9645 (time 1787710291.84 total 400.07 last 24.80) - open/close 9914 (time 1787710302.14 total 410.37 last 26.12) - open/close 10000 (time 1787710305.60 total 413.83 last 24.84) - open/close 10232 (time 1787710316.47 total 424.70 last 21.35) - open/close 10444 (time 1787710326.92 total 435.15 last 20.28) - open/close 10660 (time 1787710337.69 total 445.92 last 20.06) - open/close 10898 (time 1787710347.71 total 455.94 last 23.76) - open/close 11166 (time 1787710357.73 total 465.96 last 26.74) - open/close 11467 (time 1787710367.73 total 475.96 last 30.10) - open/close 11723 (time 1787710377.73 total 485.96 last 25.59) - open/close 11942 (time 1787710387.74 total 495.97 last 21.89) - open/close 12156 (time 1787710398.65 total 506.87 last 19.62) - open/close 12404 (time 1787710408.89 total 517.12 last 24.22) - open/close 12662 (time 1787710419.26 total 527.49 last 24.86) - open/close 12910 (time 1787710429.28 total 537.51 last 24.76) - open/close 13173 (time 1787710439.60 total 547.83 last 25.47) - open/close 13443 (time 1787710450.18 total 558.40 last 25.54) - open/close 13681 (time 1787710461.02 total 569.25 last 21.95) - open/close 13966 (time 1787710471.02 total 579.25 last 28.49) - open/close 14194 (time 1787710481.08 total 589.31 last 22.66) - open/close 14448 (time 1787710492.57 total 600.79 last 22.12) - open/close 14733 (time 1787710502.83 total 611.06 last 27.76) - open/close 14954 (time 1787710513.65 total 621.88 last 20.42) - open/close 15209 (time 1787710525.17 total 633.40 last 22.14) - open/close 15437 (time 1787710535.18 total 643.41 last 22.79) - open/close 15708 (time 1787710545.80 total 654.03 last 25.51) - open/close 15984 (time 1787710556.64 total 664.87 last 25.47) - open/close 16246 (time 1787710566.90 total 675.12 last 25.54) - open/close 16492 (time 1787710576.90 total 685.13 last 24.58) - open/close 16756 (time 1787710586.91 total 695.14 last 26.37) - open/close 16993 (time 1787710598.93 total 707.16 last 19.72) - open/close 17194 (time 1787710608.97 total 717.20 last 20.02) - open/close 17332 (time 1787710619.00 total 727.23 last 13.76) - open/close 17545 (time 1787710629.13 total 737.36 last 21.03) - open/close 17790 (time 1787710639.61 total 747.84 last 23.38) - open/close 17990 (time 1787710649.62 total 757.85 last 19.99) - open/close 18266 (time 1787710659.62 total 767.85 last 27.59) - open/close 18543 (time 1787710670.52 total 778.75 last 25.41) - open/close 18768 (time 1787710680.58 total 788.81 last 22.37) - open/close 18993 (time 1787710690.81 total 799.04 last 21.99) - open/close 19214 (time 1787710701.01 total 809.24 last 21.68) - open/close 19423 (time 1787710711.83 total 820.05 last 19.32) - open/close 19534 (time 1787710722.33 total 830.56 last 10.57) - open/close 19777 (time 1787710732.99 total 841.22 last 22.78) - open/close 20000 (time 1787710742.14 total 850.37 last 24.38) - open/close 20263 (time 1787710752.82 total 861.05 last 24.63) - open/close 20512 (time 1787710762.83 total 871.06 last 24.86) - open/close 21125 (time 1787710772.84 total 881.07 last 61.28) - open/close 21957 (time 1787710782.84 total 891.07 last 83.19) - open/close 22739 (time 1787710792.84 total 901.07 last 78.15) - open/close 23569 (time 1787710802.86 total 911.09 last 82.89) - open/close 24397 (time 1787710812.86 total 921.09 last 82.77) - open/close 25233 (time 1787710822.87 total 931.10 last 83.56) - open/close 26102 (time 1787710832.87 total 941.10 last 86.89) - open/close 26934 (time 1787710842.87 total 951.10 last 83.15) - open/close 27764 (time 1787710852.88 total 961.11 last 82.97) - open/close 28617 (time 1787710862.89 total 971.12 last 85.22) - open/close 29413 (time 1787710872.89 total 981.12 last 79.59) - open/close 30000 (time 1787710879.62 total 987.84 last 87.24) - open/close 30790 (time 1787710889.62 total 997.85 last 78.93) - open/close 31683 (time 1787710899.63 total 1007.86 last 89.23) - open/close 32523 (time 1787710909.64 total 1017.86 last 83.97) - open/close 33405 (time 1787710919.64 total 1027.87 last 88.13) - open/close 34217 (time 1787710929.65 total 1037.87 last 81.19) - open/close 35029 (time 1787710939.65 total 1047.88 last 81.19) - open/close 35652 (time 1787710949.65 total 1057.88 last 62.30) - open/close 36458 (time 1787710959.65 total 1067.88 last 80.56) - open/close 37300 (time 1787710969.65 total 1077.88 last 84.20) - open/close 37964 (time 1787710979.66 total 1087.89 last 66.34) - open/close 38820 (time 1787710989.67 total 1097.89 last 85.56) - open/close 39617 (time 1787710999.68 total 1107.91 last 79.59) - open/close 40000 (time 1787711004.15 total 1112.38 last 85.64) - open/close 40783 (time 1787711014.15 total 1122.38 last 78.28) - open/close 41556 (time 1787711024.17 total 1132.40 last 77.20) - open/close 42334 (time 1787711034.17 total 1142.40 last 77.76) - open/close 43162 (time 1787711044.18 total 1152.40 last 82.78) - open/close 43985 (time 1787711054.18 total 1162.41 last 82.27) - open/close 44830 (time 1787711064.18 total 1172.41 last 84.47) - open/close 45576 (time 1787711074.19 total 1182.42 last 74.51) - open/close 46391 (time 1787711084.20 total 1192.43 last 81.44) - open/close 47257 (time 1787711094.20 total 1202.43 last 86.60) - open/close 48082 (time 1787711104.21 total 1212.44 last 82.43) - open/close 48849 (time 1787711114.22 total 1222.45 last 76.63) - open/close 49633 (time 1787711124.23 total 1232.46 last 78.32) total: 50000 open/close in 1236.81 seconds: 40.43 ops/second - unlinked 0 (time 1787711132 ; total 0 ; last 0) - unlinked 10000 (time 1787711257 ; total 125 ; last 125) - unlinked 20000 (time 1787711379 ; total 247 ; last 122) - unlinked 30000 (time 1787711506 ; total 374 ; last 127) - unlinked 40000 (time 1787711635 ; total 503 ; last 129) total: 50000 unlinks in 631 seconds: 79.239304 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server unloading modules via unload_modules_local on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing unload_modules_local oleg613-server: modules unloaded. pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-client: ssh exited with exit code 2 pdsh@oleg613-client: oleg613-server: ssh exited with exit code 2