== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:31:55 (1789353115) start mds service on oleg349-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions oleg349-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg349-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg349-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg349-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 385 (time 1789353182.86 total 10.01 last 38.47) - open/close 730 (time 1789353193.62 total 20.77 last 32.07) - open/close 1015 (time 1789353204.38 total 31.52 last 26.50) - open/close 1327 (time 1789353214.88 total 42.02 last 29.71) - open/close 1613 (time 1789353225.24 total 52.38 last 27.61) - open/close 1807 (time 1789353235.80 total 62.94 last 18.37) - open/close 2073 (time 1789353245.92 total 73.06 last 26.29) - open/close 2351 (time 1789353255.92 total 83.06 last 27.80) - open/close 2611 (time 1789353266.91 total 94.05 last 23.66) - open/close 2936 (time 1789353277.85 total 104.99 last 29.71) - open/close 3302 (time 1789353288.86 total 116.00 last 33.24) - open/close 3621 (time 1789353298.86 total 126.00 last 31.90) - open/close 3967 (time 1789353309.02 total 136.16 last 34.06) - open/close 4302 (time 1789353319.07 total 146.22 last 33.32) - open/close 4566 (time 1789353329.89 total 157.03 last 24.42) - open/close 4854 (time 1789353339.89 total 167.03 last 28.79) - open/close 5158 (time 1789353350.68 total 177.82 last 28.17) - open/close 5434 (time 1789353361.05 total 188.19 last 26.62) - open/close 5684 (time 1789353371.99 total 199.14 last 22.84) - open/close 5920 (time 1789353382.75 total 209.89 last 21.94) - open/close 6120 (time 1789353393.37 total 220.52 last 18.82) - open/close 6324 (time 1789353403.93 total 231.07 last 19.33) - open/close 6602 (time 1789353414.37 total 241.51 last 26.63) - open/close 6942 (time 1789353425.05 total 252.19 last 31.84) - open/close 7201 (time 1789353435.05 total 262.19 last 25.90) - open/close 7493 (time 1789353445.05 total 272.19 last 29.19) - open/close 7817 (time 1789353455.96 total 283.10 last 29.71) - open/close 8167 (time 1789353466.77 total 293.92 last 32.36) - open/close 8360 (time 1789353477.27 total 304.41 last 18.39) - open/close 8559 (time 1789353487.90 total 315.04 last 18.73) - open/close 8695 (time 1789353498.08 total 325.22 last 13.36) - open/close 8871 (time 1789353508.38 total 335.52 last 17.09) - open/close 9106 (time 1789353519.07 total 346.21 last 21.98) - open/close 9290 (time 1789353529.50 total 356.65 last 17.63) - open/close 9520 (time 1789353540.24 total 367.39 last 21.42) - open/close 9745 (time 1789353551.13 total 378.27 last 20.67) - open/close 9962 (time 1789353561.88 total 389.02 last 20.18) - open/close 10165 (time 1789353572.44 total 399.59 last 19.22) - open/close 10454 (time 1789353582.62 total 409.76 last 28.40) - open/close 11418 (time 1789353592.62 total 419.76 last 96.39) - open/close 13419 (time 1789353602.62 total 429.76 last 200.05) - open/close 15654 (time 1789353612.62 total 439.77 last 223.47) - open/close 18046 (time 1789353622.62 total 449.77 last 239.14) - open/close 20000 (time 1789353632.01 total 459.16 last 208.13) - open/close 22013 (time 1789353642.01 total 469.16 last 201.28) - open/close 24544 (time 1789353652.02 total 479.16 last 252.97) - open/close 26418 (time 1789353662.02 total 489.16 last 187.38) - open/close 28630 (time 1789353672.02 total 499.17 last 221.14) - open/close 30000 (time 1789353677.21 total 504.35 last 264.26) - open/close 32043 (time 1789353687.21 total 514.36 last 204.22) - open/close 34285 (time 1789353697.21 total 524.36 last 224.19) - open/close 35979 (time 1789353707.22 total 534.37 last 169.23) - open/close 37676 (time 1789353717.23 total 544.37 last 169.57) - open/close 39374 (time 1789353727.23 total 554.37 last 169.78) - open/close 40000 (time 1789353731.03 total 558.18 last 164.55) - open/close 41782 (time 1789353741.04 total 568.19 last 178.06) - open/close 43477 (time 1789353751.04 total 578.19 last 169.48) - open/close 45593 (time 1789353761.05 total 588.19 last 211.55) - open/close 47804 (time 1789353771.05 total 598.20 last 221.00) - open/close 49920 (time 1789353781.05 total 608.20 last 211.55) total: 50000 open/close in 608.55 seconds: 82.16 ops/second - unlinked 0 (time 1789353783 ; total 0 ; last 0) - unlinked 10000 (time 1789353819 ; total 36 ; last 36) - unlinked 20000 (time 1789353850 ; total 67 ; last 31) - unlinked 30000 (time 1789353877 ; total 94 ; last 27) - unlinked 40000 (time 1789353900 ; total 117 ; last 23) total: 50000 unlinks in 136 seconds: 367.647064 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2