== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:06:13 (1788905173) start mds service on oleg412-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 620 (time 1788905211.22 total 10.42 last 59.53) - open/close 989 (time 1788905221.22 total 20.42 last 36.90) - open/close 1422 (time 1788905231.22 total 30.42 last 43.29) - open/close 1689 (time 1788905241.42 total 40.62 last 26.17) - open/close 2326 (time 1788905251.79 total 50.99 last 61.45) - open/close 2825 (time 1788905261.79 total 60.99 last 49.89) - open/close 3447 (time 1788905271.95 total 71.15 last 61.23) - open/close 3917 (time 1788905281.95 total 81.15 last 46.99) - open/close 4085 (time 1788905291.98 total 91.18 last 16.75) - open/close 4322 (time 1788905302.22 total 101.42 last 23.15) - open/close 4902 (time 1788905312.33 total 111.53 last 57.36) - open/close 5591 (time 1788905322.57 total 121.77 last 67.28) - open/close 6323 (time 1788905333.01 total 132.20 last 70.16) - open/close 6904 (time 1788905344.91 total 144.11 last 48.81) - open/close 7542 (time 1788905355.47 total 154.67 last 60.40) - open/close 8100 (time 1788905365.65 total 164.85 last 54.84) - open/close 8755 (time 1788905376.14 total 175.34 last 62.41) - open/close 9421 (time 1788905386.38 total 185.58 last 65.06) - open/close 10000 (time 1788905394.32 total 193.52 last 72.90) - open/close 10662 (time 1788905404.62 total 203.82 last 64.27) - open/close 11381 (time 1788905415.05 total 214.25 last 68.92) - open/close 12011 (time 1788905425.29 total 224.49 last 61.53) - open/close 12718 (time 1788905435.47 total 234.67 last 69.50) - open/close 13336 (time 1788905445.47 total 244.67 last 61.79) - open/close 13946 (time 1788905455.47 total 254.67 last 60.99) - open/close 14485 (time 1788905466.25 total 265.45 last 50.00) - open/close 15068 (time 1788905476.26 total 275.45 last 58.27) - open/close 15662 (time 1788905486.35 total 285.55 last 58.84) - open/close 16292 (time 1788905496.46 total 295.66 last 62.31) - open/close 16889 (time 1788905506.46 total 305.66 last 59.69) - open/close 17502 (time 1788905516.49 total 315.69 last 61.14) - open/close 18444 (time 1788905526.49 total 325.69 last 94.19) - open/close 20000 (time 1788905531.01 total 330.21 last 344.37) - open/close 23368 (time 1788905541.01 total 340.21 last 336.64) - open/close 26812 (time 1788905551.01 total 350.21 last 344.38) - open/close 30000 (time 1788905560.52 total 359.72 last 335.53) - open/close 31958 (time 1788905570.53 total 369.73 last 195.54) - open/close 32734 (time 1788905580.53 total 379.73 last 77.56) - open/close 33357 (time 1788905590.54 total 389.74 last 62.24) - open/close 33895 (time 1788905600.56 total 399.76 last 53.73) - open/close 34438 (time 1788905610.56 total 409.76 last 54.28) - open/close 34912 (time 1788905620.57 total 419.77 last 47.36) - open/close 35482 (time 1788905630.57 total 429.77 last 56.98) - open/close 36364 (time 1788905640.58 total 439.78 last 88.16) - open/close 37345 (time 1788905650.58 total 449.78 last 98.05) - open/close 38421 (time 1788905660.59 total 459.78 last 107.57) - open/close 39511 (time 1788905670.59 total 469.79 last 108.98) - open/close 40000 (time 1788905675.70 total 474.90 last 95.61) - open/close 41021 (time 1788905685.71 total 484.91 last 102.00) - open/close 42001 (time 1788905695.71 total 494.91 last 97.99) - open/close 43074 (time 1788905705.72 total 504.92 last 107.23) - open/close 44173 (time 1788905715.72 total 514.92 last 109.86) - open/close 45269 (time 1788905725.73 total 524.93 last 109.52) - open/close 46279 (time 1788905735.74 total 534.94 last 100.92) - open/close 47175 (time 1788905745.75 total 544.94 last 89.54) - open/close 48089 (time 1788905755.75 total 554.95 last 91.37) - open/close 49078 (time 1788905765.75 total 564.95 last 98.85) total: 50000 open/close in 574.71 seconds: 87.00 ops/second - unlinked 0 (time 1788905778 ; total 0 ; last 0) - unlinked 10000 (time 1788905844 ; total 66 ; last 66) - unlinked 20000 (time 1788905959 ; total 181 ; last 115) - unlinked 30000 (time 1788906066 ; total 288 ; last 107) - unlinked 40000 (time 1788906176 ; total 398 ; last 110) total: 50000 unlinks in 503 seconds: 99.403580 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: modules unloaded. pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2