== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:09:00 (1788905340) start mds service on oleg251-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg251-server: detected 4 online CPUs by sysfs oleg251-server: MODOPTS_LIBCFS= oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg251-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg251-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 747 (time 1788905377.77 total 10.19 last 73.29) - open/close 1519 (time 1788905387.94 total 20.36 last 75.91) - open/close 2234 (time 1788905397.98 total 30.41 last 71.16) - open/close 2957 (time 1788905408.10 total 40.52 last 71.48) - open/close 3660 (time 1788905418.10 total 50.53 last 70.28) - open/close 4417 (time 1788905428.26 total 60.68 last 74.54) - open/close 5114 (time 1788905438.30 total 70.73 last 69.37) - open/close 5894 (time 1788905448.30 total 80.73 last 78.00) - open/close 6451 (time 1788905458.91 total 91.34 last 52.50) - open/close 7219 (time 1788905469.09 total 101.51 last 75.48) - open/close 8000 (time 1788905479.26 total 111.69 last 76.75) - open/close 8799 (time 1788905489.50 total 121.93 last 78.03) - open/close 9507 (time 1788905499.62 total 132.04 last 70.00) - open/close 10000 (time 1788905505.95 total 138.37 last 77.91) - open/close 10680 (time 1788905516.00 total 148.43 last 67.62) - open/close 11433 (time 1788905526.01 total 158.43 last 75.28) - open/close 12098 (time 1788905536.87 total 169.29 last 61.22) - open/close 12804 (time 1788905546.87 total 179.29 last 70.60) - open/close 13514 (time 1788905556.87 total 189.29 last 71.00) - open/close 14213 (time 1788905566.87 total 199.30 last 69.89) - open/close 14584 (time 1788905576.87 total 209.30 last 37.08) - open/close 14882 (time 1788905586.88 total 219.31 last 29.78) - open/close 15111 (time 1788905597.61 total 230.04 last 21.35) - open/close 15381 (time 1788905608.69 total 241.12 last 24.36) - open/close 15642 (time 1788905619.50 total 251.92 last 24.16) - open/close 15824 (time 1788905629.61 total 262.04 last 18.00) - open/close 16084 (time 1788905640.30 total 272.73 last 24.31) - open/close 16293 (time 1788905650.92 total 283.34 last 19.69) - open/close 16482 (time 1788905661.55 total 293.97 last 17.78) - open/close 16719 (time 1788905672.11 total 304.53 last 22.44) - open/close 16991 (time 1788905682.47 total 314.90 last 26.24) - open/close 17480 (time 1788905695.52 total 327.95 last 37.47) - open/close 17733 (time 1788905706.40 total 338.83 last 23.25) - open/close 17957 (time 1788905717.22 total 349.64 last 20.71) - open/close 18693 (time 1788905727.22 total 359.64 last 73.59) - open/close 20000 (time 1788905731.83 total 364.25 last 283.49) - open/close 22938 (time 1788905741.83 total 374.26 last 293.70) - open/close 25126 (time 1788905751.84 total 384.26 last 218.68) - open/close 27108 (time 1788905761.84 total 394.27 last 198.14) - open/close 29214 (time 1788905771.84 total 404.27 last 210.54) - open/close 30000 (time 1788905775.86 total 408.28 last 195.87) - open/close 32043 (time 1788905785.86 total 418.29 last 204.22) - open/close 33791 (time 1788905795.86 total 428.29 last 174.76) - open/close 35845 (time 1788905805.86 total 438.29 last 205.39) - open/close 38659 (time 1788905815.86 total 448.29 last 281.39) - open/close 40000 (time 1788905821.16 total 453.58 last 253.26) - open/close 42730 (time 1788905831.16 total 463.59 last 272.95) - open/close 44932 (time 1788905841.16 total 473.59 last 220.11) - open/close 46968 (time 1788905851.16 total 483.59 last 203.60) - open/close 48641 (time 1788905861.17 total 493.59 last 167.23) - open/close 49797 (time 1788905871.17 total 503.60 last 115.55) total: 50000 open/close in 506.41 seconds: 98.73 ops/second - unlinked 0 (time 1788905878 ; total 0 ; last 0) - unlinked 10000 (time 1788905964 ; total 86 ; last 86) - unlinked 20000 (time 1788906021 ; total 143 ; last 57) - unlinked 30000 (time 1788906082 ; total 204 ; last 61) - unlinked 40000 (time 1788906143 ; total 265 ; last 61) total: 50000 unlinks in 325 seconds: 153.846161 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server unloading modules via unload_modules_local on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local oleg251-server: modules unloaded. pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2