== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:08:09 (1788354489) start mds service on oleg460-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 732 (time 1788354526.49 total 10.00 last 73.18) - open/close 1418 (time 1788354536.49 total 20.00 last 68.59) - open/close 2167 (time 1788354546.71 total 30.23 last 73.27) - open/close 2972 (time 1788354557.02 total 40.53 last 78.11) - open/close 3795 (time 1788354567.20 total 50.71 last 80.87) - open/close 4531 (time 1788354577.24 total 60.75 last 73.27) - open/close 5231 (time 1788354588.25 total 71.76 last 63.58) - open/close 5968 (time 1788354598.25 total 81.77 last 73.69) - open/close 6555 (time 1788354608.92 total 92.43 last 55.02) - open/close 7311 (time 1788354619.16 total 102.68 last 73.81) - open/close 8067 (time 1788354629.40 total 112.92 last 73.83) - open/close 8821 (time 1788354639.58 total 123.09 last 74.11) - open/close 9588 (time 1788354649.75 total 133.27 last 75.37) - open/close 10000 (time 1788354654.88 total 138.39 last 80.38) - open/close 10703 (time 1788354664.88 total 148.39 last 70.29) - open/close 11360 (time 1788354675.80 total 159.32 last 60.15) - open/close 12070 (time 1788354685.81 total 169.32 last 70.99) - open/close 12731 (time 1788354695.81 total 179.32 last 66.09) - open/close 13443 (time 1788354706.78 total 190.29 last 64.89) - open/close 14166 (time 1788354716.78 total 200.30 last 72.29) - open/close 14834 (time 1788354726.78 total 210.30 last 66.79) - open/close 15454 (time 1788354737.63 total 221.14 last 57.19) - open/close 16139 (time 1788354747.63 total 231.14 last 68.49) - open/close 16749 (time 1788354758.38 total 241.90 last 56.71) - open/close 17441 (time 1788354768.38 total 251.90 last 69.19) - open/close 18095 (time 1788354779.29 total 262.81 last 59.95) - open/close 18862 (time 1788354789.47 total 272.98 last 75.37) - open/close 19608 (time 1788354799.47 total 282.98 last 74.60) - open/close 20000 (time 1788354805.62 total 289.13 last 63.75) - open/close 20734 (time 1788354815.64 total 299.16 last 73.23) - open/close 21442 (time 1788354825.64 total 309.16 last 70.80) - open/close 24465 (time 1788354835.64 total 319.16 last 302.28) - open/close 28801 (time 1788354845.64 total 329.16 last 433.57) - open/close 30000 (time 1788354848.54 total 332.06 last 413.61) - open/close 34297 (time 1788354858.55 total 342.06 last 429.62) - open/close 38547 (time 1788354868.55 total 352.06 last 425.00) - open/close 40000 (time 1788354871.93 total 355.44 last 429.33) - open/close 43775 (time 1788354881.93 total 365.44 last 377.46) - open/close 47766 (time 1788354891.93 total 375.45 last 399.07) total: 50000 open/close in 381.21 seconds: 131.16 ops/second - unlinked 0 (time 1788354899 ; total 0 ; last 0) - unlinked 10000 (time 1788354914 ; total 15 ; last 15) - unlinked 20000 (time 1788354933 ; total 34 ; last 19) - unlinked 30000 (time 1788354950 ; total 51 ; last 17) - unlinked 40000 (time 1788354965 ; total 66 ; last 15) total: 50000 unlinks in 84 seconds: 595.238098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2