== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:25:49 (1784125549) start mds service on oleg460-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 262 (time 1784125591.70 total 10.51 last 24.94) - open/close 875 (time 1784125602.13 total 20.94 last 58.76) - open/close 1100 (time 1784125612.44 total 31.24 last 21.83) - open/close 1687 (time 1784125622.44 total 41.24 last 58.70) - open/close 2110 (time 1784125633.43 total 52.23 last 38.49) - open/close 2227 (time 1784125644.44 total 63.24 last 10.63) - open/close 2349 (time 1784125654.44 total 73.25 last 12.20) - open/close 2441 (time 1784125665.37 total 84.17 last 8.42) - open/close 2625 (time 1784125675.61 total 94.42 last 17.96) - open/close 2734 (time 1784125686.43 total 105.23 last 10.08) - open/close 2970 (time 1784125696.60 total 115.40 last 23.20) - open/close 3339 (time 1784125707.29 total 126.09 last 34.53) - open/close 3779 (time 1784125718.10 total 136.91 last 40.68) - open/close 4449 (time 1784125728.34 total 147.14 last 65.44) - open/close 5054 (time 1784125738.45 total 157.26 last 59.83) - open/close 5674 (time 1784125748.95 total 167.76 last 59.05) - open/close 6160 (time 1784125758.95 total 177.76 last 48.60) - open/close 6612 (time 1784125768.96 total 187.76 last 45.18) - open/close 7155 (time 1784125778.96 total 197.76 last 54.30) - open/close 7662 (time 1784125788.96 total 207.76 last 50.69) - open/close 8253 (time 1784125798.96 total 217.76 last 59.08) - open/close 8821 (time 1784125809.11 total 227.91 last 55.96) - open/close 9455 (time 1784125819.35 total 238.15 last 61.93) - open/close 10000 (time 1784125828.36 total 247.17 last 60.47) - open/close 10511 (time 1784125838.36 total 257.17 last 51.10) - open/close 11157 (time 1784125848.86 total 267.66 last 61.57) - open/close 11824 (time 1784125858.90 total 277.71 last 66.38) - open/close 12436 (time 1784125869.14 total 287.94 last 59.78) - open/close 13041 (time 1784125879.14 total 297.95 last 60.49) - open/close 13642 (time 1784125889.37 total 308.18 last 58.74) - open/close 14214 (time 1784125900.31 total 319.11 last 52.30) - open/close 14799 (time 1784125910.56 total 329.36 last 57.09) - open/close 17355 (time 1784125920.56 total 339.36 last 255.55) - open/close 20000 (time 1784125926.80 total 345.60 last 424.09) - open/close 23706 (time 1784125936.80 total 355.60 last 370.56) - open/close 27836 (time 1784125946.80 total 365.60 last 412.98) - open/close 30000 (time 1784125951.91 total 370.71 last 423.45) - open/close 34593 (time 1784125961.91 total 380.71 last 459.28) - open/close 38947 (time 1784125971.91 total 390.71 last 435.32) - open/close 40000 (time 1784125974.59 total 393.39 last 393.68) - open/close 44388 (time 1784125984.59 total 403.39 last 438.75) - open/close 48351 (time 1784125994.59 total 413.39 last 396.21) total: 50000 open/close in 417.70 seconds: 119.70 ops/second - unlinked 0 (time 1784126000 ; total 0 ; last 0) - unlinked 10000 (time 1784126015 ; total 15 ; last 15) - unlinked 20000 (time 1784126034 ; total 34 ; last 19) - unlinked 30000 (time 1784126052 ; total 52 ; last 18) - unlinked 40000 (time 1784126071 ; total 71 ; last 19) total: 50000 unlinks in 89 seconds: 561.797729 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2