== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:04:12 (1788905052) start mds service on oleg448-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 571 (time 1788905091.33 total 10.06 last 56.78) - open/close 1138 (time 1788905101.38 total 20.11 last 56.42) - open/close 1766 (time 1788905111.49 total 30.22 last 62.12) - open/close 2397 (time 1788905121.60 total 40.33 last 62.39) - open/close 2981 (time 1788905131.71 total 50.44 last 57.75) - open/close 3517 (time 1788905142.08 total 60.81 last 51.69) - open/close 4098 (time 1788905152.32 total 71.05 last 56.75) - open/close 4700 (time 1788905162.56 total 81.29 last 58.79) - open/close 5152 (time 1788905173.50 total 92.23 last 41.30) - open/close 5808 (time 1788905183.68 total 102.41 last 64.46) - open/close 6417 (time 1788905193.68 total 112.41 last 60.88) - open/close 7096 (time 1788905204.29 total 123.02 last 64.03) - open/close 7657 (time 1788905214.46 total 133.19 last 55.13) - open/close 7793 (time 1788905224.47 total 143.19 last 13.60) - open/close 8297 (time 1788905234.82 total 153.54 last 48.69) - open/close 8769 (time 1788905245.63 total 164.36 last 43.64) - open/close 9389 (time 1788905258.18 total 176.90 last 49.43) - open/close 10000 (time 1788905267.42 total 186.15 last 66.10) - open/close 10460 (time 1788905278.21 total 196.94 last 42.64) - open/close 10812 (time 1788905290.18 total 208.91 last 29.40) - open/close 11195 (time 1788905300.18 total 218.91 last 38.29) - open/close 11707 (time 1788905310.18 total 228.91 last 51.19) - open/close 12282 (time 1788905320.19 total 238.92 last 57.45) - open/close 12931 (time 1788905330.37 total 249.10 last 63.78) - open/close 13608 (time 1788905340.37 total 259.10 last 67.70) - open/close 14098 (time 1788905350.40 total 269.13 last 48.82) - open/close 14724 (time 1788905360.84 total 279.56 last 60.01) - open/close 15310 (time 1788905370.95 total 289.67 last 57.96) - open/close 15936 (time 1788905381.25 total 299.98 last 60.77) - open/close 16537 (time 1788905391.30 total 310.03 last 59.81) - open/close 17247 (time 1788905401.79 total 320.52 last 67.65) - open/close 19820 (time 1788905411.79 total 330.52 last 257.30) - open/close 23275 (time 1788905421.79 total 340.52 last 345.48) - open/close 27018 (time 1788905431.79 total 350.52 last 374.27) - open/close 30000 (time 1788905438.68 total 357.40 last 433.23) - open/close 33922 (time 1788905448.68 total 367.41 last 392.13) - open/close 37776 (time 1788905458.68 total 377.41 last 385.31) - open/close 40000 (time 1788905463.58 total 382.31 last 454.20) - open/close 44256 (time 1788905473.58 total 392.31 last 425.58) - open/close 47864 (time 1788905483.58 total 402.31 last 360.75) total: 50000 open/close in 407.92 seconds: 122.57 ops/second - unlinked 0 (time 1788905490 ; total 0 ; last 0) - unlinked 10000 (time 1788905506 ; total 16 ; last 16) - unlinked 20000 (time 1788905524 ; total 34 ; last 18) - unlinked 30000 (time 1788905542 ; total 52 ; last 18) - unlinked 40000 (time 1788905560 ; total 70 ; last 18) total: 50000 unlinks in 139 seconds: 359.712219 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded. pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2