== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:49:41 (1782218981) start mds service on oleg626-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing load_modules_local oleg626-server: Loading modules from /home/green/git/lustre-release/lustre oleg626-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg626-server: detected 4 online CPUs by sysfs oleg626-server: MODOPTS_LIBCFS= oleg626-server: Force libcfs to create 2 CPU partitions oleg626-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg626-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg626-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg626-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg626-server: oleg626-server.virtnet: executing set_default_debug -1 all pdsh@oleg626-client: oleg626-server: ssh exited with exit code 1 Started lustre-OST0000 oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg626-client.virtnet: -o user_xattr,flock 192.168.206.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 525 (time 1782219018.43 total 10.53 last 49.84) - open/close 1258 (time 1782219028.43 total 20.53 last 73.30) - open/close 1773 (time 1782219039.23 total 31.34 last 47.66) - open/close 2211 (time 1782219049.67 total 41.77 last 41.99) - open/close 2938 (time 1782219060.61 total 52.71 last 66.45) - open/close 3410 (time 1782219071.23 total 63.33 last 44.42) - open/close 3984 (time 1782219082.11 total 74.22 last 52.75) - open/close 4739 (time 1782219092.11 total 84.22 last 75.50) - open/close 5463 (time 1782219102.11 total 94.22 last 72.39) - open/close 6123 (time 1782219112.83 total 104.94 last 61.58) - open/close 6832 (time 1782219123.78 total 115.88 last 64.79) - open/close 7444 (time 1782219134.40 total 126.51 last 57.58) - open/close 7982 (time 1782219145.02 total 137.12 last 50.67) - open/close 8632 (time 1782219155.14 total 147.24 last 64.25) - open/close 8966 (time 1782219165.25 total 157.35 last 33.04) - open/close 9315 (time 1782219175.43 total 167.53 last 34.28) - open/close 9684 (time 1782219185.73 total 177.83 last 35.83) - open/close 10000 (time 1782219195.58 total 187.68 last 32.08) - open/close 10565 (time 1782219206.53 total 198.63 last 51.61) - open/close 11198 (time 1782219217.28 total 209.38 last 58.86) - open/close 11733 (time 1782219227.71 total 219.81 last 51.30) - open/close 12425 (time 1782219238.72 total 230.82 last 62.85) - open/close 13100 (time 1782219249.73 total 241.83 last 61.32) - open/close 13455 (time 1782219259.84 total 251.94 last 35.11) - open/close 14051 (time 1782219270.59 total 262.70 last 55.42) - open/close 14718 (time 1782219281.54 total 273.64 last 60.94) - open/close 15330 (time 1782219292.22 total 284.33 last 57.26) - open/close 15829 (time 1782219302.66 total 294.76 last 47.84) - open/close 16440 (time 1782219312.77 total 304.87 last 60.44) - open/close 17009 (time 1782219322.77 total 314.87 last 56.90) - open/close 17587 (time 1782219333.50 total 325.61 last 53.84) - open/close 20000 (time 1782219341.09 total 333.20 last 317.84) - open/close 24545 (time 1782219351.09 total 343.20 last 454.48) - open/close 29124 (time 1782219361.10 total 353.20 last 457.87) - open/close 30000 (time 1782219363.14 total 355.24 last 428.97) - open/close 34349 (time 1782219373.14 total 365.24 last 434.80) - open/close 38942 (time 1782219383.27 total 375.37 last 453.60) - open/close 40000 (time 1782219385.67 total 377.77 last 440.31) - open/close 44355 (time 1782219395.67 total 387.77 last 435.42) - open/close 48890 (time 1782219405.67 total 397.78 last 453.38) total: 50000 open/close in 400.50 seconds: 124.84 ops/second - unlinked 0 (time 1782219409 ; total 0 ; last 0) - unlinked 10000 (time 1782219425 ; total 16 ; last 16) - unlinked 20000 (time 1782219441 ; total 32 ; last 16) - unlinked 30000 (time 1782219456 ; total 47 ; last 15) - unlinked 40000 (time 1782219470 ; total 61 ; last 14) total: 50000 unlinks in 75 seconds: 666.666687 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg626-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg626-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg626-server stop mds service on oleg626-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg626-server unloading modules via unload_modules_local on: 'oleg626-server' oleg626-server: oleg626-server.virtnet: executing unload_modules_local oleg626-server: modules unloaded. pdsh@oleg626-client: oleg626-client: ssh exited with exit code 2 pdsh@oleg626-client: oleg626-server: ssh exited with exit code 2 pdsh@oleg626-client: oleg626-client: ssh exited with exit code 2 pdsh@oleg626-client: oleg626-server: ssh exited with exit code 2