== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:45:03 (1780386303) start mds service on oleg328-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg328-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 289 (time 1780386348.97 total 10.58 last 27.31) - open/close 507 (time 1780386359.16 total 20.76 last 21.41) - open/close 822 (time 1780386369.97 total 31.58 last 29.13) - open/close 1056 (time 1780386380.72 total 42.33 last 21.77) - open/close 1348 (time 1780386391.35 total 52.96 last 27.46) - open/close 1611 (time 1780386401.78 total 63.39 last 25.21) - open/close 1859 (time 1780386412.08 total 73.69 last 24.09) - open/close 2051 (time 1780386422.70 total 84.31 last 18.07) - open/close 2415 (time 1780386433.71 total 95.32 last 33.07) - open/close 2502 (time 1780386443.71 total 105.32 last 8.70) - open/close 2673 (time 1780386454.19 total 115.80 last 16.31) - open/close 2810 (time 1780386464.37 total 125.97 last 13.47) - open/close 2931 (time 1780386475.37 total 136.98 last 10.99) - open/close 3292 (time 1780386486.06 total 147.67 last 33.77) - open/close 3568 (time 1780386496.62 total 158.23 last 26.14) - open/close 3786 (time 1780386507.31 total 168.92 last 20.40) - open/close 3992 (time 1780386517.74 total 179.35 last 19.74) - open/close 4240 (time 1780386527.74 total 189.35 last 24.80) - open/close 4464 (time 1780386538.41 total 200.02 last 20.99) - open/close 4770 (time 1780386548.60 total 210.20 last 30.05) - open/close 5018 (time 1780386558.60 total 220.21 last 24.79) - open/close 5228 (time 1780386569.39 total 231.00 last 19.46) - open/close 5484 (time 1780386579.69 total 241.30 last 24.85) - open/close 5688 (time 1780386590.19 total 251.80 last 19.43) - open/close 5879 (time 1780386600.82 total 262.43 last 17.98) - open/close 6056 (time 1780386611.25 total 272.86 last 16.97) - open/close 6287 (time 1780386621.25 total 282.86 last 23.09) - open/close 6513 (time 1780386632.25 total 293.86 last 20.55) - open/close 6772 (time 1780386643.19 total 304.79 last 23.68) - open/close 7010 (time 1780386654.13 total 315.73 last 21.76) - open/close 7205 (time 1780386664.76 total 326.36 last 18.34) - open/close 7436 (time 1780386674.76 total 336.36 last 23.10) - open/close 7684 (time 1780386684.76 total 346.37 last 24.79) - open/close 7997 (time 1780386695.22 total 356.82 last 29.93) - open/close 8274 (time 1780386705.52 total 367.13 last 26.89) - open/close 8469 (time 1780386716.02 total 377.63 last 18.56) - open/close 8631 (time 1780386726.38 total 387.99 last 15.63) - open/close 8893 (time 1780386736.50 total 398.11 last 25.89) - open/close 10000 (time 1780386743.88 total 405.49 last 150.10) - open/close 11842 (time 1780386753.88 total 415.49 last 184.16) - open/close 13966 (time 1780386763.88 total 425.49 last 212.34) - open/close 16172 (time 1780386773.88 total 435.49 last 220.59) - open/close 18229 (time 1780386783.89 total 445.49 last 205.64) - open/close 20000 (time 1780386793.04 total 454.65 last 193.43) - open/close 22379 (time 1780386803.05 total 464.65 last 237.84) - open/close 25000 (time 1780386813.05 total 474.65 last 262.10) - open/close 28797 (time 1780386823.05 total 484.65 last 379.63) - open/close 30000 (time 1780386825.92 total 487.53 last 418.41) - open/close 33202 (time 1780386835.94 total 497.55 last 319.68) - open/close 35734 (time 1780386845.94 total 507.55 last 253.18) - open/close 38179 (time 1780386855.94 total 517.55 last 244.42) - open/close 40000 (time 1780386863.24 total 524.85 last 249.51) - open/close 42342 (time 1780386873.24 total 534.85 last 234.12) - open/close 44836 (time 1780386883.25 total 544.85 last 249.36) - open/close 46907 (time 1780386893.25 total 554.85 last 207.09) - open/close 48849 (time 1780386903.25 total 564.86 last 194.14) total: 50000 open/close in 569.58 seconds: 87.78 ops/second - unlinked 0 (time 1780386909 ; total 0 ; last 0) - unlinked 10000 (time 1780386935 ; total 26 ; last 26) - unlinked 20000 (time 1780386962 ; total 53 ; last 27) - unlinked 30000 (time 1780386989 ; total 80 ; last 27) - unlinked 40000 (time 1780387009 ; total 100 ; last 20) total: 50000 unlinks in 122 seconds: 409.836060 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local oleg328-server: modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2