== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:38:07 (1782362287) start mds service on oleg117-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg117-server: detected 4 online CPUs by sysfs oleg117-server: MODOPTS_LIBCFS= oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg117-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 636 (time 1782362327.71 total 11.01 last 57.76) - open/close 1140 (time 1782362337.71 total 21.01 last 50.40) - open/close 1514 (time 1782362348.00 total 31.30 last 36.35) - open/close 2280 (time 1782362358.05 total 41.35 last 76.20) - open/close 2942 (time 1782362369.05 total 52.36 last 60.17) - open/close 3467 (time 1782362379.68 total 62.98 last 49.42) - open/close 4065 (time 1782362390.62 total 73.92 last 54.63) - open/close 4809 (time 1782362400.74 total 84.04 last 73.58) - open/close 5406 (time 1782362412.45 total 95.75 last 50.97) - open/close 5981 (time 1782362423.20 total 106.50 last 53.49) - open/close 6683 (time 1782362433.20 total 116.50 last 70.19) - open/close 7430 (time 1782362443.20 total 126.50 last 74.69) - open/close 8125 (time 1782362454.05 total 137.35 last 64.09) - open/close 8590 (time 1782362464.48 total 147.78 last 44.57) - open/close 9252 (time 1782362475.36 total 158.66 last 60.85) - open/close 9984 (time 1782362486.30 total 169.61 last 66.88) - open/close 10533 (time 1782362496.86 total 180.16 last 52.00) - open/close 11242 (time 1782362507.81 total 191.11 last 64.78) - open/close 11905 (time 1782362518.63 total 201.93 last 61.27) - open/close 12463 (time 1782362529.38 total 212.68 last 51.91) - open/close 13170 (time 1782362539.38 total 222.68 last 70.69) - open/close 13890 (time 1782362549.38 total 232.68 last 71.99) - open/close 14526 (time 1782362560.16 total 243.46 last 58.99) - open/close 15042 (time 1782362570.98 total 254.28 last 47.69) - open/close 15920 (time 1782362581.41 total 264.71 last 84.19) - open/close 16468 (time 1782362591.41 total 274.71 last 54.78) - open/close 16916 (time 1782362602.08 total 285.38 last 42.00) - open/close 17651 (time 1782362614.37 total 297.67 last 59.78) - open/close 18224 (time 1782362624.48 total 307.78 last 56.71) - open/close 18898 (time 1782362635.49 total 318.79 last 61.21) - open/close 20000 (time 1782362641.22 total 324.52 last 192.42) - open/close 24417 (time 1782362651.22 total 334.52 last 441.67) - open/close 28864 (time 1782362661.22 total 344.52 last 444.62) - open/close 30000 (time 1782362663.73 total 347.04 last 451.55) - open/close 34457 (time 1782362673.74 total 357.04 last 445.65) - open/close 38955 (time 1782362683.74 total 367.04 last 449.70) - open/close 40000 (time 1782362685.90 total 369.20 last 483.93) - open/close 44566 (time 1782362695.97 total 379.27 last 453.33) - open/close 49071 (time 1782362705.97 total 389.27 last 450.47) total: 50000 open/close in 391.48 seconds: 127.72 ops/second - unlinked 0 (time 1782362709 ; total 0 ; last 0) - unlinked 10000 (time 1782362724 ; total 15 ; last 15) - unlinked 20000 (time 1782362741 ; total 32 ; last 17) - unlinked 30000 (time 1782362756 ; total 47 ; last 15) - unlinked 40000 (time 1782362772 ; total 63 ; last 16) total: 50000 unlinks in 78 seconds: 641.025635 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server unloading modules via unload_modules_local on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local oleg117-server: modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2