== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:44:07 (1787078647) start mds service on oleg215-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 184 (time 1787078712.04 total 10.28 last 17.91) - open/close 349 (time 1787078722.14 total 20.38 last 16.33) - open/close 619 (time 1787078732.91 total 31.14 last 25.08) - open/close 950 (time 1787078743.07 total 41.30 last 32.58) - open/close 1264 (time 1787078753.07 total 51.31 last 31.39) - open/close 1556 (time 1787078763.08 total 61.32 last 29.18) - open/close 1768 (time 1787078773.08 total 71.32 last 21.20) - open/close 1949 (time 1787078783.09 total 81.33 last 18.08) - open/close 2223 (time 1787078793.95 total 92.18 last 25.24) - open/close 2539 (time 1787078803.95 total 102.19 last 31.59) - open/close 2807 (time 1787078813.98 total 112.22 last 26.72) - open/close 2915 (time 1787078827.32 total 125.55 last 8.10) - open/close 3233 (time 1787078837.32 total 135.56 last 31.79) - open/close 3430 (time 1787078847.44 total 145.68 last 19.46) - open/close 3698 (time 1787078859.41 total 157.65 last 22.40) - open/close 3844 (time 1787078870.37 total 168.61 last 13.32) - open/close 3999 (time 1787078880.86 total 179.10 last 14.78) - open/close 4137 (time 1787078891.62 total 189.85 last 12.83) - open/close 4325 (time 1787078902.17 total 200.41 last 17.81) - open/close 4541 (time 1787078912.79 total 211.03 last 20.34) - open/close 4771 (time 1787078923.73 total 221.97 last 21.03) - open/close 4940 (time 1787078933.74 total 231.98 last 16.88) - open/close 5120 (time 1787078943.74 total 241.98 last 18.00) - open/close 5244 (time 1787078954.46 total 252.69 last 11.57) - open/close 5372 (time 1787078965.28 total 263.52 last 11.83) - open/close 5516 (time 1787078976.23 total 274.46 last 13.15) - open/close 5732 (time 1787078986.54 total 284.77 last 20.95) - open/close 5907 (time 1787078996.69 total 294.93 last 17.23) - open/close 6004 (time 1787079007.19 total 305.42 last 9.25) - open/close 6304 (time 1787079017.19 total 315.43 last 29.97) - open/close 6626 (time 1787079028.12 total 326.35 last 29.48) - open/close 6922 (time 1787079039.19 total 337.43 last 26.73) - open/close 7139 (time 1787079049.63 total 347.86 last 20.80) - open/close 7234 (time 1787079060.90 total 359.14 last 8.42) - open/close 7506 (time 1787079071.77 total 370.00 last 25.04) - open/close 7768 (time 1787079084.82 total 383.06 last 20.07) - open/close 7988 (time 1787079094.87 total 393.10 last 21.90) - open/close 8238 (time 1787079105.75 total 403.99 last 22.97) - open/close 8464 (time 1787079116.20 total 414.44 last 21.62) - open/close 8727 (time 1787079127.13 total 425.37 last 24.06) - open/close 8952 (time 1787079137.82 total 436.06 last 21.04) - open/close 9124 (time 1787079148.06 total 446.29 last 16.81) - open/close 9249 (time 1787079158.81 total 457.04 last 11.63) - open/close 9516 (time 1787079169.43 total 467.66 last 25.14) - open/close 9768 (time 1787079179.99 total 478.23 last 23.86) - open/close 10000 (time 1787079186.00 total 484.24 last 38.57) - open/close 10304 (time 1787079196.76 total 495.00 last 28.27) - open/close 10614 (time 1787079206.77 total 505.01 last 30.97) - open/close 10950 (time 1787079217.75 total 515.98 last 30.61) - open/close 11277 (time 1787079228.51 total 526.74 last 30.39) - open/close 12754 (time 1787079238.51 total 536.74 last 147.69) - open/close 14421 (time 1787079248.51 total 546.75 last 166.64) - open/close 16094 (time 1787079258.52 total 556.75 last 167.22) - open/close 17943 (time 1787079268.52 total 566.75 last 184.88) - open/close 19795 (time 1787079278.52 total 576.76 last 185.12) - open/close 21511 (time 1787079288.52 total 586.76 last 171.58) - open/close 23040 (time 1787079298.52 total 596.76 last 152.89) - open/close 24910 (time 1787079308.53 total 606.76 last 186.97) - open/close 26927 (time 1787079318.54 total 616.77 last 201.45) - open/close 27787 (time 1787079328.54 total 626.78 last 85.96) - open/close 28675 (time 1787079338.61 total 636.85 last 88.20) - open/close 29523 (time 1787079348.62 total 646.85 last 84.74) - open/close 30000 (time 1787079355.93 total 654.16 last 65.26) - open/close 30887 (time 1787079365.93 total 664.17 last 88.67) - open/close 31421 (time 1787079375.94 total 674.18 last 53.32) - open/close 31867 (time 1787079385.94 total 684.18 last 44.60) - open/close 32493 (time 1787079395.99 total 694.22 last 62.33) - open/close 33203 (time 1787079406.00 total 704.23 last 70.94) - open/close 34025 (time 1787079416.00 total 714.24 last 82.15) - open/close 35044 (time 1787079426.00 total 724.24 last 101.89) - open/close 35954 (time 1787079436.01 total 734.24 last 90.96) - open/close 36928 (time 1787079446.01 total 744.25 last 97.37) - open/close 37873 (time 1787079456.01 total 754.25 last 94.46) - open/close 38873 (time 1787079466.02 total 764.26 last 99.93) - open/close 39948 (time 1787079476.03 total 774.27 last 107.42) - open/close 40812 (time 1787079486.04 total 784.28 last 86.30) - open/close 41784 (time 1787079496.04 total 794.28 last 97.16) - open/close 42737 (time 1787079506.05 total 804.28 last 95.28) - open/close 43830 (time 1787079516.05 total 814.28 last 109.28) - open/close 44859 (time 1787079526.05 total 824.29 last 102.89) - open/close 45906 (time 1787079536.05 total 834.29 last 104.68) - open/close 46993 (time 1787079546.06 total 844.30 last 108.61) - open/close 48122 (time 1787079556.08 total 854.31 last 112.70) - open/close 49127 (time 1787079566.08 total 864.31 last 100.50) total: 50000 open/close in 872.91 seconds: 57.28 ops/second - unlinked 0 (time 1787079577 ; total 0 ; last 0) - unlinked 10000 (time 1787079670 ; total 93 ; last 93) - unlinked 20000 (time 1787079743 ; total 166 ; last 73) - unlinked 30000 (time 1787079815 ; total 238 ; last 72) - unlinked 40000 (time 1787079888 ; total 311 ; last 73) total: 50000 unlinks in 380 seconds: 131.578949 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2