== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:53:06 (1782611586) start mds service on oleg324-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg324-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg324-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1782611660.81 total 10.01 last 21.38) - open/close 487 (time 1782611670.83 total 20.03 last 27.25) - open/close 697 (time 1782611681.35 total 30.55 last 19.96) - open/close 899 (time 1782611692.15 total 41.35 last 18.70) - open/close 1108 (time 1782611702.53 total 51.73 last 20.14) - open/close 1339 (time 1782611713.50 total 62.70 last 21.06) - open/close 1546 (time 1782611723.53 total 72.73 last 20.63) - open/close 1795 (time 1782611733.54 total 82.74 last 24.89) - open/close 2032 (time 1782611744.52 total 93.72 last 21.58) - open/close 2247 (time 1782611755.28 total 104.48 last 19.98) - open/close 2469 (time 1782611765.57 total 114.77 last 21.57) - open/close 2694 (time 1782611775.58 total 124.78 last 22.48) - open/close 2900 (time 1782611785.58 total 134.78 last 20.59) - open/close 3101 (time 1782611796.54 total 145.74 last 18.33) - open/close 3316 (time 1782611806.55 total 155.75 last 21.48) - open/close 3563 (time 1782611816.57 total 165.77 last 24.65) - open/close 3781 (time 1782611826.96 total 176.16 last 20.99) - open/close 4014 (time 1782611837.70 total 186.90 last 21.69) - open/close 4233 (time 1782611848.20 total 197.40 last 20.87) - open/close 4484 (time 1782611859.21 total 208.41 last 22.80) - open/close 4727 (time 1782611870.21 total 219.42 last 22.07) - open/close 4986 (time 1782611881.16 total 230.36 last 23.67) - open/close 5140 (time 1782611891.16 total 240.37 last 15.39) - open/close 5276 (time 1782611901.55 total 250.75 last 13.10) - open/close 5425 (time 1782611912.03 total 261.23 last 14.22) - open/close 5637 (time 1782611922.08 total 271.28 last 21.09) - open/close 5803 (time 1782611932.08 total 281.28 last 16.59) - open/close 5868 (time 1782611942.49 total 291.69 last 6.24) - open/close 6008 (time 1782611952.52 total 301.72 last 13.97) - open/close 6201 (time 1782611962.76 total 311.96 last 18.83) - open/close 6422 (time 1782611973.71 total 322.91 last 20.20) - open/close 6567 (time 1782611986.46 total 335.66 last 11.37) - open/close 6748 (time 1782611996.47 total 345.67 last 18.08) - open/close 6892 (time 1782612007.69 total 356.89 last 12.84) - open/close 7080 (time 1782612017.70 total 366.90 last 18.77) - open/close 7327 (time 1782612027.72 total 376.93 last 24.65) - open/close 7576 (time 1782612037.73 total 386.93 last 24.89) - open/close 7810 (time 1782612048.64 total 397.84 last 21.45) - open/close 8063 (time 1782612059.65 total 408.85 last 22.97) - open/close 8229 (time 1782612070.24 total 419.44 last 15.68) - open/close 8459 (time 1782612081.09 total 430.29 last 21.20) - open/close 8694 (time 1782612091.84 total 441.04 last 21.86) - open/close 8897 (time 1782612102.19 total 451.39 last 19.61) - open/close 9129 (time 1782612113.12 total 462.32 last 21.23) - open/close 9411 (time 1782612123.87 total 473.07 last 26.25) - open/close 9531 (time 1782612134.02 total 483.22 last 11.82) - open/close 9734 (time 1782612145.04 total 494.24 last 18.42) - open/close 9998 (time 1782612155.55 total 504.75 last 25.11) - open/close 10227 (time 1782612167.61 total 516.81 last 18.99) - open/close 10332 (time 1782612177.63 total 526.83 last 10.48) - open/close 10520 (time 1782612187.64 total 536.84 last 18.77) - open/close 10773 (time 1782612197.65 total 546.85 last 25.29) - open/close 10975 (time 1782612208.21 total 557.41 last 19.14) - open/close 11142 (time 1782612218.22 total 567.42 last 16.68) - open/close 11310 (time 1782612228.88 total 578.08 last 15.76) - open/close 11520 (time 1782612238.88 total 588.08 last 20.99) - open/close 11742 (time 1782612249.42 total 598.62 last 21.06) - open/close 11943 (time 1782612264.74 total 613.94 last 13.13) - open/close 12130 (time 1782612275.58 total 624.79 last 17.24) - open/close 12397 (time 1782612285.89 total 635.09 last 25.92) - open/close 12574 (time 1782612295.89 total 645.09 last 17.70) - open/close 12788 (time 1782612306.78 total 655.98 last 19.66) - open/close 13043 (time 1782612316.78 total 665.98 last 25.49) - open/close 13248 (time 1782612326.78 total 675.98 last 20.49) - open/close 13493 (time 1782612336.80 total 686.00 last 24.46) - open/close 13705 (time 1782612346.88 total 696.09 last 21.02) - open/close 13934 (time 1782612357.59 total 706.80 last 21.38) - open/close 14111 (time 1782612367.60 total 716.80 last 17.70) - open/close 14313 (time 1782612377.60 total 726.80 last 20.20) - open/close 14533 (time 1782612388.36 total 737.56 last 20.45) - open/close 14697 (time 1782612399.55 total 748.75 last 14.65) - open/close 14912 (time 1782612409.56 total 758.76 last 21.49) - open/close 15162 (time 1782612419.57 total 768.77 last 24.98) - open/close 15381 (time 1782612430.11 total 779.31 last 20.77) - open/close 15622 (time 1782612441.09 total 790.29 last 21.95) - open/close 15797 (time 1782612452.23 total 801.43 last 15.71) - open/close 15989 (time 1782612462.23 total 811.43 last 19.20) - open/close 16135 (time 1782612472.77 total 821.97 last 13.85) - open/close 16371 (time 1782612482.77 total 831.97 last 23.60) - open/close 16589 (time 1782612493.57 total 842.77 last 20.19) - open/close 16798 (time 1782612504.07 total 853.27 last 19.90) - open/close 16984 (time 1782612514.09 total 863.29 last 18.56) - open/close 17083 (time 1782612524.10 total 873.30 last 9.90) - open/close 17200 (time 1782612534.16 total 883.37 last 11.62) - open/close 17380 (time 1782612544.17 total 893.37 last 18.00) - open/close 17550 (time 1782612555.03 total 904.23 last 15.64) - open/close 17674 (time 1782612566.01 total 915.21 last 11.29) - open/close 17844 (time 1782612576.02 total 925.22 last 17.00) - open/close 17959 (time 1782612587.02 total 936.22 last 10.45) - open/close 18304 (time 1782612597.03 total 946.23 last 34.44) - open/close 18943 (time 1782612607.05 total 956.25 last 63.80) - open/close 19577 (time 1782612617.07 total 966.27 last 63.27) - open/close 20000 (time 1782612623.50 total 972.70 last 65.77) - open/close 20614 (time 1782612633.50 total 982.70 last 61.39) - open/close 21183 (time 1782612643.51 total 992.71 last 56.85) - open/close 21794 (time 1782612653.51 total 1002.71 last 61.09) - open/close 22443 (time 1782612663.52 total 1012.72 last 64.87) - open/close 23081 (time 1782612673.55 total 1022.75 last 63.63) - open/close 23709 (time 1782612683.57 total 1032.77 last 62.68) - open/close 24366 (time 1782612693.58 total 1042.78 last 65.63) - open/close 24896 (time 1782612703.60 total 1052.80 last 52.87) - open/close 25333 (time 1782612713.62 total 1062.82 last 43.63) - open/close 26000 (time 1782612723.63 total 1072.83 last 66.64) - open/close 26710 (time 1782612733.63 total 1082.83 last 70.98) - open/close 27411 (time 1782612743.64 total 1092.84 last 70.04) - open/close 28099 (time 1782612753.64 total 1102.84 last 68.79) - open/close 28737 (time 1782612763.66 total 1112.86 last 63.67) - open/close 29326 (time 1782612773.67 total 1122.87 last 58.83) - open/close 30000 (time 1782612783.20 total 1132.40 last 70.75) - open/close 30565 (time 1782612793.23 total 1142.43 last 56.30) - open/close 31167 (time 1782612803.24 total 1152.44 last 60.16) - open/close 31766 (time 1782612813.25 total 1162.45 last 59.84) - open/close 32407 (time 1782612823.26 total 1172.46 last 64.06) - open/close 33007 (time 1782612833.26 total 1182.46 last 59.97) - open/close 33567 (time 1782612843.27 total 1192.47 last 55.98) - open/close 34124 (time 1782612853.27 total 1202.47 last 55.68) - open/close 34758 (time 1782612863.28 total 1212.48 last 63.37) - open/close 35327 (time 1782612873.29 total 1222.49 last 56.80) - open/close 35956 (time 1782612883.30 total 1232.50 last 62.84) - open/close 36594 (time 1782612893.31 total 1242.51 last 63.77) - open/close 37205 (time 1782612903.33 total 1252.53 last 60.97) - open/close 37762 (time 1782612913.33 total 1262.53 last 55.68) - open/close 38376 (time 1782612923.34 total 1272.54 last 61.35) - open/close 38973 (time 1782612933.34 total 1282.54 last 59.68) - open/close 39599 (time 1782612943.35 total 1292.55 last 62.55) - open/close 40000 (time 1782612950.48 total 1299.68 last 56.24) - open/close 40639 (time 1782612960.49 total 1309.69 last 63.83) - open/close 41151 (time 1782612970.50 total 1319.71 last 51.13) - open/close 41782 (time 1782612980.51 total 1329.71 last 63.09) - open/close 42475 (time 1782612990.51 total 1339.71 last 69.28) - open/close 43157 (time 1782613000.52 total 1349.72 last 68.16) - open/close 43737 (time 1782613010.52 total 1359.72 last 57.96) - open/close 44380 (time 1782613020.54 total 1369.74 last 64.22) - open/close 44974 (time 1782613030.55 total 1379.75 last 59.30) - open/close 45534 (time 1782613040.58 total 1389.78 last 55.86) - open/close 46092 (time 1782613050.58 total 1399.78 last 55.79) - open/close 46646 (time 1782613060.59 total 1409.79 last 55.33) - open/close 47209 (time 1782613070.60 total 1419.80 last 56.26) - open/close 47810 (time 1782613080.61 total 1429.81 last 60.02) - open/close 48322 (time 1782613090.62 total 1439.82 last 51.16) - open/close 48850 (time 1782613100.63 total 1449.83 last 52.76) - open/close 49421 (time 1782613110.64 total 1459.84 last 57.05) total: 50000 open/close in 1469.64 seconds: 34.02 ops/second - unlinked 0 (time 1782613125 ; total 0 ; last 0) - unlinked 10000 (time 1782613293 ; total 168 ; last 168) - unlinked 20000 (time 1782613457 ; total 332 ; last 164) - unlinked 30000 (time 1782613622 ; total 497 ; last 165) - unlinked 40000 (time 1782613781 ; total 656 ; last 159) total: 50000 unlinks in 815 seconds: 61.349693 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2