== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:02:44 (1787497364) start mds service on oleg209-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 150 (time 1787497450.78 total 13.30 last 11.28) - open/close 373 (time 1787497460.78 total 23.30 last 22.28) - open/close 550 (time 1787497471.02 total 33.54 last 17.29) - open/close 743 (time 1787497481.03 total 43.55 last 19.29) - open/close 913 (time 1787497491.06 total 53.57 last 16.95) - open/close 1069 (time 1787497501.06 total 63.58 last 15.59) - open/close 1297 (time 1787497513.69 total 76.21 last 18.05) - open/close 1476 (time 1787497524.56 total 87.07 last 16.47) - open/close 1586 (time 1787497537.50 total 100.02 last 8.50) - open/close 1748 (time 1787497547.53 total 110.05 last 16.15) - open/close 1948 (time 1787497557.54 total 120.06 last 19.98) - open/close 2163 (time 1787497568.32 total 130.84 last 19.95) - open/close 2288 (time 1787497578.53 total 141.04 last 12.25) - open/close 2379 (time 1787497588.57 total 151.09 last 9.06) - open/close 2493 (time 1787497598.58 total 161.10 last 11.39) - open/close 2608 (time 1787497608.60 total 171.12 last 11.48) - open/close 2721 (time 1787497618.62 total 181.14 last 11.28) - open/close 2934 (time 1787497628.78 total 191.30 last 20.96) - open/close 3084 (time 1787497639.51 total 202.03 last 13.98) - open/close 3223 (time 1787497650.59 total 213.11 last 12.55) - open/close 3256 (time 1787497660.76 total 223.28 last 3.24) - open/close 3338 (time 1787497671.62 total 234.13 last 7.55) - open/close 3459 (time 1787497681.62 total 244.14 last 12.09) - open/close 3550 (time 1787497692.07 total 254.58 last 8.71) - open/close 3663 (time 1787497702.10 total 264.62 last 11.26) - open/close 3838 (time 1787497712.11 total 274.63 last 17.48) - open/close 3952 (time 1787497722.30 total 284.82 last 11.19) - open/close 4133 (time 1787497732.32 total 294.83 last 18.07) - open/close 4174 (time 1787497742.80 total 305.31 last 3.91) - open/close 4318 (time 1787497753.52 total 316.04 last 13.43) - open/close 4457 (time 1787497764.57 total 327.09 last 12.58) - open/close 4659 (time 1787497775.27 total 337.79 last 18.88) - open/close 4768 (time 1787497786.07 total 348.59 last 10.09) - open/close 4817 (time 1787497796.96 total 359.48 last 4.50) - open/close 4959 (time 1787497806.98 total 369.50 last 14.17) - open/close 5006 (time 1787497817.50 total 380.02 last 4.47) - open/close 5147 (time 1787497828.00 total 390.52 last 13.43) - open/close 5252 (time 1787497841.05 total 403.57 last 8.05) - open/close 5311 (time 1787497853.03 total 415.55 last 4.92) - open/close 5495 (time 1787497863.59 total 426.11 last 17.42) - open/close 5708 (time 1787497874.59 total 437.11 last 19.36) - open/close 5863 (time 1787497885.27 total 447.79 last 14.52) - open/close 6003 (time 1787497895.28 total 457.80 last 13.99) - open/close 6150 (time 1787497906.99 total 469.51 last 12.55) - open/close 6297 (time 1787497918.69 total 481.21 last 12.56) - open/close 6444 (time 1787497928.70 total 491.22 last 14.69) - open/close 6640 (time 1787497939.26 total 501.78 last 18.57) - open/close 6830 (time 1787497950.08 total 512.60 last 17.56) - open/close 7071 (time 1787497960.10 total 522.62 last 24.05) - open/close 7150 (time 1787497971.18 total 533.70 last 7.13) - open/close 7334 (time 1787497982.01 total 544.53 last 17.00) - open/close 7434 (time 1787497992.68 total 555.20 last 9.37) - open/close 7516 (time 1787498002.69 total 565.21 last 8.20) - open/close 7683 (time 1787498012.70 total 575.22 last 16.68) - open/close 7872 (time 1787498023.21 total 585.73 last 17.98) - open/close 8000 (time 1787498033.87 total 596.39 last 12.01) - open/close 8143 (time 1787498045.42 total 607.93 last 12.39) - open/close 8224 (time 1787498055.89 total 618.40 last 7.74) - open/close 8458 (time 1787498068.00 total 630.51 last 19.33) - open/close 8585 (time 1787498078.01 total 640.53 last 12.68) - open/close 8770 (time 1787498088.97 total 651.49 last 16.88) - open/close 8986 (time 1787498099.79 total 662.31 last 19.96) - open/close 9101 (time 1787498109.82 total 672.34 last 11.47) - open/close 9223 (time 1787498120.02 total 682.53 last 11.97) - open/close 9350 (time 1787498131.99 total 694.50 last 10.61) - open/close 9555 (time 1787498142.88 total 705.40 last 18.82) - open/close 9765 (time 1787498153.26 total 715.78 last 20.23) - open/close 9906 (time 1787498163.26 total 725.78 last 14.10) - open/close 10000 (time 1787498167.19 total 729.71 last 23.90) - open/close 10179 (time 1787498177.20 total 739.71 last 17.89) - open/close 10368 (time 1787498187.42 total 749.94 last 18.49) - open/close 10526 (time 1787498198.11 total 760.63 last 14.78) - open/close 10783 (time 1787498208.11 total 770.63 last 25.70) - open/close 10999 (time 1787498218.22 total 780.74 last 21.36) - open/close 11187 (time 1787498229.07 total 791.59 last 17.33) - open/close 11351 (time 1787498239.98 total 802.49 last 15.04) - open/close 11450 (time 1787498250.26 total 812.78 last 9.62) - open/close 11616 (time 1787498260.28 total 822.80 last 16.57) - open/close 11743 (time 1787498270.83 total 833.34 last 12.05) - open/close 11934 (time 1787498280.84 total 843.36 last 19.08) - open/close 12132 (time 1787498291.76 total 854.27 last 18.13) - open/close 12382 (time 1787498302.81 total 865.33 last 22.61) - open/close 12622 (time 1787498312.99 total 875.51 last 23.58) - open/close 12878 (time 1787498323.00 total 885.52 last 25.59) - open/close 13011 (time 1787498333.00 total 895.52 last 13.30) - open/close 13139 (time 1787498343.26 total 905.78 last 12.47) - open/close 13283 (time 1787498353.27 total 915.79 last 14.39) - open/close 13433 (time 1787498365.20 total 927.72 last 12.57) - open/close 13590 (time 1787498376.21 total 938.73 last 14.26) - open/close 13735 (time 1787498386.26 total 948.77 last 14.44) - open/close 13997 (time 1787498396.31 total 958.83 last 26.07) - open/close 14152 (time 1787498406.31 total 968.83 last 15.49) - open/close 14292 (time 1787498416.41 total 978.93 last 13.87) - open/close 14393 (time 1787498426.96 total 989.48 last 9.57) - open/close 14499 (time 1787498437.59 total 1000.11 last 9.97) - open/close 14675 (time 1787498448.61 total 1011.12 last 15.98) - open/close 14816 (time 1787498459.43 total 1021.95 last 13.02) - open/close 14964 (time 1787498469.72 total 1032.24 last 14.38) - open/close 15098 (time 1787498479.73 total 1042.25 last 13.39) - open/close 15276 (time 1787498490.80 total 1053.32 last 16.08) - open/close 15438 (time 1787498500.83 total 1063.35 last 16.16) - open/close 15789 (time 1787498510.83 total 1073.35 last 35.09) - open/close 16382 (time 1787498520.83 total 1083.35 last 59.27) - open/close 17047 (time 1787498530.84 total 1093.35 last 66.49) - open/close 17662 (time 1787498540.84 total 1103.36 last 61.48) - open/close 18403 (time 1787498550.85 total 1113.37 last 74.02) - open/close 19119 (time 1787498560.85 total 1123.37 last 71.59) - open/close 19890 (time 1787498570.86 total 1133.37 last 77.06) - open/close 20595 (time 1787498580.86 total 1143.38 last 70.46) - open/close 21324 (time 1787498590.87 total 1153.39 last 72.84) - open/close 21982 (time 1787498600.88 total 1163.40 last 65.70) - open/close 22698 (time 1787498610.89 total 1173.40 last 71.58) - open/close 23381 (time 1787498620.90 total 1183.42 last 68.20) - open/close 24043 (time 1787498630.91 total 1193.43 last 66.12) - open/close 24694 (time 1787498640.92 total 1203.44 last 65.03) - open/close 25419 (time 1787498650.93 total 1213.45 last 72.46) - open/close 26242 (time 1787498660.93 total 1223.45 last 82.30) - open/close 26754 (time 1787498670.93 total 1233.45 last 51.17) - open/close 27280 (time 1787498680.95 total 1243.46 last 52.54) - open/close 27984 (time 1787498690.95 total 1253.47 last 70.38) - open/close 28687 (time 1787498700.95 total 1263.47 last 70.25) - open/close 29317 (time 1787498710.96 total 1273.48 last 62.94) - open/close 30000 (time 1787498719.45 total 1281.97 last 80.50) - open/close 30631 (time 1787498729.46 total 1291.97 last 63.05) - open/close 31266 (time 1787498739.47 total 1301.99 last 63.43) - open/close 31957 (time 1787498749.47 total 1311.99 last 69.09) - open/close 32614 (time 1787498759.48 total 1321.99 last 65.65) - open/close 33358 (time 1787498769.48 total 1331.99 last 74.40) - open/close 33987 (time 1787498779.48 total 1342.00 last 62.86) - open/close 34578 (time 1787498789.52 total 1352.04 last 58.88) - open/close 35077 (time 1787498799.54 total 1362.05 last 49.82) - open/close 35650 (time 1787498809.54 total 1372.06 last 57.28) - open/close 36199 (time 1787498819.55 total 1382.06 last 54.86) - open/close 36703 (time 1787498829.55 total 1392.07 last 50.39) - open/close 37206 (time 1787498839.56 total 1402.08 last 50.24) - open/close 37716 (time 1787498849.58 total 1412.10 last 50.92) - open/close 38385 (time 1787498859.59 total 1422.11 last 66.83) - open/close 39093 (time 1787498869.59 total 1432.11 last 70.78) - open/close 39760 (time 1787498879.59 total 1442.11 last 66.68) - open/close 40000 (time 1787498883.10 total 1445.62 last 68.48) - open/close 40623 (time 1787498893.10 total 1455.62 last 62.29) - open/close 41255 (time 1787498903.11 total 1465.63 last 63.10) - open/close 41889 (time 1787498913.12 total 1475.64 last 63.37) - open/close 42610 (time 1787498923.13 total 1485.65 last 72.00) - open/close 43240 (time 1787498933.14 total 1495.66 last 62.93) - open/close 43839 (time 1787498943.19 total 1505.71 last 59.63) - open/close 44431 (time 1787498953.19 total 1515.71 last 59.18) - open/close 45173 (time 1787498963.21 total 1525.73 last 74.09) - open/close 45772 (time 1787498973.21 total 1535.73 last 59.85) - open/close 46310 (time 1787498983.22 total 1545.74 last 53.75) - open/close 47024 (time 1787498993.24 total 1555.76 last 71.27) - open/close 47722 (time 1787499003.25 total 1565.76 last 69.77) - open/close 48440 (time 1787499013.25 total 1575.77 last 71.75) - open/close 49133 (time 1787499023.28 total 1585.80 last 69.11) - open/close 49906 (time 1787499033.29 total 1595.81 last 77.25) total: 50000 open/close in 1597.01 seconds: 31.31 ops/second - unlinked 0 (time 1787499040 ; total 0 ; last 0) - unlinked 10000 (time 1787499179 ; total 139 ; last 139) - unlinked 20000 (time 1787499315 ; total 275 ; last 136) - unlinked 30000 (time 1787499441 ; total 401 ; last 126) - unlinked 40000 (time 1787499558 ; total 518 ; last 117) total: 50000 unlinks in 638 seconds: 78.369904 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2