== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:19:34 (1781223574) start mds service on oleg218-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg218-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 103 (time 1781223664.52 total 10.24 last 10.06) - open/close 256 (time 1781223674.54 total 20.26 last 15.27) - open/close 377 (time 1781223684.54 total 30.26 last 12.10) - open/close 582 (time 1781223694.55 total 40.27 last 20.48) - open/close 769 (time 1781223704.95 total 50.68 last 17.97) - open/close 1003 (time 1781223715.13 total 60.86 last 22.99) - open/close 1047 (time 1781223725.58 total 71.30 last 4.21) - open/close 1084 (time 1781223736.06 total 81.78 last 3.53) - open/close 1171 (time 1781223746.39 total 92.12 last 8.42) - open/close 1227 (time 1781223757.38 total 103.10 last 5.10) - open/close 1303 (time 1781223767.95 total 113.67 last 7.19) - open/close 1440 (time 1781223778.75 total 124.48 last 12.68) - open/close 1573 (time 1781223789.31 total 135.03 last 12.60) - open/close 1701 (time 1781223799.71 total 145.43 last 12.31) - open/close 1778 (time 1781223810.08 total 155.80 last 7.42) - open/close 1939 (time 1781223820.09 total 165.81 last 16.09) - open/close 2200 (time 1781223830.10 total 175.82 last 26.07) - open/close 2341 (time 1781223840.10 total 185.83 last 14.09) - open/close 2515 (time 1781223850.89 total 196.61 last 16.13) - open/close 2659 (time 1781223860.90 total 206.62 last 14.39) - open/close 2717 (time 1781223873.68 total 219.40 last 4.54) - open/close 2820 (time 1781223883.69 total 229.41 last 10.29) - open/close 2879 (time 1781223895.59 total 241.31 last 4.96) - open/close 3025 (time 1781223905.61 total 251.33 last 14.57) - open/close 3173 (time 1781223915.84 total 261.57 last 14.46) - open/close 3404 (time 1781223925.86 total 271.59 last 23.05) - open/close 3593 (time 1781223936.39 total 282.11 last 17.96) - open/close 3724 (time 1781223946.77 total 292.50 last 12.61) - open/close 3835 (time 1781223956.79 total 302.51 last 11.09) - open/close 4074 (time 1781223967.83 total 313.55 last 21.64) - open/close 4239 (time 1781223978.25 total 323.97 last 15.83) - open/close 4469 (time 1781223988.81 total 334.53 last 21.78) - open/close 4624 (time 1781223999.35 total 345.07 last 14.71) - open/close 4750 (time 1781224010.19 total 355.91 last 11.63) - open/close 4868 (time 1781224020.20 total 365.93 last 11.78) - open/close 5076 (time 1781224030.22 total 375.94 last 20.77) - open/close 5208 (time 1781224040.23 total 385.95 last 13.19) - open/close 5258 (time 1781224051.95 total 397.67 last 4.27) - open/close 5496 (time 1781224062.94 total 408.66 last 21.66) - open/close 5762 (time 1781224072.94 total 418.66 last 26.59) - open/close 5934 (time 1781224083.78 total 429.50 last 15.87) - open/close 5992 (time 1781224093.80 total 439.52 last 5.79) - open/close 6099 (time 1781224103.82 total 449.54 last 10.68) - open/close 6274 (time 1781224113.87 total 459.59 last 17.42) - open/close 6408 (time 1781224123.88 total 469.60 last 13.38) - open/close 6665 (time 1781224134.60 total 480.32 last 23.98) - open/close 6810 (time 1781224145.41 total 491.13 last 13.41) - open/close 6916 (time 1781224156.48 total 502.20 last 9.58) - open/close 7189 (time 1781224167.49 total 513.21 last 24.80) - open/close 7343 (time 1781224177.49 total 523.21 last 15.40) - open/close 7492 (time 1781224187.58 total 533.30 last 14.77) - open/close 7688 (time 1781224198.62 total 544.34 last 17.75) - open/close 7769 (time 1781224209.35 total 555.08 last 7.55) - open/close 7871 (time 1781224220.17 total 565.89 last 9.43) - open/close 7947 (time 1781224230.52 total 576.24 last 7.35) - open/close 8131 (time 1781224240.53 total 586.25 last 18.37) - open/close 8252 (time 1781224250.54 total 596.26 last 12.10) - open/close 8427 (time 1781224261.25 total 606.97 last 16.34) - open/close 8568 (time 1781224271.26 total 616.98 last 14.09) - open/close 8714 (time 1781224281.27 total 626.99 last 14.58) - open/close 8924 (time 1781224292.04 total 637.77 last 19.49) - open/close 9077 (time 1781224302.05 total 647.77 last 15.29) - open/close 9177 (time 1781224312.34 total 658.06 last 9.72) - open/close 9305 (time 1781224323.48 total 669.20 last 11.49) - open/close 9593 (time 1781224333.57 total 679.29 last 28.55) - open/close 9774 (time 1781224344.59 total 690.31 last 16.42) - open/close 9928 (time 1781224354.79 total 700.51 last 15.11) - open/close 10000 (time 1781224357.59 total 703.31 last 25.73) - open/close 10061 (time 1781224368.64 total 714.37 last 5.52) - open/close 10316 (time 1781224378.65 total 724.37 last 25.50) - open/close 10575 (time 1781224388.90 total 734.62 last 25.25) - open/close 10843 (time 1781224398.91 total 744.63 last 26.79) - open/close 10922 (time 1781224408.93 total 754.65 last 7.88) - open/close 11011 (time 1781224418.94 total 764.66 last 8.89) - open/close 11244 (time 1781224428.94 total 774.66 last 23.30) - open/close 11441 (time 1781224438.95 total 784.67 last 19.69) - open/close 11614 (time 1781224451.21 total 796.93 last 14.11) - open/close 11690 (time 1781224461.24 total 806.96 last 7.58) - open/close 11899 (time 1781224471.25 total 816.97 last 20.88) - open/close 12090 (time 1781224481.55 total 827.27 last 18.54) - open/close 12264 (time 1781224492.17 total 837.89 last 16.39) - open/close 12439 (time 1781224502.17 total 847.89 last 17.49) - open/close 12621 (time 1781224512.59 total 858.31 last 17.47) - open/close 12788 (time 1781224523.59 total 869.31 last 15.18) - open/close 12879 (time 1781224534.16 total 879.88 last 8.61) - open/close 13095 (time 1781224544.84 total 890.56 last 20.22) - open/close 13258 (time 1781224555.19 total 900.92 last 15.74) - open/close 13421 (time 1781224566.11 total 911.83 last 14.94) - open/close 13547 (time 1781224577.02 total 922.74 last 11.54) - open/close 13733 (time 1781224588.29 total 934.01 last 16.50) - open/close 13910 (time 1781224599.30 total 945.02 last 16.08) - open/close 14072 (time 1781224609.30 total 955.02 last 16.20) - open/close 14174 (time 1781224619.34 total 965.06 last 10.16) - open/close 14369 (time 1781224630.43 total 976.15 last 17.59) - open/close 14429 (time 1781224640.44 total 986.16 last 6.00) - open/close 14660 (time 1781224650.78 total 996.50 last 22.34) - open/close 14780 (time 1781224660.80 total 1006.52 last 11.97) - open/close 14921 (time 1781224671.49 total 1017.22 last 13.19) - open/close 15061 (time 1781224681.51 total 1027.23 last 13.98) - open/close 15265 (time 1781224691.67 total 1037.39 last 20.09) - open/close 15366 (time 1781224701.68 total 1047.40 last 10.09) - open/close 15453 (time 1781224712.02 total 1057.74 last 8.42) - open/close 15513 (time 1781224725.14 total 1070.86 last 4.57) - open/close 15585 (time 1781224735.15 total 1080.87 last 7.19) - open/close 15690 (time 1781224746.01 total 1091.73 last 9.67) - open/close 15789 (time 1781224756.01 total 1101.73 last 9.90) - open/close 16409 (time 1781224766.04 total 1111.76 last 61.86) - open/close 16952 (time 1781224776.04 total 1121.76 last 54.28) - open/close 17579 (time 1781224786.05 total 1131.77 last 62.60) - open/close 18141 (time 1781224796.06 total 1141.78 last 56.18) - open/close 18887 (time 1781224806.07 total 1151.79 last 74.52) - open/close 19438 (time 1781224816.09 total 1161.81 last 54.96) - open/close 20000 (time 1781224825.03 total 1170.75 last 62.86) - open/close 20770 (time 1781224835.03 total 1180.76 last 76.98) - open/close 21462 (time 1781224845.04 total 1190.77 last 69.13) - open/close 22103 (time 1781224855.05 total 1200.77 last 64.05) - open/close 22744 (time 1781224865.06 total 1210.78 last 64.06) - open/close 23351 (time 1781224875.07 total 1220.79 last 60.66) - open/close 23930 (time 1781224885.07 total 1230.79 last 57.87) - open/close 24506 (time 1781224895.07 total 1240.80 last 57.58) - open/close 25209 (time 1781224905.08 total 1250.80 last 70.28) - open/close 25847 (time 1781224915.08 total 1260.80 last 63.78) - open/close 26452 (time 1781224925.09 total 1270.81 last 60.47) - open/close 27205 (time 1781224935.10 total 1280.82 last 75.22) - open/close 27863 (time 1781224945.10 total 1290.82 last 65.78) - open/close 28468 (time 1781224955.12 total 1300.84 last 60.40) - open/close 28937 (time 1781224965.13 total 1310.85 last 46.85) - open/close 29492 (time 1781224975.14 total 1320.86 last 55.43) - open/close 30000 (time 1781224983.67 total 1329.39 last 59.54) - open/close 30648 (time 1781224993.67 total 1339.39 last 64.78) - open/close 31383 (time 1781225003.68 total 1349.40 last 73.48) - open/close 32021 (time 1781225013.69 total 1359.41 last 63.73) - open/close 32582 (time 1781225023.70 total 1369.42 last 56.05) - open/close 33219 (time 1781225033.72 total 1379.44 last 63.56) - open/close 33847 (time 1781225043.73 total 1389.45 last 62.75) - open/close 34567 (time 1781225053.74 total 1399.46 last 71.91) - open/close 35176 (time 1781225063.74 total 1409.46 last 60.88) - open/close 35815 (time 1781225073.76 total 1419.48 last 63.82) - open/close 36498 (time 1781225083.77 total 1429.49 last 68.20) - open/close 37101 (time 1781225093.78 total 1439.50 last 60.26) - open/close 37772 (time 1781225103.79 total 1449.51 last 67.03) - open/close 38483 (time 1781225113.80 total 1459.52 last 71.04) - open/close 39121 (time 1781225123.80 total 1469.52 last 63.77) - open/close 39830 (time 1781225133.81 total 1479.54 last 70.81) - open/close 40000 (time 1781225136.29 total 1482.01 last 68.72) - open/close 40772 (time 1781225146.30 total 1492.02 last 77.10) - open/close 41528 (time 1781225156.31 total 1502.03 last 75.56) - open/close 42184 (time 1781225166.36 total 1512.08 last 65.27) - open/close 42827 (time 1781225176.37 total 1522.09 last 64.24) - open/close 43466 (time 1781225186.37 total 1532.09 last 63.89) - open/close 44148 (time 1781225196.37 total 1542.09 last 68.18) - open/close 44888 (time 1781225206.37 total 1552.09 last 74.00) - open/close 45530 (time 1781225216.38 total 1562.10 last 64.16) - open/close 46217 (time 1781225226.38 total 1572.10 last 68.67) - open/close 46904 (time 1781225236.39 total 1582.11 last 68.62) - open/close 47541 (time 1781225246.40 total 1592.12 last 63.65) - open/close 48350 (time 1781225256.42 total 1602.14 last 80.72) - open/close 48988 (time 1781225266.43 total 1612.15 last 63.77) - open/close 49586 (time 1781225276.43 total 1622.15 last 59.76) total: 50000 open/close in 1627.66 seconds: 30.72 ops/second - unlinked 0 (time 1781225287 ; total 0 ; last 0) - unlinked 10000 (time 1781225424 ; total 137 ; last 137) - unlinked 20000 (time 1781225567 ; total 280 ; last 143) - unlinked 30000 (time 1781225709 ; total 422 ; last 142) - unlinked 40000 (time 1781225856 ; total 569 ; last 147) total: 50000 unlinks in 709 seconds: 70.521858 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2