== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:06:35 (1787551595) start mds service on oleg619-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing load_modules_local oleg619-server: Loading modules from /home/green/git/lustre-release/lustre oleg619-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg619-server: detected 4 online CPUs by sysfs oleg619-server: MODOPTS_LIBCFS= oleg619-server: Force libcfs to create 2 CPU partitions oleg619-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg619-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg619-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg619-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 153 (time 1787551703.01 total 10.47 last 14.62) - open/close 396 (time 1787551713.04 total 20.50 last 24.22) - open/close 599 (time 1787551723.72 total 31.18 last 19.01) - open/close 896 (time 1787551734.75 total 42.21 last 26.91) - open/close 1071 (time 1787551745.65 total 53.11 last 16.06) - open/close 1280 (time 1787551757.59 total 65.05 last 17.50) - open/close 1476 (time 1787551767.61 total 75.07 last 19.56) - open/close 1664 (time 1787551777.63 total 85.09 last 18.77) - open/close 1893 (time 1787551787.64 total 95.10 last 22.88) - open/close 2013 (time 1787551797.64 total 105.10 last 11.99) - open/close 2206 (time 1787551807.65 total 115.11 last 19.29) - open/close 2401 (time 1787551818.42 total 125.88 last 18.11) - open/close 2577 (time 1787551829.44 total 136.90 last 15.97) - open/close 2740 (time 1787551839.47 total 146.93 last 16.25) - open/close 2831 (time 1787551850.34 total 157.80 last 8.38) - open/close 2977 (time 1787551860.91 total 168.37 last 13.81) - open/close 3116 (time 1787551871.92 total 179.38 last 12.62) - open/close 3340 (time 1787551881.93 total 189.39 last 22.37) - open/close 3533 (time 1787551892.85 total 200.31 last 17.68) - open/close 3734 (time 1787551903.46 total 210.92 last 18.94) - open/close 3900 (time 1787551913.47 total 220.93 last 16.58) - open/close 4102 (time 1787551923.94 total 231.40 last 19.29) - open/close 4257 (time 1787551933.98 total 241.44 last 15.44) - open/close 4399 (time 1787551946.22 total 253.68 last 11.60) - open/close 4547 (time 1787551956.23 total 263.69 last 14.79) - open/close 4639 (time 1787551967.15 total 274.61 last 8.42) - open/close 4741 (time 1787551977.32 total 284.78 last 10.03) - open/close 4791 (time 1787551987.76 total 295.22 last 4.79) - open/close 4995 (time 1787551998.47 total 305.93 last 19.06) - open/close 5059 (time 1787552009.30 total 316.76 last 5.91) - open/close 5135 (time 1787552019.32 total 326.78 last 7.59) - open/close 5166 (time 1787552029.54 total 337.00 last 3.03) - open/close 5236 (time 1787552039.66 total 347.12 last 6.92) - open/close 5332 (time 1787552050.10 total 357.56 last 9.20) - open/close 5581 (time 1787552060.68 total 368.14 last 23.53) - open/close 5655 (time 1787552071.08 total 378.54 last 7.12) - open/close 5813 (time 1787552083.80 total 391.26 last 12.42) - open/close 5971 (time 1787552094.64 total 402.10 last 14.57) - open/close 6068 (time 1787552105.19 total 412.65 last 9.20) - open/close 6173 (time 1787552115.57 total 423.03 last 10.12) - open/close 6290 (time 1787552127.39 total 434.85 last 9.89) - open/close 6451 (time 1787552140.27 total 447.73 last 12.50) - open/close 6647 (time 1787552150.33 total 457.79 last 19.50) - open/close 6881 (time 1787552160.48 total 467.94 last 23.04) - open/close 7102 (time 1787552171.44 total 478.90 last 20.16) - open/close 7272 (time 1787552181.47 total 488.93 last 16.96) - open/close 7403 (time 1787552191.47 total 498.93 last 13.10) - open/close 7635 (time 1787552201.47 total 508.93 last 23.20) - open/close 7789 (time 1787552212.02 total 519.48 last 14.59) - open/close 7928 (time 1787552222.02 total 529.48 last 13.90) - open/close 8170 (time 1787552232.03 total 539.49 last 24.19) - open/close 8340 (time 1787552243.56 total 551.02 last 14.74) - open/close 8458 (time 1787552253.56 total 561.02 last 11.80) - open/close 8661 (time 1787552263.74 total 571.20 last 19.94) - open/close 8732 (time 1787552273.75 total 581.21 last 7.09) - open/close 8949 (time 1787552283.78 total 591.24 last 21.65) - open/close 9140 (time 1787552294.37 total 601.83 last 18.03) - open/close 9321 (time 1787552304.48 total 611.94 last 17.91) - open/close 9510 (time 1787552314.49 total 621.95 last 18.88) - open/close 9685 (time 1787552324.50 total 631.96 last 17.48) - open/close 9832 (time 1787552334.71 total 642.17 last 14.40) - open/close 10000 (time 1787552342.75 total 650.21 last 20.90) - open/close 10100 (time 1787552353.77 total 661.23 last 9.07) - open/close 10281 (time 1787552364.20 total 671.66 last 17.36) - open/close 10535 (time 1787552374.21 total 681.67 last 25.38) - open/close 10760 (time 1787552386.67 total 694.13 last 18.06) - open/close 10984 (time 1787552397.03 total 704.49 last 21.61) - open/close 11212 (time 1787552407.71 total 715.17 last 21.35) - open/close 11247 (time 1787552417.92 total 725.38 last 3.43) - open/close 11477 (time 1787552428.79 total 736.25 last 21.16) - open/close 11714 (time 1787552439.72 total 747.18 last 21.68) - open/close 11896 (time 1787552451.65 total 759.11 last 15.25) - open/close 12019 (time 1787552461.65 total 769.11 last 12.30) - open/close 12134 (time 1787552471.67 total 779.13 last 11.47) - open/close 12343 (time 1787552481.68 total 789.14 last 20.89) - open/close 12623 (time 1787552491.68 total 799.14 last 28.00) - open/close 12882 (time 1787552502.66 total 810.12 last 23.58) - open/close 13102 (time 1787552512.99 total 820.45 last 21.30) - open/close 13344 (time 1787552522.99 total 830.45 last 24.19) - open/close 13561 (time 1787552533.02 total 840.48 last 21.65) - open/close 13793 (time 1787552543.91 total 851.37 last 21.30) - open/close 14016 (time 1787552554.21 total 861.67 last 21.64) - open/close 14235 (time 1787552564.21 total 871.67 last 21.90) - open/close 14345 (time 1787552574.25 total 881.70 last 10.97) - open/close 14464 (time 1787552585.28 total 892.73 last 10.79) - open/close 14633 (time 1787552596.26 total 903.72 last 15.38) - open/close 14818 (time 1787552606.98 total 914.44 last 17.27) - open/close 14917 (time 1787552617.77 total 925.23 last 9.17) - open/close 15017 (time 1787552629.66 total 937.12 last 8.41) - open/close 15263 (time 1787552639.90 total 947.36 last 24.02) - open/close 15483 (time 1787552650.48 total 957.94 last 20.81) - open/close 15667 (time 1787552660.91 total 968.37 last 17.63) - open/close 15894 (time 1787552670.93 total 978.39 last 22.66) - open/close 15980 (time 1787552681.25 total 988.71 last 8.33) - open/close 16059 (time 1787552691.65 total 999.11 last 7.60) - open/close 16170 (time 1787552702.65 total 1010.11 last 10.09) - open/close 16425 (time 1787552714.72 total 1022.18 last 21.12) - open/close 16911 (time 1787552724.73 total 1032.19 last 48.59) - open/close 17590 (time 1787552734.73 total 1042.19 last 67.84) - open/close 18349 (time 1787552744.75 total 1052.21 last 75.81) - open/close 19105 (time 1787552754.75 total 1062.21 last 75.57) - open/close 19812 (time 1787552764.75 total 1072.21 last 70.69) - open/close 20000 (time 1787552767.39 total 1074.85 last 71.24) - open/close 20650 (time 1787552777.43 total 1084.89 last 64.73) - open/close 21237 (time 1787552787.44 total 1094.90 last 58.65) - open/close 21887 (time 1787552797.46 total 1104.92 last 64.88) - open/close 22580 (time 1787552807.47 total 1114.93 last 69.25) - open/close 23239 (time 1787552817.47 total 1124.93 last 65.86) - open/close 23956 (time 1787552827.49 total 1134.95 last 71.56) - open/close 24629 (time 1787552837.50 total 1144.96 last 67.27) - open/close 25308 (time 1787552847.50 total 1154.96 last 67.88) - open/close 25993 (time 1787552857.51 total 1164.97 last 68.43) - open/close 26598 (time 1787552867.51 total 1174.97 last 60.50) - open/close 27198 (time 1787552877.52 total 1184.98 last 59.93) - open/close 27815 (time 1787552887.56 total 1195.02 last 61.48) - open/close 28422 (time 1787552897.56 total 1205.02 last 60.69) - open/close 29053 (time 1787552907.57 total 1215.03 last 63.04) - open/close 29644 (time 1787552917.58 total 1225.04 last 59.02) - open/close 30000 (time 1787552924.22 total 1231.68 last 53.64) - open/close 30604 (time 1787552934.22 total 1241.68 last 60.38) - open/close 31237 (time 1787552944.24 total 1251.70 last 63.21) - open/close 31881 (time 1787552954.26 total 1261.72 last 64.27) - open/close 32484 (time 1787552964.26 total 1271.72 last 60.28) - open/close 33191 (time 1787552974.26 total 1281.72 last 70.68) - open/close 33789 (time 1787552984.28 total 1291.74 last 59.68) - open/close 34329 (time 1787552994.29 total 1301.75 last 53.97) - open/close 34956 (time 1787553004.30 total 1311.76 last 62.66) - open/close 35662 (time 1787553014.30 total 1321.76 last 70.57) - open/close 36328 (time 1787553024.31 total 1331.77 last 66.55) - open/close 37066 (time 1787553034.31 total 1341.77 last 73.76) - open/close 37736 (time 1787553044.32 total 1351.78 last 66.99) - open/close 38238 (time 1787553054.32 total 1361.78 last 50.16) - open/close 38878 (time 1787553064.33 total 1371.79 last 63.93) - open/close 39425 (time 1787553074.37 total 1381.83 last 54.49) - open/close 39983 (time 1787553084.38 total 1391.84 last 55.77) - open/close 40613 (time 1787553094.38 total 1401.84 last 62.95) - open/close 41243 (time 1787553104.39 total 1411.85 last 62.96) - open/close 41886 (time 1787553114.39 total 1421.85 last 64.30) - open/close 42565 (time 1787553124.39 total 1431.85 last 67.90) - open/close 43266 (time 1787553134.39 total 1441.85 last 70.08) - open/close 43878 (time 1787553144.40 total 1451.86 last 61.16) - open/close 44575 (time 1787553154.41 total 1461.87 last 69.66) - open/close 45276 (time 1787553164.41 total 1471.87 last 70.04) - open/close 45958 (time 1787553174.42 total 1481.88 last 68.16) - open/close 46666 (time 1787553184.44 total 1491.90 last 70.69) - open/close 47363 (time 1787553194.45 total 1501.91 last 69.64) - open/close 48071 (time 1787553204.46 total 1511.92 last 70.68) - open/close 48778 (time 1787553214.47 total 1521.93 last 70.64) - open/close 49431 (time 1787553224.48 total 1531.94 last 65.21) total: 50000 open/close in 1540.60 seconds: 32.45 ops/second - unlinked 0 (time 1787553237 ; total 0 ; last 0) - unlinked 10000 (time 1787553396 ; total 159 ; last 159) - unlinked 20000 (time 1787553547 ; total 310 ; last 151) - unlinked 30000 (time 1787553703 ; total 466 ; last 156) - unlinked 40000 (time 1787553853 ; total 616 ; last 150) total: 50000 unlinks in 768 seconds: 65.104164 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg619-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg619-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server stop mds service on oleg619-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server unloading modules via unload_modules_local on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing unload_modules_local oleg619-server: modules unloaded. pdsh@oleg619-client: oleg619-server: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-client: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-client: ssh exited with exit code 2 pdsh@oleg619-client: oleg619-server: ssh exited with exit code 2