== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:42:27 (1784587347) start mds service on oleg245-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg245-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 136 (time 1784587439.51 total 10.35 last 13.14) - open/close 326 (time 1784587450.25 total 21.08 last 17.70) - open/close 530 (time 1784587460.55 total 31.39 last 19.80) - open/close 742 (time 1784587471.51 total 42.35 last 19.34) - open/close 896 (time 1784587481.87 total 52.71 last 14.87) - open/close 1121 (time 1784587491.87 total 62.71 last 22.50) - open/close 1202 (time 1784587503.50 total 74.34 last 6.97) - open/close 1281 (time 1784587514.07 total 84.91 last 7.47) - open/close 1371 (time 1784587525.04 total 95.88 last 8.21) - open/close 1488 (time 1784587535.04 total 105.88 last 11.69) - open/close 1663 (time 1784587545.94 total 116.78 last 16.06) - open/close 1761 (time 1784587556.69 total 127.52 last 9.12) - open/close 1932 (time 1784587566.71 total 137.54 last 17.06) - open/close 2132 (time 1784587576.72 total 147.56 last 19.96) - open/close 2299 (time 1784587589.02 total 159.85 last 13.59) - open/close 2420 (time 1784587599.13 total 169.96 last 11.97) - open/close 2548 (time 1784587611.43 total 182.26 last 10.41) - open/close 2699 (time 1784587622.34 total 193.18 last 13.83) - open/close 2832 (time 1784587633.45 total 204.28 last 11.98) - open/close 2990 (time 1784587643.46 total 214.29 last 15.79) - open/close 3104 (time 1784587653.47 total 224.31 last 11.38) - open/close 3263 (time 1784587664.47 total 235.31 last 14.45) - open/close 3482 (time 1784587674.50 total 245.34 last 21.85) - open/close 3595 (time 1784587684.51 total 255.34 last 11.29) - open/close 3671 (time 1784587696.81 total 267.65 last 6.17) - open/close 3816 (time 1784587706.82 total 277.66 last 14.49) - open/close 3935 (time 1784587717.85 total 288.68 last 10.80) - open/close 4016 (time 1784587727.87 total 298.70 last 8.08) - open/close 4221 (time 1784587738.59 total 309.42 last 19.13) - open/close 4333 (time 1784587748.88 total 319.71 last 10.88) - open/close 4439 (time 1784587759.06 total 329.90 last 10.40) - open/close 4557 (time 1784587769.19 total 340.02 last 11.66) - open/close 4657 (time 1784587779.92 total 350.76 last 9.31) - open/close 4780 (time 1784587789.93 total 360.77 last 12.29) - open/close 5019 (time 1784587801.44 total 372.27 last 20.78) - open/close 5215 (time 1784587811.44 total 382.28 last 19.58) - open/close 5364 (time 1784587821.47 total 392.30 last 14.87) - open/close 5529 (time 1784587831.91 total 402.74 last 15.81) - open/close 5583 (time 1784587841.91 total 412.75 last 5.40) - open/close 5681 (time 1784587851.92 total 422.76 last 9.79) - open/close 5850 (time 1784587863.32 total 434.15 last 14.83) - open/close 6057 (time 1784587873.34 total 444.18 last 20.66) - open/close 6145 (time 1784587883.64 total 454.47 last 8.54) - open/close 6278 (time 1784587893.64 total 464.48 last 13.29) - open/close 6463 (time 1784587903.65 total 474.49 last 18.48) - open/close 6625 (time 1784587916.40 total 487.23 last 12.71) - open/close 6859 (time 1784587927.29 total 498.12 last 21.49) - open/close 7012 (time 1784587937.29 total 508.13 last 15.30) - open/close 7174 (time 1784587949.83 total 520.67 last 12.92) - open/close 7268 (time 1784587960.85 total 531.69 last 8.53) - open/close 7428 (time 1784587971.67 total 542.50 last 14.80) - open/close 7625 (time 1784587982.43 total 553.26 last 18.31) - open/close 7807 (time 1784587993.38 total 564.21 last 16.62) - open/close 8011 (time 1784588003.40 total 574.24 last 20.35) - open/close 8213 (time 1784588014.50 total 585.34 last 18.20) - open/close 8405 (time 1784588024.50 total 595.34 last 19.19) - open/close 8649 (time 1784588034.51 total 605.35 last 24.38) - open/close 8771 (time 1784588045.59 total 616.43 last 11.02) - open/close 8854 (time 1784588055.85 total 626.69 last 8.09) - open/close 9053 (time 1784588066.10 total 636.94 last 19.41) - open/close 9275 (time 1784588076.89 total 647.73 last 20.58) - open/close 9440 (time 1784588087.11 total 657.95 last 16.14) - open/close 9606 (time 1784588097.33 total 668.17 last 16.24) - open/close 9817 (time 1784588108.14 total 678.98 last 19.52) - open/close 9996 (time 1784588118.81 total 689.65 last 16.77) - open/close 10121 (time 1784588129.79 total 700.63 last 11.38) - open/close 10313 (time 1784588139.82 total 710.66 last 19.15) - open/close 10453 (time 1784588150.03 total 720.87 last 13.71) - open/close 10578 (time 1784588160.05 total 730.89 last 12.48) - open/close 10789 (time 1784588170.16 total 741.00 last 20.86) - open/close 10998 (time 1784588180.19 total 751.02 last 20.85) - open/close 11120 (time 1784588190.21 total 761.05 last 12.17) - open/close 11272 (time 1784588200.35 total 771.18 last 14.99) - open/close 11461 (time 1784588210.54 total 781.37 last 18.55) - open/close 11651 (time 1784588220.56 total 791.40 last 18.95) - open/close 11834 (time 1784588232.86 total 803.70 last 14.88) - open/close 12050 (time 1784588244.58 total 815.42 last 18.42) - open/close 12273 (time 1784588254.67 total 825.51 last 22.11) - open/close 12459 (time 1784588264.98 total 835.82 last 18.04) - open/close 12698 (time 1784588274.99 total 845.82 last 23.88) - open/close 12904 (time 1784588285.00 total 855.84 last 20.58) - open/close 13012 (time 1784588295.01 total 865.85 last 10.79) - open/close 13262 (time 1784588305.03 total 875.87 last 24.94) - open/close 13339 (time 1784588315.30 total 886.14 last 7.50) - open/close 13447 (time 1784588325.32 total 896.16 last 10.77) - open/close 13484 (time 1784588335.45 total 906.28 last 3.66) - open/close 13686 (time 1784588345.81 total 916.64 last 19.49) - open/close 13825 (time 1784588357.66 total 928.49 last 11.73) - open/close 13990 (time 1784588371.64 total 942.47 last 11.80) - open/close 14209 (time 1784588381.64 total 952.47 last 21.90) - open/close 14346 (time 1784588392.60 total 963.43 last 12.50) - open/close 14556 (time 1784588405.21 total 976.05 last 16.65) - open/close 14804 (time 1784588416.10 total 986.94 last 22.76) - open/close 15029 (time 1784588426.12 total 996.96 last 22.46) - open/close 15304 (time 1784588437.14 total 1007.98 last 24.96) - open/close 15566 (time 1784588447.14 total 1017.98 last 26.20) - open/close 15797 (time 1784588457.17 total 1028.01 last 23.03) - open/close 15955 (time 1784588467.17 total 1038.01 last 15.80) - open/close 16223 (time 1784588477.19 total 1048.03 last 26.75) - open/close 16538 (time 1784588487.21 total 1058.04 last 31.45) - open/close 17268 (time 1784588497.21 total 1068.05 last 72.99) - open/close 17900 (time 1784588507.23 total 1078.07 last 63.05) - open/close 18548 (time 1784588517.24 total 1088.08 last 64.75) - open/close 19177 (time 1784588527.24 total 1098.08 last 62.88) - open/close 19749 (time 1784588537.25 total 1108.08 last 57.18) - open/close 20000 (time 1784588541.80 total 1112.63 last 55.15) - open/close 20658 (time 1784588551.80 total 1122.64 last 65.78) - open/close 21197 (time 1784588561.81 total 1132.64 last 53.87) - open/close 21928 (time 1784588571.81 total 1142.65 last 73.05) - open/close 22695 (time 1784588581.82 total 1152.66 last 76.63) - open/close 23290 (time 1784588591.83 total 1162.67 last 59.43) - open/close 23831 (time 1784588601.84 total 1172.67 last 54.09) - open/close 24445 (time 1784588611.85 total 1182.68 last 61.33) - open/close 25244 (time 1784588621.85 total 1192.69 last 79.84) - open/close 25986 (time 1784588631.86 total 1202.70 last 74.16) - open/close 26688 (time 1784588641.87 total 1212.71 last 70.09) - open/close 27433 (time 1784588651.88 total 1222.72 last 74.43) - open/close 28210 (time 1784588661.88 total 1232.72 last 77.70) - open/close 28906 (time 1784588671.89 total 1242.73 last 69.57) - open/close 29576 (time 1784588681.90 total 1252.73 last 66.95) - open/close 30000 (time 1784588687.66 total 1258.50 last 73.56) - open/close 30722 (time 1784588697.67 total 1268.50 last 72.14) - open/close 31399 (time 1784588707.69 total 1278.52 last 67.57) - open/close 32103 (time 1784588717.70 total 1288.54 last 70.32) - open/close 32817 (time 1784588727.71 total 1298.55 last 71.30) - open/close 33451 (time 1784588737.72 total 1308.56 last 63.35) - open/close 34210 (time 1784588747.73 total 1318.57 last 75.82) - open/close 34932 (time 1784588757.73 total 1328.57 last 72.18) - open/close 35642 (time 1784588767.74 total 1338.58 last 70.92) - open/close 36310 (time 1784588777.76 total 1348.60 last 66.69) - open/close 36937 (time 1784588787.76 total 1358.60 last 62.68) - open/close 37618 (time 1784588797.77 total 1368.61 last 68.03) - open/close 38141 (time 1784588807.78 total 1378.62 last 52.27) - open/close 38863 (time 1784588817.79 total 1388.62 last 72.14) - open/close 39596 (time 1784588827.79 total 1398.63 last 73.25) - open/close 40000 (time 1784588833.21 total 1404.05 last 74.61) - open/close 40781 (time 1784588843.21 total 1414.05 last 78.10) - open/close 41515 (time 1784588853.22 total 1424.05 last 73.35) - open/close 42206 (time 1784588863.22 total 1434.06 last 69.04) - open/close 42918 (time 1784588873.23 total 1444.07 last 71.13) - open/close 43570 (time 1784588883.24 total 1454.08 last 65.16) - open/close 44256 (time 1784588893.25 total 1464.09 last 68.50) - open/close 44872 (time 1784588903.26 total 1474.10 last 61.53) - open/close 45527 (time 1784588913.27 total 1484.10 last 65.49) - open/close 46142 (time 1784588923.28 total 1494.11 last 61.43) - open/close 46890 (time 1784588933.28 total 1504.12 last 74.80) - open/close 47652 (time 1784588943.28 total 1514.12 last 76.17) - open/close 48385 (time 1784588953.28 total 1524.12 last 73.29) - open/close 49104 (time 1784588963.31 total 1534.15 last 71.69) - open/close 49800 (time 1784588973.32 total 1544.16 last 69.53) total: 50000 open/close in 1547.37 seconds: 32.31 ops/second - unlinked 0 (time 1784588982 ; total 0 ; last 0) - unlinked 10000 (time 1784589137 ; total 155 ; last 155) - unlinked 20000 (time 1784589292 ; total 310 ; last 155) - unlinked 30000 (time 1784589465 ; total 483 ; last 173) - unlinked 40000 (time 1784589632 ; total 650 ; last 167) total: 50000 unlinks in 820 seconds: 60.975609 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2