== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:58:58 (1780383538) start mds service on oleg225-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1780383632.00 total 10.06 last 19.58) - open/close 435 (time 1780383642.02 total 20.08 last 23.76) - open/close 703 (time 1780383652.79 total 30.85 last 24.88) - open/close 998 (time 1780383663.76 total 41.82 last 26.88) - open/close 1247 (time 1780383673.77 total 51.83 last 24.87) - open/close 1478 (time 1780383683.93 total 61.99 last 22.75) - open/close 1670 (time 1780383694.83 total 72.89 last 17.62) - open/close 1767 (time 1780383705.39 total 83.45 last 9.19) - open/close 1947 (time 1780383715.99 total 94.05 last 16.98) - open/close 2155 (time 1780383726.31 total 104.37 last 20.15) - open/close 2415 (time 1780383736.32 total 114.38 last 25.98) - open/close 2582 (time 1780383746.98 total 125.04 last 15.66) - open/close 2783 (time 1780383757.92 total 135.99 last 18.37) - open/close 2994 (time 1780383768.16 total 146.22 last 20.62) - open/close 3211 (time 1780383779.05 total 157.11 last 19.93) - open/close 3442 (time 1780383789.56 total 167.63 last 21.97) - open/close 3572 (time 1780383799.73 total 177.79 last 12.79) - open/close 3766 (time 1780383809.74 total 187.80 last 19.37) - open/close 4016 (time 1780383820.65 total 198.71 last 22.92) - open/close 4235 (time 1780383830.83 total 208.89 last 21.51) - open/close 4477 (time 1780383841.33 total 219.40 last 23.04) - open/close 4677 (time 1780383852.07 total 230.13 last 18.63) - open/close 4916 (time 1780383862.50 total 240.56 last 22.92) - open/close 5126 (time 1780383873.33 total 251.39 last 19.39) - open/close 5293 (time 1780383883.39 total 261.45 last 16.59) - open/close 5562 (time 1780383893.41 total 271.47 last 26.84) - open/close 5837 (time 1780383904.24 total 282.30 last 25.40) - open/close 5951 (time 1780383916.15 total 294.21 last 9.58) - open/close 6137 (time 1780383927.12 total 305.18 last 16.95) - open/close 6367 (time 1780383937.12 total 315.18 last 22.98) - open/close 6602 (time 1780383947.17 total 325.23 last 23.40) - open/close 6827 (time 1780383957.19 total 335.25 last 22.45) - open/close 6939 (time 1780383967.93 total 345.99 last 10.43) - open/close 7176 (time 1780383978.69 total 356.75 last 22.03) - open/close 7407 (time 1780383989.58 total 367.64 last 21.21) - open/close 7635 (time 1780384000.39 total 378.45 last 21.09) - open/close 7867 (time 1780384010.41 total 388.48 last 23.14) - open/close 8085 (time 1780384020.42 total 398.49 last 21.78) - open/close 8366 (time 1780384030.57 total 408.63 last 27.70) - open/close 8621 (time 1780384040.58 total 418.64 last 25.48) - open/close 8768 (time 1780384051.31 total 429.37 last 13.70) - open/close 8857 (time 1780384062.05 total 440.11 last 8.29) - open/close 8956 (time 1780384072.81 total 450.87 last 9.20) - open/close 9156 (time 1780384083.64 total 461.70 last 18.47) - open/close 9440 (time 1780384093.73 total 471.80 last 28.13) - open/close 9631 (time 1780384103.74 total 481.81 last 19.08) - open/close 9838 (time 1780384113.83 total 491.89 last 20.52) - open/close 9977 (time 1780384124.02 total 502.08 last 13.65) - open/close 10130 (time 1780384134.25 total 512.31 last 14.95) - open/close 10214 (time 1780384144.63 total 522.69 last 8.09) - open/close 10455 (time 1780384155.19 total 533.25 last 22.82) - open/close 10723 (time 1780384165.19 total 543.25 last 26.79) - open/close 10950 (time 1780384175.27 total 553.34 last 22.52) - open/close 11180 (time 1780384185.84 total 563.90 last 21.76) - open/close 11450 (time 1780384196.65 total 574.72 last 24.97) - open/close 11711 (time 1780384206.66 total 584.72 last 26.08) - open/close 11908 (time 1780384217.63 total 595.69 last 17.96) - open/close 12072 (time 1780384227.95 total 606.01 last 15.89) - open/close 12302 (time 1780384238.70 total 616.76 last 21.39) - open/close 12520 (time 1780384248.90 total 626.96 last 21.37) - open/close 12792 (time 1780384258.90 total 636.96 last 27.20) - open/close 13025 (time 1780384269.61 total 647.67 last 21.75) - open/close 13192 (time 1780384279.62 total 657.69 last 16.68) - open/close 13419 (time 1780384290.03 total 668.09 last 21.82) - open/close 13612 (time 1780384300.05 total 678.12 last 19.25) - open/close 13762 (time 1780384310.07 total 688.13 last 14.98) - open/close 14029 (time 1780384321.01 total 699.07 last 24.41) - open/close 14284 (time 1780384331.59 total 709.66 last 24.09) - open/close 14487 (time 1780384342.24 total 720.30 last 19.07) - open/close 14713 (time 1780384352.49 total 730.56 last 22.04) - open/close 14931 (time 1780384362.52 total 740.58 last 21.75) - open/close 15026 (time 1780384372.84 total 750.90 last 9.20) - open/close 15259 (time 1780384382.88 total 760.94 last 23.21) - open/close 15496 (time 1780384392.88 total 770.94 last 23.70) - open/close 15714 (time 1780384403.02 total 781.08 last 21.50) - open/close 15927 (time 1780384413.37 total 791.44 last 20.56) - open/close 16060 (time 1780384423.38 total 801.44 last 13.29) - open/close 16323 (time 1780384433.40 total 811.46 last 26.25) - open/close 16535 (time 1780384443.87 total 821.93 last 20.24) - open/close 16783 (time 1780384453.89 total 831.95 last 24.77) - open/close 16930 (time 1780384463.90 total 841.96 last 14.69) - open/close 17163 (time 1780384473.90 total 851.96 last 23.29) - open/close 17342 (time 1780384484.84 total 862.90 last 16.36) - open/close 17497 (time 1780384495.53 total 873.59 last 14.50) - open/close 17724 (time 1780384505.91 total 883.97 last 21.87) - open/close 17977 (time 1780384516.97 total 895.04 last 22.87) - open/close 18264 (time 1780384527.26 total 905.32 last 27.90) - open/close 18505 (time 1780384537.27 total 915.34 last 24.07) - open/close 18723 (time 1780384547.29 total 925.35 last 21.76) - open/close 18942 (time 1780384557.67 total 935.73 last 21.10) - open/close 19373 (time 1780384567.67 total 945.74 last 43.08) - open/close 20000 (time 1780384575.31 total 953.37 last 82.09) - open/close 20854 (time 1780384585.33 total 963.40 last 85.21) - open/close 21752 (time 1780384595.34 total 973.40 last 89.78) - open/close 22664 (time 1780384605.34 total 983.41 last 91.12) - open/close 23525 (time 1780384615.35 total 993.41 last 86.06) - open/close 24389 (time 1780384625.35 total 1003.42 last 86.36) - open/close 25248 (time 1780384635.37 total 1013.43 last 85.80) - open/close 26094 (time 1780384645.37 total 1023.43 last 84.55) - open/close 26994 (time 1780384655.38 total 1033.44 last 89.93) - open/close 27862 (time 1780384665.38 total 1043.44 last 86.78) - open/close 28695 (time 1780384675.38 total 1053.44 last 83.29) - open/close 29577 (time 1780384685.39 total 1063.45 last 88.16) - open/close 30000 (time 1780384692.40 total 1070.46 last 60.31) - open/close 30609 (time 1780384702.41 total 1080.47 last 60.84) - open/close 31475 (time 1780384712.41 total 1090.48 last 86.59) - open/close 32369 (time 1780384722.42 total 1100.48 last 89.39) - open/close 33218 (time 1780384732.42 total 1110.48 last 84.87) - open/close 34109 (time 1780384742.43 total 1120.49 last 89.04) - open/close 35059 (time 1780384752.43 total 1130.49 last 94.97) - open/close 36003 (time 1780384762.43 total 1140.50 last 94.34) - open/close 36858 (time 1780384772.44 total 1150.50 last 85.48) - open/close 37778 (time 1780384782.44 total 1160.50 last 91.96) - open/close 38590 (time 1780384792.44 total 1170.50 last 81.20) - open/close 39531 (time 1780384802.45 total 1180.51 last 94.05) - open/close 40000 (time 1780384807.38 total 1185.44 last 95.09) - open/close 40950 (time 1780384817.38 total 1195.44 last 94.98) - open/close 41895 (time 1780384827.42 total 1205.48 last 94.15) - open/close 42749 (time 1780384837.42 total 1215.48 last 85.40) - open/close 43597 (time 1780384847.42 total 1225.48 last 84.78) - open/close 44452 (time 1780384857.44 total 1235.50 last 85.32) - open/close 45285 (time 1780384867.45 total 1245.51 last 83.25) - open/close 46169 (time 1780384877.46 total 1255.52 last 88.28) - open/close 46970 (time 1780384887.46 total 1265.52 last 80.08) - open/close 47911 (time 1780384897.46 total 1275.52 last 94.09) - open/close 48948 (time 1780384907.47 total 1285.53 last 103.66) total: 50000 open/close in 1295.41 seconds: 38.60 ops/second - unlinked 0 (time 1780384920 ; total 0 ; last 0) - unlinked 10000 (time 1780385033 ; total 113 ; last 113) - unlinked 20000 (time 1780385133 ; total 213 ; last 100) - unlinked 30000 (time 1780385197 ; total 277 ; last 64) - unlinked 40000 (time 1780385262 ; total 342 ; last 65) total: 50000 unlinks in 402 seconds: 124.378113 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2