== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:37:46 (1787387866) start mds service on oleg641-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing load_modules_local oleg641-server: Loading modules from /home/green/git/lustre-release/lustre oleg641-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg641-server: detected 4 online CPUs by sysfs oleg641-server: MODOPTS_LIBCFS= oleg641-server: Force libcfs to create 2 CPU partitions oleg641-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg641-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg641-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg641-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 247 (time 1787387945.63 total 10.38 last 23.79) - open/close 530 (time 1787387956.53 total 21.28 last 25.98) - open/close 806 (time 1787387966.54 total 31.29 last 27.57) - open/close 1041 (time 1787387976.61 total 41.36 last 23.33) - open/close 1279 (time 1787387986.61 total 51.36 last 23.80) - open/close 1534 (time 1787387996.63 total 61.38 last 25.46) - open/close 1728 (time 1787388006.63 total 71.38 last 19.39) - open/close 1825 (time 1787388017.71 total 82.46 last 8.75) - open/close 2008 (time 1787388027.72 total 92.47 last 18.29) - open/close 2232 (time 1787388038.19 total 102.94 last 21.40) - open/close 2510 (time 1787388049.07 total 113.82 last 25.55) - open/close 2677 (time 1787388059.08 total 123.83 last 16.68) - open/close 2834 (time 1787388069.11 total 133.86 last 15.67) - open/close 3030 (time 1787388079.14 total 143.89 last 19.53) - open/close 3185 (time 1787388089.52 total 154.27 last 14.93) - open/close 3389 (time 1787388099.55 total 164.30 last 20.33) - open/close 3534 (time 1787388110.18 total 174.93 last 13.64) - open/close 3721 (time 1787388121.66 total 186.41 last 16.29) - open/close 3926 (time 1787388131.70 total 196.45 last 20.43) - open/close 4071 (time 1787388142.05 total 206.80 last 14.00) - open/close 4267 (time 1787388154.92 total 219.67 last 15.23) - open/close 4456 (time 1787388165.46 total 230.21 last 17.92) - open/close 4715 (time 1787388176.23 total 240.98 last 24.06) - open/close 4927 (time 1787388186.24 total 250.99 last 21.17) - open/close 5158 (time 1787388196.27 total 261.02 last 23.05) - open/close 5304 (time 1787388206.62 total 271.37 last 14.10) - open/close 5404 (time 1787388218.54 total 283.29 last 8.39) - open/close 5564 (time 1787388229.55 total 294.30 last 14.54) - open/close 5715 (time 1787388240.75 total 305.50 last 13.48) - open/close 5978 (time 1787388250.76 total 315.51 last 26.28) - open/close 6122 (time 1787388260.89 total 325.64 last 14.21) - open/close 6340 (time 1787388271.20 total 335.95 last 21.15) - open/close 6597 (time 1787388282.27 total 347.02 last 23.22) - open/close 6838 (time 1787388293.21 total 357.96 last 22.03) - open/close 7053 (time 1787388303.34 total 368.09 last 21.24) - open/close 7295 (time 1787388313.62 total 378.37 last 23.53) - open/close 7531 (time 1787388323.63 total 388.38 last 23.58) - open/close 7782 (time 1787388334.58 total 399.33 last 22.93) - open/close 8049 (time 1787388344.80 total 409.55 last 26.12) - open/close 8296 (time 1787388354.80 total 419.55 last 24.69) - open/close 8339 (time 1787388365.08 total 429.83 last 4.18) - open/close 8545 (time 1787388375.21 total 439.96 last 20.33) - open/close 8668 (time 1787388385.70 total 450.45 last 11.72) - open/close 8857 (time 1787388395.84 total 460.59 last 18.64) - open/close 8934 (time 1787388407.55 total 472.30 last 6.58) - open/close 9225 (time 1787388418.21 total 482.96 last 27.31) - open/close 9450 (time 1787388429.10 total 493.85 last 20.66) - open/close 9729 (time 1787388439.12 total 503.87 last 27.85) - open/close 9952 (time 1787388449.50 total 514.25 last 21.47) - open/close 10206 (time 1787388459.68 total 524.43 last 24.97) - open/close 10430 (time 1787388469.68 total 534.43 last 22.39) - open/close 10634 (time 1787388479.72 total 544.47 last 20.32) - open/close 10824 (time 1787388489.72 total 554.47 last 18.99) - open/close 11102 (time 1787388500.51 total 565.26 last 25.77) - open/close 11333 (time 1787388510.52 total 575.27 last 23.08) - open/close 11517 (time 1787388520.52 total 585.27 last 18.40) - open/close 11641 (time 1787388530.85 total 595.60 last 12.01) - open/close 11850 (time 1787388540.86 total 605.61 last 20.88) - open/close 12072 (time 1787388551.33 total 616.08 last 21.20) - open/close 12335 (time 1787388561.39 total 626.14 last 26.14) - open/close 12564 (time 1787388571.40 total 636.15 last 22.88) - open/close 12782 (time 1787388581.73 total 646.48 last 21.10) - open/close 13032 (time 1787388592.38 total 657.13 last 23.47) - open/close 13328 (time 1787388602.85 total 667.60 last 28.28) - open/close 13499 (time 1787388613.41 total 678.16 last 16.19) - open/close 13737 (time 1787388623.91 total 688.66 last 22.66) - open/close 14012 (time 1787388633.92 total 698.67 last 27.48) - open/close 14240 (time 1787388644.06 total 708.81 last 22.48) - open/close 14458 (time 1787388654.07 total 718.82 last 21.78) - open/close 14667 (time 1787388664.11 total 728.86 last 20.82) - open/close 14900 (time 1787388674.11 total 738.87 last 23.29) - open/close 15135 (time 1787388685.04 total 749.79 last 21.52) - open/close 15249 (time 1787388695.72 total 760.47 last 10.67) - open/close 15489 (time 1787388705.73 total 770.48 last 23.97) - open/close 15700 (time 1787388716.80 total 781.55 last 19.07) - open/close 15944 (time 1787388727.60 total 792.35 last 22.57) - open/close 16189 (time 1787388738.22 total 802.97 last 23.09) - open/close 16415 (time 1787388748.22 total 812.97 last 22.58) - open/close 16609 (time 1787388758.26 total 823.01 last 19.33) - open/close 16841 (time 1787388768.67 total 833.42 last 22.28) - open/close 17073 (time 1787388778.72 total 843.47 last 23.08) - open/close 17258 (time 1787388789.62 total 854.37 last 16.98) - open/close 17432 (time 1787388800.35 total 865.10 last 16.21) - open/close 17688 (time 1787388810.99 total 875.74 last 24.07) - open/close 17898 (time 1787388821.03 total 885.78 last 20.91) - open/close 18170 (time 1787388831.04 total 895.79 last 27.19) - open/close 18399 (time 1787388841.04 total 905.79 last 22.89) - open/close 18617 (time 1787388851.04 total 915.79 last 21.79) - open/close 18871 (time 1787388861.41 total 926.16 last 24.50) - open/close 19085 (time 1787388871.53 total 936.28 last 21.15) - open/close 19274 (time 1787388881.54 total 946.29 last 18.88) - open/close 19861 (time 1787388891.55 total 956.30 last 58.65) - open/close 20000 (time 1787388893.67 total 958.42 last 65.67) - open/close 20760 (time 1787388903.68 total 968.43 last 75.93) - open/close 21479 (time 1787388913.70 total 978.46 last 71.70) - open/close 22253 (time 1787388923.73 total 988.48 last 77.20) - open/close 23015 (time 1787388933.74 total 998.49 last 76.13) - open/close 23760 (time 1787388943.74 total 1008.49 last 74.49) - open/close 24504 (time 1787388953.75 total 1018.50 last 74.32) - open/close 25192 (time 1787388963.76 total 1028.51 last 68.72) - open/close 26004 (time 1787388973.77 total 1038.52 last 81.14) - open/close 26752 (time 1787388983.78 total 1048.53 last 74.75) - open/close 27440 (time 1787388993.78 total 1058.53 last 68.77) - open/close 28204 (time 1787389003.79 total 1068.54 last 76.35) - open/close 28889 (time 1787389013.79 total 1078.54 last 68.47) - open/close 29555 (time 1787389023.79 total 1088.54 last 66.59) - open/close 30000 (time 1787389030.12 total 1094.87 last 70.37) - open/close 30636 (time 1787389040.12 total 1104.87 last 63.56) - open/close 31291 (time 1787389050.14 total 1114.89 last 65.41) - open/close 32015 (time 1787389060.17 total 1124.92 last 72.17) - open/close 32719 (time 1787389070.18 total 1134.93 last 70.35) - open/close 33403 (time 1787389080.19 total 1144.94 last 68.32) - open/close 34172 (time 1787389090.20 total 1154.95 last 76.85) - open/close 34952 (time 1787389100.20 total 1164.95 last 78.00) - open/close 35675 (time 1787389110.20 total 1174.95 last 72.27) - open/close 36381 (time 1787389120.23 total 1184.98 last 70.38) - open/close 37078 (time 1787389130.25 total 1195.00 last 69.58) - open/close 37748 (time 1787389140.25 total 1205.00 last 67.00) - open/close 38464 (time 1787389150.26 total 1215.01 last 71.50) - open/close 39239 (time 1787389160.27 total 1225.02 last 77.42) - open/close 39985 (time 1787389170.28 total 1235.03 last 74.52) - open/close 40686 (time 1787389180.28 total 1245.03 last 70.10) - open/close 41377 (time 1787389190.30 total 1255.05 last 68.99) - open/close 42076 (time 1787389200.32 total 1265.07 last 69.77) - open/close 42656 (time 1787389210.32 total 1275.07 last 57.99) - open/close 43258 (time 1787389220.33 total 1285.08 last 60.14) - open/close 43855 (time 1787389230.34 total 1295.09 last 59.61) - open/close 44583 (time 1787389240.34 total 1305.09 last 72.80) - open/close 45271 (time 1787389250.35 total 1315.10 last 68.76) - open/close 45953 (time 1787389260.35 total 1325.10 last 68.19) - open/close 46640 (time 1787389270.36 total 1335.11 last 68.64) - open/close 47407 (time 1787389280.37 total 1345.12 last 76.63) - open/close 48135 (time 1787389290.38 total 1355.13 last 72.73) - open/close 48885 (time 1787389300.38 total 1365.13 last 75.00) - open/close 49627 (time 1787389310.39 total 1375.14 last 74.16) total: 50000 open/close in 1379.90 seconds: 36.23 ops/second - unlinked 0 (time 1787389318 ; total 0 ; last 0) - unlinked 10000 (time 1787389466 ; total 148 ; last 148) - unlinked 20000 (time 1787389618 ; total 300 ; last 152) - unlinked 30000 (time 1787389759 ; total 441 ; last 141) - unlinked 40000 (time 1787389906 ; total 588 ; last 147) total: 50000 unlinks in 736 seconds: 67.934784 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg641-server unloading modules via unload_modules_local on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing unload_modules_local oleg641-server: modules unloaded. pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2