== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:35:04 (1779701704) start mds service on oleg313-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg313-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 91 (time 1779701782.91 total 10.01 last 9.09) - open/close 261 (time 1779701792.93 total 20.03 last 16.97) - open/close 485 (time 1779701803.52 total 30.62 last 21.14) - open/close 730 (time 1779701814.02 total 41.12 last 23.33) - open/close 958 (time 1779701824.84 total 51.94 last 21.07) - open/close 1217 (time 1779701835.47 total 62.57 last 24.37) - open/close 1471 (time 1779701845.96 total 73.05 last 24.22) - open/close 1703 (time 1779701856.64 total 83.74 last 21.71) - open/close 1909 (time 1779701866.65 total 93.75 last 20.58) - open/close 2127 (time 1779701876.80 total 103.90 last 21.47) - open/close 2293 (time 1779701887.81 total 114.91 last 15.08) - open/close 2448 (time 1779701898.19 total 125.29 last 14.93) - open/close 2606 (time 1779701908.58 total 135.68 last 15.21) - open/close 2829 (time 1779701918.59 total 145.69 last 22.29) - open/close 3069 (time 1779701928.60 total 155.70 last 23.96) - open/close 3256 (time 1779701938.96 total 166.06 last 18.06) - open/close 3367 (time 1779701948.96 total 176.06 last 11.10) - open/close 3552 (time 1779701961.47 total 188.57 last 14.79) - open/close 3793 (time 1779701971.48 total 198.58 last 24.08) - open/close 3933 (time 1779701981.49 total 208.59 last 13.98) - open/close 4146 (time 1779701991.56 total 218.66 last 21.16) - open/close 4393 (time 1779702002.50 total 229.60 last 22.59) - open/close 4642 (time 1779702012.69 total 239.79 last 24.44) - open/close 4784 (time 1779702023.18 total 250.28 last 13.53) - open/close 4923 (time 1779702033.93 total 261.03 last 12.93) - open/close 5120 (time 1779702043.93 total 271.03 last 19.69) - open/close 5302 (time 1779702056.72 total 283.81 last 14.24) - open/close 5525 (time 1779702067.34 total 294.44 last 21.00) - open/close 5735 (time 1779702077.60 total 304.70 last 20.46) - open/close 5921 (time 1779702087.60 total 314.70 last 18.59) - open/close 6150 (time 1779702098.58 total 325.68 last 20.86) - open/close 6413 (time 1779702108.62 total 335.72 last 26.18) - open/close 6590 (time 1779702118.73 total 345.83 last 17.51) - open/close 6686 (time 1779702128.99 total 356.09 last 9.36) - open/close 6781 (time 1779702139.02 total 366.12 last 9.48) - open/close 6853 (time 1779702149.21 total 376.30 last 7.07) - open/close 6958 (time 1779702159.22 total 386.32 last 10.48) - open/close 7171 (time 1779702169.80 total 396.89 last 20.15) - open/close 7361 (time 1779702179.80 total 406.90 last 18.99) - open/close 7599 (time 1779702190.48 total 417.58 last 22.28) - open/close 7747 (time 1779702200.80 total 427.90 last 14.34) - open/close 7972 (time 1779702210.82 total 437.92 last 22.45) - open/close 8104 (time 1779702221.73 total 448.83 last 12.10) - open/close 8292 (time 1779702233.73 total 460.82 last 15.68) - open/close 8550 (time 1779702244.80 total 471.90 last 23.30) - open/close 8693 (time 1779702254.87 total 481.97 last 14.20) - open/close 8810 (time 1779702265.53 total 492.63 last 10.97) - open/close 8971 (time 1779702275.54 total 502.64 last 16.09) - open/close 9137 (time 1779702286.60 total 513.70 last 15.00) - open/close 9284 (time 1779702297.42 total 524.52 last 13.59) - open/close 9517 (time 1779702307.82 total 534.92 last 22.41) - open/close 9628 (time 1779702317.96 total 545.06 last 10.94) - open/close 9819 (time 1779702328.64 total 555.74 last 17.88) - open/close 9896 (time 1779702341.06 total 568.16 last 6.20) - open/close 10000 (time 1779702351.00 total 578.10 last 10.46) - open/close 10245 (time 1779702361.01 total 588.11 last 24.49) - open/close 10482 (time 1779702371.67 total 598.77 last 22.23) - open/close 10778 (time 1779702381.67 total 608.77 last 29.60) - open/close 10963 (time 1779702391.69 total 618.79 last 18.47) - open/close 11180 (time 1779702402.20 total 629.30 last 20.64) - open/close 11429 (time 1779702412.21 total 639.31 last 24.89) - open/close 11601 (time 1779702422.73 total 649.83 last 16.34) - open/close 11749 (time 1779702433.48 total 660.58 last 13.77) - open/close 11858 (time 1779702443.50 total 670.60 last 10.88) - open/close 12030 (time 1779702453.50 total 680.60 last 17.19) - open/close 12188 (time 1779702464.20 total 691.30 last 14.77) - open/close 12317 (time 1779702474.21 total 701.31 last 12.88) - open/close 12560 (time 1779702485.13 total 712.23 last 22.26) - open/close 12810 (time 1779702495.13 total 722.23 last 24.99) - open/close 13012 (time 1779702505.69 total 732.79 last 19.14) - open/close 13161 (time 1779702515.72 total 742.82 last 14.85) - open/close 13283 (time 1779702525.72 total 752.82 last 12.20) - open/close 13510 (time 1779702536.47 total 763.57 last 21.12) - open/close 13748 (time 1779702546.47 total 773.57 last 23.79) - open/close 13972 (time 1779702557.38 total 784.48 last 20.54) - open/close 14039 (time 1779702567.39 total 794.49 last 6.69) - open/close 14169 (time 1779702578.02 total 805.12 last 12.23) - open/close 14264 (time 1779702588.03 total 815.13 last 9.49) - open/close 14434 (time 1779702598.06 total 825.16 last 16.95) - open/close 14621 (time 1779702608.20 total 835.30 last 18.45) - open/close 14796 (time 1779702618.21 total 845.31 last 17.47) - open/close 15033 (time 1779702629.28 total 856.38 last 21.42) - open/close 15213 (time 1779702640.34 total 867.44 last 16.26) - open/close 15389 (time 1779702651.08 total 878.18 last 16.39) - open/close 15513 (time 1779702666.83 total 893.93 last 7.87) - open/close 15739 (time 1779702677.19 total 904.29 last 21.81) - open/close 15914 (time 1779702687.35 total 914.45 last 17.23) - open/close 16149 (time 1779702697.37 total 924.47 last 23.46) - open/close 16352 (time 1779702707.38 total 934.48 last 20.28) - open/close 16583 (time 1779702717.39 total 944.49 last 23.06) - open/close 16814 (time 1779702727.40 total 954.50 last 23.09) - open/close 17018 (time 1779702738.37 total 965.47 last 18.59) - open/close 17193 (time 1779702748.88 total 975.98 last 16.66) - open/close 17397 (time 1779702759.89 total 986.99 last 18.52) - open/close 17617 (time 1779702770.77 total 997.87 last 20.22) - open/close 18235 (time 1779702780.78 total 1007.88 last 61.73) - open/close 18853 (time 1779702790.79 total 1017.89 last 61.77) - open/close 19533 (time 1779702800.79 total 1027.89 last 67.96) - open/close 20000 (time 1779702807.74 total 1034.84 last 67.20) - open/close 20700 (time 1779702817.75 total 1044.85 last 69.96) - open/close 21416 (time 1779702827.75 total 1054.85 last 71.60) - open/close 22076 (time 1779702837.77 total 1064.87 last 65.86) - open/close 22802 (time 1779702847.78 total 1074.88 last 72.53) - open/close 23453 (time 1779702857.78 total 1084.88 last 65.09) - open/close 24170 (time 1779702867.78 total 1094.88 last 71.67) - open/close 24848 (time 1779702877.79 total 1104.89 last 67.76) - open/close 25540 (time 1779702887.81 total 1114.90 last 69.09) - open/close 26237 (time 1779702897.82 total 1124.92 last 69.57) - open/close 26926 (time 1779702907.84 total 1134.94 last 68.79) - open/close 27629 (time 1779702917.85 total 1144.95 last 70.23) - open/close 28341 (time 1779702927.86 total 1154.96 last 71.09) - open/close 29036 (time 1779702937.89 total 1164.99 last 69.30) - open/close 29659 (time 1779702947.89 total 1174.99 last 62.29) - open/close 30000 (time 1779702953.62 total 1180.72 last 59.57) - open/close 30674 (time 1779702963.62 total 1190.72 last 67.40) - open/close 31374 (time 1779702973.62 total 1200.72 last 70.00) - open/close 32042 (time 1779702983.63 total 1210.73 last 66.75) - open/close 32616 (time 1779702993.63 total 1220.73 last 57.37) - open/close 33234 (time 1779703003.64 total 1230.74 last 61.73) - open/close 33890 (time 1779703013.64 total 1240.74 last 65.59) - open/close 34446 (time 1779703023.65 total 1250.75 last 55.56) - open/close 35084 (time 1779703033.66 total 1260.76 last 63.77) - open/close 35747 (time 1779703043.66 total 1270.76 last 66.29) - open/close 36403 (time 1779703053.67 total 1280.76 last 65.55) - open/close 36921 (time 1779703063.68 total 1290.78 last 51.72) - open/close 37461 (time 1779703073.68 total 1300.78 last 53.99) - open/close 38125 (time 1779703083.70 total 1310.80 last 66.30) - open/close 38826 (time 1779703093.72 total 1320.82 last 69.94) - open/close 39470 (time 1779703103.73 total 1330.82 last 64.37) - open/close 40000 (time 1779703112.49 total 1339.59 last 60.45) - open/close 40624 (time 1779703122.50 total 1349.60 last 62.34) - open/close 41288 (time 1779703132.51 total 1359.61 last 66.37) - open/close 41908 (time 1779703142.52 total 1369.62 last 61.93) - open/close 42482 (time 1779703152.53 total 1379.63 last 57.31) - open/close 43103 (time 1779703162.56 total 1389.65 last 61.97) - open/close 43680 (time 1779703172.56 total 1399.66 last 57.68) - open/close 44248 (time 1779703182.56 total 1409.66 last 56.78) - open/close 44835 (time 1779703192.57 total 1419.67 last 58.63) - open/close 45489 (time 1779703202.57 total 1429.67 last 65.39) - open/close 46091 (time 1779703212.58 total 1439.68 last 60.15) - open/close 46673 (time 1779703222.60 total 1449.70 last 58.12) - open/close 47250 (time 1779703232.60 total 1459.70 last 57.66) - open/close 47744 (time 1779703242.62 total 1469.72 last 49.34) - open/close 48314 (time 1779703252.62 total 1479.72 last 56.99) - open/close 48918 (time 1779703262.62 total 1489.72 last 60.38) - open/close 49507 (time 1779703272.63 total 1499.73 last 58.84) total: 50000 open/close in 1508.79 seconds: 33.14 ops/second - unlinked 0 (time 1779703287 ; total 0 ; last 0) - unlinked 10000 (time 1779703458 ; total 171 ; last 171) - unlinked 20000 (time 1779703632 ; total 345 ; last 174) - unlinked 30000 (time 1779703801 ; total 514 ; last 169) - unlinked 40000 (time 1779703964 ; total 677 ; last 163) total: 50000 unlinks in 835 seconds: 59.880241 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules via unload_modules_local on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local oleg313-server: modules unloaded. pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2