== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:20:03 (1787138403) start mds service on oleg139-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 89 (time 1787138494.08 total 10.56 last 8.43) - open/close 297 (time 1787138504.11 total 20.58 last 20.74) - open/close 366 (time 1787138514.11 total 30.59 last 6.90) - open/close 482 (time 1787138524.23 total 40.71 last 11.47) - open/close 649 (time 1787138534.24 total 50.72 last 16.68) - open/close 712 (time 1787138544.25 total 60.72 last 6.30) - open/close 870 (time 1787138556.14 total 72.61 last 13.29) - open/close 1033 (time 1787138567.07 total 83.55 last 14.91) - open/close 1137 (time 1787138577.08 total 93.56 last 10.39) - open/close 1243 (time 1787138587.94 total 104.41 last 9.77) - open/close 1475 (time 1787138597.96 total 114.43 last 23.15) - open/close 1564 (time 1787138608.81 total 125.29 last 8.20) - open/close 1671 (time 1787138618.82 total 135.30 last 10.69) - open/close 1796 (time 1787138628.82 total 145.30 last 12.50) - open/close 1967 (time 1787138638.83 total 155.31 last 17.09) - open/close 2183 (time 1787138649.50 total 165.98 last 20.24) - open/close 2323 (time 1787138659.61 total 176.08 last 13.85) - open/close 2514 (time 1787138672.23 total 188.71 last 15.13) - open/close 2724 (time 1787138683.06 total 199.54 last 19.39) - open/close 2842 (time 1787138693.74 total 210.22 last 11.05) - open/close 2913 (time 1787138703.98 total 220.45 last 6.94) - open/close 3066 (time 1787138714.00 total 230.47 last 15.27) - open/close 3231 (time 1787138724.51 total 240.99 last 15.69) - open/close 3304 (time 1787138735.11 total 251.59 last 6.89) - open/close 3357 (time 1787138745.20 total 261.68 last 5.26) - open/close 3444 (time 1787138755.21 total 271.69 last 8.69) - open/close 3512 (time 1787138766.63 total 283.10 last 5.96) - open/close 3573 (time 1787138776.68 total 293.15 last 6.07) - open/close 3794 (time 1787138786.70 total 303.18 last 22.05) - open/close 3896 (time 1787138796.79 total 313.27 last 10.11) - open/close 4044 (time 1787138806.79 total 323.27 last 14.80) - open/close 4204 (time 1787138816.80 total 333.27 last 16.00) - open/close 4354 (time 1787138827.83 total 344.30 last 13.60) - open/close 4524 (time 1787138838.91 total 355.39 last 15.34) - open/close 4711 (time 1787138849.20 total 365.68 last 18.16) - open/close 4865 (time 1787138859.23 total 375.70 last 15.36) - open/close 5012 (time 1787138869.80 total 386.28 last 13.90) - open/close 5092 (time 1787138879.82 total 396.30 last 7.98) - open/close 5237 (time 1787138890.42 total 406.90 last 13.68) - open/close 5354 (time 1787138903.50 total 419.98 last 8.94) - open/close 5548 (time 1787138914.17 total 430.65 last 18.18) - open/close 5752 (time 1787138925.23 total 441.70 last 18.45) - open/close 5873 (time 1787138935.24 total 451.71 last 12.09) - open/close 5995 (time 1787138945.31 total 461.79 last 12.11) - open/close 6087 (time 1787138955.77 total 472.25 last 8.79) - open/close 6252 (time 1787138965.78 total 482.26 last 16.49) - open/close 6385 (time 1787138975.79 total 492.26 last 13.29) - open/close 6528 (time 1787138986.62 total 503.10 last 13.20) - open/close 6697 (time 1787138996.74 total 513.22 last 16.70) - open/close 6824 (time 1787139007.47 total 523.95 last 11.83) - open/close 6930 (time 1787139018.23 total 534.71 last 9.85) - open/close 7089 (time 1787139028.87 total 545.34 last 14.95) - open/close 7216 (time 1787139038.91 total 555.39 last 12.64) - open/close 7379 (time 1787139051.01 total 567.49 last 13.47) - open/close 7549 (time 1787139061.40 total 577.87 last 16.37) - open/close 7721 (time 1787139071.85 total 588.33 last 16.45) - open/close 7840 (time 1787139082.47 total 598.95 last 11.21) - open/close 7950 (time 1787139092.72 total 609.20 last 10.73) - open/close 8056 (time 1787139105.86 total 622.34 last 8.07) - open/close 8142 (time 1787139116.70 total 633.18 last 7.93) - open/close 8245 (time 1787139127.07 total 643.54 last 9.94) - open/close 8336 (time 1787139137.72 total 654.20 last 8.54) - open/close 8433 (time 1787139147.73 total 664.20 last 9.69) - open/close 8597 (time 1787139158.78 total 675.25 last 14.84) - open/close 8629 (time 1787139169.09 total 685.57 last 3.10) - open/close 8719 (time 1787139180.17 total 696.65 last 8.12) - open/close 8806 (time 1787139190.20 total 706.68 last 8.68) - open/close 8887 (time 1787139201.14 total 717.62 last 7.40) - open/close 8983 (time 1787139211.82 total 728.30 last 8.99) - open/close 9062 (time 1787139222.25 total 738.73 last 7.58) - open/close 9189 (time 1787139232.95 total 749.42 last 11.87) - open/close 9250 (time 1787139242.97 total 759.44 last 6.09) - open/close 9385 (time 1787139252.98 total 769.45 last 13.49) - open/close 9521 (time 1787139263.60 total 780.07 last 12.81) - open/close 9616 (time 1787139274.35 total 790.82 last 8.84) - open/close 9765 (time 1787139284.37 total 800.84 last 14.87) - open/close 9894 (time 1787139294.64 total 811.12 last 12.56) - open/close 10000 (time 1787139299.27 total 815.74 last 22.91) - open/close 10083 (time 1787139309.55 total 826.03 last 8.07) - open/close 10237 (time 1787139320.30 total 836.77 last 14.34) - open/close 10400 (time 1787139330.49 total 846.96 last 15.99) - open/close 10505 (time 1787139340.50 total 856.98 last 10.49) - open/close 10563 (time 1787139350.63 total 867.11 last 5.72) - open/close 10652 (time 1787139361.19 total 877.67 last 8.43) - open/close 10808 (time 1787139371.21 total 887.69 last 15.57) - open/close 11032 (time 1787139384.23 total 900.71 last 17.21) - open/close 11260 (time 1787139394.27 total 910.75 last 22.71) - open/close 11471 (time 1787139406.44 total 922.92 last 17.33) - open/close 11586 (time 1787139417.07 total 933.55 last 10.82) - open/close 11697 (time 1787139427.09 total 943.57 last 11.08) - open/close 11875 (time 1787139437.62 total 954.09 last 16.91) - open/close 11983 (time 1787139447.63 total 964.11 last 10.78) - open/close 12072 (time 1787139458.92 total 975.40 last 7.88) - open/close 12171 (time 1787139469.16 total 985.63 last 9.68) - open/close 12330 (time 1787139479.47 total 995.95 last 15.42) - open/close 12512 (time 1787139490.17 total 1006.65 last 17.00) - open/close 12689 (time 1787139501.30 total 1017.78 last 15.91) - open/close 12892 (time 1787139512.23 total 1028.70 last 18.58) - open/close 13085 (time 1787139522.92 total 1039.40 last 18.04) - open/close 13224 (time 1787139533.88 total 1050.36 last 12.68) - open/close 13401 (time 1787139544.32 total 1060.80 last 16.95) - open/close 13507 (time 1787139554.74 total 1071.22 last 10.17) - open/close 13576 (time 1787139566.96 total 1083.44 last 5.65) - open/close 13713 (time 1787139578.00 total 1094.48 last 12.41) - open/close 13927 (time 1787139588.01 total 1104.49 last 21.38) - open/close 14145 (time 1787139598.96 total 1115.44 last 19.90) - open/close 14328 (time 1787139608.96 total 1125.44 last 18.29) - open/close 14961 (time 1787139618.98 total 1135.46 last 63.18) - open/close 15567 (time 1787139629.00 total 1145.47 last 60.52) - open/close 16337 (time 1787139639.00 total 1155.48 last 76.96) - open/close 16993 (time 1787139649.01 total 1165.48 last 65.57) - open/close 17649 (time 1787139659.06 total 1175.54 last 65.22) - open/close 18189 (time 1787139669.08 total 1185.55 last 53.93) - open/close 18765 (time 1787139679.08 total 1195.56 last 57.59) - open/close 19471 (time 1787139689.08 total 1205.56 last 70.58) - open/close 20000 (time 1787139695.98 total 1212.46 last 76.64) - open/close 20654 (time 1787139705.99 total 1222.47 last 65.36) - open/close 21265 (time 1787139716.01 total 1232.49 last 60.97) - open/close 21961 (time 1787139726.02 total 1242.49 last 69.59) - open/close 22750 (time 1787139736.03 total 1252.50 last 78.82) - open/close 23303 (time 1787139746.03 total 1262.50 last 55.30) - open/close 23948 (time 1787139756.03 total 1272.51 last 64.44) - open/close 24678 (time 1787139766.07 total 1282.54 last 72.77) - open/close 25356 (time 1787139776.07 total 1292.55 last 67.75) - open/close 25944 (time 1787139786.08 total 1302.56 last 58.74) - open/close 26501 (time 1787139796.09 total 1312.57 last 55.66) - open/close 27143 (time 1787139806.09 total 1322.57 last 64.18) - open/close 27728 (time 1787139816.12 total 1332.60 last 58.33) - open/close 28328 (time 1787139826.13 total 1342.60 last 59.97) - open/close 28988 (time 1787139836.14 total 1352.61 last 65.95) - open/close 29615 (time 1787139846.15 total 1362.63 last 62.58) - open/close 30000 (time 1787139853.69 total 1370.17 last 51.09) - open/close 30750 (time 1787139863.72 total 1380.19 last 74.81) - open/close 31469 (time 1787139873.72 total 1390.20 last 71.84) - open/close 32144 (time 1787139883.77 total 1400.24 last 67.21) - open/close 32729 (time 1787139893.77 total 1410.25 last 58.46) - open/close 33423 (time 1787139903.79 total 1420.27 last 69.27) - open/close 34134 (time 1787139913.80 total 1430.28 last 71.06) - open/close 34864 (time 1787139923.81 total 1440.29 last 72.91) - open/close 35498 (time 1787139933.81 total 1450.29 last 63.38) - open/close 36094 (time 1787139943.82 total 1460.30 last 59.55) - open/close 36625 (time 1787139953.83 total 1470.30 last 53.06) - open/close 37282 (time 1787139963.83 total 1480.31 last 65.66) - open/close 37891 (time 1787139973.83 total 1490.31 last 60.89) - open/close 38596 (time 1787139983.85 total 1500.33 last 70.40) - open/close 39197 (time 1787139993.86 total 1510.34 last 60.04) - open/close 39921 (time 1787140003.86 total 1520.34 last 72.38) - open/close 40591 (time 1787140013.87 total 1530.35 last 66.92) - open/close 41237 (time 1787140023.88 total 1540.36 last 64.55) - open/close 41920 (time 1787140033.89 total 1550.37 last 68.25) - open/close 42604 (time 1787140043.91 total 1560.38 last 68.27) - open/close 43195 (time 1787140053.91 total 1570.39 last 59.10) - open/close 43847 (time 1787140063.91 total 1580.39 last 65.18) - open/close 44473 (time 1787140073.92 total 1590.40 last 62.56) - open/close 45181 (time 1787140083.97 total 1600.45 last 70.45) - open/close 45843 (time 1787140093.98 total 1610.46 last 66.13) - open/close 46550 (time 1787140103.98 total 1620.46 last 70.69) - open/close 47205 (time 1787140113.99 total 1630.46 last 65.47) - open/close 47945 (time 1787140123.99 total 1640.47 last 73.93) - open/close 48513 (time 1787140134.00 total 1650.48 last 56.76) - open/close 49117 (time 1787140144.00 total 1660.48 last 60.38) - open/close 49753 (time 1787140154.01 total 1670.49 last 63.57) total: 50000 open/close in 1675.15 seconds: 29.85 ops/second - unlinked 0 (time 1787140164 ; total 0 ; last 0) - unlinked 10000 (time 1787140347 ; total 183 ; last 183) - unlinked 20000 (time 1787140526 ; total 362 ; last 179) - unlinked 30000 (time 1787140718 ; total 554 ; last 192) - unlinked 40000 (time 1787140898 ; total 734 ; last 180) total: 50000 unlinks in 904 seconds: 55.309734 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2