== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:11:53 (1781230313) start mds service on oleg407-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 185 (time 1781230400.91 total 10.02 last 18.47) - open/close 387 (time 1781230411.63 total 20.74 last 18.84) - open/close 537 (time 1781230422.36 total 31.46 last 13.98) - open/close 732 (time 1781230433.04 total 42.14 last 18.26) - open/close 917 (time 1781230443.10 total 52.20 last 18.39) - open/close 1059 (time 1781230453.97 total 63.08 last 13.06) - open/close 1271 (time 1781230464.91 total 74.02 last 19.37) - open/close 1387 (time 1781230475.02 total 84.13 last 11.48) - open/close 1514 (time 1781230485.65 total 94.75 last 11.95) - open/close 1622 (time 1781230497.92 total 107.03 last 8.80) - open/close 1741 (time 1781230508.11 total 117.21 last 11.69) - open/close 1847 (time 1781230518.12 total 127.22 last 10.59) - open/close 1986 (time 1781230528.84 total 137.95 last 12.96) - open/close 2154 (time 1781230539.02 total 148.13 last 16.51) - open/close 2284 (time 1781230549.03 total 158.14 last 12.99) - open/close 2347 (time 1781230559.18 total 168.28 last 6.21) - open/close 2477 (time 1781230569.57 total 178.67 last 12.51) - open/close 2581 (time 1781230579.72 total 188.82 last 10.25) - open/close 2716 (time 1781230590.02 total 199.13 last 13.11) - open/close 2876 (time 1781230600.29 total 209.40 last 15.58) - open/close 2996 (time 1781230610.56 total 219.67 last 11.68) - open/close 3152 (time 1781230621.45 total 230.56 last 14.32) - open/close 3334 (time 1781230631.77 total 240.88 last 17.64) - open/close 3386 (time 1781230641.78 total 250.89 last 5.20) - open/close 3474 (time 1781230652.23 total 261.34 last 8.42) - open/close 3564 (time 1781230662.80 total 271.91 last 8.51) - open/close 3706 (time 1781230672.82 total 281.93 last 14.17) - open/close 3960 (time 1781230683.76 total 292.86 last 23.23) - open/close 4226 (time 1781230693.98 total 303.09 last 26.01) - open/close 4395 (time 1781230704.08 total 313.18 last 16.74) - open/close 4616 (time 1781230714.84 total 323.95 last 20.53) - open/close 4738 (time 1781230726.91 total 336.02 last 10.10) - open/close 4834 (time 1781230739.66 total 348.77 last 7.53) - open/close 5036 (time 1781230750.42 total 359.53 last 18.77) - open/close 5154 (time 1781230760.44 total 369.55 last 11.78) - open/close 5362 (time 1781230771.27 total 380.38 last 19.21) - open/close 5594 (time 1781230782.29 total 391.39 last 21.06) - open/close 5813 (time 1781230792.88 total 401.99 last 20.66) - open/close 5979 (time 1781230803.84 total 412.95 last 15.15) - open/close 6166 (time 1781230815.01 total 424.12 last 16.75) - open/close 6332 (time 1781230825.03 total 434.14 last 16.56) - open/close 6521 (time 1781230835.28 total 444.39 last 18.44) - open/close 6679 (time 1781230848.27 total 457.38 last 12.16) - open/close 6874 (time 1781230858.63 total 467.74 last 18.82) - open/close 7081 (time 1781230869.18 total 478.29 last 19.62) - open/close 7286 (time 1781230879.62 total 488.73 last 19.64) - open/close 7509 (time 1781230890.33 total 499.44 last 20.83) - open/close 7731 (time 1781230901.35 total 510.46 last 20.15) - open/close 7904 (time 1781230911.36 total 520.47 last 17.28) - open/close 8094 (time 1781230921.37 total 530.48 last 18.98) - open/close 8282 (time 1781230931.55 total 540.66 last 18.48) - open/close 8437 (time 1781230941.92 total 551.02 last 14.95) - open/close 8608 (time 1781230952.26 total 561.37 last 16.53) - open/close 8822 (time 1781230963.22 total 572.32 last 19.53) - open/close 9035 (time 1781230973.71 total 582.82 last 20.30) - open/close 9257 (time 1781230984.39 total 593.49 last 20.80) - open/close 9492 (time 1781230995.22 total 604.32 last 21.70) - open/close 9726 (time 1781231005.23 total 614.34 last 23.37) - open/close 9930 (time 1781231015.60 total 624.71 last 19.67) - open/close 10000 (time 1781231018.21 total 627.32 last 26.83) - open/close 10213 (time 1781231028.95 total 638.05 last 19.84) - open/close 10418 (time 1781231039.64 total 648.75 last 19.17) - open/close 10472 (time 1781231050.49 total 659.60 last 4.98) - open/close 10622 (time 1781231061.03 total 670.14 last 14.24) - open/close 10841 (time 1781231071.49 total 680.60 last 20.92) - open/close 11077 (time 1781231081.50 total 690.61 last 23.58) - open/close 11312 (time 1781231091.50 total 700.61 last 23.49) - open/close 11533 (time 1781231101.51 total 710.61 last 22.10) - open/close 11729 (time 1781231112.07 total 721.18 last 18.56) - open/close 11927 (time 1781231123.09 total 732.20 last 17.96) - open/close 12072 (time 1781231133.98 total 743.09 last 13.32) - open/close 12225 (time 1781231144.48 total 753.58 last 14.58) - open/close 12393 (time 1781231154.48 total 763.59 last 16.80) - open/close 12621 (time 1781231165.51 total 774.62 last 20.66) - open/close 12837 (time 1781231176.01 total 785.12 last 20.57) - open/close 12963 (time 1781231186.46 total 795.57 last 12.06) - open/close 13080 (time 1781231196.70 total 805.80 last 11.43) - open/close 13272 (time 1781231207.69 total 816.80 last 17.47) - open/close 13551 (time 1781231217.69 total 826.80 last 27.89) - open/close 13721 (time 1781231227.99 total 837.10 last 16.51) - open/close 13908 (time 1781231238.17 total 847.27 last 18.38) - open/close 14070 (time 1781231248.89 total 858.00 last 15.11) - open/close 14160 (time 1781231259.11 total 868.21 last 8.81) - open/close 14344 (time 1781231269.12 total 878.23 last 18.37) - open/close 14526 (time 1781231279.13 total 888.24 last 18.19) - open/close 14713 (time 1781231289.37 total 898.48 last 18.26) - open/close 14823 (time 1781231301.53 total 910.64 last 9.04) - open/close 14970 (time 1781231311.64 total 920.75 last 14.54) - open/close 15092 (time 1781231321.82 total 930.93 last 11.99) - open/close 15300 (time 1781231332.27 total 941.38 last 19.90) - open/close 15447 (time 1781231342.62 total 951.73 last 14.20) - open/close 15629 (time 1781231352.84 total 961.95 last 17.81) - open/close 15727 (time 1781231365.45 total 974.56 last 7.77) - open/close 15896 (time 1781231375.46 total 984.56 last 16.89) - open/close 16024 (time 1781231385.46 total 994.57 last 12.79) - open/close 16259 (time 1781231396.37 total 1005.47 last 21.55) - open/close 16475 (time 1781231409.09 total 1018.20 last 16.97) - open/close 16584 (time 1781231420.74 total 1029.85 last 9.36) - open/close 16758 (time 1781231430.76 total 1039.87 last 17.36) - open/close 17327 (time 1781231440.78 total 1049.89 last 56.82) - open/close 17925 (time 1781231450.79 total 1059.90 last 59.75) - open/close 18435 (time 1781231460.79 total 1069.90 last 51.00) - open/close 18936 (time 1781231470.80 total 1079.90 last 50.06) - open/close 19487 (time 1781231480.80 total 1089.91 last 55.09) - open/close 20000 (time 1781231489.64 total 1098.74 last 58.05) - open/close 20567 (time 1781231499.65 total 1108.76 last 56.64) - open/close 21102 (time 1781231509.65 total 1118.76 last 53.47) - open/close 21666 (time 1781231519.67 total 1128.78 last 56.31) - open/close 22227 (time 1781231529.67 total 1138.78 last 56.09) - open/close 22759 (time 1781231539.68 total 1148.79 last 53.14) - open/close 23336 (time 1781231549.71 total 1158.82 last 57.54) - open/close 23908 (time 1781231559.73 total 1168.84 last 57.08) - open/close 24503 (time 1781231569.73 total 1178.84 last 59.47) - open/close 25030 (time 1781231579.75 total 1188.85 last 52.64) - open/close 25586 (time 1781231589.75 total 1198.86 last 55.58) - open/close 26175 (time 1781231599.75 total 1208.86 last 58.86) - open/close 26755 (time 1781231609.76 total 1218.87 last 57.98) - open/close 27304 (time 1781231619.77 total 1228.88 last 54.83) - open/close 27830 (time 1781231629.78 total 1238.89 last 52.55) - open/close 28370 (time 1781231639.80 total 1248.91 last 53.89) - open/close 28972 (time 1781231649.81 total 1258.92 last 60.14) - open/close 29549 (time 1781231659.81 total 1268.92 last 57.68) - open/close 30000 (time 1781231667.47 total 1276.58 last 58.92) - open/close 30614 (time 1781231677.47 total 1286.58 last 61.37) - open/close 31061 (time 1781231687.48 total 1296.59 last 44.66) - open/close 31618 (time 1781231697.48 total 1306.59 last 55.68) - open/close 32218 (time 1781231707.51 total 1316.62 last 59.84) - open/close 32801 (time 1781231717.53 total 1326.64 last 58.21) - open/close 33381 (time 1781231727.54 total 1336.65 last 57.91) - open/close 33936 (time 1781231737.56 total 1346.67 last 55.41) - open/close 34534 (time 1781231747.56 total 1356.67 last 59.78) - open/close 35066 (time 1781231757.59 total 1366.70 last 53.04) - open/close 35648 (time 1781231767.60 total 1376.71 last 58.16) - open/close 36240 (time 1781231777.61 total 1386.72 last 59.14) - open/close 36777 (time 1781231787.62 total 1396.73 last 53.65) - open/close 37322 (time 1781231797.63 total 1406.74 last 54.43) - open/close 37904 (time 1781231807.64 total 1416.75 last 58.17) - open/close 38554 (time 1781231817.65 total 1426.75 last 64.95) - open/close 39175 (time 1781231827.65 total 1436.76 last 62.05) - open/close 39821 (time 1781231837.66 total 1446.77 last 64.55) - open/close 40000 (time 1781231840.57 total 1449.68 last 61.45) - open/close 40610 (time 1781231850.58 total 1459.68 last 60.99) - open/close 41192 (time 1781231860.58 total 1469.69 last 58.14) - open/close 41825 (time 1781231870.59 total 1479.69 last 63.29) - open/close 42440 (time 1781231880.59 total 1489.70 last 61.48) - open/close 42956 (time 1781231890.60 total 1499.71 last 51.53) - open/close 43490 (time 1781231900.63 total 1509.74 last 53.27) - open/close 43974 (time 1781231910.65 total 1519.76 last 48.28) - open/close 44455 (time 1781231920.66 total 1529.76 last 48.09) - open/close 44956 (time 1781231930.68 total 1539.79 last 49.98) - open/close 45447 (time 1781231940.68 total 1549.79 last 49.09) - open/close 45985 (time 1781231950.69 total 1559.80 last 53.77) - open/close 46508 (time 1781231960.72 total 1569.83 last 52.13) - open/close 47035 (time 1781231970.72 total 1579.83 last 52.69) - open/close 47587 (time 1781231980.73 total 1589.84 last 55.16) - open/close 48136 (time 1781231990.74 total 1599.85 last 54.82) - open/close 48678 (time 1781232000.75 total 1609.86 last 54.16) - open/close 49225 (time 1781232010.76 total 1619.87 last 54.62) - open/close 49783 (time 1781232020.77 total 1629.88 last 55.76) total: 50000 open/close in 1632.99 seconds: 30.62 ops/second - unlinked 0 (time 1781232028 ; total 0 ; last 0) - unlinked 10000 (time 1781232210 ; total 182 ; last 182) - unlinked 20000 (time 1781232400 ; total 372 ; last 190) - unlinked 30000 (time 1781232586 ; total 558 ; last 186) - unlinked 40000 (time 1781232775 ; total 747 ; last 189) total: 50000 unlinks in 934 seconds: 53.533192 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: modules unloaded. pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2