== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:10:22 (1788444622) start mds service on oleg405-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions oleg405-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg405-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg405-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 90 (time 1788444727.16 total 10.02 last 8.99) - open/close 332 (time 1788444737.17 total 20.02 last 24.18) - open/close 561 (time 1788444748.12 total 30.98 last 20.91) - open/close 769 (time 1788444758.13 total 40.98 last 20.78) - open/close 963 (time 1788444768.33 total 51.19 last 19.02) - open/close 1160 (time 1788444778.77 total 61.62 last 18.87) - open/close 1376 (time 1788444789.77 total 72.62 last 19.64) - open/close 1606 (time 1788444799.84 total 82.69 last 22.84) - open/close 1838 (time 1788444810.27 total 93.12 last 22.25) - open/close 2062 (time 1788444820.49 total 103.35 last 21.90) - open/close 2222 (time 1788444831.14 total 113.99 last 15.03) - open/close 2343 (time 1788444841.62 total 124.48 last 11.54) - open/close 2413 (time 1788444852.37 total 135.23 last 6.51) - open/close 2449 (time 1788444862.92 total 145.78 last 3.41) - open/close 2522 (time 1788444872.99 total 155.85 last 7.25) - open/close 2685 (time 1788444883.53 total 166.39 last 15.46) - open/close 2831 (time 1788444894.28 total 177.14 last 13.58) - open/close 2987 (time 1788444905.29 total 188.14 last 14.18) - open/close 3119 (time 1788444915.29 total 198.15 last 13.19) - open/close 3250 (time 1788444925.39 total 208.24 last 12.98) - open/close 3370 (time 1788444936.10 total 218.95 last 11.21) - open/close 3513 (time 1788444947.60 total 230.46 last 12.43) - open/close 3667 (time 1788444957.72 total 240.58 last 15.22) - open/close 3781 (time 1788444967.97 total 250.83 last 11.12) - open/close 3979 (time 1788444978.90 total 261.76 last 18.11) - open/close 4077 (time 1788444989.77 total 272.63 last 9.02) - open/close 4218 (time 1788445000.46 total 283.32 last 13.19) - open/close 4376 (time 1788445010.47 total 293.33 last 15.78) - open/close 4473 (time 1788445021.00 total 303.85 last 9.21) - open/close 4589 (time 1788445031.75 total 314.61 last 10.79) - open/close 4731 (time 1788445042.71 total 325.57 last 12.96) - open/close 4846 (time 1788445052.74 total 335.59 last 11.47) - open/close 5001 (time 1788445062.76 total 345.61 last 15.47) - open/close 5177 (time 1788445073.17 total 356.03 last 16.90) - open/close 5336 (time 1788445083.17 total 366.03 last 15.89) - open/close 5514 (time 1788445093.19 total 376.04 last 17.78) - open/close 5721 (time 1788445103.70 total 386.56 last 19.69) - open/close 5868 (time 1788445114.02 total 396.88 last 14.24) - open/close 5939 (time 1788445124.34 total 407.20 last 6.88) - open/close 6107 (time 1788445134.35 total 417.21 last 16.78) - open/close 6293 (time 1788445145.24 total 428.09 last 17.09) - open/close 6451 (time 1788445155.27 total 438.12 last 15.76) - open/close 6560 (time 1788445166.02 total 448.88 last 10.13) - open/close 6700 (time 1788445176.26 total 459.12 last 13.67) - open/close 6912 (time 1788445186.26 total 469.12 last 21.20) - open/close 7096 (time 1788445196.72 total 479.58 last 17.59) - open/close 7179 (time 1788445207.68 total 490.54 last 7.57) - open/close 7240 (time 1788445217.69 total 500.54 last 6.10) - open/close 7312 (time 1788445228.18 total 511.04 last 6.86) - open/close 7474 (time 1788445238.19 total 521.04 last 16.19) - open/close 7629 (time 1788445248.66 total 531.51 last 14.80) - open/close 7743 (time 1788445258.67 total 541.53 last 11.39) - open/close 7917 (time 1788445269.58 total 552.43 last 15.95) - open/close 8109 (time 1788445279.61 total 562.46 last 19.14) - open/close 8213 (time 1788445290.19 total 573.04 last 9.83) - open/close 8368 (time 1788445301.12 total 583.98 last 14.17) - open/close 8535 (time 1788445311.82 total 594.68 last 15.61) - open/close 8565 (time 1788445321.83 total 604.68 last 3.00) - open/close 8701 (time 1788445332.71 total 615.56 last 12.50) - open/close 8800 (time 1788445342.72 total 625.57 last 9.89) - open/close 8894 (time 1788445353.43 total 636.29 last 8.77) - open/close 9140 (time 1788445366.22 total 649.08 last 19.23) - open/close 9283 (time 1788445376.47 total 659.33 last 13.96) - open/close 9482 (time 1788445387.31 total 670.17 last 18.36) - open/close 9568 (time 1788445398.09 total 680.95 last 7.98) - open/close 9648 (time 1788445408.35 total 691.20 last 7.80) - open/close 9830 (time 1788445418.53 total 701.38 last 17.87) - open/close 10000 (time 1788445427.84 total 710.70 last 18.25) - open/close 10159 (time 1788445437.85 total 720.70 last 15.89) - open/close 10277 (time 1788445447.86 total 730.71 last 11.79) - open/close 10369 (time 1788445457.89 total 740.74 last 9.17) - open/close 10485 (time 1788445467.89 total 750.75 last 11.59) - open/close 10602 (time 1788445478.57 total 761.42 last 10.96) - open/close 10779 (time 1788445488.58 total 771.43 last 17.68) - open/close 10912 (time 1788445498.98 total 781.83 last 12.79) - open/close 11091 (time 1788445509.36 total 792.21 last 17.25) - open/close 11192 (time 1788445519.62 total 802.47 last 9.84) - open/close 11269 (time 1788445531.74 total 814.60 last 6.35) - open/close 11393 (time 1788445541.90 total 824.76 last 12.20) - open/close 11461 (time 1788445552.86 total 835.72 last 6.21) - open/close 11600 (time 1788445563.11 total 845.96 last 13.57) - open/close 11702 (time 1788445573.47 total 856.33 last 9.84) - open/close 11806 (time 1788445586.21 total 869.07 last 8.17) - open/close 11989 (time 1788445597.85 total 880.70 last 15.73) - open/close 12150 (time 1788445609.68 total 892.53 last 13.61) - open/close 12258 (time 1788445620.50 total 903.36 last 9.98) - open/close 12322 (time 1788445630.95 total 913.80 last 6.13) - open/close 12397 (time 1788445641.20 total 924.06 last 7.31) - open/close 12520 (time 1788445651.62 total 934.47 last 11.81) - open/close 12632 (time 1788445661.63 total 944.49 last 11.19) - open/close 12815 (time 1788445671.64 total 954.49 last 18.28) - open/close 12929 (time 1788445681.84 total 964.70 last 11.17) - open/close 13026 (time 1788445691.86 total 974.71 last 9.69) - open/close 13154 (time 1788445702.04 total 984.89 last 12.57) - open/close 13315 (time 1788445712.26 total 995.11 last 15.75) - open/close 13454 (time 1788445722.62 total 1005.47 last 13.42) - open/close 13621 (time 1788445732.62 total 1015.47 last 16.70) - open/close 13816 (time 1788445742.74 total 1025.60 last 19.26) - open/close 13917 (time 1788445754.89 total 1037.74 last 8.32) - open/close 14057 (time 1788445764.91 total 1047.77 last 13.97) - open/close 14221 (time 1788445775.90 total 1058.75 last 14.93) - open/close 14345 (time 1788445786.06 total 1068.92 last 12.20) - open/close 14507 (time 1788445796.75 total 1079.61 last 15.15) - open/close 14748 (time 1788445806.75 total 1089.61 last 24.10) - open/close 14913 (time 1788445816.78 total 1099.64 last 16.45) - open/close 14936 (time 1788445827.80 total 1110.65 last 2.09) - open/close 15369 (time 1788445837.80 total 1120.66 last 43.30) - open/close 15892 (time 1788445847.81 total 1130.66 last 52.26) - open/close 16504 (time 1788445857.82 total 1140.68 last 61.13) - open/close 17125 (time 1788445867.83 total 1150.69 last 62.02) - open/close 17776 (time 1788445877.85 total 1160.70 last 65.01) - open/close 18451 (time 1788445887.85 total 1170.70 last 67.49) - open/close 19075 (time 1788445897.85 total 1180.71 last 62.35) - open/close 19633 (time 1788445907.88 total 1190.73 last 55.66) - open/close 20000 (time 1788445914.42 total 1197.27 last 56.14) - open/close 20644 (time 1788445924.42 total 1207.28 last 64.38) - open/close 21262 (time 1788445934.43 total 1217.28 last 61.77) - open/close 21793 (time 1788445944.43 total 1227.28 last 53.09) - open/close 22274 (time 1788445954.43 total 1237.28 last 48.09) - open/close 22848 (time 1788445964.44 total 1247.29 last 57.35) - open/close 23506 (time 1788445974.45 total 1257.31 last 65.70) - open/close 24182 (time 1788445984.45 total 1267.31 last 67.59) - open/close 24804 (time 1788445994.46 total 1277.31 last 62.16) - open/close 25460 (time 1788446004.51 total 1287.37 last 65.26) - open/close 26087 (time 1788446014.52 total 1297.38 last 62.62) - open/close 26692 (time 1788446024.55 total 1307.40 last 60.36) - open/close 27279 (time 1788446034.56 total 1317.41 last 58.65) - open/close 27915 (time 1788446044.56 total 1327.41 last 63.58) - open/close 28360 (time 1788446054.59 total 1337.45 last 44.37) - open/close 28754 (time 1788446064.60 total 1347.45 last 39.37) - open/close 29246 (time 1788446074.60 total 1357.46 last 49.17) - open/close 29831 (time 1788446084.61 total 1367.46 last 58.48) - open/close 30000 (time 1788446087.25 total 1370.11 last 63.94) - open/close 30661 (time 1788446097.26 total 1380.11 last 66.07) - open/close 31292 (time 1788446107.26 total 1390.12 last 63.05) - open/close 31853 (time 1788446117.26 total 1400.12 last 56.09) - open/close 32491 (time 1788446127.27 total 1410.13 last 63.76) - open/close 33026 (time 1788446137.28 total 1420.13 last 53.48) - open/close 33680 (time 1788446147.29 total 1430.15 last 65.28) - open/close 34336 (time 1788446157.30 total 1440.16 last 65.53) - open/close 35027 (time 1788446167.31 total 1450.17 last 69.04) - open/close 35720 (time 1788446177.33 total 1460.18 last 69.21) - open/close 36259 (time 1788446187.34 total 1470.19 last 53.84) - open/close 36862 (time 1788446197.34 total 1480.20 last 60.27) - open/close 37438 (time 1788446207.35 total 1490.21 last 57.55) - open/close 38061 (time 1788446217.36 total 1500.22 last 62.21) - open/close 38680 (time 1788446227.37 total 1510.23 last 61.86) - open/close 39270 (time 1788446237.40 total 1520.25 last 58.83) - open/close 39888 (time 1788446247.42 total 1530.27 last 61.68) - open/close 40479 (time 1788446257.45 total 1540.30 last 58.93) - open/close 41070 (time 1788446267.46 total 1550.31 last 59.06) - open/close 41693 (time 1788446277.49 total 1560.34 last 62.10) - open/close 42391 (time 1788446287.50 total 1570.35 last 69.73) - open/close 43047 (time 1788446297.52 total 1580.38 last 65.45) - open/close 43715 (time 1788446307.53 total 1590.38 last 66.77) - open/close 44330 (time 1788446317.54 total 1600.39 last 61.42) - open/close 44962 (time 1788446327.54 total 1610.40 last 63.16) - open/close 45588 (time 1788446337.55 total 1620.40 last 62.58) - open/close 46164 (time 1788446347.56 total 1630.42 last 57.53) - open/close 46789 (time 1788446357.56 total 1640.42 last 62.47) - open/close 47387 (time 1788446367.57 total 1650.42 last 59.79) - open/close 48025 (time 1788446377.58 total 1660.44 last 63.69) - open/close 48650 (time 1788446387.61 total 1670.46 last 62.37) - open/close 49207 (time 1788446397.61 total 1680.47 last 55.65) - open/close 49870 (time 1788446407.62 total 1690.47 last 66.29) total: 50000 open/close in 1692.53 seconds: 29.54 ops/second - unlinked 0 (time 1788446414 ; total 0 ; last 0) - unlinked 10000 (time 1788446584 ; total 170 ; last 170) - unlinked 20000 (time 1788446750 ; total 336 ; last 166) - unlinked 30000 (time 1788446913 ; total 499 ; last 163) - unlinked 40000 (time 1788447074 ; total 660 ; last 161) total: 50000 unlinks in 821 seconds: 60.901340 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: modules unloaded. pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2