== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:35:17 (1789130117) start mds service on oleg245-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg245-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg245-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 123 (time 1789130227.26 total 10.00 last 12.30) - open/close 197 (time 1789130237.72 total 20.47 last 7.07) - open/close 386 (time 1789130248.80 total 31.55 last 17.06) - open/close 581 (time 1789130261.47 total 44.22 last 15.39) - open/close 733 (time 1789130271.61 total 54.36 last 15.00) - open/close 839 (time 1789130281.62 total 64.37 last 10.58) - open/close 950 (time 1789130292.52 total 75.27 last 10.19) - open/close 1073 (time 1789130303.19 total 85.94 last 11.53) - open/close 1202 (time 1789130314.01 total 96.75 last 11.93) - open/close 1434 (time 1789130324.03 total 106.78 last 23.14) - open/close 1564 (time 1789130334.83 total 117.58 last 12.04) - open/close 1669 (time 1789130344.83 total 127.58 last 10.50) - open/close 1801 (time 1789130355.86 total 138.61 last 11.97) - open/close 2043 (time 1789130366.73 total 149.48 last 22.26) - open/close 2265 (time 1789130376.75 total 159.50 last 22.16) - open/close 2478 (time 1789130386.76 total 169.51 last 21.28) - open/close 2718 (time 1789130396.89 total 179.64 last 23.68) - open/close 2910 (time 1789130409.20 total 191.95 last 15.60) - open/close 2981 (time 1789130419.41 total 202.16 last 6.95) - open/close 3194 (time 1789130429.69 total 212.44 last 20.72) - open/close 3313 (time 1789130440.49 total 223.24 last 11.02) - open/close 3536 (time 1789130450.51 total 233.25 last 22.27) - open/close 3624 (time 1789130461.02 total 243.77 last 8.37) - open/close 3844 (time 1789130471.05 total 253.80 last 21.93) - open/close 3970 (time 1789130481.07 total 263.82 last 12.58) - open/close 4108 (time 1789130491.07 total 273.82 last 13.80) - open/close 4218 (time 1789130501.07 total 283.82 last 11.00) - open/close 4322 (time 1789130511.37 total 294.12 last 10.10) - open/close 4431 (time 1789130521.40 total 304.15 last 10.87) - open/close 4579 (time 1789130531.41 total 314.16 last 14.78) - open/close 4638 (time 1789130546.28 total 329.03 last 3.97) - open/close 4761 (time 1789130556.78 total 339.53 last 11.71) - open/close 4861 (time 1789130566.80 total 349.55 last 9.99) - open/close 5023 (time 1789130576.80 total 359.55 last 16.20) - open/close 5214 (time 1789130587.17 total 369.92 last 18.42) - open/close 5407 (time 1789130597.67 total 380.42 last 18.38) - open/close 5548 (time 1789130608.34 total 391.09 last 13.21) - open/close 5665 (time 1789130618.48 total 401.23 last 11.54) - open/close 5799 (time 1789130630.75 total 413.50 last 10.92) - open/close 5913 (time 1789130640.76 total 423.50 last 11.39) - open/close 6044 (time 1789130651.13 total 433.87 last 12.63) - open/close 6208 (time 1789130662.12 total 444.87 last 14.92) - open/close 6342 (time 1789130672.79 total 455.54 last 12.56) - open/close 6449 (time 1789130684.51 total 467.26 last 9.13) - open/close 6524 (time 1789130694.63 total 477.38 last 7.41) - open/close 6674 (time 1789130705.32 total 488.07 last 14.03) - open/close 6825 (time 1789130715.32 total 498.07 last 15.10) - open/close 6937 (time 1789130725.48 total 508.23 last 11.03) - open/close 7132 (time 1789130735.65 total 518.40 last 19.17) - open/close 7291 (time 1789130745.66 total 528.41 last 15.88) - open/close 7476 (time 1789130756.51 total 539.26 last 17.05) - open/close 7675 (time 1789130767.21 total 549.95 last 18.61) - open/close 7738 (time 1789130777.22 total 559.97 last 6.29) - open/close 7905 (time 1789130787.22 total 569.97 last 16.70) - open/close 8092 (time 1789130797.79 total 580.54 last 17.69) - open/close 8302 (time 1789130807.80 total 590.55 last 20.97) - open/close 8486 (time 1789130817.82 total 600.57 last 18.37) - open/close 8659 (time 1789130828.31 total 611.06 last 16.48) - open/close 8792 (time 1789130838.77 total 621.52 last 12.71) - open/close 8938 (time 1789130848.98 total 631.73 last 14.30) - open/close 9119 (time 1789130859.56 total 642.31 last 17.11) - open/close 9195 (time 1789130869.56 total 652.31 last 7.60) - open/close 9411 (time 1789130879.57 total 662.32 last 21.59) - open/close 9559 (time 1789130890.21 total 672.95 last 13.92) - open/close 9670 (time 1789130900.22 total 682.97 last 11.08) - open/close 9836 (time 1789130910.57 total 693.32 last 16.03) - open/close 9971 (time 1789130921.59 total 704.33 last 12.26) - open/close 10049 (time 1789130931.77 total 714.52 last 7.66) - open/close 10290 (time 1789130942.50 total 725.25 last 22.45) - open/close 10470 (time 1789130952.54 total 735.28 last 17.94) - open/close 10617 (time 1789130962.56 total 745.31 last 14.67) - open/close 10838 (time 1789130973.54 total 756.29 last 20.12) - open/close 10977 (time 1789130983.97 total 766.71 last 13.33) - open/close 11034 (time 1789130993.97 total 776.72 last 5.70) - open/close 11127 (time 1789131004.26 total 787.01 last 9.04) - open/close 11228 (time 1789131014.28 total 797.03 last 10.08) - open/close 11375 (time 1789131024.99 total 807.74 last 13.73) - open/close 11523 (time 1789131035.55 total 818.30 last 14.01) - open/close 11629 (time 1789131045.57 total 828.32 last 10.58) - open/close 11814 (time 1789131055.90 total 838.65 last 17.91) - open/close 11882 (time 1789131066.03 total 848.78 last 6.72) - open/close 12022 (time 1789131076.04 total 858.79 last 13.98) - open/close 12192 (time 1789131086.04 total 868.79 last 17.00) - open/close 12344 (time 1789131096.04 total 878.79 last 15.19) - open/close 12408 (time 1789131106.09 total 888.84 last 6.37) - open/close 12488 (time 1789131116.34 total 899.09 last 7.81) - open/close 12637 (time 1789131126.35 total 909.10 last 14.88) - open/close 12769 (time 1789131136.41 total 919.16 last 13.12) - open/close 12869 (time 1789131146.60 total 929.35 last 9.82) - open/close 12972 (time 1789131156.61 total 939.36 last 10.29) - open/close 13131 (time 1789131167.04 total 949.79 last 15.24) - open/close 13190 (time 1789131178.08 total 960.83 last 5.34) - open/close 13278 (time 1789131189.05 total 971.80 last 8.03) - open/close 13433 (time 1789131199.59 total 982.33 last 14.71) - open/close 13564 (time 1789131210.60 total 993.35 last 11.89) - open/close 13661 (time 1789131222.37 total 1005.12 last 8.24) - open/close 13785 (time 1789131232.43 total 1015.18 last 12.33) - open/close 13961 (time 1789131242.48 total 1025.23 last 17.51) - open/close 14115 (time 1789131253.27 total 1036.02 last 14.27) - open/close 14149 (time 1789131265.69 total 1048.44 last 2.74) - open/close 14336 (time 1789131275.72 total 1058.47 last 18.65) - open/close 14568 (time 1789131286.24 total 1068.99 last 22.04) - open/close 14706 (time 1789131296.89 total 1079.63 last 12.97) - open/close 14840 (time 1789131307.56 total 1090.30 last 12.56) - open/close 14934 (time 1789131317.57 total 1100.32 last 9.39) - open/close 15030 (time 1789131327.60 total 1110.35 last 9.57) - open/close 15187 (time 1789131338.57 total 1121.32 last 14.32) - open/close 15288 (time 1789131349.68 total 1132.43 last 9.09) - open/close 15698 (time 1789131359.68 total 1142.43 last 41.00) - open/close 16293 (time 1789131369.68 total 1152.43 last 59.49) - open/close 16847 (time 1789131379.69 total 1162.44 last 55.34) - open/close 17484 (time 1789131389.71 total 1172.46 last 63.58) - open/close 18112 (time 1789131399.72 total 1182.47 last 62.77) - open/close 18866 (time 1789131409.72 total 1192.47 last 75.40) - open/close 19607 (time 1789131419.72 total 1202.47 last 74.04) - open/close 20000 (time 1789131425.81 total 1208.55 last 64.63) - open/close 20739 (time 1789131435.83 total 1218.58 last 73.72) - open/close 21408 (time 1789131445.83 total 1228.58 last 66.87) - open/close 22158 (time 1789131455.84 total 1238.59 last 74.95) - open/close 22805 (time 1789131465.84 total 1248.59 last 64.69) - open/close 23466 (time 1789131475.85 total 1258.60 last 66.04) - open/close 24190 (time 1789131485.86 total 1268.60 last 72.37) - open/close 24799 (time 1789131495.87 total 1278.62 last 60.81) - open/close 25445 (time 1789131505.90 total 1288.65 last 64.42) - open/close 26054 (time 1789131515.90 total 1298.65 last 60.87) - open/close 26678 (time 1789131525.92 total 1308.67 last 62.31) - open/close 27398 (time 1789131535.92 total 1318.67 last 71.99) - open/close 27998 (time 1789131545.93 total 1328.68 last 59.96) - open/close 28687 (time 1789131555.93 total 1338.68 last 68.87) - open/close 29448 (time 1789131565.95 total 1348.69 last 76.00) - open/close 30000 (time 1789131575.16 total 1357.91 last 59.90) - open/close 30676 (time 1789131585.17 total 1367.92 last 67.57) - open/close 31391 (time 1789131595.18 total 1377.93 last 71.41) - open/close 32107 (time 1789131605.19 total 1387.94 last 71.54) - open/close 32800 (time 1789131615.20 total 1397.95 last 69.19) - open/close 33467 (time 1789131625.24 total 1407.98 last 66.48) - open/close 34124 (time 1789131635.24 total 1417.99 last 65.68) - open/close 34862 (time 1789131645.26 total 1428.01 last 73.65) - open/close 35418 (time 1789131655.27 total 1438.02 last 55.52) - open/close 36038 (time 1789131665.30 total 1448.04 last 61.85) - open/close 36582 (time 1789131675.32 total 1458.07 last 54.24) - open/close 37164 (time 1789131685.33 total 1468.08 last 58.16) - open/close 37620 (time 1789131695.35 total 1478.10 last 45.53) - open/close 38242 (time 1789131705.35 total 1488.10 last 62.16) - open/close 38832 (time 1789131715.36 total 1498.11 last 58.97) - open/close 39504 (time 1789131725.36 total 1508.11 last 67.17) - open/close 40000 (time 1789131733.55 total 1516.30 last 60.59) - open/close 40560 (time 1789131743.56 total 1526.31 last 55.91) - open/close 41229 (time 1789131753.58 total 1536.32 last 66.83) - open/close 41888 (time 1789131763.58 total 1546.33 last 65.89) - open/close 42645 (time 1789131773.59 total 1556.34 last 75.63) - open/close 43347 (time 1789131783.59 total 1566.34 last 70.17) - open/close 43979 (time 1789131793.59 total 1576.34 last 63.19) - open/close 44625 (time 1789131803.60 total 1586.35 last 64.56) - open/close 45235 (time 1789131813.61 total 1596.36 last 60.91) - open/close 45900 (time 1789131823.62 total 1606.37 last 66.47) - open/close 46490 (time 1789131833.63 total 1616.38 last 58.94) - open/close 47101 (time 1789131843.64 total 1626.39 last 61.05) - open/close 47688 (time 1789131853.64 total 1636.39 last 58.67) - open/close 48310 (time 1789131863.64 total 1646.39 last 62.20) - open/close 48889 (time 1789131873.67 total 1656.42 last 57.73) - open/close 49330 (time 1789131883.68 total 1666.43 last 44.07) - open/close 49850 (time 1789131893.68 total 1676.43 last 51.97) total: 50000 open/close in 1678.87 seconds: 29.78 ops/second - unlinked 0 (time 1789131901 ; total 0 ; last 0) - unlinked 10000 (time 1789132074 ; total 173 ; last 173) - unlinked 20000 (time 1789132252 ; total 351 ; last 178) - unlinked 30000 (time 1789132420 ; total 519 ; last 168) - unlinked 40000 (time 1789132599 ; total 698 ; last 179) total: 50000 unlinks in 877 seconds: 57.012543 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local oleg245-server: modules unloaded. pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2