== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:57:57 (1709787477) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg214-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1709787527.46 total 10.00 last 24.19) - open/close 457 (time 1709787537.46 total 20.01 last 21.49) - open/close 715 (time 1709787547.62 total 30.17 last 25.41) - open/close 928 (time 1709787558.41 total 40.95 last 19.75) - open/close 1108 (time 1709787568.43 total 50.97 last 17.96) - open/close 1286 (time 1709787579.34 total 61.88 last 16.32) - open/close 1446 (time 1709787590.35 total 72.89 last 14.53) - open/close 1583 (time 1709787600.69 total 83.24 last 13.24) - open/close 1747 (time 1709787610.70 total 93.25 last 16.39) - open/close 1933 (time 1709787621.10 total 103.65 last 17.89) - open/close 2059 (time 1709787631.54 total 114.09 last 12.06) - open/close 2230 (time 1709787642.48 total 125.02 last 15.64) - open/close 2353 (time 1709787652.57 total 135.12 last 12.18) - open/close 2625 (time 1709787663.03 total 145.58 last 26.01) - open/close 2787 (time 1709787673.19 total 155.74 last 15.94) - open/close 2834 (time 1709787683.21 total 165.76 last 4.69) - open/close 3035 (time 1709787694.09 total 176.64 last 18.47) - open/close 3362 (time 1709787704.16 total 186.70 last 32.50) - open/close 3631 (time 1709787714.61 total 197.15 last 25.73) - open/close 3795 (time 1709787725.32 total 207.86 last 15.32) - open/close 3916 (time 1709787735.45 total 217.99 last 11.94) - open/close 4043 (time 1709787746.14 total 228.69 last 11.88) - open/close 4288 (time 1709787756.15 total 238.70 last 24.47) - open/close 4577 (time 1709787767.09 total 249.64 last 26.41) - open/close 4692 (time 1709787780.23 total 262.78 last 8.75) - open/close 4853 (time 1709787790.29 total 272.84 last 16.01) - open/close 5032 (time 1709787800.77 total 283.32 last 17.08) - open/close 5156 (time 1709787812.20 total 294.75 last 10.85) - open/close 5317 (time 1709787822.69 total 305.24 last 15.35) - open/close 5463 (time 1709787833.37 total 315.91 last 13.67) - open/close 5583 (time 1709787843.80 total 326.35 last 11.50) - open/close 5755 (time 1709787853.80 total 336.35 last 17.20) - open/close 5949 (time 1709787863.81 total 346.36 last 19.39) - open/close 6169 (time 1709787873.82 total 356.37 last 21.97) - open/close 6305 (time 1709787883.83 total 366.37 last 13.59) - open/close 6449 (time 1709787894.04 total 376.59 last 14.10) - open/close 6576 (time 1709787912.09 total 394.64 last 7.04) - open/close 6708 (time 1709787923.05 total 405.60 last 12.04) - open/close 6842 (time 1709787933.11 total 415.66 last 13.32) - open/close 6947 (time 1709787943.12 total 425.66 last 10.49) - open/close 7164 (time 1709787953.74 total 436.29 last 20.42) - open/close 7299 (time 1709787963.84 total 446.39 last 13.37) - open/close 7619 (time 1709787974.23 total 456.78 last 30.79) - open/close 7848 (time 1709787984.25 total 466.79 last 22.87) - open/close 8096 (time 1709787994.26 total 476.80 last 24.77) - open/close 8282 (time 1709788004.94 total 487.48 last 17.42) - open/close 8668 (time 1709788014.94 total 497.49 last 38.57) - open/close 8920 (time 1709788025.93 total 508.48 last 22.93) - open/close 9264 (time 1709788035.94 total 518.49 last 34.37) - open/close 9615 (time 1709788045.98 total 528.53 last 34.96) - open/close 10000 (time 1709788053.40 total 535.95 last 51.89) - open/close 10814 (time 1709788063.41 total 545.95 last 81.37) - open/close 11494 (time 1709788073.41 total 555.96 last 67.95) - open/close 12214 (time 1709788083.42 total 565.96 last 71.99) - open/close 12918 (time 1709788093.42 total 575.97 last 70.34) - open/close 13557 (time 1709788103.43 total 585.98 last 63.86) - open/close 14188 (time 1709788113.44 total 595.99 last 63.04) - open/close 14817 (time 1709788123.45 total 606.00 last 62.83) - open/close 15428 (time 1709788133.46 total 616.01 last 61.03) - open/close 16007 (time 1709788143.47 total 626.02 last 57.84) - open/close 16584 (time 1709788153.47 total 636.02 last 57.68) - open/close 17162 (time 1709788163.47 total 646.02 last 57.80) - open/close 17743 (time 1709788173.48 total 656.03 last 58.07) - open/close 18292 (time 1709788183.49 total 666.03 last 54.87) - open/close 18831 (time 1709788193.49 total 676.04 last 53.88) - open/close 19340 (time 1709788203.50 total 686.05 last 50.86) - open/close 19871 (time 1709788213.51 total 696.06 last 53.02) - open/close 20000 (time 1709788215.89 total 698.44 last 54.23) - open/close 20522 (time 1709788225.90 total 708.44 last 52.17) - open/close 21029 (time 1709788235.90 total 718.44 last 50.70) - open/close 21524 (time 1709788245.91 total 728.46 last 49.43) - open/close 22015 (time 1709788255.92 total 738.47 last 49.04) - open/close 22467 (time 1709788265.93 total 748.48 last 45.16) - open/close 22947 (time 1709788275.95 total 758.50 last 47.91) - open/close 23368 (time 1709788285.95 total 768.50 last 42.09) - open/close 23790 (time 1709788295.96 total 778.51 last 42.16) - open/close 24196 (time 1709788305.99 total 788.54 last 40.50) - open/close 24611 (time 1709788315.99 total 798.54 last 41.49) - open/close 25012 (time 1709788326.01 total 808.56 last 40.02) - open/close 25389 (time 1709788336.04 total 818.58 last 37.60) - open/close 25779 (time 1709788346.06 total 828.60 last 38.93) - open/close 26159 (time 1709788356.08 total 838.63 last 37.91) - open/close 26551 (time 1709788366.10 total 848.64 last 39.14) - open/close 26918 (time 1709788376.11 total 858.66 last 36.64) - open/close 27286 (time 1709788386.14 total 868.69 last 36.69) - open/close 27644 (time 1709788396.15 total 878.70 last 35.78) - open/close 28034 (time 1709788406.16 total 888.70 last 38.98) - open/close 28411 (time 1709788416.17 total 898.71 last 37.66) - open/close 28786 (time 1709788426.20 total 908.74 last 37.39) - open/close 29175 (time 1709788436.21 total 918.75 last 38.86) - open/close 29555 (time 1709788446.21 total 928.76 last 37.97) - open/close 29933 (time 1709788456.22 total 938.76 last 37.79) - open/close 30285 (time 1709788466.24 total 948.78 last 35.12) - open/close 30660 (time 1709788476.26 total 958.81 last 37.42) - open/close 31046 (time 1709788486.26 total 968.81 last 38.58) - open/close 31428 (time 1709788496.28 total 978.82 last 38.15) - open/close 31809 (time 1709788506.28 total 988.82 last 38.10) - open/close 32189 (time 1709788516.29 total 998.83 last 37.97) - open/close 32564 (time 1709788526.30 total 1008.85 last 37.44) - open/close 32934 (time 1709788536.32 total 1018.86 last 36.94) - open/close 33299 (time 1709788546.33 total 1028.88 last 36.45) - open/close 33674 (time 1709788556.34 total 1038.89 last 37.46) - open/close 34028 (time 1709788566.35 total 1048.90 last 35.37) - open/close 34377 (time 1709788576.36 total 1058.91 last 34.86) - open/close 34738 (time 1709788586.39 total 1068.93 last 36.02) - open/close 35097 (time 1709788596.40 total 1078.94 last 35.87) - open/close 35448 (time 1709788606.41 total 1088.96 last 35.04) - open/close 35791 (time 1709788616.42 total 1098.96 last 34.28) - open/close 36133 (time 1709788626.42 total 1108.97 last 34.17) - open/close 36462 (time 1709788636.43 total 1118.98 last 32.88) - open/close 36783 (time 1709788646.44 total 1128.99 last 32.06) - open/close 37096 (time 1709788656.45 total 1138.99 last 31.28) - open/close 37429 (time 1709788666.45 total 1149.00 last 33.29) - open/close 37750 (time 1709788676.46 total 1159.01 last 32.06) - open/close 38065 (time 1709788686.49 total 1169.04 last 31.40) - open/close 38390 (time 1709788696.52 total 1179.06 last 32.42) - open/close 38718 (time 1709788706.52 total 1189.06 last 32.79) - open/close 39035 (time 1709788716.55 total 1199.09 last 31.61) - open/close 39361 (time 1709788726.55 total 1209.10 last 32.59) - open/close 39677 (time 1709788736.55 total 1219.10 last 31.60) - open/close 39998 (time 1709788746.58 total 1229.13 last 32.00) - open/close 40302 (time 1709788756.59 total 1239.14 last 30.37) - open/close 40616 (time 1709788766.62 total 1249.17 last 31.32) - open/close 40925 (time 1709788776.63 total 1259.18 last 30.86) - open/close 41229 (time 1709788786.66 total 1269.20 last 30.32) - open/close 41535 (time 1709788796.69 total 1279.23 last 30.51) - open/close 41849 (time 1709788806.72 total 1289.26 last 31.31) - open/close 42149 (time 1709788816.73 total 1299.28 last 29.96) - open/close 42438 (time 1709788826.74 total 1309.28 last 28.89) - open/close 42730 (time 1709788836.75 total 1319.29 last 29.17) - open/close 43017 (time 1709788846.75 total 1329.29 last 28.70) - open/close 43305 (time 1709788856.76 total 1339.30 last 28.77) - open/close 43596 (time 1709788866.78 total 1349.33 last 29.03) - open/close 43889 (time 1709788876.79 total 1359.34 last 29.27) - open/close 44174 (time 1709788886.79 total 1369.34 last 28.50) - open/close 44455 (time 1709788896.82 total 1379.36 last 28.04) - open/close 44741 (time 1709788906.82 total 1389.37 last 28.59) - open/close 45019 (time 1709788916.83 total 1399.38 last 27.76) - open/close 45295 (time 1709788926.86 total 1409.40 last 27.54) - open/close 45574 (time 1709788936.89 total 1419.44 last 27.80) - open/close 45849 (time 1709788946.91 total 1429.46 last 27.45) - open/close 46118 (time 1709788956.92 total 1439.47 last 26.87) - open/close 46384 (time 1709788966.94 total 1449.49 last 26.55) - open/close 46644 (time 1709788976.96 total 1459.51 last 25.94) - open/close 46902 (time 1709788986.97 total 1469.51 last 25.79) - open/close 47158 (time 1709788996.98 total 1479.53 last 25.56) - open/close 47410 (time 1709789007.00 total 1489.54 last 25.16) - open/close 47659 (time 1709789017.02 total 1499.56 last 24.85) - open/close 47913 (time 1709789027.06 total 1509.61 last 25.29) - open/close 48164 (time 1709789037.06 total 1519.61 last 25.09) - open/close 48416 (time 1709789047.10 total 1529.65 last 25.10) - open/close 48664 (time 1709789057.12 total 1539.67 last 24.74) - open/close 48907 (time 1709789067.15 total 1549.69 last 24.24) - open/close 49151 (time 1709789077.16 total 1559.71 last 24.37) - open/close 49398 (time 1709789087.18 total 1569.73 last 24.65) - open/close 49648 (time 1709789097.19 total 1579.74 last 24.98) - open/close 49888 (time 1709789107.20 total 1589.75 last 23.96) total: 50000 open/close in 1594.38 seconds: 31.36 ops/second - unlinked 0 (time 1709789112 ; total 0 ; last 0) - unlinked 10000 (time 1709789570 ; total 458 ; last 458) - unlinked 20000 (time 1709789925 ; total 813 ; last 355) - unlinked 30000 (time 1709790135 ; total 1023 ; last 210) - unlinked 40000 (time 1709790209 ; total 1097 ; last 74) total: 50000 unlinks in 1129 seconds: 44.286980 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2