== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:34 (1709786974) start mds service on oleg113-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg113-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock oleg113-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 77 (time 1709787023.43 total 12.09 last 6.37) - open/close 388 (time 1709787033.91 total 22.57 last 29.68) - open/close 749 (time 1709787043.92 total 32.58 last 36.06) - open/close 804 (time 1709787056.63 total 45.29 last 4.33) - open/close 1088 (time 1709787066.65 total 55.31 last 28.35) - open/close 1304 (time 1709787077.05 total 65.71 last 20.77) - open/close 1512 (time 1709787087.73 total 76.39 last 19.48) - open/close 1699 (time 1709787098.28 total 86.94 last 17.72) - open/close 1867 (time 1709787109.56 total 98.22 last 14.89) - open/close 2105 (time 1709787120.08 total 108.74 last 22.62) - open/close 2237 (time 1709787130.71 total 119.37 last 12.42) - open/close 2420 (time 1709787140.97 total 129.63 last 17.83) - open/close 2594 (time 1709787151.81 total 140.47 last 16.05) - open/close 2742 (time 1709787162.09 total 150.74 last 14.41) - open/close 2917 (time 1709787172.09 total 160.75 last 17.49) - open/close 3125 (time 1709787182.10 total 170.76 last 20.78) - open/close 3278 (time 1709787192.75 total 181.40 last 14.37) - open/close 3414 (time 1709787202.76 total 191.41 last 13.59) - open/close 3617 (time 1709787213.64 total 202.29 last 18.66) - open/close 3784 (time 1709787223.64 total 212.30 last 16.69) - open/close 4025 (time 1709787233.91 total 222.56 last 23.49) - open/close 4166 (time 1709787243.91 total 232.57 last 14.09) - open/close 4314 (time 1709787256.43 total 245.09 last 11.82) - open/close 4503 (time 1709787268.65 total 257.31 last 15.47) - open/close 4624 (time 1709787278.92 total 267.58 last 11.78) - open/close 4696 (time 1709787289.40 total 278.06 last 6.87) - open/close 4878 (time 1709787300.02 total 288.68 last 17.13) - open/close 5020 (time 1709787311.06 total 299.72 last 12.86) - open/close 5196 (time 1709787321.42 total 310.08 last 16.99) - open/close 5283 (time 1709787331.45 total 320.11 last 8.68) - open/close 5488 (time 1709787341.48 total 330.13 last 20.45) - open/close 5690 (time 1709787351.49 total 340.15 last 20.18) - open/close 5894 (time 1709787361.49 total 350.15 last 20.39) - open/close 6058 (time 1709787372.01 total 360.67 last 15.59) - open/close 6198 (time 1709787382.02 total 370.67 last 13.99) - open/close 6455 (time 1709787392.10 total 380.75 last 25.50) - open/close 6638 (time 1709787403.08 total 391.73 last 16.67) - open/close 6779 (time 1709787414.08 total 402.74 last 12.81) - open/close 6929 (time 1709787424.63 total 413.29 last 14.22) - open/close 7054 (time 1709787435.57 total 424.22 last 11.43) - open/close 7241 (time 1709787454.82 total 443.48 last 9.71) - open/close 7399 (time 1709787464.83 total 453.49 last 15.78) - open/close 7511 (time 1709787474.84 total 463.50 last 11.19) - open/close 7635 (time 1709787485.11 total 473.77 last 12.07) - open/close 7852 (time 1709787495.12 total 483.78 last 21.68) - open/close 8043 (time 1709787505.78 total 494.44 last 17.91) - open/close 8159 (time 1709787515.80 total 504.45 last 11.58) - open/close 8278 (time 1709787525.88 total 514.54 last 11.80) - open/close 8504 (time 1709787536.68 total 525.34 last 20.92) - open/close 8739 (time 1709787546.74 total 535.40 last 23.37) - open/close 8932 (time 1709787557.15 total 545.81 last 18.53) - open/close 9104 (time 1709787567.18 total 555.84 last 17.15) - open/close 9308 (time 1709787577.57 total 566.23 last 19.64) - open/close 9417 (time 1709787588.20 total 576.86 last 10.25) - open/close 9537 (time 1709787598.20 total 586.86 last 12.00) - open/close 9922 (time 1709787608.23 total 596.89 last 38.41) - open/close 10424 (time 1709787618.24 total 606.90 last 50.12) - open/close 10894 (time 1709787628.26 total 616.91 last 46.94) - open/close 11339 (time 1709787638.27 total 626.93 last 44.44) - open/close 11814 (time 1709787648.28 total 636.94 last 47.45) - open/close 12227 (time 1709787658.29 total 646.95 last 41.26) - open/close 12656 (time 1709787668.29 total 656.95 last 42.89) - open/close 13075 (time 1709787678.31 total 666.97 last 41.81) - open/close 13484 (time 1709787688.34 total 677.00 last 40.79) - open/close 13889 (time 1709787698.35 total 687.01 last 40.46) - open/close 14313 (time 1709787708.35 total 697.01 last 42.39) - open/close 14734 (time 1709787718.37 total 707.02 last 42.04) - open/close 15181 (time 1709787728.38 total 717.04 last 44.63) - open/close 15596 (time 1709787738.38 total 727.04 last 41.50) - open/close 15953 (time 1709787748.40 total 737.06 last 35.62) - open/close 16296 (time 1709787758.44 total 747.10 last 34.18) - open/close 16656 (time 1709787768.45 total 757.11 last 35.96) - open/close 17023 (time 1709787778.46 total 767.12 last 36.65) - open/close 17352 (time 1709787788.48 total 777.14 last 32.85) - open/close 17670 (time 1709787798.49 total 787.15 last 31.77) - open/close 17848 (time 1709787814.64 total 803.29 last 11.02) - open/close 18206 (time 1709787824.65 total 813.31 last 35.74) - open/close 18513 (time 1709787834.68 total 823.33 last 30.63) - open/close 18810 (time 1709787844.68 total 833.34 last 29.69) - open/close 19115 (time 1709787854.74 total 843.39 last 30.33) - open/close 19426 (time 1709787864.74 total 853.39 last 31.10) - open/close 19745 (time 1709787874.74 total 863.40 last 31.89) - open/close 20000 (time 1709787882.84 total 871.50 last 31.47) - open/close 20303 (time 1709787892.87 total 881.52 last 30.22) - open/close 20612 (time 1709787902.87 total 891.53 last 30.89) - open/close 20902 (time 1709787912.91 total 901.57 last 28.89) - open/close 20937 (time 1709787925.22 total 913.88 last 2.84) - open/close 21218 (time 1709787935.27 total 923.93 last 27.97) - open/close 21495 (time 1709787945.29 total 933.95 last 27.63) - open/close 21814 (time 1709787955.31 total 943.97 last 31.84) - open/close 22159 (time 1709787965.32 total 953.97 last 34.49) - open/close 22465 (time 1709787975.34 total 964.00 last 30.52) - open/close 22821 (time 1709787985.35 total 974.01 last 35.56) - open/close 23175 (time 1709787995.36 total 984.02 last 35.37) - open/close 23491 (time 1709788005.37 total 994.02 last 31.58) - open/close 23820 (time 1709788015.39 total 1004.05 last 32.81) - open/close 24127 (time 1709788025.41 total 1014.07 last 30.65) - open/close 24445 (time 1709788035.43 total 1024.08 last 31.75) - open/close 24776 (time 1709788045.43 total 1034.09 last 33.09) - open/close 25109 (time 1709788055.46 total 1044.12 last 33.20) - open/close 25422 (time 1709788065.50 total 1054.16 last 31.18) - open/close 25747 (time 1709788075.51 total 1064.17 last 32.45) - open/close 26078 (time 1709788085.55 total 1074.21 last 32.99) - open/close 26396 (time 1709788095.55 total 1084.21 last 31.78) - open/close 26720 (time 1709788105.57 total 1094.23 last 32.36) - open/close 27024 (time 1709788115.58 total 1104.24 last 30.36) - open/close 27309 (time 1709788125.59 total 1114.25 last 28.48) - open/close 27625 (time 1709788135.59 total 1124.25 last 31.59) - open/close 27922 (time 1709788145.60 total 1134.25 last 29.69) - open/close 28209 (time 1709788155.61 total 1144.27 last 28.66) - open/close 28493 (time 1709788165.64 total 1154.30 last 28.33) - open/close 28778 (time 1709788175.65 total 1164.30 last 28.47) - open/close 29062 (time 1709788185.66 total 1174.31 last 28.38) - open/close 29336 (time 1709788195.70 total 1184.36 last 27.28) - open/close 29654 (time 1709788205.71 total 1194.37 last 31.77) - open/close 29958 (time 1709788215.74 total 1204.39 last 30.33) - open/close 30240 (time 1709788225.74 total 1214.39 last 28.19) - open/close 30557 (time 1709788235.78 total 1224.43 last 31.58) - open/close 30846 (time 1709788245.79 total 1234.44 last 28.87) - open/close 31105 (time 1709788255.79 total 1244.44 last 25.90) - open/close 31348 (time 1709788265.82 total 1254.48 last 24.22) - open/close 31624 (time 1709788275.85 total 1264.50 last 27.53) - open/close 31905 (time 1709788285.87 total 1274.53 last 28.03) - open/close 32187 (time 1709788295.89 total 1284.54 last 28.16) - open/close 32455 (time 1709788305.91 total 1294.56 last 26.74) - open/close 32730 (time 1709788315.92 total 1304.58 last 27.45) - open/close 33009 (time 1709788325.94 total 1314.60 last 27.86) - open/close 33274 (time 1709788335.95 total 1324.61 last 26.46) - open/close 33535 (time 1709788345.96 total 1334.61 last 26.10) - open/close 33807 (time 1709788355.96 total 1344.62 last 27.18) - open/close 34076 (time 1709788365.98 total 1354.63 last 26.86) - open/close 34334 (time 1709788375.99 total 1364.65 last 25.77) - open/close 34607 (time 1709788386.02 total 1374.68 last 27.21) - open/close 34867 (time 1709788396.03 total 1384.69 last 25.97) - open/close 35133 (time 1709788406.05 total 1394.71 last 26.55) - open/close 35402 (time 1709788416.06 total 1404.72 last 26.89) - open/close 35643 (time 1709788426.08 total 1414.73 last 24.06) - open/close 35904 (time 1709788436.09 total 1424.74 last 26.07) - open/close 36153 (time 1709788446.10 total 1434.76 last 24.86) - open/close 36407 (time 1709788456.11 total 1444.77 last 25.37) - open/close 36655 (time 1709788466.13 total 1454.78 last 24.77) - open/close 36900 (time 1709788476.17 total 1464.83 last 24.39) - open/close 37144 (time 1709788486.18 total 1474.83 last 24.38) - open/close 37397 (time 1709788496.20 total 1484.85 last 25.25) - open/close 37643 (time 1709788506.22 total 1494.87 last 24.55) - open/close 37900 (time 1709788516.24 total 1504.89 last 25.65) - open/close 38145 (time 1709788526.26 total 1514.92 last 24.43) - open/close 38387 (time 1709788536.28 total 1524.93 last 24.17) - open/close 38620 (time 1709788546.29 total 1534.95 last 23.26) - open/close 38846 (time 1709788556.31 total 1544.96 last 22.57) - open/close 39070 (time 1709788566.34 total 1554.99 last 22.33) - open/close 39293 (time 1709788576.36 total 1565.02 last 22.25) - open/close 39521 (time 1709788586.37 total 1575.03 last 22.78) - open/close 39750 (time 1709788596.37 total 1585.03 last 22.90) - open/close 39980 (time 1709788606.38 total 1595.03 last 22.99) - open/close 40207 (time 1709788616.40 total 1605.06 last 22.64) - open/close 40447 (time 1709788626.40 total 1615.06 last 23.99) - open/close 40693 (time 1709788636.41 total 1625.07 last 24.59) - open/close 40926 (time 1709788646.41 total 1635.07 last 23.30) - open/close 41158 (time 1709788656.42 total 1645.07 last 23.18) - open/close 41390 (time 1709788666.44 total 1655.09 last 23.15) - open/close 41613 (time 1709788676.45 total 1665.11 last 22.27) - open/close 41829 (time 1709788686.48 total 1675.14 last 21.53) - open/close 42051 (time 1709788696.49 total 1685.15 last 22.17) - open/close 42273 (time 1709788706.53 total 1695.19 last 22.11) - open/close 42510 (time 1709788716.55 total 1705.20 last 23.67) - open/close 42738 (time 1709788726.56 total 1715.22 last 22.76) - open/close 42961 (time 1709788736.59 total 1725.25 last 22.24) - open/close 43174 (time 1709788746.60 total 1735.26 last 21.27) - open/close 43375 (time 1709788756.63 total 1745.29 last 20.04) - open/close 43588 (time 1709788766.63 total 1755.29 last 21.30) - open/close 43800 (time 1709788776.66 total 1765.32 last 21.14) - open/close 44013 (time 1709788786.70 total 1775.36 last 21.21) - open/close 44235 (time 1709788796.72 total 1785.38 last 22.17) - open/close 44446 (time 1709788806.75 total 1795.41 last 21.03) - open/close 44665 (time 1709788816.78 total 1805.43 last 21.85) - open/close 44876 (time 1709788826.81 total 1815.47 last 21.02) - open/close 45096 (time 1709788836.84 total 1825.50 last 21.95) - open/close 45306 (time 1709788846.86 total 1835.52 last 20.94) - open/close 45511 (time 1709788856.92 total 1845.58 last 20.39) - open/close 45709 (time 1709788866.93 total 1855.59 last 19.78) - open/close 45910 (time 1709788876.95 total 1865.61 last 20.06) - open/close 46104 (time 1709788886.96 total 1875.62 last 19.38) - open/close 46306 (time 1709788896.99 total 1885.65 last 20.14) - open/close 46503 (time 1709788906.99 total 1895.65 last 19.69) - open/close 46699 (time 1709788917.04 total 1905.70 last 19.51) - open/close 46890 (time 1709788927.05 total 1915.71 last 19.08) - open/close 47097 (time 1709788937.07 total 1925.73 last 20.66) - open/close 47295 (time 1709788947.10 total 1935.76 last 19.75) - open/close 47496 (time 1709788957.11 total 1945.77 last 20.07) - open/close 47689 (time 1709788967.14 total 1955.79 last 19.25) - open/close 47890 (time 1709788977.18 total 1965.83 last 20.02) - open/close 48081 (time 1709788987.18 total 1975.84 last 19.09) - open/close 48275 (time 1709788997.19 total 1985.85 last 19.38) - open/close 48470 (time 1709789007.22 total 1995.88 last 19.44) - open/close 48664 (time 1709789017.23 total 2005.89 last 19.38) - open/close 48871 (time 1709789027.27 total 2015.93 last 20.62) - open/close 49068 (time 1709789037.28 total 2025.94 last 19.68) - open/close 49254 (time 1709789047.28 total 2035.94 last 18.60) - open/close 49444 (time 1709789057.32 total 2045.98 last 18.92) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 2054.44 seconds: 24.14 ops/second