== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:56:36 (1709787396) start mds service on oleg310-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg310-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg310-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock oleg310-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 132 (time 1709787453.07 total 10.95 last 12.06) - open/close 340 (time 1709787463.08 total 20.96 last 20.77) - open/close 538 (time 1709787473.10 total 30.98 last 19.77) - open/close 748 (time 1709787483.10 total 40.98 last 20.99) - open/close 908 (time 1709787493.31 total 51.19 last 15.67) - open/close 1110 (time 1709787504.02 total 61.90 last 18.86) - open/close 1267 (time 1709787514.63 total 72.51 last 14.79) - open/close 1484 (time 1709787525.52 total 83.40 last 19.93) - open/close 1671 (time 1709787535.57 total 93.45 last 18.61) - open/close 1858 (time 1709787545.59 total 103.47 last 18.67) - open/close 2034 (time 1709787555.62 total 113.50 last 17.54) - open/close 2200 (time 1709787565.65 total 123.53 last 16.55) - open/close 2367 (time 1709787575.97 total 133.85 last 16.19) - open/close 2615 (time 1709787585.97 total 143.85 last 24.79) - open/close 2771 (time 1709787595.99 total 153.87 last 15.58) - open/close 2983 (time 1709787606.00 total 163.88 last 21.18) - open/close 3203 (time 1709787616.24 total 174.12 last 21.47) - open/close 3379 (time 1709787626.25 total 184.13 last 17.58) - open/close 3561 (time 1709787636.96 total 194.84 last 17.00) - open/close 3760 (time 1709787646.98 total 204.86 last 19.87) - open/close 3877 (time 1709787657.69 total 215.57 last 10.93) - open/close 4047 (time 1709787668.19 total 226.07 last 16.19) - open/close 4258 (time 1709787678.20 total 236.08 last 21.07) - open/close 4472 (time 1709787688.84 total 246.72 last 20.12) - open/close 4631 (time 1709787698.85 total 256.74 last 15.88) - open/close 4790 (time 1709787708.87 total 266.75 last 15.88) - open/close 4948 (time 1709787719.51 total 277.39 last 14.84) - open/close 5043 (time 1709787729.52 total 287.40 last 9.49) - open/close 5190 (time 1709787739.53 total 297.41 last 14.69) - open/close 5436 (time 1709787749.53 total 307.41 last 24.58) - open/close 5560 (time 1709787759.54 total 317.42 last 12.39) - open/close 5723 (time 1709787769.54 total 327.43 last 16.29) - open/close 5878 (time 1709787780.25 total 338.13 last 14.48) - open/close 5924 (time 1709787795.29 total 353.17 last 3.06) - open/close 6085 (time 1709787805.30 total 363.18 last 16.09) - open/close 6276 (time 1709787815.73 total 373.61 last 18.31) - open/close 6395 (time 1709787826.35 total 384.23 last 11.21) - open/close 6579 (time 1709787836.35 total 394.23 last 18.40) - open/close 6691 (time 1709787847.37 total 405.25 last 10.16) - open/close 6803 (time 1709787857.53 total 415.41 last 11.03) - open/close 6923 (time 1709787867.85 total 425.73 last 11.62) - open/close 7078 (time 1709787878.33 total 436.21 last 14.79) - open/close 7227 (time 1709787889.14 total 447.02 last 13.79) - open/close 7297 (time 1709787900.64 total 458.53 last 6.08) - open/close 7582 (time 1709787911.47 total 469.35 last 26.32) - open/close 7669 (time 1709787922.36 total 480.24 last 7.99) - open/close 7876 (time 1709787932.88 total 490.76 last 19.68) - open/close 8027 (time 1709787942.88 total 500.76 last 15.10) - open/close 8174 (time 1709787952.88 total 510.76 last 14.69) - open/close 8299 (time 1709787963.33 total 521.22 last 11.96) - open/close 8557 (time 1709787976.30 total 534.18 last 19.91) - open/close 8796 (time 1709787986.30 total 544.18 last 23.89) - open/close 9026 (time 1709787996.49 total 554.37 last 22.57) - open/close 9179 (time 1709788006.49 total 564.37 last 15.30) - open/close 9361 (time 1709788016.85 total 574.73 last 17.57) - open/close 9569 (time 1709788026.96 total 584.84 last 20.58) - open/close 10000 (time 1709788035.62 total 593.50 last 49.75) - open/close 10548 (time 1709788045.63 total 603.52 last 54.72) - open/close 11100 (time 1709788055.65 total 613.53 last 55.14) - open/close 11660 (time 1709788065.66 total 623.54 last 55.90) - open/close 12239 (time 1709788075.66 total 633.55 last 57.89) - open/close 12863 (time 1709788085.68 total 643.56 last 62.33) - open/close 13464 (time 1709788095.69 total 653.57 last 60.04) - open/close 14044 (time 1709788105.71 total 663.59 last 57.87) - open/close 14587 (time 1709788115.71 total 673.60 last 54.26) - open/close 15161 (time 1709788125.73 total 683.61 last 57.33) - open/close 15695 (time 1709788135.73 total 693.61 last 53.36) - open/close 16234 (time 1709788145.75 total 703.63 last 53.79) - open/close 16726 (time 1709788155.77 total 713.65 last 49.11) - open/close 17209 (time 1709788165.79 total 723.67 last 48.22) - open/close 17699 (time 1709788175.79 total 733.67 last 48.99) - open/close 18163 (time 1709788185.80 total 743.68 last 46.37) - open/close 18610 (time 1709788195.81 total 753.69 last 44.66) - open/close 19054 (time 1709788205.83 total 763.72 last 44.28) - open/close 19473 (time 1709788215.83 total 773.72 last 41.90) - open/close 19941 (time 1709788225.84 total 783.72 last 46.76) - open/close 20381 (time 1709788235.86 total 793.74 last 43.92) - open/close 20806 (time 1709788245.86 total 803.75 last 42.48) - open/close 21217 (time 1709788255.87 total 813.75 last 41.06) - open/close 21630 (time 1709788265.89 total 823.77 last 41.22) - open/close 22031 (time 1709788275.90 total 833.78 last 40.08) - open/close 22410 (time 1709788285.90 total 843.78 last 37.89) - open/close 22773 (time 1709788295.92 total 853.80 last 36.24) - open/close 23130 (time 1709788305.93 total 863.81 last 35.67) - open/close 23525 (time 1709788315.95 total 873.83 last 39.43) - open/close 23908 (time 1709788325.95 total 883.83 last 38.28) - open/close 24313 (time 1709788335.96 total 893.84 last 40.45) - open/close 24729 (time 1709788345.98 total 903.86 last 41.52) - open/close 25141 (time 1709788355.98 total 913.86 last 41.19) - open/close 25538 (time 1709788366.00 total 923.88 last 39.62) - open/close 25911 (time 1709788376.03 total 933.91 last 37.19) - open/close 26303 (time 1709788386.05 total 943.94 last 39.12) - open/close 26676 (time 1709788396.07 total 953.95 last 37.23) - open/close 27069 (time 1709788406.10 total 963.98 last 39.20) - open/close 27438 (time 1709788416.11 total 973.99 last 36.86) - open/close 27827 (time 1709788426.14 total 984.02 last 38.80) - open/close 28206 (time 1709788436.16 total 994.04 last 37.82) - open/close 28588 (time 1709788446.16 total 1004.04 last 38.19) - open/close 28931 (time 1709788456.19 total 1014.07 last 34.22) - open/close 29285 (time 1709788466.20 total 1024.08 last 35.34) - open/close 29651 (time 1709788476.22 total 1034.10 last 36.53) - open/close 29992 (time 1709788486.24 total 1044.12 last 34.03) - open/close 30310 (time 1709788496.26 total 1054.14 last 31.74) - open/close 30658 (time 1709788506.28 total 1064.16 last 34.75) - open/close 31008 (time 1709788516.28 total 1074.16 last 34.99) - open/close 31355 (time 1709788526.30 total 1084.18 last 34.63) - open/close 31696 (time 1709788536.31 total 1094.19 last 34.07) - open/close 32048 (time 1709788546.32 total 1104.20 last 35.14) - open/close 32379 (time 1709788556.35 total 1114.23 last 33.00) - open/close 32698 (time 1709788566.38 total 1124.26 last 31.81) - open/close 33049 (time 1709788576.41 total 1134.29 last 35.00) - open/close 33389 (time 1709788586.42 total 1144.30 last 33.97) - open/close 33723 (time 1709788596.42 total 1154.30 last 33.39) - open/close 34061 (time 1709788606.45 total 1164.33 last 33.71) - open/close 34397 (time 1709788616.47 total 1174.35 last 33.53) - open/close 34723 (time 1709788626.47 total 1184.35 last 32.59) - open/close 35035 (time 1709788636.49 total 1194.37 last 31.14) - open/close 35294 (time 1709788646.51 total 1204.39 last 25.85) - open/close 35520 (time 1709788656.53 total 1214.41 last 22.54) - open/close 35753 (time 1709788666.57 total 1224.45 last 23.22) - open/close 36030 (time 1709788676.57 total 1234.45 last 27.69) - open/close 36312 (time 1709788686.61 total 1244.49 last 28.09) - open/close 36605 (time 1709788696.63 total 1254.51 last 29.24) - open/close 36893 (time 1709788706.64 total 1264.52 last 28.78) - open/close 37167 (time 1709788716.67 total 1274.55 last 27.32) - open/close 37459 (time 1709788726.68 total 1284.56 last 29.17) - open/close 37774 (time 1709788736.70 total 1294.58 last 31.43) - open/close 38087 (time 1709788746.72 total 1304.60 last 31.25) - open/close 38383 (time 1709788756.75 total 1314.63 last 29.50) - open/close 38667 (time 1709788766.79 total 1324.67 last 28.30) - open/close 38917 (time 1709788776.80 total 1334.68 last 24.97) - open/close 39192 (time 1709788786.80 total 1344.68 last 27.50) - open/close 39446 (time 1709788796.81 total 1354.69 last 25.38) - open/close 39706 (time 1709788806.82 total 1364.70 last 25.98) - open/close 39955 (time 1709788816.84 total 1374.72 last 24.85) - open/close 40196 (time 1709788826.84 total 1384.72 last 24.10) - open/close 40479 (time 1709788836.85 total 1394.73 last 28.25) - open/close 40753 (time 1709788846.88 total 1404.76 last 27.33) - open/close 41011 (time 1709788856.90 total 1414.78 last 25.75) - open/close 41257 (time 1709788866.93 total 1424.81 last 24.52) - open/close 41505 (time 1709788876.94 total 1434.82 last 24.79) - open/close 41763 (time 1709788886.95 total 1444.83 last 25.77) - open/close 42037 (time 1709788896.98 total 1454.86 last 27.33) - open/close 42293 (time 1709788906.99 total 1464.87 last 25.57) - open/close 42558 (time 1709788917.02 total 1474.90 last 26.42) - open/close 42822 (time 1709788927.03 total 1484.91 last 26.36) - open/close 43082 (time 1709788937.04 total 1494.92 last 25.99) - open/close 43339 (time 1709788947.06 total 1504.94 last 25.64) - open/close 43594 (time 1709788957.06 total 1514.95 last 25.48) - open/close 43856 (time 1709788967.09 total 1524.97 last 26.15) - open/close 44125 (time 1709788977.11 total 1534.99 last 26.83) - open/close 44393 (time 1709788987.14 total 1545.02 last 26.73) - open/close 44647 (time 1709788997.17 total 1555.05 last 25.34) - open/close 44908 (time 1709789007.18 total 1565.07 last 26.05) - open/close 45167 (time 1709789017.21 total 1575.09 last 25.84) - open/close 45421 (time 1709789027.21 total 1585.10 last 25.39) - open/close 45672 (time 1709789037.23 total 1595.11 last 25.07) - open/close 45920 (time 1709789047.23 total 1605.12 last 24.78) - open/close 46168 (time 1709789057.26 total 1615.14 last 24.74) - open/close 46409 (time 1709789067.27 total 1625.15 last 24.06) - open/close 46629 (time 1709789077.28 total 1635.17 last 21.98) - open/close 46848 (time 1709789087.30 total 1645.18 last 21.87) - open/close 47054 (time 1709789097.33 total 1655.21 last 20.55) - open/close 47291 (time 1709789107.36 total 1665.24 last 23.61) - open/close 47523 (time 1709789117.40 total 1675.28 last 23.11) - open/close 47745 (time 1709789127.42 total 1685.30 last 22.17) - open/close 47966 (time 1709789137.42 total 1695.30 last 22.10) - open/close 48194 (time 1709789147.46 total 1705.34 last 22.71) - open/close 48424 (time 1709789157.48 total 1715.36 last 22.96) - open/close 48658 (time 1709789167.51 total 1725.39 last 23.31) - open/close 48899 (time 1709789177.54 total 1735.42 last 24.03) - open/close 49127 (time 1709789187.59 total 1745.47 last 22.70) - open/close 49352 (time 1709789197.64 total 1755.52 last 22.38) - open/close 49576 (time 1709789207.65 total 1765.53 last 22.38) - open/close 49801 (time 1709789217.70 total 1775.58 last 22.40) total: 50000 open/close in 1784.38 seconds: 28.02 ops/second - unlinked 0 (time 1709789227 ; total 0 ; last 0) - unlinked 10000 (time 1709789750 ; total 523 ; last 523) - unlinked 20000 (time 1709790132 ; total 905 ; last 382) - unlinked 30000 (time 1709790332 ; total 1105 ; last 200) - unlinked 40000 (time 1709790412 ; total 1185 ; last 80) total: 50000 unlinks in 1213 seconds: 41.220116 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server unloading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2