== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:43 (1709786803) start mds service on oleg123-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock oleg123-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1709786833.76 total 10.16 last 19.30) - open/close 247 (time 1709786844.51 total 20.91 last 4.74) - open/close 354 (time 1709786855.20 total 31.60 last 10.01) - open/close 483 (time 1709786865.21 total 41.61 last 12.89) - open/close 647 (time 1709786875.65 total 52.05 last 15.71) - open/close 923 (time 1709786885.97 total 62.37 last 26.74) - open/close 1096 (time 1709786895.97 total 72.37 last 17.30) - open/close 1237 (time 1709786905.98 total 82.38 last 14.08) - open/close 1486 (time 1709786916.35 total 92.75 last 24.01) - open/close 1636 (time 1709786926.50 total 102.90 last 14.77) - open/close 1820 (time 1709786937.10 total 113.50 last 17.37) - open/close 1999 (time 1709786947.31 total 123.71 last 17.53) - open/close 2132 (time 1709786957.31 total 133.71 last 13.29) - open/close 2356 (time 1709786967.82 total 144.22 last 21.32) - open/close 2532 (time 1709786978.28 total 154.68 last 16.83) - open/close 2792 (time 1709786989.11 total 165.51 last 24.00) - open/close 3095 (time 1709786999.93 total 176.33 last 28.00) - open/close 3406 (time 1709787010.67 total 187.07 last 28.96) - open/close 3575 (time 1709787021.08 total 197.48 last 16.24) - open/close 3952 (time 1709787031.09 total 207.49 last 37.67) - open/close 4180 (time 1709787041.30 total 217.70 last 22.33) - open/close 4516 (time 1709787051.32 total 227.72 last 33.53) - open/close 4756 (time 1709787061.33 total 237.73 last 23.99) - open/close 4906 (time 1709787071.33 total 247.73 last 15.00) - open/close 5158 (time 1709787082.30 total 258.70 last 22.97) - open/close 5248 (time 1709787092.96 total 269.36 last 8.44) - open/close 5422 (time 1709787103.21 total 279.61 last 16.98) - open/close 5616 (time 1709787113.57 total 289.97 last 18.72) - open/close 5728 (time 1709787123.74 total 300.14 last 11.01) - open/close 5897 (time 1709787134.00 total 310.40 last 16.48) - open/close 6021 (time 1709787144.61 total 321.01 last 11.68) - open/close 6103 (time 1709787155.44 total 331.84 last 7.57) - open/close 6287 (time 1709787165.46 total 341.86 last 18.37) - open/close 6500 (time 1709787175.55 total 351.95 last 21.11) - open/close 6668 (time 1709787185.57 total 361.97 last 16.77) - open/close 6875 (time 1709787196.02 total 372.42 last 19.81) - open/close 6948 (time 1709787206.02 total 382.42 last 7.30) - open/close 7124 (time 1709787216.04 total 392.44 last 17.58) - open/close 7328 (time 1709787226.70 total 403.10 last 19.13) - open/close 7459 (time 1709787236.71 total 413.11 last 13.09) - open/close 7677 (time 1709787247.08 total 423.48 last 21.02) - open/close 7892 (time 1709787257.10 total 433.50 last 21.46) - open/close 8024 (time 1709787267.11 total 443.51 last 13.19) - open/close 8159 (time 1709787277.12 total 453.52 last 13.48) - open/close 8344 (time 1709787287.13 total 463.53 last 18.48) - open/close 8549 (time 1709787298.04 total 474.44 last 18.79) - open/close 8665 (time 1709787308.90 total 485.30 last 10.69) - open/close 8843 (time 1709787318.90 total 495.30 last 17.79) - open/close 8985 (time 1709787329.32 total 505.72 last 13.63) - open/close 9185 (time 1709787340.34 total 516.74 last 18.16) - open/close 9380 (time 1709787350.34 total 526.74 last 19.49) - open/close 9493 (time 1709787361.24 total 537.64 last 10.37) - open/close 9619 (time 1709787371.62 total 548.02 last 12.13) - open/close 10000 (time 1709787379.44 total 555.84 last 48.76) - open/close 10404 (time 1709787389.44 total 565.84 last 40.40) - open/close 10841 (time 1709787399.48 total 575.88 last 43.52) - open/close 11243 (time 1709787409.49 total 585.89 last 40.15) - open/close 11692 (time 1709787419.50 total 595.90 last 44.83) - open/close 12108 (time 1709787429.52 total 605.92 last 41.55) - open/close 12489 (time 1709787439.52 total 615.92 last 38.09) - open/close 12882 (time 1709787449.52 total 625.92 last 39.28) - open/close 13234 (time 1709787459.53 total 635.93 last 35.16) - open/close 13606 (time 1709787469.55 total 645.95 last 37.13) - open/close 13934 (time 1709787479.56 total 655.96 last 32.79) - open/close 13957 (time 1709787491.18 total 667.58 last 1.98) - open/close 14308 (time 1709787501.21 total 677.61 last 34.99) - open/close 14533 (time 1709787518.41 total 694.81 last 13.08) - open/close 14907 (time 1709787528.42 total 704.82 last 37.34) - open/close 15282 (time 1709787538.44 total 714.84 last 37.45) - open/close 15579 (time 1709787548.45 total 724.85 last 29.65) - open/close 15891 (time 1709787558.49 total 734.89 last 31.09) - open/close 16180 (time 1709787568.49 total 744.89 last 28.89) - open/close 16499 (time 1709787578.51 total 754.91 last 31.84) - open/close 16809 (time 1709787588.53 total 764.93 last 30.94) - open/close 17119 (time 1709787598.56 total 774.96 last 30.91) - open/close 17437 (time 1709787608.63 total 785.03 last 31.58) - open/close 17744 (time 1709787618.63 total 795.03 last 30.70) - open/close 18057 (time 1709787628.66 total 805.06 last 31.21) - open/close 18360 (time 1709787638.68 total 815.08 last 30.24) - open/close 18670 (time 1709787648.69 total 825.09 last 30.95) - open/close 18971 (time 1709787658.71 total 835.11 last 30.06) - open/close 19279 (time 1709787668.72 total 845.12 last 30.77) - open/close 19573 (time 1709787678.73 total 855.13 last 29.37) - open/close 19872 (time 1709787688.74 total 865.14 last 29.85) - open/close 20000 (time 1709787692.99 total 869.39 last 30.17) - open/close 20293 (time 1709787702.99 total 879.39 last 29.28) - open/close 20609 (time 1709787713.03 total 889.43 last 31.48) - open/close 20926 (time 1709787723.04 total 899.44 last 31.66) - open/close 21253 (time 1709787733.05 total 909.45 last 32.68) - open/close 21543 (time 1709787743.07 total 919.47 last 28.94) - open/close 21808 (time 1709787753.08 total 929.48 last 26.47) - open/close 22083 (time 1709787763.14 total 939.54 last 27.35) - open/close 22377 (time 1709787773.16 total 949.56 last 29.32) - open/close 22664 (time 1709787783.18 total 959.58 last 28.66) - open/close 22927 (time 1709787793.22 total 969.62 last 26.19) - open/close 23183 (time 1709787803.24 total 979.64 last 25.56) - open/close 23476 (time 1709787813.26 total 989.65 last 29.25) - open/close 23778 (time 1709787823.27 total 999.67 last 30.15) - open/close 24032 (time 1709787833.37 total 1009.77 last 25.16) - open/close 24263 (time 1709787843.38 total 1019.78 last 23.06) - open/close 24498 (time 1709787853.39 total 1029.79 last 23.48) - open/close 24756 (time 1709787863.41 total 1039.81 last 25.75) - open/close 25014 (time 1709787873.44 total 1049.84 last 25.72) - open/close 25264 (time 1709787883.47 total 1059.87 last 24.92) - open/close 25517 (time 1709787893.51 total 1069.91 last 25.22) - open/close 25773 (time 1709787903.52 total 1079.92 last 25.57) - open/close 26010 (time 1709787913.52 total 1089.92 last 23.68) - open/close 26231 (time 1709787923.58 total 1099.97 last 21.99) - open/close 26462 (time 1709787933.58 total 1109.98 last 23.08) - open/close 26720 (time 1709787943.60 total 1120.00 last 25.76) - open/close 26979 (time 1709787953.63 total 1130.03 last 25.83) - open/close 27258 (time 1709787963.66 total 1140.06 last 27.82) - open/close 27527 (time 1709787973.68 total 1150.07 last 26.85) - open/close 27825 (time 1709787983.68 total 1160.08 last 29.79) - open/close 28126 (time 1709787993.71 total 1170.11 last 30.01) - open/close 28399 (time 1709788003.73 total 1180.13 last 27.24) - open/close 28688 (time 1709788013.75 total 1190.15 last 28.86) - open/close 28932 (time 1709788023.77 total 1200.17 last 24.35) - open/close 29220 (time 1709788033.80 total 1210.19 last 28.72) - open/close 29496 (time 1709788043.83 total 1220.23 last 27.51) - open/close 29776 (time 1709788053.87 total 1230.27 last 27.88) - open/close 30000 (time 1709788062.64 total 1239.04 last 25.53) - open/close 30274 (time 1709788072.66 total 1249.06 last 27.35) - open/close 30559 (time 1709788082.68 total 1259.08 last 28.44) - open/close 30831 (time 1709788092.69 total 1269.09 last 27.17) - open/close 31102 (time 1709788102.70 total 1279.10 last 27.06) - open/close 31354 (time 1709788112.73 total 1289.13 last 25.14) - open/close 31608 (time 1709788122.73 total 1299.13 last 25.39) - open/close 31865 (time 1709788132.76 total 1309.16 last 25.64) - open/close 32128 (time 1709788142.80 total 1319.20 last 26.19) - open/close 32363 (time 1709788152.85 total 1329.25 last 23.39) - open/close 32600 (time 1709788162.87 total 1339.27 last 23.65) - open/close 32846 (time 1709788172.88 total 1349.28 last 24.57) - open/close 33098 (time 1709788182.89 total 1359.29 last 25.16) - open/close 33335 (time 1709788192.91 total 1369.31 last 23.67) - open/close 33591 (time 1709788202.94 total 1379.34 last 25.52) - open/close 33872 (time 1709788212.94 total 1389.34 last 28.09) - open/close 34132 (time 1709788222.98 total 1399.38 last 25.89) - open/close 34422 (time 1709788233.00 total 1409.40 last 28.95) - open/close 34682 (time 1709788243.02 total 1419.41 last 25.96) - open/close 34926 (time 1709788253.05 total 1429.45 last 24.32) - open/close 35165 (time 1709788263.06 total 1439.46 last 23.86) - open/close 35414 (time 1709788273.10 total 1449.50 last 24.82) - open/close 35662 (time 1709788283.13 total 1459.53 last 24.72) - open/close 35922 (time 1709788293.14 total 1469.53 last 25.98) - open/close 36163 (time 1709788303.14 total 1479.54 last 24.08) - open/close 36409 (time 1709788313.20 total 1489.60 last 24.47) - open/close 36664 (time 1709788323.22 total 1499.62 last 25.44) - open/close 36917 (time 1709788333.26 total 1509.65 last 25.21) - open/close 37158 (time 1709788343.28 total 1519.68 last 24.05) - open/close 37410 (time 1709788353.28 total 1529.68 last 25.20) - open/close 37658 (time 1709788363.29 total 1539.69 last 24.78) - open/close 37903 (time 1709788373.32 total 1549.72 last 24.43) - open/close 38150 (time 1709788383.34 total 1559.74 last 24.65) - open/close 38383 (time 1709788393.37 total 1569.77 last 23.22) - open/close 38618 (time 1709788403.37 total 1579.77 last 23.49) - open/close 38872 (time 1709788413.40 total 1589.80 last 25.33) - open/close 39096 (time 1709788423.40 total 1599.80 last 22.40) - open/close 39324 (time 1709788433.43 total 1609.83 last 22.73) - open/close 39543 (time 1709788443.44 total 1619.84 last 21.89) - open/close 39762 (time 1709788453.47 total 1629.87 last 21.83) - open/close 39981 (time 1709788463.47 total 1639.87 last 21.89) - open/close 40186 (time 1709788473.51 total 1649.91 last 20.42) - open/close 40405 (time 1709788483.51 total 1659.91 last 21.89) - open/close 40634 (time 1709788493.55 total 1669.94 last 22.83) - open/close 40859 (time 1709788503.55 total 1679.95 last 22.49) - open/close 41098 (time 1709788513.58 total 1689.98 last 23.83) - open/close 41330 (time 1709788523.58 total 1699.98 last 23.19) - open/close 41561 (time 1709788533.59 total 1709.99 last 23.09) - open/close 41791 (time 1709788543.61 total 1720.01 last 22.94) - open/close 42019 (time 1709788553.63 total 1730.03 last 22.76) - open/close 42251 (time 1709788563.63 total 1740.03 last 23.20) - open/close 42489 (time 1709788573.64 total 1750.04 last 23.79) - open/close 42702 (time 1709788583.65 total 1760.05 last 21.27) - open/close 42924 (time 1709788593.66 total 1770.06 last 22.19) - open/close 43149 (time 1709788603.70 total 1780.09 last 22.41) - open/close 43378 (time 1709788613.73 total 1790.13 last 22.82) - open/close 43624 (time 1709788623.73 total 1800.13 last 24.59) - open/close 43876 (time 1709788633.76 total 1810.16 last 25.14) - open/close 44104 (time 1709788643.76 total 1820.16 last 22.79) - open/close 44330 (time 1709788653.80 total 1830.20 last 22.51) - open/close 44557 (time 1709788663.83 total 1840.23 last 22.64) - open/close 44769 (time 1709788673.86 total 1850.26 last 21.13) - open/close 44978 (time 1709788683.90 total 1860.30 last 20.82) - open/close 45186 (time 1709788693.92 total 1870.32 last 20.76) - open/close 45392 (time 1709788703.96 total 1880.36 last 20.52) - open/close 45592 (time 1709788713.98 total 1890.38 last 19.95) - open/close 45798 (time 1709788724.01 total 1900.41 last 20.54) - open/close 45997 (time 1709788734.06 total 1910.46 last 19.80) - open/close 46185 (time 1709788744.09 total 1920.48 last 18.75) - open/close 46373 (time 1709788754.12 total 1930.52 last 18.73) - open/close 46564 (time 1709788764.17 total 1940.57 last 19.00) - open/close 46751 (time 1709788774.24 total 1950.64 last 18.57) - open/close 46944 (time 1709788784.26 total 1960.66 last 19.26) - open/close 47150 (time 1709788794.26 total 1970.66 last 20.60) - open/close 47356 (time 1709788804.27 total 1980.67 last 20.59) - open/close 47571 (time 1709788814.29 total 1990.69 last 21.45) - open/close 47781 (time 1709788824.32 total 2000.72 last 20.95) - open/close 47994 (time 1709788834.33 total 2010.73 last 21.28) - open/close 48206 (time 1709788844.37 total 2020.77 last 21.12) - open/close 48419 (time 1709788854.39 total 2030.79 last 21.26) - open/close 48615 (time 1709788864.42 total 2040.82 last 19.54) - open/close 48830 (time 1709788874.44 total 2050.84 last 21.46) - open/close 49042 (time 1709788884.49 total 2060.89 last 21.09) - open/close 49249 (time 1709788894.49 total 2070.89 last 20.70) - open/close 49451 (time 1709788904.50 total 2080.90 last 20.19) - open/close 49657 (time 1709788914.50 total 2090.90 last 20.60) - open/close 49853 (time 1709788924.52 total 2100.91 last 19.57) total: 50000 open/close in 2108.14 seconds: 23.72 ops/second