== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:30 (1709786910) start mds service on oleg338-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg338-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg338-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock oleg338-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 168 (time 1709786956.60 total 10.35 last 16.23) - open/close 392 (time 1709786967.02 total 20.77 last 21.49) - open/close 739 (time 1709786977.71 total 31.46 last 32.48) - open/close 906 (time 1709786988.56 total 42.31 last 15.38) - open/close 1124 (time 1709786998.57 total 52.32 last 21.79) - open/close 1299 (time 1709787008.60 total 62.35 last 17.44) - open/close 1428 (time 1709787028.13 total 81.88 last 6.61) - open/close 1591 (time 1709787038.13 total 91.89 last 16.29) - open/close 1838 (time 1709787049.41 total 103.16 last 21.91) - open/close 2097 (time 1709787059.41 total 113.16 last 25.89) - open/close 2226 (time 1709787069.48 total 123.23 last 12.82) - open/close 2411 (time 1709787079.97 total 133.72 last 17.64) - open/close 2562 (time 1709787092.75 total 146.50 last 11.81) - open/close 2732 (time 1709787103.12 total 156.88 last 16.39) - open/close 2841 (time 1709787113.18 total 166.93 last 10.84) - open/close 3077 (time 1709787123.93 total 177.68 last 21.97) - open/close 3317 (time 1709787134.08 total 187.83 last 23.63) - open/close 3554 (time 1709787144.08 total 197.83 last 23.70) - open/close 3765 (time 1709787154.42 total 208.17 last 20.42) - open/close 3900 (time 1709787164.86 total 218.61 last 12.93) - open/close 4051 (time 1709787175.19 total 228.94 last 14.61) - open/close 4262 (time 1709787186.18 total 239.93 last 19.20) - open/close 4412 (time 1709787197.16 total 250.91 last 13.66) - open/close 4609 (time 1709787207.16 total 260.91 last 19.70) - open/close 4912 (time 1709787217.17 total 270.92 last 30.27) - open/close 5107 (time 1709787227.57 total 281.32 last 18.75) - open/close 5269 (time 1709787237.79 total 291.54 last 15.86) - open/close 5477 (time 1709787248.20 total 301.95 last 19.97) - open/close 5600 (time 1709787258.21 total 311.96 last 12.29) - open/close 5774 (time 1709787268.22 total 321.97 last 17.38) - open/close 5946 (time 1709787278.23 total 331.98 last 17.18) - open/close 6048 (time 1709787288.47 total 342.22 last 9.96) - open/close 6193 (time 1709787298.49 total 352.24 last 14.47) - open/close 6293 (time 1709787309.35 total 363.10 last 9.21) - open/close 6551 (time 1709787320.23 total 373.98 last 23.72) - open/close 6759 (time 1709787330.73 total 384.48 last 19.81) - open/close 6872 (time 1709787341.13 total 394.88 last 10.86) - open/close 7013 (time 1709787351.52 total 405.27 last 13.58) - open/close 7131 (time 1709787361.53 total 415.28 last 11.78) - open/close 7319 (time 1709787371.77 total 425.52 last 18.36) - open/close 7513 (time 1709787382.68 total 436.43 last 17.79) - open/close 7672 (time 1709787393.32 total 447.07 last 14.94) - open/close 7889 (time 1709787403.32 total 457.07 last 21.69) - open/close 8115 (time 1709787413.71 total 467.46 last 21.77) - open/close 8258 (time 1709787423.71 total 477.46 last 14.29) - open/close 8400 (time 1709787434.10 total 487.85 last 13.68) - open/close 8587 (time 1709787444.64 total 498.39 last 17.73) - open/close 8673 (time 1709787454.77 total 508.52 last 8.49) - open/close 8919 (time 1709787464.78 total 518.53 last 24.59) - open/close 9138 (time 1709787474.81 total 528.56 last 21.83) - open/close 9280 (time 1709787485.33 total 539.08 last 13.49) - open/close 9477 (time 1709787495.36 total 549.11 last 19.65) - open/close 9600 (time 1709787505.40 total 559.15 last 12.25) - open/close 9740 (time 1709787516.23 total 569.98 last 12.93) - open/close 9876 (time 1709787526.28 total 580.03 last 13.53) - open/close 10000 (time 1709787529.81 total 583.56 last 35.09) - open/close 10158 (time 1709787539.87 total 593.62 last 15.72) - open/close 10435 (time 1709787549.87 total 603.62 last 27.67) - open/close 10872 (time 1709787559.88 total 613.63 last 43.67) - open/close 11270 (time 1709787569.89 total 623.64 last 39.76) - open/close 11670 (time 1709787579.91 total 633.67 last 39.90) - open/close 12139 (time 1709787589.92 total 643.67 last 46.86) - open/close 12513 (time 1709787599.96 total 653.71 last 37.26) - open/close 12904 (time 1709787609.98 total 663.73 last 39.04) - open/close 13264 (time 1709787620.00 total 673.75 last 35.92) - open/close 13672 (time 1709787630.02 total 683.77 last 40.71) - open/close 14075 (time 1709787640.02 total 693.77 last 40.30) - open/close 14410 (time 1709787650.03 total 703.78 last 33.46) - open/close 14420 (time 1709787661.31 total 715.06 last 0.89) - open/close 14799 (time 1709787671.33 total 725.08 last 37.83) - open/close 15132 (time 1709787681.34 total 735.09 last 33.27) - open/close 15248 (time 1709787696.25 total 750.00 last 7.78) - open/close 15575 (time 1709787706.27 total 760.02 last 32.64) - open/close 15881 (time 1709787716.29 total 770.04 last 30.53) - open/close 16226 (time 1709787726.32 total 780.07 last 34.42) - open/close 16581 (time 1709787736.35 total 790.10 last 35.39) - open/close 16873 (time 1709787746.36 total 800.11 last 29.17) - open/close 17164 (time 1709787756.39 total 810.14 last 29.01) - open/close 17518 (time 1709787766.42 total 820.17 last 35.29) - open/close 17841 (time 1709787776.43 total 830.18 last 32.28) - open/close 18211 (time 1709787786.46 total 840.21 last 36.89) - open/close 18542 (time 1709787796.50 total 850.25 last 32.96) - open/close 18909 (time 1709787806.53 total 860.28 last 36.59) - open/close 19270 (time 1709787816.55 total 870.30 last 36.04) - open/close 19569 (time 1709787826.56 total 880.31 last 29.86) - open/close 19908 (time 1709787836.57 total 890.32 last 33.86) - open/close 20000 (time 1709787839.17 total 892.92 last 35.35) - open/close 20331 (time 1709787849.17 total 902.92 last 33.10) - open/close 20634 (time 1709787859.19 total 912.94 last 30.26) - open/close 20941 (time 1709787869.22 total 922.97 last 30.61) - open/close 21231 (time 1709787879.23 total 932.98 last 28.96) - open/close 21534 (time 1709787889.25 total 943.00 last 30.24) - open/close 21832 (time 1709787899.28 total 953.03 last 29.72) - open/close 22163 (time 1709787909.29 total 963.04 last 33.06) - open/close 22453 (time 1709787919.32 total 973.07 last 28.91) - open/close 22749 (time 1709787929.33 total 983.08 last 29.56) - open/close 23030 (time 1709787939.34 total 993.09 last 28.07) - open/close 23321 (time 1709787949.37 total 1003.12 last 29.03) - open/close 23619 (time 1709787959.40 total 1013.15 last 29.72) - open/close 23902 (time 1709787969.41 total 1023.16 last 28.27) - open/close 24208 (time 1709787979.42 total 1033.17 last 30.55) - open/close 24516 (time 1709787989.45 total 1043.20 last 30.72) - open/close 24818 (time 1709787999.47 total 1053.22 last 30.15) - open/close 25106 (time 1709788009.48 total 1063.23 last 28.76) - open/close 25398 (time 1709788019.53 total 1073.28 last 29.06) - open/close 25687 (time 1709788029.57 total 1083.32 last 28.77) - open/close 25959 (time 1709788039.59 total 1093.34 last 27.15) - open/close 26225 (time 1709788049.61 total 1103.36 last 26.55) - open/close 26485 (time 1709788059.61 total 1113.36 last 26.00) - open/close 26749 (time 1709788069.67 total 1123.42 last 26.25) - open/close 27062 (time 1709788079.69 total 1133.44 last 31.23) - open/close 27401 (time 1709788089.69 total 1143.44 last 33.89) - open/close 27744 (time 1709788099.70 total 1153.45 last 34.29) - open/close 28085 (time 1709788109.71 total 1163.46 last 34.06) - open/close 28421 (time 1709788119.73 total 1173.48 last 33.54) - open/close 28770 (time 1709788129.74 total 1183.49 last 34.88) - open/close 29119 (time 1709788139.74 total 1193.49 last 34.88) - open/close 29450 (time 1709788149.75 total 1203.50 last 33.06) - open/close 29781 (time 1709788159.76 total 1213.51 last 33.06) - open/close 30000 (time 1709788166.40 total 1220.15 last 33.01) - open/close 30318 (time 1709788176.41 total 1230.16 last 31.76) - open/close 30648 (time 1709788186.43 total 1240.18 last 32.92) - open/close 30951 (time 1709788196.44 total 1250.19 last 30.29) - open/close 31238 (time 1709788206.46 total 1260.21 last 28.65) - open/close 31532 (time 1709788216.47 total 1270.22 last 29.36) - open/close 31849 (time 1709788226.49 total 1280.24 last 31.63) - open/close 32163 (time 1709788236.52 total 1290.27 last 31.32) - open/close 32471 (time 1709788246.53 total 1300.28 last 30.76) - open/close 32752 (time 1709788256.53 total 1310.28 last 28.09) - open/close 33034 (time 1709788266.55 total 1320.30 last 28.16) - open/close 33306 (time 1709788276.58 total 1330.33 last 27.12) - open/close 33586 (time 1709788286.60 total 1340.35 last 27.93) - open/close 33841 (time 1709788296.61 total 1350.36 last 25.47) - open/close 34103 (time 1709788306.62 total 1360.37 last 26.18) - open/close 34377 (time 1709788316.64 total 1370.39 last 27.35) - open/close 34645 (time 1709788326.67 total 1380.42 last 26.71) - open/close 34929 (time 1709788336.68 total 1390.43 last 28.38) - open/close 35213 (time 1709788346.71 total 1400.46 last 28.31) - open/close 35502 (time 1709788356.73 total 1410.48 last 28.85) - open/close 35768 (time 1709788366.75 total 1420.50 last 26.55) - open/close 36021 (time 1709788376.76 total 1430.51 last 25.27) - open/close 36278 (time 1709788386.80 total 1440.55 last 25.60) - open/close 36526 (time 1709788396.80 total 1450.55 last 24.79) - open/close 36793 (time 1709788406.83 total 1460.58 last 26.62) - open/close 37060 (time 1709788416.85 total 1470.60 last 26.66) - open/close 37345 (time 1709788426.88 total 1480.63 last 28.39) - open/close 37623 (time 1709788436.91 total 1490.66 last 27.73) - open/close 37885 (time 1709788446.95 total 1500.70 last 26.10) - open/close 38152 (time 1709788456.96 total 1510.71 last 26.67) - open/close 38413 (time 1709788466.98 total 1520.73 last 26.06) - open/close 38687 (time 1709788477.00 total 1530.75 last 27.34) - open/close 38948 (time 1709788487.01 total 1540.76 last 26.08) - open/close 39190 (time 1709788497.06 total 1550.81 last 24.08) - open/close 39440 (time 1709788507.09 total 1560.84 last 24.92) - open/close 39709 (time 1709788517.11 total 1570.86 last 26.85) - open/close 39971 (time 1709788527.11 total 1580.86 last 26.19) - open/close 40229 (time 1709788537.13 total 1590.88 last 25.75) - open/close 40494 (time 1709788547.15 total 1600.90 last 26.45) - open/close 40751 (time 1709788557.17 total 1610.92 last 25.64) - open/close 41012 (time 1709788567.21 total 1620.96 last 26.01) - open/close 41287 (time 1709788577.23 total 1630.98 last 27.44) - open/close 41561 (time 1709788587.23 total 1640.98 last 27.39) - open/close 41823 (time 1709788597.23 total 1650.98 last 26.20) - open/close 42097 (time 1709788607.25 total 1661.00 last 27.34) - open/close 42369 (time 1709788617.29 total 1671.04 last 27.10) - open/close 42646 (time 1709788627.32 total 1681.07 last 27.62) - open/close 42903 (time 1709788637.35 total 1691.10 last 25.64) - open/close 43152 (time 1709788647.37 total 1701.12 last 24.84) - open/close 43379 (time 1709788657.42 total 1711.17 last 22.59) - open/close 43610 (time 1709788667.43 total 1721.18 last 23.07) - open/close 43854 (time 1709788677.45 total 1731.20 last 24.36) - open/close 44095 (time 1709788687.48 total 1741.23 last 24.03) - open/close 44334 (time 1709788697.50 total 1751.25 last 23.85) - open/close 44578 (time 1709788707.53 total 1761.28 last 24.32) - open/close 44816 (time 1709788717.54 total 1771.29 last 23.79) - open/close 45074 (time 1709788727.57 total 1781.32 last 25.72) - open/close 45337 (time 1709788737.60 total 1791.35 last 26.22) - open/close 45604 (time 1709788747.61 total 1801.36 last 26.66) - open/close 45848 (time 1709788757.65 total 1811.40 last 24.30) - open/close 46075 (time 1709788767.66 total 1821.41 last 22.68) - open/close 46284 (time 1709788777.68 total 1831.43 last 20.87) - open/close 46503 (time 1709788787.72 total 1841.47 last 21.82) - open/close 46728 (time 1709788797.76 total 1851.51 last 22.41) - open/close 46942 (time 1709788807.79 total 1861.54 last 21.34) - open/close 47167 (time 1709788817.82 total 1871.57 last 22.44) - open/close 47373 (time 1709788827.82 total 1881.57 last 20.60) - open/close 47591 (time 1709788837.82 total 1891.57 last 21.79) - open/close 47806 (time 1709788847.87 total 1901.62 last 21.40) - open/close 47997 (time 1709788857.90 total 1911.65 last 19.05) - open/close 48199 (time 1709788867.90 total 1921.65 last 20.19) - open/close 48417 (time 1709788877.93 total 1931.68 last 21.73) - open/close 48632 (time 1709788887.96 total 1941.71 last 21.45) - open/close 48872 (time 1709788897.96 total 1951.71 last 23.99) - open/close 49094 (time 1709788907.96 total 1961.71 last 22.19) - open/close 49321 (time 1709788918.00 total 1971.75 last 22.61) - open/close 49539 (time 1709788928.04 total 1981.79 last 21.71) - open/close 49754 (time 1709788938.08 total 1991.83 last 21.41) - open/close 49961 (time 1709788948.11 total 2001.86 last 20.64) total: 50000 open/close in 2003.60 seconds: 24.96 ops/second - unlinked 0 (time 1709788951 ; total 0 ; last 0) - unlinked 10000 (time 1709789461 ; total 510 ; last 510) - unlinked 20000 (time 1709789921 ; total 970 ; last 460) - unlinked 30000 (time 1709790238 ; total 1287 ; last 317) - unlinked 40000 (time 1709790376 ; total 1425 ; last 138) total: 50000 unlinks in 1467 seconds: 34.083164 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg338-server unloading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2