== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:50 (1709786810) start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1709786842.53 total 10.29 last 18.75) - open/close 443 (time 1709786853.25 total 21.01 last 23.33) - open/close 698 (time 1709786863.26 total 31.02 last 25.47) - open/close 1033 (time 1709786873.75 total 41.51 last 31.94) - open/close 1236 (time 1709786884.32 total 52.08 last 19.20) - open/close 1448 (time 1709786894.92 total 62.68 last 20.01) - open/close 1683 (time 1709786905.71 total 73.47 last 21.77) - open/close 1981 (time 1709786916.09 total 83.85 last 28.73) - open/close 2169 (time 1709786926.84 total 94.60 last 17.48) - open/close 2373 (time 1709786939.76 total 107.52 last 15.80) - open/close 2597 (time 1709786949.77 total 117.53 last 22.38) - open/close 2746 (time 1709786960.19 total 127.95 last 14.30) - open/close 3041 (time 1709786970.21 total 137.97 last 29.45) - open/close 3260 (time 1709786983.17 total 150.93 last 16.89) - open/close 3473 (time 1709786994.09 total 161.85 last 19.51) - open/close 3730 (time 1709787005.03 total 172.79 last 23.50) - open/close 3933 (time 1709787015.50 total 183.26 last 19.38) - open/close 4159 (time 1709787026.16 total 193.92 last 21.21) - open/close 4338 (time 1709787037.16 total 204.92 last 16.26) - open/close 4580 (time 1709787047.94 total 215.70 last 22.45) - open/close 4756 (time 1709787058.58 total 226.34 last 16.54) - open/close 4939 (time 1709787068.59 total 236.35 last 18.29) - open/close 5109 (time 1709787078.59 total 246.35 last 16.99) - open/close 5270 (time 1709787088.62 total 256.38 last 16.06) - open/close 5400 (time 1709787100.89 total 268.65 last 10.60) - open/close 5576 (time 1709787111.89 total 279.65 last 16.01) - open/close 5776 (time 1709787122.54 total 290.30 last 18.77) - open/close 5943 (time 1709787132.69 total 300.45 last 16.46) - open/close 6093 (time 1709787142.69 total 310.45 last 15.00) - open/close 6284 (time 1709787153.40 total 321.16 last 17.82) - open/close 6426 (time 1709787163.41 total 331.17 last 14.20) - open/close 6474 (time 1709787175.35 total 343.11 last 4.02) - open/close 6611 (time 1709787195.04 total 362.80 last 6.96) - open/close 6770 (time 1709787205.21 total 372.97 last 15.62) - open/close 6974 (time 1709787217.80 total 385.56 last 16.21) - open/close 7222 (time 1709787228.40 total 396.17 last 23.38) - open/close 7426 (time 1709787238.73 total 406.49 last 19.77) - open/close 7585 (time 1709787248.74 total 416.50 last 15.87) - open/close 7709 (time 1709787258.76 total 426.52 last 12.38) - open/close 7877 (time 1709787268.98 total 436.74 last 16.43) - open/close 8025 (time 1709787279.53 total 447.29 last 14.03) - open/close 8174 (time 1709787289.55 total 457.31 last 14.87) - open/close 8371 (time 1709787299.97 total 467.73 last 18.89) - open/close 8523 (time 1709787310.64 total 478.40 last 14.25) - open/close 8742 (time 1709787321.49 total 489.25 last 20.18) - open/close 8900 (time 1709787331.72 total 499.48 last 15.45) - open/close 9060 (time 1709787341.88 total 509.64 last 15.75) - open/close 9203 (time 1709787351.91 total 519.67 last 14.25) - open/close 9388 (time 1709787361.92 total 529.68 last 18.50) - open/close 9529 (time 1709787372.05 total 539.81 last 13.91) - open/close 9709 (time 1709787382.08 total 549.84 last 17.95) - open/close 10000 (time 1709787391.38 total 559.14 last 31.30) - open/close 10397 (time 1709787401.43 total 569.19 last 39.50) - open/close 10803 (time 1709787411.43 total 579.19 last 40.59) - open/close 11238 (time 1709787421.43 total 589.19 last 43.48) - open/close 11692 (time 1709787431.45 total 599.21 last 45.32) - open/close 11861 (time 1709787446.53 total 614.29 last 11.21) - open/close 12258 (time 1709787456.56 total 624.32 last 39.58) - open/close 12662 (time 1709787466.56 total 634.32 last 40.38) - open/close 12987 (time 1709787486.57 total 654.33 last 16.25) - open/close 13420 (time 1709787496.61 total 664.37 last 43.11) - open/close 13841 (time 1709787506.61 total 674.37 last 42.09) - open/close 14204 (time 1709787516.64 total 684.40 last 36.20) - open/close 14564 (time 1709787526.65 total 694.41 last 35.98) - open/close 14918 (time 1709787536.65 total 704.41 last 35.39) - open/close 15226 (time 1709787546.68 total 714.44 last 30.71) - open/close 15535 (time 1709787556.71 total 724.47 last 30.80) - open/close 15882 (time 1709787566.72 total 734.48 last 34.69) - open/close 16248 (time 1709787576.72 total 744.48 last 36.59) - open/close 16574 (time 1709787586.76 total 754.52 last 32.48) - open/close 16914 (time 1709787596.76 total 764.52 last 33.98) - open/close 17245 (time 1709787606.77 total 774.53 last 33.07) - open/close 17560 (time 1709787616.79 total 784.55 last 31.44) - open/close 17894 (time 1709787626.80 total 794.56 last 33.37) - open/close 18254 (time 1709787636.81 total 804.57 last 35.98) - open/close 18604 (time 1709787646.82 total 814.58 last 34.95) - open/close 18900 (time 1709787656.84 total 824.60 last 29.55) - open/close 19232 (time 1709787666.87 total 834.63 last 33.11) - open/close 19528 (time 1709787676.88 total 844.64 last 29.56) - open/close 19846 (time 1709787686.89 total 854.65 last 31.77) - open/close 20000 (time 1709787692.55 total 860.31 last 27.19) - open/close 20314 (time 1709787702.58 total 870.34 last 31.32) - open/close 20625 (time 1709787712.59 total 880.35 last 31.08) - open/close 20889 (time 1709787722.65 total 890.41 last 26.23) - open/close 21177 (time 1709787732.66 total 900.42 last 28.79) - open/close 21466 (time 1709787742.66 total 910.42 last 28.88) - open/close 21747 (time 1709787752.67 total 920.43 last 28.07) - open/close 22019 (time 1709787762.72 total 930.48 last 27.08) - open/close 22290 (time 1709787772.75 total 940.51 last 27.03) - open/close 22588 (time 1709787782.76 total 950.52 last 29.75) - open/close 22880 (time 1709787792.78 total 960.54 last 29.15) - open/close 23185 (time 1709787802.79 total 970.55 last 30.49) - open/close 23194 (time 1709787814.13 total 981.89 last 0.79) - open/close 23455 (time 1709787824.17 total 991.93 last 25.99) - open/close 23715 (time 1709787843.72 total 1011.48 last 13.30) - open/close 23958 (time 1709787853.74 total 1021.50 last 24.24) - open/close 24204 (time 1709787863.80 total 1031.56 last 24.47) - open/close 24461 (time 1709787873.81 total 1041.57 last 25.67) - open/close 24723 (time 1709787883.82 total 1051.58 last 26.16) - open/close 24969 (time 1709787893.86 total 1061.62 last 24.51) - open/close 25196 (time 1709787903.94 total 1071.70 last 22.52) - open/close 25463 (time 1709787913.98 total 1081.74 last 26.60) - open/close 25708 (time 1709787924.02 total 1091.78 last 24.39) - open/close 25957 (time 1709787934.03 total 1101.79 last 24.87) - open/close 26228 (time 1709787944.05 total 1111.81 last 27.06) - open/close 26538 (time 1709787954.07 total 1121.83 last 30.93) - open/close 26872 (time 1709787964.08 total 1131.84 last 33.38) - open/close 27175 (time 1709787974.08 total 1141.84 last 30.29) - open/close 27483 (time 1709787984.10 total 1151.86 last 30.74) - open/close 27804 (time 1709787994.12 total 1161.88 last 32.02) - open/close 28163 (time 1709788004.13 total 1171.89 last 35.85) - open/close 28537 (time 1709788014.14 total 1181.90 last 37.40) - open/close 28934 (time 1709788024.15 total 1191.91 last 39.65) - open/close 29327 (time 1709788034.17 total 1201.93 last 39.22) - open/close 29739 (time 1709788044.17 total 1211.93 last 41.18) - open/close 30000 (time 1709788050.75 total 1218.51 last 39.68) - open/close 30394 (time 1709788060.75 total 1228.51 last 39.39) - open/close 30762 (time 1709788070.75 total 1238.51 last 36.80) - open/close 31141 (time 1709788080.76 total 1248.52 last 37.89) - open/close 31531 (time 1709788090.77 total 1258.53 last 38.96) - open/close 31893 (time 1709788100.79 total 1268.55 last 36.12) - open/close 32260 (time 1709788110.80 total 1278.56 last 36.64) - open/close 32627 (time 1709788120.81 total 1288.57 last 36.70) - open/close 32988 (time 1709788130.81 total 1298.57 last 36.07) - open/close 33348 (time 1709788140.82 total 1308.58 last 35.99) - open/close 33703 (time 1709788150.83 total 1318.59 last 35.45) - open/close 34060 (time 1709788160.84 total 1328.60 last 35.66) - open/close 34429 (time 1709788170.86 total 1338.62 last 36.83) - open/close 34790 (time 1709788180.88 total 1348.64 last 36.04) - open/close 35149 (time 1709788190.90 total 1358.66 last 35.83) - open/close 35502 (time 1709788200.92 total 1368.68 last 35.23) - open/close 35853 (time 1709788210.92 total 1378.68 last 35.08) - open/close 36209 (time 1709788220.93 total 1388.69 last 35.59) - open/close 36558 (time 1709788230.94 total 1398.70 last 34.84) - open/close 36895 (time 1709788240.95 total 1408.71 last 33.68) - open/close 37238 (time 1709788250.96 total 1418.72 last 34.25) - open/close 37583 (time 1709788260.99 total 1428.75 last 34.42) - open/close 37907 (time 1709788271.01 total 1438.77 last 32.32) - open/close 38230 (time 1709788281.02 total 1448.78 last 32.28) - open/close 38533 (time 1709788291.04 total 1458.80 last 30.22) - open/close 38848 (time 1709788301.06 total 1468.82 last 31.46) - open/close 39157 (time 1709788311.06 total 1478.82 last 30.90) - open/close 39468 (time 1709788321.09 total 1488.85 last 31.01) - open/close 39773 (time 1709788331.12 total 1498.88 last 30.41) - open/close 40000 (time 1709788338.96 total 1506.72 last 28.96) - open/close 40296 (time 1709788348.97 total 1516.73 last 29.57) - open/close 40590 (time 1709788358.98 total 1526.74 last 29.38) - open/close 40882 (time 1709788368.98 total 1536.74 last 29.19) - open/close 41168 (time 1709788378.99 total 1546.75 last 28.58) - open/close 41446 (time 1709788389.00 total 1556.76 last 27.77) - open/close 41736 (time 1709788399.02 total 1566.78 last 28.92) - open/close 42039 (time 1709788409.05 total 1576.81 last 30.21) - open/close 42327 (time 1709788419.05 total 1586.81 last 28.80) - open/close 42623 (time 1709788429.05 total 1596.82 last 29.59) - open/close 42926 (time 1709788439.07 total 1606.83 last 30.24) - open/close 43212 (time 1709788449.11 total 1616.87 last 28.50) - open/close 43504 (time 1709788459.12 total 1626.88 last 29.18) - open/close 43786 (time 1709788469.13 total 1636.89 last 28.16) - open/close 44083 (time 1709788479.14 total 1646.90 last 29.66) - open/close 44375 (time 1709788489.16 total 1656.92 last 29.15) - open/close 44670 (time 1709788499.18 total 1666.94 last 29.44) - open/close 44959 (time 1709788509.19 total 1676.95 last 28.87) - open/close 45254 (time 1709788519.22 total 1686.98 last 29.41) - open/close 45542 (time 1709788529.25 total 1697.01 last 28.72) - open/close 45828 (time 1709788539.26 total 1707.02 last 28.56) - open/close 46108 (time 1709788549.28 total 1717.04 last 27.94) - open/close 46398 (time 1709788559.29 total 1727.05 last 28.97) - open/close 46682 (time 1709788569.31 total 1737.07 last 28.35) - open/close 46971 (time 1709788579.32 total 1747.08 last 28.86) - open/close 47263 (time 1709788589.33 total 1757.09 last 29.17) - open/close 47557 (time 1709788599.36 total 1767.12 last 29.32) - open/close 47848 (time 1709788609.38 total 1777.14 last 29.05) - open/close 48141 (time 1709788619.38 total 1787.14 last 29.30) - open/close 48421 (time 1709788629.41 total 1797.17 last 27.91) - open/close 48690 (time 1709788639.42 total 1807.18 last 26.86) - open/close 48955 (time 1709788649.46 total 1817.22 last 26.41) - open/close 49227 (time 1709788659.48 total 1827.24 last 27.14) - open/close 49496 (time 1709788669.50 total 1837.26 last 26.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1844.66 seconds: 26.94 ops/second - unlinked 0 (time 1709788678 ; total 1 ; last 1) - unlinked 10000 (time 1709789122 ; total 445 ; last 444) - unlinked 20000 (time 1709789510 ; total 833 ; last 388) - unlinked 30000 (time 1709789777 ; total 1100 ; last 267) - unlinked 40000 (time 1709789912 ; total 1235 ; last 135) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1279 seconds: 38.856918 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2