== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:45 (1709786745) start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 247 (time 1709786774.54 total 10.50 last 23.52) - open/close 601 (time 1709786784.75 total 20.71 last 34.68) - open/close 904 (time 1709786795.40 total 31.37 last 28.44) - open/close 1044 (time 1709786805.40 total 41.37 last 14.00) - open/close 1317 (time 1709786815.85 total 51.82 last 26.13) - open/close 1548 (time 1709786826.49 total 62.46 last 21.71) - open/close 1863 (time 1709786837.39 total 73.36 last 28.89) - open/close 2113 (time 1709786847.65 total 83.62 last 24.37) - open/close 2538 (time 1709786858.39 total 94.36 last 39.57) - open/close 2799 (time 1709786868.40 total 104.37 last 26.08) - open/close 2904 (time 1709786878.53 total 114.50 last 10.36) - open/close 3004 (time 1709786889.63 total 125.60 last 9.01) - open/close 3194 (time 1709786900.39 total 136.36 last 17.65) - open/close 3491 (time 1709786910.43 total 146.39 last 29.60) - open/close 3661 (time 1709786920.43 total 156.40 last 16.99) - open/close 3801 (time 1709786930.47 total 166.44 last 13.95) - open/close 3964 (time 1709786941.16 total 177.13 last 15.25) - open/close 4309 (time 1709786951.18 total 187.15 last 34.43) - open/close 4446 (time 1709786963.16 total 199.12 last 11.44) - open/close 4657 (time 1709786973.73 total 209.70 last 19.96) - open/close 4836 (time 1709786983.73 total 219.70 last 17.89) - open/close 5179 (time 1709786994.16 total 230.13 last 32.88) - open/close 5445 (time 1709787005.16 total 241.13 last 24.19) - open/close 5572 (time 1709787016.05 total 252.01 last 11.67) - open/close 5794 (time 1709787026.60 total 262.57 last 21.04) - open/close 6052 (time 1709787037.33 total 273.29 last 24.05) - open/close 6266 (time 1709787047.77 total 283.74 last 20.49) - open/close 6451 (time 1709787057.81 total 293.78 last 18.43) - open/close 6615 (time 1709787068.25 total 304.21 last 15.72) - open/close 6716 (time 1709787078.25 total 314.22 last 10.09) - open/close 6890 (time 1709787088.28 total 324.24 last 17.36) - open/close 7092 (time 1709787098.28 total 334.25 last 20.19) - open/close 7407 (time 1709787108.29 total 344.26 last 31.47) - open/close 7576 (time 1709787118.63 total 354.60 last 16.34) - open/close 7862 (time 1709787128.81 total 364.78 last 28.09) - open/close 8012 (time 1709787139.34 total 375.30 last 14.25) - open/close 8208 (time 1709787149.39 total 385.36 last 19.50) - open/close 8476 (time 1709787159.39 total 395.36 last 26.79) - open/close 8692 (time 1709787169.92 total 405.89 last 20.52) - open/close 8970 (time 1709787179.92 total 415.89 last 27.79) - open/close 9283 (time 1709787190.85 total 426.81 last 28.66) - open/close 9454 (time 1709787200.87 total 436.84 last 17.06) - open/close 9640 (time 1709787211.48 total 447.45 last 17.53) - open/close 9813 (time 1709787222.15 total 458.12 last 16.21) - open/close 10000 (time 1709787231.29 total 467.26 last 20.46) - open/close 10502 (time 1709787250.38 total 486.35 last 26.30) - open/close 11108 (time 1709787260.39 total 496.35 last 60.55) - open/close 11703 (time 1709787270.39 total 506.36 last 59.49) - open/close 12208 (time 1709787280.50 total 516.47 last 49.93) - open/close 12747 (time 1709787290.53 total 526.49 last 53.77) - open/close 13275 (time 1709787300.54 total 536.50 last 52.76) - open/close 13720 (time 1709787310.54 total 546.50 last 44.49) - open/close 14268 (time 1709787320.55 total 556.52 last 54.70) - open/close 14756 (time 1709787330.58 total 566.54 last 48.70) - open/close 15184 (time 1709787340.59 total 576.55 last 42.75) - open/close 15666 (time 1709787350.59 total 586.56 last 48.18) - open/close 16173 (time 1709787360.61 total 596.58 last 50.60) - open/close 16606 (time 1709787370.63 total 606.60 last 43.23) - open/close 17043 (time 1709787380.64 total 616.61 last 43.65) - open/close 17507 (time 1709787390.65 total 626.61 last 46.37) - open/close 17946 (time 1709787400.66 total 636.63 last 43.84) - open/close 18346 (time 1709787410.66 total 646.63 last 39.98) - open/close 18730 (time 1709787420.70 total 656.66 last 38.27) - open/close 18825 (time 1709787435.34 total 671.31 last 6.49) - open/close 19197 (time 1709787445.37 total 681.33 last 37.11) - open/close 19259 (time 1709787458.13 total 694.10 last 4.86) - open/close 19600 (time 1709787468.14 total 704.10 last 34.09) - open/close 19699 (time 1709787481.85 total 717.82 last 7.22) - open/close 20000 (time 1709787489.78 total 725.75 last 37.95) - open/close 20341 (time 1709787499.82 total 735.79 last 33.97) - open/close 20663 (time 1709787509.85 total 745.81 last 32.12) - open/close 20959 (time 1709787519.86 total 755.83 last 29.56) - open/close 21276 (time 1709787529.88 total 765.85 last 31.64) - open/close 21625 (time 1709787539.91 total 775.87 last 34.81) - open/close 21985 (time 1709787549.93 total 785.90 last 35.90) - open/close 22326 (time 1709787559.96 total 795.93 last 34.00) - open/close 22694 (time 1709787569.98 total 805.94 last 36.75) - open/close 23071 (time 1709787579.98 total 815.95 last 37.69) - open/close 23321 (time 1709787597.42 total 833.38 last 14.34) - open/close 23698 (time 1709787607.45 total 843.42 last 37.56) - open/close 24059 (time 1709787617.47 total 853.44 last 36.04) - open/close 24395 (time 1709787627.49 total 863.46 last 33.52) - open/close 24737 (time 1709787637.52 total 873.48 last 34.12) - open/close 25100 (time 1709787647.53 total 883.49 last 36.27) - open/close 25472 (time 1709787657.53 total 893.50 last 37.18) - open/close 25835 (time 1709787667.54 total 903.50 last 36.29) - open/close 26154 (time 1709787677.54 total 913.50 last 31.90) - open/close 26450 (time 1709787687.54 total 923.51 last 29.59) - open/close 26750 (time 1709787697.55 total 933.52 last 29.97) - open/close 27037 (time 1709787707.56 total 943.53 last 28.68) - open/close 27327 (time 1709787717.57 total 953.54 last 28.96) - open/close 27635 (time 1709787727.60 total 963.56 last 30.72) - open/close 27983 (time 1709787737.61 total 973.58 last 34.74) - open/close 28366 (time 1709787747.64 total 983.60 last 38.22) - open/close 28690 (time 1709787757.65 total 993.62 last 32.36) - open/close 28983 (time 1709787767.68 total 1003.65 last 29.21) - open/close 29268 (time 1709787777.68 total 1013.65 last 28.48) - open/close 29528 (time 1709787787.69 total 1023.66 last 25.97) - open/close 29782 (time 1709787797.70 total 1033.67 last 25.38) - open/close 30000 (time 1709787804.55 total 1040.51 last 31.86) - open/close 30298 (time 1709787814.57 total 1050.54 last 29.72) - open/close 30579 (time 1709787824.61 total 1060.58 last 28.00) - open/close 30861 (time 1709787834.64 total 1070.60 last 28.12) - open/close 31179 (time 1709787844.65 total 1080.62 last 31.76) - open/close 31500 (time 1709787854.66 total 1090.63 last 32.05) - open/close 31852 (time 1709787864.68 total 1100.65 last 35.15) - open/close 32190 (time 1709787874.69 total 1110.65 last 33.77) - open/close 32536 (time 1709787884.71 total 1120.67 last 34.53) - open/close 32876 (time 1709787894.74 total 1130.70 last 33.89) - open/close 33203 (time 1709787904.75 total 1140.72 last 32.66) - open/close 33511 (time 1709787914.76 total 1150.73 last 30.76) - open/close 33831 (time 1709787924.78 total 1160.74 last 31.96) - open/close 34139 (time 1709787934.79 total 1170.76 last 30.76) - open/close 34454 (time 1709787944.80 total 1180.76 last 31.47) - open/close 34765 (time 1709787954.82 total 1190.79 last 31.03) - open/close 35070 (time 1709787964.82 total 1200.79 last 30.48) - open/close 35378 (time 1709787974.85 total 1210.82 last 30.71) - open/close 35681 (time 1709787984.86 total 1220.82 last 30.29) - open/close 35989 (time 1709787994.87 total 1230.84 last 30.76) - open/close 36303 (time 1709788004.90 total 1240.86 last 31.32) - open/close 36610 (time 1709788014.90 total 1250.86 last 30.69) - open/close 36915 (time 1709788024.91 total 1260.88 last 30.46) - open/close 37212 (time 1709788034.95 total 1270.91 last 29.58) - open/close 37503 (time 1709788044.97 total 1280.94 last 29.03) - open/close 37784 (time 1709788054.99 total 1290.96 last 28.04) - open/close 38076 (time 1709788065.02 total 1300.99 last 29.12) - open/close 38360 (time 1709788075.05 total 1311.01 last 28.33) - open/close 38651 (time 1709788085.06 total 1321.03 last 29.05) - open/close 38943 (time 1709788095.09 total 1331.06 last 29.11) - open/close 39232 (time 1709788105.11 total 1341.08 last 28.85) - open/close 39526 (time 1709788115.13 total 1351.10 last 29.35) - open/close 39812 (time 1709788125.14 total 1361.11 last 28.56) - open/close 40000 (time 1709788131.92 total 1367.89 last 27.74) - open/close 40268 (time 1709788141.94 total 1377.91 last 26.74) - open/close 40543 (time 1709788151.99 total 1387.96 last 27.37) - open/close 40783 (time 1709788162.04 total 1398.00 last 23.89) - open/close 41051 (time 1709788172.05 total 1408.02 last 26.76) - open/close 41317 (time 1709788182.08 total 1418.05 last 26.54) - open/close 41592 (time 1709788192.11 total 1428.08 last 27.41) - open/close 41859 (time 1709788202.15 total 1438.12 last 26.60) - open/close 42115 (time 1709788212.18 total 1448.15 last 25.51) - open/close 42360 (time 1709788222.20 total 1458.17 last 24.46) - open/close 42619 (time 1709788232.22 total 1468.18 last 25.85) - open/close 42862 (time 1709788242.25 total 1478.21 last 24.23) - open/close 43103 (time 1709788252.25 total 1488.21 last 24.10) - open/close 43368 (time 1709788262.26 total 1498.23 last 26.46) - open/close 43619 (time 1709788272.28 total 1508.25 last 25.06) - open/close 43874 (time 1709788282.29 total 1518.25 last 25.48) - open/close 44135 (time 1709788292.31 total 1528.28 last 26.03) - open/close 44385 (time 1709788302.33 total 1538.30 last 24.96) - open/close 44627 (time 1709788312.34 total 1548.30 last 24.18) - open/close 44867 (time 1709788322.38 total 1558.34 last 23.91) - open/close 45115 (time 1709788332.39 total 1568.36 last 24.76) - open/close 45352 (time 1709788342.45 total 1578.41 last 23.57) - open/close 45582 (time 1709788352.48 total 1588.45 last 22.92) - open/close 45812 (time 1709788362.50 total 1598.47 last 22.96) - open/close 46041 (time 1709788372.54 total 1608.51 last 22.80) - open/close 46270 (time 1709788382.56 total 1618.52 last 22.86) - open/close 46498 (time 1709788392.59 total 1628.56 last 22.72) - open/close 46731 (time 1709788402.62 total 1638.58 last 23.25) - open/close 46962 (time 1709788412.64 total 1648.60 last 23.05) - open/close 47191 (time 1709788422.67 total 1658.64 last 22.82) - open/close 47420 (time 1709788432.69 total 1668.66 last 22.86) - open/close 47652 (time 1709788442.70 total 1678.67 last 23.18) - open/close 47884 (time 1709788452.70 total 1688.67 last 23.20) - open/close 48120 (time 1709788462.74 total 1698.71 last 23.51) - open/close 48351 (time 1709788472.74 total 1708.71 last 23.09) - open/close 48579 (time 1709788482.76 total 1718.73 last 22.76) - open/close 48818 (time 1709788492.80 total 1728.77 last 23.81) - open/close 49052 (time 1709788502.80 total 1738.77 last 23.40) - open/close 49295 (time 1709788512.83 total 1748.79 last 24.25) - open/close 49512 (time 1709788522.86 total 1758.83 last 21.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1767.11 seconds: 28.12 ops/second - unlinked 0 (time 1709788532 ; total 0 ; last 0) - unlinked 10000 (time 1709789019 ; total 487 ; last 487) - unlinked 20000 (time 1709789423 ; total 891 ; last 404) - unlinked 30000 (time 1709789730 ; total 1198 ; last 307) - unlinked 40000 (time 1709789900 ; total 1368 ; last 170) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1430 seconds: 34.753845 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2