== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:56 (1709786816) start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1709786850.71 total 10.76 last 21.75) - open/close 432 (time 1709786860.73 total 20.78 last 19.76) - open/close 588 (time 1709786871.58 total 31.63 last 14.38) - open/close 772 (time 1709786881.78 total 41.83 last 18.03) - open/close 1011 (time 1709786891.80 total 51.84 last 23.87) - open/close 1313 (time 1709786902.67 total 62.72 last 27.77) - open/close 1496 (time 1709786912.90 total 72.94 last 17.90) - open/close 1725 (time 1709786922.90 total 82.95 last 22.90) - open/close 1955 (time 1709786933.72 total 93.77 last 21.25) - open/close 2167 (time 1709786943.73 total 103.77 last 21.19) - open/close 2425 (time 1709786954.34 total 114.39 last 24.30) - open/close 2579 (time 1709786965.90 total 125.94 last 13.33) - open/close 2817 (time 1709786976.33 total 136.38 last 22.81) - open/close 3022 (time 1709786987.05 total 147.10 last 19.12) - open/close 3189 (time 1709786997.46 total 157.51 last 16.04) - open/close 3382 (time 1709787008.47 total 168.52 last 17.53) - open/close 3572 (time 1709787018.47 total 178.52 last 19.00) - open/close 3704 (time 1709787028.67 total 188.72 last 12.94) - open/close 3888 (time 1709787039.59 total 199.64 last 16.84) - open/close 4048 (time 1709787050.40 total 210.45 last 14.80) - open/close 4227 (time 1709787060.69 total 220.74 last 17.41) - open/close 4335 (time 1709787070.70 total 230.75 last 10.79) - open/close 4472 (time 1709787080.96 total 241.01 last 13.35) - open/close 4652 (time 1709787091.94 total 251.98 last 16.41) - open/close 4757 (time 1709787102.63 total 262.67 last 9.82) - open/close 4914 (time 1709787113.35 total 273.40 last 14.64) - open/close 5070 (time 1709787123.36 total 283.41 last 15.59) - open/close 5204 (time 1709787133.40 total 293.44 last 13.35) - open/close 5303 (time 1709787143.41 total 303.46 last 9.89) - open/close 5434 (time 1709787153.93 total 313.98 last 12.45) - open/close 5541 (time 1709787163.94 total 323.99 last 10.69) - open/close 5696 (time 1709787173.97 total 334.02 last 15.46) - open/close 5740 (time 1709787190.81 total 350.86 last 2.61) - open/close 5894 (time 1709787201.06 total 361.11 last 15.02) - open/close 6211 (time 1709787211.70 total 371.74 last 29.80) - open/close 6352 (time 1709787222.26 total 382.31 last 13.35) - open/close 6426 (time 1709787232.86 total 392.91 last 6.98) - open/close 6632 (time 1709787243.15 total 403.20 last 20.02) - open/close 6815 (time 1709787253.16 total 413.21 last 18.29) - open/close 7054 (time 1709787263.17 total 423.22 last 23.87) - open/close 7230 (time 1709787274.16 total 434.21 last 16.01) - open/close 7370 (time 1709787284.56 total 444.61 last 13.46) - open/close 7544 (time 1709787295.53 total 455.58 last 15.87) - open/close 7804 (time 1709787305.54 total 465.59 last 25.96) - open/close 7930 (time 1709787316.50 total 476.55 last 11.50) - open/close 8141 (time 1709787326.51 total 486.56 last 21.08) - open/close 8432 (time 1709787336.86 total 496.91 last 28.12) - open/close 8687 (time 1709787346.87 total 506.91 last 25.48) - open/close 8821 (time 1709787356.88 total 516.92 last 13.39) - open/close 9024 (time 1709787367.00 total 527.05 last 20.04) - open/close 9222 (time 1709787377.04 total 537.08 last 19.74) - open/close 9410 (time 1709787387.05 total 547.10 last 18.78) - open/close 9722 (time 1709787397.06 total 557.11 last 31.14) - open/close 10000 (time 1709787403.77 total 563.82 last 41.48) - open/close 10456 (time 1709787413.78 total 573.83 last 45.55) - open/close 10899 (time 1709787423.78 total 583.83 last 44.29) - open/close 11338 (time 1709787433.79 total 593.84 last 43.86) - open/close 11365 (time 1709787445.69 total 605.74 last 2.27) - open/close 11806 (time 1709787455.69 total 615.74 last 44.10) - open/close 12249 (time 1709787465.72 total 625.77 last 44.17) - open/close 12595 (time 1709787475.75 total 635.80 last 34.51) - open/close 13052 (time 1709787485.76 total 645.80 last 45.68) - open/close 13499 (time 1709787495.77 total 655.82 last 44.62) - open/close 13909 (time 1709787505.80 total 665.85 last 40.90) - open/close 14293 (time 1709787515.81 total 675.86 last 38.35) - open/close 14669 (time 1709787525.81 total 685.86 last 37.60) - open/close 15026 (time 1709787535.85 total 695.89 last 35.57) - open/close 15391 (time 1709787545.87 total 705.92 last 36.41) - open/close 15763 (time 1709787555.87 total 715.92 last 37.20) - open/close 16115 (time 1709787565.90 total 725.95 last 35.11) - open/close 16486 (time 1709787575.93 total 735.98 last 36.98) - open/close 16863 (time 1709787585.94 total 745.99 last 37.66) - open/close 17216 (time 1709787595.98 total 756.02 last 35.18) - open/close 17578 (time 1709787605.98 total 766.03 last 36.18) - open/close 17929 (time 1709787616.00 total 776.05 last 35.05) - open/close 18275 (time 1709787626.02 total 786.07 last 34.53) - open/close 18650 (time 1709787636.05 total 796.10 last 37.37) - open/close 19022 (time 1709787646.05 total 806.10 last 37.19) - open/close 19298 (time 1709787656.06 total 816.10 last 27.60) - open/close 19635 (time 1709787666.08 total 826.13 last 33.63) - open/close 19928 (time 1709787676.09 total 836.13 last 29.27) - open/close 20000 (time 1709787678.54 total 838.59 last 29.37) - open/close 20316 (time 1709787688.54 total 848.59 last 31.59) - open/close 20646 (time 1709787698.56 total 858.61 last 32.94) - open/close 20959 (time 1709787708.56 total 868.61 last 31.30) - open/close 21238 (time 1709787718.57 total 878.62 last 27.88) - open/close 21501 (time 1709787728.58 total 888.63 last 26.27) - open/close 21795 (time 1709787738.60 total 898.64 last 29.36) - open/close 22076 (time 1709787748.60 total 908.65 last 28.09) - open/close 22360 (time 1709787758.61 total 918.66 last 28.36) - open/close 22650 (time 1709787768.65 total 928.70 last 28.89) - open/close 22941 (time 1709787778.66 total 938.71 last 29.06) - open/close 23217 (time 1709787799.24 total 959.29 last 13.41) - open/close 23493 (time 1709787809.25 total 969.30 last 27.58) - open/close 23783 (time 1709787819.27 total 979.31 last 28.95) - open/close 24075 (time 1709787829.28 total 989.33 last 29.15) - open/close 24372 (time 1709787839.31 total 999.35 last 29.63) - open/close 24628 (time 1709787849.35 total 1009.40 last 25.48) - open/close 24878 (time 1709787859.35 total 1019.40 last 25.00) - open/close 25109 (time 1709787869.36 total 1029.41 last 23.07) - open/close 25386 (time 1709787879.37 total 1039.41 last 27.70) - open/close 25643 (time 1709787889.37 total 1049.42 last 25.68) - open/close 25881 (time 1709787899.41 total 1059.46 last 23.72) - open/close 26123 (time 1709787909.43 total 1069.48 last 24.15) - open/close 26388 (time 1709787919.44 total 1079.49 last 26.47) - open/close 26637 (time 1709787929.44 total 1089.49 last 24.89) - open/close 26906 (time 1709787939.44 total 1099.49 last 26.90) - open/close 27186 (time 1709787949.46 total 1109.50 last 27.96) - open/close 27504 (time 1709787959.46 total 1119.51 last 31.80) - open/close 27790 (time 1709787969.47 total 1129.52 last 28.56) - open/close 28085 (time 1709787979.48 total 1139.53 last 29.47) - open/close 28379 (time 1709787989.50 total 1149.55 last 29.36) - open/close 28706 (time 1709787999.50 total 1159.55 last 32.69) - open/close 29051 (time 1709788009.55 total 1169.60 last 34.35) - open/close 29424 (time 1709788019.55 total 1179.60 last 37.27) - open/close 29814 (time 1709788029.56 total 1189.61 last 38.96) - open/close 30000 (time 1709788034.26 total 1194.31 last 39.63) - open/close 30382 (time 1709788044.26 total 1204.31 last 38.18) - open/close 30762 (time 1709788054.28 total 1214.33 last 37.95) - open/close 31129 (time 1709788064.30 total 1224.35 last 36.60) - open/close 31474 (time 1709788074.32 total 1234.37 last 34.45) - open/close 31852 (time 1709788084.33 total 1244.38 last 37.74) - open/close 32212 (time 1709788094.34 total 1254.39 last 35.97) - open/close 32557 (time 1709788104.36 total 1264.41 last 34.44) - open/close 32905 (time 1709788114.36 total 1274.41 last 34.79) - open/close 33250 (time 1709788124.37 total 1284.42 last 34.47) - open/close 33588 (time 1709788134.40 total 1294.44 last 33.72) - open/close 33912 (time 1709788144.41 total 1304.46 last 32.35) - open/close 34237 (time 1709788154.43 total 1314.48 last 32.44) - open/close 34571 (time 1709788164.45 total 1324.50 last 33.33) - open/close 34908 (time 1709788174.46 total 1334.51 last 33.68) - open/close 35233 (time 1709788184.47 total 1344.52 last 32.45) - open/close 35559 (time 1709788194.47 total 1354.52 last 32.60) - open/close 35880 (time 1709788204.49 total 1364.54 last 32.03) - open/close 36206 (time 1709788214.50 total 1374.55 last 32.57) - open/close 36536 (time 1709788224.52 total 1384.57 last 32.95) - open/close 36858 (time 1709788234.54 total 1394.59 last 32.13) - open/close 37172 (time 1709788244.56 total 1404.61 last 31.33) - open/close 37488 (time 1709788254.58 total 1414.62 last 31.57) - open/close 37795 (time 1709788264.61 total 1424.65 last 30.61) - open/close 38103 (time 1709788274.61 total 1434.66 last 30.80) - open/close 38392 (time 1709788284.63 total 1444.68 last 28.84) - open/close 38667 (time 1709788294.64 total 1454.69 last 27.48) - open/close 38945 (time 1709788304.65 total 1464.70 last 27.76) - open/close 39221 (time 1709788314.68 total 1474.72 last 27.54) - open/close 39493 (time 1709788324.70 total 1484.75 last 27.14) - open/close 39757 (time 1709788334.73 total 1494.78 last 26.32) - open/close 40000 (time 1709788343.81 total 1503.86 last 26.75) - open/close 40279 (time 1709788353.84 total 1513.89 last 27.82) - open/close 40561 (time 1709788363.84 total 1523.89 last 28.19) - open/close 40838 (time 1709788373.85 total 1533.90 last 27.68) - open/close 41116 (time 1709788383.88 total 1543.93 last 27.71) - open/close 41375 (time 1709788393.91 total 1553.96 last 25.83) - open/close 41663 (time 1709788403.94 total 1563.99 last 28.72) - open/close 41947 (time 1709788413.96 total 1574.01 last 28.34) - open/close 42225 (time 1709788423.98 total 1584.03 last 27.74) - open/close 42508 (time 1709788433.99 total 1594.04 last 28.28) - open/close 42780 (time 1709788444.03 total 1604.07 last 27.10) - open/close 43033 (time 1709788454.03 total 1614.07 last 25.30) - open/close 43290 (time 1709788464.04 total 1624.09 last 25.65) - open/close 43465 (time 1709788482.05 total 1642.10 last 9.72) - open/close 43716 (time 1709788492.06 total 1652.10 last 25.09) - open/close 43978 (time 1709788502.06 total 1662.11 last 26.18) - open/close 44249 (time 1709788512.07 total 1672.12 last 27.08) - open/close 44543 (time 1709788522.10 total 1682.15 last 29.31) - open/close 44823 (time 1709788532.13 total 1692.18 last 27.93) - open/close 45106 (time 1709788542.16 total 1702.21 last 28.21) - open/close 45385 (time 1709788552.19 total 1712.24 last 27.82) - open/close 45663 (time 1709788562.22 total 1722.27 last 27.71) - open/close 45947 (time 1709788572.22 total 1732.27 last 28.39) - open/close 46225 (time 1709788582.23 total 1742.28 last 27.78) - open/close 46510 (time 1709788592.23 total 1752.28 last 28.49) - open/close 46771 (time 1709788602.23 total 1762.28 last 26.10) - open/close 47028 (time 1709788612.26 total 1772.31 last 25.64) - open/close 47288 (time 1709788622.28 total 1782.33 last 25.95) - open/close 47543 (time 1709788632.29 total 1792.34 last 25.46) - open/close 47791 (time 1709788642.33 total 1802.38 last 24.70) - open/close 48044 (time 1709788652.37 total 1812.42 last 25.20) - open/close 48295 (time 1709788662.41 total 1822.46 last 25.01) - open/close 48560 (time 1709788672.43 total 1832.48 last 26.44) - open/close 48819 (time 1709788682.46 total 1842.50 last 25.84) - open/close 49068 (time 1709788692.47 total 1852.52 last 24.86) - open/close 49325 (time 1709788702.47 total 1862.52 last 25.69) - open/close 49578 (time 1709788712.48 total 1872.53 last 25.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1877.28 seconds: 26.47 ops/second - unlinked 0 (time 1709788718 ; total 0 ; last 0) - unlinked 10000 (time 1709789184 ; total 466 ; last 466) - unlinked 20000 (time 1709789581 ; total 863 ; last 397) - unlinked 30000 (time 1709789866 ; total 1148 ; last 285) - unlinked 40000 (time 1709790013 ; total 1295 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1336 seconds: 37.199100 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2