== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:00 (1709786820) start mds service on oleg442-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg442-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg442-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock oleg442-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1709786867.48 total 10.38 last 24.48) - open/close 552 (time 1709786877.55 total 20.45 last 29.60) - open/close 796 (time 1709786888.21 total 31.11 last 22.88) - open/close 960 (time 1709786899.09 total 41.98 last 15.08) - open/close 1143 (time 1709786909.12 total 52.01 last 18.25) - open/close 1401 (time 1709786919.31 total 62.20 last 25.32) - open/close 1647 (time 1709786929.66 total 72.55 last 23.77) - open/close 1891 (time 1709786939.74 total 82.64 last 24.19) - open/close 2099 (time 1709786949.75 total 92.64 last 20.79) - open/close 2302 (time 1709786959.86 total 102.75 last 20.08) - open/close 2437 (time 1709786970.11 total 113.01 last 13.16) - open/close 2569 (time 1709786980.11 total 123.01 last 13.20) - open/close 2869 (time 1709786990.12 total 133.01 last 29.99) - open/close 3039 (time 1709787000.65 total 143.54 last 16.14) - open/close 3170 (time 1709787010.66 total 153.56 last 13.08) - open/close 3439 (time 1709787021.23 total 164.13 last 25.45) - open/close 3681 (time 1709787031.24 total 174.13 last 24.18) - open/close 3853 (time 1709787041.33 total 184.23 last 17.04) - open/close 3994 (time 1709787051.43 total 194.33 last 13.96) - open/close 4326 (time 1709787062.38 total 205.27 last 30.34) - open/close 4646 (time 1709787072.39 total 215.28 last 31.97) - open/close 4821 (time 1709787082.44 total 225.33 last 17.41) - open/close 5018 (time 1709787092.45 total 235.35 last 19.67) - open/close 5152 (time 1709787103.26 total 246.16 last 12.40) - open/close 5362 (time 1709787113.27 total 256.17 last 20.98) - open/close 5388 (time 1709787124.94 total 267.83 last 2.23) - open/close 5699 (time 1709787134.99 total 277.89 last 30.94) - open/close 5860 (time 1709787145.99 total 288.89 last 14.63) - open/close 6142 (time 1709787156.92 total 299.82 last 25.80) - open/close 6376 (time 1709787167.24 total 310.13 last 22.69) - open/close 6536 (time 1709787177.29 total 320.19 last 15.91) - open/close 6731 (time 1709787187.68 total 330.58 last 18.77) - open/close 6938 (time 1709787198.43 total 341.33 last 19.26) - open/close 7083 (time 1709787208.44 total 351.34 last 14.48) - open/close 7254 (time 1709787218.68 total 361.58 last 16.70) - open/close 7395 (time 1709787229.32 total 372.21 last 13.26) - open/close 7553 (time 1709787239.36 total 382.26 last 15.73) - open/close 7722 (time 1709787250.25 total 393.14 last 15.52) - open/close 7860 (time 1709787260.31 total 403.20 last 13.72) - open/close 8028 (time 1709787270.32 total 413.21 last 16.78) - open/close 8266 (time 1709787280.32 total 423.21 last 23.80) - open/close 8390 (time 1709787290.37 total 433.26 last 12.34) - open/close 8602 (time 1709787300.39 total 443.29 last 21.15) - open/close 8689 (time 1709787310.40 total 453.30 last 8.69) - open/close 8900 (time 1709787320.75 total 463.65 last 20.39) - open/close 9093 (time 1709787330.91 total 473.80 last 19.01) - open/close 9342 (time 1709787341.12 total 484.01 last 24.39) - open/close 9590 (time 1709787352.09 total 494.99 last 22.59) - open/close 9723 (time 1709787363.08 total 505.98 last 12.10) - open/close 9931 (time 1709787373.09 total 515.99 last 20.78) - open/close 10456 (time 1709787383.10 total 525.99 last 52.49) - open/close 10800 (time 1709787400.77 total 543.67 last 19.46) - open/close 11297 (time 1709787410.77 total 553.67 last 49.70) - open/close 11716 (time 1709787420.79 total 563.68 last 41.85) - open/close 12115 (time 1709787430.81 total 573.71 last 39.79) - open/close 12537 (time 1709787440.83 total 583.73 last 42.13) - open/close 12968 (time 1709787450.85 total 593.75 last 43.01) - open/close 13366 (time 1709787460.86 total 603.76 last 39.76) - open/close 13776 (time 1709787470.89 total 613.79 last 40.88) - open/close 14164 (time 1709787480.91 total 623.80 last 38.74) - open/close 14545 (time 1709787490.92 total 633.82 last 38.05) - open/close 14888 (time 1709787500.94 total 643.83 last 34.25) - open/close 15238 (time 1709787510.99 total 653.88 last 34.82) - open/close 15545 (time 1709787531.36 total 674.26 last 15.07) - open/close 15906 (time 1709787541.38 total 684.28 last 36.03) - open/close 16268 (time 1709787551.40 total 694.30 last 36.12) - open/close 16658 (time 1709787561.41 total 704.30 last 38.98) - open/close 17077 (time 1709787571.41 total 714.30 last 41.89) - open/close 17448 (time 1709787581.43 total 724.33 last 37.01) - open/close 17867 (time 1709787591.45 total 734.34 last 41.85) - open/close 17945 (time 1709787604.39 total 747.28 last 6.03) - open/close 18332 (time 1709787614.40 total 757.29 last 38.65) - open/close 18734 (time 1709787624.41 total 767.30 last 40.15) - open/close 19093 (time 1709787634.42 total 777.31 last 35.87) - open/close 19445 (time 1709787644.44 total 787.34 last 35.11) - open/close 19816 (time 1709787654.46 total 797.35 last 37.04) - open/close 20000 (time 1709787659.08 total 801.98 last 39.78) - open/close 20394 (time 1709787669.09 total 811.99 last 39.36) - open/close 20721 (time 1709787679.10 total 821.99 last 32.69) - open/close 21079 (time 1709787689.11 total 832.00 last 35.77) - open/close 21405 (time 1709787699.11 total 842.00 last 32.58) - open/close 21734 (time 1709787709.14 total 852.03 last 32.82) - open/close 22015 (time 1709787719.17 total 862.07 last 27.99) - open/close 22339 (time 1709787729.22 total 872.11 last 32.26) - open/close 22611 (time 1709787748.16 total 891.06 last 14.36) - open/close 22912 (time 1709787758.16 total 901.06 last 30.10) - open/close 23177 (time 1709787768.20 total 911.10 last 26.41) - open/close 23459 (time 1709787778.22 total 921.12 last 28.14) - open/close 23710 (time 1709787788.24 total 931.14 last 25.04) - open/close 23974 (time 1709787798.26 total 941.15 last 26.36) - open/close 24292 (time 1709787808.27 total 951.16 last 31.78) - open/close 24601 (time 1709787818.28 total 961.17 last 30.85) - open/close 24890 (time 1709787828.30 total 971.20 last 28.84) - open/close 25207 (time 1709787838.31 total 981.20 last 31.69) - open/close 25539 (time 1709787848.32 total 991.21 last 33.16) - open/close 25886 (time 1709787858.32 total 1001.21 last 34.69) - open/close 26253 (time 1709787868.34 total 1011.23 last 36.63) - open/close 26555 (time 1709787887.87 total 1030.77 last 15.46) - open/close 26913 (time 1709787897.88 total 1040.78 last 35.76) - open/close 27253 (time 1709787907.89 total 1050.79 last 33.98) - open/close 27577 (time 1709787917.90 total 1060.79 last 32.38) - open/close 27913 (time 1709787927.90 total 1070.79 last 33.60) - open/close 28253 (time 1709787937.91 total 1080.81 last 33.95) - open/close 28588 (time 1709787947.93 total 1090.83 last 33.43) - open/close 28924 (time 1709787957.95 total 1100.85 last 33.53) - open/close 29252 (time 1709787967.97 total 1110.87 last 32.73) - open/close 29569 (time 1709787977.99 total 1120.88 last 31.65) - open/close 29892 (time 1709787988.00 total 1130.89 last 32.27) - open/close 30000 (time 1709787991.30 total 1134.20 last 32.70) - open/close 30321 (time 1709788001.31 total 1144.21 last 32.07) - open/close 30615 (time 1709788011.33 total 1154.22 last 29.35) - open/close 30911 (time 1709788021.36 total 1164.25 last 29.51) - open/close 31221 (time 1709788031.38 total 1174.28 last 30.93) - open/close 31535 (time 1709788041.39 total 1184.29 last 31.36) - open/close 31830 (time 1709788051.43 total 1194.33 last 29.39) - open/close 32131 (time 1709788061.43 total 1204.33 last 30.10) - open/close 32432 (time 1709788071.44 total 1214.34 last 30.07) - open/close 32737 (time 1709788081.45 total 1224.34 last 30.49) - open/close 33044 (time 1709788091.46 total 1234.35 last 30.67) - open/close 33345 (time 1709788101.48 total 1244.38 last 30.03) - open/close 33642 (time 1709788111.51 total 1254.41 last 29.61) - open/close 33946 (time 1709788121.52 total 1264.41 last 30.38) - open/close 34232 (time 1709788131.54 total 1274.43 last 28.54) - open/close 34509 (time 1709788141.57 total 1284.46 last 27.62) - open/close 34791 (time 1709788151.59 total 1294.49 last 28.13) - open/close 35043 (time 1709788161.62 total 1304.52 last 25.13) - open/close 35306 (time 1709788171.64 total 1314.54 last 26.24) - open/close 35582 (time 1709788181.65 total 1324.54 last 27.58) - open/close 35662 (time 1709788195.67 total 1338.56 last 5.71) - open/close 35896 (time 1709788205.69 total 1348.59 last 23.35) - open/close 36158 (time 1709788215.72 total 1358.62 last 26.11) - open/close 36406 (time 1709788225.73 total 1368.62 last 24.79) - open/close 36666 (time 1709788235.73 total 1378.63 last 25.99) - open/close 36894 (time 1709788245.76 total 1388.65 last 22.74) - open/close 37170 (time 1709788255.78 total 1398.67 last 27.55) - open/close 37454 (time 1709788265.78 total 1408.67 last 28.39) - open/close 37710 (time 1709788275.80 total 1418.69 last 25.55) - open/close 37973 (time 1709788285.81 total 1428.71 last 26.26) - open/close 38246 (time 1709788295.82 total 1438.71 last 27.28) - open/close 38510 (time 1709788305.84 total 1448.73 last 26.34) - open/close 38768 (time 1709788315.88 total 1458.78 last 25.69) - open/close 39028 (time 1709788325.90 total 1468.80 last 25.95) - open/close 39293 (time 1709788335.94 total 1478.83 last 26.41) - open/close 39538 (time 1709788345.94 total 1488.84 last 24.48) - open/close 39787 (time 1709788355.96 total 1498.85 last 24.86) - open/close 40000 (time 1709788364.74 total 1507.63 last 24.26) - open/close 40243 (time 1709788374.77 total 1517.66 last 24.23) - open/close 40481 (time 1709788384.79 total 1527.68 last 23.74) - open/close 40720 (time 1709788394.83 total 1537.72 last 23.81) - open/close 40964 (time 1709788404.86 total 1547.75 last 24.34) - open/close 41200 (time 1709788414.91 total 1557.80 last 23.48) - open/close 41425 (time 1709788424.92 total 1567.81 last 22.48) - open/close 41667 (time 1709788434.94 total 1577.84 last 24.14) - open/close 41895 (time 1709788444.95 total 1587.85 last 22.77) - open/close 42129 (time 1709788454.96 total 1597.85 last 23.39) - open/close 42362 (time 1709788464.98 total 1607.87 last 23.26) - open/close 42608 (time 1709788475.01 total 1617.90 last 24.53) - open/close 42844 (time 1709788485.02 total 1627.92 last 23.57) - open/close 43095 (time 1709788495.02 total 1637.92 last 25.09) - open/close 43347 (time 1709788505.03 total 1647.93 last 25.18) - open/close 43601 (time 1709788515.07 total 1657.96 last 25.31) - open/close 43831 (time 1709788525.09 total 1667.98 last 22.95) - open/close 44068 (time 1709788535.13 total 1678.02 last 23.61) - open/close 44314 (time 1709788545.14 total 1688.04 last 24.55) - open/close 44557 (time 1709788555.15 total 1698.05 last 24.29) - open/close 44782 (time 1709788565.19 total 1708.08 last 22.42) - open/close 45021 (time 1709788575.20 total 1718.09 last 23.87) - open/close 45256 (time 1709788585.24 total 1728.13 last 23.41) - open/close 45508 (time 1709788595.25 total 1738.15 last 25.16) - open/close 45760 (time 1709788605.27 total 1748.17 last 25.15) - open/close 46000 (time 1709788615.27 total 1758.17 last 24.00) - open/close 46249 (time 1709788625.29 total 1768.18 last 24.86) - open/close 46502 (time 1709788635.31 total 1778.20 last 25.25) - open/close 46754 (time 1709788645.34 total 1788.23 last 25.13) - open/close 46994 (time 1709788655.34 total 1798.23 last 24.00) - open/close 47228 (time 1709788665.35 total 1808.24 last 23.38) - open/close 47440 (time 1709788675.35 total 1818.24 last 21.19) - open/close 47646 (time 1709788685.36 total 1828.26 last 20.57) - open/close 47868 (time 1709788695.40 total 1838.30 last 22.11) - open/close 48097 (time 1709788705.44 total 1848.33 last 22.82) - open/close 48322 (time 1709788715.45 total 1858.35 last 22.46) - open/close 48542 (time 1709788725.49 total 1868.39 last 21.91) - open/close 48767 (time 1709788735.50 total 1878.40 last 22.48) - open/close 48988 (time 1709788745.54 total 1888.43 last 22.03) - open/close 49211 (time 1709788755.58 total 1898.47 last 22.21) - open/close 49439 (time 1709788765.60 total 1908.49 last 22.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1915.44 seconds: 25.89 ops/second - unlinked 0 (time 1709788773 ; total 0 ; last 0) - unlinked 10000 (time 1709789274 ; total 501 ; last 501) - unlinked 20000 (time 1709789619 ; total 846 ; last 345) - unlinked 30000 (time 1709789828 ; total 1055 ; last 209) - unlinked 40000 (time 1709789920 ; total 1147 ; last 92) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1189 seconds: 41.715729 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2