== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:42 (1709787282) start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1709787318.07 total 10.18 last 13.65) - open/close 392 (time 1709787328.59 total 20.70 last 24.05) - open/close 491 (time 1709787339.18 total 31.29 last 9.35) - open/close 626 (time 1709787350.09 total 42.20 last 12.37) - open/close 875 (time 1709787360.40 total 52.51 last 24.14) - open/close 1018 (time 1709787370.82 total 62.93 last 13.73) - open/close 1181 (time 1709787380.85 total 72.96 last 16.24) - open/close 1398 (time 1709787391.77 total 83.88 last 19.88) - open/close 1653 (time 1709787402.23 total 94.34 last 24.37) - open/close 1900 (time 1709787413.12 total 105.23 last 22.69) - open/close 2114 (time 1709787423.64 total 115.75 last 20.34) - open/close 2244 (time 1709787434.30 total 126.41 last 12.20) - open/close 2477 (time 1709787444.58 total 136.69 last 22.67) - open/close 2703 (time 1709787455.55 total 147.66 last 20.59) - open/close 2904 (time 1709787466.16 total 158.27 last 18.94) - open/close 3069 (time 1709787476.34 total 168.45 last 16.21) - open/close 3190 (time 1709787486.34 total 178.46 last 12.10) - open/close 3383 (time 1709787497.03 total 189.14 last 18.07) - open/close 3612 (time 1709787508.03 total 200.14 last 20.81) - open/close 3703 (time 1709787523.79 total 215.90 last 5.77) - open/close 3945 (time 1709787533.80 total 225.91 last 24.17) - open/close 4097 (time 1709787543.90 total 236.01 last 15.05) - open/close 4269 (time 1709787554.20 total 246.31 last 16.71) - open/close 4457 (time 1709787564.92 total 257.03 last 17.53) - open/close 4607 (time 1709787575.51 total 267.62 last 14.16) - open/close 4764 (time 1709787585.55 total 277.66 last 15.65) - open/close 4971 (time 1709787595.66 total 287.77 last 20.48) - open/close 5113 (time 1709787605.71 total 297.82 last 14.13) - open/close 5333 (time 1709787616.16 total 308.27 last 21.04) - open/close 5538 (time 1709787626.78 total 318.89 last 19.30) - open/close 5686 (time 1709787636.79 total 328.90 last 14.79) - open/close 5874 (time 1709787646.80 total 338.91 last 18.78) - open/close 6090 (time 1709787656.81 total 348.92 last 21.57) - open/close 6198 (time 1709787667.12 total 359.23 last 10.48) - open/close 6323 (time 1709787677.17 total 369.28 last 12.44) - open/close 6505 (time 1709787687.19 total 379.30 last 18.17) - open/close 6657 (time 1709787697.90 total 390.01 last 14.18) - open/close 6820 (time 1709787708.37 total 400.48 last 15.57) - open/close 7033 (time 1709787718.65 total 410.76 last 20.73) - open/close 7162 (time 1709787728.93 total 421.04 last 12.54) - open/close 7283 (time 1709787738.96 total 431.07 last 12.07) - open/close 7421 (time 1709787748.98 total 441.09 last 13.77) - open/close 7548 (time 1709787758.99 total 451.10 last 12.68) - open/close 7765 (time 1709787769.00 total 461.11 last 21.68) - open/close 7947 (time 1709787779.02 total 471.13 last 18.17) - open/close 8089 (time 1709787789.04 total 481.15 last 14.18) - open/close 8090 (time 1709787800.12 total 492.23 last 0.09) - open/close 8281 (time 1709787811.11 total 503.22 last 17.38) - open/close 8455 (time 1709787822.04 total 514.15 last 15.92) - open/close 8599 (time 1709787832.07 total 524.18 last 14.36) - open/close 8752 (time 1709787842.98 total 535.09 last 14.02) - open/close 8964 (time 1709787853.66 total 545.77 last 19.84) - open/close 9066 (time 1709787863.67 total 555.78 last 10.19) - open/close 9185 (time 1709787873.70 total 565.81 last 11.87) - open/close 9344 (time 1709787883.72 total 575.83 last 15.87) - open/close 9509 (time 1709787893.73 total 585.84 last 16.48) - open/close 9648 (time 1709787904.77 total 596.88 last 12.59) - open/close 9775 (time 1709787915.25 total 607.36 last 12.11) - open/close 9935 (time 1709787925.27 total 617.38 last 15.98) - open/close 10091 (time 1709787939.73 total 631.84 last 10.79) - open/close 10640 (time 1709787949.73 total 641.84 last 54.88) - open/close 11219 (time 1709787959.75 total 651.86 last 57.83) - open/close 11787 (time 1709787969.76 total 661.87 last 56.72) - open/close 12331 (time 1709787979.78 total 671.89 last 54.31) - open/close 12852 (time 1709787989.78 total 681.89 last 52.08) - open/close 13408 (time 1709787999.79 total 691.90 last 55.58) - open/close 14014 (time 1709788009.80 total 701.91 last 60.53) - open/close 14644 (time 1709788019.81 total 711.92 last 62.95) - open/close 15279 (time 1709788029.81 total 721.92 last 63.48) - open/close 15944 (time 1709788039.81 total 731.92 last 66.48) - open/close 16568 (time 1709788049.83 total 741.94 last 62.27) - open/close 17186 (time 1709788059.84 total 751.95 last 61.76) - open/close 17745 (time 1709788069.85 total 761.96 last 55.84) - open/close 18304 (time 1709788079.85 total 771.96 last 55.88) - open/close 18893 (time 1709788089.86 total 781.97 last 58.86) - open/close 19410 (time 1709788099.87 total 791.98 last 51.66) - open/close 19954 (time 1709788109.89 total 802.00 last 54.28) - open/close 20475 (time 1709788119.90 total 812.01 last 52.04) - open/close 20986 (time 1709788129.92 total 822.03 last 51.02) - open/close 21502 (time 1709788139.93 total 832.04 last 51.52) - open/close 21995 (time 1709788149.94 total 842.05 last 49.25) - open/close 22484 (time 1709788159.94 total 852.05 last 48.89) - open/close 22995 (time 1709788169.96 total 862.07 last 51.02) - open/close 23494 (time 1709788179.96 total 872.07 last 49.88) - open/close 23980 (time 1709788189.97 total 882.08 last 48.57) - open/close 24457 (time 1709788199.99 total 892.10 last 47.61) - open/close 24936 (time 1709788209.99 total 902.10 last 47.89) - open/close 25416 (time 1709788219.99 total 912.10 last 47.99) - open/close 25883 (time 1709788230.01 total 922.12 last 46.61) - open/close 26325 (time 1709788240.03 total 932.14 last 44.11) - open/close 26786 (time 1709788250.04 total 942.15 last 46.05) - open/close 27225 (time 1709788260.05 total 952.16 last 43.87) - open/close 27641 (time 1709788270.05 total 962.16 last 41.59) - open/close 28052 (time 1709788280.07 total 972.18 last 41.01) - open/close 28426 (time 1709788290.07 total 982.18 last 37.40) - open/close 28826 (time 1709788300.08 total 992.19 last 39.96) - open/close 29211 (time 1709788310.10 total 1002.21 last 38.45) - open/close 29598 (time 1709788320.13 total 1012.24 last 38.59) - open/close 29974 (time 1709788330.15 total 1022.26 last 37.53) - open/close 30331 (time 1709788340.16 total 1032.27 last 35.67) - open/close 30700 (time 1709788350.18 total 1042.29 last 36.81) - open/close 31056 (time 1709788360.20 total 1052.31 last 35.55) - open/close 31417 (time 1709788370.22 total 1062.33 last 36.02) - open/close 31766 (time 1709788380.24 total 1072.35 last 34.82) - open/close 32089 (time 1709788390.24 total 1082.35 last 32.30) - open/close 32413 (time 1709788400.26 total 1092.37 last 32.35) - open/close 32774 (time 1709788410.28 total 1102.39 last 36.01) - open/close 33118 (time 1709788420.28 total 1112.39 last 34.40) - open/close 33474 (time 1709788430.29 total 1122.40 last 35.56) - open/close 33824 (time 1709788440.31 total 1132.42 last 34.93) - open/close 34171 (time 1709788450.32 total 1142.43 last 34.69) - open/close 34517 (time 1709788460.32 total 1152.43 last 34.58) - open/close 34856 (time 1709788470.33 total 1162.44 last 33.87) - open/close 35213 (time 1709788480.35 total 1172.46 last 35.64) - open/close 35554 (time 1709788490.37 total 1182.48 last 34.04) - open/close 35894 (time 1709788500.38 total 1192.49 last 33.96) - open/close 36232 (time 1709788510.38 total 1202.49 last 33.79) - open/close 36540 (time 1709788520.40 total 1212.51 last 30.75) - open/close 36851 (time 1709788530.40 total 1222.51 last 31.10) - open/close 37163 (time 1709788540.40 total 1232.51 last 31.18) - open/close 37478 (time 1709788550.43 total 1242.54 last 31.41) - open/close 37785 (time 1709788560.43 total 1252.54 last 30.69) - open/close 38087 (time 1709788570.45 total 1262.56 last 30.16) - open/close 38387 (time 1709788580.45 total 1272.56 last 29.99) - open/close 38700 (time 1709788590.48 total 1282.59 last 31.23) - open/close 39012 (time 1709788600.51 total 1292.62 last 31.10) - open/close 39317 (time 1709788610.53 total 1302.64 last 30.42) - open/close 39608 (time 1709788620.54 total 1312.65 last 29.07) - open/close 39896 (time 1709788630.58 total 1322.69 last 28.70) - open/close 40000 (time 1709788634.35 total 1326.46 last 27.56) - open/close 40266 (time 1709788644.37 total 1336.49 last 26.54) - open/close 40526 (time 1709788654.38 total 1346.49 last 25.98) - open/close 40800 (time 1709788664.41 total 1356.52 last 27.32) - open/close 41067 (time 1709788674.44 total 1366.55 last 26.63) - open/close 41334 (time 1709788684.47 total 1376.58 last 26.63) - open/close 41613 (time 1709788694.50 total 1386.61 last 27.81) - open/close 41894 (time 1709788704.51 total 1396.62 last 28.08) - open/close 42171 (time 1709788714.52 total 1406.63 last 27.65) - open/close 42459 (time 1709788724.55 total 1416.66 last 28.71) - open/close 42736 (time 1709788734.59 total 1426.70 last 27.61) - open/close 43029 (time 1709788744.62 total 1436.73 last 29.20) - open/close 43301 (time 1709788754.63 total 1446.74 last 27.16) - open/close 43574 (time 1709788764.66 total 1456.77 last 27.24) - open/close 43834 (time 1709788774.70 total 1466.81 last 25.90) - open/close 44096 (time 1709788784.70 total 1476.81 last 26.19) - open/close 44364 (time 1709788794.73 total 1486.84 last 26.72) - open/close 44635 (time 1709788804.75 total 1496.86 last 27.06) - open/close 44908 (time 1709788814.75 total 1506.86 last 27.29) - open/close 45182 (time 1709788824.77 total 1516.88 last 27.34) - open/close 45442 (time 1709788834.79 total 1526.90 last 25.95) - open/close 45715 (time 1709788844.83 total 1536.94 last 27.20) - open/close 45986 (time 1709788854.85 total 1546.96 last 27.03) - open/close 46257 (time 1709788864.88 total 1556.99 last 27.02) - open/close 46530 (time 1709788874.90 total 1567.01 last 27.25) - open/close 46799 (time 1709788884.92 total 1577.03 last 26.83) - open/close 47057 (time 1709788894.94 total 1587.05 last 25.76) - open/close 47317 (time 1709788904.95 total 1597.06 last 25.97) - open/close 47574 (time 1709788914.96 total 1607.07 last 25.68) - open/close 47822 (time 1709788924.97 total 1617.08 last 24.78) - open/close 48079 (time 1709788934.99 total 1627.10 last 25.65) - open/close 48327 (time 1709788945.00 total 1637.11 last 24.76) - open/close 48582 (time 1709788955.02 total 1647.13 last 25.46) - open/close 48837 (time 1709788965.02 total 1657.13 last 25.50) - open/close 49088 (time 1709788975.02 total 1667.13 last 25.08) - open/close 49335 (time 1709788985.02 total 1677.13 last 24.70) - open/close 49580 (time 1709788995.05 total 1687.16 last 24.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1691.92 seconds: 29.37 ops/second - unlinked 0 (time 1709789000 ; total 0 ; last 0) - unlinked 10000 (time 1709789484 ; total 484 ; last 484) - unlinked 20000 (time 1709789856 ; total 856 ; last 372) - unlinked 30000 (time 1709790075 ; total 1075 ; last 219) - unlinked 40000 (time 1709790144 ; total 1144 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1169 seconds: 42.513260 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2