== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:47 (1709786867) start mds service on oleg438-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg438-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1709786898.22 total 10.29 last 23.33) - open/close 475 (time 1709786908.26 total 20.33 last 23.41) - open/close 753 (time 1709786918.26 total 30.33 last 27.79) - open/close 1035 (time 1709786928.27 total 40.34 last 28.17) - open/close 1245 (time 1709786938.64 total 50.71 last 20.25) - open/close 1434 (time 1709786948.69 total 60.76 last 18.81) - open/close 1659 (time 1709786958.70 total 70.77 last 22.48) - open/close 1985 (time 1709786969.13 total 81.20 last 31.26) - open/close 2056 (time 1709786980.17 total 92.24 last 6.43) - open/close 2319 (time 1709786990.84 total 102.91 last 24.65) - open/close 2755 (time 1709787001.60 total 113.67 last 40.51) - open/close 2930 (time 1709787011.83 total 123.90 last 17.11) - open/close 3154 (time 1709787022.10 total 134.16 last 21.83) - open/close 3421 (time 1709787032.37 total 144.43 last 26.00) - open/close 3686 (time 1709787043.85 total 155.92 last 23.08) - open/close 3886 (time 1709787054.61 total 166.68 last 18.59) - open/close 4072 (time 1709787066.21 total 178.27 last 16.04) - open/close 4365 (time 1709787076.21 total 188.28 last 29.28) - open/close 4645 (time 1709787086.81 total 198.88 last 26.42) - open/close 4811 (time 1709787096.81 total 208.88 last 16.59) - open/close 5082 (time 1709787107.67 total 219.74 last 24.95) - open/close 5283 (time 1709787118.44 total 230.51 last 18.66) - open/close 5446 (time 1709787128.49 total 240.56 last 16.22) - open/close 5662 (time 1709787138.49 total 250.56 last 21.59) - open/close 5951 (time 1709787148.53 total 260.59 last 28.81) - open/close 6180 (time 1709787159.20 total 271.27 last 21.45) - open/close 6401 (time 1709787169.33 total 281.40 last 21.82) - open/close 6792 (time 1709787179.35 total 291.42 last 39.03) - open/close 6959 (time 1709787189.87 total 301.94 last 15.87) - open/close 7125 (time 1709787200.26 total 312.33 last 15.97) - open/close 7201 (time 1709787210.60 total 322.67 last 7.35) - open/close 7384 (time 1709787220.62 total 332.69 last 18.27) - open/close 7478 (time 1709787231.35 total 343.42 last 8.76) - open/close 7740 (time 1709787251.77 total 363.84 last 12.83) - open/close 7841 (time 1709787261.89 total 373.96 last 9.98) - open/close 7989 (time 1709787272.57 total 384.64 last 13.85) - open/close 8121 (time 1709787282.94 total 395.01 last 12.73) - open/close 8313 (time 1709787292.95 total 405.01 last 19.19) - open/close 8496 (time 1709787303.76 total 415.83 last 16.93) - open/close 8706 (time 1709787313.79 total 425.85 last 20.94) - open/close 9058 (time 1709787323.79 total 435.85 last 35.20) - open/close 9292 (time 1709787334.01 total 446.07 last 22.90) - open/close 9430 (time 1709787344.75 total 456.82 last 12.85) - open/close 9627 (time 1709787354.77 total 466.84 last 19.66) - open/close 9734 (time 1709787367.19 total 479.26 last 8.62) - open/close 9870 (time 1709787377.90 total 489.97 last 12.70) - open/close 10000 (time 1709787383.38 total 495.45 last 23.71) - open/close 10335 (time 1709787393.40 total 505.46 last 33.45) - open/close 10937 (time 1709787403.40 total 515.47 last 60.16) - open/close 11457 (time 1709787413.41 total 525.48 last 51.97) - open/close 11949 (time 1709787423.41 total 535.48 last 49.19) - open/close 12453 (time 1709787433.42 total 545.49 last 50.37) - open/close 12961 (time 1709787443.47 total 555.54 last 50.53) - open/close 13465 (time 1709787453.47 total 565.54 last 50.39) - open/close 13958 (time 1709787463.48 total 575.55 last 49.27) - open/close 14433 (time 1709787473.52 total 585.59 last 47.31) - open/close 14894 (time 1709787483.53 total 595.60 last 46.05) - open/close 15331 (time 1709787493.60 total 605.67 last 43.41) - open/close 15733 (time 1709787503.62 total 615.69 last 40.10) - open/close 16116 (time 1709787513.64 total 625.70 last 38.25) - open/close 16555 (time 1709787523.64 total 635.71 last 43.88) - open/close 17006 (time 1709787533.64 total 645.71 last 45.09) - open/close 17457 (time 1709787543.65 total 655.71 last 45.09) - open/close 17942 (time 1709787553.65 total 665.72 last 48.49) - open/close 18401 (time 1709787563.67 total 675.74 last 45.80) - open/close 18880 (time 1709787573.68 total 685.74 last 47.87) - open/close 19295 (time 1709787583.69 total 695.75 last 41.46) - open/close 19789 (time 1709787593.70 total 705.77 last 49.32) - open/close 20000 (time 1709787598.12 total 710.19 last 47.77) - open/close 20494 (time 1709787608.14 total 720.21 last 49.30) - open/close 20921 (time 1709787618.15 total 730.22 last 42.67) - open/close 21331 (time 1709787628.15 total 740.22 last 40.99) - open/close 21763 (time 1709787638.15 total 750.22 last 43.19) - open/close 22173 (time 1709787648.17 total 760.23 last 40.95) - open/close 22640 (time 1709787658.17 total 770.24 last 46.66) - open/close 23075 (time 1709787668.18 total 780.25 last 43.47) - open/close 23457 (time 1709787678.19 total 790.26 last 38.17) - open/close 23847 (time 1709787688.21 total 800.28 last 38.90) - open/close 24245 (time 1709787698.23 total 810.30 last 39.72) - open/close 24630 (time 1709787708.27 total 820.34 last 38.36) - open/close 24968 (time 1709787718.28 total 830.35 last 33.77) - open/close 25327 (time 1709787728.29 total 840.36 last 35.84) - open/close 25734 (time 1709787738.31 total 850.38 last 40.63) - open/close 26159 (time 1709787748.32 total 860.38 last 42.48) - open/close 26517 (time 1709787758.33 total 870.40 last 35.74) - open/close 26847 (time 1709787768.35 total 880.41 last 32.95) - open/close 27125 (time 1709787778.36 total 890.43 last 27.76) - open/close 27410 (time 1709787788.36 total 900.43 last 28.49) - open/close 27710 (time 1709787798.37 total 910.44 last 29.98) - open/close 28078 (time 1709787808.39 total 920.46 last 36.74) - open/close 28399 (time 1709787818.40 total 930.47 last 32.05) - open/close 28716 (time 1709787828.43 total 940.50 last 31.61) - open/close 29068 (time 1709787838.43 total 950.50 last 35.20) - open/close 29446 (time 1709787848.45 total 960.52 last 37.72) - open/close 29855 (time 1709787858.48 total 970.54 last 40.81) - open/close 30000 (time 1709787861.87 total 973.94 last 42.67) - open/close 30408 (time 1709787871.89 total 983.96 last 40.73) - open/close 30817 (time 1709787881.91 total 993.98 last 40.82) - open/close 31210 (time 1709787891.93 total 1003.99 last 39.23) - open/close 31619 (time 1709787901.95 total 1014.01 last 40.82) - open/close 31973 (time 1709787911.97 total 1024.04 last 35.32) - open/close 32357 (time 1709787921.97 total 1034.04 last 38.39) - open/close 32720 (time 1709787931.99 total 1044.06 last 36.22) - open/close 33113 (time 1709787942.01 total 1054.08 last 39.23) - open/close 33494 (time 1709787952.02 total 1064.09 last 38.07) - open/close 33501 (time 1709787963.21 total 1075.28 last 0.63) - open/close 33860 (time 1709787973.22 total 1085.29 last 35.88) - open/close 34231 (time 1709787983.23 total 1095.30 last 37.05) - open/close 34645 (time 1709787993.23 total 1105.30 last 41.39) - open/close 34976 (time 1709788003.27 total 1115.34 last 32.99) - open/close 35244 (time 1709788013.29 total 1125.36 last 26.73) - open/close 35524 (time 1709788023.30 total 1135.37 last 27.99) - open/close 35897 (time 1709788033.31 total 1145.38 last 37.24) - open/close 36274 (time 1709788043.32 total 1155.39 last 37.66) - open/close 36617 (time 1709788053.36 total 1165.42 last 34.19) - open/close 36969 (time 1709788063.38 total 1175.45 last 35.10) - open/close 37321 (time 1709788073.40 total 1185.47 last 35.15) - open/close 37503 (time 1709788089.52 total 1201.59 last 11.29) - open/close 37843 (time 1709788099.52 total 1211.59 last 34.00) - open/close 38178 (time 1709788109.53 total 1221.60 last 33.45) - open/close 38513 (time 1709788119.54 total 1231.61 last 33.47) - open/close 38833 (time 1709788129.55 total 1241.62 last 31.97) - open/close 39127 (time 1709788139.56 total 1251.63 last 29.37) - open/close 39426 (time 1709788149.57 total 1261.64 last 29.88) - open/close 39694 (time 1709788159.59 total 1271.65 last 26.76) - open/close 39948 (time 1709788169.63 total 1281.70 last 25.28) - open/close 40258 (time 1709788179.65 total 1291.72 last 30.93) - open/close 40583 (time 1709788189.69 total 1301.76 last 32.37) - open/close 40849 (time 1709788199.76 total 1311.82 last 26.44) - open/close 41163 (time 1709788209.79 total 1321.85 last 31.30) - open/close 41483 (time 1709788219.79 total 1331.86 last 31.99) - open/close 41797 (time 1709788229.80 total 1341.87 last 31.38) - open/close 42102 (time 1709788239.82 total 1351.89 last 30.43) - open/close 42410 (time 1709788249.84 total 1361.90 last 30.75) - open/close 42740 (time 1709788259.86 total 1371.93 last 32.91) - open/close 43054 (time 1709788269.87 total 1381.93 last 31.39) - open/close 43336 (time 1709788279.91 total 1391.97 last 28.09) - open/close 43642 (time 1709788289.91 total 1401.98 last 30.59) - open/close 43955 (time 1709788299.95 total 1412.01 last 31.18) - open/close 44236 (time 1709788309.96 total 1422.02 last 28.07) - open/close 44530 (time 1709788319.96 total 1432.03 last 29.38) - open/close 44841 (time 1709788329.99 total 1442.06 last 31.00) - open/close 45145 (time 1709788340.03 total 1452.10 last 30.28) - open/close 45439 (time 1709788350.07 total 1462.14 last 29.28) - open/close 45711 (time 1709788360.09 total 1472.16 last 27.16) - open/close 46021 (time 1709788370.10 total 1482.17 last 30.95) - open/close 46319 (time 1709788380.13 total 1492.20 last 29.72) - open/close 46619 (time 1709788390.13 total 1502.20 last 30.00) - open/close 46923 (time 1709788400.15 total 1512.22 last 30.35) - open/close 47213 (time 1709788410.15 total 1522.22 last 28.99) - open/close 47470 (time 1709788420.18 total 1532.25 last 25.63) - open/close 47731 (time 1709788430.18 total 1542.25 last 26.08) - open/close 47993 (time 1709788440.23 total 1552.29 last 26.09) - open/close 48253 (time 1709788450.24 total 1562.30 last 25.97) - open/close 48519 (time 1709788460.27 total 1572.34 last 26.51) - open/close 48793 (time 1709788470.30 total 1582.37 last 27.31) - open/close 49067 (time 1709788480.31 total 1592.38 last 27.38) - open/close 49317 (time 1709788490.32 total 1602.38 last 24.98) - open/close 49571 (time 1709788500.32 total 1612.39 last 25.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49696) error: No space left on device total: 49696 open/close in 1616.85 seconds: 30.74 ops/second - unlinked 0 (time 1709788505 ; total 0 ; last 0) - unlinked 10000 (time 1709788936 ; total 431 ; last 431) - unlinked 20000 (time 1709789286 ; total 781 ; last 350) - unlinked 30000 (time 1709789543 ; total 1038 ; last 257) - unlinked 40000 (time 1709789682 ; total 1177 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49696) error: No such file or directory total: 49696 unlinks in 1221 seconds: 40.701065 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2