== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:29 (1709786969) start mds service on oleg347-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock oleg347-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 385 (time 1709787021.27 total 10.00 last 38.50) - open/close 539 (time 1709787031.28 total 20.01 last 15.39) - open/close 670 (time 1709787041.88 total 30.61 last 12.36) - open/close 976 (time 1709787052.27 total 41.00 last 29.44) - open/close 1260 (time 1709787063.17 total 51.89 last 26.07) - open/close 1533 (time 1709787073.21 total 61.94 last 27.18) - open/close 1704 (time 1709787083.64 total 72.37 last 16.39) - open/close 1847 (time 1709787093.65 total 82.37 last 14.29) - open/close 2074 (time 1709787103.65 total 92.38 last 22.69) - open/close 2294 (time 1709787114.68 total 103.40 last 19.96) - open/close 2369 (time 1709787124.88 total 113.61 last 7.35) - open/close 2489 (time 1709787134.89 total 123.61 last 11.99) - open/close 2632 (time 1709787145.83 total 134.55 last 13.07) - open/close 2765 (time 1709787156.53 total 145.26 last 12.43) - open/close 2950 (time 1709787167.37 total 156.10 last 17.07) - open/close 3198 (time 1709787177.38 total 166.10 last 24.78) - open/close 3422 (time 1709787188.14 total 176.86 last 20.82) - open/close 3517 (time 1709787198.67 total 187.40 last 9.02) - open/close 3693 (time 1709787209.38 total 198.10 last 16.44) - open/close 3907 (time 1709787219.46 total 208.19 last 21.22) - open/close 4187 (time 1709787230.38 total 219.11 last 25.64) - open/close 4393 (time 1709787240.42 total 229.15 last 20.52) - open/close 4520 (time 1709787250.89 total 239.61 last 12.14) - open/close 4647 (time 1709787261.24 total 249.97 last 12.27) - open/close 4813 (time 1709787271.26 total 259.98 last 16.57) - open/close 5027 (time 1709787281.91 total 270.63 last 20.10) - open/close 5129 (time 1709787291.93 total 280.66 last 10.17) - open/close 5260 (time 1709787303.52 total 292.25 last 11.31) - open/close 5491 (time 1709787313.68 total 302.40 last 22.74) - open/close 5628 (time 1709787324.30 total 313.02 last 12.90) - open/close 5828 (time 1709787334.33 total 323.05 last 19.94) - open/close 5963 (time 1709787344.88 total 333.60 last 12.80) - open/close 6077 (time 1709787355.18 total 343.91 last 11.06) - open/close 6201 (time 1709787365.88 total 354.61 last 11.59) - open/close 6340 (time 1709787375.89 total 364.61 last 13.90) - open/close 6433 (time 1709787385.92 total 374.64 last 9.27) - open/close 6614 (time 1709787396.06 total 384.78 last 17.85) - open/close 6804 (time 1709787406.06 total 394.79 last 18.99) - open/close 7029 (time 1709787416.09 total 404.82 last 22.43) - open/close 7142 (time 1709787426.46 total 415.19 last 10.90) - open/close 7321 (time 1709787436.48 total 425.21 last 17.87) - open/close 7509 (time 1709787446.48 total 435.21 last 18.79) - open/close 7789 (time 1709787456.49 total 445.22 last 27.98) - open/close 7990 (time 1709787466.81 total 455.53 last 19.48) - open/close 8168 (time 1709787477.16 total 465.89 last 17.19) - open/close 8331 (time 1709787487.65 total 476.38 last 15.54) - open/close 8572 (time 1709787498.25 total 486.98 last 22.73) - open/close 8743 (time 1709787508.26 total 496.99 last 17.08) - open/close 8994 (time 1709787518.28 total 507.01 last 25.05) - open/close 9184 (time 1709787529.16 total 517.89 last 17.47) - open/close 9311 (time 1709787539.17 total 527.90 last 12.68) - open/close 9396 (time 1709787549.18 total 537.91 last 8.49) - open/close 9547 (time 1709787559.20 total 547.92 last 15.08) - open/close 9671 (time 1709787579.85 total 568.57 last 6.01) - open/close 9776 (time 1709787589.86 total 578.58 last 10.49) - open/close 9941 (time 1709787599.89 total 588.62 last 16.45) - open/close 10000 (time 1709787602.29 total 591.01 last 24.59) - open/close 10359 (time 1709787612.30 total 601.03 last 35.86) - open/close 10789 (time 1709787622.31 total 611.04 last 42.94) - open/close 11241 (time 1709787632.33 total 621.06 last 45.13) - open/close 11642 (time 1709787642.35 total 631.07 last 40.03) - open/close 12039 (time 1709787652.35 total 641.08 last 39.68) - open/close 12445 (time 1709787662.36 total 651.09 last 40.55) - open/close 12828 (time 1709787672.38 total 661.11 last 38.22) - open/close 13161 (time 1709787682.39 total 671.12 last 33.27) - open/close 13525 (time 1709787692.41 total 681.14 last 36.34) - open/close 13845 (time 1709787702.46 total 691.19 last 31.83) - open/close 14157 (time 1709787712.49 total 701.22 last 31.10) - open/close 14422 (time 1709787731.15 total 719.88 last 14.20) - open/close 14759 (time 1709787741.17 total 729.89 last 33.65) - open/close 15081 (time 1709787751.18 total 739.90 last 32.16) - open/close 15420 (time 1709787761.22 total 749.94 last 33.76) - open/close 15769 (time 1709787771.23 total 759.95 last 34.87) - open/close 16079 (time 1709787781.25 total 769.98 last 30.92) - open/close 16409 (time 1709787791.26 total 779.98 last 32.99) - open/close 16777 (time 1709787801.27 total 790.00 last 36.75) - open/close 17134 (time 1709787811.27 total 800.00 last 35.68) - open/close 17461 (time 1709787821.28 total 810.01 last 32.67) - open/close 17637 (time 1709787837.70 total 826.42 last 10.72) - open/close 17970 (time 1709787847.73 total 836.45 last 33.21) - open/close 18314 (time 1709787868.27 total 856.99 last 16.75) - open/close 18638 (time 1709787878.28 total 867.00 last 32.37) - open/close 18958 (time 1709787888.29 total 877.02 last 31.95) - open/close 19272 (time 1709787898.30 total 887.03 last 31.37) - open/close 19632 (time 1709787908.31 total 897.04 last 35.96) - open/close 19959 (time 1709787918.32 total 907.05 last 32.68) - open/close 20266 (time 1709787928.33 total 917.06 last 30.67) - open/close 20559 (time 1709787938.37 total 927.09 last 29.19) - open/close 20858 (time 1709787948.37 total 937.10 last 29.89) - open/close 21182 (time 1709787958.40 total 947.13 last 32.31) - open/close 21500 (time 1709787968.42 total 957.14 last 31.75) - open/close 21813 (time 1709787978.43 total 967.15 last 31.26) - open/close 22144 (time 1709787988.43 total 977.16 last 33.08) - open/close 22494 (time 1709787998.46 total 987.18 last 34.93) - open/close 22809 (time 1709788008.48 total 997.21 last 31.42) - open/close 23137 (time 1709788018.50 total 1007.23 last 32.73) - open/close 23457 (time 1709788028.52 total 1017.25 last 31.94) - open/close 23756 (time 1709788038.55 total 1027.27 last 29.83) - open/close 24055 (time 1709788048.57 total 1037.30 last 29.81) - open/close 24345 (time 1709788058.60 total 1047.33 last 28.92) - open/close 24588 (time 1709788078.24 total 1066.96 last 12.37) - open/close 24980 (time 1709788088.27 total 1076.99 last 39.09) - open/close 25356 (time 1709788098.30 total 1087.03 last 37.47) - open/close 25731 (time 1709788108.33 total 1097.05 last 37.40) - open/close 26085 (time 1709788118.35 total 1107.07 last 35.34) - open/close 26451 (time 1709788128.35 total 1117.08 last 36.57) - open/close 26837 (time 1709788138.38 total 1127.10 last 38.51) - open/close 27204 (time 1709788148.40 total 1137.12 last 36.63) - open/close 27549 (time 1709788158.40 total 1147.12 last 34.50) - open/close 27907 (time 1709788168.41 total 1157.14 last 35.75) - open/close 28250 (time 1709788178.43 total 1167.15 last 34.25) - open/close 28606 (time 1709788188.43 total 1177.16 last 35.58) - open/close 28934 (time 1709788198.43 total 1187.16 last 32.79) - open/close 29256 (time 1709788208.45 total 1197.18 last 32.14) - open/close 29584 (time 1709788218.46 total 1207.19 last 32.77) - open/close 29922 (time 1709788228.49 total 1217.22 last 33.71) - open/close 30000 (time 1709788230.90 total 1219.63 last 32.32) - open/close 30326 (time 1709788240.92 total 1229.65 last 32.53) - open/close 30640 (time 1709788250.95 total 1239.68 last 31.32) - open/close 30941 (time 1709788260.97 total 1249.70 last 30.04) - open/close 31248 (time 1709788271.00 total 1259.73 last 30.61) - open/close 31556 (time 1709788281.01 total 1269.74 last 30.76) - open/close 31844 (time 1709788291.05 total 1279.77 last 28.70) - open/close 32138 (time 1709788301.09 total 1289.82 last 29.27) - open/close 32421 (time 1709788311.12 total 1299.84 last 28.23) - open/close 32716 (time 1709788321.13 total 1309.86 last 29.47) - open/close 33020 (time 1709788331.16 total 1319.88 last 30.32) - open/close 33318 (time 1709788341.17 total 1329.90 last 29.76) - open/close 33620 (time 1709788351.18 total 1339.91 last 30.16) - open/close 33918 (time 1709788361.21 total 1349.93 last 29.72) - open/close 34211 (time 1709788371.22 total 1359.94 last 29.27) - open/close 34504 (time 1709788381.23 total 1369.96 last 29.26) - open/close 34796 (time 1709788391.26 total 1379.98 last 29.13) - open/close 35087 (time 1709788401.27 total 1389.99 last 29.06) - open/close 35366 (time 1709788411.29 total 1400.01 last 27.85) - open/close 35669 (time 1709788421.29 total 1410.02 last 30.29) - open/close 35946 (time 1709788431.29 total 1420.02 last 27.69) - open/close 36244 (time 1709788441.31 total 1430.04 last 29.74) - open/close 36522 (time 1709788451.33 total 1440.06 last 27.74) - open/close 36804 (time 1709788461.36 total 1450.08 last 28.13) - open/close 37075 (time 1709788471.39 total 1460.11 last 27.02) - open/close 37357 (time 1709788481.41 total 1470.13 last 28.14) - open/close 37613 (time 1709788491.41 total 1480.14 last 25.60) - open/close 37869 (time 1709788501.41 total 1490.14 last 25.59) - open/close 38138 (time 1709788511.42 total 1500.15 last 26.88) - open/close 38419 (time 1709788521.44 total 1510.16 last 28.06) - open/close 38686 (time 1709788531.45 total 1520.18 last 26.65) - open/close 38963 (time 1709788541.49 total 1530.22 last 27.60) - open/close 39238 (time 1709788551.53 total 1540.25 last 27.40) - open/close 39496 (time 1709788561.53 total 1550.25 last 25.80) - open/close 39765 (time 1709788571.53 total 1560.26 last 26.88) - open/close 40000 (time 1709788580.53 total 1569.25 last 26.14) - open/close 40258 (time 1709788590.55 total 1579.28 last 25.74) - open/close 40529 (time 1709788600.58 total 1589.31 last 27.01) - open/close 40810 (time 1709788610.61 total 1599.34 last 28.02) - open/close 41091 (time 1709788620.61 total 1609.34 last 28.10) - open/close 41365 (time 1709788630.63 total 1619.35 last 27.36) - open/close 41623 (time 1709788640.66 total 1629.38 last 25.72) - open/close 41848 (time 1709788650.68 total 1639.41 last 22.44) - open/close 42063 (time 1709788660.69 total 1649.41 last 21.49) - open/close 42286 (time 1709788670.69 total 1659.41 last 22.30) - open/close 42532 (time 1709788680.71 total 1669.43 last 24.56) - open/close 42775 (time 1709788690.72 total 1679.44 last 24.27) - open/close 43028 (time 1709788700.73 total 1689.45 last 25.28) - open/close 43277 (time 1709788710.74 total 1699.47 last 24.86) - open/close 43514 (time 1709788720.79 total 1709.52 last 23.58) - open/close 43782 (time 1709788730.81 total 1719.53 last 26.77) - open/close 44054 (time 1709788740.82 total 1729.55 last 27.16) - open/close 44323 (time 1709788750.85 total 1739.58 last 26.81) - open/close 44563 (time 1709788760.88 total 1749.61 last 23.92) - open/close 44793 (time 1709788770.92 total 1759.65 last 22.92) - open/close 45017 (time 1709788780.96 total 1769.69 last 22.31) - open/close 45244 (time 1709788790.99 total 1779.72 last 22.62) - open/close 45464 (time 1709788801.00 total 1789.73 last 21.98) - open/close 45684 (time 1709788811.04 total 1799.77 last 21.91) - open/close 45911 (time 1709788821.05 total 1809.78 last 22.68) - open/close 46129 (time 1709788831.06 total 1819.78 last 21.78) - open/close 46366 (time 1709788841.09 total 1829.81 last 23.64) - open/close 46596 (time 1709788851.12 total 1839.85 last 22.92) - open/close 46817 (time 1709788861.13 total 1849.86 last 22.08) - open/close 47041 (time 1709788871.13 total 1859.86 last 22.39) - open/close 47260 (time 1709788881.16 total 1869.88 last 21.85) - open/close 47494 (time 1709788891.16 total 1879.89 last 23.39) - open/close 47728 (time 1709788901.19 total 1889.92 last 23.32) - open/close 47957 (time 1709788911.23 total 1899.96 last 22.81) - open/close 48195 (time 1709788921.26 total 1909.99 last 23.73) - open/close 48429 (time 1709788931.27 total 1920.00 last 23.37) - open/close 48646 (time 1709788941.27 total 1930.00 last 21.70) - open/close 48869 (time 1709788951.28 total 1940.00 last 22.30) - open/close 49087 (time 1709788961.32 total 1950.04 last 21.71) - open/close 49311 (time 1709788971.34 total 1960.06 last 22.36) - open/close 49546 (time 1709788981.36 total 1970.09 last 23.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1972.43 seconds: 25.15 ops/second - unlinked 0 (time 1709788985 ; total 0 ; last 0) - unlinked 10000 (time 1709789479 ; total 494 ; last 494) - unlinked 20000 (time 1709789911 ; total 926 ; last 432) - unlinked 30000 (time 1709790211 ; total 1226 ; last 300) - unlinked 40000 (time 1709790345 ; total 1360 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1401 seconds: 35.403282 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2