== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:48 (1709787108) start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock oleg204-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1709787145.59 total 10.95 last 13.42) - open/close 358 (time 1709787155.63 total 20.99 last 21.03) - open/close 587 (time 1709787166.46 total 31.82 last 21.14) - open/close 694 (time 1709787176.47 total 41.83 last 10.68) - open/close 875 (time 1709787186.56 total 51.92 last 17.95) - open/close 1076 (time 1709787196.58 total 61.94 last 20.05) - open/close 1314 (time 1709787207.16 total 72.52 last 22.51) - open/close 1508 (time 1709787217.73 total 83.09 last 18.34) - open/close 1740 (time 1709787227.90 total 93.26 last 22.82) - open/close 1944 (time 1709787238.21 total 103.57 last 19.79) - open/close 2227 (time 1709787249.12 total 114.48 last 25.94) - open/close 2377 (time 1709787259.14 total 124.50 last 14.97) - open/close 2543 (time 1709787270.00 total 135.36 last 15.27) - open/close 2757 (time 1709787280.66 total 146.02 last 20.08) - open/close 2933 (time 1709787291.64 total 157.00 last 16.04) - open/close 3105 (time 1709787301.64 total 167.00 last 17.20) - open/close 3219 (time 1709787312.58 total 177.94 last 10.42) - open/close 3446 (time 1709787322.99 total 188.35 last 21.81) - open/close 3629 (time 1709787333.01 total 198.37 last 18.27) - open/close 3831 (time 1709787343.71 total 209.07 last 18.87) - open/close 4025 (time 1709787354.51 total 219.87 last 17.96) - open/close 4149 (time 1709787364.79 total 230.16 last 12.06) - open/close 4295 (time 1709787374.81 total 240.17 last 14.58) - open/close 4452 (time 1709787385.01 total 250.37 last 15.39) - open/close 4621 (time 1709787395.28 total 260.64 last 16.45) - open/close 4801 (time 1709787405.28 total 270.64 last 18.00) - open/close 5019 (time 1709787415.28 total 280.64 last 21.80) - open/close 5180 (time 1709787425.50 total 290.86 last 15.76) - open/close 5421 (time 1709787435.52 total 300.88 last 24.06) - open/close 5628 (time 1709787446.14 total 311.50 last 19.49) - open/close 5815 (time 1709787457.10 total 322.47 last 17.05) - open/close 5958 (time 1709787467.12 total 332.48 last 14.27) - open/close 6128 (time 1709787477.82 total 343.18 last 15.90) - open/close 6243 (time 1709787487.83 total 353.19 last 11.48) - open/close 6432 (time 1709787497.84 total 363.20 last 18.89) - open/close 6761 (time 1709787507.84 total 373.20 last 32.90) - open/close 6903 (time 1709787517.86 total 383.22 last 14.17) - open/close 7128 (time 1709787528.26 total 393.62 last 21.63) - open/close 7284 (time 1709787538.45 total 403.81 last 15.32) - open/close 7457 (time 1709787548.57 total 413.93 last 17.08) - open/close 7636 (time 1709787558.83 total 424.19 last 17.45) - open/close 7868 (time 1709787569.80 total 435.16 last 21.15) - open/close 8093 (time 1709787579.81 total 445.17 last 22.49) - open/close 8256 (time 1709787589.83 total 455.19 last 16.27) - open/close 8379 (time 1709787599.85 total 465.21 last 12.27) - open/close 8537 (time 1709787610.54 total 475.90 last 14.79) - open/close 8645 (time 1709787629.65 total 495.01 last 5.65) - open/close 8823 (time 1709787639.66 total 505.02 last 17.77) - open/close 9023 (time 1709787649.69 total 515.05 last 19.94) - open/close 9244 (time 1709787660.61 total 525.97 last 20.25) - open/close 9410 (time 1709787670.61 total 535.97 last 16.59) - open/close 9569 (time 1709787681.43 total 546.79 last 14.70) - open/close 9769 (time 1709787691.54 total 556.90 last 19.79) - open/close 9915 (time 1709787701.54 total 566.90 last 14.60) - open/close 10070 (time 1709787711.56 total 576.92 last 15.47) - open/close 10206 (time 1709787721.80 total 587.16 last 13.27) - open/close 10321 (time 1709787731.81 total 597.17 last 11.49) - open/close 10436 (time 1709787741.83 total 607.19 last 11.48) - open/close 10872 (time 1709787751.84 total 617.20 last 43.54) - open/close 11332 (time 1709787761.85 total 627.21 last 45.96) - open/close 11775 (time 1709787771.86 total 637.23 last 44.22) - open/close 12188 (time 1709787781.87 total 647.23 last 41.28) - open/close 12608 (time 1709787791.89 total 657.25 last 41.92) - open/close 13044 (time 1709787801.90 total 667.26 last 43.54) - open/close 13448 (time 1709787811.91 total 677.27 last 40.37) - open/close 13833 (time 1709787821.93 total 687.29 last 38.43) - open/close 14229 (time 1709787831.95 total 697.31 last 39.52) - open/close 14611 (time 1709787841.97 total 707.33 last 38.11) - open/close 14944 (time 1709787852.00 total 717.36 last 33.20) - open/close 15266 (time 1709787862.01 total 727.37 last 32.17) - open/close 15601 (time 1709787872.04 total 737.40 last 33.41) - open/close 15941 (time 1709787882.04 total 747.40 last 33.98) - open/close 16256 (time 1709787892.05 total 757.41 last 31.48) - open/close 16527 (time 1709787902.05 total 767.41 last 27.10) - open/close 16844 (time 1709787912.05 total 777.41 last 31.68) - open/close 17175 (time 1709787922.05 total 787.41 last 33.10) - open/close 17488 (time 1709787932.06 total 797.42 last 31.28) - open/close 17813 (time 1709787942.07 total 807.44 last 32.45) - open/close 18173 (time 1709787952.09 total 817.45 last 35.94) - open/close 18592 (time 1709787962.11 total 827.47 last 41.82) - open/close 18978 (time 1709787972.13 total 837.49 last 38.53) - open/close 19340 (time 1709787982.14 total 847.51 last 36.14) - open/close 19716 (time 1709787992.15 total 857.51 last 37.59) - open/close 20000 (time 1709787999.07 total 864.43 last 41.04) - open/close 20452 (time 1709788009.08 total 874.44 last 45.14) - open/close 20913 (time 1709788019.09 total 884.45 last 46.06) - open/close 21390 (time 1709788029.10 total 894.46 last 47.64) - open/close 21887 (time 1709788039.11 total 904.47 last 49.66) - open/close 22355 (time 1709788049.13 total 914.49 last 46.73) - open/close 22837 (time 1709788059.14 total 924.50 last 48.12) - open/close 23274 (time 1709788069.15 total 934.51 last 43.68) - open/close 23708 (time 1709788079.16 total 944.52 last 43.35) - open/close 24163 (time 1709788089.17 total 954.53 last 45.45) - open/close 24589 (time 1709788099.18 total 964.54 last 42.58) - open/close 25016 (time 1709788109.18 total 974.54 last 42.69) - open/close 25437 (time 1709788119.18 total 984.54 last 42.08) - open/close 25856 (time 1709788129.20 total 994.56 last 41.83) - open/close 26273 (time 1709788139.21 total 1004.57 last 41.67) - open/close 26676 (time 1709788149.22 total 1014.58 last 40.26) - open/close 27083 (time 1709788159.23 total 1024.59 last 40.65) - open/close 27493 (time 1709788169.24 total 1034.60 last 40.97) - open/close 27907 (time 1709788179.26 total 1044.62 last 41.33) - open/close 28302 (time 1709788189.26 total 1054.62 last 39.49) - open/close 28692 (time 1709788199.26 total 1064.62 last 38.98) - open/close 29080 (time 1709788209.27 total 1074.63 last 38.77) - open/close 29482 (time 1709788219.27 total 1084.63 last 40.19) - open/close 29876 (time 1709788229.29 total 1094.65 last 39.33) - open/close 30000 (time 1709788232.61 total 1097.97 last 37.33) - open/close 30373 (time 1709788242.61 total 1107.98 last 37.30) - open/close 30756 (time 1709788252.63 total 1117.99 last 38.23) - open/close 31118 (time 1709788262.64 total 1128.00 last 36.18) - open/close 31480 (time 1709788272.64 total 1138.00 last 36.18) - open/close 31804 (time 1709788282.65 total 1148.02 last 32.36) - open/close 32121 (time 1709788292.66 total 1158.02 last 31.68) - open/close 32450 (time 1709788302.68 total 1168.04 last 32.84) - open/close 32782 (time 1709788312.69 total 1178.05 last 33.18) - open/close 33100 (time 1709788322.70 total 1188.06 last 31.75) - open/close 33414 (time 1709788332.71 total 1198.07 last 31.38) - open/close 33729 (time 1709788342.73 total 1208.09 last 31.44) - open/close 34039 (time 1709788352.75 total 1218.11 last 30.94) - open/close 34349 (time 1709788362.75 total 1228.11 last 30.98) - open/close 34660 (time 1709788372.76 total 1238.12 last 31.07) - open/close 34963 (time 1709788382.78 total 1248.14 last 30.26) - open/close 35248 (time 1709788392.81 total 1258.17 last 28.40) - open/close 35548 (time 1709788402.83 total 1268.19 last 29.95) - open/close 35845 (time 1709788412.84 total 1278.20 last 29.66) - open/close 36136 (time 1709788422.87 total 1288.23 last 29.02) - open/close 36435 (time 1709788432.89 total 1298.25 last 29.82) - open/close 36739 (time 1709788442.93 total 1308.29 last 30.30) - open/close 37038 (time 1709788452.96 total 1318.32 last 29.80) - open/close 37330 (time 1709788462.99 total 1328.35 last 29.11) - open/close 37636 (time 1709788473.01 total 1338.37 last 30.55) - open/close 37953 (time 1709788483.02 total 1348.38 last 31.66) - open/close 38263 (time 1709788493.05 total 1358.41 last 30.91) - open/close 38581 (time 1709788503.07 total 1368.43 last 31.74) - open/close 38900 (time 1709788513.07 total 1378.43 last 31.89) - open/close 39209 (time 1709788523.09 total 1388.45 last 30.84) - open/close 39517 (time 1709788533.09 total 1398.45 last 30.79) - open/close 39830 (time 1709788543.11 total 1408.47 last 31.24) - open/close 40000 (time 1709788548.72 total 1414.08 last 30.34) - open/close 40308 (time 1709788558.74 total 1424.10 last 30.73) - open/close 40616 (time 1709788568.75 total 1434.11 last 30.76) - open/close 40919 (time 1709788578.78 total 1444.14 last 30.20) - open/close 41224 (time 1709788588.79 total 1454.15 last 30.48) - open/close 41513 (time 1709788598.81 total 1464.17 last 28.83) - open/close 41808 (time 1709788608.85 total 1474.21 last 29.40) - open/close 42092 (time 1709788618.86 total 1484.22 last 28.37) - open/close 42371 (time 1709788628.87 total 1494.23 last 27.87) - open/close 42638 (time 1709788638.89 total 1504.25 last 26.64) - open/close 42895 (time 1709788648.91 total 1514.27 last 25.66) - open/close 43157 (time 1709788658.92 total 1524.28 last 26.18) - open/close 43423 (time 1709788668.94 total 1534.30 last 26.54) - open/close 43688 (time 1709788678.94 total 1544.30 last 26.49) - open/close 43957 (time 1709788688.95 total 1554.31 last 26.87) - open/close 44228 (time 1709788698.98 total 1564.34 last 27.04) - open/close 44494 (time 1709788708.98 total 1574.34 last 26.59) - open/close 44771 (time 1709788719.01 total 1584.37 last 27.62) - open/close 45043 (time 1709788729.03 total 1594.39 last 27.13) - open/close 45320 (time 1709788739.05 total 1604.41 last 27.65) - open/close 45591 (time 1709788749.06 total 1614.42 last 27.08) - open/close 45856 (time 1709788759.06 total 1624.42 last 26.49) - open/close 46124 (time 1709788769.09 total 1634.45 last 26.72) - open/close 46384 (time 1709788779.10 total 1644.46 last 25.97) - open/close 46643 (time 1709788789.10 total 1654.46 last 25.89) - open/close 46906 (time 1709788799.12 total 1664.48 last 26.26) - open/close 47169 (time 1709788809.14 total 1674.50 last 26.26) - open/close 47429 (time 1709788819.16 total 1684.52 last 25.93) - open/close 47675 (time 1709788829.20 total 1694.56 last 24.51) - open/close 47923 (time 1709788839.22 total 1704.58 last 24.74) - open/close 48170 (time 1709788849.25 total 1714.61 last 24.63) - open/close 48412 (time 1709788859.28 total 1724.64 last 24.12) - open/close 48644 (time 1709788869.30 total 1734.66 last 23.16) - open/close 48885 (time 1709788879.31 total 1744.67 last 24.09) - open/close 49133 (time 1709788889.33 total 1754.70 last 24.73) - open/close 49387 (time 1709788899.34 total 1764.70 last 25.39) - open/close 49636 (time 1709788909.35 total 1774.71 last 24.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1777.26 seconds: 27.96 ops/second - unlinked 0 (time 1709788913 ; total 1 ; last 1) - unlinked 10000 (time 1709789371 ; total 459 ; last 458) - unlinked 20000 (time 1709789752 ; total 840 ; last 381) - unlinked 30000 (time 1709790008 ; total 1096 ; last 256) - unlinked 40000 (time 1709790117 ; total 1205 ; last 109) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1233 seconds: 40.306568 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2