== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:44:52 (1709786692) start mds service on oleg444-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock oleg444-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1709786721.53 total 10.13 last 22.81) - open/close 450 (time 1709786732.31 total 20.91 last 20.32) - open/close 631 (time 1709786742.66 total 31.26 last 17.48) - open/close 846 (time 1709786752.81 total 41.40 last 21.19) - open/close 978 (time 1709786762.96 total 51.55 last 13.00) - open/close 1323 (time 1709786773.21 total 61.80 last 33.66) - open/close 1569 (time 1709786783.89 total 72.48 last 23.04) - open/close 1747 (time 1709786793.89 total 82.49 last 17.79) - open/close 1929 (time 1709786804.64 total 93.24 last 16.93) - open/close 2103 (time 1709786814.65 total 103.25 last 17.39) - open/close 2272 (time 1709786825.51 total 114.11 last 15.56) - open/close 2665 (time 1709786835.52 total 124.11 last 39.28) - open/close 2946 (time 1709786846.22 total 134.81 last 26.26) - open/close 3235 (time 1709786856.23 total 144.82 last 28.87) - open/close 3455 (time 1709786866.75 total 155.35 last 20.90) - open/close 3698 (time 1709786877.07 total 165.66 last 23.56) - open/close 3867 (time 1709786887.28 total 175.88 last 16.54) - open/close 4128 (time 1709786897.37 total 185.96 last 25.88) - open/close 4288 (time 1709786907.74 total 196.33 last 15.43) - open/close 4521 (time 1709786918.38 total 206.98 last 21.89) - open/close 4715 (time 1709786928.75 total 217.34 last 18.72) - open/close 4949 (time 1709786938.76 total 227.36 last 23.37) - open/close 5171 (time 1709786949.31 total 237.90 last 21.05) - open/close 5456 (time 1709786960.04 total 248.63 last 26.56) - open/close 5697 (time 1709786970.05 total 258.65 last 24.07) - open/close 5911 (time 1709786980.44 total 269.04 last 20.60) - open/close 6140 (time 1709786990.71 total 279.31 last 22.30) - open/close 6468 (time 1709787001.36 total 289.96 last 30.79) - open/close 6644 (time 1709787011.38 total 299.97 last 17.58) - open/close 6664 (time 1709787022.77 total 311.36 last 1.76) - open/close 6737 (time 1709787032.91 total 321.51 last 7.20) - open/close 6789 (time 1709787042.94 total 331.54 last 5.18) - open/close 7008 (time 1709787053.89 total 342.48 last 20.01) - open/close 7151 (time 1709787064.07 total 352.67 last 14.04) - open/close 7327 (time 1709787074.08 total 362.68 last 17.58) - open/close 7558 (time 1709787085.07 total 373.67 last 21.01) - open/close 7798 (time 1709787095.19 total 383.79 last 23.72) - open/close 7927 (time 1709787105.34 total 393.93 last 12.72) - open/close 8092 (time 1709787117.50 total 406.10 last 13.56) - open/close 8289 (time 1709787127.86 total 416.45 last 19.02) - open/close 8460 (time 1709787137.88 total 426.47 last 17.07) - open/close 8673 (time 1709787147.88 total 436.48 last 21.29) - open/close 8849 (time 1709787158.60 total 447.19 last 16.43) - open/close 9107 (time 1709787169.02 total 457.62 last 24.74) - open/close 9404 (time 1709787179.03 total 467.63 last 29.67) - open/close 9540 (time 1709787192.34 total 480.93 last 10.22) - open/close 10000 (time 1709787200.30 total 488.89 last 57.78) - open/close 10527 (time 1709787210.30 total 498.89 last 52.70) - open/close 10724 (time 1709787225.26 total 513.86 last 13.17) - open/close 11246 (time 1709787235.28 total 523.88 last 52.08) - open/close 11733 (time 1709787245.30 total 533.89 last 48.63) - open/close 12208 (time 1709787255.30 total 543.90 last 47.48) - open/close 12734 (time 1709787265.31 total 553.91 last 52.53) - open/close 13227 (time 1709787275.33 total 563.92 last 49.22) - open/close 13698 (time 1709787285.34 total 573.94 last 47.04) - open/close 14146 (time 1709787295.35 total 583.94 last 44.78) - open/close 14626 (time 1709787305.35 total 593.94 last 48.00) - open/close 15063 (time 1709787315.38 total 603.97 last 43.56) - open/close 15555 (time 1709787325.38 total 613.98 last 49.19) - open/close 15992 (time 1709787335.42 total 624.02 last 43.53) - open/close 16452 (time 1709787345.45 total 634.04 last 45.88) - open/close 16867 (time 1709787355.47 total 644.06 last 41.41) - open/close 17236 (time 1709787365.48 total 654.08 last 36.85) - open/close 17598 (time 1709787375.50 total 664.09 last 36.15) - open/close 17963 (time 1709787385.52 total 674.11 last 36.43) - open/close 18340 (time 1709787395.53 total 684.12 last 37.66) - open/close 18703 (time 1709787416.19 total 704.78 last 17.57) - open/close 19043 (time 1709787426.19 total 714.79 last 34.00) - open/close 19365 (time 1709787436.20 total 724.79 last 32.18) - open/close 19690 (time 1709787446.22 total 734.82 last 32.42) - open/close 19980 (time 1709787456.24 total 744.84 last 28.94) - open/close 20295 (time 1709787466.26 total 754.86 last 31.43) - open/close 20490 (time 1709787483.54 total 772.14 last 11.29) - open/close 20795 (time 1709787493.57 total 782.17 last 30.42) - open/close 21093 (time 1709787503.58 total 792.17 last 29.78) - open/close 21375 (time 1709787513.61 total 802.20 last 28.12) - open/close 21692 (time 1709787523.61 total 812.21 last 31.68) - open/close 21984 (time 1709787533.65 total 822.24 last 29.10) - open/close 22274 (time 1709787543.67 total 832.26 last 28.95) - open/close 22310 (time 1709787555.78 total 844.37 last 2.97) - open/close 22662 (time 1709787565.80 total 854.39 last 35.13) - open/close 23012 (time 1709787575.82 total 864.42 last 34.92) - open/close 23383 (time 1709787585.84 total 874.44 last 37.03) - open/close 23769 (time 1709787595.85 total 884.45 last 38.56) - open/close 24138 (time 1709787605.86 total 894.45 last 36.87) - open/close 24500 (time 1709787615.87 total 904.47 last 36.15) - open/close 24862 (time 1709787625.87 total 914.47 last 36.20) - open/close 25204 (time 1709787635.90 total 924.49 last 34.11) - open/close 25542 (time 1709787645.90 total 934.50 last 33.78) - open/close 25573 (time 1709787657.90 total 946.49 last 2.58) - open/close 25959 (time 1709787667.92 total 956.51 last 38.54) - open/close 26292 (time 1709787677.95 total 966.54 last 33.20) - open/close 26651 (time 1709787687.95 total 976.54 last 35.90) - open/close 26976 (time 1709787697.95 total 986.54 last 32.50) - open/close 27300 (time 1709787707.98 total 996.58 last 32.28) - open/close 27601 (time 1709787718.03 total 1006.62 last 29.97) - open/close 27908 (time 1709787728.04 total 1016.64 last 30.66) - open/close 28253 (time 1709787738.06 total 1026.66 last 34.42) - open/close 28608 (time 1709787748.07 total 1036.66 last 35.48) - open/close 28892 (time 1709787758.10 total 1046.69 last 28.32) - open/close 29173 (time 1709787768.15 total 1056.75 last 27.95) - open/close 29431 (time 1709787778.15 total 1066.75 last 25.79) - open/close 29665 (time 1709787788.21 total 1076.80 last 23.28) - open/close 29909 (time 1709787798.24 total 1086.83 last 24.32) - open/close 30000 (time 1709787801.35 total 1089.95 last 29.23) - open/close 30299 (time 1709787811.37 total 1099.97 last 29.84) - open/close 30573 (time 1709787821.41 total 1110.01 last 27.29) - open/close 30825 (time 1709787831.42 total 1120.02 last 25.17) - open/close 31099 (time 1709787841.45 total 1130.04 last 27.34) - open/close 31422 (time 1709787851.48 total 1140.08 last 32.18) - open/close 31753 (time 1709787861.51 total 1150.10 last 33.03) - open/close 32074 (time 1709787871.51 total 1160.11 last 32.07) - open/close 32413 (time 1709787881.53 total 1170.13 last 33.84) - open/close 32746 (time 1709787891.54 total 1180.14 last 33.26) - open/close 33069 (time 1709787901.58 total 1190.17 last 32.20) - open/close 33360 (time 1709787911.59 total 1200.18 last 29.06) - open/close 33666 (time 1709787921.60 total 1210.20 last 30.56) - open/close 33959 (time 1709787931.64 total 1220.23 last 29.20) - open/close 34270 (time 1709787941.64 total 1230.23 last 31.10) - open/close 34581 (time 1709787951.69 total 1240.29 last 30.93) - open/close 34916 (time 1709787961.73 total 1250.33 last 33.38) - open/close 35207 (time 1709787971.77 total 1260.36 last 29.00) - open/close 35501 (time 1709787981.77 total 1270.36 last 29.39) - open/close 35822 (time 1709787991.78 total 1280.38 last 32.06) - open/close 36121 (time 1709788001.80 total 1290.39 last 29.84) - open/close 36377 (time 1709788011.82 total 1300.41 last 25.56) - open/close 36636 (time 1709788021.83 total 1310.42 last 25.88) - open/close 36934 (time 1709788031.84 total 1320.43 last 29.76) - open/close 37248 (time 1709788041.86 total 1330.45 last 31.35) - open/close 37561 (time 1709788051.88 total 1340.48 last 31.22) - open/close 37865 (time 1709788061.91 total 1350.50 last 30.33) - open/close 38176 (time 1709788071.95 total 1360.54 last 30.98) - open/close 38474 (time 1709788081.99 total 1370.58 last 29.68) - open/close 38778 (time 1709788092.01 total 1380.60 last 30.34) - open/close 39065 (time 1709788102.04 total 1390.63 last 28.62) - open/close 39352 (time 1709788112.05 total 1400.65 last 28.65) - open/close 39642 (time 1709788122.06 total 1410.65 last 28.99) - open/close 39912 (time 1709788132.09 total 1420.69 last 26.90) - open/close 40000 (time 1709788135.78 total 1424.37 last 23.87) - open/close 40260 (time 1709788145.82 total 1434.41 last 25.91) - open/close 40501 (time 1709788155.84 total 1444.43 last 24.05) - open/close 40725 (time 1709788165.86 total 1454.45 last 22.35) - open/close 40969 (time 1709788175.87 total 1464.46 last 24.38) - open/close 41245 (time 1709788185.88 total 1474.48 last 27.56) - open/close 41508 (time 1709788195.94 total 1484.53 last 26.16) - open/close 41752 (time 1709788205.96 total 1494.55 last 24.35) - open/close 42028 (time 1709788215.99 total 1504.58 last 27.52) - open/close 42228 (time 1709788234.70 total 1523.30 last 10.68) - open/close 42475 (time 1709788244.73 total 1533.33 last 24.63) - open/close 42751 (time 1709788254.74 total 1543.33 last 27.59) - open/close 43040 (time 1709788264.74 total 1553.34 last 28.89) - open/close 43302 (time 1709788274.75 total 1563.34 last 26.18) - open/close 43550 (time 1709788284.77 total 1573.36 last 24.75) - open/close 43812 (time 1709788294.78 total 1583.37 last 26.17) - open/close 44076 (time 1709788304.79 total 1593.38 last 26.37) - open/close 44332 (time 1709788314.80 total 1603.40 last 25.56) - open/close 44578 (time 1709788324.83 total 1613.42 last 24.54) - open/close 44836 (time 1709788334.86 total 1623.46 last 25.71) - open/close 45069 (time 1709788344.88 total 1633.47 last 23.26) - open/close 45312 (time 1709788354.88 total 1643.47 last 24.30) - open/close 45553 (time 1709788364.89 total 1653.48 last 24.08) - open/close 45812 (time 1709788374.90 total 1663.49 last 25.88) - open/close 46062 (time 1709788384.91 total 1673.50 last 24.97) - open/close 46320 (time 1709788394.94 total 1683.53 last 25.72) - open/close 46579 (time 1709788404.95 total 1693.55 last 25.87) - open/close 46823 (time 1709788414.98 total 1703.58 last 24.33) - open/close 47056 (time 1709788425.01 total 1713.60 last 23.24) - open/close 47293 (time 1709788435.04 total 1723.64 last 23.62) - open/close 47528 (time 1709788445.08 total 1733.68 last 23.40) - open/close 47763 (time 1709788455.10 total 1743.70 last 23.46) - open/close 47998 (time 1709788465.14 total 1753.73 last 23.42) - open/close 48244 (time 1709788475.15 total 1763.74 last 24.58) - open/close 48470 (time 1709788485.16 total 1773.75 last 22.57) - open/close 48478 (time 1709788496.55 total 1785.14 last 0.70) - open/close 48725 (time 1709788506.58 total 1795.18 last 24.62) - open/close 48974 (time 1709788516.61 total 1805.20 last 24.84) - open/close 49189 (time 1709788526.62 total 1815.22 last 21.46) - open/close 49421 (time 1709788536.64 total 1825.23 last 23.17) - open/close 49668 (time 1709788546.67 total 1835.26 last 24.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1836.57 seconds: 27.06 ops/second - unlinked 0 (time 1709788549 ; total 0 ; last 0) - unlinked 10000 (time 1709789013 ; total 464 ; last 464) - unlinked 20000 (time 1709789390 ; total 841 ; last 377) - unlinked 30000 (time 1709789640 ; total 1091 ; last 250) - unlinked 40000 (time 1709789759 ; total 1210 ; last 119) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1257 seconds: 39.536991 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2