== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:40 (1709786920) start mds service on oleg239-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg239-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock oleg239-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1709786969.24 total 10.37 last 19.97) - open/close 461 (time 1709786980.06 total 21.19 last 23.48) - open/close 715 (time 1709786990.88 total 32.00 last 23.49) - open/close 878 (time 1709787001.14 total 42.26 last 15.88) - open/close 1146 (time 1709787012.99 total 54.11 last 22.61) - open/close 1291 (time 1709787023.89 total 65.01 last 13.30) - open/close 1547 (time 1709787034.51 total 75.63 last 24.11) - open/close 1813 (time 1709787044.98 total 86.11 last 25.40) - open/close 1937 (time 1709787054.98 total 96.11 last 12.40) - open/close 2109 (time 1709787065.60 total 106.72 last 16.21) - open/close 2156 (time 1709787078.44 total 119.56 last 3.66) - open/close 2323 (time 1709787088.59 total 129.71 last 16.44) - open/close 2580 (time 1709787099.00 total 140.12 last 24.70) - open/close 2831 (time 1709787109.63 total 150.75 last 23.62) - open/close 2948 (time 1709787120.41 total 161.53 last 10.85) - open/close 3113 (time 1709787130.98 total 172.10 last 15.60) - open/close 3285 (time 1709787141.55 total 182.68 last 16.27) - open/close 3430 (time 1709787151.80 total 192.92 last 14.15) - open/close 3751 (time 1709787162.73 total 203.85 last 29.37) - open/close 3926 (time 1709787172.74 total 213.86 last 17.49) - open/close 3989 (time 1709787189.82 total 230.94 last 3.69) - open/close 4131 (time 1709787200.40 total 241.52 last 13.42) - open/close 4211 (time 1709787212.75 total 253.88 last 6.48) - open/close 4457 (time 1709787225.50 total 266.62 last 19.30) - open/close 4593 (time 1709787235.92 total 277.04 last 13.05) - open/close 4700 (time 1709787245.92 total 287.04 last 10.70) - open/close 4863 (time 1709787256.85 total 297.97 last 14.92) - open/close 5076 (time 1709787267.33 total 308.46 last 20.31) - open/close 5266 (time 1709787277.84 total 318.96 last 18.09) - open/close 5395 (time 1709787287.88 total 329.00 last 12.84) - open/close 5490 (time 1709787298.11 total 339.24 last 9.28) - open/close 5616 (time 1709787308.59 total 349.71 last 12.03) - open/close 5765 (time 1709787318.62 total 359.74 last 14.85) - open/close 6004 (time 1709787328.66 total 369.78 last 23.80) - open/close 6227 (time 1709787339.58 total 380.70 last 20.42) - open/close 6508 (time 1709787349.61 total 390.73 last 28.02) - open/close 6734 (time 1709787360.53 total 401.65 last 20.69) - open/close 6964 (time 1709787371.39 total 412.51 last 21.18) - open/close 7128 (time 1709787381.90 total 423.02 last 15.61) - open/close 7288 (time 1709787392.43 total 433.55 last 15.19) - open/close 7473 (time 1709787402.46 total 443.58 last 18.45) - open/close 7697 (time 1709787413.26 total 454.38 last 20.73) - open/close 7906 (time 1709787423.69 total 464.81 last 20.03) - open/close 8079 (time 1709787434.74 total 475.86 last 15.66) - open/close 8204 (time 1709787445.41 total 486.53 last 11.72) - open/close 8353 (time 1709787455.41 total 496.53 last 14.90) - open/close 8514 (time 1709787465.42 total 506.54 last 16.09) - open/close 8702 (time 1709787475.43 total 516.55 last 18.78) - open/close 8850 (time 1709787485.73 total 526.85 last 14.37) - open/close 9060 (time 1709787496.47 total 537.59 last 19.55) - open/close 9271 (time 1709787506.65 total 547.77 last 20.74) - open/close 9439 (time 1709787516.68 total 557.80 last 16.75) - open/close 9845 (time 1709787526.68 total 567.80 last 40.59) - open/close 10000 (time 1709787529.27 total 570.40 last 59.74) - open/close 10460 (time 1709787539.29 total 580.41 last 45.92) - open/close 10914 (time 1709787549.30 total 590.42 last 45.36) - open/close 11384 (time 1709787559.31 total 600.43 last 46.97) - open/close 11885 (time 1709787569.31 total 610.44 last 50.05) - open/close 12389 (time 1709787579.32 total 620.44 last 50.40) - open/close 12816 (time 1709787589.32 total 630.44 last 42.69) - open/close 13227 (time 1709787599.34 total 640.47 last 40.99) - open/close 13663 (time 1709787609.36 total 650.49 last 43.52) - open/close 14043 (time 1709787619.39 total 660.51 last 37.90) - open/close 14493 (time 1709787629.41 total 670.53 last 44.91) - open/close 14968 (time 1709787639.41 total 680.53 last 47.50) - open/close 15389 (time 1709787649.41 total 690.53 last 42.09) - open/close 15789 (time 1709787659.44 total 700.56 last 39.88) - open/close 16225 (time 1709787669.46 total 710.59 last 43.50) - open/close 16599 (time 1709787679.48 total 720.61 last 37.32) - open/close 16992 (time 1709787689.50 total 730.62 last 39.26) - open/close 17347 (time 1709787699.50 total 740.63 last 35.47) - open/close 17702 (time 1709787709.51 total 750.64 last 35.46) - open/close 18014 (time 1709787719.52 total 760.64 last 31.19) - open/close 18325 (time 1709787729.54 total 770.67 last 31.02) - open/close 18647 (time 1709787739.60 total 780.72 last 32.01) - open/close 18960 (time 1709787749.61 total 790.74 last 31.26) - open/close 19272 (time 1709787759.65 total 800.77 last 31.09) - open/close 19622 (time 1709787769.67 total 810.79 last 34.93) - open/close 20000 (time 1709787779.66 total 820.79 last 37.82) - open/close 20336 (time 1709787789.67 total 830.80 last 33.57) - open/close 20677 (time 1709787799.68 total 840.80 last 34.09) - open/close 21023 (time 1709787809.68 total 850.80 last 34.58) - open/close 21337 (time 1709787819.68 total 860.81 last 31.39) - open/close 21672 (time 1709787829.70 total 870.82 last 33.45) - open/close 22004 (time 1709787839.77 total 880.89 last 32.98) - open/close 22287 (time 1709787849.78 total 890.90 last 28.27) - open/close 22572 (time 1709787859.78 total 900.90 last 28.49) - open/close 22853 (time 1709787869.79 total 910.92 last 28.06) - open/close 23162 (time 1709787879.83 total 920.95 last 30.79) - open/close 23460 (time 1709787889.87 total 931.00 last 29.67) - open/close 23754 (time 1709787899.89 total 941.01 last 29.35) - open/close 24021 (time 1709787909.91 total 951.03 last 26.65) - open/close 24336 (time 1709787919.92 total 961.04 last 31.46) - open/close 24608 (time 1709787929.98 total 971.10 last 27.05) - open/close 24919 (time 1709787939.99 total 981.11 last 31.06) - open/close 25238 (time 1709787950.01 total 991.14 last 31.83) - open/close 25590 (time 1709787960.05 total 1001.17 last 35.08) - open/close 25950 (time 1709787970.05 total 1011.17 last 35.99) - open/close 26289 (time 1709787980.07 total 1021.20 last 33.82) - open/close 26627 (time 1709787990.09 total 1031.21 last 33.75) - open/close 26988 (time 1709788000.10 total 1041.23 last 36.05) - open/close 27382 (time 1709788010.13 total 1051.25 last 39.29) - open/close 27802 (time 1709788020.14 total 1061.26 last 41.97) - open/close 28233 (time 1709788030.16 total 1071.28 last 43.03) - open/close 28659 (time 1709788040.16 total 1081.28 last 42.59) - open/close 29061 (time 1709788050.17 total 1091.30 last 40.14) - open/close 29479 (time 1709788060.20 total 1101.32 last 41.70) - open/close 29871 (time 1709788070.21 total 1111.33 last 39.17) - open/close 30000 (time 1709788073.67 total 1114.79 last 37.26) - open/close 30377 (time 1709788083.67 total 1124.80 last 37.67) - open/close 30771 (time 1709788093.70 total 1134.82 last 39.29) - open/close 31139 (time 1709788103.71 total 1144.83 last 36.77) - open/close 31495 (time 1709788113.73 total 1154.85 last 35.52) - open/close 31870 (time 1709788123.74 total 1164.86 last 37.48) - open/close 32235 (time 1709788133.75 total 1174.88 last 36.43) - open/close 32594 (time 1709788143.77 total 1184.89 last 35.86) - open/close 32956 (time 1709788153.79 total 1194.91 last 36.11) - open/close 33320 (time 1709788163.81 total 1204.93 last 36.33) - open/close 33692 (time 1709788173.82 total 1214.94 last 37.15) - open/close 34048 (time 1709788183.83 total 1224.96 last 35.56) - open/close 34388 (time 1709788193.86 total 1234.98 last 33.92) - open/close 34723 (time 1709788203.88 total 1245.00 last 33.43) - open/close 35063 (time 1709788213.90 total 1255.02 last 33.92) - open/close 35408 (time 1709788223.91 total 1265.03 last 34.48) - open/close 35732 (time 1709788233.92 total 1275.04 last 32.37) - open/close 36056 (time 1709788243.93 total 1285.06 last 32.35) - open/close 36377 (time 1709788253.95 total 1295.07 last 32.06) - open/close 36696 (time 1709788263.96 total 1305.08 last 31.85) - open/close 37016 (time 1709788273.98 total 1315.10 last 31.95) - open/close 37328 (time 1709788284.01 total 1325.13 last 31.10) - open/close 37615 (time 1709788294.01 total 1335.13 last 28.70) - open/close 37910 (time 1709788304.04 total 1345.16 last 29.42) - open/close 38198 (time 1709788314.04 total 1355.16 last 28.80) - open/close 38480 (time 1709788324.05 total 1365.17 last 28.18) - open/close 38758 (time 1709788334.08 total 1375.20 last 27.71) - open/close 39030 (time 1709788344.10 total 1385.22 last 27.15) - open/close 39307 (time 1709788354.12 total 1395.24 last 27.64) - open/close 39588 (time 1709788364.14 total 1405.27 last 28.04) - open/close 39862 (time 1709788374.18 total 1415.30 last 27.31) - open/close 40000 (time 1709788379.39 total 1420.51 last 26.45) - open/close 40273 (time 1709788389.42 total 1430.54 last 27.23) - open/close 40540 (time 1709788399.45 total 1440.58 last 26.60) - open/close 40830 (time 1709788409.47 total 1450.59 last 28.96) - open/close 41122 (time 1709788419.47 total 1460.59 last 29.19) - open/close 41414 (time 1709788429.50 total 1470.62 last 29.12) - open/close 41722 (time 1709788439.50 total 1480.63 last 30.78) - open/close 42021 (time 1709788449.51 total 1490.64 last 29.87) - open/close 42304 (time 1709788459.54 total 1500.66 last 28.24) - open/close 42580 (time 1709788469.57 total 1510.69 last 27.51) - open/close 42868 (time 1709788479.57 total 1520.69 last 28.79) - open/close 43167 (time 1709788489.58 total 1530.71 last 29.87) - open/close 43471 (time 1709788499.61 total 1540.74 last 30.31) - open/close 43777 (time 1709788509.64 total 1550.76 last 30.52) - open/close 44082 (time 1709788519.66 total 1560.78 last 30.44) - open/close 44378 (time 1709788529.68 total 1570.80 last 29.53) - open/close 44681 (time 1709788539.69 total 1580.81 last 30.27) - open/close 44979 (time 1709788549.71 total 1590.84 last 29.73) - open/close 45274 (time 1709788559.74 total 1600.86 last 29.43) - open/close 45565 (time 1709788569.76 total 1610.88 last 29.04) - open/close 45849 (time 1709788579.76 total 1620.88 last 28.39) - open/close 46140 (time 1709788589.78 total 1630.90 last 29.04) - open/close 46430 (time 1709788599.81 total 1640.93 last 28.92) - open/close 46724 (time 1709788609.83 total 1650.95 last 29.34) - open/close 47014 (time 1709788619.86 total 1660.98 last 28.91) - open/close 47297 (time 1709788629.88 total 1671.00 last 28.25) - open/close 47574 (time 1709788639.89 total 1681.02 last 27.65) - open/close 47837 (time 1709788649.93 total 1691.05 last 26.20) - open/close 48101 (time 1709788659.95 total 1701.08 last 26.33) - open/close 48383 (time 1709788669.98 total 1711.11 last 28.11) - open/close 48648 (time 1709788680.01 total 1721.14 last 26.42) - open/close 48919 (time 1709788690.02 total 1731.15 last 27.08) - open/close 49189 (time 1709788700.06 total 1741.18 last 26.91) - open/close 49464 (time 1709788710.09 total 1751.21 last 27.40) - open/close 49743 (time 1709788720.10 total 1761.23 last 27.87) total: 50000 open/close in 1770.43 seconds: 28.24 ops/second - unlinked 0 (time 1709788730 ; total 0 ; last 0) - unlinked 10000 (time 1709789153 ; total 423 ; last 423) - unlinked 20000 (time 1709789508 ; total 778 ; last 355) - unlinked 30000 (time 1709789764 ; total 1034 ; last 256) - unlinked 40000 (time 1709789894 ; total 1164 ; last 130) total: 50000 unlinks in 1217 seconds: 41.084633 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg239-server unloading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2