== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:58 (1709787058) start mds service on oleg246-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg246-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock oleg246-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 146 (time 1709787106.52 total 12.36 last 11.81) - open/close 329 (time 1709787117.32 total 23.16 last 16.95) - open/close 476 (time 1709787127.51 total 33.35 last 14.42) - open/close 704 (time 1709787138.07 total 43.91 last 21.58) - open/close 825 (time 1709787148.75 total 54.59 last 11.33) - open/close 976 (time 1709787159.29 total 65.13 last 14.33) - open/close 1113 (time 1709787170.72 total 76.56 last 11.99) - open/close 1338 (time 1709787180.73 total 86.57 last 22.49) - open/close 1411 (time 1709787190.92 total 96.76 last 7.16) - open/close 1532 (time 1709787200.92 total 106.76 last 12.10) - open/close 1666 (time 1709787211.11 total 116.96 last 13.14) - open/close 1850 (time 1709787221.28 total 127.12 last 18.10) - open/close 2057 (time 1709787232.30 total 138.14 last 18.78) - open/close 2246 (time 1709787242.63 total 148.47 last 18.31) - open/close 2397 (time 1709787253.46 total 159.30 last 13.94) - open/close 2667 (time 1709787263.48 total 169.32 last 26.95) - open/close 2856 (time 1709787273.50 total 179.34 last 18.86) - open/close 3013 (time 1709787283.86 total 189.70 last 15.14) - open/close 3192 (time 1709787294.07 total 199.91 last 17.53) - open/close 3275 (time 1709787304.73 total 210.58 last 7.79) - open/close 3485 (time 1709787314.78 total 220.63 last 20.90) - open/close 3784 (time 1709787325.60 total 231.44 last 27.65) - open/close 4014 (time 1709787335.62 total 241.46 last 22.96) - open/close 4206 (time 1709787345.64 total 251.48 last 19.15) - open/close 4356 (time 1709787356.42 total 262.26 last 13.91) - open/close 4608 (time 1709787366.93 total 272.77 last 23.98) - open/close 4805 (time 1709787376.93 total 282.77 last 19.70) - open/close 4954 (time 1709787387.42 total 293.26 last 14.20) - open/close 5179 (time 1709787397.66 total 303.50 last 21.99) - open/close 5373 (time 1709787408.39 total 314.23 last 18.08) - open/close 5564 (time 1709787419.41 total 325.25 last 17.33) - open/close 5629 (time 1709787429.96 total 335.80 last 6.16) - open/close 5866 (time 1709787440.47 total 346.31 last 22.57) - open/close 6028 (time 1709787450.62 total 356.47 last 15.95) - open/close 6223 (time 1709787461.06 total 366.90 last 18.69) - open/close 6438 (time 1709787471.34 total 377.18 last 20.92) - open/close 6623 (time 1709787482.34 total 388.18 last 16.81) - open/close 6834 (time 1709787492.36 total 398.20 last 21.06) - open/close 7013 (time 1709787503.31 total 409.15 last 16.34) - open/close 7197 (time 1709787513.32 total 419.16 last 18.39) - open/close 7404 (time 1709787523.93 total 429.77 last 19.51) - open/close 7615 (time 1709787533.93 total 439.77 last 21.09) - open/close 7776 (time 1709787544.32 total 450.16 last 15.50) - open/close 7978 (time 1709787554.34 total 460.18 last 20.16) - open/close 8142 (time 1709787564.76 total 470.60 last 15.73) - open/close 8238 (time 1709787575.41 total 481.25 last 9.02) - open/close 8363 (time 1709787585.55 total 491.39 last 12.32) - open/close 8540 (time 1709787596.45 total 502.29 last 16.24) - open/close 8671 (time 1709787606.73 total 512.57 last 12.74) - open/close 8797 (time 1709787616.83 total 522.67 last 12.48) - open/close 8956 (time 1709787626.83 total 532.67 last 15.89) - open/close 9091 (time 1709787637.45 total 543.29 last 12.71) - open/close 9282 (time 1709787647.74 total 553.58 last 18.57) - open/close 9525 (time 1709787658.44 total 564.28 last 22.72) - open/close 9660 (time 1709787668.86 total 574.70 last 12.95) - open/close 9815 (time 1709787678.99 total 584.83 last 15.31) - open/close 9939 (time 1709787689.01 total 594.85 last 12.38) - open/close 10000 (time 1709787695.40 total 601.24 last 9.55) - open/close 10438 (time 1709787705.40 total 611.24 last 43.79) - open/close 10823 (time 1709787715.43 total 621.27 last 38.40) - open/close 11221 (time 1709787725.44 total 631.28 last 39.74) - open/close 11633 (time 1709787735.45 total 641.29 last 41.16) - open/close 12005 (time 1709787745.47 total 651.31 last 37.11) - open/close 12394 (time 1709787755.48 total 661.32 last 38.88) - open/close 12764 (time 1709787765.49 total 671.33 last 36.96) - open/close 13155 (time 1709787775.49 total 681.33 last 39.09) - open/close 13559 (time 1709787785.50 total 691.34 last 40.37) - open/close 13900 (time 1709787795.50 total 701.34 last 34.10) - open/close 14313 (time 1709787805.51 total 711.35 last 41.26) - open/close 14702 (time 1709787815.52 total 721.36 last 38.87) - open/close 15057 (time 1709787825.52 total 731.36 last 35.48) - open/close 15447 (time 1709787835.56 total 741.40 last 38.86) - open/close 15778 (time 1709787845.59 total 751.43 last 33.00) - open/close 16097 (time 1709787855.60 total 761.44 last 31.85) - open/close 16382 (time 1709787865.63 total 771.47 last 28.44) - open/close 16717 (time 1709787875.63 total 781.47 last 33.50) - open/close 17038 (time 1709787885.65 total 791.49 last 32.01) - open/close 17349 (time 1709787895.67 total 801.51 last 31.05) - open/close 17624 (time 1709787905.69 total 811.53 last 27.44) - open/close 17928 (time 1709787915.70 total 821.54 last 30.37) - open/close 18221 (time 1709787925.74 total 831.58 last 29.19) - open/close 18523 (time 1709787935.75 total 841.59 last 30.18) - open/close 18842 (time 1709787945.77 total 851.61 last 31.82) - open/close 19203 (time 1709787955.79 total 861.63 last 36.03) - open/close 19589 (time 1709787965.81 total 871.65 last 38.52) - open/close 19950 (time 1709787975.82 total 881.66 last 36.08) - open/close 20295 (time 1709787985.83 total 891.67 last 34.45) - open/close 20666 (time 1709787995.84 total 901.68 last 37.07) - open/close 21070 (time 1709788005.87 total 911.71 last 40.29) - open/close 21479 (time 1709788015.88 total 921.72 last 40.85) - open/close 21915 (time 1709788025.89 total 931.73 last 43.57) - open/close 22344 (time 1709788035.90 total 941.74 last 42.84) - open/close 22783 (time 1709788045.91 total 951.76 last 43.84) - open/close 23209 (time 1709788055.93 total 961.77 last 42.52) - open/close 23636 (time 1709788065.95 total 971.79 last 42.65) - open/close 24049 (time 1709788075.96 total 981.80 last 41.23) - open/close 24493 (time 1709788085.97 total 991.81 last 44.35) - open/close 24909 (time 1709788095.99 total 1001.83 last 41.53) - open/close 25310 (time 1709788106.00 total 1011.84 last 40.05) - open/close 25687 (time 1709788116.02 total 1021.86 last 37.63) - open/close 26082 (time 1709788126.04 total 1031.88 last 39.42) - open/close 26493 (time 1709788136.04 total 1041.88 last 41.09) - open/close 26882 (time 1709788146.05 total 1051.89 last 38.86) - open/close 27280 (time 1709788156.06 total 1061.90 last 39.77) - open/close 27687 (time 1709788166.07 total 1071.91 last 40.66) - open/close 28093 (time 1709788176.08 total 1081.92 last 40.55) - open/close 28495 (time 1709788186.09 total 1091.93 last 40.15) - open/close 28880 (time 1709788196.11 total 1101.96 last 38.41) - open/close 29253 (time 1709788206.13 total 1111.97 last 37.24) - open/close 29648 (time 1709788216.14 total 1121.98 last 39.46) - open/close 30000 (time 1709788225.19 total 1131.03 last 38.88) - open/close 30375 (time 1709788235.20 total 1141.04 last 37.48) - open/close 30746 (time 1709788245.21 total 1151.05 last 37.05) - open/close 31110 (time 1709788255.22 total 1161.06 last 36.38) - open/close 31453 (time 1709788265.22 total 1171.06 last 34.29) - open/close 31816 (time 1709788275.23 total 1181.07 last 36.28) - open/close 32144 (time 1709788285.24 total 1191.08 last 32.75) - open/close 32466 (time 1709788295.26 total 1201.10 last 32.14) - open/close 32787 (time 1709788305.27 total 1211.11 last 32.07) - open/close 33110 (time 1709788315.28 total 1221.12 last 32.27) - open/close 33427 (time 1709788325.30 total 1231.14 last 31.64) - open/close 33732 (time 1709788335.32 total 1241.16 last 30.42) - open/close 34037 (time 1709788345.33 total 1251.17 last 30.48) - open/close 34341 (time 1709788355.34 total 1261.18 last 30.38) - open/close 34654 (time 1709788365.37 total 1271.21 last 31.21) - open/close 34945 (time 1709788375.40 total 1281.24 last 29.01) - open/close 35243 (time 1709788385.41 total 1291.25 last 29.77) - open/close 35527 (time 1709788395.42 total 1301.26 last 28.37) - open/close 35839 (time 1709788405.45 total 1311.29 last 31.10) - open/close 36148 (time 1709788415.46 total 1321.30 last 30.86) - open/close 36462 (time 1709788425.47 total 1331.31 last 31.37) - open/close 36784 (time 1709788435.49 total 1341.33 last 32.16) - open/close 37105 (time 1709788445.49 total 1351.33 last 32.07) - open/close 37425 (time 1709788455.51 total 1361.35 last 31.95) - open/close 37721 (time 1709788465.54 total 1371.38 last 29.52) - open/close 38032 (time 1709788475.54 total 1381.38 last 31.09) - open/close 38354 (time 1709788485.54 total 1391.38 last 32.19) - open/close 38677 (time 1709788495.56 total 1401.40 last 32.23) - open/close 38996 (time 1709788505.58 total 1411.42 last 31.83) - open/close 39321 (time 1709788515.60 total 1421.44 last 32.45) - open/close 39633 (time 1709788525.60 total 1431.44 last 31.19) - open/close 39941 (time 1709788535.63 total 1441.47 last 30.71) - open/close 40254 (time 1709788545.66 total 1451.50 last 31.21) - open/close 40561 (time 1709788555.66 total 1461.50 last 30.69) - open/close 40860 (time 1709788565.67 total 1471.51 last 29.87) - open/close 41155 (time 1709788575.70 total 1481.54 last 29.41) - open/close 41451 (time 1709788585.73 total 1491.57 last 29.53) - open/close 41762 (time 1709788595.75 total 1501.59 last 31.04) - open/close 42062 (time 1709788605.75 total 1511.59 last 29.99) - open/close 42361 (time 1709788615.77 total 1521.61 last 29.84) - open/close 42658 (time 1709788625.77 total 1531.61 last 29.70) - open/close 42939 (time 1709788635.78 total 1541.62 last 28.08) - open/close 43216 (time 1709788645.79 total 1551.64 last 27.66) - open/close 43482 (time 1709788655.82 total 1561.66 last 26.53) - open/close 43768 (time 1709788665.84 total 1571.69 last 28.53) - open/close 44041 (time 1709788675.86 total 1581.70 last 27.25) - open/close 44320 (time 1709788685.89 total 1591.73 last 27.82) - open/close 44606 (time 1709788695.92 total 1601.76 last 28.51) - open/close 44894 (time 1709788705.95 total 1611.79 last 28.73) - open/close 45180 (time 1709788715.97 total 1621.81 last 28.54) - open/close 45483 (time 1709788725.98 total 1631.82 last 30.27) - open/close 45770 (time 1709788736.00 total 1641.84 last 28.63) - open/close 46065 (time 1709788746.00 total 1651.85 last 29.49) - open/close 46347 (time 1709788756.03 total 1661.87 last 28.13) - open/close 46637 (time 1709788766.05 total 1671.89 last 28.93) - open/close 46923 (time 1709788776.09 total 1681.93 last 28.51) - open/close 47201 (time 1709788786.09 total 1691.93 last 27.80) - open/close 47472 (time 1709788796.09 total 1701.94 last 27.07) - open/close 47745 (time 1709788806.13 total 1711.97 last 27.21) - open/close 48016 (time 1709788816.14 total 1721.98 last 27.08) - open/close 48274 (time 1709788826.15 total 1731.99 last 25.77) - open/close 48534 (time 1709788836.17 total 1742.01 last 25.95) - open/close 48795 (time 1709788846.20 total 1752.04 last 26.02) - open/close 49059 (time 1709788856.21 total 1762.05 last 26.36) - open/close 49321 (time 1709788866.24 total 1772.08 last 26.14) - open/close 49584 (time 1709788876.24 total 1782.08 last 26.29) - open/close 49841 (time 1709788886.28 total 1792.12 last 25.60) total: 50000 open/close in 1798.32 seconds: 27.80 ops/second - unlinked 0 (time 1709788893 ; total 0 ; last 0) - unlinked 10000 (time 1709789377 ; total 484 ; last 484) - unlinked 20000 (time 1709789733 ; total 840 ; last 356) - unlinked 30000 (time 1709789982 ; total 1089 ; last 249) - unlinked 40000 (time 1709790091 ; total 1198 ; last 109) total: 50000 unlinks in 1233 seconds: 40.551502 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server unloading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2