== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:13 (1709787013) start mds service on oleg355-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg355-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg355-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock oleg355-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1709787057.03 total 10.41 last 23.05) - open/close 515 (time 1709787067.05 total 20.43 last 27.45) - open/close 720 (time 1709787077.61 total 30.99 last 19.42) - open/close 861 (time 1709787087.61 total 40.99 last 14.10) - open/close 1013 (time 1709787097.88 total 51.26 last 14.80) - open/close 1228 (time 1709787108.69 total 62.07 last 19.89) - open/close 1417 (time 1709787119.43 total 72.81 last 17.60) - open/close 1651 (time 1709787130.08 total 83.46 last 21.97) - open/close 1777 (time 1709787140.92 total 94.30 last 11.63) - open/close 1984 (time 1709787150.92 total 104.30 last 20.69) - open/close 2139 (time 1709787163.31 total 116.69 last 12.51) - open/close 2444 (time 1709787174.29 total 127.67 last 27.78) - open/close 2640 (time 1709787184.56 total 137.94 last 19.08) - open/close 2840 (time 1709787195.26 total 148.64 last 18.69) - open/close 2969 (time 1709787205.53 total 158.92 last 12.56) - open/close 3163 (time 1709787224.28 total 177.66 last 10.35) - open/close 3327 (time 1709787234.29 total 187.67 last 16.38) - open/close 3517 (time 1709787245.01 total 198.40 last 17.72) - open/close 3658 (time 1709787255.92 total 209.30 last 12.93) - open/close 3856 (time 1709787266.16 total 219.54 last 19.33) - open/close 4029 (time 1709787276.19 total 229.57 last 17.25) - open/close 4202 (time 1709787286.94 total 240.32 last 16.09) - open/close 4401 (time 1709787297.22 total 250.61 last 19.35) - open/close 4507 (time 1709787308.15 total 261.53 last 9.70) - open/close 4665 (time 1709787318.18 total 271.56 last 15.76) - open/close 4778 (time 1709787328.44 total 281.83 last 11.00) - open/close 4983 (time 1709787338.46 total 291.84 last 20.47) - open/close 5155 (time 1709787348.46 total 301.85 last 17.19) - open/close 5343 (time 1709787359.32 total 312.70 last 17.32) - open/close 5515 (time 1709787369.89 total 323.28 last 16.26) - open/close 5637 (time 1709787380.29 total 333.67 last 11.74) - open/close 5796 (time 1709787390.70 total 344.09 last 15.26) - open/close 5908 (time 1709787401.41 total 354.79 last 10.46) - open/close 6045 (time 1709787411.81 total 365.19 last 13.17) - open/close 6249 (time 1709787423.45 total 376.83 last 17.53) - open/close 6373 (time 1709787434.07 total 387.45 last 11.68) - open/close 6506 (time 1709787445.10 total 398.48 last 12.06) - open/close 6655 (time 1709787455.40 total 408.79 last 14.46) - open/close 6820 (time 1709787465.63 total 419.01 last 16.14) - open/close 7009 (time 1709787476.08 total 429.46 last 18.09) - open/close 7218 (time 1709787487.00 total 440.39 last 19.13) - open/close 7327 (time 1709787497.93 total 451.31 last 9.97) - open/close 7532 (time 1709787507.97 total 461.36 last 20.42) - open/close 7682 (time 1709787518.63 total 472.02 last 14.07) - open/close 7899 (time 1709787529.10 total 482.48 last 20.73) - open/close 8097 (time 1709787539.99 total 493.37 last 18.19) - open/close 8321 (time 1709787550.00 total 503.38 last 22.36) - open/close 8486 (time 1709787560.89 total 514.27 last 15.16) - open/close 8682 (time 1709787571.71 total 525.09 last 18.11) - open/close 8834 (time 1709787582.15 total 535.53 last 14.56) - open/close 8967 (time 1709787593.05 total 546.44 last 12.20) - open/close 9148 (time 1709787603.56 total 556.94 last 17.23) - open/close 9297 (time 1709787613.59 total 566.98 last 14.85) - open/close 9475 (time 1709787623.61 total 576.99 last 17.77) - open/close 9619 (time 1709787634.59 total 587.97 last 13.12) - open/close 9790 (time 1709787645.51 total 598.90 last 15.65) - open/close 9976 (time 1709787655.52 total 608.91 last 18.58) - open/close 10142 (time 1709787665.53 total 618.91 last 16.59) - open/close 10503 (time 1709787675.54 total 628.92 last 36.07) - open/close 10903 (time 1709787685.55 total 638.93 last 39.97) - open/close 11294 (time 1709787695.58 total 648.96 last 39.00) - open/close 11660 (time 1709787705.58 total 658.96 last 36.58) - open/close 12043 (time 1709787715.62 total 669.00 last 38.17) - open/close 12483 (time 1709787725.62 total 679.00 last 44.00) - open/close 12906 (time 1709787735.64 total 689.02 last 42.20) - open/close 13275 (time 1709787745.66 total 699.04 last 36.82) - open/close 13572 (time 1709787755.74 total 709.12 last 29.46) - open/close 13958 (time 1709787765.75 total 719.13 last 38.57) - open/close 14335 (time 1709787775.76 total 729.15 last 37.64) - open/close 14693 (time 1709787785.80 total 739.18 last 35.68) - open/close 15051 (time 1709787795.80 total 749.18 last 35.80) - open/close 15482 (time 1709787805.80 total 759.19 last 43.08) - open/close 15899 (time 1709787815.84 total 769.22 last 41.57) - open/close 16251 (time 1709787825.88 total 779.26 last 35.06) - open/close 16647 (time 1709787835.89 total 789.27 last 39.54) - open/close 17005 (time 1709787845.92 total 799.30 last 35.71) - open/close 17374 (time 1709787855.93 total 809.31 last 36.86) - open/close 17741 (time 1709787865.94 total 819.33 last 36.63) - open/close 18046 (time 1709787875.95 total 829.34 last 30.47) - open/close 18383 (time 1709787885.96 total 839.34 last 33.68) - open/close 18710 (time 1709787895.96 total 849.35 last 32.69) - open/close 19074 (time 1709787906.00 total 859.38 last 36.27) - open/close 19404 (time 1709787916.01 total 869.39 last 32.96) - open/close 19718 (time 1709787926.02 total 879.40 last 31.37) - open/close 20000 (time 1709787935.11 total 888.49 last 31.01) - open/close 20310 (time 1709787945.13 total 898.51 last 30.94) - open/close 20625 (time 1709787955.16 total 908.54 last 31.41) - open/close 20948 (time 1709787965.18 total 918.56 last 32.24) - open/close 21273 (time 1709787975.18 total 928.56 last 32.48) - open/close 21617 (time 1709787985.20 total 938.58 last 34.35) - open/close 21693 (time 1709787998.32 total 951.71 last 5.79) - open/close 22031 (time 1709788008.35 total 961.73 last 33.72) - open/close 22364 (time 1709788018.37 total 971.75 last 33.23) - open/close 22669 (time 1709788028.39 total 981.77 last 30.44) - open/close 22992 (time 1709788038.40 total 991.78 last 32.26) - open/close 23281 (time 1709788048.41 total 1001.79 last 28.87) - open/close 23584 (time 1709788058.43 total 1011.81 last 30.25) - open/close 23868 (time 1709788068.45 total 1021.83 last 28.35) - open/close 24217 (time 1709788078.45 total 1031.83 last 34.88) - open/close 24614 (time 1709788088.47 total 1041.85 last 39.63) - open/close 25006 (time 1709788098.49 total 1051.88 last 39.11) - open/close 25394 (time 1709788108.51 total 1061.89 last 38.75) - open/close 25777 (time 1709788118.52 total 1071.90 last 38.25) - open/close 26156 (time 1709788128.53 total 1081.91 last 37.87) - open/close 26549 (time 1709788138.54 total 1091.92 last 39.26) - open/close 26911 (time 1709788148.54 total 1101.92 last 36.18) - open/close 27254 (time 1709788158.56 total 1111.94 last 34.23) - open/close 27622 (time 1709788168.58 total 1121.96 last 36.74) - open/close 27952 (time 1709788178.58 total 1131.96 last 33.00) - open/close 28287 (time 1709788188.59 total 1141.98 last 33.45) - open/close 28605 (time 1709788198.61 total 1151.99 last 31.75) - open/close 28935 (time 1709788208.63 total 1162.02 last 32.93) - open/close 29257 (time 1709788218.65 total 1172.03 last 32.15) - open/close 29590 (time 1709788228.66 total 1182.04 last 33.26) - open/close 29904 (time 1709788238.68 total 1192.06 last 31.35) - open/close 30000 (time 1709788241.67 total 1195.06 last 32.04) - open/close 30296 (time 1709788251.68 total 1205.07 last 29.57) - open/close 30601 (time 1709788261.69 total 1215.07 last 30.49) - open/close 30896 (time 1709788271.71 total 1225.09 last 29.43) - open/close 31197 (time 1709788281.74 total 1235.13 last 30.00) - open/close 31477 (time 1709788291.76 total 1245.14 last 27.97) - open/close 31766 (time 1709788301.78 total 1255.16 last 28.84) - open/close 32046 (time 1709788311.81 total 1265.19 last 27.90) - open/close 32338 (time 1709788321.83 total 1275.21 last 29.14) - open/close 32643 (time 1709788331.83 total 1285.22 last 30.50) - open/close 32942 (time 1709788341.84 total 1295.22 last 29.88) - open/close 33241 (time 1709788351.87 total 1305.25 last 29.81) - open/close 33543 (time 1709788361.90 total 1315.28 last 30.13) - open/close 33825 (time 1709788371.90 total 1325.28 last 28.19) - open/close 34108 (time 1709788381.94 total 1335.32 last 28.19) - open/close 34382 (time 1709788391.95 total 1345.33 last 27.38) - open/close 34678 (time 1709788401.99 total 1355.37 last 29.48) - open/close 34943 (time 1709788422.77 total 1376.16 last 12.75) - open/close 35222 (time 1709788432.78 total 1386.16 last 27.89) - open/close 35513 (time 1709788442.79 total 1396.17 last 29.07) - open/close 35788 (time 1709788452.80 total 1406.18 last 27.48) - open/close 36066 (time 1709788462.82 total 1416.20 last 27.75) - open/close 36356 (time 1709788472.85 total 1426.23 last 28.91) - open/close 36639 (time 1709788482.86 total 1436.24 last 28.27) - open/close 36889 (time 1709788492.89 total 1446.27 last 24.91) - open/close 37157 (time 1709788502.90 total 1456.29 last 26.77) - open/close 37432 (time 1709788512.91 total 1466.29 last 27.49) - open/close 37712 (time 1709788522.93 total 1476.32 last 27.93) - open/close 37977 (time 1709788532.96 total 1486.34 last 26.44) - open/close 38246 (time 1709788542.97 total 1496.35 last 26.87) - open/close 38515 (time 1709788552.98 total 1506.36 last 26.87) - open/close 38779 (time 1709788563.01 total 1516.39 last 26.32) - open/close 39055 (time 1709788573.01 total 1526.40 last 27.57) - open/close 39323 (time 1709788583.02 total 1536.40 last 26.79) - open/close 39596 (time 1709788593.02 total 1546.40 last 27.29) - open/close 39861 (time 1709788603.02 total 1556.41 last 26.50) - open/close 40000 (time 1709788607.90 total 1561.28 last 28.51) - open/close 40279 (time 1709788617.93 total 1571.31 last 27.82) - open/close 40558 (time 1709788627.93 total 1581.32 last 27.89) - open/close 40820 (time 1709788637.95 total 1591.33 last 26.16) - open/close 41069 (time 1709788647.99 total 1601.37 last 24.80) - open/close 41304 (time 1709788658.01 total 1611.40 last 23.44) - open/close 41544 (time 1709788668.02 total 1621.40 last 24.00) - open/close 41797 (time 1709788678.05 total 1631.43 last 25.21) - open/close 42037 (time 1709788688.07 total 1641.45 last 23.94) - open/close 42292 (time 1709788698.08 total 1651.46 last 25.48) - open/close 42544 (time 1709788708.11 total 1661.49 last 25.14) - open/close 42784 (time 1709788718.12 total 1671.50 last 23.96) - open/close 43046 (time 1709788728.13 total 1681.51 last 26.17) - open/close 43296 (time 1709788738.15 total 1691.53 last 24.96) - open/close 43557 (time 1709788748.19 total 1701.57 last 26.00) - open/close 43801 (time 1709788758.19 total 1711.57 last 24.40) - open/close 44040 (time 1709788768.21 total 1721.59 last 23.85) - open/close 44258 (time 1709788778.27 total 1731.65 last 21.66) - open/close 44491 (time 1709788788.28 total 1741.67 last 23.27) - open/close 44717 (time 1709788798.29 total 1751.67 last 22.59) - open/close 44939 (time 1709788808.34 total 1761.72 last 22.10) - open/close 45166 (time 1709788818.35 total 1771.74 last 22.66) - open/close 45381 (time 1709788828.38 total 1781.76 last 21.44) - open/close 45607 (time 1709788838.40 total 1791.79 last 22.55) - open/close 45827 (time 1709788848.40 total 1801.79 last 22.00) - open/close 46050 (time 1709788858.43 total 1811.81 last 22.24) - open/close 46267 (time 1709788868.45 total 1821.83 last 21.66) - open/close 46489 (time 1709788878.46 total 1831.84 last 22.19) - open/close 46724 (time 1709788888.49 total 1841.87 last 23.44) - open/close 46975 (time 1709788898.50 total 1851.88 last 25.06) - open/close 47203 (time 1709788908.50 total 1861.88 last 22.80) - open/close 47445 (time 1709788918.50 total 1871.88 last 24.20) - open/close 47686 (time 1709788928.50 total 1881.88 last 24.10) - open/close 47912 (time 1709788938.52 total 1891.91 last 22.55) - open/close 48134 (time 1709788948.54 total 1901.92 last 22.18) - open/close 48352 (time 1709788958.54 total 1911.92 last 21.80) - open/close 48573 (time 1709788968.56 total 1921.95 last 22.04) - open/close 48813 (time 1709788978.57 total 1931.95 last 23.98) - open/close 49039 (time 1709788988.58 total 1941.96 last 22.59) - open/close 49259 (time 1709788998.58 total 1951.96 last 21.99) - open/close 49484 (time 1709789008.61 total 1961.99 last 22.44) - open/close 49700 (time 1709789018.63 total 1972.01 last 21.55) - open/close 49916 (time 1709789028.67 total 1982.05 last 21.52) total: 50000 open/close in 1985.92 seconds: 25.18 ops/second - unlinked 0 (time 1709789034 ; total 1 ; last 1) - unlinked 10000 (time 1709789544 ; total 511 ; last 510) - unlinked 20000 (time 1709789977 ; total 944 ; last 433) - unlinked 30000 (time 1709790274 ; total 1241 ; last 297) - unlinked 40000 (time 1709790394 ; total 1361 ; last 120) total: 50000 unlinks in 1397 seconds: 35.790981 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server unloading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2