== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:56 (1709787176) start mds service on oleg420-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1709787207.12 total 10.00 last 18.10) - open/close 284 (time 1709787217.52 total 20.41 last 9.90) - open/close 478 (time 1709787228.52 total 31.40 last 17.64) - open/close 725 (time 1709787239.20 total 42.09 last 23.12) - open/close 803 (time 1709787249.21 total 52.09 last 7.80) - open/close 1042 (time 1709787260.12 total 63.00 last 21.90) - open/close 1177 (time 1709787271.87 total 74.76 last 11.48) - open/close 1395 (time 1709787282.82 total 85.70 last 19.92) - open/close 1581 (time 1709787293.79 total 96.67 last 16.95) - open/close 1710 (time 1709787304.76 total 107.64 last 11.76) - open/close 1912 (time 1709787314.76 total 117.65 last 20.19) - open/close 2066 (time 1709787324.82 total 127.70 last 15.32) - open/close 2215 (time 1709787335.17 total 138.05 last 14.40) - open/close 2431 (time 1709787345.29 total 148.17 last 21.34) - open/close 2646 (time 1709787356.22 total 159.10 last 19.68) - open/close 2783 (time 1709787366.23 total 169.11 last 13.69) - open/close 2968 (time 1709787377.11 total 179.99 last 17.00) - open/close 3159 (time 1709787387.17 total 190.05 last 18.99) - open/close 3297 (time 1709787397.21 total 200.09 last 13.74) - open/close 3535 (time 1709787407.28 total 210.16 last 23.63) - open/close 3748 (time 1709787418.17 total 221.05 last 19.57) - open/close 3878 (time 1709787428.53 total 231.41 last 12.55) - open/close 4057 (time 1709787438.54 total 241.43 last 17.87) - open/close 4213 (time 1709787448.55 total 251.43 last 15.59) - open/close 4443 (time 1709787458.86 total 261.74 last 22.31) - open/close 4727 (time 1709787468.88 total 271.77 last 28.32) - open/close 4845 (time 1709787479.97 total 282.85 last 10.65) - open/close 4978 (time 1709787490.08 total 292.96 last 13.15) - open/close 5113 (time 1709787500.63 total 303.51 last 12.80) - open/close 5324 (time 1709787511.28 total 314.17 last 19.80) - open/close 5553 (time 1709787521.38 total 324.26 last 22.68) - open/close 5703 (time 1709787532.21 total 335.10 last 13.85) - open/close 5878 (time 1709787542.79 total 345.67 last 16.54) - open/close 6029 (time 1709787553.56 total 356.45 last 14.02) - open/close 6242 (time 1709787564.18 total 367.06 last 20.06) - open/close 6441 (time 1709787574.19 total 377.07 last 19.88) - open/close 6674 (time 1709787586.43 total 389.31 last 19.04) - open/close 6889 (time 1709787596.76 total 399.64 last 20.81) - open/close 7224 (time 1709787607.55 total 410.43 last 31.04) - open/close 7429 (time 1709787617.70 total 420.58 last 20.21) - open/close 7689 (time 1709787627.70 total 430.59 last 25.98) - open/close 7875 (time 1709787637.71 total 440.59 last 18.59) - open/close 8182 (time 1709787648.55 total 451.43 last 28.31) - open/close 8456 (time 1709787658.56 total 461.44 last 27.37) - open/close 8751 (time 1709787669.44 total 472.32 last 27.12) - open/close 8885 (time 1709787680.37 total 483.26 last 12.26) - open/close 9051 (time 1709787691.25 total 494.14 last 15.26) - open/close 9190 (time 1709787702.00 total 504.89 last 12.93) - open/close 9337 (time 1709787712.01 total 514.89 last 14.69) - open/close 9496 (time 1709787722.02 total 524.90 last 15.89) - open/close 9903 (time 1709787732.02 total 534.91 last 40.67) - open/close 10654 (time 1709787742.03 total 544.92 last 75.04) - open/close 11284 (time 1709787752.03 total 554.92 last 62.99) - open/close 11853 (time 1709787762.03 total 564.92 last 56.89) - open/close 12325 (time 1709787782.37 total 585.25 last 23.21) - open/close 12804 (time 1709787792.39 total 595.27 last 47.82) - open/close 13340 (time 1709787802.40 total 605.28 last 53.54) - open/close 13885 (time 1709787812.41 total 615.29 last 54.45) - open/close 14385 (time 1709787822.43 total 625.32 last 49.87) - open/close 14860 (time 1709787832.44 total 635.32 last 47.49) - open/close 15362 (time 1709787842.44 total 645.33 last 50.16) - open/close 15885 (time 1709787852.45 total 655.34 last 52.25) - open/close 16451 (time 1709787862.46 total 665.35 last 56.54) - open/close 16988 (time 1709787872.48 total 675.36 last 53.62) - open/close 17540 (time 1709787882.48 total 685.36 last 55.19) - open/close 18050 (time 1709787892.49 total 695.38 last 50.92) - open/close 18552 (time 1709787902.50 total 705.38 last 50.17) - open/close 19012 (time 1709787912.50 total 715.38 last 45.99) - open/close 19483 (time 1709787922.52 total 725.40 last 47.03) - open/close 19948 (time 1709787932.52 total 735.40 last 46.50) - open/close 20395 (time 1709787942.53 total 745.41 last 44.66) - open/close 20837 (time 1709787952.55 total 755.43 last 44.10) - open/close 21276 (time 1709787962.57 total 765.45 last 43.80) - open/close 21714 (time 1709787972.59 total 775.47 last 43.72) - open/close 22117 (time 1709787982.62 total 785.50 last 40.19) - open/close 22551 (time 1709787992.63 total 795.51 last 43.36) - open/close 22983 (time 1709788002.64 total 805.52 last 43.15) - open/close 23410 (time 1709788012.65 total 815.53 last 42.66) - open/close 23827 (time 1709788022.66 total 825.55 last 41.62) - open/close 24234 (time 1709788032.71 total 835.59 last 40.52) - open/close 24633 (time 1709788042.72 total 845.61 last 39.84) - open/close 25003 (time 1709788052.75 total 855.63 last 36.91) - open/close 25396 (time 1709788062.76 total 865.64 last 39.27) - open/close 25778 (time 1709788072.76 total 875.64 last 38.20) - open/close 26153 (time 1709788082.77 total 885.65 last 37.45) - open/close 26542 (time 1709788092.79 total 895.67 last 38.83) - open/close 26918 (time 1709788102.81 total 905.70 last 37.52) - open/close 27289 (time 1709788112.83 total 915.71 last 37.04) - open/close 27673 (time 1709788122.84 total 925.72 last 38.37) - open/close 28028 (time 1709788132.84 total 935.72 last 35.48) - open/close 28374 (time 1709788142.86 total 945.75 last 34.52) - open/close 28730 (time 1709788152.88 total 955.77 last 35.53) - open/close 29036 (time 1709788162.94 total 965.82 last 30.43) - open/close 29359 (time 1709788172.97 total 975.85 last 32.21) - open/close 29708 (time 1709788182.97 total 985.86 last 34.88) - open/close 30000 (time 1709788191.53 total 994.42 last 34.12) - open/close 30317 (time 1709788201.55 total 1004.44 last 31.63) - open/close 30651 (time 1709788211.57 total 1014.45 last 33.35) - open/close 30954 (time 1709788221.57 total 1024.46 last 30.30) - open/close 31280 (time 1709788231.60 total 1034.49 last 32.50) - open/close 31584 (time 1709788241.61 total 1044.50 last 30.37) - open/close 31896 (time 1709788251.63 total 1054.51 last 31.14) - open/close 32238 (time 1709788261.66 total 1064.54 last 34.11) - open/close 32555 (time 1709788271.67 total 1074.55 last 31.67) - open/close 32862 (time 1709788281.68 total 1084.57 last 30.65) - open/close 33190 (time 1709788291.69 total 1094.58 last 32.77) - open/close 33506 (time 1709788301.71 total 1104.59 last 31.54) - open/close 33810 (time 1709788311.71 total 1114.60 last 30.40) - open/close 34107 (time 1709788321.72 total 1124.60 last 29.68) - open/close 34416 (time 1709788331.75 total 1134.63 last 30.81) - open/close 34707 (time 1709788341.76 total 1144.65 last 29.06) - open/close 34993 (time 1709788351.80 total 1154.68 last 28.50) - open/close 35271 (time 1709788361.80 total 1164.68 last 27.79) - open/close 35552 (time 1709788371.80 total 1174.69 last 28.08) - open/close 35827 (time 1709788381.82 total 1184.70 last 27.46) - open/close 36102 (time 1709788391.82 total 1194.70 last 27.50) - open/close 36382 (time 1709788401.85 total 1204.73 last 27.93) - open/close 36659 (time 1709788411.88 total 1214.76 last 27.61) - open/close 36934 (time 1709788421.88 total 1224.76 last 27.49) - open/close 37215 (time 1709788431.92 total 1234.80 last 28.01) - open/close 37488 (time 1709788441.94 total 1244.82 last 27.23) - open/close 37767 (time 1709788451.97 total 1254.85 last 27.83) - open/close 38047 (time 1709788461.99 total 1264.87 last 27.93) - open/close 38322 (time 1709788472.02 total 1274.91 last 27.41) - open/close 38595 (time 1709788482.05 total 1284.93 last 27.23) - open/close 38877 (time 1709788492.08 total 1294.96 last 28.13) - open/close 39161 (time 1709788502.10 total 1304.99 last 28.33) - open/close 39453 (time 1709788512.13 total 1315.01 last 29.12) - open/close 39709 (time 1709788522.15 total 1325.03 last 25.56) - open/close 39975 (time 1709788532.15 total 1335.03 last 26.59) - open/close 40246 (time 1709788542.17 total 1345.06 last 27.03) - open/close 40511 (time 1709788552.19 total 1355.07 last 26.46) - open/close 40776 (time 1709788562.21 total 1365.10 last 26.44) - open/close 41042 (time 1709788572.23 total 1375.11 last 26.56) - open/close 41310 (time 1709788582.25 total 1385.13 last 26.75) - open/close 41584 (time 1709788592.28 total 1395.17 last 27.31) - open/close 41856 (time 1709788602.29 total 1405.17 last 27.19) - open/close 42119 (time 1709788612.32 total 1415.21 last 26.20) - open/close 42386 (time 1709788622.32 total 1425.21 last 26.70) - open/close 42667 (time 1709788632.35 total 1435.23 last 28.04) - open/close 42942 (time 1709788642.36 total 1445.24 last 27.47) - open/close 43214 (time 1709788652.38 total 1455.26 last 27.15) - open/close 43476 (time 1709788662.39 total 1465.27 last 26.18) - open/close 43735 (time 1709788672.39 total 1475.28 last 25.88) - open/close 43992 (time 1709788682.43 total 1485.31 last 25.61) - open/close 44258 (time 1709788692.44 total 1495.33 last 26.56) - open/close 44521 (time 1709788702.46 total 1505.34 last 26.26) - open/close 44777 (time 1709788712.46 total 1515.34 last 25.60) - open/close 45019 (time 1709788722.50 total 1525.38 last 24.10) - open/close 45268 (time 1709788732.52 total 1535.40 last 24.84) - open/close 45511 (time 1709788742.53 total 1545.42 last 24.27) - open/close 45762 (time 1709788752.55 total 1555.43 last 25.06) - open/close 46013 (time 1709788762.56 total 1565.44 last 25.07) - open/close 46270 (time 1709788772.58 total 1575.47 last 25.64) - open/close 46511 (time 1709788782.59 total 1585.48 last 24.08) - open/close 46751 (time 1709788792.62 total 1595.50 last 23.94) - open/close 46983 (time 1709788802.63 total 1605.51 last 23.19) - open/close 47215 (time 1709788812.63 total 1615.51 last 23.20) - open/close 47447 (time 1709788822.67 total 1625.55 last 23.10) - open/close 47683 (time 1709788832.68 total 1635.56 last 23.58) - open/close 47917 (time 1709788842.69 total 1645.57 last 23.37) - open/close 48143 (time 1709788852.71 total 1655.59 last 22.56) - open/close 48376 (time 1709788862.74 total 1665.62 last 23.22) - open/close 48611 (time 1709788872.76 total 1675.65 last 23.45) - open/close 48840 (time 1709788882.79 total 1685.67 last 22.84) - open/close 49059 (time 1709788892.83 total 1695.71 last 21.82) - open/close 49283 (time 1709788902.85 total 1705.73 last 22.35) - open/close 49511 (time 1709788912.86 total 1715.74 last 22.77) - open/close 49731 (time 1709788922.86 total 1725.75 last 21.99) - open/close 49954 (time 1709788932.88 total 1735.77 last 22.25) total: 50000 open/close in 1737.74 seconds: 28.77 ops/second - unlinked 0 (time 1709788936 ; total 0 ; last 0) - unlinked 10000 (time 1709789439 ; total 503 ; last 503) - unlinked 20000 (time 1709789853 ; total 917 ; last 414) - unlinked 30000 (time 1709790130 ; total 1194 ; last 277) - unlinked 40000 (time 1709790240 ; total 1304 ; last 110) total: 50000 unlinks in 1344 seconds: 37.202381 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server unloading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2