== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:57:36 (1709787456) start mds service on oleg336-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg336-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock oleg336-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 185 (time 1709787512.23 total 10.09 last 18.33) - open/close 387 (time 1709787522.60 total 20.47 last 19.47) - open/close 631 (time 1709787532.62 total 30.48 last 24.37) - open/close 756 (time 1709787542.70 total 40.57 last 12.39) - open/close 914 (time 1709787552.71 total 50.58 last 15.78) - open/close 1163 (time 1709787562.73 total 60.59 last 24.86) - open/close 1362 (time 1709787573.30 total 71.16 last 18.83) - open/close 1563 (time 1709787583.31 total 81.18 last 20.07) - open/close 1806 (time 1709787593.92 total 91.79 last 22.91) - open/close 2039 (time 1709787604.26 total 102.12 last 22.54) - open/close 2316 (time 1709787614.27 total 112.13 last 27.68) - open/close 2468 (time 1709787624.59 total 122.46 last 14.72) - open/close 2667 (time 1709787634.90 total 132.77 last 19.30) - open/close 2845 (time 1709787645.70 total 143.56 last 16.49) - open/close 2994 (time 1709787655.99 total 153.85 last 14.48) - open/close 3211 (time 1709787666.01 total 163.87 last 21.65) - open/close 3401 (time 1709787676.87 total 174.73 last 17.50) - open/close 3529 (time 1709787686.88 total 184.75 last 12.78) - open/close 3707 (time 1709787696.90 total 194.76 last 17.77) - open/close 3851 (time 1709787706.93 total 204.79 last 14.36) - open/close 4075 (time 1709787716.93 total 214.80 last 22.40) - open/close 4342 (time 1709787726.99 total 224.85 last 26.56) - open/close 4520 (time 1709787737.64 total 235.51 last 16.70) - open/close 4706 (time 1709787747.75 total 245.61 last 18.40) - open/close 4869 (time 1709787757.75 total 255.61 last 16.30) - open/close 5058 (time 1709787768.63 total 266.49 last 17.37) - open/close 5180 (time 1709787778.75 total 276.62 last 12.05) - open/close 5310 (time 1709787789.21 total 287.07 last 12.43) - open/close 5464 (time 1709787799.22 total 297.08 last 15.39) - open/close 5567 (time 1709787809.23 total 307.09 last 10.29) - open/close 5696 (time 1709787820.15 total 318.01 last 11.81) - open/close 5911 (time 1709787830.15 total 328.02 last 21.49) - open/close 6062 (time 1709787841.14 total 339.00 last 13.74) - open/close 6210 (time 1709787852.08 total 349.94 last 13.53) - open/close 6308 (time 1709787862.09 total 359.95 last 9.79) - open/close 6574 (time 1709787872.62 total 370.48 last 25.26) - open/close 6724 (time 1709787882.62 total 380.49 last 14.99) - open/close 6914 (time 1709787892.63 total 390.49 last 18.99) - open/close 6990 (time 1709787903.22 total 401.08 last 7.18) - open/close 7133 (time 1709787913.22 total 411.09 last 14.29) - open/close 7324 (time 1709787923.96 total 421.83 last 17.78) - open/close 7504 (time 1709787934.15 total 432.01 last 17.67) - open/close 7724 (time 1709787944.17 total 442.03 last 21.96) - open/close 7852 (time 1709787954.32 total 452.18 last 12.61) - open/close 8007 (time 1709787965.28 total 463.15 last 14.13) - open/close 8252 (time 1709787975.29 total 473.15 last 24.50) - open/close 8400 (time 1709787985.33 total 483.19 last 14.74) - open/close 8674 (time 1709787996.26 total 494.12 last 25.07) - open/close 8850 (time 1709788007.62 total 505.48 last 15.50) - open/close 8977 (time 1709788018.38 total 516.24 last 11.80) - open/close 9130 (time 1709788028.81 total 526.67 last 14.67) - open/close 9263 (time 1709788039.64 total 537.50 last 12.28) - open/close 9442 (time 1709788050.80 total 548.66 last 16.04) - open/close 9653 (time 1709788061.82 total 559.69 last 19.14) - open/close 9839 (time 1709788071.83 total 569.70 last 18.59) - open/close 10000 (time 1709788077.58 total 575.45 last 27.99) - open/close 10379 (time 1709788087.59 total 585.46 last 37.86) - open/close 10994 (time 1709788097.60 total 595.47 last 61.45) - open/close 11594 (time 1709788107.62 total 605.49 last 59.89) - open/close 12179 (time 1709788117.63 total 615.49 last 58.45) - open/close 12779 (time 1709788127.64 total 625.50 last 59.96) - open/close 13365 (time 1709788137.64 total 635.51 last 58.56) - open/close 13942 (time 1709788147.65 total 645.51 last 57.66) - open/close 14450 (time 1709788157.66 total 655.52 last 50.75) - open/close 14974 (time 1709788167.67 total 665.53 last 52.35) - open/close 15474 (time 1709788177.67 total 675.54 last 49.99) - open/close 15978 (time 1709788187.68 total 685.54 last 50.37) - open/close 16433 (time 1709788197.69 total 695.55 last 45.46) - open/close 16895 (time 1709788207.70 total 705.57 last 46.13) - open/close 17349 (time 1709788217.72 total 715.59 last 45.31) - open/close 17835 (time 1709788227.73 total 725.59 last 48.58) - open/close 18289 (time 1709788237.74 total 735.60 last 45.36) - open/close 18728 (time 1709788247.76 total 745.62 last 43.81) - open/close 19128 (time 1709788257.77 total 755.63 last 39.97) - open/close 19555 (time 1709788267.77 total 765.63 last 42.68) - open/close 19959 (time 1709788277.79 total 775.65 last 40.33) - open/close 20354 (time 1709788287.80 total 785.67 last 39.43) - open/close 20736 (time 1709788297.82 total 795.69 last 38.12) - open/close 21112 (time 1709788307.85 total 805.71 last 37.52) - open/close 21491 (time 1709788317.85 total 815.71 last 37.89) - open/close 21861 (time 1709788327.87 total 825.73 last 36.94) - open/close 22264 (time 1709788337.88 total 835.74 last 40.26) - open/close 22672 (time 1709788347.89 total 845.76 last 40.73) - open/close 23084 (time 1709788357.91 total 855.78 last 41.11) - open/close 23449 (time 1709788367.93 total 865.79 last 36.45) - open/close 23825 (time 1709788377.94 total 875.80 last 37.56) - open/close 24183 (time 1709788387.97 total 885.83 last 35.69) - open/close 24556 (time 1709788397.99 total 895.86 last 37.20) - open/close 24921 (time 1709788408.01 total 905.88 last 36.43) - open/close 25281 (time 1709788418.03 total 915.89 last 35.96) - open/close 25648 (time 1709788428.03 total 925.90 last 36.68) - open/close 25999 (time 1709788438.05 total 935.91 last 35.05) - open/close 26335 (time 1709788448.07 total 945.94 last 33.51) - open/close 26674 (time 1709788458.08 total 955.94 last 33.88) - open/close 26995 (time 1709788468.11 total 965.97 last 32.00) - open/close 27343 (time 1709788478.12 total 975.98 last 34.77) - open/close 27639 (time 1709788488.14 total 986.01 last 29.52) - open/close 27935 (time 1709788498.15 total 996.01 last 29.59) - open/close 28256 (time 1709788508.18 total 1006.04 last 32.01) - open/close 28596 (time 1709788518.20 total 1016.06 last 33.92) - open/close 28931 (time 1709788528.23 total 1026.09 last 33.39) - open/close 29259 (time 1709788538.26 total 1036.12 last 32.71) - open/close 29595 (time 1709788548.28 total 1046.14 last 33.54) - open/close 29904 (time 1709788558.29 total 1056.16 last 30.85) - open/close 30000 (time 1709788561.50 total 1059.36 last 29.92) - open/close 30316 (time 1709788571.51 total 1069.38 last 31.56) - open/close 30631 (time 1709788581.52 total 1079.38 last 31.48) - open/close 30962 (time 1709788591.53 total 1089.39 last 33.07) - open/close 31293 (time 1709788601.54 total 1099.40 last 33.07) - open/close 31625 (time 1709788611.56 total 1109.42 last 33.13) - open/close 31961 (time 1709788621.58 total 1119.45 last 33.52) - open/close 32292 (time 1709788631.61 total 1129.48 last 33.00) - open/close 32601 (time 1709788641.63 total 1139.49 last 30.85) - open/close 32890 (time 1709788651.64 total 1149.50 last 28.89) - open/close 33177 (time 1709788661.66 total 1159.52 last 28.64) - open/close 33458 (time 1709788671.66 total 1169.52 last 28.09) - open/close 33754 (time 1709788681.67 total 1179.54 last 29.57) - open/close 34038 (time 1709788691.70 total 1189.56 last 28.33) - open/close 34330 (time 1709788701.72 total 1199.59 last 29.12) - open/close 34621 (time 1709788711.74 total 1209.60 last 29.05) - open/close 34908 (time 1709788721.74 total 1219.61 last 28.69) - open/close 35224 (time 1709788731.78 total 1229.65 last 31.48) - open/close 35540 (time 1709788741.79 total 1239.66 last 31.57) - open/close 35849 (time 1709788751.81 total 1249.67 last 30.85) - open/close 36119 (time 1709788761.84 total 1259.71 last 26.90) - open/close 36361 (time 1709788771.89 total 1269.75 last 24.09) - open/close 36608 (time 1709788781.91 total 1279.77 last 24.65) - open/close 36866 (time 1709788791.93 total 1289.79 last 25.76) - open/close 37101 (time 1709788801.93 total 1299.79 last 23.50) - open/close 37349 (time 1709788811.95 total 1309.82 last 24.73) - open/close 37586 (time 1709788821.96 total 1319.82 last 23.69) - open/close 37834 (time 1709788831.97 total 1329.83 last 24.78) - open/close 38105 (time 1709788842.00 total 1339.86 last 27.02) - open/close 38350 (time 1709788852.03 total 1349.90 last 24.41) - open/close 38584 (time 1709788862.08 total 1359.94 last 23.29) - open/close 38821 (time 1709788872.11 total 1369.97 last 23.63) - open/close 39058 (time 1709788882.16 total 1380.02 last 23.58) - open/close 39336 (time 1709788892.19 total 1390.05 last 27.73) - open/close 39607 (time 1709788902.21 total 1400.07 last 27.05) - open/close 39871 (time 1709788912.21 total 1410.07 last 26.39) - open/close 40000 (time 1709788916.89 total 1414.76 last 27.55) - open/close 40272 (time 1709788926.91 total 1424.77 last 27.16) - open/close 40523 (time 1709788936.92 total 1434.78 last 25.07) - open/close 40764 (time 1709788946.94 total 1444.80 last 24.05) - open/close 41012 (time 1709788956.98 total 1454.84 last 24.71) - open/close 41261 (time 1709788967.00 total 1464.86 last 24.84) - open/close 41532 (time 1709788977.01 total 1474.88 last 27.06) - open/close 41793 (time 1709788987.05 total 1484.91 last 26.01) - open/close 42038 (time 1709788997.08 total 1494.94 last 24.42) - open/close 42283 (time 1709789007.09 total 1504.95 last 24.47) - open/close 42533 (time 1709789017.10 total 1514.97 last 24.97) - open/close 42769 (time 1709789027.12 total 1524.98 last 23.56) - open/close 43004 (time 1709789037.16 total 1535.03 last 23.40) - open/close 43240 (time 1709789047.19 total 1545.05 last 23.54) - open/close 43493 (time 1709789057.20 total 1555.07 last 25.26) - open/close 43735 (time 1709789067.21 total 1565.07 last 24.19) - open/close 43962 (time 1709789077.23 total 1575.10 last 22.64) - open/close 44184 (time 1709789087.27 total 1585.13 last 22.12) - open/close 44395 (time 1709789097.30 total 1595.17 last 21.03) - open/close 44640 (time 1709789107.33 total 1605.20 last 24.43) - open/close 44872 (time 1709789117.35 total 1615.21 last 23.17) - open/close 45099 (time 1709789127.36 total 1625.22 last 22.68) - open/close 45323 (time 1709789137.37 total 1635.23 last 22.38) - open/close 45546 (time 1709789147.39 total 1645.26 last 22.24) - open/close 45775 (time 1709789157.43 total 1655.29 last 22.82) - open/close 46016 (time 1709789167.47 total 1665.33 last 24.01) - open/close 46260 (time 1709789177.50 total 1675.36 last 24.32) - open/close 46491 (time 1709789187.54 total 1685.40 last 23.01) - open/close 46708 (time 1709789197.58 total 1695.44 last 21.61) - open/close 46919 (time 1709789207.63 total 1705.49 last 21.00) - open/close 47135 (time 1709789217.63 total 1715.50 last 21.59) - open/close 47362 (time 1709789227.69 total 1725.55 last 22.57) - open/close 47578 (time 1709789237.70 total 1735.57 last 21.57) - open/close 47789 (time 1709789247.74 total 1745.60 last 21.02) - open/close 47996 (time 1709789257.77 total 1755.63 last 20.64) - open/close 48210 (time 1709789267.78 total 1765.64 last 21.38) - open/close 48415 (time 1709789277.78 total 1775.65 last 20.49) - open/close 48633 (time 1709789287.82 total 1785.69 last 21.72) - open/close 48843 (time 1709789297.83 total 1795.70 last 20.97) - open/close 49055 (time 1709789307.85 total 1805.72 last 21.16) - open/close 49269 (time 1709789317.88 total 1815.74 last 21.34) - open/close 49484 (time 1709789327.90 total 1825.77 last 21.45) - open/close 49683 (time 1709789337.94 total 1835.81 last 19.82) - open/close 49881 (time 1709789347.96 total 1845.83 last 19.76) total: 50000 open/close in 1851.90 seconds: 27.00 ops/second - unlinked 0 (time 1709789355 ; total 0 ; last 0) - unlinked 10000 (time 1709789901 ; total 546 ; last 546) - unlinked 20000 (time 1709790298 ; total 943 ; last 397) - unlinked 30000 (time 1709790478 ; total 1123 ; last 180) - unlinked 40000 (time 1709790554 ; total 1199 ; last 76) total: 50000 unlinks in 1249 seconds: 40.032024 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2