== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:38 (1709787038) start mds service on oleg314-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg314-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg314-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock oleg314-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 172 (time 1709787085.35 total 10.04 last 17.13) - open/close 339 (time 1709787095.78 total 20.48 last 16.01) - open/close 602 (time 1709787105.83 total 30.53 last 26.16) - open/close 704 (time 1709787116.08 total 40.77 last 9.96) - open/close 976 (time 1709787126.71 total 51.41 last 25.57) - open/close 1138 (time 1709787136.72 total 61.41 last 16.19) - open/close 1400 (time 1709787147.39 total 72.09 last 24.55) - open/close 1556 (time 1709787157.40 total 82.10 last 15.59) - open/close 1716 (time 1709787167.56 total 92.25 last 15.75) - open/close 1882 (time 1709787178.36 total 103.06 last 15.37) - open/close 2106 (time 1709787188.98 total 113.67 last 21.10) - open/close 2201 (time 1709787199.61 total 124.30 last 8.94) - open/close 2276 (time 1709787212.81 total 137.51 last 5.68) - open/close 2465 (time 1709787222.99 total 147.68 last 18.57) - open/close 2657 (time 1709787233.07 total 157.77 last 19.04) - open/close 2789 (time 1709787243.08 total 167.77 last 13.19) - open/close 2927 (time 1709787253.68 total 178.37 last 13.02) - open/close 3140 (time 1709787264.42 total 189.11 last 19.83) - open/close 3293 (time 1709787275.29 total 199.98 last 14.08) - open/close 3567 (time 1709787285.30 total 209.99 last 27.38) - open/close 3716 (time 1709787295.97 total 220.67 last 13.95) - open/close 3878 (time 1709787306.01 total 230.70 last 16.14) - open/close 4168 (time 1709787316.03 total 240.73 last 28.93) - open/close 4322 (time 1709787326.15 total 250.84 last 15.22) - open/close 4479 (time 1709787336.86 total 261.55 last 14.66) - open/close 4796 (time 1709787346.89 total 271.58 last 31.61) - open/close 5014 (time 1709787357.68 total 282.38 last 20.19) - open/close 5159 (time 1709787368.51 total 293.20 last 13.40) - open/close 5265 (time 1709787380.68 total 305.37 last 8.71) - open/close 5506 (time 1709787390.71 total 315.41 last 24.02) - open/close 5705 (time 1709787401.00 total 325.70 last 19.34) - open/close 5871 (time 1709787411.27 total 335.96 last 16.17) - open/close 6010 (time 1709787421.28 total 345.97 last 13.88) - open/close 6192 (time 1709787431.67 total 356.36 last 17.52) - open/close 6372 (time 1709787441.99 total 366.68 last 17.45) - open/close 6557 (time 1709787452.53 total 377.22 last 17.55) - open/close 6801 (time 1709787463.32 total 388.02 last 22.59) - open/close 7003 (time 1709787473.35 total 398.05 last 20.14) - open/close 7197 (time 1709787483.35 total 408.05 last 19.40) - open/close 7332 (time 1709787494.11 total 418.80 last 12.55) - open/close 7577 (time 1709787504.11 total 428.80 last 24.50) - open/close 7798 (time 1709787514.13 total 438.82 last 22.06) - open/close 7967 (time 1709787524.22 total 448.92 last 16.74) - open/close 8111 (time 1709787534.57 total 459.26 last 13.92) - open/close 8137 (time 1709787546.17 total 470.86 last 2.24) - open/close 8302 (time 1709787556.17 total 480.86 last 16.50) - open/close 8500 (time 1709787566.20 total 490.90 last 19.74) - open/close 8699 (time 1709787577.22 total 501.91 last 18.06) - open/close 8870 (time 1709787587.22 total 511.92 last 17.10) - open/close 9072 (time 1709787597.25 total 521.95 last 20.13) - open/close 9163 (time 1709787607.80 total 532.50 last 8.63) - open/close 9388 (time 1709787617.82 total 542.51 last 22.46) - open/close 9520 (time 1709787628.19 total 552.89 last 12.72) - open/close 9622 (time 1709787638.62 total 563.31 last 9.79) - open/close 9784 (time 1709787648.71 total 573.40 last 16.06) - open/close 10000 (time 1709787657.93 total 582.62 last 23.42) - open/close 10182 (time 1709787667.95 total 592.65 last 18.16) - open/close 10424 (time 1709787677.97 total 602.67 last 24.15) - open/close 10907 (time 1709787688.03 total 612.72 last 48.03) - open/close 11381 (time 1709787698.03 total 622.72 last 47.40) - open/close 11876 (time 1709787708.03 total 632.73 last 49.48) - open/close 12294 (time 1709787728.50 total 653.20 last 20.42) - open/close 12777 (time 1709787738.51 total 663.20 last 48.26) - open/close 13243 (time 1709787748.51 total 673.21 last 46.59) - open/close 13685 (time 1709787758.52 total 683.21 last 44.16) - open/close 14134 (time 1709787768.53 total 693.23 last 44.84) - open/close 14598 (time 1709787778.55 total 703.24 last 46.34) - open/close 15061 (time 1709787788.56 total 713.26 last 46.23) - open/close 15529 (time 1709787798.59 total 723.28 last 46.70) - open/close 15997 (time 1709787808.59 total 733.28 last 46.78) - open/close 16431 (time 1709787818.61 total 743.31 last 43.31) - open/close 16835 (time 1709787828.64 total 753.33 last 40.29) - open/close 17269 (time 1709787838.64 total 763.34 last 43.38) - open/close 17657 (time 1709787848.74 total 773.43 last 38.44) - open/close 18065 (time 1709787858.74 total 783.44 last 40.77) - open/close 18417 (time 1709787868.75 total 793.44 last 35.19) - open/close 18709 (time 1709787887.23 total 811.92 last 15.80) - open/close 19084 (time 1709787897.26 total 821.96 last 37.36) - open/close 19497 (time 1709787907.28 total 831.97 last 41.25) - open/close 19863 (time 1709787917.31 total 842.01 last 36.47) - open/close 20000 (time 1709787920.95 total 845.64 last 37.69) - open/close 20349 (time 1709787930.98 total 855.67 last 34.80) - open/close 20704 (time 1709787940.99 total 865.68 last 35.45) - open/close 21062 (time 1709787950.99 total 875.69 last 35.78) - open/close 21414 (time 1709787961.01 total 885.71 last 35.13) - open/close 21766 (time 1709787971.03 total 895.72 last 35.15) - open/close 22118 (time 1709787981.04 total 905.74 last 35.14) - open/close 22493 (time 1709787991.05 total 915.75 last 37.47) - open/close 22873 (time 1709788001.06 total 925.75 last 37.99) - open/close 23201 (time 1709788011.08 total 935.77 last 32.72) - open/close 23552 (time 1709788021.10 total 945.79 last 35.05) - open/close 23871 (time 1709788031.11 total 955.80 last 31.87) - open/close 24023 (time 1709788047.07 total 971.77 last 9.52) - open/close 24327 (time 1709788057.08 total 981.77 last 30.38) - open/close 24640 (time 1709788067.10 total 991.79 last 31.23) - open/close 24979 (time 1709788077.11 total 1001.80 last 33.88) - open/close 25379 (time 1709788087.12 total 1011.81 last 39.96) - open/close 25773 (time 1709788097.13 total 1021.83 last 39.33) - open/close 26164 (time 1709788107.15 total 1031.84 last 39.05) - open/close 26555 (time 1709788117.16 total 1041.86 last 39.04) - open/close 26931 (time 1709788127.18 total 1051.88 last 37.53) - open/close 27322 (time 1709788137.19 total 1061.88 last 39.09) - open/close 27724 (time 1709788147.23 total 1071.92 last 40.03) - open/close 28089 (time 1709788157.24 total 1081.93 last 36.47) - open/close 28443 (time 1709788167.25 total 1091.95 last 35.35) - open/close 28809 (time 1709788177.26 total 1101.96 last 36.57) - open/close 29184 (time 1709788187.28 total 1111.97 last 37.43) - open/close 29539 (time 1709788197.28 total 1121.97 last 35.49) - open/close 29883 (time 1709788207.31 total 1132.00 last 34.30) - open/close 30000 (time 1709788210.69 total 1135.38 last 34.66) - open/close 30338 (time 1709788220.69 total 1145.38 last 33.80) - open/close 30724 (time 1709788230.69 total 1155.38 last 38.60) - open/close 31078 (time 1709788240.70 total 1165.39 last 35.37) - open/close 31388 (time 1709788250.71 total 1175.41 last 30.94) - open/close 31700 (time 1709788260.75 total 1185.45 last 31.09) - open/close 32019 (time 1709788270.80 total 1195.49 last 31.75) - open/close 32342 (time 1709788280.85 total 1205.55 last 32.12) - open/close 32635 (time 1709788290.87 total 1215.57 last 29.24) - open/close 32925 (time 1709788300.90 total 1225.59 last 28.94) - open/close 33226 (time 1709788310.91 total 1235.61 last 30.05) - open/close 33532 (time 1709788320.92 total 1245.61 last 30.59) - open/close 33854 (time 1709788330.94 total 1255.64 last 32.12) - open/close 34157 (time 1709788340.95 total 1265.65 last 30.28) - open/close 34468 (time 1709788350.97 total 1275.67 last 31.04) - open/close 34796 (time 1709788361.00 total 1285.69 last 32.71) - open/close 35082 (time 1709788371.04 total 1295.73 last 28.49) - open/close 35384 (time 1709788381.07 total 1305.76 last 30.11) - open/close 35693 (time 1709788391.10 total 1315.79 last 30.80) - open/close 36002 (time 1709788401.13 total 1325.82 last 30.82) - open/close 36293 (time 1709788411.14 total 1335.83 last 29.07) - open/close 36615 (time 1709788421.15 total 1345.84 last 32.16) - open/close 36903 (time 1709788431.18 total 1355.87 last 28.72) - open/close 37198 (time 1709788441.21 total 1365.91 last 29.40) - open/close 37482 (time 1709788451.22 total 1375.92 last 28.37) - open/close 37786 (time 1709788461.24 total 1385.93 last 30.35) - open/close 38054 (time 1709788471.26 total 1395.95 last 26.75) - open/close 38353 (time 1709788481.27 total 1405.96 last 29.88) - open/close 38620 (time 1709788491.28 total 1415.97 last 26.67) - open/close 38891 (time 1709788501.28 total 1425.97 last 27.10) - open/close 39166 (time 1709788511.30 total 1436.00 last 27.43) - open/close 39448 (time 1709788521.32 total 1446.02 last 28.15) - open/close 39726 (time 1709788531.33 total 1456.02 last 27.79) - open/close 40000 (time 1709788540.91 total 1465.60 last 28.60) - open/close 40283 (time 1709788550.93 total 1475.63 last 28.23) - open/close 40564 (time 1709788560.95 total 1485.65 last 28.04) - open/close 40849 (time 1709788570.97 total 1495.66 last 28.45) - open/close 41137 (time 1709788580.98 total 1505.67 last 28.78) - open/close 41418 (time 1709788591.01 total 1515.70 last 28.02) - open/close 41710 (time 1709788601.03 total 1525.72 last 29.14) - open/close 41992 (time 1709788611.04 total 1535.73 last 28.17) - open/close 42283 (time 1709788621.06 total 1545.75 last 29.04) - open/close 42576 (time 1709788631.09 total 1555.78 last 29.21) - open/close 42857 (time 1709788641.10 total 1565.79 last 28.08) - open/close 43144 (time 1709788651.13 total 1575.82 last 28.61) - open/close 43425 (time 1709788661.14 total 1585.84 last 28.05) - open/close 43702 (time 1709788671.16 total 1595.86 last 27.65) - open/close 43967 (time 1709788681.18 total 1605.87 last 26.46) - open/close 44217 (time 1709788691.20 total 1615.89 last 24.95) - open/close 44480 (time 1709788701.23 total 1625.92 last 26.21) - open/close 44745 (time 1709788711.24 total 1635.93 last 26.48) - open/close 45000 (time 1709788721.26 total 1645.95 last 25.45) - open/close 45275 (time 1709788731.27 total 1655.96 last 27.47) - open/close 45547 (time 1709788741.28 total 1665.97 last 27.17) - open/close 45819 (time 1709788751.32 total 1676.01 last 27.10) - open/close 46070 (time 1709788761.32 total 1686.01 last 25.09) - open/close 46305 (time 1709788771.35 total 1696.04 last 23.43) - open/close 46544 (time 1709788781.35 total 1706.04 last 23.90) - open/close 46787 (time 1709788791.37 total 1716.06 last 24.26) - open/close 47021 (time 1709788801.38 total 1726.07 last 23.37) - open/close 47236 (time 1709788811.38 total 1736.08 last 21.49) - open/close 47459 (time 1709788821.42 total 1746.11 last 22.23) - open/close 47679 (time 1709788831.44 total 1756.14 last 21.95) - open/close 47916 (time 1709788841.45 total 1766.14 last 23.69) - open/close 48152 (time 1709788851.47 total 1776.17 last 23.53) - open/close 48383 (time 1709788861.51 total 1786.21 last 23.02) - open/close 48604 (time 1709788871.52 total 1796.21 last 22.08) - open/close 48825 (time 1709788881.53 total 1806.23 last 22.07) - open/close 49069 (time 1709788891.55 total 1816.24 last 24.36) - open/close 49317 (time 1709788901.57 total 1826.27 last 24.74) - open/close 49555 (time 1709788911.58 total 1836.28 last 23.77) - open/close 49799 (time 1709788921.59 total 1846.29 last 24.38) total: 50000 open/close in 1854.50 seconds: 26.96 ops/second - unlinked 0 (time 1709788931 ; total 0 ; last 0) - unlinked 10000 (time 1709789439 ; total 508 ; last 508) - unlinked 20000 (time 1709789889 ; total 958 ; last 450) - unlinked 30000 (time 1709790216 ; total 1285 ; last 327) - unlinked 40000 (time 1709790362 ; total 1431 ; last 146) total: 50000 unlinks in 1471 seconds: 33.990482 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server unloading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-client: ssh exited with exit code 2 pdsh@oleg314-client: oleg314-server: ssh exited with exit code 2