== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:28 (1709787028) start mds service on oleg321-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg321-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg321-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock oleg321-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 237 (time 1709787072.62 total 10.04 last 23.62) - open/close 510 (time 1709787083.10 total 20.52 last 26.03) - open/close 720 (time 1709787093.89 total 31.31 last 19.46) - open/close 915 (time 1709787104.02 total 41.44 last 19.25) - open/close 1090 (time 1709787114.72 total 52.14 last 16.36) - open/close 1232 (time 1709787124.76 total 62.18 last 14.14) - open/close 1401 (time 1709787135.33 total 72.75 last 15.99) - open/close 1650 (time 1709787147.93 total 85.35 last 19.76) - open/close 1926 (time 1709787158.12 total 95.54 last 27.09) - open/close 2053 (time 1709787168.23 total 105.64 last 12.57) - open/close 2200 (time 1709787178.74 total 116.16 last 13.98) - open/close 2431 (time 1709787191.76 total 129.18 last 17.75) - open/close 2667 (time 1709787202.22 total 139.63 last 22.58) - open/close 2848 (time 1709787212.22 total 149.64 last 18.09) - open/close 3033 (time 1709787222.82 total 160.24 last 17.45) - open/close 3263 (time 1709787232.95 total 170.37 last 22.71) - open/close 3424 (time 1709787243.65 total 181.07 last 15.05) - open/close 3586 (time 1709787253.66 total 191.08 last 16.18) - open/close 3800 (time 1709787264.52 total 201.94 last 19.71) - open/close 3948 (time 1709787274.53 total 211.95 last 14.78) - open/close 4182 (time 1709787284.56 total 221.97 last 23.35) - open/close 4372 (time 1709787294.87 total 232.29 last 18.42) - open/close 4615 (time 1709787305.39 total 242.81 last 23.09) - open/close 4696 (time 1709787316.10 total 253.52 last 7.57) - open/close 4944 (time 1709787326.12 total 263.54 last 24.74) - open/close 5145 (time 1709787336.14 total 273.56 last 20.06) - open/close 5312 (time 1709787346.81 total 284.23 last 15.65) - open/close 5465 (time 1709787356.82 total 294.23 last 15.30) - open/close 5597 (time 1709787367.58 total 305.00 last 12.26) - open/close 5857 (time 1709787377.58 total 315.00 last 25.99) - open/close 6018 (time 1709787387.58 total 325.00 last 16.10) - open/close 6117 (time 1709787403.71 total 341.13 last 6.14) - open/close 6298 (time 1709787414.37 total 351.79 last 16.98) - open/close 6437 (time 1709787424.38 total 361.80 last 13.88) - open/close 6650 (time 1709787435.34 total 372.76 last 19.43) - open/close 6863 (time 1709787445.86 total 383.27 last 20.26) - open/close 6891 (time 1709787458.55 total 395.97 last 2.20) - open/close 7059 (time 1709787468.98 total 406.40 last 16.12) - open/close 7224 (time 1709787479.61 total 417.03 last 15.52) - open/close 7344 (time 1709787489.62 total 427.04 last 11.99) - open/close 7501 (time 1709787499.63 total 437.05 last 15.68) - open/close 7666 (time 1709787510.28 total 447.70 last 15.50) - open/close 7757 (time 1709787520.53 total 457.94 last 8.88) - open/close 7924 (time 1709787530.54 total 467.96 last 16.68) - open/close 8092 (time 1709787540.54 total 477.96 last 16.80) - open/close 8245 (time 1709787550.57 total 487.98 last 15.27) - open/close 8476 (time 1709787560.58 total 498.00 last 23.07) - open/close 8697 (time 1709787570.59 total 508.01 last 22.06) - open/close 8817 (time 1709787580.61 total 518.03 last 11.98) - open/close 8968 (time 1709787591.31 total 528.73 last 14.12) - open/close 9161 (time 1709787601.32 total 538.74 last 19.28) - open/close 9325 (time 1709787611.34 total 548.76 last 16.36) - open/close 9460 (time 1709787621.35 total 558.77 last 13.49) - open/close 9651 (time 1709787631.37 total 568.78 last 19.07) - open/close 9796 (time 1709787641.83 total 579.25 last 13.85) - open/close 10000 (time 1709787650.79 total 588.21 last 22.78) - open/close 10103 (time 1709787660.82 total 598.24 last 10.26) - open/close 10289 (time 1709787670.83 total 608.24 last 18.60) - open/close 10521 (time 1709787680.84 total 618.25 last 23.18) - open/close 10892 (time 1709787690.85 total 628.27 last 37.04) - open/close 11261 (time 1709787700.85 total 638.27 last 36.88) - open/close 11643 (time 1709787710.87 total 648.29 last 38.15) - open/close 12003 (time 1709787720.89 total 658.30 last 35.94) - open/close 12449 (time 1709787730.89 total 668.31 last 44.59) - open/close 12568 (time 1709787745.26 total 682.68 last 8.28) - open/close 12894 (time 1709787755.27 total 692.69 last 32.58) - open/close 13011 (time 1709787769.39 total 706.80 last 8.29) - open/close 13381 (time 1709787779.40 total 716.82 last 36.96) - open/close 13779 (time 1709787789.40 total 726.82 last 39.79) - open/close 14151 (time 1709787799.43 total 736.85 last 37.08) - open/close 14534 (time 1709787809.45 total 746.86 last 38.24) - open/close 14892 (time 1709787819.45 total 756.86 last 35.80) - open/close 15103 (time 1709787836.67 total 774.09 last 12.25) - open/close 15477 (time 1709787846.69 total 784.11 last 37.32) - open/close 15849 (time 1709787856.69 total 794.11 last 37.19) - open/close 16211 (time 1709787866.71 total 804.13 last 36.13) - open/close 16543 (time 1709787876.73 total 814.14 last 33.16) - open/close 16895 (time 1709787886.75 total 824.17 last 35.12) - open/close 17210 (time 1709787896.77 total 834.19 last 31.42) - open/close 17583 (time 1709787906.80 total 844.22 last 37.20) - open/close 17932 (time 1709787916.80 total 854.22 last 34.89) - open/close 18253 (time 1709787926.81 total 864.23 last 32.06) - open/close 18553 (time 1709787936.83 total 874.25 last 29.96) - open/close 18880 (time 1709787946.86 total 884.28 last 32.59) - open/close 19199 (time 1709787956.88 total 894.30 last 31.84) - open/close 19534 (time 1709787966.90 total 904.32 last 33.44) - open/close 19856 (time 1709787976.92 total 914.34 last 32.13) - open/close 20000 (time 1709787981.20 total 918.62 last 33.66) - open/close 20354 (time 1709787991.21 total 928.63 last 35.36) - open/close 20711 (time 1709788001.24 total 938.66 last 35.60) - open/close 21053 (time 1709788011.27 total 948.69 last 34.09) - open/close 21369 (time 1709788021.27 total 958.69 last 31.59) - open/close 21675 (time 1709788031.30 total 968.71 last 30.52) - open/close 21983 (time 1709788041.31 total 978.73 last 30.75) - open/close 22264 (time 1709788051.37 total 988.79 last 27.93) - open/close 22571 (time 1709788061.38 total 998.80 last 30.67) - open/close 22885 (time 1709788071.41 total 1008.83 last 31.32) - open/close 23267 (time 1709788081.43 total 1018.84 last 38.14) - open/close 23661 (time 1709788091.44 total 1028.86 last 39.36) - open/close 24040 (time 1709788101.45 total 1038.86 last 37.86) - open/close 24423 (time 1709788111.45 total 1048.87 last 38.27) - open/close 24807 (time 1709788121.49 total 1058.90 last 38.28) - open/close 25185 (time 1709788131.51 total 1068.93 last 37.72) - open/close 25569 (time 1709788141.53 total 1078.95 last 38.32) - open/close 25929 (time 1709788151.54 total 1088.95 last 35.98) - open/close 26284 (time 1709788161.56 total 1098.98 last 35.42) - open/close 26651 (time 1709788171.57 total 1108.99 last 36.67) - open/close 26990 (time 1709788181.59 total 1119.01 last 33.83) - open/close 27337 (time 1709788191.60 total 1129.02 last 34.66) - open/close 27685 (time 1709788201.62 total 1139.03 last 34.76) - open/close 28022 (time 1709788211.62 total 1149.04 last 33.69) - open/close 28360 (time 1709788221.64 total 1159.06 last 33.74) - open/close 28699 (time 1709788231.67 total 1169.08 last 33.80) - open/close 29023 (time 1709788241.68 total 1179.10 last 32.35) - open/close 29333 (time 1709788251.71 total 1189.13 last 30.90) - open/close 29653 (time 1709788261.73 total 1199.15 last 31.95) - open/close 29963 (time 1709788271.75 total 1209.17 last 30.94) - open/close 30275 (time 1709788281.75 total 1219.17 last 31.20) - open/close 30565 (time 1709788291.75 total 1229.17 last 28.99) - open/close 30864 (time 1709788301.77 total 1239.19 last 29.85) - open/close 31149 (time 1709788311.80 total 1249.22 last 28.42) - open/close 31453 (time 1709788321.80 total 1259.22 last 30.39) - open/close 31771 (time 1709788331.80 total 1269.22 last 31.79) - open/close 32082 (time 1709788341.81 total 1279.23 last 31.07) - open/close 32396 (time 1709788351.82 total 1289.24 last 31.39) - open/close 32709 (time 1709788361.82 total 1299.24 last 31.28) - open/close 33008 (time 1709788371.85 total 1309.27 last 29.82) - open/close 33317 (time 1709788381.88 total 1319.30 last 30.82) - open/close 33614 (time 1709788391.90 total 1329.32 last 29.64) - open/close 33920 (time 1709788401.93 total 1339.35 last 30.49) - open/close 34205 (time 1709788411.95 total 1349.37 last 28.44) - open/close 34523 (time 1709788421.98 total 1359.40 last 31.72) - open/close 34807 (time 1709788431.98 total 1369.40 last 28.40) - open/close 35118 (time 1709788442.01 total 1379.43 last 31.01) - open/close 35404 (time 1709788452.01 total 1389.43 last 28.60) - open/close 35699 (time 1709788462.02 total 1399.44 last 29.48) - open/close 35836 (time 1709788477.87 total 1415.29 last 8.64) - open/close 36109 (time 1709788487.89 total 1425.30 last 27.25) - open/close 36361 (time 1709788497.90 total 1435.32 last 25.16) - open/close 36634 (time 1709788507.91 total 1445.33 last 27.27) - open/close 36912 (time 1709788517.93 total 1455.35 last 27.74) - open/close 37179 (time 1709788527.95 total 1465.37 last 26.64) - open/close 37445 (time 1709788537.98 total 1475.39 last 26.54) - open/close 37718 (time 1709788547.98 total 1485.40 last 27.29) - open/close 37975 (time 1709788557.99 total 1495.41 last 25.66) - open/close 38243 (time 1709788568.02 total 1505.44 last 26.73) - open/close 38538 (time 1709788578.03 total 1515.45 last 29.46) - open/close 38825 (time 1709788588.06 total 1525.48 last 28.63) - open/close 39094 (time 1709788598.06 total 1535.48 last 26.89) - open/close 39375 (time 1709788608.08 total 1545.50 last 28.04) - open/close 39654 (time 1709788618.11 total 1555.53 last 27.83) - open/close 39939 (time 1709788628.12 total 1565.54 last 28.47) - open/close 40000 (time 1709788630.40 total 1567.82 last 26.77) - open/close 40256 (time 1709788640.44 total 1577.86 last 25.50) - open/close 40478 (time 1709788650.48 total 1587.90 last 22.11) - open/close 40690 (time 1709788660.51 total 1597.92 last 21.14) - open/close 40913 (time 1709788670.51 total 1607.93 last 22.28) - open/close 41166 (time 1709788680.52 total 1617.94 last 25.28) - open/close 41414 (time 1709788690.52 total 1627.94 last 24.80) - open/close 41669 (time 1709788700.53 total 1637.95 last 25.47) - open/close 41913 (time 1709788710.57 total 1647.99 last 24.32) - open/close 42154 (time 1709788720.57 total 1657.99 last 24.09) - open/close 42419 (time 1709788730.60 total 1668.02 last 26.41) - open/close 42684 (time 1709788740.64 total 1678.06 last 26.40) - open/close 42949 (time 1709788750.67 total 1688.09 last 26.43) - open/close 43196 (time 1709788760.68 total 1698.10 last 24.66) - open/close 43420 (time 1709788770.71 total 1708.13 last 22.34) - open/close 43638 (time 1709788791.22 total 1728.63 last 10.63) - open/close 43862 (time 1709788801.23 total 1738.65 last 22.36) - open/close 44095 (time 1709788811.25 total 1748.67 last 23.25) - open/close 44325 (time 1709788821.26 total 1758.67 last 23.00) - open/close 44547 (time 1709788831.29 total 1768.71 last 22.12) - open/close 44787 (time 1709788841.31 total 1778.73 last 23.96) - open/close 45012 (time 1709788851.34 total 1788.75 last 22.44) - open/close 45236 (time 1709788861.38 total 1798.80 last 22.30) - open/close 45458 (time 1709788871.42 total 1808.83 last 22.12) - open/close 45681 (time 1709788881.42 total 1818.84 last 22.28) - open/close 45925 (time 1709788891.45 total 1828.87 last 24.34) - open/close 46161 (time 1709788901.46 total 1838.88 last 23.58) - open/close 46387 (time 1709788911.48 total 1848.90 last 22.54) - open/close 46622 (time 1709788921.51 total 1858.93 last 23.43) - open/close 46856 (time 1709788931.55 total 1868.97 last 23.32) - open/close 47076 (time 1709788941.59 total 1879.01 last 21.90) - open/close 47297 (time 1709788951.60 total 1889.01 last 22.09) - open/close 47516 (time 1709788961.60 total 1899.02 last 21.89) - open/close 47747 (time 1709788971.64 total 1909.06 last 23.01) - open/close 47983 (time 1709788981.67 total 1919.09 last 23.53) - open/close 48215 (time 1709788991.70 total 1929.12 last 23.13) - open/close 48435 (time 1709789001.71 total 1939.12 last 21.99) - open/close 48668 (time 1709789011.74 total 1949.16 last 23.22) - open/close 48889 (time 1709789021.75 total 1959.17 last 22.07) - open/close 49112 (time 1709789031.76 total 1969.18 last 22.28) - open/close 49324 (time 1709789041.79 total 1979.21 last 21.13) - open/close 49551 (time 1709789051.80 total 1989.22 last 22.69) - open/close 49781 (time 1709789061.81 total 1999.23 last 22.98) - open/close 49997 (time 1709789071.82 total 2009.24 last 21.57) total: 50000 open/close in 2009.38 seconds: 24.88 ops/second - unlinked 0 (time 1709789073 ; total 0 ; last 0) - unlinked 10000 (time 1709789566 ; total 493 ; last 493) - unlinked 20000 (time 1709789971 ; total 898 ; last 405) - unlinked 30000 (time 1709790253 ; total 1180 ; last 282) - unlinked 40000 (time 1709790367 ; total 1294 ; last 114) total: 50000 unlinks in 1335 seconds: 37.453182 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server unloading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2