== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:44:54 (1709786694) start mds service on oleg448-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg448-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock oleg448-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1709786723.11 total 10.00 last 20.79) - open/close 374 (time 1709786733.61 total 20.50 last 15.81) - open/close 683 (time 1709786743.68 total 30.57 last 30.70) - open/close 908 (time 1709786754.47 total 41.37 last 20.84) - open/close 1357 (time 1709786764.48 total 51.37 last 44.89) - open/close 1620 (time 1709786775.25 total 62.14 last 24.42) - open/close 1948 (time 1709786785.96 total 72.85 last 30.61) - open/close 2247 (time 1709786796.93 total 83.82 last 27.26) - open/close 2627 (time 1709786807.83 total 94.72 last 34.85) - open/close 2864 (time 1709786818.10 total 104.99 last 23.08) - open/close 3228 (time 1709786828.54 total 115.43 last 34.88) - open/close 3639 (time 1709786839.49 total 126.38 last 37.51) - open/close 3947 (time 1709786850.32 total 137.21 last 28.45) - open/close 4263 (time 1709786860.93 total 147.82 last 29.77) - open/close 4508 (time 1709786870.94 total 157.83 last 24.47) - open/close 4798 (time 1709786881.15 total 168.04 last 28.40) - open/close 5060 (time 1709786891.61 total 178.51 last 25.05) - open/close 5277 (time 1709786902.63 total 189.52 last 19.70) - open/close 5475 (time 1709786912.65 total 199.54 last 19.76) - open/close 5677 (time 1709786922.66 total 209.55 last 20.18) - open/close 5770 (time 1709786933.41 total 220.30 last 8.65) - open/close 6167 (time 1709786944.16 total 231.05 last 36.93) - open/close 6331 (time 1709786954.59 total 241.48 last 15.73) - open/close 6439 (time 1709786965.42 total 252.31 last 9.97) - open/close 6724 (time 1709786976.36 total 263.25 last 26.06) - open/close 6936 (time 1709786987.11 total 274.00 last 19.71) - open/close 6969 (time 1709786999.63 total 286.52 last 2.64) - open/close 7178 (time 1709787010.81 total 297.70 last 18.69) - open/close 7387 (time 1709787021.61 total 308.50 last 19.36) - open/close 7635 (time 1709787032.14 total 319.03 last 23.54) - open/close 7729 (time 1709787042.89 total 329.78 last 8.75) - open/close 7834 (time 1709787052.91 total 339.80 last 10.47) - open/close 8022 (time 1709787063.14 total 350.03 last 18.38) - open/close 8154 (time 1709787073.21 total 360.10 last 13.10) - open/close 8358 (time 1709787083.53 total 370.42 last 19.78) - open/close 8581 (time 1709787094.23 total 381.12 last 20.84) - open/close 8710 (time 1709787104.70 total 391.59 last 12.32) - open/close 9049 (time 1709787115.75 total 402.64 last 30.68) - open/close 9367 (time 1709787126.75 total 413.64 last 28.90) - open/close 9577 (time 1709787137.50 total 424.39 last 19.55) - open/close 9779 (time 1709787147.50 total 434.39 last 20.19) - open/close 10000 (time 1709787153.61 total 440.50 last 36.16) - open/close 10922 (time 1709787163.62 total 450.51 last 92.16) - open/close 11784 (time 1709787173.62 total 460.51 last 86.16) - open/close 12405 (time 1709787183.63 total 470.52 last 62.08) - open/close 13001 (time 1709787193.63 total 480.52 last 59.58) - open/close 13546 (time 1709787203.63 total 490.52 last 54.50) - open/close 14023 (time 1709787213.63 total 500.52 last 47.69) - open/close 14537 (time 1709787223.65 total 510.54 last 51.32) - open/close 15032 (time 1709787233.65 total 520.54 last 49.50) - open/close 15519 (time 1709787243.66 total 530.55 last 48.63) - open/close 15769 (time 1709787259.80 total 546.69 last 15.49) - open/close 16289 (time 1709787269.80 total 556.69 last 51.96) - open/close 16756 (time 1709787279.81 total 566.70 last 46.70) - open/close 17127 (time 1709787289.81 total 576.70 last 37.08) - open/close 17585 (time 1709787299.82 total 586.71 last 45.76) - open/close 18013 (time 1709787309.84 total 596.73 last 42.71) - open/close 18473 (time 1709787319.84 total 606.73 last 45.99) - open/close 18924 (time 1709787329.85 total 616.74 last 45.08) - open/close 19335 (time 1709787339.87 total 626.76 last 40.99) - open/close 19751 (time 1709787349.91 total 636.80 last 41.47) - open/close 20000 (time 1709787355.95 total 642.84 last 41.17) - open/close 20351 (time 1709787365.98 total 652.87 last 35.01) - open/close 20442 (time 1709787379.47 total 666.36 last 6.74) - open/close 20832 (time 1709787389.48 total 676.37 last 38.98) - open/close 21193 (time 1709787399.48 total 686.37 last 36.10) - open/close 21567 (time 1709787409.51 total 696.40 last 37.27) - open/close 21906 (time 1709787419.52 total 706.41 last 33.89) - open/close 22212 (time 1709787429.52 total 716.41 last 30.58) - open/close 22516 (time 1709787439.54 total 726.43 last 30.35) - open/close 22860 (time 1709787449.54 total 736.43 last 34.40) - open/close 23172 (time 1709787459.57 total 746.46 last 31.11) - open/close 23505 (time 1709787469.60 total 756.49 last 33.20) - open/close 23820 (time 1709787479.61 total 766.50 last 31.48) - open/close 24133 (time 1709787489.62 total 776.51 last 31.25) - open/close 24455 (time 1709787499.65 total 786.54 last 32.12) - open/close 24751 (time 1709787509.68 total 796.57 last 29.49) - open/close 24786 (time 1709787522.64 total 809.53 last 2.70) - open/close 25106 (time 1709787532.67 total 819.56 last 31.89) - open/close 25408 (time 1709787542.72 total 829.62 last 30.04) - open/close 25749 (time 1709787552.74 total 839.63 last 34.03) - open/close 26076 (time 1709787562.74 total 849.64 last 32.70) - open/close 26446 (time 1709787572.82 total 859.71 last 36.74) - open/close 26780 (time 1709787582.85 total 869.74 last 33.28) - open/close 27162 (time 1709787592.86 total 879.75 last 38.19) - open/close 27283 (time 1709787607.33 total 894.22 last 8.36) - open/close 27643 (time 1709787617.33 total 904.22 last 36.00) - open/close 27985 (time 1709787627.36 total 914.25 last 34.09) - open/close 28326 (time 1709787637.39 total 924.28 last 34.03) - open/close 28663 (time 1709787647.41 total 934.30 last 33.61) - open/close 29046 (time 1709787657.43 total 944.32 last 38.22) - open/close 29407 (time 1709787667.44 total 954.33 last 36.09) - open/close 29744 (time 1709787677.49 total 964.38 last 33.53) - open/close 30000 (time 1709787685.38 total 972.27 last 32.44) - open/close 30347 (time 1709787695.41 total 982.30 last 34.60) - open/close 30686 (time 1709787705.41 total 992.30 last 33.88) - open/close 30973 (time 1709787715.42 total 1002.31 last 28.68) - open/close 31249 (time 1709787725.42 total 1012.31 last 27.58) - open/close 31606 (time 1709787735.46 total 1022.35 last 35.57) - open/close 31964 (time 1709787745.49 total 1032.38 last 35.68) - open/close 32275 (time 1709787755.52 total 1042.41 last 31.02) - open/close 32545 (time 1709787765.53 total 1052.42 last 26.97) - open/close 32794 (time 1709787775.56 total 1062.45 last 24.84) - open/close 33023 (time 1709787785.57 total 1072.46 last 22.87) - open/close 33275 (time 1709787795.60 total 1082.49 last 25.13) - open/close 33571 (time 1709787805.62 total 1092.51 last 29.53) - open/close 33872 (time 1709787815.64 total 1102.53 last 30.04) - open/close 34137 (time 1709787825.67 total 1112.56 last 26.43) - open/close 34427 (time 1709787835.69 total 1122.58 last 28.94) - open/close 34749 (time 1709787845.72 total 1132.61 last 32.09) - open/close 35097 (time 1709787855.74 total 1142.63 last 34.73) - open/close 35460 (time 1709787865.75 total 1152.64 last 36.27) - open/close 35806 (time 1709787875.76 total 1162.65 last 34.58) - open/close 36159 (time 1709787885.78 total 1172.67 last 35.23) - open/close 36508 (time 1709787895.80 total 1182.69 last 34.83) - open/close 36848 (time 1709787905.81 total 1192.70 last 33.94) - open/close 37168 (time 1709787915.83 total 1202.72 last 31.93) - open/close 37496 (time 1709787925.86 total 1212.75 last 32.72) - open/close 37827 (time 1709787935.86 total 1222.75 last 33.09) - open/close 38157 (time 1709787945.86 total 1232.75 last 32.99) - open/close 38509 (time 1709787955.87 total 1242.76 last 35.16) - open/close 38837 (time 1709787965.90 total 1252.79 last 32.72) - open/close 39158 (time 1709787975.90 total 1262.79 last 32.08) - open/close 39500 (time 1709787985.93 total 1272.82 last 34.10) - open/close 39860 (time 1709787995.94 total 1282.83 last 35.98) - open/close 40000 (time 1709788001.09 total 1287.98 last 27.14) - open/close 40233 (time 1709788011.13 total 1298.02 last 23.23) - open/close 40472 (time 1709788021.13 total 1308.02 last 23.88) - open/close 40796 (time 1709788031.15 total 1318.04 last 32.35) - open/close 41088 (time 1709788051.23 total 1338.12 last 14.54) - open/close 41393 (time 1709788061.25 total 1348.14 last 30.42) - open/close 41719 (time 1709788071.28 total 1358.17 last 32.51) - open/close 42027 (time 1709788081.30 total 1368.19 last 30.74) - open/close 42341 (time 1709788091.32 total 1378.21 last 31.32) - open/close 42645 (time 1709788101.33 total 1388.22 last 30.37) - open/close 42936 (time 1709788111.37 total 1398.26 last 28.98) - open/close 43244 (time 1709788121.40 total 1408.29 last 30.71) - open/close 43535 (time 1709788131.42 total 1418.31 last 29.06) - open/close 43803 (time 1709788141.45 total 1428.34 last 26.71) - open/close 44078 (time 1709788151.47 total 1438.36 last 27.46) - open/close 44305 (time 1709788161.48 total 1448.37 last 22.67) - open/close 44532 (time 1709788171.51 total 1458.40 last 22.64) - open/close 44826 (time 1709788181.55 total 1468.44 last 29.28) - open/close 45109 (time 1709788191.58 total 1478.47 last 28.21) - open/close 45350 (time 1709788201.62 total 1488.51 last 24.00) - open/close 45629 (time 1709788211.64 total 1498.53 last 27.86) - open/close 45916 (time 1709788221.64 total 1508.53 last 28.68) - open/close 46199 (time 1709788231.65 total 1518.54 last 28.27) - open/close 46456 (time 1709788241.66 total 1528.55 last 25.68) - open/close 46724 (time 1709788251.71 total 1538.60 last 26.68) - open/close 47003 (time 1709788261.72 total 1548.61 last 27.86) - open/close 47267 (time 1709788271.74 total 1558.63 last 26.36) - open/close 47513 (time 1709788281.77 total 1568.66 last 24.52) - open/close 47771 (time 1709788291.78 total 1578.67 last 25.78) - open/close 48046 (time 1709788301.78 total 1588.67 last 27.49) - open/close 48312 (time 1709788311.80 total 1598.69 last 26.55) - open/close 48572 (time 1709788321.82 total 1608.71 last 25.94) - open/close 48854 (time 1709788331.83 total 1618.72 last 28.17) - open/close 49127 (time 1709788341.85 total 1628.74 last 27.26) - open/close 49391 (time 1709788351.85 total 1638.74 last 26.38) - open/close 49638 (time 1709788361.87 total 1648.76 last 24.67) - open/close 49920 (time 1709788371.88 total 1658.77 last 28.17) total: 50000 open/close in 1661.63 seconds: 30.09 ops/second - unlinked 0 (time 1709788376 ; total 1 ; last 1) - unlinked 10000 (time 1709788811 ; total 436 ; last 435) - unlinked 20000 (time 1709789169 ; total 794 ; last 358) - unlinked 30000 (time 1709789440 ; total 1065 ; last 271) - unlinked 40000 (time 1709789584 ; total 1209 ; last 144) total: 50000 unlinks in 1266 seconds: 39.494473 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server unloading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2