== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:50 (1709787050) start mds service on oleg248-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg248-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg248-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock oleg248-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 194 (time 1709787098.35 total 10.40 last 18.65) - open/close 324 (time 1709787109.06 total 21.12 last 12.13) - open/close 481 (time 1709787119.08 total 31.13 last 15.68) - open/close 685 (time 1709787129.11 total 41.16 last 20.35) - open/close 879 (time 1709787139.12 total 51.17 last 19.37) - open/close 1086 (time 1709787149.24 total 61.30 last 20.45) - open/close 1239 (time 1709787159.67 total 71.72 last 14.67) - open/close 1318 (time 1709787171.37 total 83.42 last 6.75) - open/close 1509 (time 1709787182.21 total 94.26 last 17.63) - open/close 1640 (time 1709787193.16 total 105.21 last 11.96) - open/close 1843 (time 1709787203.68 total 115.73 last 19.30) - open/close 2043 (time 1709787213.68 total 125.73 last 20.00) - open/close 2221 (time 1709787224.70 total 136.75 last 16.16) - open/close 2449 (time 1709787235.60 total 147.65 last 20.91) - open/close 2655 (time 1709787245.60 total 157.65 last 20.60) - open/close 2874 (time 1709787255.91 total 167.97 last 21.23) - open/close 3117 (time 1709787266.39 total 178.44 last 23.19) - open/close 3276 (time 1709787276.40 total 188.45 last 15.89) - open/close 3451 (time 1709787287.28 total 199.34 last 16.08) - open/close 3618 (time 1709787297.30 total 209.35 last 16.67) - open/close 3729 (time 1709787307.60 total 219.65 last 10.78) - open/close 3885 (time 1709787317.60 total 229.66 last 15.59) - open/close 4025 (time 1709787328.09 total 240.14 last 13.36) - open/close 4129 (time 1709787338.38 total 250.43 last 10.11) - open/close 4340 (time 1709787348.38 total 260.44 last 21.08) - open/close 4572 (time 1709787358.45 total 270.51 last 23.04) - open/close 4774 (time 1709787369.29 total 281.35 last 18.64) - open/close 4888 (time 1709787380.24 total 292.29 last 10.41) - open/close 5005 (time 1709787390.40 total 302.45 last 11.52) - open/close 5218 (time 1709787400.41 total 312.47 last 21.27) - open/close 5448 (time 1709787410.59 total 322.64 last 22.61) - open/close 5543 (time 1709787420.61 total 332.66 last 9.48) - open/close 5572 (time 1709787434.36 total 346.41 last 2.11) - open/close 5712 (time 1709787444.37 total 356.42 last 13.99) - open/close 5902 (time 1709787454.38 total 366.44 last 18.97) - open/close 6074 (time 1709787464.44 total 376.49 last 17.10) - open/close 6217 (time 1709787475.22 total 387.28 last 13.26) - open/close 6376 (time 1709787485.24 total 397.29 last 15.88) - open/close 6575 (time 1709787495.27 total 407.33 last 19.83) - open/close 6748 (time 1709787505.36 total 417.41 last 17.16) - open/close 6904 (time 1709787515.37 total 427.42 last 15.58) - open/close 7094 (time 1709787525.80 total 437.85 last 18.22) - open/close 7247 (time 1709787535.81 total 447.86 last 15.29) - open/close 7437 (time 1709787545.83 total 457.89 last 18.95) - open/close 7626 (time 1709787555.85 total 467.90 last 18.88) - open/close 7893 (time 1709787565.85 total 477.90 last 26.69) - open/close 8027 (time 1709787575.85 total 487.91 last 13.40) - open/close 8206 (time 1709787586.67 total 498.73 last 16.54) - open/close 8439 (time 1709787596.69 total 508.74 last 23.26) - open/close 8584 (time 1709787607.03 total 519.08 last 14.02) - open/close 8763 (time 1709787617.88 total 529.94 last 16.50) - open/close 8892 (time 1709787627.98 total 540.03 last 12.78) - open/close 9100 (time 1709787638.00 total 550.05 last 20.76) - open/close 9285 (time 1709787648.58 total 560.63 last 17.48) - open/close 9431 (time 1709787659.25 total 571.30 last 13.69) - open/close 9595 (time 1709787669.40 total 581.45 last 16.15) - open/close 9835 (time 1709787679.41 total 591.46 last 23.97) - open/close 10000 (time 1709787686.00 total 598.05 last 25.04) - open/close 10414 (time 1709787696.01 total 608.07 last 41.35) - open/close 10839 (time 1709787706.02 total 618.07 last 42.47) - open/close 11219 (time 1709787716.02 total 628.08 last 37.99) - open/close 11572 (time 1709787726.03 total 638.09 last 35.27) - open/close 11952 (time 1709787736.04 total 648.09 last 37.97) - open/close 12354 (time 1709787746.06 total 658.12 last 40.11) - open/close 12723 (time 1709787756.08 total 668.14 last 36.83) - open/close 13112 (time 1709787766.09 total 678.15 last 38.86) - open/close 13502 (time 1709787776.12 total 688.18 last 38.88) - open/close 13913 (time 1709787786.15 total 698.20 last 41.00) - open/close 14308 (time 1709787796.16 total 708.21 last 39.47) - open/close 14726 (time 1709787806.20 total 718.25 last 41.62) - open/close 15082 (time 1709787816.21 total 728.26 last 35.58) - open/close 15447 (time 1709787826.21 total 738.26 last 36.48) - open/close 15861 (time 1709787836.22 total 748.28 last 41.35) - open/close 16180 (time 1709787846.24 total 758.30 last 31.83) - open/close 16505 (time 1709787856.26 total 768.31 last 32.45) - open/close 16815 (time 1709787866.26 total 778.31 last 31.00) - open/close 17170 (time 1709787876.26 total 788.32 last 35.49) - open/close 17495 (time 1709787886.28 total 798.33 last 32.44) - open/close 17788 (time 1709787896.34 total 808.40 last 29.12) - open/close 18084 (time 1709787906.36 total 818.42 last 29.54) - open/close 18403 (time 1709787916.39 total 828.44 last 31.82) - open/close 18697 (time 1709787926.40 total 838.45 last 29.37) - open/close 19012 (time 1709787936.41 total 848.46 last 31.45) - open/close 19343 (time 1709787946.43 total 858.49 last 33.02) - open/close 19730 (time 1709787956.45 total 868.50 last 38.65) - open/close 20000 (time 1709787963.46 total 875.51 last 38.50) - open/close 20347 (time 1709787973.50 total 885.56 last 34.55) - open/close 20693 (time 1709787983.51 total 895.56 last 34.60) - open/close 21071 (time 1709787993.52 total 905.58 last 37.73) - open/close 21486 (time 1709788003.52 total 915.58 last 41.50) - open/close 21909 (time 1709788013.54 total 925.59 last 42.24) - open/close 22351 (time 1709788023.54 total 935.60 last 44.18) - open/close 22789 (time 1709788033.54 total 945.60 last 43.80) - open/close 23227 (time 1709788043.55 total 955.61 last 43.75) - open/close 23662 (time 1709788053.57 total 965.63 last 43.41) - open/close 24083 (time 1709788063.58 total 975.63 last 42.09) - open/close 24478 (time 1709788073.59 total 985.64 last 39.45) - open/close 24902 (time 1709788083.61 total 995.67 last 42.30) - open/close 25312 (time 1709788093.63 total 1005.69 last 40.91) - open/close 25712 (time 1709788103.65 total 1015.71 last 39.92) - open/close 26103 (time 1709788113.67 total 1025.72 last 39.05) - open/close 26484 (time 1709788123.68 total 1035.74 last 38.05) - open/close 26881 (time 1709788133.68 total 1045.74 last 39.69) - open/close 27281 (time 1709788143.71 total 1055.76 last 39.91) - open/close 27680 (time 1709788153.71 total 1065.76 last 39.88) - open/close 28081 (time 1709788163.73 total 1075.78 last 40.02) - open/close 28501 (time 1709788173.74 total 1085.79 last 41.97) - open/close 28874 (time 1709788183.75 total 1095.80 last 37.26) - open/close 29279 (time 1709788193.75 total 1105.80 last 40.50) - open/close 29674 (time 1709788203.77 total 1115.82 last 39.41) - open/close 30000 (time 1709788212.25 total 1124.30 last 38.46) - open/close 30406 (time 1709788222.25 total 1134.30 last 40.58) - open/close 30823 (time 1709788232.27 total 1144.32 last 41.63) - open/close 31214 (time 1709788242.28 total 1154.34 last 39.04) - open/close 31633 (time 1709788252.28 total 1164.34 last 41.89) - open/close 32044 (time 1709788262.29 total 1174.34 last 41.08) - open/close 32442 (time 1709788272.30 total 1184.35 last 39.76) - open/close 32842 (time 1709788282.32 total 1194.38 last 39.91) - open/close 33212 (time 1709788292.33 total 1204.38 last 36.97) - open/close 33604 (time 1709788302.35 total 1214.40 last 39.13) - open/close 33984 (time 1709788312.36 total 1224.42 last 37.95) - open/close 34354 (time 1709788322.37 total 1234.42 last 36.99) - open/close 34717 (time 1709788332.38 total 1244.43 last 36.25) - open/close 35079 (time 1709788342.39 total 1254.44 last 36.16) - open/close 35438 (time 1709788352.40 total 1264.45 last 35.88) - open/close 35789 (time 1709788362.41 total 1274.47 last 35.05) - open/close 36138 (time 1709788372.42 total 1284.47 last 34.89) - open/close 36484 (time 1709788382.45 total 1294.50 last 34.50) - open/close 36812 (time 1709788392.47 total 1304.52 last 32.74) - open/close 37173 (time 1709788402.49 total 1314.54 last 36.01) - open/close 37520 (time 1709788412.49 total 1324.54 last 34.69) - open/close 37862 (time 1709788422.51 total 1334.56 last 34.15) - open/close 38210 (time 1709788432.52 total 1344.58 last 34.74) - open/close 38540 (time 1709788442.53 total 1354.58 last 32.98) - open/close 38874 (time 1709788452.56 total 1364.61 last 33.31) - open/close 39204 (time 1709788462.56 total 1374.61 last 32.99) - open/close 39534 (time 1709788472.56 total 1384.61 last 33.00) - open/close 39881 (time 1709788482.58 total 1394.64 last 34.62) - open/close 40000 (time 1709788486.40 total 1398.45 last 31.20) - open/close 40321 (time 1709788496.40 total 1408.46 last 32.08) - open/close 40648 (time 1709788506.40 total 1418.46 last 32.70) - open/close 40987 (time 1709788516.41 total 1428.47 last 33.87) - open/close 41308 (time 1709788526.44 total 1438.49 last 32.03) - open/close 41627 (time 1709788536.46 total 1448.52 last 31.81) - open/close 41944 (time 1709788546.49 total 1458.54 last 31.62) - open/close 42264 (time 1709788556.50 total 1468.55 last 31.96) - open/close 42595 (time 1709788566.51 total 1478.57 last 33.05) - open/close 42918 (time 1709788576.53 total 1488.58 last 32.26) - open/close 43244 (time 1709788586.53 total 1498.58 last 32.59) - open/close 43575 (time 1709788596.56 total 1508.61 last 33.01) - open/close 43906 (time 1709788606.56 total 1518.61 last 33.09) - open/close 44228 (time 1709788616.57 total 1528.62 last 32.18) - open/close 44550 (time 1709788626.58 total 1538.64 last 32.15) - open/close 44858 (time 1709788636.61 total 1548.66 last 30.72) - open/close 45159 (time 1709788646.61 total 1558.66 last 30.09) - open/close 45467 (time 1709788656.64 total 1568.69 last 30.71) - open/close 45765 (time 1709788666.64 total 1578.70 last 29.79) - open/close 46070 (time 1709788676.66 total 1588.72 last 30.43) - open/close 46366 (time 1709788686.70 total 1598.75 last 29.50) - open/close 46667 (time 1709788696.71 total 1608.76 last 30.08) - open/close 46961 (time 1709788706.71 total 1618.76 last 29.38) - open/close 47258 (time 1709788716.72 total 1628.77 last 29.68) - open/close 47553 (time 1709788726.74 total 1638.79 last 29.43) - open/close 47850 (time 1709788736.76 total 1648.81 last 29.66) - open/close 48131 (time 1709788746.76 total 1658.81 last 28.10) - open/close 48408 (time 1709788756.78 total 1668.83 last 27.65) - open/close 48688 (time 1709788766.80 total 1678.85 last 27.94) - open/close 48965 (time 1709788776.80 total 1688.85 last 27.69) - open/close 49235 (time 1709788786.82 total 1698.87 last 26.95) - open/close 49517 (time 1709788796.83 total 1708.89 last 28.16) - open/close 49793 (time 1709788806.85 total 1718.90 last 27.56) total: 50000 open/close in 1726.43 seconds: 28.96 ops/second - unlinked 0 (time 1709788815 ; total 0 ; last 0) - unlinked 10000 (time 1709789280 ; total 465 ; last 465) - unlinked 20000 (time 1709789668 ; total 853 ; last 388) - unlinked 30000 (time 1709789933 ; total 1118 ; last 265) - unlinked 40000 (time 1709790046 ; total 1231 ; last 113) total: 50000 unlinks in 1271 seconds: 39.339104 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server unloading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2