== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:42 (1709787222) start mds service on oleg258-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg258-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock oleg258-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 154 (time 1709787256.67 total 10.46 last 14.73) - open/close 351 (time 1709787266.69 total 20.47 last 19.67) - open/close 486 (time 1709787276.84 total 30.62 last 13.30) - open/close 626 (time 1709787287.79 total 41.57 last 12.79) - open/close 902 (time 1709787300.31 total 54.10 last 22.04) - open/close 1047 (time 1709787310.32 total 64.11 last 14.48) - open/close 1343 (time 1709787321.36 total 75.14 last 26.82) - open/close 1511 (time 1709787332.09 total 85.87 last 15.66) - open/close 1753 (time 1709787343.09 total 96.88 last 21.99) - open/close 1942 (time 1709787353.14 total 106.92 last 18.81) - open/close 2145 (time 1709787364.02 total 117.81 last 18.65) - open/close 2282 (time 1709787374.35 total 128.14 last 13.26) - open/close 2411 (time 1709787384.95 total 138.74 last 12.17) - open/close 2620 (time 1709787395.12 total 148.91 last 20.56) - open/close 2774 (time 1709787405.89 total 159.68 last 14.30) - open/close 2930 (time 1709787416.63 total 170.41 last 14.53) - open/close 3111 (time 1709787428.55 total 182.34 last 15.18) - open/close 3361 (time 1709787438.99 total 192.77 last 23.96) - open/close 3633 (time 1709787449.00 total 202.79 last 27.17) - open/close 3928 (time 1709787459.10 total 212.89 last 29.20) - open/close 4163 (time 1709787469.58 total 223.36 last 22.44) - open/close 4303 (time 1709787480.47 total 234.26 last 12.85) - open/close 4492 (time 1709787491.04 total 244.82 last 17.89) - open/close 4661 (time 1709787501.92 total 255.70 last 15.53) - open/close 4778 (time 1709787512.06 total 265.85 last 11.53) - open/close 4976 (time 1709787522.80 total 276.58 last 18.44) - open/close 5118 (time 1709787533.03 total 286.81 last 13.88) - open/close 5375 (time 1709787543.04 total 296.83 last 25.67) - open/close 5595 (time 1709787553.24 total 307.02 last 21.58) - open/close 5787 (time 1709787564.21 total 318.00 last 17.50) - open/close 6021 (time 1709787574.48 total 328.27 last 22.78) - open/close 6192 (time 1709787584.94 total 338.73 last 16.35) - open/close 6275 (time 1709787595.23 total 349.02 last 8.07) - open/close 6371 (time 1709787605.30 total 359.09 last 9.53) - open/close 6570 (time 1709787615.42 total 369.20 last 19.67) - open/close 6695 (time 1709787626.17 total 379.95 last 11.63) - open/close 6870 (time 1709787636.17 total 389.96 last 17.50) - open/close 7078 (time 1709787646.44 total 400.22 last 20.26) - open/close 7242 (time 1709787657.27 total 411.06 last 15.13) - open/close 7355 (time 1709787667.28 total 421.06 last 11.30) - open/close 7536 (time 1709787677.29 total 431.07 last 18.08) - open/close 7638 (time 1709787687.46 total 441.25 last 10.02) - open/close 7856 (time 1709787698.17 total 451.96 last 20.35) - open/close 7960 (time 1709787708.67 total 462.45 last 9.91) - open/close 8118 (time 1709787718.69 total 472.48 last 15.76) - open/close 8309 (time 1709787729.71 total 483.49 last 17.34) - open/close 8473 (time 1709787740.63 total 494.41 last 15.02) - open/close 8611 (time 1709787750.63 total 504.42 last 13.79) - open/close 8706 (time 1709787761.06 total 514.85 last 9.11) - open/close 8867 (time 1709787771.98 total 525.76 last 14.75) - open/close 8963 (time 1709787782.00 total 535.79 last 9.58) - open/close 9132 (time 1709787792.82 total 546.61 last 15.62) - open/close 9313 (time 1709787803.43 total 557.22 last 17.06) - open/close 9450 (time 1709787814.21 total 568.00 last 12.70) - open/close 9589 (time 1709787824.36 total 578.15 last 13.70) - open/close 9784 (time 1709787834.37 total 588.15 last 19.49) - open/close 10000 (time 1709787839.44 total 593.23 last 42.57) - open/close 10499 (time 1709787849.44 total 603.23 last 49.88) - open/close 10948 (time 1709787859.45 total 613.23 last 44.89) - open/close 11435 (time 1709787869.45 total 623.24 last 48.68) - open/close 11927 (time 1709787879.46 total 633.25 last 49.15) - open/close 12348 (time 1709787889.46 total 643.25 last 42.08) - open/close 12754 (time 1709787899.49 total 653.27 last 40.50) - open/close 13189 (time 1709787909.50 total 663.28 last 43.47) - open/close 13633 (time 1709787919.51 total 673.29 last 44.34) - open/close 14053 (time 1709787929.52 total 683.31 last 41.94) - open/close 14492 (time 1709787939.54 total 693.32 last 43.84) - open/close 14957 (time 1709787949.54 total 703.33 last 46.48) - open/close 15480 (time 1709787959.54 total 713.33 last 52.29) - open/close 15961 (time 1709787969.55 total 723.34 last 48.05) - open/close 16462 (time 1709787979.57 total 733.35 last 50.04) - open/close 16959 (time 1709787989.57 total 743.36 last 49.67) - open/close 17471 (time 1709787999.58 total 753.36 last 51.19) - open/close 18069 (time 1709788009.58 total 763.37 last 59.74) - open/close 18646 (time 1709788019.59 total 773.38 last 57.65) - open/close 19216 (time 1709788029.60 total 783.38 last 56.97) - open/close 19820 (time 1709788039.60 total 793.39 last 60.36) - open/close 20000 (time 1709788042.72 total 796.50 last 57.86) - open/close 20582 (time 1709788052.73 total 806.52 last 58.12) - open/close 21138 (time 1709788062.73 total 816.52 last 55.58) - open/close 21655 (time 1709788072.73 total 826.52 last 51.69) - open/close 21819 (time 1709788086.86 total 840.64 last 11.61) - open/close 22324 (time 1709788096.87 total 850.65 last 50.46) - open/close 22856 (time 1709788106.87 total 860.66 last 53.15) - open/close 23376 (time 1709788116.89 total 870.67 last 51.93) - open/close 23867 (time 1709788126.91 total 880.69 last 49.00) - open/close 24380 (time 1709788136.91 total 890.70 last 51.29) - open/close 24890 (time 1709788146.91 total 900.70 last 50.98) - open/close 25379 (time 1709788156.94 total 910.72 last 48.80) - open/close 25889 (time 1709788166.96 total 920.74 last 50.89) - open/close 26378 (time 1709788176.96 total 930.75 last 48.88) - open/close 26862 (time 1709788186.97 total 940.75 last 48.38) - open/close 27350 (time 1709788196.97 total 950.75 last 48.80) - open/close 27823 (time 1709788206.97 total 960.75 last 47.30) - open/close 28300 (time 1709788216.99 total 970.78 last 47.59) - open/close 28772 (time 1709788227.00 total 980.79 last 47.16) - open/close 29223 (time 1709788237.00 total 990.79 last 45.09) - open/close 29686 (time 1709788247.00 total 1000.79 last 46.29) - open/close 30000 (time 1709788254.03 total 1007.81 last 44.72) - open/close 30433 (time 1709788264.03 total 1017.81 last 43.30) - open/close 30872 (time 1709788274.04 total 1027.83 last 43.84) - open/close 31273 (time 1709788284.06 total 1037.85 last 40.02) - open/close 31682 (time 1709788294.08 total 1047.87 last 40.82) - open/close 32096 (time 1709788304.08 total 1057.87 last 41.38) - open/close 32503 (time 1709788314.08 total 1067.87 last 40.70) - open/close 32901 (time 1709788324.10 total 1077.88 last 39.76) - open/close 33288 (time 1709788334.12 total 1087.90 last 38.61) - open/close 33689 (time 1709788344.13 total 1097.91 last 40.07) - open/close 34070 (time 1709788354.15 total 1107.93 last 38.02) - open/close 34448 (time 1709788364.15 total 1117.94 last 37.78) - open/close 34820 (time 1709788374.17 total 1127.95 last 37.15) - open/close 35191 (time 1709788384.18 total 1137.97 last 37.03) - open/close 35541 (time 1709788394.19 total 1147.97 last 35.00) - open/close 35927 (time 1709788404.19 total 1157.98 last 38.59) - open/close 36299 (time 1709788414.19 total 1167.98 last 37.19) - open/close 36668 (time 1709788424.19 total 1177.98 last 36.89) - open/close 37034 (time 1709788434.20 total 1187.98 last 36.58) - open/close 37377 (time 1709788444.20 total 1197.99 last 34.28) - open/close 37736 (time 1709788454.21 total 1208.00 last 35.87) - open/close 38081 (time 1709788464.23 total 1218.02 last 34.42) - open/close 38432 (time 1709788474.24 total 1228.03 last 35.07) - open/close 38795 (time 1709788484.26 total 1238.04 last 36.25) - open/close 39137 (time 1709788494.27 total 1248.06 last 34.15) - open/close 39483 (time 1709788504.30 total 1258.08 last 34.52) - open/close 39846 (time 1709788514.30 total 1268.09 last 36.28) - open/close 40000 (time 1709788518.77 total 1272.55 last 34.48) - open/close 40343 (time 1709788528.78 total 1282.57 last 34.26) - open/close 40687 (time 1709788538.81 total 1292.59 last 34.31) - open/close 41019 (time 1709788548.81 total 1302.59 last 33.20) - open/close 41360 (time 1709788558.82 total 1312.61 last 34.05) - open/close 41710 (time 1709788568.82 total 1322.61 last 35.00) - open/close 42055 (time 1709788578.82 total 1332.61 last 34.50) - open/close 42401 (time 1709788588.82 total 1342.61 last 34.60) - open/close 42754 (time 1709788598.84 total 1352.63 last 35.23) - open/close 43094 (time 1709788608.84 total 1362.63 last 34.00) - open/close 43434 (time 1709788618.85 total 1372.64 last 33.97) - open/close 43771 (time 1709788628.86 total 1382.64 last 33.67) - open/close 44086 (time 1709788638.88 total 1392.66 last 31.44) - open/close 44402 (time 1709788648.89 total 1402.67 last 31.57) - open/close 44720 (time 1709788658.91 total 1412.70 last 31.72) - open/close 45029 (time 1709788668.92 total 1422.71 last 30.86) - open/close 45346 (time 1709788678.93 total 1432.72 last 31.67) - open/close 45648 (time 1709788688.95 total 1442.73 last 30.15) - open/close 45959 (time 1709788698.98 total 1452.76 last 31.01) - open/close 46260 (time 1709788708.98 total 1462.77 last 30.08) - open/close 46573 (time 1709788719.00 total 1472.79 last 31.24) - open/close 46874 (time 1709788729.03 total 1482.82 last 30.02) - open/close 47177 (time 1709788739.06 total 1492.85 last 30.21) - open/close 47468 (time 1709788749.06 total 1502.85 last 29.09) - open/close 47754 (time 1709788759.07 total 1512.86 last 28.57) - open/close 48048 (time 1709788769.10 total 1522.89 last 29.32) - open/close 48329 (time 1709788779.13 total 1532.92 last 28.02) - open/close 48614 (time 1709788789.14 total 1542.93 last 28.46) - open/close 48903 (time 1709788799.16 total 1552.95 last 28.85) - open/close 49182 (time 1709788809.20 total 1562.98 last 27.81) - open/close 49459 (time 1709788819.23 total 1573.02 last 27.61) - open/close 49727 (time 1709788829.24 total 1583.02 last 26.78) total: 50000 open/close in 1592.83 seconds: 31.39 ops/second - unlinked 0 (time 1709788840 ; total 1 ; last 1) - unlinked 10000 (time 1709789304 ; total 465 ; last 464) - unlinked 20000 (time 1709789684 ; total 845 ; last 380) - unlinked 30000 (time 1709789946 ; total 1107 ; last 262) - unlinked 40000 (time 1709790052 ; total 1213 ; last 106) total: 50000 unlinks in 1248 seconds: 40.064102 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server stop mds service on oleg258-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server unloading modules on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2