== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:49 (1709787229) start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1709787262.33 total 10.04 last 25.59) - open/close 497 (time 1709787272.36 total 20.07 last 23.93) - open/close 714 (time 1709787282.52 total 30.23 last 21.36) - open/close 888 (time 1709787297.12 total 44.83 last 11.92) - open/close 1069 (time 1709787307.14 total 54.85 last 18.06) - open/close 1288 (time 1709787317.50 total 65.21 last 21.14) - open/close 1474 (time 1709787328.09 total 75.80 last 17.56) - open/close 1619 (time 1709787338.31 total 86.03 last 14.18) - open/close 1801 (time 1709787350.90 total 98.61 last 14.46) - open/close 2036 (time 1709787361.35 total 109.07 last 22.47) - open/close 2216 (time 1709787372.36 total 120.07 last 16.36) - open/close 2363 (time 1709787382.68 total 130.40 last 14.24) - open/close 2575 (time 1709787393.11 total 140.82 last 20.33) - open/close 2780 (time 1709787403.12 total 150.83 last 20.48) - open/close 2956 (time 1709787413.86 total 161.57 last 16.39) - open/close 3158 (time 1709787424.26 total 171.98 last 19.42) - open/close 3386 (time 1709787434.43 total 182.14 last 22.43) - open/close 3524 (time 1709787444.92 total 192.63 last 13.16) - open/close 3634 (time 1709787455.89 total 203.60 last 10.02) - open/close 3853 (time 1709787466.91 total 214.62 last 19.88) - open/close 4083 (time 1709787476.92 total 224.63 last 22.98) - open/close 4172 (time 1709787488.42 total 236.14 last 7.73) - open/close 4337 (time 1709787498.53 total 246.25 last 16.32) - open/close 4531 (time 1709787508.55 total 256.26 last 19.37) - open/close 4723 (time 1709787519.15 total 266.86 last 18.11) - open/close 4892 (time 1709787529.30 total 277.02 last 16.64) - open/close 5040 (time 1709787539.43 total 287.15 last 14.61) - open/close 5247 (time 1709787550.93 total 298.64 last 18.01) - open/close 5501 (time 1709787560.93 total 308.65 last 25.39) - open/close 5812 (time 1709787571.85 total 319.56 last 28.50) - open/close 6124 (time 1709787581.85 total 329.57 last 31.17) - open/close 6319 (time 1709787592.81 total 340.52 last 17.80) - open/close 6539 (time 1709787602.81 total 350.53 last 21.99) - open/close 6769 (time 1709787613.38 total 361.09 last 21.77) - open/close 6992 (time 1709787625.74 total 373.45 last 18.05) - open/close 7270 (time 1709787636.12 total 383.83 last 26.77) - open/close 7468 (time 1709787648.15 total 395.86 last 16.46) - open/close 7717 (time 1709787667.91 total 415.62 last 12.60) - open/close 7877 (time 1709787677.91 total 425.63 last 15.99) - open/close 8001 (time 1709787688.26 total 435.97 last 11.99) - open/close 8262 (time 1709787698.28 total 446.00 last 26.03) - open/close 8367 (time 1709787708.29 total 456.00 last 10.50) - open/close 8623 (time 1709787718.31 total 466.03 last 25.53) - open/close 8809 (time 1709787728.34 total 476.06 last 18.54) - open/close 8981 (time 1709787738.88 total 486.60 last 16.32) - open/close 9177 (time 1709787748.91 total 496.62 last 19.56) - open/close 9403 (time 1709787758.91 total 506.62 last 22.60) - open/close 9609 (time 1709787768.92 total 516.64 last 20.57) - open/close 9961 (time 1709787778.94 total 526.65 last 35.15) - open/close 10503 (time 1709787788.94 total 536.66 last 54.16) - open/close 11023 (time 1709787798.95 total 546.66 last 51.98) - open/close 11645 (time 1709787809.05 total 556.77 last 61.56) - open/close 12195 (time 1709787819.06 total 566.78 last 54.95) - open/close 12665 (time 1709787829.06 total 576.78 last 47.00) - open/close 13215 (time 1709787839.07 total 586.78 last 54.96) - open/close 13773 (time 1709787849.09 total 596.80 last 55.68) - open/close 13981 (time 1709787863.97 total 611.68 last 13.98) - open/close 14528 (time 1709787873.97 total 621.69 last 54.67) - open/close 15098 (time 1709787883.99 total 631.71 last 56.88) - open/close 15631 (time 1709787894.00 total 641.72 last 53.25) - open/close 16149 (time 1709787904.01 total 651.73 last 51.75) - open/close 16633 (time 1709787914.04 total 661.75 last 48.27) - open/close 17100 (time 1709787924.06 total 671.77 last 46.60) - open/close 17569 (time 1709787934.07 total 681.79 last 46.84) - open/close 18019 (time 1709787944.09 total 691.81 last 44.91) - open/close 18484 (time 1709787954.10 total 701.82 last 46.45) - open/close 18935 (time 1709787964.12 total 711.84 last 45.01) - open/close 19372 (time 1709787974.15 total 721.86 last 43.60) - open/close 19801 (time 1709787984.15 total 731.87 last 42.87) - open/close 20000 (time 1709787988.65 total 736.37 last 44.22) - open/close 20453 (time 1709787998.67 total 746.39 last 45.22) - open/close 20900 (time 1709788008.68 total 756.39 last 44.67) - open/close 21317 (time 1709788018.70 total 766.41 last 41.61) - open/close 21748 (time 1709788028.70 total 776.42 last 43.09) - open/close 22165 (time 1709788038.72 total 786.44 last 41.62) - open/close 22542 (time 1709788048.74 total 796.46 last 37.63) - open/close 22928 (time 1709788058.75 total 806.46 last 38.58) - open/close 23301 (time 1709788068.77 total 816.48 last 37.22) - open/close 23710 (time 1709788078.79 total 826.50 last 40.82) - open/close 24103 (time 1709788088.79 total 836.51 last 39.29) - open/close 24484 (time 1709788098.81 total 846.52 last 38.03) - open/close 24851 (time 1709788108.82 total 856.53 last 36.67) - open/close 25231 (time 1709788118.83 total 866.55 last 37.94) - open/close 25607 (time 1709788128.85 total 876.57 last 37.54) - open/close 25953 (time 1709788138.89 total 886.60 last 34.48) - open/close 26301 (time 1709788148.92 total 896.63 last 34.70) - open/close 26636 (time 1709788158.92 total 906.63 last 33.49) - open/close 26957 (time 1709788168.94 total 916.66 last 32.03) - open/close 27282 (time 1709788178.94 total 926.66 last 32.49) - open/close 27642 (time 1709788188.95 total 936.66 last 35.99) - open/close 27967 (time 1709788198.97 total 946.69 last 32.41) - open/close 28288 (time 1709788208.98 total 956.69 last 32.08) - open/close 28610 (time 1709788219.00 total 966.71 last 32.14) - open/close 28929 (time 1709788229.02 total 976.73 last 31.84) - open/close 29236 (time 1709788239.03 total 986.75 last 30.64) - open/close 29548 (time 1709788249.06 total 996.77 last 31.13) - open/close 29895 (time 1709788259.07 total 1006.79 last 34.65) - open/close 30000 (time 1709788262.12 total 1009.83 last 34.50) - open/close 30313 (time 1709788272.13 total 1019.85 last 31.24) - open/close 30619 (time 1709788282.15 total 1029.86 last 30.56) - open/close 30943 (time 1709788292.16 total 1039.88 last 32.34) - open/close 31262 (time 1709788302.19 total 1049.91 last 31.81) - open/close 31572 (time 1709788312.19 total 1059.91 last 31.00) - open/close 31879 (time 1709788322.21 total 1069.92 last 30.64) - open/close 32182 (time 1709788332.22 total 1079.93 last 30.27) - open/close 32478 (time 1709788342.22 total 1089.94 last 29.58) - open/close 32763 (time 1709788352.23 total 1099.94 last 28.50) - open/close 33041 (time 1709788362.24 total 1109.95 last 27.77) - open/close 33324 (time 1709788372.26 total 1119.97 last 28.24) - open/close 33608 (time 1709788382.27 total 1129.98 last 28.37) - open/close 33891 (time 1709788392.28 total 1140.00 last 28.26) - open/close 34181 (time 1709788402.32 total 1150.03 last 28.90) - open/close 34462 (time 1709788412.35 total 1160.07 last 28.00) - open/close 34743 (time 1709788422.38 total 1170.10 last 28.01) - open/close 35023 (time 1709788432.41 total 1180.12 last 27.93) - open/close 35307 (time 1709788442.43 total 1190.15 last 28.33) - open/close 35590 (time 1709788452.44 total 1200.15 last 28.29) - open/close 35854 (time 1709788462.46 total 1210.18 last 26.33) - open/close 36124 (time 1709788472.48 total 1220.20 last 26.95) - open/close 36398 (time 1709788482.49 total 1230.21 last 27.37) - open/close 36685 (time 1709788492.49 total 1240.21 last 28.70) - open/close 36974 (time 1709788502.49 total 1250.21 last 28.89) - open/close 37274 (time 1709788512.52 total 1260.23 last 29.93) - open/close 37538 (time 1709788522.55 total 1270.26 last 26.32) - open/close 37809 (time 1709788532.56 total 1280.27 last 27.07) - open/close 38075 (time 1709788542.58 total 1290.30 last 26.54) - open/close 38345 (time 1709788552.61 total 1300.32 last 26.94) - open/close 38613 (time 1709788562.61 total 1310.33 last 26.79) - open/close 38882 (time 1709788572.63 total 1320.35 last 26.84) - open/close 39154 (time 1709788582.64 total 1330.35 last 27.19) - open/close 39434 (time 1709788592.65 total 1340.36 last 27.96) - open/close 39716 (time 1709788602.66 total 1350.38 last 28.16) - open/close 39980 (time 1709788612.68 total 1360.40 last 26.35) - open/close 40251 (time 1709788622.71 total 1370.43 last 27.02) - open/close 40536 (time 1709788632.73 total 1380.44 last 28.45) - open/close 40820 (time 1709788642.73 total 1390.44 last 28.39) - open/close 41099 (time 1709788652.75 total 1400.46 last 27.85) - open/close 41366 (time 1709788662.75 total 1410.47 last 26.69) - open/close 41630 (time 1709788672.76 total 1420.47 last 26.39) - open/close 41891 (time 1709788682.78 total 1430.50 last 26.03) - open/close 42162 (time 1709788692.81 total 1440.52 last 27.02) - open/close 42421 (time 1709788702.81 total 1450.53 last 25.89) - open/close 42677 (time 1709788712.84 total 1460.56 last 25.53) - open/close 42919 (time 1709788722.87 total 1470.58 last 24.13) - open/close 43170 (time 1709788732.90 total 1480.61 last 25.04) - open/close 43414 (time 1709788742.90 total 1490.62 last 24.38) - open/close 43656 (time 1709788752.92 total 1500.64 last 24.15) - open/close 43906 (time 1709788762.92 total 1510.64 last 25.00) - open/close 44163 (time 1709788772.94 total 1520.66 last 25.65) - open/close 44399 (time 1709788782.96 total 1530.67 last 23.56) - open/close 44640 (time 1709788792.96 total 1540.68 last 24.09) - open/close 44877 (time 1709788802.98 total 1550.70 last 23.65) - open/close 45114 (time 1709788812.99 total 1560.71 last 23.68) - open/close 45349 (time 1709788823.03 total 1570.75 last 23.41) - open/close 45587 (time 1709788833.04 total 1580.76 last 23.77) - open/close 45824 (time 1709788843.07 total 1590.78 last 23.64) - open/close 46046 (time 1709788853.11 total 1600.82 last 22.12) - open/close 46281 (time 1709788863.11 total 1610.83 last 23.49) - open/close 46521 (time 1709788873.12 total 1620.83 last 23.99) - open/close 46758 (time 1709788883.15 total 1630.86 last 23.63) - open/close 46981 (time 1709788893.19 total 1640.91 last 22.20) - open/close 47209 (time 1709788903.21 total 1650.92 last 22.76) - open/close 47442 (time 1709788913.23 total 1660.95 last 23.24) - open/close 47666 (time 1709788923.26 total 1670.97 last 22.35) - open/close 47890 (time 1709788933.27 total 1680.99 last 22.36) - open/close 48118 (time 1709788943.31 total 1691.02 last 22.72) - open/close 48342 (time 1709788953.34 total 1701.06 last 22.32) - open/close 48569 (time 1709788963.39 total 1711.10 last 22.60) - open/close 48789 (time 1709788973.43 total 1721.14 last 21.92) - open/close 49005 (time 1709788983.44 total 1731.16 last 21.56) - open/close 49222 (time 1709788993.49 total 1741.20 last 21.61) - open/close 49446 (time 1709789003.51 total 1751.22 last 22.36) - open/close 49669 (time 1709789013.55 total 1761.26 last 22.22) - open/close 49885 (time 1709789023.55 total 1771.26 last 21.60) total: 50000 open/close in 1776.31 seconds: 28.15 ops/second - unlinked 0 (time 1709789029 ; total 0 ; last 0) - unlinked 10000 (time 1709789554 ; total 525 ; last 525) - unlinked 20000 (time 1709789964 ; total 935 ; last 410) - unlinked 30000 (time 1709790212 ; total 1183 ; last 248) - unlinked 40000 (time 1709790297 ; total 1268 ; last 85) total: 50000 unlinks in 1295 seconds: 38.610039 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2