== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:53 (1709786813) start mds service on oleg331-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock oleg331-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 303 (time 1709786842.03 total 11.84 last 25.59) - open/close 539 (time 1709786854.21 total 24.02 last 19.38) - open/close 807 (time 1709786864.40 total 34.20 last 26.32) - open/close 1077 (time 1709786875.07 total 44.88 last 25.29) - open/close 1300 (time 1709786885.49 total 55.30 last 21.40) - open/close 1494 (time 1709786896.06 total 65.86 last 18.36) - open/close 1654 (time 1709786906.11 total 75.91 last 15.92) - open/close 1855 (time 1709786916.14 total 85.95 last 20.03) - open/close 2037 (time 1709786926.57 total 96.38 last 17.44) - open/close 2339 (time 1709786936.74 total 106.54 last 29.71) - open/close 2487 (time 1709786947.01 total 116.81 last 14.41) - open/close 2693 (time 1709786960.00 total 129.81 last 15.85) - open/close 2822 (time 1709786970.77 total 140.58 last 11.98) - open/close 3025 (time 1709786981.45 total 151.26 last 19.01) - open/close 3257 (time 1709786992.02 total 161.83 last 21.95) - open/close 3481 (time 1709787002.03 total 171.83 last 22.39) - open/close 3662 (time 1709787012.24 total 182.04 last 17.72) - open/close 3850 (time 1709787022.36 total 192.16 last 18.58) - open/close 4074 (time 1709787032.37 total 202.17 last 22.38) - open/close 4355 (time 1709787043.37 total 213.17 last 25.54) - open/close 4576 (time 1709787054.25 total 224.05 last 20.31) - open/close 4735 (time 1709787064.42 total 234.23 last 15.63) - open/close 5092 (time 1709787074.64 total 244.45 last 34.93) - open/close 5293 (time 1709787085.60 total 255.40 last 18.35) - open/close 5456 (time 1709787096.23 total 266.03 last 15.34) - open/close 5740 (time 1709787106.61 total 276.42 last 27.35) - open/close 5944 (time 1709787116.61 total 286.42 last 20.40) - open/close 6067 (time 1709787126.63 total 296.44 last 12.28) - open/close 6283 (time 1709787136.65 total 306.46 last 21.56) - open/close 6417 (time 1709787146.95 total 316.75 last 13.01) - open/close 6659 (time 1709787157.76 total 327.56 last 22.39) - open/close 6701 (time 1709787170.43 total 340.24 last 3.31) - open/close 6841 (time 1709787181.05 total 350.86 last 13.18) - open/close 7007 (time 1709787191.27 total 361.07 last 16.25) - open/close 7150 (time 1709787203.17 total 372.97 last 12.02) - open/close 7314 (time 1709787213.50 total 383.31 last 15.87) - open/close 7507 (time 1709787223.52 total 393.33 last 19.26) - open/close 7656 (time 1709787233.60 total 403.40 last 14.79) - open/close 7846 (time 1709787244.05 total 413.85 last 18.19) - open/close 8047 (time 1709787255.05 total 424.86 last 18.26) - open/close 8279 (time 1709787265.07 total 434.88 last 23.15) - open/close 8502 (time 1709787275.29 total 445.09 last 21.83) - open/close 8693 (time 1709787285.94 total 455.75 last 17.93) - open/close 8894 (time 1709787295.96 total 465.77 last 20.06) - open/close 9129 (time 1709787306.11 total 475.92 last 23.15) - open/close 9222 (time 1709787316.46 total 486.26 last 8.99) - open/close 9335 (time 1709787326.82 total 496.62 last 10.91) - open/close 9546 (time 1709787337.56 total 507.37 last 19.64) - open/close 9995 (time 1709787347.58 total 517.38 last 44.83) - open/close 10515 (time 1709787357.58 total 527.38 last 51.99) - open/close 11067 (time 1709787367.59 total 537.39 last 55.16) - open/close 11484 (time 1709787377.61 total 547.42 last 41.60) - open/close 11989 (time 1709787387.65 total 557.46 last 50.30) - open/close 12444 (time 1709787397.66 total 567.47 last 45.46) - open/close 12881 (time 1709787407.66 total 577.47 last 43.70) - open/close 13337 (time 1709787417.66 total 587.47 last 45.59) - open/close 13869 (time 1709787427.67 total 597.47 last 53.19) - open/close 14353 (time 1709787437.68 total 607.49 last 48.31) - open/close 14855 (time 1709787447.69 total 617.50 last 50.17) - open/close 15331 (time 1709787457.70 total 627.51 last 47.55) - open/close 15720 (time 1709787467.72 total 637.52 last 38.84) - open/close 16127 (time 1709787477.76 total 647.56 last 40.53) - open/close 16569 (time 1709787487.79 total 657.60 last 44.05) - open/close 16972 (time 1709787497.80 total 667.60 last 40.27) - open/close 17368 (time 1709787507.81 total 677.61 last 39.58) - open/close 17795 (time 1709787517.82 total 687.62 last 42.64) - open/close 18144 (time 1709787527.82 total 697.63 last 34.89) - open/close 18521 (time 1709787537.83 total 707.63 last 37.69) - open/close 18924 (time 1709787547.83 total 717.64 last 40.27) - open/close 19274 (time 1709787557.85 total 727.66 last 34.93) - open/close 19632 (time 1709787567.85 total 737.66 last 35.79) - open/close 19961 (time 1709787577.87 total 747.67 last 32.85) - open/close 20295 (time 1709787587.89 total 757.70 last 33.32) - open/close 20640 (time 1709787597.92 total 767.72 last 34.42) - open/close 20958 (time 1709787607.92 total 777.72 last 31.79) - open/close 21300 (time 1709787617.93 total 787.74 last 34.15) - open/close 21624 (time 1709787627.94 total 797.75 last 32.38) - open/close 22001 (time 1709787637.94 total 807.75 last 37.69) - open/close 22296 (time 1709787647.96 total 817.76 last 29.46) - open/close 22604 (time 1709787657.97 total 827.78 last 30.74) - open/close 22921 (time 1709787667.98 total 837.79 last 31.68) - open/close 23209 (time 1709787678.00 total 847.80 last 28.75) - open/close 23506 (time 1709787688.01 total 857.81 last 29.68) - open/close 23767 (time 1709787698.01 total 867.82 last 26.08) - open/close 24015 (time 1709787708.02 total 877.83 last 24.78) - open/close 24263 (time 1709787718.07 total 887.87 last 24.69) - open/close 24567 (time 1709787728.08 total 897.88 last 30.36) - open/close 24865 (time 1709787738.10 total 907.90 last 29.75) - open/close 25137 (time 1709787748.13 total 917.93 last 27.12) - open/close 25379 (time 1709787758.13 total 927.93 last 24.20) - open/close 25683 (time 1709787768.20 total 938.00 last 30.20) - open/close 25945 (time 1709787778.21 total 948.02 last 26.15) - open/close 26238 (time 1709787788.23 total 958.03 last 29.27) - open/close 26488 (time 1709787798.26 total 968.06 last 24.92) - open/close 26806 (time 1709787808.26 total 978.07 last 31.78) - open/close 27100 (time 1709787818.27 total 988.07 last 29.39) - open/close 27387 (time 1709787828.28 total 998.08 last 28.66) - open/close 27720 (time 1709787838.32 total 1008.13 last 33.16) - open/close 28012 (time 1709787848.33 total 1018.14 last 29.18) - open/close 28316 (time 1709787858.41 total 1028.21 last 30.18) - open/close 28589 (time 1709787868.41 total 1038.22 last 27.28) - open/close 28866 (time 1709787878.44 total 1048.25 last 27.62) - open/close 28995 (time 1709787894.53 total 1064.34 last 8.02) - open/close 29304 (time 1709787904.55 total 1074.35 last 30.85) - open/close 29615 (time 1709787914.57 total 1084.38 last 31.02) - open/close 29896 (time 1709787924.59 total 1094.39 last 28.06) - open/close 30000 (time 1709787928.54 total 1098.35 last 26.31) - open/close 30278 (time 1709787938.56 total 1108.36 last 27.77) - open/close 30545 (time 1709787948.56 total 1118.37 last 26.67) - open/close 30813 (time 1709787958.60 total 1128.41 last 26.70) - open/close 31069 (time 1709787968.61 total 1138.41 last 25.58) - open/close 31337 (time 1709787978.62 total 1148.42 last 26.77) - open/close 31634 (time 1709787988.62 total 1158.43 last 29.69) - open/close 31936 (time 1709787998.65 total 1168.46 last 30.10) - open/close 32224 (time 1709788008.67 total 1178.48 last 28.74) - open/close 32503 (time 1709788018.70 total 1188.50 last 27.84) - open/close 32779 (time 1709788028.75 total 1198.56 last 27.44) - open/close 33055 (time 1709788038.77 total 1208.57 last 27.56) - open/close 33306 (time 1709788048.78 total 1218.59 last 25.07) - open/close 33557 (time 1709788058.79 total 1228.59 last 25.09) - open/close 33816 (time 1709788068.82 total 1238.63 last 25.81) - open/close 34125 (time 1709788078.84 total 1248.64 last 30.85) - open/close 34469 (time 1709788088.86 total 1258.67 last 34.31) - open/close 34811 (time 1709788098.87 total 1268.68 last 34.17) - open/close 35152 (time 1709788108.89 total 1278.69 last 34.05) - open/close 35487 (time 1709788118.91 total 1288.71 last 33.42) - open/close 35829 (time 1709788128.92 total 1298.72 last 34.17) - open/close 36169 (time 1709788138.93 total 1308.73 last 33.97) - open/close 36496 (time 1709788148.94 total 1318.74 last 32.67) - open/close 36809 (time 1709788158.95 total 1328.75 last 31.26) - open/close 37130 (time 1709788168.98 total 1338.78 last 32.02) - open/close 37452 (time 1709788178.99 total 1348.80 last 32.14) - open/close 37777 (time 1709788189.03 total 1358.83 last 32.38) - open/close 38086 (time 1709788199.03 total 1368.84 last 30.89) - open/close 38386 (time 1709788209.03 total 1378.84 last 29.99) - open/close 38696 (time 1709788219.04 total 1388.84 last 30.99) - open/close 39021 (time 1709788229.06 total 1398.86 last 32.44) - open/close 39321 (time 1709788239.09 total 1408.90 last 29.89) - open/close 39609 (time 1709788249.12 total 1418.92 last 28.73) - open/close 39884 (time 1709788259.12 total 1428.93 last 27.48) - open/close 40000 (time 1709788262.86 total 1432.67 last 31.03) - open/close 40271 (time 1709788272.89 total 1442.69 last 27.03) - open/close 40533 (time 1709788282.92 total 1452.72 last 26.12) - open/close 40775 (time 1709788292.93 total 1462.73 last 24.18) - open/close 41031 (time 1709788302.97 total 1472.77 last 25.50) - open/close 41270 (time 1709788312.97 total 1482.78 last 23.88) - open/close 41520 (time 1709788322.98 total 1492.78 last 25.00) - open/close 41781 (time 1709788333.01 total 1502.81 last 26.02) - open/close 42036 (time 1709788343.04 total 1512.84 last 25.42) - open/close 42285 (time 1709788353.06 total 1522.86 last 24.85) - open/close 42530 (time 1709788363.10 total 1532.90 last 24.39) - open/close 42766 (time 1709788373.13 total 1542.93 last 23.53) - open/close 43010 (time 1709788383.15 total 1552.95 last 24.35) - open/close 43249 (time 1709788393.17 total 1562.98 last 23.84) - open/close 43500 (time 1709788403.20 total 1573.01 last 25.03) - open/close 43756 (time 1709788413.23 total 1583.03 last 25.53) - open/close 44018 (time 1709788423.24 total 1593.04 last 26.18) - open/close 44256 (time 1709788433.26 total 1603.06 last 23.75) - open/close 44510 (time 1709788443.27 total 1613.08 last 25.37) - open/close 44760 (time 1709788453.28 total 1623.08 last 24.98) - open/close 45021 (time 1709788463.28 total 1633.09 last 26.09) - open/close 45272 (time 1709788473.32 total 1643.13 last 25.00) - open/close 45527 (time 1709788483.33 total 1653.13 last 25.49) - open/close 45773 (time 1709788493.34 total 1663.14 last 24.57) - open/close 46009 (time 1709788503.36 total 1673.17 last 23.54) - open/close 46258 (time 1709788513.41 total 1683.22 last 24.78) - open/close 46508 (time 1709788523.44 total 1693.25 last 24.92) - open/close 46758 (time 1709788533.48 total 1703.29 last 24.91) - open/close 47010 (time 1709788543.50 total 1713.30 last 25.16) - open/close 47264 (time 1709788553.51 total 1723.31 last 25.38) - open/close 47495 (time 1709788563.51 total 1733.31 last 23.10) - open/close 47734 (time 1709788573.54 total 1743.34 last 23.83) - open/close 47965 (time 1709788583.56 total 1753.36 last 23.05) - open/close 48191 (time 1709788593.57 total 1763.37 last 22.57) - open/close 48430 (time 1709788603.58 total 1773.39 last 23.87) - open/close 48682 (time 1709788613.62 total 1783.42 last 25.11) - open/close 48756 (time 1709788627.55 total 1797.36 last 5.31) - open/close 49010 (time 1709788637.57 total 1807.38 last 25.35) - open/close 49273 (time 1709788647.59 total 1817.40 last 26.24) - open/close 49528 (time 1709788657.63 total 1827.43 last 25.41) - open/close 49783 (time 1709788667.67 total 1837.48 last 25.39) total: 50000 open/close in 1846.58 seconds: 27.08 ops/second - unlinked 0 (time 1709788678 ; total 0 ; last 0) - unlinked 10000 (time 1709789154 ; total 476 ; last 476) - unlinked 20000 (time 1709789580 ; total 902 ; last 426) - unlinked 30000 (time 1709789918 ; total 1240 ; last 338) - unlinked 40000 (time 1709790119 ; total 1441 ; last 201) total: 50000 unlinks in 1506 seconds: 33.200531 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2