== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:14 (1709787134) start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1709787165.30 total 10.51 last 17.22) - open/close 433 (time 1709787175.65 total 20.85 last 24.36) - open/close 692 (time 1709787185.87 total 31.08 last 25.33) - open/close 873 (time 1709787196.30 total 41.51 last 17.36) - open/close 1119 (time 1709787207.08 total 52.29 last 22.82) - open/close 1411 (time 1709787217.08 total 62.29 last 29.19) - open/close 1559 (time 1709787228.11 total 73.31 last 13.43) - open/close 1786 (time 1709787239.13 total 84.34 last 20.59) - open/close 1898 (time 1709787250.03 total 95.24 last 10.27) - open/close 2111 (time 1709787260.11 total 105.31 last 21.14) - open/close 2410 (time 1709787270.11 total 115.32 last 29.89) - open/close 2573 (time 1709787280.90 total 126.10 last 15.11) - open/close 2729 (time 1709787291.73 total 136.93 last 14.41) - open/close 2868 (time 1709787302.10 total 147.31 last 13.39) - open/close 3053 (time 1709787312.47 total 157.68 last 17.84) - open/close 3267 (time 1709787322.49 total 167.70 last 21.35) - open/close 3501 (time 1709787333.30 total 178.51 last 21.65) - open/close 3734 (time 1709787343.31 total 188.51 last 23.29) - open/close 3896 (time 1709787353.81 total 199.02 last 15.42) - open/close 4112 (time 1709787364.02 total 209.22 last 21.17) - open/close 4280 (time 1709787374.03 total 219.24 last 16.78) - open/close 4451 (time 1709787384.15 total 229.35 last 16.90) - open/close 4680 (time 1709787394.54 total 239.75 last 22.04) - open/close 4924 (time 1709787404.55 total 249.76 last 24.37) - open/close 5132 (time 1709787414.56 total 259.76 last 20.79) - open/close 5298 (time 1709787427.09 total 272.29 last 13.25) - open/close 5521 (time 1709787437.69 total 282.89 last 21.03) - open/close 5622 (time 1709787447.71 total 292.92 last 10.08) - open/close 5756 (time 1709787457.72 total 302.93 last 13.38) - open/close 5985 (time 1709787467.84 total 313.05 last 22.63) - open/close 6176 (time 1709787477.94 total 323.15 last 18.91) - open/close 6339 (time 1709787488.13 total 333.33 last 16.00) - open/close 6500 (time 1709787498.15 total 343.36 last 16.06) - open/close 6710 (time 1709787508.16 total 353.37 last 20.98) - open/close 6917 (time 1709787518.16 total 363.37 last 20.69) - open/close 7108 (time 1709787528.18 total 373.38 last 19.08) - open/close 7231 (time 1709787538.91 total 384.11 last 11.46) - open/close 7339 (time 1709787548.92 total 394.12 last 10.79) - open/close 7513 (time 1709787559.81 total 405.01 last 15.98) - open/close 7812 (time 1709787569.82 total 415.02 last 29.87) - open/close 7978 (time 1709787580.84 total 426.05 last 15.06) - open/close 8207 (time 1709787591.33 total 436.53 last 21.84) - open/close 8442 (time 1709787601.34 total 446.55 last 23.46) - open/close 8628 (time 1709787611.49 total 456.70 last 18.33) - open/close 8851 (time 1709787622.96 total 468.17 last 19.44) - open/close 9180 (time 1709787633.79 total 478.99 last 30.39) - open/close 9289 (time 1709787643.80 total 489.00 last 10.89) - open/close 9525 (time 1709787654.75 total 499.96 last 21.54) - open/close 9694 (time 1709787665.75 total 510.95 last 15.37) - open/close 9876 (time 1709787676.37 total 521.58 last 17.13) - open/close 10000 (time 1709787682.24 total 527.45 last 21.14) - open/close 10581 (time 1709787692.24 total 537.45 last 58.08) - open/close 11169 (time 1709787702.25 total 547.46 last 58.75) - open/close 11715 (time 1709787712.26 total 557.47 last 54.53) - open/close 12245 (time 1709787722.27 total 567.48 last 52.96) - open/close 12822 (time 1709787732.29 total 577.49 last 57.61) - open/close 13424 (time 1709787742.29 total 587.50 last 60.17) - open/close 13992 (time 1709787752.30 total 597.50 last 56.76) - open/close 14468 (time 1709787762.31 total 607.51 last 47.57) - open/close 14947 (time 1709787772.34 total 617.55 last 47.72) - open/close 15398 (time 1709787782.38 total 627.58 last 44.95) - open/close 15846 (time 1709787792.40 total 637.60 last 44.71) - open/close 16309 (time 1709787802.41 total 647.61 last 46.25) - open/close 16777 (time 1709787812.43 total 657.63 last 46.72) - open/close 17200 (time 1709787822.44 total 667.65 last 42.23) - open/close 17580 (time 1709787832.45 total 677.66 last 37.96) - open/close 18023 (time 1709787842.46 total 687.66 last 44.29) - open/close 18459 (time 1709787852.47 total 697.68 last 43.52) - open/close 18930 (time 1709787862.48 total 707.69 last 47.07) - open/close 19385 (time 1709787872.49 total 717.69 last 45.47) - open/close 19858 (time 1709787882.50 total 727.70 last 47.25) - open/close 20000 (time 1709787885.48 total 730.69 last 47.61) - open/close 20435 (time 1709787895.50 total 740.70 last 43.43) - open/close 20868 (time 1709787905.52 total 750.72 last 43.22) - open/close 21262 (time 1709787915.54 total 760.75 last 39.29) - open/close 21681 (time 1709787925.55 total 770.76 last 41.85) - open/close 22091 (time 1709787935.57 total 780.78 last 40.94) - open/close 22478 (time 1709787945.58 total 790.79 last 38.65) - open/close 22878 (time 1709787955.59 total 800.80 last 39.96) - open/close 23268 (time 1709787965.60 total 810.80 last 38.98) - open/close 23649 (time 1709787975.63 total 820.83 last 37.98) - open/close 24019 (time 1709787985.65 total 830.85 last 36.94) - open/close 24411 (time 1709787995.67 total 840.87 last 39.11) - open/close 24818 (time 1709788005.69 total 850.89 last 40.63) - open/close 25212 (time 1709788015.71 total 860.91 last 39.32) - open/close 25578 (time 1709788025.71 total 870.92 last 36.58) - open/close 25908 (time 1709788035.74 total 880.94 last 32.91) - open/close 26248 (time 1709788045.74 total 890.95 last 33.98) - open/close 26592 (time 1709788055.76 total 900.96 last 34.36) - open/close 26935 (time 1709788065.76 total 910.96 last 34.29) - open/close 27292 (time 1709788075.77 total 920.98 last 35.65) - open/close 27641 (time 1709788085.78 total 930.98 last 34.89) - open/close 27999 (time 1709788095.80 total 941.01 last 35.71) - open/close 28349 (time 1709788105.86 total 951.06 last 34.80) - open/close 28687 (time 1709788115.88 total 961.08 last 33.74) - open/close 29034 (time 1709788125.88 total 971.09 last 34.67) - open/close 29346 (time 1709788135.91 total 981.12 last 31.12) - open/close 29668 (time 1709788145.92 total 991.12 last 32.18) - open/close 29978 (time 1709788155.93 total 1001.13 last 30.97) - open/close 30262 (time 1709788165.93 total 1011.14 last 28.39) - open/close 30565 (time 1709788175.95 total 1021.15 last 30.25) - open/close 30895 (time 1709788185.98 total 1031.18 last 32.90) - open/close 31223 (time 1709788196.01 total 1041.22 last 32.70) - open/close 31517 (time 1709788206.02 total 1051.23 last 29.36) - open/close 31820 (time 1709788216.03 total 1061.23 last 30.29) - open/close 32105 (time 1709788226.05 total 1071.25 last 28.45) - open/close 32407 (time 1709788236.07 total 1081.28 last 30.11) - open/close 32679 (time 1709788246.08 total 1091.28 last 27.19) - open/close 32975 (time 1709788256.12 total 1101.32 last 29.48) - open/close 33293 (time 1709788266.12 total 1111.33 last 31.78) - open/close 33578 (time 1709788276.13 total 1121.34 last 28.48) - open/close 33882 (time 1709788286.16 total 1131.37 last 30.30) - open/close 34189 (time 1709788296.18 total 1141.38 last 30.65) - open/close 34487 (time 1709788306.18 total 1151.38 last 29.80) - open/close 34776 (time 1709788316.19 total 1161.39 last 28.88) - open/close 35060 (time 1709788326.20 total 1171.41 last 28.36) - open/close 35349 (time 1709788336.23 total 1181.43 last 28.82) - open/close 35620 (time 1709788346.26 total 1191.46 last 27.02) - open/close 35885 (time 1709788356.26 total 1201.47 last 26.49) - open/close 36133 (time 1709788366.29 total 1211.49 last 24.74) - open/close 36399 (time 1709788376.29 total 1221.50 last 26.59) - open/close 36666 (time 1709788386.32 total 1231.53 last 26.62) - open/close 36930 (time 1709788396.34 total 1241.54 last 26.35) - open/close 37202 (time 1709788406.35 total 1251.56 last 27.16) - open/close 37469 (time 1709788416.39 total 1261.59 last 26.61) - open/close 37723 (time 1709788426.42 total 1271.62 last 25.33) - open/close 37997 (time 1709788436.44 total 1281.65 last 27.33) - open/close 38258 (time 1709788446.45 total 1291.65 last 26.08) - open/close 38528 (time 1709788456.46 total 1301.67 last 26.96) - open/close 38797 (time 1709788466.49 total 1311.69 last 26.84) - open/close 39065 (time 1709788476.49 total 1321.69 last 26.79) - open/close 39324 (time 1709788486.51 total 1331.72 last 25.84) - open/close 39592 (time 1709788496.53 total 1341.73 last 26.76) - open/close 39668 (time 1709788510.53 total 1355.73 last 5.43) - open/close 39929 (time 1709788520.55 total 1365.76 last 26.03) - open/close 40000 (time 1709788523.69 total 1368.89 last 22.66) - open/close 40260 (time 1709788533.73 total 1378.93 last 25.90) - open/close 40517 (time 1709788543.74 total 1388.95 last 25.66) - open/close 40780 (time 1709788553.77 total 1398.98 last 26.22) - open/close 41037 (time 1709788563.78 total 1408.99 last 25.68) - open/close 41294 (time 1709788573.82 total 1419.02 last 25.61) - open/close 41553 (time 1709788583.84 total 1429.04 last 25.84) - open/close 41821 (time 1709788593.85 total 1439.05 last 26.77) - open/close 42087 (time 1709788603.85 total 1449.06 last 26.58) - open/close 42342 (time 1709788613.88 total 1459.08 last 25.45) - open/close 42607 (time 1709788623.88 total 1469.09 last 26.48) - open/close 42881 (time 1709788633.90 total 1479.11 last 27.34) - open/close 43153 (time 1709788643.93 total 1489.14 last 27.12) - open/close 43419 (time 1709788653.95 total 1499.16 last 26.55) - open/close 43682 (time 1709788663.97 total 1509.18 last 26.25) - open/close 43940 (time 1709788674.00 total 1519.21 last 25.72) - open/close 44205 (time 1709788684.03 total 1529.24 last 26.43) - open/close 44480 (time 1709788694.05 total 1539.25 last 27.46) - open/close 44740 (time 1709788704.05 total 1549.25 last 26.00) - open/close 44989 (time 1709788714.09 total 1559.29 last 24.80) - open/close 45221 (time 1709788724.10 total 1569.30 last 23.17) - open/close 45467 (time 1709788734.10 total 1579.30 last 24.60) - open/close 45711 (time 1709788744.13 total 1589.33 last 24.33) - open/close 45944 (time 1709788754.14 total 1599.34 last 23.28) - open/close 46188 (time 1709788764.16 total 1609.36 last 24.35) - open/close 46410 (time 1709788774.17 total 1619.37 last 22.19) - open/close 46634 (time 1709788784.17 total 1629.37 last 22.39) - open/close 46869 (time 1709788794.19 total 1639.40 last 23.44) - open/close 47095 (time 1709788804.20 total 1649.40 last 22.59) - open/close 47322 (time 1709788814.22 total 1659.42 last 22.66) - open/close 47548 (time 1709788824.24 total 1669.45 last 22.54) - open/close 47780 (time 1709788834.26 total 1679.46 last 23.16) - open/close 48006 (time 1709788844.29 total 1689.50 last 22.52) - open/close 48217 (time 1709788854.31 total 1699.51 last 21.07) - open/close 48445 (time 1709788864.33 total 1709.53 last 22.75) - open/close 48676 (time 1709788874.36 total 1719.56 last 23.03) - open/close 48900 (time 1709788884.40 total 1729.60 last 22.32) - open/close 49105 (time 1709788894.45 total 1739.66 last 20.39) - open/close 49311 (time 1709788904.48 total 1749.69 last 20.53) - open/close 49524 (time 1709788914.49 total 1759.70 last 21.28) - open/close 49725 (time 1709788924.49 total 1769.70 last 20.09) - open/close 49931 (time 1709788934.53 total 1779.73 last 20.53) total: 50000 open/close in 1782.99 seconds: 28.04 ops/second - unlinked 0 (time 1709788939 ; total 0 ; last 0) - unlinked 10000 (time 1709789458 ; total 519 ; last 519) - unlinked 20000 (time 1709789880 ; total 941 ; last 422) - unlinked 30000 (time 1709790158 ; total 1219 ; last 278) - unlinked 40000 (time 1709790261 ; total 1322 ; last 103) total: 50000 unlinks in 1357 seconds: 36.845985 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2