== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:18 (1709786778) start mds service on oleg441-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg441-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 300 (time 1709786820.56 total 10.00 last 29.99) - open/close 540 (time 1709786830.87 total 20.31 last 23.28) - open/close 681 (time 1709786840.87 total 30.32 last 14.09) - open/close 1013 (time 1709786850.88 total 40.32 last 33.19) - open/close 1340 (time 1709786861.61 total 51.05 last 30.46) - open/close 1658 (time 1709786871.61 total 61.06 last 31.78) - open/close 1881 (time 1709786881.62 total 71.07 last 22.28) - open/close 2061 (time 1709786891.89 total 81.33 last 17.53) - open/close 2303 (time 1709786901.92 total 91.36 last 24.13) - open/close 2603 (time 1709786912.28 total 101.72 last 28.96) - open/close 2750 (time 1709786922.76 total 112.21 last 14.02) - open/close 3013 (time 1709786932.77 total 122.22 last 26.26) - open/close 3262 (time 1709786942.78 total 132.22 last 24.89) - open/close 3475 (time 1709786953.62 total 143.06 last 19.65) - open/close 3558 (time 1709786964.37 total 153.81 last 7.72) - open/close 3708 (time 1709786975.06 total 164.50 last 14.03) - open/close 3885 (time 1709786985.39 total 174.84 last 17.13) - open/close 4050 (time 1709786996.18 total 185.63 last 15.29) - open/close 4242 (time 1709787008.37 total 197.81 last 15.76) - open/close 4515 (time 1709787018.61 total 208.05 last 26.65) - open/close 4817 (time 1709787029.46 total 218.90 last 27.84) - open/close 4984 (time 1709787040.46 total 229.91 last 15.17) - open/close 5214 (time 1709787050.74 total 240.18 last 22.38) - open/close 5412 (time 1709787062.88 total 252.33 last 16.31) - open/close 5551 (time 1709787073.32 total 262.77 last 13.31) - open/close 5736 (time 1709787083.34 total 272.78 last 18.47) - open/close 5953 (time 1709787093.41 total 282.85 last 21.55) - open/close 6241 (time 1709787103.55 total 293.00 last 28.39) - open/close 6405 (time 1709787114.17 total 303.62 last 15.44) - open/close 6604 (time 1709787124.51 total 313.96 last 19.24) - open/close 6762 (time 1709787134.52 total 323.96 last 15.79) - open/close 7000 (time 1709787146.40 total 335.85 last 20.03) - open/close 7247 (time 1709787157.00 total 346.45 last 23.29) - open/close 7388 (time 1709787167.01 total 356.46 last 14.09) - open/close 7667 (time 1709787177.14 total 366.58 last 27.54) - open/close 7911 (time 1709787187.93 total 377.38 last 22.61) - open/close 8060 (time 1709787198.17 total 387.62 last 14.55) - open/close 8062 (time 1709787209.31 total 398.75 last 0.18) - open/close 8182 (time 1709787222.22 total 411.67 last 9.29) - open/close 8359 (time 1709787232.67 total 422.12 last 16.94) - open/close 8491 (time 1709787244.40 total 433.85 last 11.25) - open/close 8672 (time 1709787255.44 total 444.89 last 16.40) - open/close 8842 (time 1709787267.11 total 456.55 last 14.57) - open/close 8976 (time 1709787277.93 total 467.37 last 12.38) - open/close 9132 (time 1709787287.94 total 477.38 last 15.58) - open/close 9274 (time 1709787297.95 total 487.39 last 14.19) - open/close 9736 (time 1709787307.96 total 497.40 last 46.15) - open/close 10000 (time 1709787312.77 total 502.21 last 54.91) - open/close 10515 (time 1709787322.78 total 512.23 last 51.44) - open/close 11022 (time 1709787332.79 total 522.24 last 50.64) - open/close 11528 (time 1709787342.81 total 532.26 last 50.51) - open/close 11985 (time 1709787352.84 total 542.28 last 45.58) - open/close 12436 (time 1709787362.88 total 552.32 last 44.92) - open/close 12900 (time 1709787372.89 total 562.33 last 46.37) - open/close 13330 (time 1709787382.91 total 572.36 last 42.89) - open/close 13730 (time 1709787392.92 total 582.36 last 39.98) - open/close 14107 (time 1709787402.93 total 592.37 last 37.66) - open/close 14476 (time 1709787412.97 total 602.41 last 36.76) - open/close 14817 (time 1709787422.99 total 612.43 last 34.02) - open/close 15154 (time 1709787432.99 total 622.44 last 33.68) - open/close 15509 (time 1709787443.03 total 632.47 last 35.39) - open/close 15856 (time 1709787453.03 total 642.47 last 34.69) - open/close 16215 (time 1709787463.05 total 652.49 last 35.83) - open/close 16553 (time 1709787473.07 total 662.51 last 33.73) - open/close 16909 (time 1709787483.09 total 672.53 last 35.54) - open/close 17240 (time 1709787493.12 total 682.56 last 33.00) - open/close 17550 (time 1709787503.14 total 692.58 last 30.94) - open/close 17837 (time 1709787513.18 total 702.62 last 28.58) - open/close 18032 (time 1709787530.64 total 720.09 last 11.17) - open/close 18337 (time 1709787540.65 total 730.09 last 30.49) - open/close 18682 (time 1709787550.66 total 740.10 last 34.46) - open/close 18992 (time 1709787560.66 total 750.11 last 30.98) - open/close 19360 (time 1709787570.70 total 760.14 last 36.68) - open/close 19678 (time 1709787580.71 total 770.15 last 31.76) - open/close 20000 (time 1709787589.79 total 779.24 last 35.44) - open/close 20360 (time 1709787599.80 total 789.24 last 35.98) - open/close 20706 (time 1709787609.81 total 799.26 last 34.55) - open/close 21034 (time 1709787619.82 total 809.26 last 32.78) - open/close 21366 (time 1709787629.84 total 819.28 last 33.14) - open/close 21687 (time 1709787639.85 total 829.29 last 32.07) - open/close 22012 (time 1709787649.86 total 839.30 last 32.46) - open/close 22354 (time 1709787659.96 total 849.40 last 33.87) - open/close 22712 (time 1709787669.97 total 859.42 last 35.74) - open/close 23007 (time 1709787680.00 total 869.44 last 29.44) - open/close 23335 (time 1709787690.01 total 879.46 last 32.75) - open/close 23660 (time 1709787700.03 total 889.48 last 32.43) - open/close 23950 (time 1709787710.04 total 899.49 last 28.97) - open/close 24209 (time 1709787720.06 total 909.50 last 25.86) - open/close 24522 (time 1709787730.08 total 919.53 last 31.22) - open/close 24839 (time 1709787740.09 total 929.54 last 31.67) - open/close 25158 (time 1709787750.10 total 939.55 last 31.88) - open/close 25433 (time 1709787760.13 total 949.58 last 27.41) - open/close 25689 (time 1709787770.21 total 959.65 last 25.41) - open/close 25924 (time 1709787780.22 total 969.66 last 23.47) - open/close 26149 (time 1709787790.23 total 979.67 last 22.49) - open/close 26398 (time 1709787800.25 total 989.69 last 24.84) - open/close 26687 (time 1709787810.25 total 999.70 last 28.89) - open/close 26961 (time 1709787820.30 total 1009.74 last 27.29) - open/close 27209 (time 1709787830.31 total 1019.75 last 24.77) - open/close 27488 (time 1709787840.32 total 1029.76 last 27.87) - open/close 27779 (time 1709787850.35 total 1039.79 last 29.01) - open/close 28079 (time 1709787860.38 total 1049.82 last 29.90) - open/close 28386 (time 1709787870.39 total 1059.83 last 30.67) - open/close 28692 (time 1709787880.41 total 1069.86 last 30.53) - open/close 29004 (time 1709787890.42 total 1079.86 last 31.18) - open/close 29314 (time 1709787900.44 total 1089.88 last 30.93) - open/close 29602 (time 1709787910.46 total 1099.90 last 28.74) - open/close 29886 (time 1709787920.50 total 1109.94 last 28.30) - open/close 30000 (time 1709787924.54 total 1113.99 last 28.16) - open/close 30293 (time 1709787934.56 total 1124.00 last 29.27) - open/close 30583 (time 1709787944.58 total 1134.03 last 28.92) - open/close 30860 (time 1709787954.62 total 1144.07 last 27.59) - open/close 31132 (time 1709787964.65 total 1154.10 last 27.11) - open/close 31396 (time 1709787974.69 total 1164.14 last 26.29) - open/close 31664 (time 1709787984.73 total 1174.17 last 26.72) - open/close 31959 (time 1709787994.76 total 1184.21 last 29.39) - open/close 32227 (time 1709788004.80 total 1194.24 last 26.70) - open/close 32494 (time 1709788014.81 total 1204.25 last 26.68) - open/close 32762 (time 1709788024.84 total 1214.28 last 26.72) - open/close 33076 (time 1709788034.86 total 1224.31 last 31.32) - open/close 33406 (time 1709788044.87 total 1234.31 last 32.99) - open/close 33719 (time 1709788054.90 total 1244.34 last 31.21) - open/close 34057 (time 1709788064.91 total 1254.36 last 33.74) - open/close 34414 (time 1709788074.92 total 1264.36 last 35.68) - open/close 34760 (time 1709788084.93 total 1274.37 last 34.57) - open/close 35103 (time 1709788094.94 total 1284.38 last 34.27) - open/close 35442 (time 1709788104.94 total 1294.39 last 33.87) - open/close 35752 (time 1709788114.97 total 1304.42 last 30.91) - open/close 36103 (time 1709788124.98 total 1314.43 last 35.07) - open/close 36378 (time 1709788135.01 total 1324.45 last 27.42) - open/close 36665 (time 1709788145.01 total 1334.46 last 28.70) - open/close 36925 (time 1709788155.04 total 1344.49 last 25.92) - open/close 37167 (time 1709788165.06 total 1354.51 last 24.15) - open/close 37431 (time 1709788175.09 total 1364.54 last 26.32) - open/close 37733 (time 1709788185.11 total 1374.56 last 30.14) - open/close 38020 (time 1709788195.12 total 1384.56 last 28.69) - open/close 38287 (time 1709788205.14 total 1394.59 last 26.63) - open/close 38591 (time 1709788215.16 total 1404.61 last 30.33) - open/close 38890 (time 1709788225.18 total 1414.62 last 29.85) - open/close 39193 (time 1709788235.19 total 1424.63 last 30.28) - open/close 39467 (time 1709788245.19 total 1434.64 last 27.38) - open/close 39777 (time 1709788255.21 total 1444.66 last 30.94) - open/close 40000 (time 1709788261.99 total 1451.44 last 32.88) - open/close 40313 (time 1709788272.02 total 1461.47 last 31.21) - open/close 40586 (time 1709788282.03 total 1471.47 last 27.28) - open/close 40877 (time 1709788292.06 total 1481.51 last 29.00) - open/close 41180 (time 1709788302.09 total 1491.53 last 30.22) - open/close 41468 (time 1709788312.12 total 1501.56 last 28.71) - open/close 41752 (time 1709788322.13 total 1511.57 last 28.38) - open/close 42057 (time 1709788332.15 total 1521.60 last 30.42) - open/close 42358 (time 1709788342.18 total 1531.63 last 30.01) - open/close 42648 (time 1709788352.19 total 1541.63 last 28.98) - open/close 42917 (time 1709788362.22 total 1551.66 last 26.83) - open/close 43220 (time 1709788372.23 total 1561.67 last 30.27) - open/close 43519 (time 1709788382.24 total 1571.68 last 29.87) - open/close 43818 (time 1709788392.24 total 1581.68 last 29.89) - open/close 44117 (time 1709788402.25 total 1591.69 last 29.88) - open/close 44392 (time 1709788412.27 total 1601.71 last 27.44) - open/close 44637 (time 1709788422.28 total 1611.73 last 24.46) - open/close 44882 (time 1709788432.29 total 1621.73 last 24.49) - open/close 45138 (time 1709788442.31 total 1631.75 last 25.56) - open/close 45401 (time 1709788452.31 total 1641.75 last 26.30) - open/close 45661 (time 1709788462.33 total 1651.77 last 25.94) - open/close 45936 (time 1709788472.34 total 1661.78 last 27.48) - open/close 46184 (time 1709788482.36 total 1671.80 last 24.75) - open/close 46437 (time 1709788492.36 total 1681.80 last 25.30) - open/close 46701 (time 1709788502.36 total 1691.80 last 26.40) - open/close 46986 (time 1709788512.36 total 1701.81 last 28.49) - open/close 47247 (time 1709788522.45 total 1711.89 last 25.88) - open/close 47495 (time 1709788532.46 total 1721.90 last 24.77) - open/close 47767 (time 1709788542.48 total 1731.93 last 27.13) - open/close 48043 (time 1709788552.50 total 1741.95 last 27.55) - open/close 48285 (time 1709788562.53 total 1751.97 last 24.14) - open/close 48546 (time 1709788572.55 total 1762.00 last 26.03) - open/close 48801 (time 1709788582.57 total 1772.02 last 25.44) - open/close 49066 (time 1709788592.61 total 1782.05 last 26.41) - open/close 49341 (time 1709788602.62 total 1792.06 last 27.47) - open/close 49596 (time 1709788612.63 total 1802.08 last 25.46) - open/close 49855 (time 1709788622.66 total 1812.11 last 25.82) total: 50000 open/close in 1817.30 seconds: 27.51 ops/second - unlinked 0 (time 1709788629 ; total 0 ; last 0) - unlinked 10000 (time 1709789081 ; total 452 ; last 452) - unlinked 20000 (time 1709789443 ; total 814 ; last 362) - unlinked 30000 (time 1709789697 ; total 1068 ; last 254) - unlinked 40000 (time 1709789817 ; total 1188 ; last 120) total: 50000 unlinks in 1242 seconds: 40.257648 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server unloading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2