== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:51 (1709787051) start mds service on oleg119-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg119-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg119-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock oleg119-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1709787102.14 total 10.01 last 24.79) - open/close 412 (time 1709787123.09 total 30.95 last 7.83) - open/close 597 (time 1709787133.10 total 40.96 last 18.48) - open/close 809 (time 1709787143.12 total 50.98 last 21.16) - open/close 1087 (time 1709787154.07 total 61.93 last 25.39) - open/close 1280 (time 1709787164.07 total 71.94 last 19.29) - open/close 1406 (time 1709787174.07 total 81.94 last 12.60) - open/close 1607 (time 1709787184.76 total 92.63 last 18.81) - open/close 1853 (time 1709787195.55 total 103.42 last 22.80) - open/close 2063 (time 1709787205.78 total 113.65 last 20.53) - open/close 2259 (time 1709787216.58 total 124.44 last 18.16) - open/close 2384 (time 1709787226.59 total 134.45 last 12.49) - open/close 2565 (time 1709787236.59 total 144.46 last 18.09) - open/close 2686 (time 1709787247.47 total 155.33 last 11.13) - open/close 2873 (time 1709787258.25 total 166.12 last 17.34) - open/close 3067 (time 1709787269.00 total 176.87 last 18.05) - open/close 3343 (time 1709787279.01 total 186.88 last 27.58) - open/close 3537 (time 1709787289.03 total 196.89 last 19.37) - open/close 3735 (time 1709787300.04 total 207.90 last 17.98) - open/close 3974 (time 1709787310.48 total 218.34 last 22.89) - open/close 4152 (time 1709787320.86 total 228.72 last 17.15) - open/close 4155 (time 1709787331.94 total 239.80 last 0.27) - open/close 4341 (time 1709787342.59 total 250.45 last 17.47) - open/close 4551 (time 1709787353.49 total 261.35 last 19.27) - open/close 4727 (time 1709787363.63 total 271.49 last 17.36) - open/close 4931 (time 1709787374.04 total 281.91 last 19.59) - open/close 5071 (time 1709787391.79 total 299.65 last 7.89) - open/close 5230 (time 1709787401.81 total 309.67 last 15.87) - open/close 5390 (time 1709787411.81 total 319.67 last 16.00) - open/close 5504 (time 1709787421.81 total 329.68 last 11.39) - open/close 5718 (time 1709787432.83 total 340.69 last 19.42) - open/close 5887 (time 1709787443.32 total 351.18 last 16.12) - open/close 5998 (time 1709787453.51 total 361.37 last 10.89) - open/close 6153 (time 1709787463.72 total 371.58 last 15.18) - open/close 6295 (time 1709787473.72 total 381.58 last 14.19) - open/close 6458 (time 1709787484.36 total 392.23 last 15.32) - open/close 6683 (time 1709787494.36 total 402.23 last 22.49) - open/close 6858 (time 1709787504.51 total 412.37 last 17.25) - open/close 7011 (time 1709787514.62 total 422.49 last 15.13) - open/close 7175 (time 1709787524.98 total 432.85 last 15.83) - open/close 7324 (time 1709787535.52 total 443.39 last 14.14) - open/close 7418 (time 1709787545.53 total 453.39 last 9.39) - open/close 7550 (time 1709787558.60 total 466.46 last 10.10) - open/close 7754 (time 1709787568.61 total 476.48 last 20.37) - open/close 7916 (time 1709787579.15 total 487.02 last 15.37) - open/close 8045 (time 1709787589.87 total 497.73 last 12.04) - open/close 8198 (time 1709787600.47 total 508.33 last 14.43) - open/close 8330 (time 1709787611.19 total 519.06 last 12.31) - open/close 8524 (time 1709787621.20 total 529.07 last 19.38) - open/close 8723 (time 1709787632.21 total 540.08 last 18.07) - open/close 8946 (time 1709787642.22 total 550.08 last 22.29) - open/close 9166 (time 1709787652.23 total 560.09 last 21.98) - open/close 9336 (time 1709787662.25 total 570.12 last 16.96) - open/close 9551 (time 1709787672.27 total 580.13 last 21.47) - open/close 10000 (time 1709787679.43 total 587.29 last 62.73) - open/close 10577 (time 1709787689.44 total 597.30 last 57.63) - open/close 11108 (time 1709787699.44 total 607.30 last 53.09) - open/close 11717 (time 1709787709.45 total 617.32 last 60.81) - open/close 12310 (time 1709787719.46 total 627.32 last 59.28) - open/close 12889 (time 1709787740.03 total 647.89 last 28.15) - open/close 13385 (time 1709787750.04 total 657.90 last 49.55) - open/close 13906 (time 1709787760.04 total 667.90 last 52.09) - open/close 14357 (time 1709787770.05 total 677.91 last 45.05) - open/close 14870 (time 1709787780.07 total 687.94 last 51.17) - open/close 15311 (time 1709787790.08 total 697.95 last 44.07) - open/close 15731 (time 1709787800.09 total 707.95 last 41.98) - open/close 15811 (time 1709787812.91 total 720.77 last 6.24) - open/close 16301 (time 1709787822.92 total 730.79 last 48.92) - open/close 16721 (time 1709787832.94 total 740.81 last 41.92) - open/close 17107 (time 1709787842.95 total 750.81 last 38.58) - open/close 17510 (time 1709787852.96 total 760.83 last 40.22) - open/close 17908 (time 1709787873.16 total 781.03 last 19.71) - open/close 18311 (time 1709787883.17 total 791.03 last 40.29) - open/close 18716 (time 1709787893.17 total 801.04 last 40.47) - open/close 19100 (time 1709787903.19 total 811.05 last 38.34) - open/close 19464 (time 1709787913.21 total 821.07 last 36.34) - open/close 19831 (time 1709787923.25 total 831.11 last 36.55) - open/close 20000 (time 1709787927.96 total 835.82 last 35.86) - open/close 20357 (time 1709787937.97 total 845.83 last 35.66) - open/close 20724 (time 1709787947.98 total 855.84 last 36.66) - open/close 21155 (time 1709787957.99 total 865.85 last 43.08) - open/close 21539 (time 1709787968.01 total 875.88 last 38.30) - open/close 21929 (time 1709787978.03 total 885.89 last 38.93) - open/close 22336 (time 1709787988.05 total 895.91 last 40.63) - open/close 22761 (time 1709787998.07 total 905.93 last 42.40) - open/close 23147 (time 1709788008.11 total 915.97 last 38.46) - open/close 23540 (time 1709788018.12 total 925.98 last 39.25) - open/close 23926 (time 1709788028.13 total 935.99 last 38.57) - open/close 24325 (time 1709788038.16 total 946.02 last 39.78) - open/close 24713 (time 1709788048.16 total 956.03 last 38.78) - open/close 25103 (time 1709788058.17 total 966.03 last 38.98) - open/close 25475 (time 1709788068.18 total 976.04 last 37.17) - open/close 25845 (time 1709788078.18 total 986.04 last 36.99) - open/close 26217 (time 1709788088.19 total 996.05 last 37.17) - open/close 26591 (time 1709788098.21 total 1006.07 last 37.32) - open/close 26941 (time 1709788108.24 total 1016.10 last 34.89) - open/close 27287 (time 1709788118.26 total 1026.13 last 34.52) - open/close 27646 (time 1709788128.28 total 1036.14 last 35.86) - open/close 28001 (time 1709788138.29 total 1046.15 last 35.45) - open/close 28318 (time 1709788148.30 total 1056.17 last 31.66) - open/close 28657 (time 1709788158.30 total 1066.17 last 33.90) - open/close 28961 (time 1709788168.32 total 1076.18 last 30.35) - open/close 29281 (time 1709788178.33 total 1086.20 last 31.96) - open/close 29595 (time 1709788188.35 total 1096.21 last 31.36) - open/close 29904 (time 1709788198.35 total 1106.22 last 30.88) - open/close 30000 (time 1709788201.24 total 1109.10 last 33.28) - open/close 30334 (time 1709788211.28 total 1119.14 last 33.27) - open/close 30671 (time 1709788221.28 total 1129.14 last 33.70) - open/close 31022 (time 1709788231.29 total 1139.15 last 35.07) - open/close 31362 (time 1709788241.31 total 1149.17 last 33.93) - open/close 31698 (time 1709788251.32 total 1159.18 last 33.57) - open/close 31999 (time 1709788261.35 total 1169.21 last 30.01) - open/close 32310 (time 1709788271.36 total 1179.23 last 31.06) - open/close 32629 (time 1709788281.38 total 1189.25 last 31.83) - open/close 32958 (time 1709788291.39 total 1199.25 last 32.88) - open/close 33268 (time 1709788301.39 total 1209.26 last 30.99) - open/close 33587 (time 1709788311.41 total 1219.28 last 31.83) - open/close 33912 (time 1709788321.44 total 1229.30 last 32.42) - open/close 34225 (time 1709788331.47 total 1239.33 last 31.21) - open/close 34547 (time 1709788341.47 total 1249.34 last 32.19) - open/close 34860 (time 1709788351.49 total 1259.35 last 31.26) - open/close 35155 (time 1709788361.56 total 1269.43 last 29.27) - open/close 35472 (time 1709788371.56 total 1279.43 last 31.69) - open/close 35778 (time 1709788381.57 total 1289.44 last 30.58) - open/close 36062 (time 1709788391.59 total 1299.46 last 28.35) - open/close 36360 (time 1709788401.60 total 1309.47 last 29.76) - open/close 36674 (time 1709788411.63 total 1319.49 last 31.33) - open/close 36963 (time 1709788421.63 total 1329.50 last 28.89) - open/close 37249 (time 1709788431.65 total 1339.52 last 28.53) - open/close 37536 (time 1709788441.68 total 1349.54 last 28.63) - open/close 37818 (time 1709788451.69 total 1359.55 last 28.18) - open/close 38105 (time 1709788461.70 total 1369.56 last 28.67) - open/close 38355 (time 1709788471.72 total 1379.59 last 24.93) - open/close 38635 (time 1709788481.73 total 1389.59 last 27.99) - open/close 38908 (time 1709788491.76 total 1399.62 last 27.21) - open/close 39179 (time 1709788501.77 total 1409.63 last 27.07) - open/close 39455 (time 1709788511.78 total 1419.65 last 27.57) - open/close 39733 (time 1709788521.79 total 1429.65 last 27.79) - open/close 40000 (time 1709788531.54 total 1439.40 last 27.38) - open/close 40273 (time 1709788541.57 total 1449.43 last 27.21) - open/close 40538 (time 1709788551.59 total 1459.45 last 26.45) - open/close 40802 (time 1709788561.60 total 1469.46 last 26.36) - open/close 41071 (time 1709788571.64 total 1479.50 last 26.80) - open/close 41315 (time 1709788581.68 total 1489.54 last 24.30) - open/close 41574 (time 1709788591.69 total 1499.56 last 25.86) - open/close 41839 (time 1709788601.71 total 1509.57 last 26.46) - open/close 42107 (time 1709788611.75 total 1519.61 last 26.70) - open/close 42377 (time 1709788621.77 total 1529.64 last 26.92) - open/close 42656 (time 1709788631.79 total 1539.66 last 27.85) - open/close 42923 (time 1709788641.83 total 1549.69 last 26.61) - open/close 43184 (time 1709788651.86 total 1559.73 last 26.01) - open/close 43437 (time 1709788661.90 total 1569.77 last 25.20) - open/close 43691 (time 1709788671.90 total 1579.77 last 25.40) - open/close 43939 (time 1709788681.93 total 1589.79 last 24.73) - open/close 44187 (time 1709788691.93 total 1599.80 last 24.79) - open/close 44437 (time 1709788701.94 total 1609.81 last 24.97) - open/close 44683 (time 1709788711.95 total 1619.82 last 24.58) - open/close 44939 (time 1709788721.99 total 1629.85 last 25.51) - open/close 45188 (time 1709788732.00 total 1639.86 last 24.88) - open/close 45430 (time 1709788742.03 total 1649.90 last 24.11) - open/close 45653 (time 1709788752.07 total 1659.94 last 22.21) - open/close 45884 (time 1709788762.08 total 1669.95 last 23.07) - open/close 46106 (time 1709788772.09 total 1679.95 last 22.19) - open/close 46330 (time 1709788782.09 total 1689.96 last 22.40) - open/close 46570 (time 1709788792.11 total 1699.97 last 23.96) - open/close 46810 (time 1709788802.12 total 1709.98 last 23.97) - open/close 47047 (time 1709788812.13 total 1720.00 last 23.66) - open/close 47283 (time 1709788822.17 total 1730.03 last 23.53) - open/close 47512 (time 1709788832.19 total 1740.05 last 22.85) - open/close 47747 (time 1709788842.23 total 1750.09 last 23.41) - open/close 47983 (time 1709788852.26 total 1760.13 last 23.52) - open/close 48217 (time 1709788862.29 total 1770.15 last 23.35) - open/close 48456 (time 1709788872.30 total 1780.17 last 23.86) - open/close 48699 (time 1709788882.33 total 1790.19 last 24.24) - open/close 48941 (time 1709788892.33 total 1800.20 last 24.19) - open/close 49182 (time 1709788902.34 total 1810.20 last 24.08) - open/close 49413 (time 1709788912.36 total 1820.23 last 23.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1828.49 seconds: 27.13 ops/second - unlinked 0 (time 1709788921 ; total 0 ; last 0) - unlinked 10000 (time 1709789442 ; total 521 ; last 521) - unlinked 20000 (time 1709789842 ; total 921 ; last 400) - unlinked 30000 (time 1709790126 ; total 1205 ; last 284) - unlinked 40000 (time 1709790257 ; total 1336 ; last 131) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1386 seconds: 35.786434 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2