== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:30 (1709786910) start mds service on oleg217-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg217-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock oleg217-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1709786943.38 total 10.98 last 14.57) - open/close 372 (time 1709786953.84 total 21.43 last 20.27) - open/close 601 (time 1709786964.81 total 32.40 last 20.88) - open/close 831 (time 1709786974.81 total 42.41 last 22.99) - open/close 1078 (time 1709786984.81 total 52.41 last 24.69) - open/close 1211 (time 1709786997.62 total 65.22 last 10.39) - open/close 1390 (time 1709787007.63 total 75.23 last 17.88) - open/close 1541 (time 1709787017.63 total 85.23 last 15.10) - open/close 1694 (time 1709787027.69 total 95.28 last 15.22) - open/close 1910 (time 1709787037.69 total 105.29 last 21.59) - open/close 2139 (time 1709787047.72 total 115.31 last 22.84) - open/close 2313 (time 1709787057.80 total 125.40 last 17.26) - open/close 2455 (time 1709787068.32 total 135.91 last 13.50) - open/close 2703 (time 1709787079.32 total 146.91 last 22.55) - open/close 2902 (time 1709787089.32 total 156.92 last 19.89) - open/close 3160 (time 1709787100.09 total 167.68 last 23.96) - open/close 3398 (time 1709787110.09 total 177.69 last 23.78) - open/close 3536 (time 1709787120.38 total 187.97 last 13.42) - open/close 3701 (time 1709787131.26 total 198.86 last 15.16) - open/close 3783 (time 1709787143.96 total 211.56 last 6.46) - open/close 4063 (time 1709787154.90 total 222.50 last 25.59) - open/close 4277 (time 1709787165.64 total 233.24 last 19.92) - open/close 4476 (time 1709787175.81 total 243.40 last 19.58) - open/close 4602 (time 1709787185.93 total 253.53 last 12.44) - open/close 4780 (time 1709787196.47 total 264.06 last 16.90) - open/close 4974 (time 1709787207.06 total 274.66 last 18.31) - open/close 5161 (time 1709787217.16 total 284.75 last 18.52) - open/close 5403 (time 1709787227.32 total 294.91 last 23.82) - open/close 5663 (time 1709787237.33 total 304.93 last 25.97) - open/close 5851 (time 1709787248.35 total 315.95 last 17.06) - open/close 6019 (time 1709787258.35 total 325.95 last 16.80) - open/close 6222 (time 1709787268.69 total 336.29 last 19.64) - open/close 6434 (time 1709787279.29 total 346.89 last 20.00) - open/close 6572 (time 1709787290.23 total 357.83 last 12.61) - open/close 6737 (time 1709787300.80 total 368.40 last 15.62) - open/close 6959 (time 1709787310.84 total 378.44 last 22.11) - open/close 7226 (time 1709787320.85 total 388.45 last 26.68) - open/close 7329 (time 1709787331.01 total 398.61 last 10.13) - open/close 7523 (time 1709787341.02 total 408.62 last 19.39) - open/close 7675 (time 1709787351.61 total 419.20 last 14.36) - open/close 7809 (time 1709787362.27 total 429.87 last 12.57) - open/close 7977 (time 1709787372.27 total 439.87 last 16.79) - open/close 8155 (time 1709787383.10 total 450.70 last 16.45) - open/close 8370 (time 1709787393.11 total 460.70 last 21.48) - open/close 8504 (time 1709787403.11 total 470.71 last 13.39) - open/close 8633 (time 1709787415.77 total 483.36 last 10.19) - open/close 8747 (time 1709787425.85 total 493.45 last 11.31) - open/close 8851 (time 1709787435.85 total 503.45 last 10.40) - open/close 9025 (time 1709787446.69 total 514.28 last 16.06) - open/close 9202 (time 1709787456.69 total 524.28 last 17.70) - open/close 9349 (time 1709787466.69 total 534.29 last 14.69) - open/close 9490 (time 1709787477.00 total 544.60 last 13.68) - open/close 9710 (time 1709787487.01 total 554.60 last 21.98) - open/close 9835 (time 1709787497.29 total 564.89 last 12.15) - open/close 10000 (time 1709787505.18 total 572.78 last 20.91) - open/close 10301 (time 1709787515.31 total 582.91 last 29.73) - open/close 10768 (time 1709787525.32 total 592.91 last 46.66) - open/close 11184 (time 1709787535.34 total 602.94 last 41.50) - open/close 11638 (time 1709787545.38 total 612.97 last 45.24) - open/close 12015 (time 1709787555.41 total 623.00 last 37.59) - open/close 12146 (time 1709787570.01 total 637.61 last 8.97) - open/close 12596 (time 1709787580.02 total 647.62 last 44.95) - open/close 13012 (time 1709787590.06 total 657.66 last 41.44) - open/close 13457 (time 1709787600.08 total 667.68 last 44.41) - open/close 13872 (time 1709787610.09 total 677.69 last 41.46) - open/close 14249 (time 1709787620.10 total 687.69 last 37.67) - open/close 14621 (time 1709787630.12 total 697.71 last 37.14) - open/close 15039 (time 1709787640.12 total 707.72 last 41.78) - open/close 15429 (time 1709787650.14 total 717.74 last 38.91) - open/close 15784 (time 1709787660.14 total 727.74 last 35.50) - open/close 16159 (time 1709787670.15 total 737.74 last 37.49) - open/close 16501 (time 1709787680.18 total 747.77 last 34.10) - open/close 16875 (time 1709787690.20 total 757.80 last 37.32) - open/close 17236 (time 1709787700.20 total 767.80 last 36.09) - open/close 17594 (time 1709787710.21 total 777.81 last 35.78) - open/close 17920 (time 1709787720.21 total 787.81 last 32.58) - open/close 18248 (time 1709787730.23 total 797.83 last 32.75) - open/close 18591 (time 1709787740.25 total 807.85 last 34.23) - open/close 18885 (time 1709787750.30 total 817.90 last 29.26) - open/close 19213 (time 1709787760.30 total 827.90 last 32.79) - open/close 19547 (time 1709787770.36 total 837.96 last 33.21) - open/close 19884 (time 1709787780.37 total 847.97 last 33.66) - open/close 20000 (time 1709787784.25 total 851.84 last 29.94) - open/close 20339 (time 1709787794.26 total 861.85 last 33.87) - open/close 20456 (time 1709787808.52 total 876.12 last 8.20) - open/close 20768 (time 1709787818.56 total 886.16 last 31.07) - open/close 21110 (time 1709787828.57 total 896.17 last 34.16) - open/close 21429 (time 1709787838.60 total 906.20 last 31.81) - open/close 21724 (time 1709787848.63 total 916.23 last 29.40) - open/close 21980 (time 1709787858.69 total 926.28 last 25.46) - open/close 22261 (time 1709787868.72 total 936.31 last 28.02) - open/close 22400 (time 1709787884.43 total 952.02 last 8.85) - open/close 22680 (time 1709787894.49 total 962.08 last 27.83) - open/close 22935 (time 1709787904.50 total 972.09 last 25.48) - open/close 23210 (time 1709787914.52 total 982.12 last 27.43) - open/close 23503 (time 1709787924.56 total 992.16 last 29.18) - open/close 23792 (time 1709787934.58 total 1002.18 last 28.84) - open/close 24081 (time 1709787944.60 total 1012.20 last 28.83) - open/close 24427 (time 1709787954.61 total 1022.21 last 34.58) - open/close 24761 (time 1709787964.63 total 1032.22 last 33.34) - open/close 25086 (time 1709787974.65 total 1042.25 last 32.42) - open/close 25391 (time 1709787984.66 total 1052.26 last 30.47) - open/close 25730 (time 1709787994.69 total 1062.29 last 33.80) - open/close 26119 (time 1709788004.69 total 1072.29 last 38.89) - open/close 26505 (time 1709788014.69 total 1082.29 last 38.60) - open/close 26923 (time 1709788024.71 total 1092.30 last 41.75) - open/close 27338 (time 1709788034.71 total 1102.31 last 41.46) - open/close 27773 (time 1709788044.73 total 1112.33 last 43.43) - open/close 28189 (time 1709788054.74 total 1122.34 last 41.56) - open/close 28581 (time 1709788064.74 total 1132.34 last 39.20) - open/close 28952 (time 1709788074.76 total 1142.35 last 37.04) - open/close 29360 (time 1709788084.77 total 1152.36 last 40.76) - open/close 29747 (time 1709788094.78 total 1162.38 last 38.65) - open/close 30000 (time 1709788101.67 total 1169.27 last 36.71) - open/close 30376 (time 1709788111.68 total 1179.28 last 37.57) - open/close 30751 (time 1709788121.68 total 1189.28 last 37.49) - open/close 31115 (time 1709788131.69 total 1199.29 last 36.37) - open/close 31487 (time 1709788141.70 total 1209.29 last 37.18) - open/close 31863 (time 1709788151.70 total 1219.30 last 37.57) - open/close 32239 (time 1709788161.72 total 1229.31 last 37.55) - open/close 32621 (time 1709788171.74 total 1239.33 last 38.13) - open/close 32993 (time 1709788181.74 total 1249.33 last 37.19) - open/close 33360 (time 1709788191.74 total 1259.34 last 36.70) - open/close 33725 (time 1709788201.75 total 1269.34 last 36.48) - open/close 34088 (time 1709788211.76 total 1279.36 last 36.24) - open/close 34459 (time 1709788221.77 total 1289.36 last 37.08) - open/close 34820 (time 1709788231.78 total 1299.38 last 36.04) - open/close 35168 (time 1709788241.81 total 1309.40 last 34.72) - open/close 35525 (time 1709788251.82 total 1319.42 last 35.64) - open/close 35871 (time 1709788261.83 total 1329.42 last 34.58) - open/close 36203 (time 1709788271.84 total 1339.44 last 33.16) - open/close 36528 (time 1709788281.85 total 1349.44 last 32.48) - open/close 36835 (time 1709788291.87 total 1359.47 last 30.62) - open/close 37154 (time 1709788301.90 total 1369.50 last 31.81) - open/close 37474 (time 1709788311.91 total 1379.51 last 31.97) - open/close 37787 (time 1709788321.94 total 1389.53 last 31.22) - open/close 38093 (time 1709788331.95 total 1399.54 last 30.57) - open/close 38397 (time 1709788341.98 total 1409.57 last 30.31) - open/close 38700 (time 1709788351.98 total 1419.57 last 30.30) - open/close 38997 (time 1709788361.98 total 1429.58 last 29.69) - open/close 39294 (time 1709788371.99 total 1439.59 last 29.65) - open/close 39586 (time 1709788382.01 total 1449.61 last 29.16) - open/close 39858 (time 1709788392.03 total 1459.62 last 27.15) - open/close 40000 (time 1709788396.83 total 1464.42 last 29.59) - open/close 40307 (time 1709788406.85 total 1474.44 last 30.63) - open/close 40606 (time 1709788416.87 total 1484.46 last 29.84) - open/close 40908 (time 1709788426.88 total 1494.47 last 30.17) - open/close 41202 (time 1709788436.91 total 1504.50 last 29.31) - open/close 41497 (time 1709788446.91 total 1514.51 last 29.49) - open/close 41799 (time 1709788456.94 total 1524.54 last 30.11) - open/close 42085 (time 1709788466.95 total 1534.54 last 28.59) - open/close 42382 (time 1709788476.95 total 1544.55 last 29.69) - open/close 42689 (time 1709788486.95 total 1554.55 last 30.69) - open/close 42989 (time 1709788496.96 total 1564.55 last 30.00) - open/close 43286 (time 1709788506.98 total 1574.57 last 29.63) - open/close 43594 (time 1709788516.99 total 1584.59 last 30.76) - open/close 43892 (time 1709788527.02 total 1594.61 last 29.72) - open/close 44185 (time 1709788537.02 total 1604.62 last 29.29) - open/close 44474 (time 1709788547.03 total 1614.63 last 28.87) - open/close 44764 (time 1709788557.05 total 1624.65 last 28.93) - open/close 45057 (time 1709788567.08 total 1634.67 last 29.23) - open/close 45351 (time 1709788577.09 total 1644.69 last 29.35) - open/close 45642 (time 1709788587.11 total 1654.71 last 29.05) - open/close 45943 (time 1709788597.14 total 1664.74 last 30.02) - open/close 46231 (time 1709788607.17 total 1674.76 last 28.72) - open/close 46516 (time 1709788617.19 total 1684.79 last 28.43) - open/close 46802 (time 1709788627.20 total 1694.80 last 28.58) - open/close 47075 (time 1709788637.21 total 1704.80 last 27.29) - open/close 47347 (time 1709788647.23 total 1714.83 last 27.13) - open/close 47622 (time 1709788657.25 total 1724.84 last 27.46) - open/close 47899 (time 1709788667.26 total 1734.86 last 27.66) - open/close 48173 (time 1709788677.28 total 1744.87 last 27.35) - open/close 48443 (time 1709788687.31 total 1754.91 last 26.91) - open/close 48716 (time 1709788697.32 total 1764.91 last 27.29) - open/close 48985 (time 1709788707.32 total 1774.92 last 26.88) - open/close 49252 (time 1709788717.33 total 1784.93 last 26.68) - open/close 49526 (time 1709788727.34 total 1794.93 last 27.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1801.35 seconds: 27.59 ops/second - unlinked 0 (time 1709788734 ; total 0 ; last 0) - unlinked 10000 (time 1709789185 ; total 451 ; last 451) - unlinked 20000 (time 1709789571 ; total 837 ; last 386) - unlinked 30000 (time 1709789846 ; total 1112 ; last 275) - unlinked 40000 (time 1709789980 ; total 1246 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1288 seconds: 38.585403 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server unloading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2