== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:36 (1709787036) start mds service on oleg320-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg320-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg320-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock oleg320-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 266 (time 1709787084.81 total 10.01 last 26.56) - open/close 488 (time 1709787094.85 total 20.05 last 22.12) - open/close 801 (time 1709787104.93 total 30.13 last 31.04) - open/close 1023 (time 1709787114.94 total 40.14 last 22.20) - open/close 1167 (time 1709787124.97 total 50.17 last 14.35) - open/close 1339 (time 1709787135.32 total 60.52 last 16.61) - open/close 1561 (time 1709787146.23 total 71.43 last 20.36) - open/close 1786 (time 1709787157.08 total 82.28 last 20.74) - open/close 1998 (time 1709787167.09 total 92.29 last 21.17) - open/close 2166 (time 1709787177.76 total 102.96 last 15.76) - open/close 2346 (time 1709787188.37 total 113.57 last 16.95) - open/close 2574 (time 1709787198.97 total 124.17 last 21.52) - open/close 2829 (time 1709787209.80 total 135.00 last 23.54) - open/close 3020 (time 1709787219.83 total 145.03 last 19.04) - open/close 3204 (time 1709787229.87 total 155.07 last 18.33) - open/close 3336 (time 1709787239.88 total 165.08 last 13.19) - open/close 3596 (time 1709787250.07 total 175.27 last 25.52) - open/close 3782 (time 1709787260.08 total 185.28 last 18.58) - open/close 4033 (time 1709787270.52 total 195.72 last 24.04) - open/close 4162 (time 1709787281.11 total 206.31 last 12.18) - open/close 4422 (time 1709787291.38 total 216.58 last 25.33) - open/close 4632 (time 1709787301.78 total 226.98 last 20.19) - open/close 4807 (time 1709787311.78 total 236.98 last 17.49) - open/close 5045 (time 1709787321.82 total 247.01 last 23.72) - open/close 5229 (time 1709787331.82 total 257.02 last 18.39) - open/close 5476 (time 1709787342.76 total 267.96 last 22.59) - open/close 5624 (time 1709787353.26 total 278.46 last 14.09) - open/close 5804 (time 1709787363.68 total 288.88 last 17.28) - open/close 5948 (time 1709787374.60 total 299.80 last 13.19) - open/close 6091 (time 1709787384.64 total 309.84 last 14.24) - open/close 6251 (time 1709787394.66 total 319.86 last 15.97) - open/close 6395 (time 1709787405.04 total 330.24 last 13.87) - open/close 6576 (time 1709787415.85 total 341.05 last 16.75) - open/close 6793 (time 1709787425.86 total 351.06 last 21.67) - open/close 6962 (time 1709787436.12 total 361.32 last 16.47) - open/close 7132 (time 1709787446.13 total 371.33 last 16.98) - open/close 7236 (time 1709787456.14 total 381.34 last 10.40) - open/close 7438 (time 1709787466.14 total 391.34 last 20.20) - open/close 7602 (time 1709787476.15 total 401.35 last 16.38) - open/close 7812 (time 1709787486.87 total 412.07 last 19.58) - open/close 7977 (time 1709787497.01 total 422.21 last 16.28) - open/close 8162 (time 1709787507.04 total 432.24 last 18.44) - open/close 8315 (time 1709787517.33 total 442.53 last 14.86) - open/close 8509 (time 1709787527.88 total 453.08 last 18.39) - open/close 8645 (time 1709787537.89 total 463.09 last 13.60) - open/close 8746 (time 1709787547.98 total 473.18 last 10.01) - open/close 8885 (time 1709787558.50 total 483.70 last 13.21) - open/close 9089 (time 1709787568.51 total 493.71 last 20.38) - open/close 9257 (time 1709787579.37 total 504.57 last 15.47) - open/close 9400 (time 1709787589.40 total 514.60 last 14.26) - open/close 9493 (time 1709787600.04 total 525.24 last 8.74) - open/close 9680 (time 1709787610.06 total 535.26 last 18.66) - open/close 9839 (time 1709787620.09 total 545.29 last 15.85) - open/close 9978 (time 1709787630.96 total 556.16 last 12.79) - open/close 10000 (time 1709787633.58 total 558.78 last 8.41) - open/close 10169 (time 1709787644.42 total 569.62 last 15.58) - open/close 10390 (time 1709787654.43 total 579.63 last 22.08) - open/close 10507 (time 1709787665.19 total 590.39 last 10.87) - open/close 10679 (time 1709787675.84 total 601.04 last 16.16) - open/close 10819 (time 1709787685.84 total 611.04 last 14.00) - open/close 11292 (time 1709787695.86 total 621.06 last 47.22) - open/close 11736 (time 1709787705.87 total 631.07 last 44.33) - open/close 12207 (time 1709787715.91 total 641.11 last 46.91) - open/close 12736 (time 1709787725.93 total 651.13 last 52.81) - open/close 13252 (time 1709787735.94 total 661.14 last 51.56) - open/close 13673 (time 1709787745.95 total 671.15 last 42.05) - open/close 14077 (time 1709787755.95 total 681.15 last 40.40) - open/close 14543 (time 1709787765.95 total 691.15 last 46.59) - open/close 14967 (time 1709787775.97 total 701.17 last 42.33) - open/close 15405 (time 1709787785.97 total 711.17 last 43.79) - open/close 15834 (time 1709787795.98 total 721.18 last 42.84) - open/close 16298 (time 1709787806.00 total 731.20 last 46.33) - open/close 16757 (time 1709787816.00 total 741.20 last 45.88) - open/close 17181 (time 1709787826.02 total 751.22 last 42.34) - open/close 17639 (time 1709787836.03 total 761.23 last 45.74) - open/close 18031 (time 1709787846.04 total 771.24 last 39.18) - open/close 18177 (time 1709787860.88 total 786.08 last 9.84) - open/close 18565 (time 1709787870.90 total 796.10 last 38.71) - open/close 18886 (time 1709787890.65 total 815.85 last 16.26) - open/close 19283 (time 1709787900.67 total 825.87 last 39.64) - open/close 19698 (time 1709787910.68 total 835.88 last 41.42) - open/close 20000 (time 1709787918.76 total 843.96 last 37.41) - open/close 20383 (time 1709787928.76 total 853.96 last 38.30) - open/close 20736 (time 1709787938.78 total 863.98 last 35.21) - open/close 21098 (time 1709787948.80 total 873.99 last 36.16) - open/close 21472 (time 1709787958.82 total 884.02 last 37.32) - open/close 21847 (time 1709787968.83 total 894.03 last 37.44) - open/close 22211 (time 1709787978.84 total 904.04 last 36.37) - open/close 22574 (time 1709787988.86 total 914.06 last 36.23) - open/close 22952 (time 1709787998.87 total 924.07 last 37.79) - open/close 23320 (time 1709788008.89 total 934.09 last 36.69) - open/close 23704 (time 1709788018.91 total 944.11 last 38.35) - open/close 24047 (time 1709788028.92 total 954.12 last 34.26) - open/close 24378 (time 1709788038.92 total 964.12 last 33.10) - open/close 24717 (time 1709788048.93 total 974.13 last 33.87) - open/close 25034 (time 1709788058.96 total 984.16 last 31.62) - open/close 25362 (time 1709788068.98 total 994.18 last 32.74) - open/close 25753 (time 1709788078.98 total 1004.18 last 39.09) - open/close 26136 (time 1709788089.00 total 1014.20 last 38.23) - open/close 26518 (time 1709788099.02 total 1024.22 last 38.12) - open/close 26899 (time 1709788109.03 total 1034.23 last 38.05) - open/close 27269 (time 1709788119.06 total 1044.26 last 36.90) - open/close 27662 (time 1709788129.07 total 1054.27 last 39.24) - open/close 28039 (time 1709788139.09 total 1064.29 last 37.62) - open/close 28412 (time 1709788149.10 total 1074.30 last 37.27) - open/close 28753 (time 1709788159.13 total 1084.33 last 34.01) - open/close 29123 (time 1709788169.14 total 1094.34 last 36.95) - open/close 29483 (time 1709788179.16 total 1104.36 last 35.95) - open/close 29851 (time 1709788189.18 total 1114.38 last 36.70) - open/close 30000 (time 1709788193.76 total 1118.96 last 32.54) - open/close 30326 (time 1709788203.78 total 1128.98 last 32.56) - open/close 30655 (time 1709788213.79 total 1138.99 last 32.84) - open/close 30987 (time 1709788223.81 total 1149.01 last 33.14) - open/close 31325 (time 1709788233.84 total 1159.04 last 33.71) - open/close 31648 (time 1709788243.84 total 1169.04 last 32.30) - open/close 31960 (time 1709788253.86 total 1179.06 last 31.16) - open/close 32263 (time 1709788263.86 total 1189.06 last 30.29) - open/close 32565 (time 1709788273.87 total 1199.07 last 30.17) - open/close 32879 (time 1709788283.87 total 1209.07 last 31.39) - open/close 33159 (time 1709788293.88 total 1219.08 last 27.97) - open/close 33453 (time 1709788303.91 total 1229.11 last 29.31) - open/close 33753 (time 1709788313.92 total 1239.12 last 29.98) - open/close 34051 (time 1709788323.95 total 1249.15 last 29.70) - open/close 34374 (time 1709788333.97 total 1259.17 last 32.24) - open/close 34695 (time 1709788343.99 total 1269.19 last 32.04) - open/close 35007 (time 1709788354.01 total 1279.21 last 31.15) - open/close 35325 (time 1709788364.02 total 1289.21 last 31.78) - open/close 35619 (time 1709788374.03 total 1299.23 last 29.35) - open/close 35909 (time 1709788384.05 total 1309.25 last 28.95) - open/close 36190 (time 1709788394.05 total 1319.25 last 28.10) - open/close 36500 (time 1709788404.07 total 1329.27 last 30.93) - open/close 36794 (time 1709788414.07 total 1339.27 last 29.40) - open/close 37109 (time 1709788424.09 total 1349.29 last 31.46) - open/close 37406 (time 1709788434.10 total 1359.30 last 29.67) - open/close 37719 (time 1709788444.11 total 1369.31 last 31.27) - open/close 38003 (time 1709788454.11 total 1379.31 last 28.40) - open/close 38283 (time 1709788464.13 total 1389.33 last 27.94) - open/close 38567 (time 1709788474.13 total 1399.33 last 28.38) - open/close 38858 (time 1709788484.14 total 1409.34 last 29.08) - open/close 39113 (time 1709788494.14 total 1419.34 last 25.50) - open/close 39373 (time 1709788504.15 total 1429.34 last 25.99) - open/close 39662 (time 1709788514.18 total 1439.38 last 28.80) - open/close 39945 (time 1709788524.18 total 1449.38 last 28.29) - open/close 40212 (time 1709788534.21 total 1459.41 last 26.63) - open/close 40494 (time 1709788544.24 total 1469.44 last 28.13) - open/close 40774 (time 1709788554.24 total 1479.44 last 27.98) - open/close 41047 (time 1709788564.28 total 1489.48 last 27.21) - open/close 41332 (time 1709788574.30 total 1499.50 last 28.44) - open/close 41608 (time 1709788584.32 total 1509.52 last 27.53) - open/close 41886 (time 1709788594.33 total 1519.53 last 27.78) - open/close 42161 (time 1709788604.34 total 1529.54 last 27.48) - open/close 42440 (time 1709788614.35 total 1539.55 last 27.85) - open/close 42716 (time 1709788624.36 total 1549.56 last 27.57) - open/close 42982 (time 1709788634.37 total 1559.57 last 26.59) - open/close 43218 (time 1709788644.41 total 1569.61 last 23.51) - open/close 43418 (time 1709788654.43 total 1579.63 last 19.95) - open/close 43620 (time 1709788664.46 total 1589.66 last 20.14) - open/close 43837 (time 1709788674.50 total 1599.70 last 21.61) - open/close 44082 (time 1709788684.51 total 1609.71 last 24.47) - open/close 44311 (time 1709788694.54 total 1619.74 last 22.84) - open/close 44545 (time 1709788704.55 total 1629.75 last 23.37) - open/close 44788 (time 1709788714.58 total 1639.78 last 24.23) - open/close 45045 (time 1709788724.61 total 1649.81 last 25.62) - open/close 45314 (time 1709788734.64 total 1659.84 last 26.82) - open/close 45581 (time 1709788744.64 total 1669.84 last 26.69) - open/close 45841 (time 1709788754.65 total 1679.85 last 25.99) - open/close 46091 (time 1709788764.68 total 1689.88 last 24.92) - open/close 46325 (time 1709788774.71 total 1699.91 last 23.33) - open/close 46550 (time 1709788784.75 total 1709.95 last 22.40) - open/close 46767 (time 1709788794.79 total 1719.99 last 21.63) - open/close 46997 (time 1709788804.80 total 1730.00 last 22.96) - open/close 47214 (time 1709788814.86 total 1740.06 last 21.57) - open/close 47427 (time 1709788824.90 total 1750.10 last 21.23) - open/close 47644 (time 1709788834.91 total 1760.11 last 21.68) - open/close 47870 (time 1709788844.94 total 1770.14 last 22.52) - open/close 48097 (time 1709788854.97 total 1780.17 last 22.63) - open/close 48324 (time 1709788865.02 total 1790.22 last 22.59) - open/close 48550 (time 1709788875.06 total 1800.26 last 22.51) - open/close 48776 (time 1709788885.06 total 1810.26 last 22.59) - open/close 49019 (time 1709788895.08 total 1820.28 last 24.26) - open/close 49253 (time 1709788905.12 total 1830.32 last 23.31) - open/close 49488 (time 1709788915.14 total 1840.34 last 23.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1845.09 seconds: 26.88 ops/second - unlinked 0 (time 1709788921 ; total 0 ; last 0) - unlinked 10000 (time 1709789391 ; total 470 ; last 470) - unlinked 20000 (time 1709789765 ; total 844 ; last 374) - unlinked 30000 (time 1709790008 ; total 1087 ; last 243) - unlinked 40000 (time 1709790121 ; total 1200 ; last 113) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1251 seconds: 39.648281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server unloading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-client: ssh exited with exit code 2 pdsh@oleg320-client: oleg320-server: ssh exited with exit code 2