== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:46 (1709787166) start mds service on oleg224-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 168 (time 1709787214.04 total 10.07 last 16.69) - open/close 259 (time 1709787224.20 total 20.22 last 8.96) - open/close 456 (time 1709787234.47 total 30.50 last 19.17) - open/close 554 (time 1709787245.00 total 41.02 last 9.31) - open/close 877 (time 1709787255.24 total 51.26 last 31.56) - open/close 1052 (time 1709787265.24 total 61.26 last 17.49) - open/close 1166 (time 1709787276.09 total 72.11 last 10.51) - open/close 1383 (time 1709787286.19 total 82.21 last 21.49) - open/close 1563 (time 1709787296.20 total 92.22 last 17.98) - open/close 1709 (time 1709787306.87 total 102.89 last 13.68) - open/close 1871 (time 1709787317.48 total 113.50 last 15.26) - open/close 2083 (time 1709787327.51 total 123.53 last 21.14) - open/close 2194 (time 1709787337.59 total 133.62 last 11.01) - open/close 2359 (time 1709787347.61 total 143.63 last 16.47) - open/close 2483 (time 1709787357.63 total 153.65 last 12.38) - open/close 2748 (time 1709787367.63 total 163.65 last 26.49) - open/close 2906 (time 1709787377.64 total 173.66 last 15.78) - open/close 3061 (time 1709787388.45 total 184.47 last 14.34) - open/close 3209 (time 1709787400.52 total 196.54 last 12.26) - open/close 3410 (time 1709787410.53 total 206.55 last 20.08) - open/close 3592 (time 1709787420.75 total 216.77 last 17.81) - open/close 3743 (time 1709787430.77 total 226.80 last 15.06) - open/close 3943 (time 1709787441.21 total 237.24 last 19.16) - open/close 4095 (time 1709787451.23 total 247.25 last 15.18) - open/close 4233 (time 1709787462.21 total 258.23 last 12.57) - open/close 4417 (time 1709787473.15 total 269.18 last 16.81) - open/close 4652 (time 1709787483.55 total 279.57 last 22.60) - open/close 4869 (time 1709787494.00 total 290.02 last 20.77) - open/close 5122 (time 1709787504.54 total 300.56 last 24.01) - open/close 5313 (time 1709787515.32 total 311.35 last 17.71) - open/close 5549 (time 1709787525.33 total 321.36 last 23.57) - open/close 5723 (time 1709787535.34 total 331.36 last 17.39) - open/close 5911 (time 1709787546.14 total 342.16 last 17.41) - open/close 6145 (time 1709787556.57 total 352.60 last 22.42) - open/close 6281 (time 1709787566.59 total 362.61 last 13.58) - open/close 6432 (time 1709787577.61 total 373.64 last 13.70) - open/close 6566 (time 1709787587.75 total 383.77 last 13.22) - open/close 6740 (time 1709787597.83 total 393.85 last 17.26) - open/close 6868 (time 1709787607.88 total 403.90 last 12.74) - open/close 7094 (time 1709787617.89 total 413.92 last 22.56) - open/close 7250 (time 1709787627.91 total 423.94 last 15.57) - open/close 7364 (time 1709787638.26 total 434.28 last 11.02) - open/close 7567 (time 1709787649.20 total 445.23 last 18.55) - open/close 7690 (time 1709787667.32 total 463.34 last 6.79) - open/close 7852 (time 1709787678.07 total 474.09 last 15.07) - open/close 7995 (time 1709787688.93 total 484.95 last 13.17) - open/close 8132 (time 1709787699.02 total 495.04 last 13.58) - open/close 8301 (time 1709787709.03 total 505.06 last 16.87) - open/close 8454 (time 1709787719.56 total 515.58 last 14.54) - open/close 8606 (time 1709787729.57 total 525.59 last 15.18) - open/close 8694 (time 1709787740.14 total 536.16 last 8.32) - open/close 8841 (time 1709787750.83 total 546.85 last 13.75) - open/close 8981 (time 1709787761.47 total 557.49 last 13.16) - open/close 9162 (time 1709787772.49 total 568.51 last 16.42) - open/close 9338 (time 1709787791.23 total 587.25 last 9.39) - open/close 9418 (time 1709787801.37 total 597.39 last 7.89) - open/close 9662 (time 1709787811.39 total 607.41 last 24.35) - open/close 10000 (time 1709787818.44 total 614.46 last 47.97) - open/close 10304 (time 1709787836.00 total 632.02 last 17.31) - open/close 10780 (time 1709787846.01 total 642.03 last 47.56) - open/close 11236 (time 1709787856.02 total 652.04 last 45.54) - open/close 11616 (time 1709787866.04 total 662.06 last 37.93) - open/close 12069 (time 1709787876.04 total 672.07 last 45.28) - open/close 12481 (time 1709787886.06 total 682.08 last 41.15) - open/close 12871 (time 1709787896.08 total 692.10 last 38.90) - open/close 13261 (time 1709787906.10 total 702.12 last 38.94) - open/close 13485 (time 1709787923.17 total 719.19 last 13.12) - open/close 13861 (time 1709787933.19 total 729.21 last 37.50) - open/close 14276 (time 1709787943.26 total 739.28 last 41.22) - open/close 14720 (time 1709787953.28 total 749.30 last 44.31) - open/close 15188 (time 1709787963.30 total 759.32 last 46.70) - open/close 15637 (time 1709787973.31 total 769.34 last 44.85) - open/close 16072 (time 1709787983.32 total 779.35 last 43.45) - open/close 16499 (time 1709787993.33 total 789.35 last 42.67) - open/close 16992 (time 1709788003.34 total 799.37 last 49.24) - open/close 17496 (time 1709788013.36 total 809.38 last 50.31) - open/close 18025 (time 1709788023.37 total 819.39 last 52.84) - open/close 18549 (time 1709788033.38 total 829.41 last 52.34) - open/close 19075 (time 1709788043.38 total 839.41 last 52.60) - open/close 19586 (time 1709788053.39 total 849.41 last 51.07) - open/close 20000 (time 1709788061.64 total 857.66 last 50.17) - open/close 20464 (time 1709788071.65 total 867.67 last 46.37) - open/close 20949 (time 1709788081.67 total 877.69 last 48.41) - open/close 21423 (time 1709788091.67 total 887.69 last 47.38) - open/close 21874 (time 1709788101.68 total 897.70 last 45.07) - open/close 22322 (time 1709788111.69 total 907.71 last 44.75) - open/close 22765 (time 1709788121.71 total 917.73 last 44.21) - open/close 23185 (time 1709788131.73 total 927.75 last 41.91) - open/close 23626 (time 1709788141.73 total 937.76 last 44.08) - open/close 24036 (time 1709788151.75 total 947.78 last 40.92) - open/close 24456 (time 1709788161.76 total 957.78 last 41.98) - open/close 24897 (time 1709788171.77 total 967.79 last 44.05) - open/close 25326 (time 1709788181.79 total 977.82 last 42.80) - open/close 25737 (time 1709788191.81 total 987.83 last 41.05) - open/close 26129 (time 1709788201.83 total 997.85 last 39.11) - open/close 26535 (time 1709788211.84 total 1007.86 last 40.57) - open/close 26943 (time 1709788221.85 total 1017.87 last 40.76) - open/close 27354 (time 1709788231.86 total 1027.89 last 41.03) - open/close 27759 (time 1709788241.89 total 1037.91 last 40.40) - open/close 28156 (time 1709788251.89 total 1047.92 last 39.68) - open/close 28550 (time 1709788261.90 total 1057.92 last 39.38) - open/close 28936 (time 1709788271.92 total 1067.94 last 38.53) - open/close 29308 (time 1709788281.95 total 1077.97 last 37.10) - open/close 29651 (time 1709788291.97 total 1087.99 last 34.22) - open/close 30000 (time 1709788301.95 total 1097.97 last 34.97) - open/close 30345 (time 1709788311.98 total 1108.00 last 34.42) - open/close 30684 (time 1709788322.00 total 1118.02 last 33.83) - open/close 31009 (time 1709788332.02 total 1128.04 last 32.44) - open/close 31324 (time 1709788342.04 total 1138.06 last 31.42) - open/close 31655 (time 1709788352.05 total 1148.07 last 33.07) - open/close 31977 (time 1709788362.06 total 1158.09 last 32.16) - open/close 32289 (time 1709788372.08 total 1168.10 last 31.14) - open/close 32595 (time 1709788382.09 total 1178.11 last 30.57) - open/close 32882 (time 1709788392.12 total 1188.14 last 28.61) - open/close 33199 (time 1709788402.13 total 1198.15 last 31.67) - open/close 33520 (time 1709788412.16 total 1208.19 last 32.00) - open/close 33827 (time 1709788422.17 total 1218.19 last 30.69) - open/close 34145 (time 1709788432.18 total 1228.20 last 31.77) - open/close 34455 (time 1709788442.18 total 1238.20 last 31.00) - open/close 34777 (time 1709788452.21 total 1248.23 last 32.11) - open/close 35075 (time 1709788462.23 total 1258.26 last 29.72) - open/close 35382 (time 1709788472.24 total 1268.27 last 30.67) - open/close 35716 (time 1709788482.25 total 1278.27 last 33.38) - open/close 36037 (time 1709788492.28 total 1288.30 last 32.02) - open/close 36363 (time 1709788502.30 total 1298.33 last 32.51) - open/close 36699 (time 1709788512.31 total 1308.33 last 33.59) - open/close 37024 (time 1709788522.32 total 1318.34 last 32.46) - open/close 37339 (time 1709788532.32 total 1328.35 last 31.49) - open/close 37667 (time 1709788542.33 total 1338.36 last 32.77) - open/close 37990 (time 1709788552.34 total 1348.36 last 32.30) - open/close 38300 (time 1709788562.36 total 1358.39 last 30.91) - open/close 38619 (time 1709788572.39 total 1368.42 last 31.81) - open/close 38928 (time 1709788582.41 total 1378.43 last 30.86) - open/close 39245 (time 1709788592.41 total 1388.43 last 31.70) - open/close 39558 (time 1709788602.42 total 1398.45 last 31.26) - open/close 39864 (time 1709788612.45 total 1408.47 last 30.52) - open/close 40000 (time 1709788616.88 total 1412.91 last 30.67) - open/close 40306 (time 1709788626.90 total 1422.93 last 30.53) - open/close 40601 (time 1709788636.91 total 1432.94 last 29.47) - open/close 40887 (time 1709788646.94 total 1442.96 last 28.53) - open/close 41169 (time 1709788656.95 total 1452.97 last 28.18) - open/close 41467 (time 1709788666.97 total 1462.99 last 29.73) - open/close 41752 (time 1709788677.00 total 1473.02 last 28.42) - open/close 42045 (time 1709788687.03 total 1483.05 last 29.21) - open/close 42338 (time 1709788697.05 total 1493.08 last 29.23) - open/close 42636 (time 1709788707.08 total 1503.11 last 29.71) - open/close 42925 (time 1709788717.09 total 1513.12 last 28.87) - open/close 43218 (time 1709788727.11 total 1523.13 last 29.25) - open/close 43501 (time 1709788737.13 total 1533.15 last 28.24) - open/close 43801 (time 1709788747.17 total 1543.19 last 29.88) - open/close 44100 (time 1709788757.20 total 1553.22 last 29.83) - open/close 44402 (time 1709788767.20 total 1563.22 last 30.19) - open/close 44699 (time 1709788777.22 total 1573.24 last 29.63) - open/close 44987 (time 1709788787.22 total 1583.24 last 28.80) - open/close 45277 (time 1709788797.23 total 1593.26 last 28.96) - open/close 45574 (time 1709788807.23 total 1603.26 last 29.70) - open/close 45862 (time 1709788817.27 total 1613.29 last 28.70) - open/close 46140 (time 1709788827.29 total 1623.32 last 27.73) - open/close 46417 (time 1709788837.30 total 1633.32 last 27.68) - open/close 46699 (time 1709788847.30 total 1643.32 last 28.20) - open/close 46983 (time 1709788857.33 total 1653.35 last 28.32) - open/close 47268 (time 1709788867.35 total 1663.37 last 28.45) - open/close 47557 (time 1709788877.35 total 1673.38 last 28.88) - open/close 47845 (time 1709788887.37 total 1683.39 last 28.76) - open/close 48130 (time 1709788897.37 total 1693.39 last 28.49) - open/close 48408 (time 1709788907.38 total 1703.41 last 27.76) - open/close 48686 (time 1709788917.42 total 1713.44 last 27.70) - open/close 48959 (time 1709788927.43 total 1723.46 last 27.26) - open/close 49239 (time 1709788937.45 total 1733.47 last 27.95) - open/close 49508 (time 1709788947.46 total 1743.48 last 26.89) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1746.88 seconds: 28.39 ops/second - unlinked 0 (time 1709788951 ; total 0 ; last 0) - unlinked 10000 (time 1709789401 ; total 450 ; last 450) - unlinked 20000 (time 1709789745 ; total 794 ; last 344) - unlinked 30000 (time 1709789978 ; total 1027 ; last 233) - unlinked 40000 (time 1709790084 ; total 1133 ; last 106) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1167 seconds: 42.502144 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server unloading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2