== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:17 (1709786837) start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 263 (time 1709786868.90 total 10.82 last 24.31) - open/close 409 (time 1709786879.67 total 21.58 last 13.56) - open/close 681 (time 1709786890.60 total 32.51 last 24.89) - open/close 882 (time 1709786901.42 total 43.33 last 18.57) - open/close 1047 (time 1709786911.52 total 53.44 last 16.33) - open/close 1297 (time 1709786922.28 total 64.20 last 23.22) - open/close 1559 (time 1709786932.29 total 74.20 last 26.19) - open/close 1921 (time 1709786943.21 total 85.13 last 33.14) - open/close 2008 (time 1709786953.53 total 95.44 last 8.43) - open/close 2161 (time 1709786963.63 total 105.55 last 15.14) - open/close 2369 (time 1709786981.00 total 122.91 last 11.98) - open/close 2628 (time 1709786991.62 total 133.53 last 24.39) - open/close 2827 (time 1709787002.24 total 144.15 last 18.73) - open/close 3036 (time 1709787012.24 total 154.16 last 20.90) - open/close 3291 (time 1709787022.25 total 164.17 last 25.48) - open/close 3467 (time 1709787032.66 total 174.58 last 16.91) - open/close 3615 (time 1709787042.66 total 184.58 last 14.79) - open/close 3762 (time 1709787053.47 total 195.38 last 13.61) - open/close 4048 (time 1709787063.83 total 205.75 last 27.59) - open/close 4263 (time 1709787073.84 total 215.76 last 21.47) - open/close 4475 (time 1709787084.54 total 226.46 last 19.82) - open/close 4775 (time 1709787094.55 total 236.47 last 29.96) - open/close 5012 (time 1709787105.09 total 247.01 last 22.49) - open/close 5130 (time 1709787125.03 total 266.95 last 5.92) - open/close 5318 (time 1709787135.04 total 276.95 last 18.80) - open/close 5450 (time 1709787145.25 total 287.17 last 12.92) - open/close 5702 (time 1709787155.26 total 297.17 last 25.18) - open/close 5924 (time 1709787166.90 total 308.82 last 19.06) - open/close 6144 (time 1709787177.73 total 319.64 last 20.33) - open/close 6442 (time 1709787187.79 total 329.70 last 29.62) - open/close 6654 (time 1709787198.66 total 340.58 last 19.49) - open/close 6751 (time 1709787209.14 total 351.06 last 9.26) - open/close 6956 (time 1709787219.81 total 361.73 last 19.22) - open/close 7131 (time 1709787230.35 total 372.27 last 16.60) - open/close 7280 (time 1709787240.42 total 382.34 last 14.79) - open/close 7529 (time 1709787250.45 total 392.37 last 24.83) - open/close 7622 (time 1709787260.46 total 402.38 last 9.29) - open/close 7783 (time 1709787270.81 total 412.73 last 15.55) - open/close 7945 (time 1709787281.01 total 422.92 last 15.89) - open/close 8095 (time 1709787291.02 total 432.94 last 14.98) - open/close 8303 (time 1709787301.68 total 443.59 last 19.52) - open/close 8542 (time 1709787311.88 total 453.80 last 23.42) - open/close 8804 (time 1709787321.89 total 463.81 last 26.18) - open/close 8992 (time 1709787332.81 total 474.73 last 17.21) - open/close 9275 (time 1709787342.82 total 484.74 last 28.27) - open/close 9484 (time 1709787353.29 total 495.21 last 19.96) - open/close 9774 (time 1709787363.30 total 505.21 last 28.98) - open/close 9892 (time 1709787374.14 total 516.06 last 10.88) - open/close 10000 (time 1709787380.47 total 522.38 last 17.08) - open/close 10468 (time 1709787390.48 total 532.39 last 46.75) - open/close 11043 (time 1709787400.49 total 542.40 last 57.45) - open/close 11547 (time 1709787410.50 total 552.41 last 50.36) - open/close 12079 (time 1709787420.53 total 562.44 last 53.04) - open/close 12663 (time 1709787430.54 total 572.45 last 58.33) - open/close 12963 (time 1709787446.92 total 588.83 last 18.32) - open/close 13470 (time 1709787456.92 total 598.84 last 50.66) - open/close 13892 (time 1709787466.94 total 608.85 last 42.16) - open/close 14268 (time 1709787476.96 total 618.87 last 37.52) - open/close 14728 (time 1709787486.97 total 628.89 last 45.93) - open/close 15216 (time 1709787496.99 total 638.91 last 48.69) - open/close 15697 (time 1709787506.99 total 648.91 last 48.09) - open/close 16162 (time 1709787517.00 total 658.92 last 46.46) - open/close 16564 (time 1709787527.03 total 668.94 last 40.10) - open/close 16965 (time 1709787537.03 total 678.94 last 40.09) - open/close 17389 (time 1709787547.03 total 688.95 last 42.38) - open/close 17763 (time 1709787557.04 total 698.96 last 37.36) - open/close 18137 (time 1709787567.07 total 708.99 last 37.30) - open/close 18495 (time 1709787577.09 total 719.00 last 35.75) - open/close 18857 (time 1709787587.09 total 729.00 last 36.19) - open/close 19179 (time 1709787597.10 total 739.01 last 32.17) - open/close 19519 (time 1709787607.13 total 749.05 last 33.89) - open/close 19746 (time 1709787624.31 total 766.23 last 13.21) - open/close 19886 (time 1709787639.11 total 781.03 last 9.46) - open/close 20000 (time 1709787642.71 total 784.63 last 31.68) - open/close 20347 (time 1709787652.73 total 794.65 last 34.63) - open/close 20467 (time 1709787667.85 total 809.76 last 7.94) - open/close 20809 (time 1709787677.86 total 819.78 last 34.15) - open/close 21139 (time 1709787687.90 total 829.82 last 32.87) - open/close 21398 (time 1709787697.93 total 839.85 last 25.82) - open/close 21686 (time 1709787707.96 total 849.88 last 28.71) - open/close 21953 (time 1709787717.98 total 859.89 last 26.67) - open/close 22286 (time 1709787727.99 total 869.90 last 33.27) - open/close 22610 (time 1709787737.99 total 879.90 last 32.39) - open/close 22912 (time 1709787748.01 total 889.93 last 30.13) - open/close 23174 (time 1709787758.03 total 899.95 last 26.14) - open/close 23481 (time 1709787768.05 total 909.96 last 30.66) - open/close 23786 (time 1709787778.06 total 919.97 last 30.47) - open/close 24111 (time 1709787788.06 total 929.97 last 32.50) - open/close 24200 (time 1709787802.27 total 944.19 last 6.26) - open/close 24542 (time 1709787812.28 total 954.19 last 34.20) - open/close 24864 (time 1709787822.29 total 964.21 last 32.15) - open/close 25205 (time 1709787832.31 total 974.23 last 34.02) - open/close 25553 (time 1709787842.36 total 984.27 last 34.65) - open/close 25877 (time 1709787852.37 total 994.28 last 32.36) - open/close 26206 (time 1709787862.39 total 1004.31 last 32.83) - open/close 26527 (time 1709787872.41 total 1014.32 last 32.06) - open/close 26833 (time 1709787882.41 total 1024.33 last 30.58) - open/close 27157 (time 1709787892.44 total 1034.36 last 32.30) - open/close 27465 (time 1709787902.57 total 1044.49 last 30.40) - open/close 27791 (time 1709787912.59 total 1054.51 last 32.53) - open/close 28081 (time 1709787922.62 total 1064.54 last 28.92) - open/close 28348 (time 1709787942.78 total 1084.69 last 13.25) - open/close 28631 (time 1709787952.79 total 1094.71 last 28.26) - open/close 28942 (time 1709787962.83 total 1104.74 last 31.00) - open/close 29260 (time 1709787972.84 total 1114.75 last 31.76) - open/close 29568 (time 1709787982.85 total 1124.77 last 30.76) - open/close 29890 (time 1709787992.89 total 1134.80 last 32.07) - open/close 30000 (time 1709787996.33 total 1138.24 last 31.99) - open/close 30323 (time 1709788006.35 total 1148.26 last 32.24) - open/close 30625 (time 1709788016.38 total 1158.30 last 30.09) - open/close 30904 (time 1709788026.39 total 1168.30 last 27.88) - open/close 31192 (time 1709788036.39 total 1178.31 last 28.79) - open/close 31467 (time 1709788046.41 total 1188.32 last 27.45) - open/close 31738 (time 1709788056.43 total 1198.35 last 27.03) - open/close 32015 (time 1709788066.46 total 1208.38 last 27.62) - open/close 32305 (time 1709788076.46 total 1218.38 last 28.99) - open/close 32661 (time 1709788086.50 total 1228.41 last 35.49) - open/close 33003 (time 1709788096.51 total 1238.43 last 34.13) - open/close 33338 (time 1709788106.52 total 1248.43 last 33.50) - open/close 33681 (time 1709788116.52 total 1258.44 last 34.27) - open/close 34019 (time 1709788126.54 total 1268.45 last 33.75) - open/close 34379 (time 1709788136.57 total 1278.48 last 35.90) - open/close 34740 (time 1709788146.58 total 1288.50 last 36.04) - open/close 35068 (time 1709788156.60 total 1298.51 last 32.75) - open/close 35401 (time 1709788166.61 total 1308.52 last 33.28) - open/close 35741 (time 1709788176.62 total 1318.53 last 33.96) - open/close 36060 (time 1709788186.64 total 1328.56 last 31.82) - open/close 36376 (time 1709788196.67 total 1338.59 last 31.51) - open/close 36688 (time 1709788206.68 total 1348.60 last 31.17) - open/close 37003 (time 1709788216.68 total 1358.60 last 31.50) - open/close 37124 (time 1709788231.48 total 1373.40 last 8.18) - open/close 37430 (time 1709788241.50 total 1383.42 last 30.54) - open/close 37724 (time 1709788251.52 total 1393.43 last 29.36) - open/close 38004 (time 1709788261.54 total 1403.46 last 27.93) - open/close 38291 (time 1709788271.58 total 1413.49 last 28.59) - open/close 38578 (time 1709788281.61 total 1423.52 last 28.62) - open/close 38853 (time 1709788291.61 total 1433.52 last 27.49) - open/close 39133 (time 1709788301.63 total 1443.55 last 27.93) - open/close 39410 (time 1709788311.63 total 1453.55 last 27.70) - open/close 39692 (time 1709788321.66 total 1463.58 last 28.12) - open/close 39975 (time 1709788331.69 total 1473.60 last 28.24) - open/close 40264 (time 1709788341.70 total 1483.61 last 28.87) - open/close 40552 (time 1709788351.72 total 1493.64 last 28.73) - open/close 40830 (time 1709788361.75 total 1503.67 last 27.71) - open/close 41080 (time 1709788371.75 total 1513.67 last 24.99) - open/close 41337 (time 1709788381.78 total 1523.70 last 25.63) - open/close 41602 (time 1709788391.79 total 1533.70 last 26.49) - open/close 41875 (time 1709788401.80 total 1543.72 last 27.26) - open/close 42141 (time 1709788411.82 total 1553.73 last 26.56) - open/close 42441 (time 1709788421.82 total 1563.74 last 29.98) - open/close 42702 (time 1709788431.85 total 1573.76 last 26.04) - open/close 42974 (time 1709788441.87 total 1583.78 last 27.14) - open/close 43233 (time 1709788451.89 total 1593.81 last 25.83) - open/close 43495 (time 1709788461.90 total 1603.81 last 26.19) - open/close 43740 (time 1709788471.93 total 1613.84 last 24.43) - open/close 44011 (time 1709788481.96 total 1623.88 last 27.01) - open/close 44252 (time 1709788491.99 total 1633.90 last 24.03) - open/close 44492 (time 1709788502.01 total 1643.93 last 23.95) - open/close 44740 (time 1709788512.04 total 1653.96 last 24.72) - open/close 45007 (time 1709788522.05 total 1663.97 last 26.68) - open/close 45265 (time 1709788532.05 total 1673.97 last 25.79) - open/close 45527 (time 1709788542.09 total 1684.00 last 26.11) - open/close 45792 (time 1709788552.10 total 1694.01 last 26.48) - open/close 46044 (time 1709788562.11 total 1704.03 last 25.15) - open/close 46311 (time 1709788572.14 total 1714.06 last 26.62) - open/close 46580 (time 1709788582.16 total 1724.07 last 26.87) - open/close 46850 (time 1709788592.18 total 1734.09 last 26.95) - open/close 47119 (time 1709788602.21 total 1744.13 last 26.80) - open/close 47384 (time 1709788612.23 total 1754.15 last 26.44) - open/close 47656 (time 1709788622.25 total 1764.16 last 27.16) - open/close 47928 (time 1709788632.28 total 1774.20 last 27.11) - open/close 48191 (time 1709788642.29 total 1784.21 last 26.28) - open/close 48462 (time 1709788652.29 total 1794.21 last 27.10) - open/close 48724 (time 1709788662.30 total 1804.21 last 26.19) - open/close 48981 (time 1709788672.30 total 1814.21 last 25.69) - open/close 49226 (time 1709788682.32 total 1824.24 last 24.44) - open/close 49461 (time 1709788692.35 total 1834.27 last 23.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1844.25 seconds: 26.95 ops/second - unlinked 0 (time 1709788703 ; total 0 ; last 0) - unlinked 10000 (time 1709789179 ; total 476 ; last 476) - unlinked 20000 (time 1709789608 ; total 905 ; last 429) - unlinked 30000 (time 1709789951 ; total 1248 ; last 343) - unlinked 40000 (time 1709790143 ; total 1440 ; last 192) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1500 seconds: 33.132000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2