== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:42:39 (1709786559) start mds service on oleg426-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg426-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock oleg426-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 344 (time 1709786587.31 total 10.27 last 33.50) - open/close 696 (time 1709786598.20 total 21.16 last 32.32) - open/close 938 (time 1709786608.75 total 31.71 last 22.94) - open/close 1197 (time 1709786618.94 total 41.90 last 25.40) - open/close 1660 (time 1709786629.94 total 52.90 last 42.12) - open/close 2039 (time 1709786640.73 total 63.69 last 35.13) - open/close 2313 (time 1709786650.80 total 73.76 last 27.20) - open/close 2740 (time 1709786661.72 total 84.68 last 39.11) - open/close 2989 (time 1709786671.73 total 94.69 last 24.88) - open/close 3258 (time 1709786682.41 total 105.37 last 25.19) - open/close 3538 (time 1709786692.57 total 115.53 last 27.55) - open/close 3723 (time 1709786702.61 total 125.57 last 18.42) - open/close 3984 (time 1709786713.21 total 136.17 last 24.64) - open/close 4192 (time 1709786723.96 total 146.92 last 19.35) - open/close 4319 (time 1709786734.85 total 157.81 last 11.66) - open/close 4558 (time 1709786744.86 total 167.82 last 23.88) - open/close 4762 (time 1709786755.08 total 178.04 last 19.96) - open/close 5104 (time 1709786765.31 total 188.27 last 33.41) - open/close 5341 (time 1709786775.50 total 198.46 last 23.26) - open/close 5660 (time 1709786786.24 total 209.20 last 29.72) - open/close 5750 (time 1709786800.60 total 223.56 last 6.27) - open/close 5986 (time 1709786811.14 total 234.10 last 22.38) - open/close 6288 (time 1709786821.15 total 244.11 last 30.18) - open/close 6478 (time 1709786832.06 total 255.02 last 17.40) - open/close 6664 (time 1709786842.68 total 265.64 last 17.52) - open/close 6966 (time 1709786852.82 total 275.78 last 29.79) - open/close 7256 (time 1709786862.83 total 285.79 last 28.96) - open/close 7451 (time 1709786872.83 total 295.80 last 19.49) - open/close 7650 (time 1709786882.84 total 305.80 last 19.88) - open/close 7842 (time 1709786892.86 total 315.82 last 19.16) - open/close 8076 (time 1709786903.80 total 326.76 last 21.39) - open/close 8273 (time 1709786914.37 total 337.33 last 18.64) - open/close 8438 (time 1709786925.12 total 348.08 last 15.35) - open/close 8621 (time 1709786935.13 total 358.09 last 18.28) - open/close 8813 (time 1709786945.95 total 368.91 last 17.75) - open/close 9058 (time 1709786956.77 total 379.73 last 22.63) - open/close 9283 (time 1709786967.74 total 390.70 last 20.52) - open/close 9435 (time 1709786977.88 total 400.84 last 14.99) - open/close 9647 (time 1709786987.88 total 410.84 last 21.20) - open/close 9987 (time 1709786997.89 total 420.85 last 33.98) - open/close 10601 (time 1709787007.90 total 430.86 last 61.33) - open/close 11260 (time 1709787017.90 total 440.86 last 65.89) - open/close 12039 (time 1709787027.90 total 450.86 last 77.89) - open/close 12659 (time 1709787037.91 total 460.87 last 61.96) - open/close 13216 (time 1709787047.91 total 470.88 last 55.66) - open/close 13775 (time 1709787057.92 total 480.88 last 55.85) - open/close 14389 (time 1709787067.93 total 490.89 last 61.35) - open/close 14956 (time 1709787077.94 total 500.90 last 56.65) - open/close 15478 (time 1709787087.95 total 510.91 last 52.15) - open/close 16085 (time 1709787097.95 total 520.91 last 60.69) - open/close 16596 (time 1709787107.96 total 530.92 last 51.07) - open/close 17176 (time 1709787117.98 total 540.94 last 57.87) - open/close 17715 (time 1709787127.99 total 550.95 last 53.86) - open/close 18189 (time 1709787138.01 total 560.97 last 47.30) - open/close 18665 (time 1709787148.03 total 570.99 last 47.50) - open/close 19199 (time 1709787158.04 total 581.00 last 53.35) - open/close 19769 (time 1709787168.04 total 591.01 last 56.98) - open/close 20000 (time 1709787172.43 total 595.39 last 52.69) - open/close 20460 (time 1709787182.44 total 605.40 last 45.95) - open/close 20889 (time 1709787192.45 total 615.41 last 42.85) - open/close 21273 (time 1709787202.46 total 625.42 last 38.39) - open/close 21658 (time 1709787212.47 total 635.43 last 38.44) - open/close 22010 (time 1709787222.48 total 645.44 last 35.16) - open/close 22372 (time 1709787232.50 total 655.46 last 36.15) - open/close 22748 (time 1709787242.50 total 665.46 last 37.57) - open/close 23113 (time 1709787252.51 total 675.47 last 36.48) - open/close 23350 (time 1709787270.32 total 693.28 last 13.31) - open/close 23661 (time 1709787280.33 total 703.29 last 31.07) - open/close 23962 (time 1709787290.34 total 713.30 last 30.07) - open/close 24306 (time 1709787300.35 total 723.31 last 34.36) - open/close 24610 (time 1709787310.35 total 733.32 last 30.38) - open/close 24939 (time 1709787320.37 total 743.33 last 32.85) - open/close 25266 (time 1709787330.38 total 753.34 last 32.66) - open/close 25416 (time 1709787346.15 total 769.11 last 9.51) - open/close 25772 (time 1709787356.17 total 779.13 last 35.53) - open/close 26091 (time 1709787366.17 total 789.13 last 31.90) - open/close 26384 (time 1709787376.17 total 799.13 last 29.29) - open/close 26699 (time 1709787386.18 total 809.14 last 31.48) - open/close 27044 (time 1709787396.19 total 819.15 last 34.46) - open/close 27349 (time 1709787406.21 total 829.17 last 30.45) - open/close 27623 (time 1709787416.24 total 839.20 last 27.32) - open/close 27897 (time 1709787426.24 total 849.20 last 27.40) - open/close 28159 (time 1709787436.28 total 859.24 last 26.09) - open/close 28441 (time 1709787446.33 total 869.29 last 28.05) - open/close 28716 (time 1709787456.34 total 879.30 last 27.47) - open/close 28964 (time 1709787466.36 total 889.32 last 24.76) - open/close 29250 (time 1709787476.42 total 899.38 last 28.44) - open/close 29550 (time 1709787486.42 total 909.38 last 29.98) - open/close 29807 (time 1709787496.45 total 919.41 last 25.63) - open/close 30000 (time 1709787503.37 total 926.33 last 27.89) - open/close 30249 (time 1709787513.38 total 936.34 last 24.86) - open/close 30480 (time 1709787523.43 total 946.39 last 23.00) - open/close 30769 (time 1709787533.46 total 956.42 last 28.82) - open/close 31021 (time 1709787543.48 total 966.44 last 25.15) - open/close 31277 (time 1709787553.52 total 976.48 last 25.48) - open/close 31535 (time 1709787563.54 total 986.50 last 25.76) - open/close 31844 (time 1709787573.58 total 996.54 last 30.79) - open/close 32139 (time 1709787583.58 total 1006.54 last 29.49) - open/close 32431 (time 1709787593.61 total 1016.57 last 29.11) - open/close 32715 (time 1709787603.64 total 1026.60 last 28.32) - open/close 33007 (time 1709787613.66 total 1036.62 last 29.13) - open/close 33304 (time 1709787623.69 total 1046.65 last 29.62) - open/close 33585 (time 1709787633.69 total 1056.65 last 28.08) - open/close 33864 (time 1709787643.70 total 1066.66 last 27.89) - open/close 34142 (time 1709787653.70 total 1076.66 last 27.79) - open/close 34446 (time 1709787663.72 total 1086.68 last 30.34) - open/close 34735 (time 1709787673.73 total 1096.69 last 28.87) - open/close 34950 (time 1709787683.75 total 1106.71 last 21.46) - open/close 35205 (time 1709787693.77 total 1116.73 last 25.44) - open/close 35445 (time 1709787703.81 total 1126.77 last 23.90) - open/close 35682 (time 1709787713.81 total 1136.78 last 23.69) - open/close 35924 (time 1709787723.82 total 1146.78 last 24.19) - open/close 36197 (time 1709787733.86 total 1156.82 last 27.19) - open/close 36520 (time 1709787743.86 total 1166.82 last 32.29) - open/close 36801 (time 1709787753.87 total 1176.83 last 28.07) - open/close 37043 (time 1709787763.88 total 1186.84 last 24.18) - open/close 37272 (time 1709787773.88 total 1196.84 last 22.89) - open/close 37520 (time 1709787783.93 total 1206.89 last 24.69) - open/close 37731 (time 1709787793.94 total 1216.90 last 21.08) - open/close 37981 (time 1709787803.97 total 1226.93 last 24.92) - open/close 38239 (time 1709787814.03 total 1236.99 last 25.66) - open/close 38479 (time 1709787824.04 total 1247.00 last 23.98) - open/close 38707 (time 1709787834.08 total 1257.04 last 22.69) - open/close 38976 (time 1709787844.11 total 1267.08 last 26.81) - open/close 39241 (time 1709787854.13 total 1277.09 last 26.46) - open/close 39550 (time 1709787864.15 total 1287.11 last 30.84) - open/close 39844 (time 1709787874.19 total 1297.15 last 29.29) - open/close 40000 (time 1709787879.47 total 1302.43 last 29.52) - open/close 40302 (time 1709787889.49 total 1312.45 last 30.13) - open/close 40594 (time 1709787899.50 total 1322.46 last 29.18) - open/close 40862 (time 1709787909.52 total 1332.48 last 26.76) - open/close 41133 (time 1709787919.54 total 1342.50 last 27.03) - open/close 41406 (time 1709787929.56 total 1352.52 last 27.25) - open/close 41691 (time 1709787939.57 total 1362.53 last 28.48) - open/close 41966 (time 1709787949.57 total 1372.53 last 27.48) - open/close 42232 (time 1709787959.59 total 1382.55 last 26.56) - open/close 42496 (time 1709787969.60 total 1392.56 last 26.37) - open/close 42757 (time 1709787979.61 total 1402.57 last 26.08) - open/close 43009 (time 1709787989.65 total 1412.61 last 25.10) - open/close 43278 (time 1709787999.68 total 1422.64 last 26.82) - open/close 43548 (time 1709788009.69 total 1432.65 last 26.96) - open/close 43814 (time 1709788019.70 total 1442.66 last 26.58) - open/close 44077 (time 1709788029.73 total 1452.69 last 26.21) - open/close 44330 (time 1709788039.78 total 1462.74 last 25.19) - open/close 44570 (time 1709788049.80 total 1472.76 last 23.95) - open/close 44825 (time 1709788059.80 total 1482.76 last 25.48) - open/close 45063 (time 1709788069.81 total 1492.77 last 23.79) - open/close 45314 (time 1709788079.83 total 1502.79 last 25.04) - open/close 45568 (time 1709788089.83 total 1512.80 last 25.39) - open/close 45824 (time 1709788099.86 total 1522.82 last 25.53) - open/close 46079 (time 1709788109.87 total 1532.83 last 25.48) - open/close 46329 (time 1709788119.90 total 1542.86 last 24.93) - open/close 46576 (time 1709788129.92 total 1552.88 last 24.64) - open/close 46822 (time 1709788139.95 total 1562.91 last 24.52) - open/close 47069 (time 1709788149.98 total 1572.94 last 24.63) - open/close 47280 (time 1709788160.00 total 1582.97 last 21.04) - open/close 47510 (time 1709788170.03 total 1592.99 last 22.93) - open/close 47745 (time 1709788180.06 total 1603.02 last 23.44) - open/close 47986 (time 1709788190.09 total 1613.05 last 24.03) - open/close 48223 (time 1709788200.14 total 1623.10 last 23.59) - open/close 48454 (time 1709788210.16 total 1633.12 last 23.05) - open/close 48681 (time 1709788220.19 total 1643.15 last 22.62) - open/close 48910 (time 1709788230.22 total 1653.18 last 22.82) - open/close 49128 (time 1709788240.27 total 1663.23 last 21.69) - open/close 49334 (time 1709788250.33 total 1673.29 last 20.48) - open/close 49574 (time 1709788260.33 total 1683.29 last 24.00) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1688.59 seconds: 29.43 ops/second - unlinked 0 (time 1709788267 ; total 0 ; last 0) - unlinked 10000 (time 1709788747 ; total 480 ; last 480) - unlinked 20000 (time 1709789143 ; total 876 ; last 396) - unlinked 30000 (time 1709789449 ; total 1182 ; last 306) - unlinked 40000 (time 1709789625 ; total 1358 ; last 176) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1419 seconds: 35.023254 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server unloading modules on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2