== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:09 (1709787069) start mds service on oleg149-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg149-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Started lustre-OST0000 oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg149-client.virtnet: -o user_xattr,flock oleg149-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1709787103.46 total 10.60 last 19.25) - open/close 441 (time 1709787114.33 total 21.47 last 21.80) - open/close 585 (time 1709787124.33 total 31.48 last 14.39) - open/close 652 (time 1709787137.31 total 44.45 last 5.16) - open/close 730 (time 1709787147.36 total 54.51 last 7.76) - open/close 908 (time 1709787157.91 total 65.05 last 16.88) - open/close 1061 (time 1709787168.03 total 75.17 last 15.11) - open/close 1242 (time 1709787178.44 total 85.58 last 17.39) - open/close 1361 (time 1709787189.01 total 96.15 last 11.26) - open/close 1567 (time 1709787199.72 total 106.86 last 19.22) - open/close 1808 (time 1709787210.64 total 117.78 last 22.08) - open/close 1965 (time 1709787220.97 total 128.11 last 15.19) - open/close 2147 (time 1709787231.72 total 138.86 last 16.93) - open/close 2275 (time 1709787241.73 total 148.87 last 12.80) - open/close 2414 (time 1709787251.80 total 158.94 last 13.79) - open/close 2652 (time 1709787263.11 total 170.25 last 21.05) - open/close 2764 (time 1709787273.77 total 180.91 last 10.51) - open/close 2986 (time 1709787283.83 total 190.97 last 22.07) - open/close 3206 (time 1709787293.84 total 200.98 last 21.98) - open/close 3490 (time 1709787304.32 total 211.46 last 27.10) - open/close 3679 (time 1709787314.51 total 221.65 last 18.53) - open/close 3822 (time 1709787325.04 total 232.19 last 13.58) - open/close 3960 (time 1709787335.05 total 242.19 last 13.79) - open/close 4144 (time 1709787345.06 total 252.20 last 18.39) - open/close 4322 (time 1709787355.07 total 262.21 last 17.78) - open/close 4561 (time 1709787365.34 total 272.48 last 23.27) - open/close 4702 (time 1709787375.80 total 282.94 last 13.47) - open/close 4890 (time 1709787386.37 total 293.52 last 17.78) - open/close 5065 (time 1709787397.06 total 304.20 last 16.37) - open/close 5258 (time 1709787407.07 total 314.21 last 19.29) - open/close 5449 (time 1709787417.07 total 324.21 last 19.09) - open/close 5570 (time 1709787427.14 total 334.28 last 12.02) - open/close 5784 (time 1709787437.14 total 344.28 last 21.40) - open/close 5917 (time 1709787456.00 total 363.14 last 7.05) - open/close 6112 (time 1709787466.60 total 373.74 last 18.40) - open/close 6291 (time 1709787477.46 total 384.60 last 16.48) - open/close 6451 (time 1709787487.57 total 394.72 last 15.81) - open/close 6655 (time 1709787497.58 total 404.72 last 20.39) - open/close 6785 (time 1709787508.14 total 415.28 last 12.32) - open/close 6983 (time 1709787519.18 total 426.32 last 17.93) - open/close 7182 (time 1709787529.19 total 436.33 last 19.88) - open/close 7365 (time 1709787539.45 total 446.59 last 17.83) - open/close 7637 (time 1709787549.87 total 457.02 last 26.09) - open/close 7721 (time 1709787560.90 total 468.04 last 7.62) - open/close 7894 (time 1709787570.92 total 478.06 last 17.27) - open/close 7993 (time 1709787581.81 total 488.95 last 9.09) - open/close 8100 (time 1709787592.28 total 499.42 last 10.21) - open/close 8338 (time 1709787603.10 total 510.24 last 22.01) - open/close 8512 (time 1709787613.84 total 520.99 last 16.19) - open/close 8650 (time 1709787624.57 total 531.71 last 12.87) - open/close 8848 (time 1709787634.58 total 541.72 last 19.78) - open/close 9146 (time 1709787644.94 total 552.08 last 28.75) - open/close 9408 (time 1709787654.95 total 562.09 last 26.17) - open/close 9590 (time 1709787665.89 total 573.03 last 16.63) - open/close 10000 (time 1709787673.80 total 580.94 last 51.83) - open/close 10579 (time 1709787683.81 total 590.95 last 57.88) - open/close 11124 (time 1709787693.86 total 601.00 last 54.19) - open/close 11672 (time 1709787703.87 total 611.01 last 54.78) - open/close 12224 (time 1709787713.88 total 621.02 last 55.15) - open/close 12825 (time 1709787723.90 total 631.04 last 59.95) - open/close 13389 (time 1709787733.91 total 641.05 last 56.35) - open/close 13905 (time 1709787743.91 total 651.05 last 51.59) - open/close 14391 (time 1709787753.92 total 661.06 last 48.58) - open/close 14857 (time 1709787763.92 total 671.06 last 46.58) - open/close 15330 (time 1709787773.93 total 681.07 last 47.23) - open/close 15829 (time 1709787783.94 total 691.08 last 49.86) - open/close 16230 (time 1709787793.94 total 701.09 last 40.08) - open/close 16637 (time 1709787803.95 total 711.09 last 40.67) - open/close 17104 (time 1709787813.95 total 721.09 last 46.69) - open/close 17544 (time 1709787823.97 total 731.11 last 43.95) - open/close 17897 (time 1709787833.98 total 741.12 last 35.25) - open/close 18264 (time 1709787843.99 total 751.13 last 36.68) - open/close 18639 (time 1709787853.99 total 761.13 last 37.48) - open/close 18891 (time 1709787871.29 total 778.43 last 14.57) - open/close 19275 (time 1709787881.29 total 788.44 last 38.37) - open/close 19659 (time 1709787891.31 total 798.45 last 38.33) - open/close 20000 (time 1709787900.47 total 807.61 last 37.26) - open/close 20360 (time 1709787910.48 total 817.62 last 35.96) - open/close 20706 (time 1709787920.48 total 827.62 last 34.58) - open/close 21037 (time 1709787930.49 total 837.63 last 33.08) - open/close 21375 (time 1709787940.49 total 847.63 last 33.78) - open/close 21714 (time 1709787950.51 total 857.65 last 33.85) - open/close 22116 (time 1709787960.53 total 867.67 last 40.09) - open/close 22451 (time 1709787970.55 total 877.69 last 33.45) - open/close 22833 (time 1709787980.56 total 887.71 last 38.14) - open/close 23246 (time 1709787990.58 total 897.72 last 41.22) - open/close 23647 (time 1709788000.61 total 907.75 last 40.01) - open/close 24034 (time 1709788010.63 total 917.77 last 38.61) - open/close 24395 (time 1709788020.64 total 927.78 last 36.07) - open/close 24800 (time 1709788030.66 total 937.80 last 40.43) - open/close 25171 (time 1709788040.67 total 947.81 last 37.03) - open/close 25545 (time 1709788050.70 total 957.84 last 37.30) - open/close 25921 (time 1709788060.73 total 967.87 last 37.49) - open/close 26294 (time 1709788070.73 total 977.87 last 37.29) - open/close 26672 (time 1709788080.74 total 987.88 last 37.77) - open/close 27028 (time 1709788090.76 total 997.90 last 35.52) - open/close 27381 (time 1709788100.79 total 1007.93 last 35.21) - open/close 27716 (time 1709788110.81 total 1017.95 last 33.42) - open/close 28052 (time 1709788120.83 total 1027.97 last 33.56) - open/close 28398 (time 1709788130.83 total 1037.97 last 34.59) - open/close 28733 (time 1709788140.84 total 1047.98 last 33.46) - open/close 29045 (time 1709788150.85 total 1057.99 last 31.18) - open/close 29344 (time 1709788160.88 total 1068.02 last 29.81) - open/close 29647 (time 1709788170.91 total 1078.05 last 30.20) - open/close 29953 (time 1709788180.93 total 1088.07 last 30.55) - open/close 30255 (time 1709788190.95 total 1098.09 last 30.13) - open/close 30565 (time 1709788200.98 total 1108.12 last 30.91) - open/close 30902 (time 1709788211.00 total 1118.14 last 33.61) - open/close 31227 (time 1709788221.01 total 1128.15 last 32.49) - open/close 31578 (time 1709788231.01 total 1138.16 last 35.07) - open/close 31914 (time 1709788241.04 total 1148.18 last 33.50) - open/close 32238 (time 1709788251.08 total 1158.22 last 32.28) - open/close 32540 (time 1709788261.10 total 1168.24 last 30.16) - open/close 32838 (time 1709788271.10 total 1178.24 last 29.78) - open/close 33147 (time 1709788281.11 total 1188.25 last 30.88) - open/close 33470 (time 1709788291.14 total 1198.28 last 32.22) - open/close 33780 (time 1709788301.16 total 1208.30 last 30.92) - open/close 34088 (time 1709788311.18 total 1218.32 last 30.74) - open/close 34403 (time 1709788321.19 total 1228.33 last 31.47) - open/close 34721 (time 1709788331.20 total 1238.34 last 31.77) - open/close 35033 (time 1709788341.21 total 1248.35 last 31.18) - open/close 35342 (time 1709788351.21 total 1258.35 last 30.88) - open/close 35631 (time 1709788361.23 total 1268.37 last 28.85) - open/close 35941 (time 1709788371.24 total 1278.38 last 30.98) - open/close 36246 (time 1709788381.25 total 1288.39 last 30.47) - open/close 36531 (time 1709788391.28 total 1298.42 last 28.41) - open/close 36829 (time 1709788401.28 total 1308.43 last 29.78) - open/close 37139 (time 1709788411.32 total 1318.46 last 30.91) - open/close 37421 (time 1709788421.34 total 1328.49 last 28.12) - open/close 37704 (time 1709788431.36 total 1338.50 last 28.27) - open/close 37983 (time 1709788441.37 total 1348.51 last 27.86) - open/close 38267 (time 1709788451.37 total 1358.52 last 28.40) - open/close 38555 (time 1709788461.41 total 1368.55 last 28.70) - open/close 38802 (time 1709788471.44 total 1378.58 last 24.62) - open/close 39071 (time 1709788481.48 total 1388.62 last 26.79) - open/close 39329 (time 1709788491.51 total 1398.65 last 25.73) - open/close 39605 (time 1709788501.53 total 1408.67 last 27.55) - open/close 39886 (time 1709788511.55 total 1418.69 last 28.03) - open/close 40000 (time 1709788515.59 total 1422.73 last 28.21) - open/close 40266 (time 1709788525.60 total 1432.74 last 26.58) - open/close 40538 (time 1709788535.62 total 1442.76 last 27.15) - open/close 40807 (time 1709788545.63 total 1452.77 last 26.87) - open/close 41061 (time 1709788555.63 total 1462.77 last 25.40) - open/close 41326 (time 1709788565.64 total 1472.78 last 26.46) - open/close 41582 (time 1709788575.66 total 1482.80 last 25.56) - open/close 41831 (time 1709788585.69 total 1492.83 last 24.83) - open/close 42087 (time 1709788595.73 total 1502.87 last 25.49) - open/close 42355 (time 1709788605.75 total 1512.90 last 26.74) - open/close 42617 (time 1709788615.79 total 1522.93 last 26.12) - open/close 42891 (time 1709788625.81 total 1532.95 last 27.34) - open/close 43160 (time 1709788635.82 total 1542.96 last 26.88) - open/close 43423 (time 1709788645.85 total 1552.99 last 26.22) - open/close 43683 (time 1709788655.87 total 1563.01 last 25.94) - open/close 43935 (time 1709788665.89 total 1573.03 last 25.14) - open/close 44178 (time 1709788675.89 total 1583.03 last 24.30) - open/close 44425 (time 1709788685.92 total 1593.07 last 24.62) - open/close 44663 (time 1709788695.95 total 1603.09 last 23.75) - open/close 44909 (time 1709788705.95 total 1613.10 last 24.58) - open/close 45155 (time 1709788715.96 total 1623.10 last 24.58) - open/close 45409 (time 1709788725.97 total 1633.11 last 25.38) - open/close 45659 (time 1709788736.00 total 1643.14 last 24.92) - open/close 45892 (time 1709788746.01 total 1653.15 last 23.27) - open/close 46102 (time 1709788756.02 total 1663.16 last 20.98) - open/close 46319 (time 1709788766.02 total 1673.16 last 21.69) - open/close 46534 (time 1709788776.04 total 1683.18 last 21.46) - open/close 46755 (time 1709788786.05 total 1693.20 last 22.07) - open/close 46997 (time 1709788796.07 total 1703.21 last 24.16) - open/close 47234 (time 1709788806.07 total 1713.21 last 23.70) - open/close 47476 (time 1709788816.08 total 1723.22 last 24.18) - open/close 47704 (time 1709788826.13 total 1733.27 last 22.70) - open/close 47942 (time 1709788836.14 total 1743.28 last 23.76) - open/close 48175 (time 1709788846.17 total 1753.31 last 23.24) - open/close 48409 (time 1709788856.20 total 1763.34 last 23.32) - open/close 48642 (time 1709788866.24 total 1773.38 last 23.21) - open/close 48885 (time 1709788876.25 total 1783.40 last 24.27) - open/close 49124 (time 1709788886.26 total 1793.40 last 23.89) - open/close 49365 (time 1709788896.28 total 1803.43 last 24.03) - open/close 49598 (time 1709788906.32 total 1813.46 last 23.22) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1817.89 seconds: 27.34 ops/second - unlinked 0 (time 1709788912 ; total 0 ; last 0) - unlinked 10000 (time 1709789437 ; total 525 ; last 525) - unlinked 20000 (time 1709789842 ; total 930 ; last 405) - unlinked 30000 (time 1709790131 ; total 1219 ; last 289) - unlinked 40000 (time 1709790262 ; total 1350 ; last 131) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1396 seconds: 35.600288 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg149-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg149-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg149-server unloading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2