== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:24:36 (1773613476) start mds service on oleg125-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg125-server: detected 4 online CPUs by sysfs oleg125-server: MODOPTS_LIBCFS= oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg125-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1773613618.77 total 10.48 last 10.02) - open/close 257 (time 1773613628.86 total 20.57 last 15.06) - open/close 341 (time 1773613639.76 total 31.46 last 7.71) - open/close 452 (time 1773613649.76 total 41.46 last 11.10) - open/close 591 (time 1773613659.76 total 51.47 last 13.90) - open/close 699 (time 1773613669.80 total 61.50 last 10.76) - open/close 817 (time 1773613680.87 total 72.58 last 10.66) - open/close 911 (time 1773613690.89 total 82.59 last 9.38) - open/close 1022 (time 1773613701.15 total 92.86 last 10.81) - open/close 1088 (time 1773613711.79 total 103.50 last 6.20) - open/close 1171 (time 1773613722.58 total 114.29 last 7.69) - open/close 1264 (time 1773613732.60 total 124.31 last 9.29) - open/close 1348 (time 1773613743.08 total 134.79 last 8.01) - open/close 1453 (time 1773613753.10 total 144.81 last 10.48) - open/close 1559 (time 1773613763.49 total 155.19 last 10.21) - open/close 1674 (time 1773613774.11 total 165.81 last 10.83) - open/close 1754 (time 1773613784.82 total 176.53 last 7.47) - open/close 1863 (time 1773613795.22 total 186.93 last 10.48) - open/close 2009 (time 1773613806.23 total 197.94 last 13.26) - open/close 2122 (time 1773613816.93 total 208.63 last 10.56) - open/close 2241 (time 1773613827.67 total 219.37 last 11.08) - open/close 2335 (time 1773613837.96 total 229.67 last 9.13) - open/close 2448 (time 1773613848.27 total 239.98 last 10.96) - open/close 2548 (time 1773613858.70 total 250.41 last 9.59) - open/close 2656 (time 1773613868.94 total 260.65 last 10.54) - open/close 2755 (time 1773613878.96 total 270.67 last 9.88) - open/close 2872 (time 1773613889.64 total 281.35 last 10.96) - open/close 2989 (time 1773613899.93 total 291.63 last 11.37) - open/close 3098 (time 1773613910.49 total 302.20 last 10.32) - open/close 3221 (time 1773613920.50 total 312.20 last 12.29) - open/close 3379 (time 1773613930.99 total 322.70 last 15.05) - open/close 3519 (time 1773613941.92 total 333.62 last 12.82) - open/close 3667 (time 1773613951.95 total 343.65 last 14.75) - open/close 3820 (time 1773613961.96 total 353.66 last 15.28) - open/close 3915 (time 1773613971.99 total 363.69 last 9.47) - open/close 4019 (time 1773613982.14 total 373.85 last 10.24) - open/close 4186 (time 1773613993.14 total 384.85 last 15.18) - open/close 4276 (time 1773614003.43 total 395.14 last 8.74) - open/close 4380 (time 1773614013.73 total 405.44 last 10.10) - open/close 4485 (time 1773614024.27 total 415.97 last 9.97) - open/close 4598 (time 1773614034.91 total 426.62 last 10.61) - open/close 4715 (time 1773614044.93 total 436.63 last 11.69) - open/close 4846 (time 1773614054.94 total 446.65 last 13.08) - open/close 4965 (time 1773614065.52 total 457.22 last 11.25) - open/close 5091 (time 1773614075.53 total 467.24 last 12.58) - open/close 5205 (time 1773614085.59 total 477.30 last 11.33) - open/close 5333 (time 1773614095.77 total 487.48 last 12.57) - open/close 5434 (time 1773614106.41 total 498.11 last 9.50) - open/close 5541 (time 1773614117.01 total 508.71 last 10.10) - open/close 5683 (time 1773614127.04 total 518.74 last 14.15) - open/close 5815 (time 1773614138.10 total 529.81 last 11.93) - open/close 5943 (time 1773614148.62 total 540.33 last 12.17) - open/close 6076 (time 1773614158.72 total 550.43 last 13.17) - open/close 6210 (time 1773614168.93 total 560.64 last 13.12) - open/close 6322 (time 1773614179.69 total 571.40 last 10.41) - open/close 6432 (time 1773614190.17 total 581.88 last 10.50) - open/close 6550 (time 1773614200.20 total 591.90 last 11.77) - open/close 6642 (time 1773614211.15 total 602.86 last 8.40) - open/close 6737 (time 1773614221.54 total 613.24 last 9.15) - open/close 6838 (time 1773614231.77 total 623.48 last 9.87) - open/close 6949 (time 1773614242.31 total 634.01 last 10.54) - open/close 7064 (time 1773614252.88 total 644.59 last 10.87) - open/close 7181 (time 1773614263.46 total 655.16 last 11.07) - open/close 7291 (time 1773614273.97 total 665.68 last 10.46) - open/close 7407 (time 1773614284.52 total 676.23 last 10.99) - open/close 7547 (time 1773614294.55 total 686.26 last 13.96) - open/close 7726 (time 1773614305.51 total 697.21 last 16.33) - open/close 7860 (time 1773614316.27 total 707.97 last 12.45) - open/close 7959 (time 1773614326.37 total 718.08 last 9.80) - open/close 8054 (time 1773614336.75 total 728.46 last 9.15) - open/close 8190 (time 1773614346.78 total 738.48 last 13.57) - open/close 8304 (time 1773614357.41 total 749.12 last 10.72) - open/close 8438 (time 1773614367.90 total 759.61 last 12.78) - open/close 8593 (time 1773614378.08 total 769.79 last 15.23) - open/close 8710 (time 1773614388.98 total 780.69 last 10.73) - open/close 8859 (time 1773614399.79 total 791.49 last 13.79) - open/close 8972 (time 1773614410.22 total 801.93 last 10.83) - open/close 9104 (time 1773614421.21 total 812.92 last 12.02) - open/close 9202 (time 1773614431.48 total 823.19 last 9.54) - open/close 9307 (time 1773614441.95 total 833.66 last 10.03) - open/close 9440 (time 1773614452.30 total 844.00 last 12.86) - open/close 9588 (time 1773614462.62 total 854.33 last 14.33) - open/close 9679 (time 1773614472.94 total 864.65 last 8.82) - open/close 9822 (time 1773614484.00 total 875.71 last 12.94) - open/close 9924 (time 1773614494.03 total 885.74 last 10.17) - open/close 10000 (time 1773614502.86 total 894.57 last 8.60) - open/close 10157 (time 1773614513.70 total 905.41 last 14.49) - open/close 10232 (time 1773614524.24 total 915.95 last 7.12) - open/close 10356 (time 1773614534.61 total 926.32 last 11.95) - open/close 10463 (time 1773614545.39 total 937.10 last 9.93) - open/close 10590 (time 1773614555.43 total 947.14 last 12.65) - open/close 10705 (time 1773614565.43 total 957.14 last 11.50) - open/close 10834 (time 1773614576.30 total 968.01 last 11.87) - open/close 10927 (time 1773614587.24 total 978.95 last 8.50) - open/close 11008 (time 1773614597.92 total 989.62 last 7.59) - open/close 11103 (time 1773614607.94 total 999.65 last 9.47) - open/close 11233 (time 1773614617.96 total 1009.67 last 12.98) - open/close 11328 (time 1773614627.97 total 1019.68 last 9.49) - open/close 11470 (time 1773614638.00 total 1029.71 last 14.16) - open/close 11553 (time 1773614648.74 total 1040.44 last 7.73) - open/close 11664 (time 1773614659.75 total 1051.45 last 10.08) - open/close 11842 (time 1773614669.75 total 1061.46 last 17.79) - open/close 11932 (time 1773614679.91 total 1071.61 last 8.86) - open/close 12067 (time 1773614690.77 total 1082.48 last 12.43) - open/close 12167 (time 1773614701.03 total 1092.74 last 9.75) - open/close 12257 (time 1773614712.09 total 1103.80 last 8.14) - open/close 12387 (time 1773614722.93 total 1114.64 last 11.99) - open/close 12494 (time 1773614733.22 total 1124.93 last 10.40) - open/close 12597 (time 1773614743.59 total 1135.30 last 9.93) - open/close 12749 (time 1773614753.76 total 1145.47 last 14.94) - open/close 12845 (time 1773614764.24 total 1155.95 last 9.16) - open/close 12966 (time 1773614775.19 total 1166.90 last 11.05) - open/close 13094 (time 1773614785.69 total 1177.40 last 12.19) - open/close 13230 (time 1773614795.70 total 1187.41 last 13.59) - open/close 13322 (time 1773614806.01 total 1197.71 last 8.92) - open/close 13475 (time 1773614816.41 total 1208.12 last 14.71) - open/close 13569 (time 1773614826.43 total 1218.14 last 9.38) - open/close 13653 (time 1773614837.20 total 1228.91 last 7.80) - open/close 13750 (time 1773614847.22 total 1238.92 last 9.69) - open/close 13875 (time 1773614857.24 total 1248.94 last 12.47) - open/close 13971 (time 1773614867.25 total 1258.96 last 9.59) - open/close 14108 (time 1773614877.84 total 1269.55 last 12.93) - open/close 14194 (time 1773614888.67 total 1280.37 last 7.95) - open/close 14332 (time 1773614898.68 total 1290.39 last 13.78) - open/close 14428 (time 1773614909.72 total 1301.43 last 8.69) - open/close 14519 (time 1773614919.84 total 1311.55 last 8.99) - open/close 14614 (time 1773614929.89 total 1321.59 last 9.46) - open/close 14719 (time 1773614940.37 total 1332.07 last 10.02) - open/close 14816 (time 1773614950.38 total 1342.09 last 9.69) - open/close 14943 (time 1773614960.41 total 1352.11 last 12.67) - open/close 15076 (time 1773614971.23 total 1362.94 last 12.29) - open/close 15206 (time 1773614981.24 total 1372.95 last 12.99) - open/close 15302 (time 1773614991.98 total 1383.68 last 8.94) - open/close 15424 (time 1773615002.52 total 1394.22 last 11.57) - open/close 15525 (time 1773615012.71 total 1404.42 last 9.91) - open/close 15702 (time 1773615023.69 total 1415.40 last 16.12) - open/close 15805 (time 1773615033.72 total 1425.43 last 10.27) - open/close 15941 (time 1773615044.32 total 1436.02 last 12.83) - open/close 16054 (time 1773615055.14 total 1446.85 last 10.44) - open/close 16163 (time 1773615065.31 total 1457.01 last 10.72) - open/close 16255 (time 1773615075.32 total 1467.02 last 9.19) - open/close 16687 (time 1773615085.33 total 1477.04 last 43.14) - open/close 17018 (time 1773615095.33 total 1487.04 last 33.08) - open/close 17467 (time 1773615105.34 total 1497.05 last 44.86) - open/close 17974 (time 1773615115.35 total 1507.05 last 50.68) - open/close 18495 (time 1773615125.37 total 1517.07 last 51.99) - open/close 19052 (time 1773615135.37 total 1527.08 last 55.66) - open/close 19647 (time 1773615145.39 total 1537.09 last 59.44) - open/close 20000 (time 1773615151.42 total 1543.13 last 58.45) - open/close 20616 (time 1773615161.44 total 1553.14 last 61.53) - open/close 21318 (time 1773615171.45 total 1563.16 last 70.08) - open/close 21779 (time 1773615181.47 total 1573.17 last 46.04) - open/close 22231 (time 1773615191.50 total 1583.21 last 45.05) - open/close 22761 (time 1773615201.51 total 1593.21 last 52.95) - open/close 23384 (time 1773615211.52 total 1603.22 last 62.24) - open/close 23871 (time 1773615221.52 total 1613.23 last 48.68) - open/close 24441 (time 1773615231.53 total 1623.24 last 56.96) - open/close 24995 (time 1773615241.54 total 1633.24 last 55.36) - open/close 25556 (time 1773615251.54 total 1643.25 last 56.08) - open/close 26071 (time 1773615261.56 total 1653.27 last 51.40) - open/close 26554 (time 1773615271.57 total 1663.28 last 48.24) - open/close 27042 (time 1773615281.58 total 1673.28 last 48.78) - open/close 27638 (time 1773615291.59 total 1683.30 last 59.52) - open/close 28261 (time 1773615301.60 total 1693.31 last 62.25) - open/close 28781 (time 1773615311.60 total 1703.31 last 51.98) - open/close 29305 (time 1773615321.61 total 1713.31 last 52.38) - open/close 29803 (time 1773615331.61 total 1723.32 last 49.76) - open/close 30000 (time 1773615335.67 total 1727.37 last 48.62) - open/close 30504 (time 1773615345.68 total 1737.38 last 50.35) - open/close 31068 (time 1773615355.68 total 1747.39 last 56.36) - open/close 31599 (time 1773615365.69 total 1757.40 last 53.05) - open/close 32067 (time 1773615375.71 total 1767.42 last 46.71) - open/close 32504 (time 1773615385.73 total 1777.43 last 43.65) - open/close 32805 (time 1773615395.73 total 1787.44 last 30.09) - open/close 33242 (time 1773615405.76 total 1797.47 last 43.55) - open/close 33692 (time 1773615415.79 total 1807.50 last 44.87) - open/close 34039 (time 1773615425.81 total 1817.52 last 34.64) - open/close 34317 (time 1773615435.83 total 1827.54 last 27.74) - open/close 34733 (time 1773615445.86 total 1837.57 last 41.47) - open/close 35269 (time 1773615455.87 total 1847.58 last 53.56) - open/close 35824 (time 1773615465.88 total 1857.59 last 55.44) - open/close 36408 (time 1773615475.90 total 1867.61 last 58.28) - open/close 36984 (time 1773615485.90 total 1877.61 last 57.60) - open/close 37567 (time 1773615495.91 total 1887.61 last 58.26) - open/close 38098 (time 1773615505.92 total 1897.63 last 53.04) - open/close 38633 (time 1773615515.93 total 1907.63 last 53.46) - open/close 39194 (time 1773615525.95 total 1917.65 last 55.98) - open/close 39821 (time 1773615535.95 total 1927.66 last 62.69) - open/close 40000 (time 1773615538.34 total 1930.05 last 74.90) - open/close 40631 (time 1773615548.37 total 1940.08 last 62.87) - open/close 41141 (time 1773615558.39 total 1950.10 last 50.90) - open/close 41570 (time 1773615568.40 total 1960.10 last 42.89) - open/close 42039 (time 1773615578.42 total 1970.13 last 46.80) - open/close 42460 (time 1773615588.43 total 1980.14 last 42.04) - open/close 43081 (time 1773615598.45 total 1990.16 last 62.00) - open/close 43590 (time 1773615608.49 total 2000.19 last 50.71) - open/close 44088 (time 1773615618.51 total 2010.21 last 49.69) - open/close 44451 (time 1773615628.52 total 2020.23 last 36.24) - open/close 44913 (time 1773615638.54 total 2030.25 last 46.11) - open/close 45340 (time 1773615648.56 total 2040.26 last 42.65) - open/close 45804 (time 1773615658.56 total 2050.27 last 46.36) - open/close 46318 (time 1773615668.58 total 2060.29 last 51.29) - open/close 46846 (time 1773615678.60 total 2070.30 last 52.74) - open/close 47348 (time 1773615688.61 total 2080.31 last 50.15) - open/close 47880 (time 1773615698.61 total 2090.32 last 53.19) - open/close 48401 (time 1773615708.64 total 2100.35 last 51.94) - open/close 48923 (time 1773615718.65 total 2110.35 last 52.16) - open/close 49567 (time 1773615728.65 total 2120.36 last 64.36) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 2121.23 seconds: 23.40 ops/second - unlinked 0 (time 1773615736 ; total 0 ; last 0) - unlinked 10000 (time 1773615932 ; total 196 ; last 196) - unlinked 20000 (time 1773616114 ; total 378 ; last 182) - unlinked 30000 (time 1773616294 ; total 558 ; last 180) - unlinked 40000 (time 1773616473 ; total 737 ; last 179) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 898 seconds: 55.269489 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules via unload_modules_local on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local oleg125-server: modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2