== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:05:52 (1774879552) start mds service on oleg415-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg415-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg415-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg415-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 80 (time 1774879666.29 total 11.35 last 7.05) - open/close 206 (time 1774879676.65 total 21.71 last 12.16) - open/close 361 (time 1774879689.33 total 34.39 last 12.22) - open/close 528 (time 1774879699.34 total 44.40 last 16.69) - open/close 638 (time 1774879710.12 total 55.18 last 10.20) - open/close 798 (time 1774879721.51 total 66.57 last 14.05) - open/close 852 (time 1774879732.72 total 77.78 last 4.82) - open/close 1043 (time 1774879743.16 total 88.22 last 18.30) - open/close 1212 (time 1774879755.19 total 100.25 last 14.04) - open/close 1367 (time 1774879765.19 total 110.25 last 15.50) - open/close 1501 (time 1774879775.54 total 120.59 last 12.95) - open/close 1592 (time 1774879786.28 total 131.34 last 8.47) - open/close 1740 (time 1774879797.03 total 142.09 last 13.77) - open/close 1966 (time 1774879807.94 total 153.00 last 20.71) - open/close 2079 (time 1774879817.97 total 163.03 last 11.26) - open/close 2244 (time 1774879830.37 total 175.43 last 13.31) - open/close 2339 (time 1774879840.95 total 186.01 last 8.98) - open/close 2393 (time 1774879852.85 total 197.91 last 4.54) - open/close 2587 (time 1774879862.86 total 207.92 last 19.38) - open/close 2787 (time 1774879873.73 total 218.79 last 18.40) - open/close 2898 (time 1774879884.27 total 229.33 last 10.53) - open/close 3011 (time 1774879894.70 total 239.76 last 10.83) - open/close 3074 (time 1774879904.71 total 249.77 last 6.30) - open/close 3194 (time 1774879914.74 total 259.80 last 11.96) - open/close 3308 (time 1774879924.78 total 269.84 last 11.35) - open/close 3423 (time 1774879934.98 total 280.04 last 11.28) - open/close 3558 (time 1774879946.43 total 291.49 last 11.79) - open/close 3701 (time 1774879956.46 total 301.51 last 14.27) - open/close 3822 (time 1774879967.15 total 312.21 last 11.32) - open/close 3983 (time 1774879977.17 total 322.23 last 16.07) - open/close 4089 (time 1774879987.17 total 332.23 last 10.59) - open/close 4238 (time 1774879997.81 total 342.87 last 14.01) - open/close 4425 (time 1774880007.81 total 352.87 last 18.70) - open/close 4604 (time 1774880018.42 total 363.47 last 16.88) - open/close 4843 (time 1774880028.43 total 373.48 last 23.87) - open/close 5021 (time 1774880039.36 total 384.42 last 16.28) - open/close 5166 (time 1774880049.36 total 394.42 last 14.50) - open/close 5390 (time 1774880060.40 total 405.46 last 20.29) - open/close 5590 (time 1774880070.95 total 416.01 last 18.96) - open/close 5775 (time 1774880084.02 total 429.08 last 14.16) - open/close 6021 (time 1774880094.02 total 439.08 last 24.59) - open/close 6112 (time 1774880104.06 total 449.12 last 9.07) - open/close 6272 (time 1774880115.74 total 460.80 last 13.69) - open/close 6503 (time 1774880126.13 total 471.19 last 22.24) - open/close 6640 (time 1774880136.68 total 481.74 last 12.99) - open/close 6823 (time 1774880146.70 total 491.76 last 18.27) - open/close 6969 (time 1774880156.71 total 501.77 last 14.59) - open/close 7151 (time 1774880166.72 total 511.77 last 18.19) - open/close 7373 (time 1774880176.72 total 521.78 last 22.19) - open/close 7600 (time 1774880187.44 total 532.49 last 21.19) - open/close 7806 (time 1774880197.47 total 542.53 last 20.54) - open/close 8075 (time 1774880207.49 total 552.55 last 26.84) - open/close 8267 (time 1774880217.50 total 562.55 last 19.19) - open/close 8341 (time 1774880227.51 total 572.57 last 7.39) - open/close 8426 (time 1774880237.53 total 582.59 last 8.48) - open/close 8626 (time 1774880247.88 total 592.93 last 19.33) - open/close 8854 (time 1774880257.91 total 602.97 last 22.73) - open/close 9072 (time 1774880267.91 total 612.97 last 21.79) - open/close 9129 (time 1774880278.61 total 623.66 last 5.33) - open/close 9278 (time 1774880289.67 total 634.73 last 13.46) - open/close 9420 (time 1774880300.03 total 645.09 last 13.71) - open/close 9583 (time 1774880310.05 total 655.11 last 16.27) - open/close 9743 (time 1774880321.03 total 666.09 last 14.57) - open/close 9900 (time 1774880331.19 total 676.25 last 15.46) - open/close 10000 (time 1774880338.59 total 683.65 last 13.51) - open/close 10138 (time 1774880348.60 total 693.66 last 13.79) - open/close 10198 (time 1774880358.68 total 703.73 last 5.96) - open/close 10359 (time 1774880369.41 total 714.47 last 15.00) - open/close 10554 (time 1774880379.65 total 724.71 last 19.05) - open/close 10743 (time 1774880389.65 total 734.71 last 18.90) - open/close 10944 (time 1774880399.74 total 744.80 last 19.91) - open/close 11278 (time 1774880410.62 total 755.68 last 30.71) - open/close 11453 (time 1774880421.41 total 766.47 last 16.21) - open/close 11683 (time 1774880431.47 total 776.53 last 22.87) - open/close 11891 (time 1774880441.70 total 786.76 last 20.33) - open/close 12081 (time 1774880452.55 total 797.61 last 17.52) - open/close 12302 (time 1774880465.15 total 810.20 last 17.54) - open/close 12492 (time 1774880475.31 total 820.37 last 18.69) - open/close 12663 (time 1774880486.28 total 831.34 last 15.59) - open/close 12747 (time 1774880496.39 total 841.44 last 8.31) - open/close 12863 (time 1774880507.35 total 852.41 last 10.58) - open/close 13018 (time 1774880518.21 total 863.27 last 14.28) - open/close 13147 (time 1774880528.63 total 873.69 last 12.38) - open/close 13296 (time 1774880538.65 total 883.71 last 14.87) - open/close 13448 (time 1774880549.38 total 894.44 last 14.17) - open/close 13541 (time 1774880560.25 total 905.31 last 8.56) - open/close 13628 (time 1774880570.27 total 915.33 last 8.68) - open/close 13720 (time 1774880581.19 total 926.25 last 8.43) - open/close 13847 (time 1774880591.43 total 936.49 last 12.40) - open/close 13949 (time 1774880602.17 total 947.22 last 9.50) - open/close 14151 (time 1774880612.39 total 957.45 last 19.76) - open/close 14397 (time 1774880623.22 total 968.28 last 22.72) - open/close 14586 (time 1774880633.22 total 978.28 last 18.89) - open/close 14808 (time 1774880643.94 total 989.00 last 20.71) - open/close 14931 (time 1774880653.95 total 999.01 last 12.29) - open/close 15257 (time 1774880663.97 total 1009.02 last 32.56) - open/close 15383 (time 1774880674.79 total 1019.85 last 11.64) - open/close 15653 (time 1774880685.43 total 1030.49 last 25.38) - open/close 15877 (time 1774880695.99 total 1041.04 last 21.22) - open/close 16118 (time 1774880708.75 total 1053.81 last 18.88) - open/close 16495 (time 1774880718.76 total 1063.82 last 37.68) - open/close 17222 (time 1774880728.76 total 1073.82 last 72.68) - open/close 17873 (time 1774880738.77 total 1083.83 last 65.02) - open/close 18492 (time 1774880748.77 total 1093.83 last 61.90) - open/close 19146 (time 1774880758.77 total 1103.83 last 65.40) - open/close 19741 (time 1774880768.78 total 1113.84 last 59.47) - open/close 20000 (time 1774880773.48 total 1118.54 last 55.03) - open/close 20502 (time 1774880783.50 total 1128.56 last 50.11) - open/close 21198 (time 1774880793.51 total 1138.57 last 69.51) - open/close 21850 (time 1774880803.52 total 1148.58 last 65.15) - open/close 22506 (time 1774880813.53 total 1158.59 last 65.56) - open/close 23193 (time 1774880823.54 total 1168.59 last 68.65) - open/close 23866 (time 1774880833.55 total 1178.60 last 67.24) - open/close 24401 (time 1774880843.56 total 1188.61 last 53.44) - open/close 25024 (time 1774880853.56 total 1198.62 last 62.29) - open/close 25657 (time 1774880863.56 total 1208.62 last 63.26) - open/close 26213 (time 1774880873.60 total 1218.66 last 55.40) - open/close 26738 (time 1774880883.62 total 1228.68 last 52.37) - open/close 27247 (time 1774880893.64 total 1238.70 last 50.80) - open/close 27759 (time 1774880903.65 total 1248.71 last 51.15) - open/close 28344 (time 1774880913.66 total 1258.71 last 58.48) - open/close 29084 (time 1774880923.67 total 1268.73 last 73.91) - open/close 29796 (time 1774880933.68 total 1278.74 last 71.12) - open/close 30000 (time 1774880937.27 total 1282.33 last 56.77) - open/close 30651 (time 1774880947.28 total 1292.34 last 65.07) - open/close 31343 (time 1774880957.29 total 1302.35 last 69.13) - open/close 31909 (time 1774880967.29 total 1312.35 last 56.59) - open/close 32482 (time 1774880977.30 total 1322.35 last 57.25) - open/close 33022 (time 1774880987.30 total 1332.36 last 53.99) - open/close 33697 (time 1774880997.30 total 1342.36 last 67.49) - open/close 34213 (time 1774881007.30 total 1352.36 last 51.60) - open/close 34687 (time 1774881017.31 total 1362.37 last 47.35) - open/close 35173 (time 1774881027.32 total 1372.38 last 48.56) - open/close 35766 (time 1774881037.34 total 1382.40 last 59.17) - open/close 36343 (time 1774881047.38 total 1392.44 last 57.44) - open/close 36848 (time 1774881057.39 total 1402.45 last 50.47) - open/close 37384 (time 1774881067.41 total 1412.47 last 53.51) - open/close 37930 (time 1774881077.41 total 1422.47 last 54.59) - open/close 38520 (time 1774881087.42 total 1432.48 last 58.93) - open/close 39110 (time 1774881097.42 total 1442.48 last 58.99) - open/close 39807 (time 1774881107.43 total 1452.48 last 69.67) - open/close 40000 (time 1774881110.39 total 1455.45 last 65.09) - open/close 40662 (time 1774881120.39 total 1465.45 last 66.18) - open/close 41402 (time 1774881130.40 total 1475.46 last 73.96) - open/close 42060 (time 1774881140.41 total 1485.47 last 65.74) - open/close 42666 (time 1774881150.42 total 1495.48 last 60.52) - open/close 43335 (time 1774881160.43 total 1505.49 last 66.85) - open/close 43911 (time 1774881170.44 total 1515.50 last 57.51) - open/close 44523 (time 1774881180.45 total 1525.51 last 61.17) - open/close 45132 (time 1774881190.46 total 1535.52 last 60.84) - open/close 45821 (time 1774881200.46 total 1545.52 last 68.87) - open/close 46475 (time 1774881210.47 total 1555.53 last 65.33) - open/close 47162 (time 1774881220.48 total 1565.54 last 68.68) - open/close 47749 (time 1774881230.50 total 1575.55 last 58.59) - open/close 48336 (time 1774881240.51 total 1585.57 last 58.63) - open/close 49031 (time 1774881250.52 total 1595.58 last 69.43) - open/close 49623 (time 1774881260.52 total 1605.58 last 59.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1605.78 seconds: 30.91 ops/second - unlinked 0 (time 1774881266 ; total 0 ; last 0) - unlinked 10000 (time 1774881438 ; total 172 ; last 172) - unlinked 20000 (time 1774881615 ; total 349 ; last 177) - unlinked 30000 (time 1774881790 ; total 524 ; last 175) - unlinked 40000 (time 1774881954 ; total 688 ; last 164) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 842 seconds: 58.945370 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server unloading modules via unload_modules_local on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local oleg415-server: modules unloaded. pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2