== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:42:06 (1787370126) start mds service on oleg609-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg609-server' oleg609-server: oleg609-server.virtnet: executing load_modules_local oleg609-server: Loading modules from /home/green/git/lustre-release/lustre oleg609-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg609-server: detected 4 online CPUs by sysfs oleg609-server: MODOPTS_LIBCFS= oleg609-server: Force libcfs to create 2 CPU partitions oleg609-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg609-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg609-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg609-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg609-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg609-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg609-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Started lustre-OST0000 oleg609-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg609-client.virtnet: -o user_xattr,flock 192.168.206.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1787370242.08 total 10.57 last 18.82) - open/close 418 (time 1787370252.68 total 21.17 last 20.66) - open/close 599 (time 1787370262.69 total 31.18 last 18.09) - open/close 738 (time 1787370272.72 total 41.21 last 13.86) - open/close 982 (time 1787370283.23 total 51.72 last 23.21) - open/close 1213 (time 1787370293.85 total 62.34 last 21.75) - open/close 1362 (time 1787370303.88 total 72.37 last 14.86) - open/close 1486 (time 1787370314.60 total 83.10 last 11.56) - open/close 1637 (time 1787370324.92 total 93.41 last 14.64) - open/close 1865 (time 1787370335.02 total 103.51 last 22.58) - open/close 2047 (time 1787370345.71 total 114.20 last 17.02) - open/close 2173 (time 1787370355.71 total 124.20 last 12.59) - open/close 2319 (time 1787370366.59 total 135.08 last 13.42) - open/close 2584 (time 1787370377.48 total 145.97 last 24.34) - open/close 2777 (time 1787370387.69 total 156.18 last 18.91) - open/close 3043 (time 1787370397.70 total 166.19 last 26.56) - open/close 3271 (time 1787370407.85 total 176.34 last 22.47) - open/close 3474 (time 1787370418.27 total 186.76 last 19.47) - open/close 3769 (time 1787370428.29 total 196.78 last 29.45) - open/close 3988 (time 1787370438.82 total 207.31 last 20.80) - open/close 4231 (time 1787370448.92 total 217.41 last 24.06) - open/close 4485 (time 1787370459.88 total 228.37 last 23.17) - open/close 4742 (time 1787370470.69 total 239.18 last 23.77) - open/close 4996 (time 1787370480.71 total 249.20 last 25.37) - open/close 5125 (time 1787370491.93 total 260.42 last 11.49) - open/close 5349 (time 1787370502.94 total 271.43 last 20.34) - open/close 5610 (time 1787370513.89 total 282.38 last 23.83) - open/close 5875 (time 1787370524.53 total 293.02 last 24.92) - open/close 6175 (time 1787370534.55 total 303.04 last 29.92) - open/close 6425 (time 1787370544.57 total 313.06 last 24.97) - open/close 6695 (time 1787370554.58 total 323.07 last 26.96) - open/close 6971 (time 1787370565.60 total 334.09 last 25.05) - open/close 7217 (time 1787370575.98 total 344.47 last 23.69) - open/close 7458 (time 1787370585.99 total 354.48 last 24.08) - open/close 7749 (time 1787370596.92 total 365.41 last 26.64) - open/close 7897 (time 1787370607.96 total 376.45 last 13.40) - open/close 8145 (time 1787370617.97 total 386.47 last 24.77) - open/close 8394 (time 1787370628.93 total 397.43 last 22.72) - open/close 8553 (time 1787370638.96 total 407.45 last 15.86) - open/close 8811 (time 1787370648.96 total 417.46 last 25.79) - open/close 8989 (time 1787370658.98 total 427.47 last 17.78) - open/close 9266 (time 1787370669.88 total 438.37 last 25.41) - open/close 9509 (time 1787370682.22 total 450.71 last 19.70) - open/close 9705 (time 1787370694.27 total 462.77 last 16.26) - open/close 9966 (time 1787370705.19 total 473.68 last 23.91) - open/close 10210 (time 1787370715.82 total 484.31 last 22.95) - open/close 10481 (time 1787370726.57 total 495.06 last 25.21) - open/close 10665 (time 1787370736.76 total 505.25 last 18.06) - open/close 10921 (time 1787370747.53 total 516.02 last 23.79) - open/close 11106 (time 1787370757.54 total 526.03 last 18.48) - open/close 11307 (time 1787370769.27 total 537.76 last 17.13) - open/close 11477 (time 1787370779.61 total 548.10 last 16.44) - open/close 11731 (time 1787370789.62 total 558.11 last 25.37) - open/close 11969 (time 1787370799.84 total 568.33 last 23.30) - open/close 12224 (time 1787370809.86 total 578.35 last 25.45) - open/close 12497 (time 1787370819.88 total 588.37 last 27.25) - open/close 12767 (time 1787370829.90 total 598.39 last 26.94) - open/close 12982 (time 1787370839.97 total 608.46 last 21.35) - open/close 13137 (time 1787370849.98 total 618.47 last 15.49) - open/close 13239 (time 1787370860.58 total 629.07 last 9.62) - open/close 13378 (time 1787370871.58 total 640.07 last 12.63) - open/close 13624 (time 1787370882.35 total 650.84 last 22.85) - open/close 13923 (time 1787370892.37 total 660.86 last 29.84) - open/close 14083 (time 1787370902.37 total 670.86 last 15.99) - open/close 14212 (time 1787370912.93 total 681.42 last 12.21) - open/close 14309 (time 1787370923.26 total 691.75 last 9.39) - open/close 14496 (time 1787370933.55 total 702.04 last 18.18) - open/close 14727 (time 1787370944.34 total 712.83 last 21.41) - open/close 14968 (time 1787370955.86 total 724.36 last 20.91) - open/close 15110 (time 1787370965.88 total 734.37 last 14.18) - open/close 15372 (time 1787370975.89 total 744.38 last 26.17) - open/close 15522 (time 1787370986.52 total 755.01 last 14.11) - open/close 15800 (time 1787370998.94 total 767.43 last 22.39) - open/close 16000 (time 1787371011.30 total 779.79 last 16.18) - open/close 16164 (time 1787371024.12 total 792.61 last 12.80) - open/close 16396 (time 1787371034.78 total 803.28 last 21.75) - open/close 16626 (time 1787371044.79 total 813.28 last 22.99) - open/close 16871 (time 1787371054.80 total 823.29 last 24.48) - open/close 17056 (time 1787371065.51 total 834.00 last 17.27) - open/close 17248 (time 1787371075.76 total 844.25 last 18.73) - open/close 17385 (time 1787371085.80 total 854.29 last 13.65) - open/close 17520 (time 1787371095.99 total 864.48 last 13.25) - open/close 17636 (time 1787371106.41 total 874.90 last 11.12) - open/close 17735 (time 1787371117.56 total 886.05 last 8.88) - open/close 18010 (time 1787371127.59 total 896.08 last 27.43) - open/close 18214 (time 1787371137.76 total 906.25 last 20.06) - open/close 18478 (time 1787371147.77 total 916.26 last 26.38) - open/close 18632 (time 1787371158.39 total 926.88 last 14.50) - open/close 18935 (time 1787371169.21 total 937.71 last 27.98) - open/close 19140 (time 1787371180.28 total 948.77 last 18.53) - open/close 19395 (time 1787371190.40 total 958.89 last 25.21) - open/close 19628 (time 1787371200.79 total 969.28 last 22.42) - open/close 19885 (time 1787371210.80 total 979.29 last 25.69) - open/close 20000 (time 1787371213.28 total 981.77 last 46.27) - open/close 20756 (time 1787371223.29 total 991.78 last 75.56) - open/close 21530 (time 1787371233.29 total 1001.79 last 77.34) - open/close 22285 (time 1787371243.30 total 1011.79 last 75.47) - open/close 23029 (time 1787371253.30 total 1021.79 last 74.36) - open/close 23736 (time 1787371263.31 total 1031.80 last 70.67) - open/close 24458 (time 1787371273.31 total 1041.80 last 72.19) - open/close 25190 (time 1787371283.32 total 1051.81 last 73.15) - open/close 26015 (time 1787371293.32 total 1061.81 last 82.44) - open/close 26715 (time 1787371303.34 total 1071.83 last 69.88) - open/close 27489 (time 1787371313.35 total 1081.84 last 77.35) - open/close 28238 (time 1787371323.37 total 1091.86 last 74.69) - open/close 29039 (time 1787371333.40 total 1101.89 last 79.92) - open/close 29781 (time 1787371343.40 total 1111.89 last 74.15) - open/close 30000 (time 1787371346.37 total 1114.86 last 73.85) - open/close 30864 (time 1787371356.38 total 1124.87 last 86.35) - open/close 31705 (time 1787371366.38 total 1134.88 last 84.02) - open/close 32357 (time 1787371376.39 total 1144.88 last 65.15) - open/close 33018 (time 1787371386.39 total 1154.89 last 66.08) - open/close 33750 (time 1787371396.40 total 1164.89 last 73.19) - open/close 34595 (time 1787371406.40 total 1174.90 last 84.43) - open/close 35303 (time 1787371416.41 total 1184.90 last 70.75) - open/close 36099 (time 1787371426.42 total 1194.91 last 79.51) - open/close 36881 (time 1787371436.43 total 1204.92 last 78.14) - open/close 37594 (time 1787371446.43 total 1214.92 last 71.28) - open/close 38385 (time 1787371456.43 total 1224.93 last 79.09) - open/close 39275 (time 1787371466.44 total 1234.93 last 88.95) - open/close 40000 (time 1787371475.32 total 1243.81 last 81.62) - open/close 40708 (time 1787371485.34 total 1253.83 last 70.72) - open/close 41453 (time 1787371495.34 total 1263.83 last 74.44) - open/close 42197 (time 1787371505.36 total 1273.85 last 74.31) - open/close 42982 (time 1787371515.36 total 1283.85 last 78.45) - open/close 43667 (time 1787371525.37 total 1293.86 last 68.46) - open/close 44332 (time 1787371535.37 total 1303.86 last 66.50) - open/close 45102 (time 1787371545.38 total 1313.87 last 76.92) - open/close 45889 (time 1787371555.39 total 1323.89 last 78.57) - open/close 46518 (time 1787371565.41 total 1333.90 last 62.83) - open/close 47156 (time 1787371575.42 total 1343.91 last 63.71) - open/close 47920 (time 1787371585.43 total 1353.92 last 76.33) - open/close 48691 (time 1787371595.43 total 1363.92 last 77.08) - open/close 49444 (time 1787371605.43 total 1373.92 last 75.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1377.38 seconds: 36.03 ops/second - unlinked 0 (time 1787371613 ; total 0 ; last 0) - unlinked 10000 (time 1787371743 ; total 130 ; last 130) - unlinked 20000 (time 1787371874 ; total 261 ; last 131) - unlinked 30000 (time 1787372003 ; total 390 ; last 129) - unlinked 40000 (time 1787372129 ; total 516 ; last 126) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 641 seconds: 77.427460 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg609-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg609-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg609-server stop mds service on oleg609-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg609-server stop mds service on oleg609-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg609-server unloading modules via unload_modules_local on: 'oleg609-server' oleg609-server: oleg609-server.virtnet: executing unload_modules_local oleg609-server: modules unloaded. pdsh@oleg609-client: oleg609-client: ssh exited with exit code 2 pdsh@oleg609-client: oleg609-server: ssh exited with exit code 2 pdsh@oleg609-client: oleg609-client: ssh exited with exit code 2 pdsh@oleg609-client: oleg609-server: ssh exited with exit code 2