== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:21:33 (1789114893) start mds service on oleg648-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing load_modules_local oleg648-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg648-server: Loading modules from /home/green/git/lustre-release/lustre oleg648-server: detected 4 online CPUs by sysfs oleg648-server: MODOPTS_LIBCFS= oleg648-server: Force libcfs to create 2 CPU partitions oleg648-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg648-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg648-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg648-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg648-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Started lustre-OST0000 oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg648-client.virtnet: -o user_xattr,flock 192.168.206.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 110 (time 1789114999.33 total 10.01 last 10.99) - open/close 276 (time 1789115010.04 total 20.72 last 15.51) - open/close 543 (time 1789115020.69 total 31.37 last 25.07) - open/close 798 (time 1789115031.57 total 42.25 last 23.42) - open/close 1027 (time 1789115041.74 total 52.42 last 22.53) - open/close 1285 (time 1789115051.98 total 62.66 last 25.19) - open/close 1513 (time 1789115062.75 total 73.43 last 21.17) - open/close 1747 (time 1789115072.85 total 83.53 last 23.17) - open/close 2033 (time 1789115083.60 total 94.28 last 26.60) - open/close 2290 (time 1789115094.30 total 104.99 last 24.01) - open/close 2441 (time 1789115104.34 total 115.03 last 15.04) - open/close 2532 (time 1789115114.35 total 125.03 last 9.09) - open/close 2781 (time 1789115124.35 total 135.03 last 24.90) - open/close 3036 (time 1789115134.74 total 145.42 last 24.54) - open/close 3270 (time 1789115144.77 total 155.45 last 23.34) - open/close 3560 (time 1789115154.77 total 165.45 last 29.00) - open/close 3806 (time 1789115165.07 total 175.75 last 23.87) - open/close 4103 (time 1789115176.02 total 186.70 last 27.12) - open/close 4380 (time 1789115186.04 total 196.72 last 27.65) - open/close 4586 (time 1789115197.14 total 207.82 last 18.56) - open/close 4882 (time 1789115207.32 total 218.00 last 29.07) - open/close 5158 (time 1789115217.37 total 228.05 last 27.47) - open/close 5409 (time 1789115227.37 total 238.05 last 25.09) - open/close 5654 (time 1789115237.46 total 248.14 last 24.29) - open/close 5933 (time 1789115247.82 total 258.50 last 26.92) - open/close 6188 (time 1789115258.07 total 268.75 last 24.88) - open/close 6422 (time 1789115268.24 total 278.92 last 23.01) - open/close 6696 (time 1789115279.32 total 290.00 last 24.73) - open/close 6913 (time 1789115289.32 total 300.00 last 21.69) - open/close 7102 (time 1789115299.94 total 310.63 last 17.80) - open/close 7385 (time 1789115309.96 total 320.64 last 28.25) - open/close 7626 (time 1789115319.97 total 330.65 last 24.08) - open/close 7899 (time 1789115330.57 total 341.25 last 25.77) - open/close 8123 (time 1789115340.57 total 351.25 last 22.40) - open/close 8381 (time 1789115350.58 total 361.26 last 25.77) - open/close 8588 (time 1789115361.42 total 372.10 last 19.10) - open/close 8748 (time 1789115371.55 total 382.23 last 15.80) - open/close 9031 (time 1789115382.46 total 393.14 last 25.94) - open/close 9237 (time 1789115392.48 total 403.16 last 20.56) - open/close 9454 (time 1789115402.78 total 413.46 last 21.06) - open/close 9604 (time 1789115413.27 total 423.95 last 14.31) - open/close 9828 (time 1789115424.54 total 435.22 last 19.88) - open/close 10000 (time 1789115431.30 total 441.98 last 25.43) - open/close 10257 (time 1789115441.33 total 452.01 last 25.63) - open/close 10451 (time 1789115451.36 total 462.04 last 19.33) - open/close 10812 (time 1789115461.79 total 472.47 last 34.62) - open/close 10914 (time 1789115472.59 total 483.27 last 9.45) - open/close 11099 (time 1789115483.29 total 493.97 last 17.29) - open/close 11256 (time 1789115493.33 total 504.01 last 15.64) - open/close 11532 (time 1789115503.34 total 514.02 last 27.56) - open/close 11759 (time 1789115513.36 total 524.04 last 22.67) - open/close 11914 (time 1789115524.51 total 535.19 last 13.90) - open/close 12169 (time 1789115534.51 total 545.20 last 25.48) - open/close 12387 (time 1789115544.53 total 555.21 last 21.77) - open/close 12653 (time 1789115554.54 total 565.22 last 26.56) - open/close 12929 (time 1789115565.53 total 576.21 last 25.12) - open/close 13206 (time 1789115576.27 total 586.95 last 25.78) - open/close 13397 (time 1789115586.51 total 597.19 last 18.66) - open/close 13636 (time 1789115597.40 total 608.08 last 21.96) - open/close 13806 (time 1789115608.92 total 619.60 last 14.75) - open/close 14054 (time 1789115619.43 total 630.12 last 23.59) - open/close 14316 (time 1789115629.91 total 640.59 last 25.02) - open/close 14563 (time 1789115640.28 total 650.96 last 23.82) - open/close 14810 (time 1789115651.03 total 661.71 last 22.96) - open/close 15101 (time 1789115662.12 total 672.80 last 26.26) - open/close 15230 (time 1789115672.18 total 682.86 last 12.82) - open/close 15292 (time 1789115682.89 total 693.57 last 5.79) - open/close 15427 (time 1789115692.92 total 703.60 last 13.46) - open/close 15754 (time 1789115702.93 total 713.62 last 32.64) - open/close 15971 (time 1789115713.05 total 723.73 last 21.46) - open/close 16223 (time 1789115723.61 total 734.29 last 23.86) - open/close 16529 (time 1789115734.45 total 745.13 last 28.24) - open/close 16815 (time 1789115744.45 total 755.13 last 28.59) - open/close 17001 (time 1789115754.46 total 765.14 last 18.58) - open/close 17200 (time 1789115764.46 total 775.14 last 19.89) - open/close 17456 (time 1789115775.18 total 785.86 last 23.88) - open/close 17725 (time 1789115785.68 total 796.37 last 25.61) - open/close 18004 (time 1789115795.69 total 806.37 last 27.88) - open/close 18169 (time 1789115806.17 total 816.85 last 15.74) - open/close 18405 (time 1789115817.21 total 827.89 last 21.37) - open/close 18625 (time 1789115827.22 total 837.90 last 21.98) - open/close 18847 (time 1789115837.74 total 848.42 last 21.11) - open/close 19119 (time 1789115848.40 total 859.08 last 25.50) - open/close 19402 (time 1789115859.35 total 870.03 last 25.86) - open/close 19628 (time 1789115869.65 total 880.33 last 21.93) - open/close 19809 (time 1789115880.46 total 891.14 last 16.74) - open/close 19980 (time 1789115890.52 total 901.20 last 17.00) - open/close 20145 (time 1789115900.75 total 911.43 last 16.12) - open/close 20242 (time 1789115910.77 total 921.45 last 9.68) - open/close 20389 (time 1789115921.31 total 931.99 last 13.94) - open/close 20565 (time 1789115931.89 total 942.57 last 16.64) - open/close 20752 (time 1789115941.90 total 952.58 last 18.70) - open/close 21564 (time 1789115951.90 total 962.58 last 81.16) - open/close 22298 (time 1789115961.91 total 972.59 last 73.35) - open/close 23087 (time 1789115971.91 total 982.59 last 78.89) - open/close 23831 (time 1789115981.92 total 992.60 last 74.35) - open/close 24628 (time 1789115991.92 total 1002.60 last 79.65) - open/close 25358 (time 1789116001.94 total 1012.62 last 72.89) - open/close 26131 (time 1789116011.95 total 1022.63 last 77.24) - open/close 26864 (time 1789116021.95 total 1032.63 last 73.27) - open/close 27702 (time 1789116031.95 total 1042.63 last 83.79) - open/close 28444 (time 1789116041.98 total 1052.66 last 73.98) - open/close 29166 (time 1789116051.99 total 1062.67 last 72.13) - open/close 29875 (time 1789116061.99 total 1072.67 last 70.89) - open/close 30682 (time 1789116072.00 total 1082.68 last 80.62) - open/close 31537 (time 1789116082.00 total 1092.68 last 85.47) - open/close 32290 (time 1789116092.01 total 1102.69 last 75.24) - open/close 32918 (time 1789116102.01 total 1112.69 last 62.80) - open/close 33700 (time 1789116112.02 total 1122.70 last 78.18) - open/close 34338 (time 1789116122.02 total 1132.70 last 63.78) - open/close 34914 (time 1789116132.03 total 1142.71 last 57.56) - open/close 35715 (time 1789116142.05 total 1152.73 last 79.91) - open/close 36465 (time 1789116152.06 total 1162.74 last 74.93) - open/close 37209 (time 1789116162.06 total 1172.75 last 74.35) - open/close 38014 (time 1789116172.07 total 1182.75 last 80.47) - open/close 38813 (time 1789116182.07 total 1192.75 last 79.85) - open/close 39564 (time 1789116192.09 total 1202.77 last 75.00) - open/close 40000 (time 1789116197.70 total 1208.38 last 77.76) - open/close 40721 (time 1789116207.71 total 1218.40 last 71.96) - open/close 41448 (time 1789116217.72 total 1228.40 last 72.64) - open/close 42212 (time 1789116227.75 total 1238.43 last 76.21) - open/close 42861 (time 1789116237.76 total 1248.44 last 64.86) - open/close 43401 (time 1789116247.78 total 1258.46 last 53.88) - open/close 43988 (time 1789116257.78 total 1268.46 last 58.66) - open/close 44657 (time 1789116267.79 total 1278.47 last 66.84) - open/close 45399 (time 1789116277.80 total 1288.48 last 74.14) - open/close 46163 (time 1789116287.80 total 1298.48 last 76.38) - open/close 46999 (time 1789116297.81 total 1308.49 last 83.52) - open/close 47762 (time 1789116307.81 total 1318.50 last 76.29) - open/close 48548 (time 1789116317.82 total 1328.51 last 78.52) - open/close 49340 (time 1789116327.83 total 1338.51 last 79.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1342.23 seconds: 36.98 ops/second - unlinked 0 (time 1789116336 ; total 0 ; last 0) - unlinked 10000 (time 1789116478 ; total 142 ; last 142) - unlinked 20000 (time 1789116610 ; total 274 ; last 132) - unlinked 30000 (time 1789116744 ; total 408 ; last 134) - unlinked 40000 (time 1789116883 ; total 547 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 671 seconds: 73.965721 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg648-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg648-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg648-server stop mds service on oleg648-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg648-server stop mds service on oleg648-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg648-server unloading modules via unload_modules_local on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing unload_modules_local oleg648-server: modules unloaded. pdsh@oleg648-client: oleg648-client: ssh exited with exit code 2 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 2 pdsh@oleg648-client: oleg648-client: ssh exited with exit code 2 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 2