== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:15:07 (1784049307) start mds service on oleg213-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg213-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 283 (time 1784049405.09 total 10.31 last 27.45) - open/close 374 (time 1784049415.09 total 20.31 last 9.10) - open/close 705 (time 1784049426.15 total 31.37 last 29.94) - open/close 902 (time 1784049438.30 total 43.52 last 16.21) - open/close 1137 (time 1784049448.32 total 53.54 last 23.45) - open/close 1297 (time 1784049458.52 total 63.74 last 15.69) - open/close 1481 (time 1784049468.52 total 73.74 last 18.40) - open/close 1613 (time 1784049479.49 total 84.71 last 12.04) - open/close 1845 (time 1784049489.81 total 95.03 last 22.47) - open/close 1995 (time 1784049499.93 total 105.14 last 14.83) - open/close 2240 (time 1784049510.15 total 115.37 last 23.95) - open/close 2467 (time 1784049520.35 total 125.57 last 22.26) - open/close 2703 (time 1784049531.00 total 136.22 last 22.17) - open/close 2956 (time 1784049541.00 total 146.22 last 25.29) - open/close 3026 (time 1784049551.74 total 156.96 last 6.52) - open/close 3274 (time 1784049562.80 total 168.02 last 22.44) - open/close 3529 (time 1784049573.74 total 178.96 last 23.31) - open/close 3771 (time 1784049584.13 total 189.34 last 23.30) - open/close 3965 (time 1784049594.14 total 199.36 last 19.37) - open/close 4212 (time 1784049604.51 total 209.73 last 23.82) - open/close 4481 (time 1784049615.59 total 220.81 last 24.27) - open/close 4748 (time 1784049625.97 total 231.18 last 25.74) - open/close 5027 (time 1784049635.99 total 241.21 last 27.84) - open/close 5167 (time 1784049645.99 total 251.21 last 13.99) - open/close 5376 (time 1784049656.01 total 261.22 last 20.87) - open/close 5592 (time 1784049666.15 total 271.36 last 21.30) - open/close 5900 (time 1784049676.15 total 281.36 last 30.79) - open/close 6114 (time 1784049686.26 total 291.48 last 21.17) - open/close 6420 (time 1784049697.30 total 302.52 last 27.70) - open/close 6575 (time 1784049709.93 total 315.15 last 12.27) - open/close 6727 (time 1784049719.93 total 325.15 last 15.20) - open/close 6989 (time 1784049730.28 total 335.49 last 25.33) - open/close 7257 (time 1784049740.28 total 345.50 last 26.79) - open/close 7497 (time 1784049750.29 total 355.51 last 23.98) - open/close 7696 (time 1784049760.32 total 365.53 last 19.85) - open/close 7784 (time 1784049770.40 total 375.62 last 8.73) - open/close 7944 (time 1784049781.21 total 386.43 last 14.80) - open/close 8085 (time 1784049791.23 total 396.45 last 14.08) - open/close 8194 (time 1784049801.25 total 406.47 last 10.87) - open/close 8364 (time 1784049812.06 total 417.28 last 15.73) - open/close 8527 (time 1784049822.96 total 428.18 last 14.95) - open/close 8629 (time 1784049833.08 total 438.30 last 10.08) - open/close 8831 (time 1784049844.00 total 449.22 last 18.50) - open/close 9053 (time 1784049854.05 total 459.27 last 22.09) - open/close 9288 (time 1784049864.54 total 469.76 last 22.41) - open/close 9452 (time 1784049874.54 total 479.76 last 16.40) - open/close 9579 (time 1784049885.03 total 490.25 last 12.11) - open/close 9705 (time 1784049897.95 total 503.17 last 9.75) - open/close 9841 (time 1784049907.96 total 513.17 last 13.60) - open/close 9962 (time 1784049919.48 total 524.70 last 10.50) - open/close 10188 (time 1784049929.49 total 534.70 last 22.59) - open/close 10355 (time 1784049939.69 total 544.91 last 16.36) - open/close 10572 (time 1784049950.65 total 555.87 last 19.80) - open/close 10825 (time 1784049963.16 total 568.38 last 20.22) - open/close 11033 (time 1784049973.19 total 578.41 last 20.75) - open/close 11308 (time 1784049983.19 total 588.41 last 27.49) - open/close 11543 (time 1784049993.21 total 598.42 last 23.47) - open/close 11777 (time 1784050003.62 total 608.84 last 22.47) - open/close 11995 (time 1784050014.30 total 619.52 last 20.41) - open/close 12212 (time 1784050024.44 total 629.66 last 21.40) - open/close 12499 (time 1784050034.44 total 639.66 last 28.70) - open/close 12751 (time 1784050044.92 total 650.14 last 24.04) - open/close 13035 (time 1784050055.68 total 660.90 last 26.39) - open/close 13294 (time 1784050066.61 total 671.83 last 23.71) - open/close 13550 (time 1784050077.48 total 682.70 last 23.55) - open/close 13733 (time 1784050087.97 total 693.19 last 17.44) - open/close 13962 (time 1784050098.22 total 703.44 last 22.33) - open/close 14118 (time 1784050108.70 total 713.91 last 14.90) - open/close 14222 (time 1784050118.70 total 723.91 last 10.40) - open/close 14419 (time 1784050129.58 total 734.79 last 18.11) - open/close 14637 (time 1784050139.59 total 744.81 last 21.77) - open/close 14928 (time 1784050149.59 total 754.81 last 29.10) - open/close 15198 (time 1784050159.59 total 764.81 last 26.99) - open/close 15443 (time 1784050172.54 total 777.76 last 18.92) - open/close 15673 (time 1784050182.55 total 787.77 last 22.98) - open/close 15789 (time 1784050192.60 total 797.82 last 11.55) - open/close 16002 (time 1784050202.61 total 807.83 last 21.28) - open/close 16215 (time 1784050213.16 total 818.37 last 20.19) - open/close 16410 (time 1784050223.29 total 828.51 last 19.25) - open/close 16654 (time 1784050233.29 total 838.51 last 24.39) - open/close 16917 (time 1784050243.29 total 848.51 last 26.29) - open/close 17040 (time 1784050253.30 total 858.52 last 12.30) - open/close 17141 (time 1784050263.64 total 868.86 last 9.76) - open/close 17381 (time 1784050273.90 total 879.11 last 23.41) - open/close 17631 (time 1784050283.90 total 889.12 last 24.98) - open/close 17879 (time 1784050294.25 total 899.46 last 23.97) - open/close 18160 (time 1784050305.25 total 910.47 last 25.53) - open/close 18384 (time 1784050316.27 total 921.49 last 20.33) - open/close 18605 (time 1784050328.00 total 933.22 last 18.84) - open/close 18741 (time 1784050338.01 total 943.23 last 13.58) - open/close 18959 (time 1784050348.02 total 953.24 last 21.79) - open/close 19152 (time 1784050359.55 total 964.76 last 16.74) - open/close 19285 (time 1784050370.60 total 975.82 last 12.03) - open/close 19441 (time 1784050380.62 total 985.84 last 15.57) - open/close 19640 (time 1784050391.27 total 996.49 last 18.68) - open/close 20000 (time 1784050400.77 total 1005.99 last 37.89) - open/close 20902 (time 1784050410.78 total 1015.99 last 90.17) - open/close 21620 (time 1784050420.78 total 1025.99 last 71.79) - open/close 22389 (time 1784050430.78 total 1036.00 last 76.85) - open/close 23240 (time 1784050440.82 total 1046.03 last 84.84) - open/close 24195 (time 1784050450.82 total 1056.04 last 95.46) - open/close 25117 (time 1784050460.82 total 1066.04 last 92.18) - open/close 25997 (time 1784050470.82 total 1076.04 last 87.98) - open/close 26981 (time 1784050480.84 total 1086.06 last 98.25) - open/close 27738 (time 1784050490.84 total 1096.06 last 75.68) - open/close 28541 (time 1784050500.84 total 1106.06 last 80.30) - open/close 29441 (time 1784050510.85 total 1116.06 last 89.96) - open/close 30000 (time 1784050516.64 total 1121.86 last 96.48) - open/close 30821 (time 1784050526.65 total 1131.87 last 82.03) - open/close 31651 (time 1784050536.65 total 1141.87 last 82.98) - open/close 32565 (time 1784050546.65 total 1151.87 last 91.40) - open/close 33167 (time 1784050556.67 total 1161.89 last 60.09) - open/close 33973 (time 1784050566.68 total 1171.90 last 80.54) - open/close 34994 (time 1784050576.68 total 1181.90 last 102.08) - open/close 35905 (time 1784050586.69 total 1191.91 last 90.98) - open/close 36694 (time 1784050596.70 total 1201.92 last 78.85) - open/close 37544 (time 1784050606.71 total 1211.92 last 84.94) - open/close 38457 (time 1784050616.71 total 1221.93 last 91.24) - open/close 39365 (time 1784050626.72 total 1231.93 last 90.77) - open/close 40000 (time 1784050635.45 total 1240.67 last 72.67) - open/close 40509 (time 1784050645.46 total 1250.67 last 50.90) - open/close 41262 (time 1784050655.46 total 1260.68 last 75.27) - open/close 42162 (time 1784050665.46 total 1270.68 last 89.99) - open/close 43080 (time 1784050675.46 total 1280.68 last 91.77) - open/close 43971 (time 1784050685.47 total 1290.68 last 89.08) - open/close 44938 (time 1784050695.47 total 1300.69 last 96.66) - open/close 45923 (time 1784050705.48 total 1310.70 last 98.42) - open/close 46736 (time 1784050715.49 total 1320.71 last 81.21) - open/close 47853 (time 1784050725.49 total 1330.71 last 111.69) - open/close 48800 (time 1784050735.49 total 1340.71 last 94.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1350.40 seconds: 36.75 ops/second - unlinked 0 (time 1784050750 ; total 0 ; last 0) - unlinked 10000 (time 1784050877 ; total 127 ; last 127) - unlinked 20000 (time 1784050987 ; total 237 ; last 110) - unlinked 30000 (time 1784051051 ; total 301 ; last 64) - unlinked 40000 (time 1784051116 ; total 366 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 430 seconds: 115.420929 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2