== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:31:09 (1788445869) start mds service on oleg654-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions oleg654-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg654-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg654-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg654-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg654-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 127 (time 1788445977.93 total 10.95 last 11.59) - open/close 356 (time 1788445987.93 total 20.95 last 22.90) - open/close 553 (time 1788445997.94 total 30.97 last 19.68) - open/close 825 (time 1788446008.70 total 41.73 last 25.27) - open/close 1039 (time 1788446018.72 total 51.74 last 21.37) - open/close 1204 (time 1788446029.24 total 62.26 last 15.69) - open/close 1470 (time 1788446039.25 total 72.28 last 26.56) - open/close 1727 (time 1788446049.26 total 82.29 last 25.67) - open/close 1909 (time 1788446059.27 total 92.29 last 18.19) - open/close 2176 (time 1788446070.00 total 103.02 last 24.89) - open/close 2381 (time 1788446080.00 total 113.02 last 20.49) - open/close 2612 (time 1788446090.42 total 123.44 last 22.17) - open/close 2852 (time 1788446100.62 total 133.65 last 23.52) - open/close 3122 (time 1788446110.96 total 143.99 last 26.11) - open/close 3338 (time 1788446121.47 total 154.49 last 20.56) - open/close 3610 (time 1788446131.58 total 164.60 last 26.90) - open/close 3852 (time 1788446141.59 total 174.61 last 24.17) - open/close 4132 (time 1788446152.32 total 185.35 last 26.09) - open/close 4359 (time 1788446163.83 total 196.86 last 19.72) - open/close 4615 (time 1788446174.37 total 207.39 last 24.31) - open/close 4880 (time 1788446184.39 total 217.41 last 26.44) - open/close 5108 (time 1788446194.88 total 227.90 last 21.74) - open/close 5309 (time 1788446204.88 total 237.91 last 20.09) - open/close 5564 (time 1788446214.90 total 247.93 last 25.46) - open/close 5640 (time 1788446224.91 total 257.94 last 7.59) - open/close 5792 (time 1788446237.19 total 270.22 last 12.38) - open/close 6084 (time 1788446247.21 total 280.23 last 29.16) - open/close 6282 (time 1788446257.96 total 290.99 last 18.40) - open/close 6524 (time 1788446267.97 total 300.99 last 24.19) - open/close 6758 (time 1788446277.98 total 311.01 last 23.37) - open/close 7040 (time 1788446288.76 total 321.79 last 26.17) - open/close 7214 (time 1788446299.83 total 332.85 last 15.72) - open/close 7421 (time 1788446309.83 total 342.85 last 20.69) - open/close 7658 (time 1788446319.87 total 352.90 last 23.60) - open/close 7895 (time 1788446329.90 total 362.93 last 23.63) - open/close 8146 (time 1788446340.27 total 373.29 last 24.22) - open/close 8442 (time 1788446351.11 total 384.13 last 27.30) - open/close 8727 (time 1788446361.12 total 394.14 last 28.47) - open/close 9003 (time 1788446371.98 total 405.00 last 25.42) - open/close 9266 (time 1788446381.99 total 415.01 last 26.28) - open/close 9534 (time 1788446391.99 total 425.02 last 26.78) - open/close 9731 (time 1788446402.00 total 435.03 last 19.68) - open/close 9852 (time 1788446412.01 total 445.04 last 12.09) - open/close 10000 (time 1788446420.17 total 453.19 last 18.15) - open/close 10277 (time 1788446430.17 total 463.19 last 27.70) - open/close 10499 (time 1788446440.18 total 473.20 last 22.18) - open/close 10744 (time 1788446450.35 total 483.38 last 24.08) - open/close 10960 (time 1788446461.18 total 494.20 last 19.96) - open/close 11241 (time 1788446471.66 total 504.69 last 26.80) - open/close 11505 (time 1788446482.10 total 515.12 last 25.29) - open/close 11767 (time 1788446492.11 total 525.13 last 26.18) - open/close 12010 (time 1788446502.97 total 535.99 last 22.38) - open/close 12272 (time 1788446512.97 total 546.00 last 26.18) - open/close 12572 (time 1788446523.12 total 556.14 last 29.58) - open/close 12855 (time 1788446534.04 total 567.06 last 25.91) - open/close 13105 (time 1788446544.04 total 577.07 last 24.99) - open/close 13354 (time 1788446554.61 total 587.64 last 23.56) - open/close 13611 (time 1788446565.14 total 598.17 last 24.41) - open/close 13857 (time 1788446575.68 total 608.70 last 23.35) - open/close 14113 (time 1788446586.33 total 619.35 last 24.05) - open/close 14335 (time 1788446596.33 total 629.35 last 22.19) - open/close 14557 (time 1788446606.35 total 639.37 last 22.16) - open/close 14855 (time 1788446616.36 total 649.39 last 29.76) - open/close 15114 (time 1788446626.73 total 659.76 last 24.97) - open/close 15341 (time 1788446636.74 total 669.76 last 22.69) - open/close 15626 (time 1788446647.48 total 680.50 last 26.54) - open/close 15867 (time 1788446657.48 total 690.51 last 24.09) - open/close 15968 (time 1788446667.51 total 700.54 last 10.07) - open/close 16236 (time 1788446678.02 total 711.04 last 25.51) - open/close 16499 (time 1788446688.03 total 721.05 last 26.27) - open/close 16707 (time 1788446698.06 total 731.08 last 20.75) - open/close 16861 (time 1788446710.23 total 743.25 last 12.65) - open/close 17077 (time 1788446722.49 total 755.51 last 17.62) - open/close 17288 (time 1788446733.51 total 766.53 last 19.15) - open/close 17486 (time 1788446744.32 total 777.35 last 18.31) - open/close 17761 (time 1788446754.93 total 787.95 last 25.93) - open/close 18069 (time 1788446764.94 total 797.97 last 30.76) - open/close 18301 (time 1788446774.95 total 807.97 last 23.19) - open/close 18539 (time 1788446784.95 total 817.98 last 23.78) - open/close 18763 (time 1788446795.82 total 828.85 last 20.61) - open/close 19000 (time 1788446805.82 total 838.85 last 23.70) - open/close 19183 (time 1788446816.55 total 849.58 last 17.06) - open/close 19396 (time 1788446826.56 total 859.58 last 21.29) - open/close 19597 (time 1788446837.20 total 870.22 last 18.89) - open/close 19860 (time 1788446847.20 total 880.22 last 26.30) - open/close 20000 (time 1788446853.44 total 886.47 last 22.42) - open/close 20300 (time 1788446863.44 total 896.47 last 30.00) - open/close 21010 (time 1788446873.45 total 906.48 last 70.95) - open/close 21824 (time 1788446883.46 total 916.49 last 81.33) - open/close 22594 (time 1788446893.46 total 926.49 last 77.00) - open/close 23337 (time 1788446903.48 total 936.50 last 74.17) - open/close 24106 (time 1788446913.48 total 946.51 last 76.86) - open/close 24851 (time 1788446923.49 total 956.51 last 74.49) - open/close 25575 (time 1788446933.49 total 966.52 last 72.36) - open/close 26257 (time 1788446943.50 total 976.53 last 68.11) - open/close 26978 (time 1788446953.53 total 986.55 last 71.94) - open/close 27698 (time 1788446963.56 total 996.58 last 71.78) - open/close 28447 (time 1788446973.57 total 1006.59 last 74.84) - open/close 29245 (time 1788446983.57 total 1016.60 last 79.75) - open/close 30000 (time 1788446992.86 total 1025.88 last 81.30) - open/close 30731 (time 1788447002.88 total 1035.91 last 72.90) - open/close 31449 (time 1788447012.89 total 1045.92 last 71.76) - open/close 32181 (time 1788447022.89 total 1055.92 last 73.20) - open/close 32889 (time 1788447032.89 total 1065.92 last 70.78) - open/close 33623 (time 1788447042.90 total 1075.93 last 73.33) - open/close 34333 (time 1788447052.92 total 1085.95 last 70.89) - open/close 35036 (time 1788447062.92 total 1095.95 last 70.30) - open/close 35892 (time 1788447072.92 total 1105.95 last 85.58) - open/close 36680 (time 1788447082.93 total 1115.96 last 78.74) - open/close 37448 (time 1788447092.95 total 1125.97 last 76.69) - open/close 38208 (time 1788447102.96 total 1135.99 last 75.89) - open/close 38989 (time 1788447112.97 total 1145.99 last 78.03) - open/close 39711 (time 1788447122.98 total 1156.00 last 72.14) - open/close 40000 (time 1788447127.49 total 1160.52 last 64.04) - open/close 40810 (time 1788447137.50 total 1170.52 last 80.96) - open/close 41586 (time 1788447147.51 total 1180.54 last 77.48) - open/close 42402 (time 1788447157.52 total 1190.54 last 81.54) - open/close 43133 (time 1788447167.53 total 1200.55 last 73.03) - open/close 43932 (time 1788447177.54 total 1210.56 last 79.83) - open/close 44684 (time 1788447187.54 total 1220.57 last 75.14) - open/close 45472 (time 1788447197.55 total 1230.58 last 78.75) - open/close 46173 (time 1788447207.56 total 1240.58 last 70.05) - open/close 46922 (time 1788447217.56 total 1250.59 last 74.88) - open/close 47697 (time 1788447227.57 total 1260.59 last 77.44) - open/close 48454 (time 1788447237.57 total 1270.60 last 75.69) - open/close 49236 (time 1788447247.57 total 1280.60 last 78.19) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1286.26 seconds: 38.59 ops/second - unlinked 0 (time 1788447257 ; total 0 ; last 0) - unlinked 10000 (time 1788447390 ; total 133 ; last 133) - unlinked 20000 (time 1788447524 ; total 267 ; last 134) - unlinked 30000 (time 1788447661 ; total 404 ; last 137) - unlinked 40000 (time 1788447793 ; total 536 ; last 132) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 663 seconds: 74.858223 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg654-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg654-server unloading modules via unload_modules_local on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing unload_modules_local oleg654-server: modules unloaded. pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2