== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:39:13 (1787791153) start mds service on oleg644-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg644-server' oleg644-server: oleg644-server.virtnet: executing load_modules_local oleg644-server: Loading modules from /home/green/git/lustre-release/lustre oleg644-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg644-server: detected 4 online CPUs by sysfs oleg644-server: MODOPTS_LIBCFS= oleg644-server: Force libcfs to create 2 CPU partitions oleg644-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg644-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg644-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg644-server: oleg644-server.virtnet: executing set_default_debug -1 all pdsh@oleg644-client: oleg644-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg644-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg644-server: oleg644-server.virtnet: executing set_default_debug -1 all pdsh@oleg644-client: oleg644-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg644-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg644-server: oleg644-server.virtnet: executing set_default_debug -1 all pdsh@oleg644-client: oleg644-server: ssh exited with exit code 1 Started lustre-OST0000 oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 255 (time 1787791240.85 total 10.37 last 24.59) - open/close 509 (time 1787791251.06 total 20.58 last 24.88) - open/close 754 (time 1787791261.95 total 31.47 last 22.49) - open/close 972 (time 1787791272.25 total 41.77 last 21.17) - open/close 1217 (time 1787791282.26 total 51.77 last 24.48) - open/close 1455 (time 1787791292.27 total 61.78 last 23.78) - open/close 1760 (time 1787791303.04 total 72.56 last 28.30) - open/close 2005 (time 1787791313.21 total 82.72 last 24.10) - open/close 2276 (time 1787791323.22 total 92.73 last 27.07) - open/close 2538 (time 1787791333.34 total 102.86 last 25.88) - open/close 2802 (time 1787791343.56 total 113.08 last 25.82) - open/close 3046 (time 1787791353.57 total 123.08 last 24.40) - open/close 3293 (time 1787791363.77 total 133.29 last 24.20) - open/close 3583 (time 1787791374.34 total 143.85 last 27.46) - open/close 3788 (time 1787791384.97 total 154.49 last 19.28) - open/close 4088 (time 1787791395.60 total 165.11 last 28.23) - open/close 4334 (time 1787791405.82 total 175.33 last 24.07) - open/close 4616 (time 1787791416.25 total 185.77 last 27.02) - open/close 4851 (time 1787791426.44 total 195.95 last 23.08) - open/close 5159 (time 1787791436.46 total 205.98 last 30.73) - open/close 5337 (time 1787791446.54 total 216.05 last 17.66) - open/close 5442 (time 1787791456.78 total 226.30 last 10.25) - open/close 5681 (time 1787791467.08 total 236.59 last 23.22) - open/close 5982 (time 1787791477.44 total 246.95 last 29.06) - open/close 6199 (time 1787791488.13 total 257.64 last 20.30) - open/close 6408 (time 1787791498.13 total 267.65 last 20.89) - open/close 6674 (time 1787791508.13 total 277.65 last 26.59) - open/close 6850 (time 1787791518.73 total 288.24 last 16.62) - open/close 7137 (time 1787791529.54 total 299.06 last 26.53) - open/close 7403 (time 1787791539.90 total 309.42 last 25.68) - open/close 7718 (time 1787791550.91 total 320.43 last 28.61) - open/close 7995 (time 1787791561.14 total 330.66 last 27.07) - open/close 8254 (time 1787791571.46 total 340.98 last 25.10) - open/close 8529 (time 1787791581.97 total 351.48 last 26.18) - open/close 8795 (time 1787791592.60 total 362.12 last 25.02) - open/close 9070 (time 1787791602.60 total 372.12 last 27.50) - open/close 9314 (time 1787791613.14 total 382.66 last 23.14) - open/close 9446 (time 1787791624.14 total 393.65 last 12.01) - open/close 9555 (time 1787791634.37 total 403.89 last 10.65) - open/close 9734 (time 1787791644.51 total 414.03 last 17.66) - open/close 9977 (time 1787791654.53 total 424.05 last 24.25) - open/close 10192 (time 1787791664.71 total 434.23 last 21.11) - open/close 10470 (time 1787791674.72 total 444.23 last 27.79) - open/close 10685 (time 1787791684.72 total 454.24 last 21.48) - open/close 10891 (time 1787791695.44 total 464.95 last 19.23) - open/close 11108 (time 1787791705.44 total 474.95 last 21.70) - open/close 11385 (time 1787791715.91 total 485.43 last 26.45) - open/close 11663 (time 1787791726.38 total 495.89 last 26.56) - open/close 11955 (time 1787791736.39 total 505.91 last 29.16) - open/close 12213 (time 1787791746.89 total 516.41 last 24.57) - open/close 12508 (time 1787791757.31 total 526.82 last 28.32) - open/close 12758 (time 1787791767.32 total 536.84 last 24.96) - open/close 13013 (time 1787791777.33 total 546.85 last 25.48) - open/close 13270 (time 1787791787.34 total 556.86 last 25.68) - open/close 13502 (time 1787791797.44 total 566.95 last 22.97) - open/close 13758 (time 1787791807.45 total 576.97 last 25.57) - open/close 13966 (time 1787791817.66 total 587.18 last 20.37) - open/close 14190 (time 1787791828.29 total 597.81 last 21.08) - open/close 14436 (time 1787791838.46 total 607.97 last 24.19) - open/close 14679 (time 1787791848.79 total 618.30 last 23.52) - open/close 14933 (time 1787791859.33 total 628.85 last 24.08) - open/close 15216 (time 1787791870.14 total 639.66 last 26.18) - open/close 15466 (time 1787791880.25 total 649.76 last 24.75) - open/close 15755 (time 1787791890.69 total 660.21 last 27.67) - open/close 16002 (time 1787791901.08 total 670.60 last 23.78) - open/close 16277 (time 1787791911.49 total 681.01 last 26.42) - open/close 16521 (time 1787791921.93 total 691.45 last 23.36) - open/close 16779 (time 1787791931.94 total 701.46 last 25.78) - open/close 17007 (time 1787791942.02 total 711.53 last 22.62) - open/close 17259 (time 1787791952.39 total 721.91 last 24.30) - open/close 17491 (time 1787791962.41 total 731.92 last 23.16) - open/close 17656 (time 1787791972.61 total 742.12 last 16.17) - open/close 17929 (time 1787791983.65 total 753.17 last 24.72) - open/close 18189 (time 1787791994.32 total 763.83 last 24.38) - open/close 18362 (time 1787792005.32 total 774.84 last 15.72) - open/close 18615 (time 1787792016.13 total 785.65 last 23.40) - open/close 18856 (time 1787792026.37 total 795.89 last 23.53) - open/close 19108 (time 1787792036.81 total 806.33 last 24.14) - open/close 19319 (time 1787792046.82 total 816.34 last 21.08) - open/close 19568 (time 1787792057.34 total 826.86 last 23.66) - open/close 19764 (time 1787792067.79 total 837.31 last 18.76) - open/close 20000 (time 1787792077.21 total 846.73 last 25.06) - open/close 20172 (time 1787792087.55 total 857.07 last 16.63) - open/close 20425 (time 1787792098.30 total 867.82 last 23.53) - open/close 21255 (time 1787792108.31 total 877.83 last 82.91) - open/close 22122 (time 1787792118.31 total 887.83 last 86.68) - open/close 22940 (time 1787792128.32 total 897.84 last 81.72) - open/close 23799 (time 1787792138.33 total 907.85 last 85.80) - open/close 24635 (time 1787792148.34 total 917.85 last 83.59) - open/close 25494 (time 1787792158.34 total 927.86 last 85.86) - open/close 26349 (time 1787792168.34 total 937.86 last 85.48) - open/close 27148 (time 1787792178.36 total 947.88 last 79.74) - open/close 27847 (time 1787792188.37 total 957.89 last 69.82) - open/close 28613 (time 1787792198.38 total 967.89 last 76.58) - open/close 29396 (time 1787792208.38 total 977.90 last 78.27) - open/close 30000 (time 1787792215.93 total 985.45 last 79.96) - open/close 30861 (time 1787792225.94 total 995.46 last 86.02) - open/close 31622 (time 1787792235.95 total 1005.46 last 76.07) - open/close 32406 (time 1787792245.95 total 1015.47 last 78.36) - open/close 33336 (time 1787792255.95 total 1025.47 last 93.00) - open/close 34218 (time 1787792265.96 total 1035.47 last 88.17) - open/close 35092 (time 1787792275.97 total 1045.49 last 87.28) - open/close 35945 (time 1787792285.98 total 1055.49 last 85.25) - open/close 36754 (time 1787792295.98 total 1065.49 last 80.89) - open/close 37412 (time 1787792305.98 total 1075.49 last 65.80) - open/close 38037 (time 1787792315.99 total 1085.50 last 62.45) - open/close 38703 (time 1787792326.00 total 1095.51 last 66.54) - open/close 39439 (time 1787792336.01 total 1105.52 last 73.52) - open/close 40000 (time 1787792342.88 total 1112.39 last 81.67) - open/close 40817 (time 1787792352.89 total 1122.40 last 81.61) - open/close 41584 (time 1787792362.90 total 1132.41 last 76.63) - open/close 42358 (time 1787792372.90 total 1142.42 last 77.35) - open/close 43186 (time 1787792382.91 total 1152.42 last 82.76) - open/close 43984 (time 1787792392.91 total 1162.42 last 79.79) - open/close 44789 (time 1787792402.91 total 1172.42 last 80.49) - open/close 45582 (time 1787792412.91 total 1182.43 last 79.25) - open/close 46354 (time 1787792422.92 total 1192.43 last 77.19) - open/close 47144 (time 1787792432.93 total 1202.44 last 78.92) - open/close 47906 (time 1787792442.93 total 1212.44 last 76.18) - open/close 48695 (time 1787792452.94 total 1222.46 last 78.80) - open/close 49421 (time 1787792462.96 total 1232.47 last 72.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1235.17 seconds: 40.18 ops/second - unlinked 0 (time 1787792470 ; total 0 ; last 0) - unlinked 10000 (time 1787792596 ; total 126 ; last 126) - unlinked 20000 (time 1787792741 ; total 271 ; last 145) - unlinked 30000 (time 1787792874 ; total 404 ; last 133) - unlinked 40000 (time 1787793003 ; total 533 ; last 129) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 658 seconds: 75.427055 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg644-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg644-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg644-server stop mds service on oleg644-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg644-server stop mds service on oleg644-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg644-server unloading modules via unload_modules_local on: 'oleg644-server' oleg644-server: oleg644-server.virtnet: executing unload_modules_local oleg644-server: modules unloaded. pdsh@oleg644-client: oleg644-server: ssh exited with exit code 2 pdsh@oleg644-client: oleg644-client: ssh exited with exit code 2 pdsh@oleg644-client: oleg644-server: ssh exited with exit code 2 pdsh@oleg644-client: oleg644-client: ssh exited with exit code 2