== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:45:51 (1787139951) start mds service on oleg336-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 285 (time 1787140057.50 total 10.02 last 28.44) - open/close 441 (time 1787140068.38 total 20.90 last 14.34) - open/close 572 (time 1787140078.42 total 30.95 last 13.04) - open/close 742 (time 1787140088.43 total 40.95 last 16.99) - open/close 961 (time 1787140098.64 total 51.16 last 21.44) - open/close 1252 (time 1787140108.64 total 61.17 last 29.09) - open/close 1457 (time 1787140118.65 total 71.18 last 20.48) - open/close 1669 (time 1787140128.82 total 81.34 last 20.86) - open/close 1934 (time 1787140138.82 total 91.34 last 26.49) - open/close 2174 (time 1787140149.58 total 102.10 last 22.31) - open/close 2411 (time 1787140160.60 total 113.12 last 21.51) - open/close 2562 (time 1787140170.94 total 123.46 last 14.60) - open/close 2747 (time 1787140181.98 total 134.50 last 16.76) - open/close 2999 (time 1787140192.84 total 145.36 last 23.22) - open/close 3260 (time 1787140203.86 total 156.39 last 23.67) - open/close 3493 (time 1787140215.44 total 167.97 last 20.12) - open/close 3705 (time 1787140225.45 total 177.97 last 21.19) - open/close 3947 (time 1787140235.46 total 187.98 last 24.18) - open/close 4190 (time 1787140245.79 total 198.32 last 23.52) - open/close 4456 (time 1787140256.72 total 209.24 last 24.35) - open/close 4716 (time 1787140266.73 total 219.25 last 25.97) - open/close 4994 (time 1787140276.74 total 229.26 last 27.78) - open/close 5234 (time 1787140287.02 total 239.55 last 23.33) - open/close 5493 (time 1787140297.03 total 249.55 last 25.88) - open/close 5691 (time 1787140307.04 total 259.56 last 19.77) - open/close 5885 (time 1787140317.06 total 269.58 last 19.37) - open/close 6119 (time 1787140328.03 total 280.56 last 21.32) - open/close 6301 (time 1787140338.60 total 291.12 last 17.23) - open/close 6569 (time 1787140348.90 total 301.42 last 26.03) - open/close 6772 (time 1787140359.91 total 312.44 last 18.43) - open/close 6963 (time 1787140370.58 total 323.10 last 17.90) - open/close 7227 (time 1787140381.42 total 333.94 last 24.37) - open/close 7413 (time 1787140391.44 total 343.96 last 18.56) - open/close 7598 (time 1787140402.18 total 354.71 last 17.22) - open/close 7713 (time 1787140412.20 total 364.72 last 11.48) - open/close 7877 (time 1787140422.20 total 374.73 last 16.39) - open/close 7935 (time 1787140432.79 total 385.32 last 5.48) - open/close 8180 (time 1787140442.79 total 395.32 last 24.50) - open/close 8371 (time 1787140452.88 total 405.40 last 18.94) - open/close 8533 (time 1787140463.50 total 416.03 last 15.25) - open/close 8754 (time 1787140473.99 total 426.52 last 21.07) - open/close 8962 (time 1787140484.83 total 437.35 last 19.19) - open/close 9189 (time 1787140494.95 total 447.47 last 22.44) - open/close 9479 (time 1787140505.94 total 458.46 last 26.39) - open/close 9768 (time 1787140516.63 total 469.16 last 27.01) - open/close 10000 (time 1787140526.46 total 478.98 last 23.61) - open/close 10188 (time 1787140536.47 total 488.99 last 18.78) - open/close 10244 (time 1787140547.21 total 499.74 last 5.21) - open/close 10446 (time 1787140559.86 total 512.39 last 15.97) - open/close 10568 (time 1787140569.88 total 522.40 last 12.18) - open/close 10805 (time 1787140580.23 total 532.76 last 22.88) - open/close 11068 (time 1787140590.24 total 542.76 last 26.30) - open/close 11312 (time 1787140600.24 total 552.77 last 24.38) - open/close 11554 (time 1787140610.28 total 562.80 last 24.12) - open/close 11718 (time 1787140620.32 total 572.84 last 16.33) - open/close 11982 (time 1787140630.32 total 582.84 last 26.40) - open/close 12221 (time 1787140642.99 total 595.51 last 18.86) - open/close 12496 (time 1787140653.00 total 605.52 last 27.48) - open/close 12755 (time 1787140663.96 total 616.48 last 23.63) - open/close 12974 (time 1787140674.16 total 626.68 last 21.46) - open/close 13190 (time 1787140684.38 total 636.90 last 21.14) - open/close 13417 (time 1787140694.95 total 647.47 last 21.49) - open/close 13664 (time 1787140704.96 total 657.49 last 24.66) - open/close 13913 (time 1787140714.98 total 667.50 last 24.85) - open/close 14012 (time 1787140725.33 total 677.86 last 9.56) - open/close 14192 (time 1787140735.33 total 687.86 last 18.00) - open/close 14331 (time 1787140747.43 total 699.95 last 11.49) - open/close 14576 (time 1787140758.45 total 710.98 last 22.22) - open/close 14812 (time 1787140770.97 total 723.49 last 18.86) - open/close 15085 (time 1787140781.86 total 734.38 last 25.06) - open/close 15332 (time 1787140792.09 total 744.61 last 24.14) - open/close 15583 (time 1787140802.53 total 755.06 last 24.03) - open/close 15831 (time 1787140813.03 total 765.56 last 23.62) - open/close 16104 (time 1787140823.04 total 775.57 last 27.27) - open/close 16256 (time 1787140833.50 total 786.02 last 14.54) - open/close 16505 (time 1787140843.52 total 796.05 last 24.84) - open/close 16773 (time 1787140854.51 total 807.04 last 24.38) - open/close 16943 (time 1787140864.52 total 817.04 last 17.00) - open/close 17107 (time 1787140875.09 total 827.62 last 15.51) - open/close 17263 (time 1787140885.10 total 837.63 last 15.58) - open/close 17499 (time 1787140895.52 total 848.05 last 22.65) - open/close 17787 (time 1787140906.53 total 859.05 last 26.18) - open/close 18052 (time 1787140917.60 total 870.12 last 23.93) - open/close 18235 (time 1787140928.28 total 880.81 last 17.12) - open/close 18445 (time 1787140938.65 total 891.17 last 20.26) - open/close 18688 (time 1787140948.72 total 901.24 last 24.13) - open/close 18863 (time 1787140959.07 total 911.59 last 16.90) - open/close 18996 (time 1787140969.12 total 921.64 last 13.24) - open/close 19127 (time 1787140979.28 total 931.81 last 12.88) - open/close 19214 (time 1787140992.50 total 945.02 last 6.58) - open/close 19470 (time 1787141003.31 total 955.84 last 23.67) - open/close 19732 (time 1787141013.32 total 965.84 last 26.18) - open/close 19982 (time 1787141023.34 total 975.86 last 24.96) - open/close 20657 (time 1787141033.37 total 985.89 last 67.30) - open/close 21379 (time 1787141043.37 total 995.90 last 72.16) - open/close 22063 (time 1787141053.38 total 1005.90 last 68.37) - open/close 22698 (time 1787141063.38 total 1015.90 last 63.49) - open/close 23356 (time 1787141073.40 total 1025.92 last 65.68) - open/close 24055 (time 1787141083.41 total 1035.93 last 69.82) - open/close 24692 (time 1787141093.42 total 1045.94 last 63.66) - open/close 25378 (time 1787141103.42 total 1055.94 last 68.56) - open/close 26119 (time 1787141113.43 total 1065.95 last 74.03) - open/close 26928 (time 1787141123.44 total 1075.96 last 80.81) - open/close 27673 (time 1787141133.45 total 1085.97 last 74.47) - open/close 28430 (time 1787141143.45 total 1095.97 last 75.68) - open/close 29090 (time 1787141153.46 total 1105.98 last 65.92) - open/close 29774 (time 1787141163.47 total 1115.99 last 68.34) - open/close 30000 (time 1787141166.67 total 1119.19 last 70.66) - open/close 30765 (time 1787141176.67 total 1129.19 last 76.49) - open/close 31555 (time 1787141186.67 total 1139.19 last 78.99) - open/close 32387 (time 1787141196.68 total 1149.21 last 83.09) - open/close 33135 (time 1787141206.70 total 1159.23 last 74.66) - open/close 33941 (time 1787141216.70 total 1169.23 last 80.59) - open/close 34770 (time 1787141226.71 total 1179.23 last 82.86) - open/close 35513 (time 1787141236.72 total 1189.24 last 74.21) - open/close 36272 (time 1787141246.76 total 1199.28 last 75.61) - open/close 37024 (time 1787141256.78 total 1209.30 last 75.03) - open/close 37734 (time 1787141266.79 total 1219.31 last 70.96) - open/close 38474 (time 1787141276.83 total 1229.35 last 73.70) - open/close 39191 (time 1787141286.83 total 1239.36 last 71.65) - open/close 39950 (time 1787141296.84 total 1249.36 last 75.87) - open/close 40581 (time 1787141306.85 total 1259.38 last 62.99) - open/close 41141 (time 1787141316.87 total 1269.39 last 55.92) - open/close 41742 (time 1787141326.89 total 1279.41 last 59.99) - open/close 42410 (time 1787141336.89 total 1289.42 last 66.77) - open/close 43190 (time 1787141346.90 total 1299.42 last 77.97) - open/close 43929 (time 1787141356.90 total 1309.43 last 73.84) - open/close 44705 (time 1787141366.91 total 1319.43 last 77.57) - open/close 45450 (time 1787141376.92 total 1329.44 last 74.44) - open/close 46247 (time 1787141386.94 total 1339.46 last 79.50) - open/close 46956 (time 1787141396.97 total 1349.49 last 70.71) - open/close 47699 (time 1787141406.97 total 1359.50 last 74.26) - open/close 48412 (time 1787141416.98 total 1369.51 last 71.23) - open/close 49075 (time 1787141426.99 total 1379.51 last 66.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1387.62 seconds: 35.77 ops/second - unlinked 0 (time 1787141440 ; total 0 ; last 0) - unlinked 10000 (time 1787141585 ; total 145 ; last 145) - unlinked 20000 (time 1787141729 ; total 289 ; last 144) - unlinked 30000 (time 1787141876 ; total 436 ; last 147) - unlinked 40000 (time 1787142028 ; total 588 ; last 152) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 728 seconds: 68.174454 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2