== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:13:22 (1789136002) start mds service on oleg250-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg250-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg250-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg250-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 266 (time 1789136138.74 total 10.89 last 24.43) - open/close 353 (time 1789136149.05 total 21.19 last 8.44) - open/close 609 (time 1789136159.05 total 31.20 last 25.59) - open/close 818 (time 1789136169.66 total 41.81 last 19.70) - open/close 978 (time 1789136180.01 total 52.16 last 15.45) - open/close 1205 (time 1789136190.57 total 62.72 last 21.50) - open/close 1410 (time 1789136200.59 total 72.74 last 20.45) - open/close 1645 (time 1789136210.60 total 82.75 last 23.48) - open/close 1840 (time 1789136220.62 total 92.76 last 19.47) - open/close 2099 (time 1789136230.63 total 102.78 last 25.86) - open/close 2181 (time 1789136241.64 total 113.79 last 7.45) - open/close 2470 (time 1789136252.72 total 124.87 last 26.08) - open/close 2603 (time 1789136262.73 total 134.88 last 13.29) - open/close 2739 (time 1789136273.21 total 145.36 last 12.98) - open/close 2872 (time 1789136283.44 total 155.59 last 13.00) - open/close 2936 (time 1789136293.45 total 165.59 last 6.40) - open/close 3108 (time 1789136303.45 total 175.60 last 17.19) - open/close 3338 (time 1789136313.47 total 185.61 last 22.97) - open/close 3597 (time 1789136324.48 total 196.62 last 23.52) - open/close 3717 (time 1789136334.49 total 206.63 last 11.99) - open/close 3782 (time 1789136345.02 total 217.16 last 6.17) - open/close 3877 (time 1789136355.03 total 227.18 last 9.48) - open/close 4059 (time 1789136365.04 total 237.19 last 18.19) - open/close 4181 (time 1789136375.27 total 247.42 last 11.92) - open/close 4432 (time 1789136385.60 total 257.74 last 24.31) - open/close 4507 (time 1789136395.60 total 267.75 last 7.50) - open/close 4737 (time 1789136408.51 total 280.66 last 17.81) - open/close 4817 (time 1789136418.94 total 291.09 last 7.67) - open/close 4993 (time 1789136429.29 total 301.44 last 17.01) - open/close 5125 (time 1789136439.30 total 311.45 last 13.18) - open/close 5280 (time 1789136450.98 total 323.13 last 13.27) - open/close 5491 (time 1789136462.26 total 334.40 last 18.71) - open/close 5625 (time 1789136472.26 total 344.41 last 13.39) - open/close 5826 (time 1789136482.27 total 354.42 last 20.09) - open/close 6029 (time 1789136492.40 total 364.55 last 20.04) - open/close 6227 (time 1789136502.44 total 374.59 last 19.72) - open/close 6388 (time 1789136513.13 total 385.28 last 15.06) - open/close 6548 (time 1789136523.16 total 395.30 last 15.96) - open/close 6642 (time 1789136533.69 total 405.84 last 8.92) - open/close 6851 (time 1789136544.16 total 416.31 last 19.96) - open/close 7102 (time 1789136554.93 total 427.08 last 23.30) - open/close 7318 (time 1789136564.93 total 437.08 last 21.60) - open/close 7593 (time 1789136575.21 total 447.36 last 26.77) - open/close 7796 (time 1789136585.21 total 457.36 last 20.29) - open/close 7926 (time 1789136595.22 total 467.37 last 12.98) - open/close 8040 (time 1789136606.06 total 478.21 last 10.52) - open/close 8297 (time 1789136616.99 total 489.14 last 23.52) - open/close 8584 (time 1789136626.99 total 499.14 last 28.70) - open/close 8808 (time 1789136638.26 total 510.41 last 19.87) - open/close 8891 (time 1789136648.28 total 520.43 last 8.29) - open/close 9172 (time 1789136658.29 total 530.44 last 28.06) - open/close 9429 (time 1789136668.79 total 540.94 last 24.47) - open/close 9687 (time 1789136681.92 total 554.06 last 19.66) - open/close 9820 (time 1789136694.38 total 566.52 last 10.67) - open/close 9971 (time 1789136704.89 total 577.04 last 14.36) - open/close 10101 (time 1789136714.90 total 587.05 last 12.99) - open/close 10253 (time 1789136725.93 total 598.08 last 13.78) - open/close 10412 (time 1789136736.64 total 608.79 last 14.85) - open/close 10594 (time 1789136747.51 total 619.66 last 16.74) - open/close 10761 (time 1789136758.48 total 630.63 last 15.23) - open/close 10882 (time 1789136768.50 total 640.64 last 12.08) - open/close 11078 (time 1789136778.55 total 650.70 last 19.49) - open/close 11281 (time 1789136789.05 total 661.20 last 19.34) - open/close 11518 (time 1789136799.07 total 671.21 last 23.66) - open/close 11651 (time 1789136809.39 total 681.54 last 12.89) - open/close 11848 (time 1789136820.34 total 692.49 last 17.99) - open/close 12096 (time 1789136830.76 total 702.91 last 23.80) - open/close 12298 (time 1789136840.94 total 713.09 last 19.84) - open/close 12377 (time 1789136852.04 total 724.18 last 7.12) - open/close 12497 (time 1789136862.05 total 734.20 last 11.98) - open/close 12794 (time 1789136872.94 total 745.08 last 27.28) - open/close 12956 (time 1789136883.89 total 756.04 last 14.79) - open/close 13179 (time 1789136893.89 total 766.04 last 22.30) - open/close 13465 (time 1789136904.93 total 777.08 last 25.90) - open/close 13585 (time 1789136914.94 total 787.09 last 11.99) - open/close 13659 (time 1789136925.88 total 798.03 last 6.76) - open/close 13797 (time 1789136936.75 total 808.90 last 12.69) - open/close 14017 (time 1789136946.88 total 819.03 last 21.73) - open/close 14159 (time 1789136956.89 total 829.03 last 14.19) - open/close 14241 (time 1789136966.91 total 839.06 last 8.18) - open/close 14436 (time 1789136976.91 total 849.06 last 19.49) - open/close 14563 (time 1789136986.92 total 859.07 last 12.69) - open/close 14644 (time 1789136997.11 total 869.26 last 7.95) - open/close 14843 (time 1789137007.37 total 879.52 last 19.40) - open/close 14951 (time 1789137017.40 total 889.55 last 10.76) - open/close 15057 (time 1789137027.52 total 899.67 last 10.47) - open/close 15168 (time 1789137037.73 total 909.88 last 10.88) - open/close 15286 (time 1789137047.99 total 920.13 last 11.51) - open/close 15346 (time 1789137058.00 total 930.14 last 5.99) - open/close 15486 (time 1789137068.23 total 940.37 last 13.69) - open/close 15676 (time 1789137079.29 total 951.44 last 17.18) - open/close 15797 (time 1789137089.99 total 962.13 last 11.31) - open/close 15973 (time 1789137099.99 total 972.14 last 17.60) - open/close 16067 (time 1789137110.59 total 982.74 last 8.87) - open/close 16279 (time 1789137121.65 total 993.79 last 19.18) - open/close 16461 (time 1789137132.04 total 1004.18 last 17.52) - open/close 16608 (time 1789137142.39 total 1014.53 last 14.20) - open/close 16745 (time 1789137152.68 total 1024.83 last 13.31) - open/close 16877 (time 1789137162.69 total 1034.84 last 13.19) - open/close 16984 (time 1789137172.97 total 1045.12 last 10.40) - open/close 17174 (time 1789137182.99 total 1055.13 last 18.98) - open/close 17601 (time 1789137192.99 total 1065.13 last 42.70) - open/close 18404 (time 1789137203.00 total 1075.15 last 80.18) - open/close 19258 (time 1789137213.01 total 1085.16 last 85.34) - open/close 20000 (time 1789137221.96 total 1094.11 last 82.91) - open/close 20823 (time 1789137231.97 total 1104.12 last 82.21) - open/close 21565 (time 1789137241.97 total 1114.12 last 74.18) - open/close 22474 (time 1789137251.99 total 1124.14 last 90.72) - open/close 23219 (time 1789137262.00 total 1134.15 last 74.44) - open/close 23802 (time 1789137272.00 total 1144.15 last 58.28) - open/close 24488 (time 1789137282.01 total 1154.16 last 68.57) - open/close 25224 (time 1789137292.02 total 1164.17 last 73.51) - open/close 26210 (time 1789137302.03 total 1174.17 last 98.54) - open/close 27150 (time 1789137312.03 total 1184.18 last 93.98) - open/close 28058 (time 1789137322.04 total 1194.19 last 90.68) - open/close 28768 (time 1789137332.05 total 1204.19 last 70.96) - open/close 29416 (time 1789137342.06 total 1214.21 last 64.70) - open/close 30000 (time 1789137350.54 total 1222.68 last 68.92) - open/close 30592 (time 1789137360.54 total 1232.68 last 59.20) - open/close 31169 (time 1789137370.54 total 1242.68 last 57.70) - open/close 31765 (time 1789137380.54 total 1252.69 last 59.58) - open/close 32481 (time 1789137390.56 total 1262.71 last 71.47) - open/close 33239 (time 1789137400.56 total 1272.71 last 75.76) - open/close 33947 (time 1789137410.57 total 1282.72 last 70.77) - open/close 34724 (time 1789137420.57 total 1292.72 last 77.69) - open/close 35444 (time 1789137430.58 total 1302.72 last 71.94) - open/close 36176 (time 1789137440.58 total 1312.73 last 73.18) - open/close 36987 (time 1789137450.59 total 1322.73 last 81.04) - open/close 37820 (time 1789137460.61 total 1332.76 last 83.08) - open/close 38533 (time 1789137470.61 total 1342.76 last 71.29) - open/close 39073 (time 1789137480.64 total 1352.78 last 53.88) - open/close 39650 (time 1789137490.64 total 1362.79 last 57.69) - open/close 40000 (time 1789137497.14 total 1369.29 last 53.85) - open/close 40799 (time 1789137507.15 total 1379.30 last 79.81) - open/close 41694 (time 1789137517.15 total 1389.30 last 89.48) - open/close 42354 (time 1789137527.16 total 1399.31 last 65.93) - open/close 43179 (time 1789137537.18 total 1409.33 last 82.37) - open/close 43941 (time 1789137547.18 total 1419.33 last 76.18) - open/close 44774 (time 1789137557.19 total 1429.34 last 83.21) - open/close 45553 (time 1789137567.20 total 1439.35 last 77.85) - open/close 46120 (time 1789137577.20 total 1449.35 last 56.67) - open/close 46642 (time 1789137587.21 total 1459.36 last 52.19) - open/close 47339 (time 1789137597.23 total 1469.38 last 69.55) - open/close 48061 (time 1789137607.24 total 1479.39 last 72.11) - open/close 48673 (time 1789137617.24 total 1489.39 last 61.18) - open/close 49439 (time 1789137627.26 total 1499.40 last 76.52) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1502.66 seconds: 33.03 ops/second - unlinked 0 (time 1789137636 ; total 0 ; last 0) - unlinked 10000 (time 1789137771 ; total 135 ; last 135) - unlinked 20000 (time 1789137905 ; total 269 ; last 134) - unlinked 30000 (time 1789138040 ; total 404 ; last 135) - unlinked 40000 (time 1789138168 ; total 532 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 658 seconds: 75.425529 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local oleg250-server: modules unloaded. pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2