== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:49:47 (1783950587) start mds service on oleg425-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg425-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1783950671.16 total 10.23 last 19.95) - open/close 455 (time 1783950682.03 total 21.10 last 23.09) - open/close 682 (time 1783950692.93 total 32.00 last 20.82) - open/close 918 (time 1783950702.94 total 42.01 last 23.57) - open/close 1221 (time 1783950713.91 total 52.98 last 27.62) - open/close 1499 (time 1783950724.60 total 63.67 last 26.01) - open/close 1762 (time 1783950734.70 total 73.77 last 26.04) - open/close 2006 (time 1783950744.82 total 83.89 last 24.11) - open/close 2283 (time 1783950755.45 total 94.51 last 26.07) - open/close 2539 (time 1783950765.61 total 104.68 last 25.18) - open/close 2769 (time 1783950775.93 total 114.99 last 22.30) - open/close 3026 (time 1783950786.29 total 125.35 last 24.81) - open/close 3251 (time 1783950796.29 total 135.36 last 22.50) - open/close 3512 (time 1783950806.91 total 145.98 last 24.57) - open/close 3742 (time 1783950816.92 total 155.99 last 22.98) - open/close 3939 (time 1783950827.89 total 166.96 last 17.96) - open/close 4188 (time 1783950837.91 total 176.97 last 24.86) - open/close 4401 (time 1783950850.93 total 190.00 last 16.35) - open/close 4631 (time 1783950863.48 total 202.55 last 18.33) - open/close 4820 (time 1783950873.50 total 212.56 last 18.87) - open/close 5050 (time 1783950883.97 total 223.03 last 21.96) - open/close 5333 (time 1783950894.85 total 233.91 last 26.01) - open/close 5558 (time 1783950906.24 total 245.31 last 19.75) - open/close 5802 (time 1783950917.88 total 256.95 last 20.97) - open/close 6004 (time 1783950928.12 total 267.19 last 19.72) - open/close 6201 (time 1783950938.12 total 277.19 last 19.70) - open/close 6394 (time 1783950948.16 total 287.23 last 19.22) - open/close 6632 (time 1783950958.17 total 297.24 last 23.77) - open/close 6793 (time 1783950968.44 total 307.51 last 15.68) - open/close 6941 (time 1783950978.47 total 317.54 last 14.75) - open/close 7155 (time 1783950988.80 total 327.87 last 20.71) - open/close 7406 (time 1783950998.81 total 337.88 last 25.08) - open/close 7612 (time 1783951008.83 total 347.89 last 20.57) - open/close 7811 (time 1783951019.44 total 358.51 last 18.74) - open/close 8046 (time 1783951029.46 total 368.53 last 23.46) - open/close 8265 (time 1783951039.54 total 378.60 last 21.73) - open/close 8505 (time 1783951050.11 total 389.17 last 22.71) - open/close 8667 (time 1783951061.17 total 400.24 last 14.64) - open/close 8824 (time 1783951073.21 total 412.28 last 13.04) - open/close 9027 (time 1783951083.76 total 422.83 last 19.24) - open/close 9211 (time 1783951094.67 total 433.74 last 16.86) - open/close 9413 (time 1783951105.73 total 444.80 last 18.27) - open/close 9665 (time 1783951115.75 total 454.81 last 25.16) - open/close 9848 (time 1783951125.77 total 464.83 last 18.26) - open/close 9956 (time 1783951136.11 total 475.18 last 10.44) - open/close 10088 (time 1783951146.29 total 485.36 last 12.97) - open/close 10324 (time 1783951157.11 total 496.17 last 21.82) - open/close 10437 (time 1783951168.14 total 507.21 last 10.24) - open/close 10620 (time 1783951178.85 total 517.92 last 17.09) - open/close 10744 (time 1783951189.70 total 528.77 last 11.43) - open/close 10849 (time 1783951202.09 total 541.16 last 8.47) - open/close 11007 (time 1783951212.43 total 551.49 last 15.29) - open/close 11199 (time 1783951222.76 total 561.83 last 18.58) - open/close 11414 (time 1783951233.46 total 572.52 last 20.10) - open/close 11699 (time 1783951244.40 total 583.47 last 26.05) - open/close 11839 (time 1783951254.70 total 593.77 last 13.59) - open/close 12081 (time 1783951267.41 total 606.47 last 19.05) - open/close 12309 (time 1783951277.65 total 616.71 last 22.26) - open/close 12514 (time 1783951287.66 total 626.73 last 20.47) - open/close 12770 (time 1783951298.18 total 637.24 last 24.34) - open/close 12870 (time 1783951308.67 total 647.74 last 9.53) - open/close 13058 (time 1783951318.67 total 657.74 last 18.80) - open/close 13312 (time 1783951328.68 total 667.75 last 25.39) - open/close 13506 (time 1783951341.64 total 680.70 last 14.97) - open/close 13672 (time 1783951351.93 total 690.99 last 16.13) - open/close 13967 (time 1783951362.54 total 701.61 last 27.80) - open/close 14061 (time 1783951372.86 total 711.92 last 9.11) - open/close 14250 (time 1783951382.88 total 721.94 last 18.86) - open/close 14516 (time 1783951392.89 total 731.95 last 26.57) - open/close 14661 (time 1783951403.28 total 742.35 last 13.95) - open/close 14794 (time 1783951413.82 total 752.88 last 12.62) - open/close 14906 (time 1783951424.13 total 763.20 last 10.86) - open/close 15014 (time 1783951435.06 total 774.12 last 9.89) - open/close 15207 (time 1783951445.08 total 784.14 last 19.27) - open/close 15401 (time 1783951455.48 total 794.54 last 18.65) - open/close 15627 (time 1783951465.59 total 804.66 last 22.34) - open/close 15846 (time 1783951475.60 total 814.67 last 21.89) - open/close 16106 (time 1783951486.26 total 825.32 last 24.39) - open/close 16374 (time 1783951496.27 total 835.33 last 26.77) - open/close 16577 (time 1783951506.42 total 845.48 last 20.00) - open/close 16851 (time 1783951517.45 total 856.52 last 24.83) - open/close 17171 (time 1783951527.45 total 866.52 last 31.99) - open/close 17426 (time 1783951537.47 total 876.54 last 25.45) - open/close 17666 (time 1783951548.20 total 887.27 last 22.37) - open/close 17871 (time 1783951558.22 total 897.28 last 20.47) - open/close 18099 (time 1783951568.57 total 907.64 last 22.02) - open/close 18197 (time 1783951578.87 total 917.93 last 9.52) - open/close 18252 (time 1783951589.43 total 928.49 last 5.21) - open/close 18387 (time 1783951599.43 total 938.49 last 13.50) - open/close 18590 (time 1783951609.47 total 948.53 last 20.22) - open/close 18753 (time 1783951619.48 total 958.54 last 16.28) - open/close 18894 (time 1783951630.01 total 969.07 last 13.39) - open/close 19146 (time 1783951640.01 total 979.08 last 25.20) - open/close 19805 (time 1783951650.02 total 989.08 last 65.86) - open/close 20000 (time 1783951652.54 total 991.61 last 77.16) - open/close 20660 (time 1783951662.55 total 1001.62 last 65.95) - open/close 21333 (time 1783951672.56 total 1011.63 last 67.23) - open/close 22023 (time 1783951682.57 total 1021.64 last 68.92) - open/close 22630 (time 1783951692.59 total 1031.66 last 60.58) - open/close 23255 (time 1783951702.61 total 1041.67 last 62.43) - open/close 23851 (time 1783951712.61 total 1051.67 last 59.58) - open/close 24462 (time 1783951722.62 total 1061.69 last 61.03) - open/close 25075 (time 1783951732.63 total 1071.69 last 61.26) - open/close 25694 (time 1783951742.63 total 1081.69 last 61.89) - open/close 26434 (time 1783951752.63 total 1091.70 last 73.96) - open/close 27163 (time 1783951762.64 total 1101.70 last 72.87) - open/close 27912 (time 1783951772.64 total 1111.71 last 74.84) - open/close 28621 (time 1783951782.66 total 1121.72 last 70.80) - open/close 29373 (time 1783951792.67 total 1131.74 last 75.11) - open/close 30000 (time 1783951801.87 total 1140.93 last 68.17) - open/close 30703 (time 1783951811.87 total 1150.94 last 70.27) - open/close 31393 (time 1783951821.88 total 1160.95 last 68.95) - open/close 31965 (time 1783951831.88 total 1170.95 last 57.18) - open/close 32673 (time 1783951841.90 total 1180.96 last 70.70) - open/close 33350 (time 1783951851.90 total 1190.97 last 67.67) - open/close 34039 (time 1783951861.91 total 1200.97 last 68.87) - open/close 34783 (time 1783951871.91 total 1210.97 last 74.38) - open/close 35535 (time 1783951881.92 total 1220.98 last 75.14) - open/close 36148 (time 1783951891.92 total 1230.99 last 61.26) - open/close 36820 (time 1783951901.93 total 1240.99 last 67.18) - open/close 37496 (time 1783951911.93 total 1250.99 last 67.58) - open/close 38214 (time 1783951921.93 total 1261.00 last 71.79) - open/close 38943 (time 1783951931.95 total 1271.02 last 72.73) - open/close 39639 (time 1783951941.96 total 1281.03 last 69.55) - open/close 40000 (time 1783951946.37 total 1285.43 last 81.95) - open/close 40727 (time 1783951956.37 total 1295.43 last 72.70) - open/close 41454 (time 1783951966.37 total 1305.44 last 72.65) - open/close 42245 (time 1783951976.37 total 1315.44 last 79.10) - open/close 42989 (time 1783951986.38 total 1325.44 last 74.38) - open/close 43752 (time 1783951996.38 total 1335.44 last 76.30) - open/close 44513 (time 1783952006.38 total 1345.45 last 76.07) - open/close 45193 (time 1783952016.38 total 1355.45 last 67.99) - open/close 45818 (time 1783952026.39 total 1365.45 last 62.46) - open/close 46542 (time 1783952036.41 total 1375.48 last 72.23) - open/close 47248 (time 1783952046.41 total 1385.48 last 70.59) - open/close 48028 (time 1783952056.41 total 1395.48 last 77.99) - open/close 48656 (time 1783952066.43 total 1405.50 last 62.68) - open/close 49217 (time 1783952076.47 total 1415.54 last 55.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No space left on device total: 49729 open/close in 1422.56 seconds: 34.96 ops/second - unlinked 0 (time 1783952087 ; total 0 ; last 0) - unlinked 10000 (time 1783952225 ; total 138 ; last 138) - unlinked 20000 (time 1783952369 ; total 282 ; last 144) - unlinked 30000 (time 1783952506 ; total 419 ; last 137) - unlinked 40000 (time 1783952646 ; total 559 ; last 140) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No such file or directory total: 49729 unlinks in 706 seconds: 70.437675 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2