== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:40:42 (1788446442) start mds service on oleg345-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg345-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg345-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1788446545.21 total 10.74 last 17.69) - open/close 463 (time 1788446556.03 total 21.56 last 25.23) - open/close 715 (time 1788446566.05 total 31.58 last 25.15) - open/close 845 (time 1788446576.89 total 42.42 last 11.99) - open/close 1088 (time 1788446587.33 total 52.86 last 23.28) - open/close 1359 (time 1788446598.16 total 63.69 last 25.02) - open/close 1494 (time 1788446609.16 total 74.69 last 12.27) - open/close 1764 (time 1788446619.18 total 84.71 last 26.95) - open/close 2001 (time 1788446629.19 total 94.72 last 23.68) - open/close 2252 (time 1788446639.36 total 104.89 last 24.68) - open/close 2515 (time 1788446649.36 total 114.89 last 26.29) - open/close 2734 (time 1788446660.10 total 125.63 last 20.40) - open/close 2973 (time 1788446670.33 total 135.86 last 23.35) - open/close 3235 (time 1788446681.20 total 146.74 last 24.10) - open/close 3438 (time 1788446691.23 total 156.76 last 20.24) - open/close 3563 (time 1788446701.36 total 166.90 last 12.34) - open/close 3717 (time 1788446711.88 total 177.41 last 14.65) - open/close 3899 (time 1788446723.52 total 189.06 last 15.63) - open/close 4231 (time 1788446734.02 total 199.55 last 31.63) - open/close 4464 (time 1788446744.06 total 209.59 last 23.21) - open/close 4714 (time 1788446754.07 total 219.60 last 24.97) - open/close 4957 (time 1788446764.69 total 230.22 last 22.88) - open/close 5227 (time 1788446775.29 total 240.82 last 25.47) - open/close 5452 (time 1788446785.98 total 251.51 last 21.05) - open/close 5693 (time 1788446795.99 total 261.52 last 24.08) - open/close 5916 (time 1788446806.01 total 271.54 last 22.27) - open/close 6163 (time 1788446816.37 total 281.90 last 23.83) - open/close 6404 (time 1788446826.95 total 292.48 last 22.78) - open/close 6661 (time 1788446837.96 total 303.49 last 23.34) - open/close 6912 (time 1788446848.00 total 313.53 last 25.00) - open/close 7023 (time 1788446860.29 total 325.82 last 9.03) - open/close 7276 (time 1788446871.23 total 336.76 last 23.14) - open/close 7414 (time 1788446881.24 total 346.77 last 13.78) - open/close 7598 (time 1788446893.50 total 359.04 last 15.00) - open/close 7745 (time 1788446903.68 total 369.21 last 14.45) - open/close 8006 (time 1788446913.71 total 379.24 last 26.03) - open/close 8233 (time 1788446923.72 total 389.25 last 22.67) - open/close 8472 (time 1788446933.72 total 399.25 last 23.90) - open/close 8737 (time 1788446944.12 total 409.65 last 25.48) - open/close 9011 (time 1788446955.08 total 420.62 last 24.99) - open/close 9221 (time 1788446966.09 total 431.62 last 19.08) - open/close 9516 (time 1788446976.10 total 441.63 last 29.47) - open/close 9749 (time 1788446986.11 total 451.65 last 23.27) - open/close 9783 (time 1788446996.13 total 461.66 last 3.40) - open/close 9981 (time 1788447007.20 total 472.73 last 17.89) - open/close 10228 (time 1788447017.91 total 483.44 last 23.05) - open/close 10472 (time 1788447028.24 total 493.77 last 23.63) - open/close 10706 (time 1788447038.46 total 503.99 last 22.88) - open/close 10999 (time 1788447048.76 total 514.29 last 28.45) - open/close 11235 (time 1788447059.00 total 524.53 last 23.06) - open/close 11475 (time 1788447070.66 total 536.19 last 20.58) - open/close 11725 (time 1788447081.72 total 547.25 last 22.60) - open/close 11958 (time 1788447094.03 total 559.56 last 18.93) - open/close 12225 (time 1788447104.89 total 570.42 last 24.60) - open/close 12506 (time 1788447114.89 total 580.42 last 28.10) - open/close 12706 (time 1788447125.38 total 590.91 last 19.06) - open/close 13016 (time 1788447135.97 total 601.50 last 29.28) - open/close 13267 (time 1788447146.74 total 612.27 last 23.30) - open/close 13463 (time 1788447156.74 total 622.28 last 19.59) - open/close 13625 (time 1788447167.42 total 632.95 last 15.17) - open/close 13783 (time 1788447177.98 total 643.51 last 14.97) - open/close 14047 (time 1788447187.99 total 653.52 last 26.37) - open/close 14269 (time 1788447200.58 total 666.11 last 17.64) - open/close 14386 (time 1788447210.77 total 676.30 last 11.48) - open/close 14502 (time 1788447221.77 total 687.31 last 10.54) - open/close 14657 (time 1788447232.00 total 697.53 last 15.16) - open/close 14816 (time 1788447242.94 total 708.47 last 14.53) - open/close 14981 (time 1788447253.22 total 718.75 last 16.05) - open/close 15230 (time 1788447264.00 total 729.53 last 23.11) - open/close 15477 (time 1788447274.01 total 739.54 last 24.67) - open/close 15784 (time 1788447284.73 total 750.26 last 28.64) - open/close 16040 (time 1788447295.24 total 760.77 last 24.36) - open/close 16299 (time 1788447305.24 total 770.77 last 25.90) - open/close 16585 (time 1788447315.25 total 780.78 last 28.58) - open/close 16805 (time 1788447325.43 total 790.96 last 21.61) - open/close 17020 (time 1788447335.74 total 801.27 last 20.86) - open/close 17238 (time 1788447345.75 total 811.28 last 21.77) - open/close 17370 (time 1788447356.65 total 822.19 last 12.11) - open/close 17566 (time 1788447367.61 total 833.14 last 17.89) - open/close 17720 (time 1788447377.63 total 843.16 last 15.37) - open/close 17920 (time 1788447388.35 total 853.88 last 18.65) - open/close 18071 (time 1788447399.44 total 864.97 last 13.62) - open/close 18372 (time 1788447410.33 total 875.86 last 27.64) - open/close 18568 (time 1788447420.34 total 885.87 last 19.57) - open/close 18786 (time 1788447430.65 total 896.18 last 21.15) - open/close 19059 (time 1788447442.57 total 908.11 last 22.90) - open/close 19112 (time 1788447453.13 total 918.66 last 5.02) - open/close 19282 (time 1788447463.99 total 929.53 last 15.65) - open/close 19439 (time 1788447474.00 total 939.53 last 15.69) - open/close 19606 (time 1788447484.12 total 949.65 last 16.51) - open/close 20000 (time 1788447491.01 total 956.54 last 57.20) - open/close 20689 (time 1788447501.01 total 966.54 last 68.86) - open/close 21319 (time 1788447511.02 total 976.56 last 62.91) - open/close 21941 (time 1788447521.03 total 986.56 last 62.18) - open/close 22530 (time 1788447531.04 total 996.57 last 58.82) - open/close 23186 (time 1788447541.08 total 1006.61 last 65.39) - open/close 23889 (time 1788447551.08 total 1016.61 last 70.26) - open/close 24593 (time 1788447561.09 total 1026.62 last 70.35) - open/close 25275 (time 1788447571.10 total 1036.63 last 68.11) - open/close 25913 (time 1788447581.11 total 1046.64 last 63.73) - open/close 26650 (time 1788447591.13 total 1056.66 last 73.60) - open/close 27343 (time 1788447601.13 total 1066.66 last 69.27) - open/close 28081 (time 1788447611.14 total 1076.67 last 73.76) - open/close 28802 (time 1788447621.15 total 1086.68 last 72.04) - open/close 29563 (time 1788447631.16 total 1096.69 last 75.99) - open/close 30000 (time 1788447636.63 total 1102.16 last 79.87) - open/close 30709 (time 1788447646.63 total 1112.16 last 70.89) - open/close 31486 (time 1788447656.63 total 1122.16 last 77.69) - open/close 32208 (time 1788447666.66 total 1132.19 last 72.02) - open/close 32910 (time 1788447676.67 total 1142.20 last 70.14) - open/close 33594 (time 1788447686.70 total 1152.23 last 68.17) - open/close 34264 (time 1788447696.70 total 1162.24 last 66.97) - open/close 34949 (time 1788447706.72 total 1172.25 last 68.41) - open/close 35660 (time 1788447716.72 total 1182.25 last 71.09) - open/close 36368 (time 1788447726.72 total 1192.25 last 70.79) - open/close 37036 (time 1788447736.73 total 1202.26 last 66.75) - open/close 37848 (time 1788447746.73 total 1212.27 last 81.15) - open/close 38520 (time 1788447756.75 total 1222.28 last 67.09) - open/close 39263 (time 1788447766.76 total 1232.29 last 74.26) - open/close 39935 (time 1788447776.76 total 1242.30 last 67.14) - open/close 40628 (time 1788447786.78 total 1252.32 last 69.17) - open/close 41211 (time 1788447796.79 total 1262.32 last 58.28) - open/close 41988 (time 1788447806.79 total 1272.33 last 77.64) - open/close 42781 (time 1788447816.80 total 1282.34 last 79.22) - open/close 43554 (time 1788447826.81 total 1292.34 last 77.24) - open/close 44280 (time 1788447836.82 total 1302.35 last 72.55) - open/close 44962 (time 1788447846.83 total 1312.36 last 68.16) - open/close 45662 (time 1788447856.83 total 1322.36 last 69.96) - open/close 46358 (time 1788447866.84 total 1332.37 last 69.52) - open/close 47042 (time 1788447876.87 total 1342.41 last 68.19) - open/close 47729 (time 1788447886.90 total 1352.43 last 68.53) - open/close 48400 (time 1788447896.91 total 1362.44 last 67.05) - open/close 49183 (time 1788447906.91 total 1372.44 last 78.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1379.55 seconds: 35.98 ops/second - unlinked 0 (time 1788447918 ; total 0 ; last 0) - unlinked 10000 (time 1788448066 ; total 148 ; last 148) - unlinked 20000 (time 1788448217 ; total 299 ; last 151) - unlinked 30000 (time 1788448363 ; total 445 ; last 146) - unlinked 40000 (time 1788448517 ; total 599 ; last 154) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 735 seconds: 67.525169 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2