== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:24:06 (1788272646) start mds service on oleg307-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg307-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 265 (time 1788272755.13 total 10.95 last 24.20) - open/close 434 (time 1788272765.31 total 21.13 last 16.60) - open/close 688 (time 1788272776.06 total 31.88 last 23.62) - open/close 944 (time 1788272786.57 total 42.39 last 24.37) - open/close 1233 (time 1788272796.58 total 52.40 last 28.86) - open/close 1498 (time 1788272806.78 total 62.61 last 25.97) - open/close 1796 (time 1788272817.86 total 73.68 last 26.91) - open/close 2027 (time 1788272828.53 total 84.35 last 21.64) - open/close 2276 (time 1788272838.71 total 94.53 last 24.46) - open/close 2575 (time 1788272849.79 total 105.61 last 26.98) - open/close 2833 (time 1788272860.22 total 116.04 last 24.74) - open/close 3121 (time 1788272871.11 total 126.93 last 26.45) - open/close 3368 (time 1788272881.93 total 137.75 last 22.83) - open/close 3646 (time 1788272892.94 total 148.76 last 25.23) - open/close 3922 (time 1788272902.95 total 158.77 last 27.59) - open/close 4110 (time 1788272913.41 total 169.24 last 17.96) - open/close 4322 (time 1788272923.42 total 179.25 last 21.18) - open/close 4579 (time 1788272933.57 total 189.39 last 25.34) - open/close 4815 (time 1788272943.58 total 199.40 last 23.56) - open/close 5075 (time 1788272953.97 total 209.80 last 25.02) - open/close 5281 (time 1788272964.35 total 220.17 last 19.85) - open/close 5505 (time 1788272974.66 total 230.48 last 21.73) - open/close 5720 (time 1788272984.67 total 240.49 last 21.48) - open/close 5887 (time 1788272995.08 total 250.91 last 16.03) - open/close 6130 (time 1788273005.69 total 261.51 last 22.92) - open/close 6417 (time 1788273015.92 total 271.74 last 28.04) - open/close 6582 (time 1788273026.56 total 282.38 last 15.51) - open/close 6749 (time 1788273040.96 total 296.78 last 11.59) - open/close 6910 (time 1788273051.46 total 307.28 last 15.34) - open/close 7111 (time 1788273061.70 total 317.52 last 19.63) - open/close 7280 (time 1788273072.32 total 328.14 last 15.91) - open/close 7533 (time 1788273082.83 total 338.65 last 24.07) - open/close 7728 (time 1788273092.83 total 348.66 last 19.49) - open/close 7826 (time 1788273103.10 total 358.92 last 9.54) - open/close 8021 (time 1788273113.11 total 368.93 last 19.49) - open/close 8298 (time 1788273124.10 total 379.92 last 25.20) - open/close 8500 (time 1788273134.10 total 389.92 last 20.19) - open/close 8579 (time 1788273144.13 total 399.95 last 7.88) - open/close 8762 (time 1788273154.14 total 409.96 last 18.29) - open/close 9005 (time 1788273164.14 total 419.97 last 24.28) - open/close 9213 (time 1788273174.16 total 429.98 last 20.77) - open/close 9477 (time 1788273184.16 total 439.98 last 26.39) - open/close 9621 (time 1788273194.17 total 449.99 last 14.39) - open/close 9794 (time 1788273209.88 total 465.70 last 11.02) - open/close 10000 (time 1788273217.28 total 473.10 last 27.83) - open/close 10254 (time 1788273227.65 total 483.48 last 24.47) - open/close 10517 (time 1788273237.96 total 493.78 last 25.53) - open/close 10751 (time 1788273247.97 total 503.79 last 23.37) - open/close 11017 (time 1788273257.97 total 513.80 last 26.58) - open/close 11284 (time 1788273267.98 total 523.80 last 26.69) - open/close 11462 (time 1788273277.98 total 533.80 last 17.79) - open/close 11716 (time 1788273288.53 total 544.35 last 24.07) - open/close 11982 (time 1788273299.33 total 555.15 last 24.64) - open/close 12268 (time 1788273309.33 total 565.15 last 28.60) - open/close 12406 (time 1788273323.20 total 579.02 last 9.95) - open/close 12706 (time 1788273333.54 total 589.36 last 29.01) - open/close 13017 (time 1788273343.54 total 599.36 last 31.09) - open/close 13250 (time 1788273353.62 total 609.44 last 23.12) - open/close 13459 (time 1788273363.62 total 619.44 last 20.90) - open/close 13708 (time 1788273374.14 total 629.96 last 23.68) - open/close 13943 (time 1788273384.65 total 640.47 last 22.35) - open/close 14194 (time 1788273395.59 total 651.41 last 22.94) - open/close 14423 (time 1788273405.96 total 661.79 last 22.08) - open/close 14650 (time 1788273416.00 total 671.82 last 22.61) - open/close 14898 (time 1788273426.94 total 682.76 last 22.68) - open/close 15123 (time 1788273437.84 total 693.66 last 20.63) - open/close 15385 (time 1788273448.39 total 704.21 last 24.84) - open/close 15653 (time 1788273458.88 total 714.71 last 25.53) - open/close 15953 (time 1788273468.90 total 724.72 last 29.95) - open/close 16177 (time 1788273479.56 total 735.38 last 21.01) - open/close 16472 (time 1788273489.57 total 745.39 last 29.48) - open/close 16711 (time 1788273499.59 total 755.41 last 23.86) - open/close 16968 (time 1788273510.13 total 765.95 last 24.37) - open/close 17222 (time 1788273520.37 total 776.19 last 24.80) - open/close 17463 (time 1788273530.38 total 786.20 last 24.08) - open/close 17692 (time 1788273540.40 total 796.23 last 22.84) - open/close 17928 (time 1788273550.42 total 806.24 last 23.56) - open/close 18160 (time 1788273560.42 total 816.24 last 23.20) - open/close 18389 (time 1788273570.43 total 826.25 last 22.89) - open/close 18591 (time 1788273580.61 total 836.44 last 19.83) - open/close 18837 (time 1788273590.63 total 846.45 last 24.57) - open/close 18968 (time 1788273600.65 total 856.47 last 13.07) - open/close 19178 (time 1788273610.66 total 866.48 last 20.98) - open/close 19427 (time 1788273620.68 total 876.50 last 24.86) - open/close 19641 (time 1788273631.36 total 887.18 last 20.04) - open/close 19867 (time 1788273641.36 total 897.18 last 22.59) - open/close 20000 (time 1788273645.41 total 901.24 last 32.82) - open/close 20435 (time 1788273655.42 total 911.24 last 43.48) - open/close 21226 (time 1788273665.43 total 921.25 last 79.00) - open/close 22061 (time 1788273675.44 total 931.26 last 83.43) - open/close 22895 (time 1788273685.44 total 941.27 last 83.36) - open/close 23749 (time 1788273695.46 total 951.28 last 85.31) - open/close 24514 (time 1788273705.46 total 961.28 last 76.44) - open/close 25401 (time 1788273715.46 total 971.29 last 88.68) - open/close 26229 (time 1788273725.47 total 981.29 last 82.78) - open/close 27027 (time 1788273735.47 total 991.29 last 79.79) - open/close 27868 (time 1788273745.49 total 1001.31 last 83.93) - open/close 28771 (time 1788273755.49 total 1011.31 last 90.29) - open/close 29618 (time 1788273765.49 total 1021.31 last 84.68) - open/close 30000 (time 1788273770.15 total 1025.97 last 82.11) - open/close 30823 (time 1788273780.15 total 1035.97 last 82.24) - open/close 31620 (time 1788273790.16 total 1045.98 last 79.63) - open/close 32453 (time 1788273800.17 total 1055.99 last 83.21) - open/close 33341 (time 1788273810.17 total 1066.00 last 88.78) - open/close 34282 (time 1788273820.18 total 1076.01 last 94.00) - open/close 35082 (time 1788273830.19 total 1086.01 last 79.98) - open/close 35835 (time 1788273840.19 total 1096.01 last 75.26) - open/close 36719 (time 1788273850.20 total 1106.02 last 88.37) - open/close 37628 (time 1788273860.20 total 1116.03 last 90.82) - open/close 38491 (time 1788273870.21 total 1126.03 last 86.27) - open/close 39301 (time 1788273880.21 total 1136.03 last 80.98) - open/close 40000 (time 1788273888.18 total 1144.00 last 87.67) - open/close 40927 (time 1788273898.19 total 1154.01 last 92.64) - open/close 41763 (time 1788273908.19 total 1164.02 last 83.56) - open/close 42498 (time 1788273918.20 total 1174.02 last 73.44) - open/close 43386 (time 1788273928.21 total 1184.03 last 88.76) - open/close 44328 (time 1788273938.21 total 1194.03 last 94.15) - open/close 45254 (time 1788273948.22 total 1204.04 last 92.58) - open/close 46175 (time 1788273958.22 total 1214.04 last 92.09) - open/close 47045 (time 1788273968.23 total 1224.05 last 86.91) - open/close 47937 (time 1788273978.24 total 1234.06 last 89.12) - open/close 48889 (time 1788273988.24 total 1244.06 last 95.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1252.27 seconds: 39.63 ops/second - unlinked 0 (time 1788273999 ; total 0 ; last 0) - unlinked 10000 (time 1788274100 ; total 101 ; last 101) - unlinked 20000 (time 1788274245 ; total 246 ; last 145) - unlinked 30000 (time 1788274316 ; total 317 ; last 71) - unlinked 40000 (time 1788274387 ; total 388 ; last 71) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 461 seconds: 107.659439 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: modules unloaded. pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2