== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:35:04 (1789140904) start mds service on oleg415-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg415-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg415-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg415-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 297 (time 1789141038.91 total 10.80 last 27.51) - open/close 438 (time 1789141049.01 total 20.90 last 13.96) - open/close 630 (time 1789141059.89 total 31.78 last 17.64) - open/close 792 (time 1789141070.94 total 42.82 last 14.67) - open/close 963 (time 1789141080.97 total 52.86 last 17.04) - open/close 1017 (time 1789141091.65 total 63.53 last 5.06) - open/close 1161 (time 1789141102.44 total 74.33 last 13.34) - open/close 1290 (time 1789141112.70 total 84.58 last 12.58) - open/close 1432 (time 1789141122.74 total 94.63 last 14.14) - open/close 1605 (time 1789141132.74 total 104.63 last 17.30) - open/close 1793 (time 1789141143.77 total 115.66 last 17.04) - open/close 1928 (time 1789141153.78 total 125.67 last 13.49) - open/close 2008 (time 1789141163.80 total 135.68 last 7.98) - open/close 2236 (time 1789141175.87 total 147.76 last 18.88) - open/close 2350 (time 1789141186.75 total 158.64 last 10.48) - open/close 2430 (time 1789141197.72 total 169.61 last 7.29) - open/close 2602 (time 1789141208.44 total 180.32 last 16.05) - open/close 2761 (time 1789141218.46 total 190.35 last 15.87) - open/close 2907 (time 1789141228.88 total 200.77 last 14.01) - open/close 2976 (time 1789141239.75 total 211.64 last 6.35) - open/close 3099 (time 1789141251.38 total 223.27 last 10.58) - open/close 3260 (time 1789141261.81 total 233.70 last 15.43) - open/close 3383 (time 1789141271.82 total 243.71 last 12.29) - open/close 3593 (time 1789141283.90 total 255.79 last 17.39) - open/close 3820 (time 1789141293.93 total 265.82 last 22.64) - open/close 4064 (time 1789141304.54 total 276.42 last 23.00) - open/close 4204 (time 1789141315.53 total 287.42 last 12.73) - open/close 4355 (time 1789141325.55 total 297.43 last 15.08) - open/close 4533 (time 1789141335.57 total 307.45 last 17.76) - open/close 4683 (time 1789141346.51 total 318.40 last 13.70) - open/close 4784 (time 1789141357.37 total 329.26 last 9.30) - open/close 4911 (time 1789141367.80 total 339.68 last 12.18) - open/close 5165 (time 1789141378.30 total 350.19 last 24.18) - open/close 5356 (time 1789141388.55 total 360.44 last 18.64) - open/close 5528 (time 1789141398.57 total 370.46 last 17.16) - open/close 5741 (time 1789141409.09 total 380.98 last 20.24) - open/close 5926 (time 1789141419.89 total 391.78 last 17.13) - open/close 5979 (time 1789141430.39 total 402.28 last 5.05) - open/close 6135 (time 1789141441.07 total 412.96 last 14.61) - open/close 6314 (time 1789141451.28 total 423.17 last 17.53) - open/close 6359 (time 1789141461.74 total 433.63 last 4.30) - open/close 6489 (time 1789141471.75 total 443.64 last 12.99) - open/close 6611 (time 1789141481.78 total 453.67 last 12.16) - open/close 6741 (time 1789141491.80 total 463.69 last 12.98) - open/close 6890 (time 1789141502.54 total 474.42 last 13.88) - open/close 7070 (time 1789141512.55 total 484.44 last 17.98) - open/close 7214 (time 1789141523.33 total 495.22 last 13.35) - open/close 7429 (time 1789141534.20 total 506.09 last 19.78) - open/close 7457 (time 1789141544.21 total 516.10 last 2.80) - open/close 7618 (time 1789141554.93 total 526.82 last 15.02) - open/close 7787 (time 1789141567.60 total 539.49 last 13.34) - open/close 7862 (time 1789141577.92 total 549.81 last 7.26) - open/close 7975 (time 1789141588.42 total 560.30 last 10.77) - open/close 8060 (time 1789141598.42 total 570.31 last 8.50) - open/close 8139 (time 1789141608.44 total 580.32 last 7.89) - open/close 8231 (time 1789141618.81 total 590.70 last 8.87) - open/close 8343 (time 1789141630.96 total 602.85 last 9.22) - open/close 8447 (time 1789141641.42 total 613.30 last 9.95) - open/close 8539 (time 1789141652.09 total 623.98 last 8.62) - open/close 8645 (time 1789141662.10 total 633.99 last 10.59) - open/close 8732 (time 1789141672.69 total 644.58 last 8.21) - open/close 8894 (time 1789141682.83 total 654.72 last 15.98) - open/close 9065 (time 1789141693.63 total 665.52 last 15.83) - open/close 9151 (time 1789141703.87 total 675.76 last 8.40) - open/close 9240 (time 1789141715.00 total 686.89 last 8.00) - open/close 9469 (time 1789141725.69 total 697.58 last 21.41) - open/close 9639 (time 1789141735.71 total 707.60 last 16.97) - open/close 9745 (time 1789141746.58 total 718.46 last 9.75) - open/close 9884 (time 1789141758.13 total 730.02 last 12.03) - open/close 9931 (time 1789141768.59 total 740.48 last 4.49) - open/close 10000 (time 1789141776.18 total 748.07 last 9.09) - open/close 10133 (time 1789141786.20 total 758.08 last 13.28) - open/close 10252 (time 1789141797.18 total 769.07 last 10.83) - open/close 10436 (time 1789141807.18 total 779.07 last 18.40) - open/close 10664 (time 1789141817.59 total 789.48 last 21.90) - open/close 10902 (time 1789141828.59 total 800.48 last 21.63) - open/close 11033 (time 1789141840.44 total 812.32 last 11.06) - open/close 11176 (time 1789141850.44 total 822.32 last 14.30) - open/close 11431 (time 1789141860.44 total 832.33 last 25.49) - open/close 11665 (time 1789141870.46 total 842.34 last 23.36) - open/close 11868 (time 1789141880.46 total 852.35 last 20.29) - open/close 12101 (time 1789141890.48 total 862.37 last 23.25) - open/close 12297 (time 1789141900.98 total 872.87 last 18.67) - open/close 12574 (time 1789141911.01 total 882.89 last 27.63) - open/close 12711 (time 1789141921.01 total 892.90 last 13.69) - open/close 12894 (time 1789141931.03 total 902.92 last 18.26) - open/close 13155 (time 1789141941.04 total 912.92 last 26.10) - open/close 13339 (time 1789141951.04 total 922.93 last 18.39) - open/close 13474 (time 1789141961.47 total 933.36 last 12.94) - open/close 13754 (time 1789141972.54 total 944.43 last 25.30) - open/close 14014 (time 1789141982.70 total 954.59 last 25.58) - open/close 14218 (time 1789141992.96 total 964.85 last 19.89) - open/close 14470 (time 1789142005.17 total 977.06 last 20.64) - open/close 14597 (time 1789142015.18 total 987.07 last 12.69) - open/close 14796 (time 1789142025.19 total 997.08 last 19.88) - open/close 14897 (time 1789142036.10 total 1007.99 last 9.26) - open/close 15037 (time 1789142046.14 total 1018.03 last 13.95) - open/close 15227 (time 1789142056.43 total 1028.32 last 18.46) - open/close 15479 (time 1789142066.44 total 1038.33 last 25.17) - open/close 15616 (time 1789142076.46 total 1048.35 last 13.68) - open/close 15852 (time 1789142086.60 total 1058.49 last 23.28) - open/close 16095 (time 1789142096.65 total 1068.54 last 24.18) - open/close 16380 (time 1789142106.65 total 1078.54 last 28.49) - open/close 16618 (time 1789142116.67 total 1088.56 last 23.76) - open/close 16985 (time 1789142126.68 total 1098.56 last 36.68) - open/close 17725 (time 1789142136.68 total 1108.57 last 73.93) - open/close 18449 (time 1789142146.69 total 1118.58 last 72.36) - open/close 19134 (time 1789142156.70 total 1128.59 last 68.45) - open/close 19846 (time 1789142166.70 total 1138.59 last 71.16) - open/close 20000 (time 1789142168.97 total 1140.86 last 67.99) - open/close 20692 (time 1789142178.98 total 1150.86 last 69.15) - open/close 21340 (time 1789142188.98 total 1160.87 last 64.77) - open/close 22137 (time 1789142198.99 total 1170.87 last 79.67) - open/close 22811 (time 1789142209.00 total 1180.89 last 67.31) - open/close 23634 (time 1789142219.01 total 1190.90 last 82.18) - open/close 24334 (time 1789142229.03 total 1200.91 last 69.89) - open/close 25036 (time 1789142239.03 total 1210.92 last 70.16) - open/close 25790 (time 1789142249.04 total 1220.92 last 75.37) - open/close 26569 (time 1789142259.05 total 1230.94 last 77.79) - open/close 27187 (time 1789142269.06 total 1240.95 last 61.73) - open/close 27847 (time 1789142279.06 total 1250.95 last 65.99) - open/close 28491 (time 1789142289.10 total 1260.98 last 64.18) - open/close 29187 (time 1789142299.11 total 1271.00 last 69.50) - open/close 29923 (time 1789142309.13 total 1281.02 last 73.46) - open/close 30747 (time 1789142319.14 total 1291.03 last 82.33) - open/close 31454 (time 1789142329.15 total 1301.04 last 70.63) - open/close 32172 (time 1789142339.15 total 1311.04 last 71.78) - open/close 32942 (time 1789142349.15 total 1321.04 last 76.98) - open/close 33700 (time 1789142359.15 total 1331.04 last 75.79) - open/close 34408 (time 1789142369.16 total 1341.05 last 70.76) - open/close 35082 (time 1789142379.18 total 1351.07 last 67.28) - open/close 35745 (time 1789142389.18 total 1361.07 last 66.27) - open/close 36456 (time 1789142399.19 total 1371.08 last 71.03) - open/close 37134 (time 1789142409.21 total 1381.10 last 67.67) - open/close 37806 (time 1789142419.22 total 1391.11 last 67.13) - open/close 38502 (time 1789142429.22 total 1401.11 last 69.58) - open/close 39286 (time 1789142439.23 total 1411.12 last 78.37) - open/close 40000 (time 1789142448.59 total 1420.48 last 76.24) - open/close 40708 (time 1789142458.60 total 1430.48 last 70.79) - open/close 41399 (time 1789142468.60 total 1440.49 last 69.04) - open/close 42143 (time 1789142478.64 total 1450.52 last 74.17) - open/close 42869 (time 1789142488.64 total 1460.53 last 72.54) - open/close 43574 (time 1789142498.65 total 1470.53 last 70.47) - open/close 44352 (time 1789142508.65 total 1480.54 last 77.76) - open/close 45045 (time 1789142518.67 total 1490.55 last 69.21) - open/close 45738 (time 1789142528.67 total 1500.56 last 69.24) - open/close 46451 (time 1789142538.68 total 1510.56 last 71.29) - open/close 47193 (time 1789142548.68 total 1520.57 last 74.14) - open/close 47883 (time 1789142558.68 total 1530.57 last 69.00) - open/close 48587 (time 1789142568.69 total 1540.58 last 70.36) - open/close 49292 (time 1789142578.71 total 1550.59 last 70.38) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1554.97 seconds: 31.92 ops/second - unlinked 0 (time 1789142587 ; total 0 ; last 0) - unlinked 10000 (time 1789142727 ; total 140 ; last 140) - unlinked 20000 (time 1789142867 ; total 280 ; last 140) - unlinked 30000 (time 1789143005 ; total 418 ; last 138) - unlinked 40000 (time 1789143148 ; total 561 ; last 143) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 693 seconds: 71.616165 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server unloading modules via unload_modules_local on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local oleg415-server: modules unloaded. pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2