== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:49:37 (1789120177) start mds service on oleg408-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1789120297.41 total 10.45 last 14.07) - open/close 323 (time 1789120308.34 total 21.38 last 16.11) - open/close 589 (time 1789120318.35 total 31.39 last 26.56) - open/close 751 (time 1789120328.35 total 41.40 last 16.20) - open/close 936 (time 1789120339.25 total 52.29 last 16.98) - open/close 1077 (time 1789120349.77 total 62.81 last 13.40) - open/close 1276 (time 1789120359.80 total 72.84 last 19.84) - open/close 1489 (time 1789120369.80 total 82.85 last 21.29) - open/close 1645 (time 1789120380.67 total 93.72 last 14.35) - open/close 1817 (time 1789120391.70 total 104.74 last 15.60) - open/close 1992 (time 1789120402.44 total 115.48 last 16.29) - open/close 2057 (time 1789120412.53 total 125.58 last 6.44) - open/close 2150 (time 1789120422.91 total 135.95 last 8.96) - open/close 2253 (time 1789120432.95 total 145.99 last 10.26) - open/close 2368 (time 1789120442.96 total 156.00 last 11.48) - open/close 2496 (time 1789120453.04 total 166.08 last 12.70) - open/close 2572 (time 1789120463.17 total 176.21 last 7.50) - open/close 2761 (time 1789120473.98 total 187.02 last 17.49) - open/close 2917 (time 1789120484.66 total 197.70 last 14.60) - open/close 3038 (time 1789120495.61 total 208.65 last 11.05) - open/close 3164 (time 1789120505.61 total 218.66 last 12.59) - open/close 3239 (time 1789120516.62 total 229.66 last 6.82) - open/close 3490 (time 1789120527.62 total 240.66 last 22.81) - open/close 3612 (time 1789120537.62 total 250.67 last 12.20) - open/close 3646 (time 1789120549.82 total 262.86 last 2.79) - open/close 3781 (time 1789120559.83 total 272.87 last 13.48) - open/close 3905 (time 1789120570.74 total 283.79 last 11.36) - open/close 4010 (time 1789120580.76 total 293.80 last 10.48) - open/close 4102 (time 1789120590.77 total 303.82 last 9.19) - open/close 4259 (time 1789120601.65 total 314.69 last 14.43) - open/close 4398 (time 1789120611.69 total 324.73 last 13.85) - open/close 4491 (time 1789120621.81 total 334.86 last 9.18) - open/close 4591 (time 1789120631.89 total 344.93 last 9.92) - open/close 4636 (time 1789120642.31 total 355.35 last 4.32) - open/close 4794 (time 1789120654.32 total 367.36 last 13.15) - open/close 4965 (time 1789120664.33 total 377.37 last 17.09) - open/close 5153 (time 1789120674.34 total 387.38 last 18.78) - open/close 5341 (time 1789120688.06 total 401.10 last 13.70) - open/close 5503 (time 1789120698.62 total 411.67 last 15.33) - open/close 5604 (time 1789120709.22 total 422.27 last 9.53) - open/close 5727 (time 1789120720.12 total 433.16 last 11.29) - open/close 5932 (time 1789120730.12 total 443.17 last 20.49) - open/close 6061 (time 1789120740.13 total 453.17 last 12.89) - open/close 6245 (time 1789120750.71 total 463.75 last 17.40) - open/close 6382 (time 1789120760.72 total 473.77 last 13.68) - open/close 6571 (time 1789120771.00 total 484.04 last 18.39) - open/close 6710 (time 1789120781.18 total 494.23 last 13.65) - open/close 6827 (time 1789120791.20 total 504.24 last 11.69) - open/close 6985 (time 1789120801.80 total 514.84 last 14.90) - open/close 7097 (time 1789120812.73 total 525.77 last 10.24) - open/close 7185 (time 1789120822.84 total 535.88 last 8.71) - open/close 7326 (time 1789120833.46 total 546.51 last 13.26) - open/close 7390 (time 1789120843.49 total 556.54 last 6.38) - open/close 7550 (time 1789120854.20 total 567.24 last 14.95) - open/close 7672 (time 1789120864.22 total 577.26 last 12.17) - open/close 7857 (time 1789120874.26 total 587.31 last 18.42) - open/close 7991 (time 1789120884.26 total 597.31 last 13.40) - open/close 8106 (time 1789120896.00 total 609.04 last 9.80) - open/close 8204 (time 1789120906.94 total 619.98 last 8.96) - open/close 8283 (time 1789120917.17 total 630.22 last 7.72) - open/close 8366 (time 1789120928.70 total 641.74 last 7.20) - open/close 8494 (time 1789120938.86 total 651.90 last 12.59) - open/close 8751 (time 1789120948.87 total 661.91 last 25.67) - open/close 8975 (time 1789120959.49 total 672.53 last 21.09) - open/close 9166 (time 1789120970.36 total 683.40 last 17.57) - open/close 9320 (time 1789120980.37 total 693.41 last 15.39) - open/close 9564 (time 1789120990.37 total 703.41 last 24.39) - open/close 9767 (time 1789121001.33 total 714.37 last 18.52) - open/close 9954 (time 1789121011.33 total 724.37 last 18.70) - open/close 10000 (time 1789121016.73 total 729.77 last 8.51) - open/close 10140 (time 1789121027.18 total 740.23 last 13.39) - open/close 10287 (time 1789121041.67 total 754.72 last 10.15) - open/close 10473 (time 1789121054.51 total 767.55 last 14.49) - open/close 10735 (time 1789121066.93 total 779.97 last 21.09) - open/close 10913 (time 1789121077.11 total 790.15 last 17.48) - open/close 10972 (time 1789121087.15 total 800.19 last 5.88) - open/close 11226 (time 1789121097.17 total 810.21 last 25.37) - open/close 11325 (time 1789121107.71 total 820.76 last 9.39) - open/close 11439 (time 1789121117.75 total 830.79 last 11.36) - open/close 11601 (time 1789121128.16 total 841.20 last 15.55) - open/close 11716 (time 1789121138.18 total 851.22 last 11.48) - open/close 11877 (time 1789121148.73 total 861.78 last 15.26) - open/close 12126 (time 1789121158.74 total 871.78 last 24.88) - open/close 12230 (time 1789121169.46 total 882.50 last 9.70) - open/close 12311 (time 1789121179.46 total 892.50 last 8.10) - open/close 12560 (time 1789121189.46 total 902.50 last 24.90) - open/close 12634 (time 1789121199.55 total 912.59 last 7.33) - open/close 12801 (time 1789121210.04 total 923.08 last 15.92) - open/close 13002 (time 1789121220.35 total 933.39 last 19.49) - open/close 13153 (time 1789121230.71 total 943.75 last 14.58) - open/close 13257 (time 1789121240.72 total 953.77 last 10.38) - open/close 13511 (time 1789121250.73 total 963.77 last 25.39) - open/close 13736 (time 1789121261.06 total 974.10 last 21.79) - open/close 14010 (time 1789121271.79 total 984.83 last 25.53) - open/close 14185 (time 1789121281.81 total 994.86 last 17.46) - open/close 14357 (time 1789121297.72 total 1010.76 last 10.81) - open/close 14559 (time 1789121307.73 total 1020.77 last 20.19) - open/close 14727 (time 1789121318.32 total 1031.36 last 15.86) - open/close 14932 (time 1789121328.65 total 1041.69 last 19.85) - open/close 15098 (time 1789121339.44 total 1052.48 last 15.39) - open/close 15292 (time 1789121350.38 total 1063.42 last 17.73) - open/close 15450 (time 1789121360.63 total 1073.67 last 15.41) - open/close 15727 (time 1789121371.06 total 1084.10 last 26.55) - open/close 15859 (time 1789121381.08 total 1094.12 last 13.18) - open/close 15952 (time 1789121391.86 total 1104.91 last 8.62) - open/close 16065 (time 1789121401.88 total 1114.92 last 11.28) - open/close 16710 (time 1789121411.90 total 1124.94 last 64.37) - open/close 17342 (time 1789121421.91 total 1134.95 last 63.16) - open/close 18000 (time 1789121431.92 total 1144.96 last 65.70) - open/close 18627 (time 1789121441.94 total 1154.99 last 62.55) - open/close 19387 (time 1789121451.97 total 1165.01 last 75.82) - open/close 20000 (time 1789121460.77 total 1173.81 last 69.67) - open/close 20787 (time 1789121470.78 total 1183.83 last 78.57) - open/close 21529 (time 1789121480.79 total 1193.83 last 74.16) - open/close 22181 (time 1789121490.79 total 1203.84 last 65.16) - open/close 22845 (time 1789121500.81 total 1213.85 last 66.33) - open/close 23487 (time 1789121510.82 total 1223.86 last 64.09) - open/close 24102 (time 1789121520.83 total 1233.87 last 61.44) - open/close 24759 (time 1789121530.84 total 1243.89 last 65.62) - open/close 25382 (time 1789121540.85 total 1253.89 last 62.28) - open/close 26059 (time 1789121550.87 total 1263.92 last 67.52) - open/close 26813 (time 1789121560.88 total 1273.92 last 75.35) - open/close 27538 (time 1789121570.90 total 1283.94 last 72.34) - open/close 28222 (time 1789121580.91 total 1293.95 last 68.38) - open/close 28958 (time 1789121590.92 total 1303.96 last 73.53) - open/close 29562 (time 1789121600.95 total 1313.99 last 60.22) - open/close 30000 (time 1789121608.22 total 1321.26 last 60.20) - open/close 30592 (time 1789121618.23 total 1331.27 last 59.13) - open/close 31289 (time 1789121628.26 total 1341.30 last 69.51) - open/close 31900 (time 1789121638.26 total 1351.31 last 61.07) - open/close 32598 (time 1789121648.27 total 1361.31 last 69.74) - open/close 33347 (time 1789121658.28 total 1371.32 last 74.86) - open/close 34045 (time 1789121668.28 total 1381.32 last 69.78) - open/close 34751 (time 1789121678.29 total 1391.33 last 70.58) - open/close 35415 (time 1789121688.29 total 1401.33 last 66.36) - open/close 36078 (time 1789121698.30 total 1411.34 last 66.23) - open/close 36693 (time 1789121708.33 total 1421.37 last 61.36) - open/close 37316 (time 1789121718.34 total 1431.38 last 62.22) - open/close 37931 (time 1789121728.34 total 1441.38 last 61.49) - open/close 38544 (time 1789121738.34 total 1451.39 last 61.27) - open/close 39271 (time 1789121748.35 total 1461.39 last 72.64) - open/close 39988 (time 1789121758.36 total 1471.41 last 71.62) - open/close 40621 (time 1789121768.37 total 1481.41 last 63.27) - open/close 41259 (time 1789121778.38 total 1491.42 last 63.72) - open/close 41737 (time 1789121788.38 total 1501.43 last 47.79) - open/close 42335 (time 1789121798.41 total 1511.45 last 59.66) - open/close 42919 (time 1789121808.42 total 1521.46 last 58.32) - open/close 43395 (time 1789121818.43 total 1531.47 last 47.57) - open/close 43830 (time 1789121828.43 total 1541.47 last 43.47) - open/close 44318 (time 1789121838.44 total 1551.48 last 48.77) - open/close 44993 (time 1789121848.44 total 1561.48 last 67.48) - open/close 45555 (time 1789121858.46 total 1571.50 last 56.10) - open/close 46171 (time 1789121868.46 total 1581.50 last 61.59) - open/close 46876 (time 1789121878.46 total 1591.51 last 70.47) - open/close 47518 (time 1789121888.47 total 1601.51 last 64.17) - open/close 48269 (time 1789121898.47 total 1611.51 last 75.10) - open/close 48973 (time 1789121908.48 total 1621.52 last 70.30) - open/close 49571 (time 1789121918.49 total 1631.53 last 59.74) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1632.56 seconds: 30.40 ops/second - unlinked 0 (time 1789121924 ; total 0 ; last 0) - unlinked 10000 (time 1789122069 ; total 145 ; last 145) - unlinked 20000 (time 1789122227 ; total 303 ; last 158) - unlinked 30000 (time 1789122396 ; total 472 ; last 169) - unlinked 40000 (time 1789122552 ; total 628 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 771 seconds: 64.370949 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2