== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:15:12 (1783970112) start mds service on oleg215-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg215-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 157 (time 1783970252.49 total 10.98 last 14.30) - open/close 264 (time 1783970262.67 total 21.16 last 10.51) - open/close 405 (time 1783970273.20 total 31.69 last 13.39) - open/close 569 (time 1783970286.02 total 44.51 last 12.80) - open/close 768 (time 1783970296.37 total 54.87 last 19.21) - open/close 995 (time 1783970306.96 total 65.45 last 21.44) - open/close 1229 (time 1783970316.97 total 75.46 last 23.38) - open/close 1347 (time 1783970327.70 total 86.19 last 11.00) - open/close 1481 (time 1783970337.72 total 96.21 last 13.37) - open/close 1599 (time 1783970348.17 total 106.66 last 11.30) - open/close 1813 (time 1783970358.60 total 117.10 last 20.51) - open/close 2084 (time 1783970369.48 total 127.97 last 24.92) - open/close 2301 (time 1783970380.10 total 138.59 last 20.44) - open/close 2483 (time 1783970390.12 total 148.61 last 18.16) - open/close 2539 (time 1783970400.74 total 159.24 last 5.27) - open/close 2668 (time 1783970412.77 total 171.26 last 10.73) - open/close 2834 (time 1783970423.83 total 182.32 last 15.01) - open/close 2921 (time 1783970436.36 total 194.85 last 6.94) - open/close 3060 (time 1783970446.37 total 204.86 last 13.89) - open/close 3242 (time 1783970456.53 total 215.03 last 17.90) - open/close 3389 (time 1783970466.69 total 225.19 last 14.47) - open/close 3471 (time 1783970476.71 total 235.21 last 8.18) - open/close 3706 (time 1783970486.73 total 245.22 last 23.47) - open/close 3855 (time 1783970496.84 total 255.33 last 14.73) - open/close 4041 (time 1783970508.91 total 267.41 last 15.41) - open/close 4272 (time 1783970518.92 total 277.42 last 23.08) - open/close 4518 (time 1783970528.92 total 287.42 last 24.59) - open/close 4675 (time 1783970539.41 total 297.90 last 14.98) - open/close 4798 (time 1783970550.36 total 308.86 last 11.23) - open/close 4967 (time 1783970560.39 total 318.88 last 16.85) - open/close 5139 (time 1783970570.39 total 328.89 last 17.20) - open/close 5311 (time 1783970582.35 total 340.84 last 14.39) - open/close 5351 (time 1783970592.77 total 351.27 last 3.84) - open/close 5462 (time 1783970602.87 total 361.37 last 10.99) - open/close 5669 (time 1783970613.85 total 372.34 last 18.87) - open/close 5830 (time 1783970624.84 total 383.34 last 14.64) - open/close 5975 (time 1783970635.36 total 393.85 last 13.79) - open/close 6070 (time 1783970645.98 total 404.47 last 8.95) - open/close 6264 (time 1783970661.62 total 420.11 last 12.41) - open/close 6450 (time 1783970672.49 total 430.98 last 17.11) - open/close 6530 (time 1783970682.50 total 440.99 last 7.99) - open/close 6645 (time 1783970693.52 total 452.02 last 10.43) - open/close 6850 (time 1783970704.10 total 462.59 last 19.38) - open/close 6929 (time 1783970717.34 total 475.84 last 5.96) - open/close 7039 (time 1783970727.87 total 486.37 last 10.44) - open/close 7179 (time 1783970738.25 total 496.74 last 13.50) - open/close 7379 (time 1783970748.42 total 506.91 last 19.67) - open/close 7523 (time 1783970759.08 total 517.57 last 13.51) - open/close 7611 (time 1783970770.32 total 528.82 last 7.82) - open/close 7893 (time 1783970780.56 total 539.05 last 27.55) - open/close 8159 (time 1783970791.26 total 549.75 last 24.85) - open/close 8222 (time 1783970801.28 total 559.78 last 6.28) - open/close 8340 (time 1783970811.58 total 570.07 last 11.46) - open/close 8573 (time 1783970822.36 total 580.86 last 21.60) - open/close 8706 (time 1783970832.37 total 590.87 last 13.29) - open/close 8870 (time 1783970843.23 total 601.73 last 15.10) - open/close 9025 (time 1783970853.60 total 612.10 last 14.95) - open/close 9104 (time 1783970863.61 total 622.11 last 7.89) - open/close 9183 (time 1783970873.87 total 632.37 last 7.70) - open/close 9309 (time 1783970884.55 total 643.04 last 11.81) - open/close 9372 (time 1783970894.55 total 653.04 last 6.30) - open/close 9490 (time 1783970907.29 total 665.79 last 9.26) - open/close 9671 (time 1783970918.24 total 676.73 last 16.54) - open/close 9811 (time 1783970930.82 total 689.32 last 11.13) - open/close 9950 (time 1783970940.83 total 699.32 last 13.89) - open/close 10000 (time 1783970944.87 total 703.36 last 12.38) - open/close 10155 (time 1783970954.96 total 713.45 last 15.36) - open/close 10319 (time 1783970965.83 total 724.32 last 15.08) - open/close 10509 (time 1783970976.77 total 735.27 last 17.36) - open/close 10604 (time 1783970986.83 total 745.32 last 9.45) - open/close 10713 (time 1783970997.32 total 755.82 last 10.39) - open/close 10901 (time 1783971008.34 total 766.83 last 17.07) - open/close 11121 (time 1783971018.89 total 777.38 last 20.85) - open/close 11208 (time 1783971028.95 total 787.45 last 8.64) - open/close 11389 (time 1783971039.03 total 797.53 last 17.95) - open/close 11516 (time 1783971050.10 total 808.59 last 11.48) - open/close 11709 (time 1783971061.14 total 819.64 last 17.47) - open/close 11960 (time 1783971071.15 total 829.65 last 25.07) - open/close 12111 (time 1783971081.30 total 839.79 last 14.89) - open/close 12198 (time 1783971091.54 total 850.04 last 8.49) - open/close 12366 (time 1783971101.55 total 860.05 last 16.78) - open/close 12517 (time 1783971111.92 total 870.41 last 14.57) - open/close 12677 (time 1783971122.76 total 881.26 last 14.75) - open/close 12727 (time 1783971133.46 total 891.96 last 4.67) - open/close 12927 (time 1783971144.35 total 902.84 last 18.37) - open/close 13084 (time 1783971154.38 total 912.87 last 15.66) - open/close 13239 (time 1783971165.03 total 923.52 last 14.55) - open/close 13402 (time 1783971175.25 total 933.74 last 15.95) - open/close 13629 (time 1783971185.27 total 943.76 last 22.66) - open/close 13688 (time 1783971195.29 total 953.78 last 5.89) - open/close 13854 (time 1783971205.30 total 963.80 last 16.58) - open/close 13958 (time 1783971216.16 total 974.65 last 9.58) - open/close 14133 (time 1783971226.16 total 984.66 last 17.49) - open/close 14261 (time 1783971236.82 total 995.31 last 12.01) - open/close 14386 (time 1783971246.85 total 1005.34 last 12.46) - open/close 14452 (time 1783971256.86 total 1015.35 last 6.60) - open/close 14571 (time 1783971267.86 total 1026.35 last 10.81) - open/close 14664 (time 1783971277.87 total 1036.37 last 9.29) - open/close 14768 (time 1783971288.72 total 1047.22 last 9.59) - open/close 14875 (time 1783971299.80 total 1058.29 last 9.66) - open/close 15049 (time 1783971309.81 total 1068.30 last 17.39) - open/close 15207 (time 1783971319.88 total 1078.38 last 15.68) - open/close 15353 (time 1783971330.02 total 1088.52 last 14.39) - open/close 15494 (time 1783971340.94 total 1099.44 last 12.91) - open/close 15574 (time 1783971351.12 total 1109.62 last 7.86) - open/close 16216 (time 1783971361.13 total 1119.62 last 64.19) - open/close 16977 (time 1783971371.13 total 1129.62 last 76.10) - open/close 17743 (time 1783971381.13 total 1139.62 last 76.58) - open/close 18420 (time 1783971391.14 total 1149.63 last 67.67) - open/close 19150 (time 1783971401.14 total 1159.63 last 72.99) - open/close 19964 (time 1783971411.14 total 1169.63 last 81.39) - open/close 20726 (time 1783971421.14 total 1179.64 last 76.15) - open/close 21500 (time 1783971431.15 total 1189.64 last 77.38) - open/close 22254 (time 1783971441.16 total 1199.65 last 75.32) - open/close 23132 (time 1783971451.18 total 1209.68 last 87.58) - open/close 24054 (time 1783971461.18 total 1219.68 last 92.20) - open/close 24952 (time 1783971471.20 total 1229.69 last 89.67) - open/close 25626 (time 1783971481.20 total 1239.70 last 67.35) - open/close 26524 (time 1783971491.22 total 1249.71 last 89.71) - open/close 27320 (time 1783971501.22 total 1259.72 last 79.54) - open/close 28234 (time 1783971511.23 total 1269.72 last 91.34) - open/close 29073 (time 1783971521.24 total 1279.73 last 83.82) - open/close 29893 (time 1783971531.24 total 1289.74 last 81.98) - open/close 30695 (time 1783971541.25 total 1299.74 last 80.17) - open/close 31499 (time 1783971551.25 total 1309.74 last 80.39) - open/close 32146 (time 1783971561.26 total 1319.75 last 64.65) - open/close 32891 (time 1783971571.26 total 1329.76 last 74.43) - open/close 33643 (time 1783971581.27 total 1339.77 last 75.12) - open/close 34351 (time 1783971591.28 total 1349.77 last 70.77) - open/close 34975 (time 1783971601.29 total 1359.79 last 62.33) - open/close 35799 (time 1783971611.29 total 1369.79 last 82.39) - open/close 36580 (time 1783971621.30 total 1379.79 last 78.08) - open/close 37281 (time 1783971631.30 total 1389.79 last 70.09) - open/close 38027 (time 1783971641.30 total 1399.80 last 74.57) - open/close 38771 (time 1783971651.30 total 1409.80 last 74.39) - open/close 39453 (time 1783971661.31 total 1419.81 last 68.14) - open/close 40000 (time 1783971669.92 total 1428.41 last 63.56) - open/close 40678 (time 1783971679.93 total 1438.43 last 67.71) - open/close 41265 (time 1783971689.95 total 1448.44 last 58.61) - open/close 41817 (time 1783971699.96 total 1458.45 last 55.15) - open/close 42433 (time 1783971709.96 total 1468.46 last 61.57) - open/close 43090 (time 1783971719.97 total 1478.47 last 65.63) - open/close 43813 (time 1783971729.97 total 1488.47 last 72.29) - open/close 44654 (time 1783971739.99 total 1498.48 last 84.00) - open/close 45466 (time 1783971749.99 total 1508.48 last 81.19) - open/close 46177 (time 1783971760.00 total 1518.49 last 71.02) - open/close 46829 (time 1783971770.01 total 1528.51 last 65.11) - open/close 47641 (time 1783971780.02 total 1538.52 last 81.11) - open/close 48450 (time 1783971790.03 total 1548.52 last 80.88) - open/close 49187 (time 1783971800.03 total 1558.53 last 73.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1564.19 seconds: 31.73 ops/second - unlinked 0 (time 1783971810 ; total 0 ; last 0) - unlinked 10000 (time 1783971962 ; total 152 ; last 152) - unlinked 20000 (time 1783972111 ; total 301 ; last 149) - unlinked 30000 (time 1783972249 ; total 439 ; last 138) - unlinked 40000 (time 1783972385 ; total 575 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 711 seconds: 69.804504 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2