== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:00:53 (1784192453) start mds service on oleg108-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg108-server: detected 4 online CPUs by sysfs oleg108-server: MODOPTS_LIBCFS= oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg108-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg108-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg108-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock 192.168.201.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1784192581.79 total 11.77 last 15.37) - open/close 448 (time 1784192592.28 total 22.26 last 25.46) - open/close 648 (time 1784192602.70 total 32.69 last 19.19) - open/close 663 (time 1784192614.42 total 44.40 last 1.28) - open/close 913 (time 1784192625.51 total 55.49 last 22.56) - open/close 1165 (time 1784192635.51 total 65.49 last 25.19) - open/close 1335 (time 1784192645.53 total 75.51 last 16.97) - open/close 1532 (time 1784192655.54 total 85.52 last 19.68) - open/close 1636 (time 1784192665.95 total 95.93 last 9.99) - open/close 1821 (time 1784192676.72 total 106.70 last 17.17) - open/close 2082 (time 1784192687.56 total 117.54 last 24.08) - open/close 2207 (time 1784192697.57 total 127.55 last 12.50) - open/close 2341 (time 1784192707.81 total 137.79 last 13.08) - open/close 2517 (time 1784192718.86 total 148.84 last 15.93) - open/close 2801 (time 1784192728.88 total 158.86 last 28.35) - open/close 2933 (time 1784192739.16 total 169.14 last 12.84) - open/close 3090 (time 1784192749.27 total 179.25 last 15.53) - open/close 3265 (time 1784192759.28 total 189.26 last 17.48) - open/close 3427 (time 1784192769.30 total 199.28 last 16.17) - open/close 3549 (time 1784192779.31 total 209.29 last 12.18) - open/close 3633 (time 1784192789.34 total 219.33 last 8.37) - open/close 3877 (time 1784192799.35 total 229.33 last 24.40) - open/close 4115 (time 1784192809.37 total 239.35 last 23.74) - open/close 4184 (time 1784192821.60 total 251.58 last 5.64) - open/close 4273 (time 1784192832.10 total 262.08 last 8.47) - open/close 4418 (time 1784192842.33 total 272.31 last 14.18) - open/close 4462 (time 1784192852.77 total 282.75 last 4.22) - open/close 4695 (time 1784192863.52 total 293.50 last 21.67) - open/close 4860 (time 1784192874.01 total 303.99 last 15.73) - open/close 5061 (time 1784192884.03 total 314.01 last 20.06) - open/close 5231 (time 1784192894.81 total 324.79 last 15.77) - open/close 5406 (time 1784192905.62 total 335.60 last 16.19) - open/close 5576 (time 1784192916.71 total 346.69 last 15.34) - open/close 5816 (time 1784192926.74 total 356.72 last 23.92) - open/close 6053 (time 1784192937.84 total 367.82 last 21.36) - open/close 6232 (time 1784192947.86 total 377.84 last 17.86) - open/close 6312 (time 1784192958.88 total 388.86 last 7.26) - open/close 6553 (time 1784192968.89 total 398.87 last 24.09) - open/close 6655 (time 1784192979.56 total 409.54 last 9.55) - open/close 6811 (time 1784192989.56 total 419.55 last 15.60) - open/close 6973 (time 1784192999.98 total 429.96 last 15.55) - open/close 7045 (time 1784193010.70 total 440.68 last 6.72) - open/close 7182 (time 1784193021.16 total 451.14 last 13.11) - open/close 7309 (time 1784193031.98 total 461.96 last 11.73) - open/close 7503 (time 1784193042.36 total 472.34 last 18.69) - open/close 7628 (time 1784193052.38 total 482.36 last 12.47) - open/close 7905 (time 1784193063.25 total 493.23 last 25.49) - open/close 8171 (time 1784193073.94 total 503.92 last 24.90) - open/close 8357 (time 1784193084.44 total 514.42 last 17.71) - open/close 8465 (time 1784193094.86 total 524.84 last 10.36) - open/close 8674 (time 1784193105.90 total 535.88 last 18.93) - open/close 8877 (time 1784193115.91 total 545.89 last 20.28) - open/close 9014 (time 1784193126.11 total 556.10 last 13.43) - open/close 9186 (time 1784193138.98 total 568.96 last 13.37) - open/close 9398 (time 1784193151.22 total 581.20 last 17.32) - open/close 9561 (time 1784193163.60 total 593.58 last 13.17) - open/close 9711 (time 1784193173.93 total 603.91 last 14.51) - open/close 9910 (time 1784193184.98 total 614.96 last 18.01) - open/close 10000 (time 1784193190.48 total 620.46 last 16.35) - open/close 10111 (time 1784193200.50 total 630.48 last 11.08) - open/close 10372 (time 1784193210.85 total 640.83 last 25.21) - open/close 10669 (time 1784193220.87 total 650.85 last 29.66) - open/close 10884 (time 1784193231.03 total 661.02 last 21.14) - open/close 11122 (time 1784193241.92 total 671.90 last 21.87) - open/close 11356 (time 1784193252.92 total 682.90 last 21.27) - open/close 11525 (time 1784193263.12 total 693.10 last 16.57) - open/close 11625 (time 1784193273.67 total 703.65 last 9.48) - open/close 11987 (time 1784193284.10 total 714.08 last 34.71) - open/close 12163 (time 1784193294.34 total 724.32 last 17.18) - open/close 12305 (time 1784193304.36 total 734.34 last 14.18) - open/close 12395 (time 1784193314.38 total 744.36 last 8.99) - open/close 12550 (time 1784193324.39 total 754.37 last 15.48) - open/close 12827 (time 1784193334.51 total 764.49 last 27.37) - open/close 13032 (time 1784193344.53 total 774.51 last 20.47) - open/close 13198 (time 1784193355.29 total 785.27 last 15.43) - open/close 13392 (time 1784193365.30 total 795.28 last 19.39) - open/close 13541 (time 1784193375.59 total 805.57 last 14.47) - open/close 13683 (time 1784193385.76 total 815.74 last 13.96) - open/close 13831 (time 1784193396.58 total 826.56 last 13.68) - open/close 13924 (time 1784193406.63 total 836.61 last 9.25) - open/close 14043 (time 1784193417.70 total 847.68 last 10.75) - open/close 14194 (time 1784193427.94 total 857.92 last 14.75) - open/close 14299 (time 1784193438.51 total 868.49 last 9.93) - open/close 14457 (time 1784193448.51 total 878.49 last 15.79) - open/close 14575 (time 1784193459.49 total 889.47 last 10.75) - open/close 14799 (time 1784193469.85 total 899.83 last 21.62) - open/close 14957 (time 1784193480.48 total 910.46 last 14.87) - open/close 15146 (time 1784193491.50 total 921.48 last 17.15) - open/close 15254 (time 1784193501.52 total 931.50 last 10.77) - open/close 15325 (time 1784193511.53 total 941.51 last 7.09) - open/close 15466 (time 1784193521.94 total 951.92 last 13.55) - open/close 15738 (time 1784193532.38 total 962.37 last 26.03) - open/close 15842 (time 1784193543.06 total 973.04 last 9.74) - open/close 16048 (time 1784193553.08 total 983.06 last 20.57) - open/close 16249 (time 1784193563.12 total 993.10 last 20.01) - open/close 16462 (time 1784193573.12 total 1003.11 last 21.29) - open/close 16570 (time 1784193583.50 total 1013.48 last 10.41) - open/close 16850 (time 1784193593.51 total 1023.49 last 27.99) - open/close 17612 (time 1784193603.51 total 1033.49 last 76.16) - open/close 18350 (time 1784193613.53 total 1043.51 last 73.66) - open/close 19004 (time 1784193623.53 total 1053.51 last 65.38) - open/close 19639 (time 1784193633.54 total 1063.52 last 63.45) - open/close 20000 (time 1784193638.48 total 1068.46 last 73.11) - open/close 20628 (time 1784193648.49 total 1078.47 last 62.73) - open/close 21394 (time 1784193658.50 total 1088.49 last 76.49) - open/close 22140 (time 1784193668.51 total 1098.49 last 74.60) - open/close 22774 (time 1784193678.52 total 1108.50 last 63.32) - open/close 23391 (time 1784193688.54 total 1118.52 last 61.55) - open/close 24151 (time 1784193698.55 total 1128.53 last 75.94) - open/close 24893 (time 1784193708.55 total 1138.53 last 74.17) - open/close 25630 (time 1784193718.57 total 1148.55 last 73.58) - open/close 26379 (time 1784193728.58 total 1158.56 last 74.86) - open/close 27156 (time 1784193738.59 total 1168.57 last 77.61) - open/close 27889 (time 1784193748.59 total 1178.57 last 73.29) - open/close 28572 (time 1784193758.59 total 1188.58 last 68.25) - open/close 29167 (time 1784193768.61 total 1198.59 last 59.41) - open/close 30000 (time 1784193778.59 total 1208.57 last 83.46) - open/close 30817 (time 1784193788.59 total 1218.57 last 81.69) - open/close 31778 (time 1784193798.60 total 1228.58 last 96.04) - open/close 32473 (time 1784193808.61 total 1238.59 last 69.44) - open/close 33235 (time 1784193818.61 total 1248.59 last 76.17) - open/close 34044 (time 1784193828.61 total 1258.59 last 80.90) - open/close 34775 (time 1784193838.61 total 1268.59 last 73.09) - open/close 35408 (time 1784193848.63 total 1278.61 last 63.20) - open/close 36081 (time 1784193858.63 total 1288.61 last 67.28) - open/close 36778 (time 1784193868.63 total 1298.61 last 69.70) - open/close 37480 (time 1784193878.64 total 1308.62 last 70.12) - open/close 38304 (time 1784193888.64 total 1318.63 last 82.37) - open/close 39113 (time 1784193898.66 total 1328.64 last 80.81) - open/close 39848 (time 1784193908.67 total 1338.65 last 73.38) - open/close 40623 (time 1784193918.67 total 1348.65 last 77.49) - open/close 41281 (time 1784193928.69 total 1358.67 last 65.69) - open/close 42000 (time 1784193938.70 total 1368.68 last 71.82) - open/close 42665 (time 1784193948.71 total 1378.69 last 66.45) - open/close 43449 (time 1784193958.71 total 1388.69 last 78.37) - open/close 44192 (time 1784193968.72 total 1398.70 last 74.23) - open/close 44990 (time 1784193978.73 total 1408.71 last 79.75) - open/close 45763 (time 1784193988.74 total 1418.72 last 77.19) - open/close 46543 (time 1784193998.74 total 1428.72 last 78.00) - open/close 47343 (time 1784194008.75 total 1438.73 last 79.96) - open/close 48262 (time 1784194018.75 total 1448.73 last 91.86) - open/close 49014 (time 1784194028.75 total 1458.73 last 75.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1467.23 seconds: 33.83 ops/second - unlinked 0 (time 1784194042 ; total 0 ; last 0) - unlinked 10000 (time 1784194180 ; total 138 ; last 138) - unlinked 20000 (time 1784194323 ; total 281 ; last 143) - unlinked 30000 (time 1784194465 ; total 423 ; last 142) - unlinked 40000 (time 1784194601 ; total 559 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 689 seconds: 72.033379 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg108-server unloading modules via unload_modules_local on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local oleg108-server: modules unloaded. pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2