Turkey - Agricultural Research Project

Document of The World Bank FOR OFFICIAL USE ONLY Report No: 23823 IMPLEMENTATION COMPLETION REPORT (CPL-34720; SCL-3472A; SCPD-3472S) ON A LOAN IN THE AMOUNT OF US$55 MILLION TO THE GOVERNMENT of TURKEY FOR THE TURKEY AGRICULTURAL RESEARCH PROJECT Marci 13, 2002 This documnent has a restricted distribution and may be used by recipients only in the perfornance of their L official duties. Its contents may not otherwise be disclosed without World Bank authorization.  CURRENCY EQUIVALENTS (Exchange Rate Effective March 13, 2002) Currency Unit = Turkish Lira (TL) 1,358,364 TL = US$ 1.00 US$ 1.00 = 0.00000074 TL FISCAL YEAR Jan Dec ABBREVIATIONS AND ACRONYMS AEARP Agricultural Extension and Applied IT Information Technology Research Project LAN Local Area Network AERI Agricultural Economics Research Institute LIS Library Information System AM Aide Memoire MAFRA Ministry of Agriculture, Forests and APK Research Planning and Coordination Board Rural Affairs (of MoF) MARA Ministry of Agriculture and Rural ARIP Agricultural Reform Implementation Project MIS Affairs ASAL Agricultural Sector Adjustment Loan MoF Management Information System CGIAR Consultative Group on International MTR Ministry of Forestry Agricultural Research NPV Mid Term Review CIMMYT International Maize and Wheat Improvement PIU Net Percent Value Center PSR Project Implementing Unit RGP Competitive Research Grant Program RI Project Status Report CRI Central Research Institute RM Research Institute ECA Europe and Central Asia Region RMIS Resident Mission ERR Economic Rate of Retum Regional Management Information FAO Food and Agriculture Organization RMP System AP Southeast Anatolia Project SAR Research Master Plan GDAR General Directorate for Agricultural Research SPO Staff Appraisal Report GDRS General Directorate of Rural Services TA State Planning Organization GEF Global Environmental Fund TARP Technical Assistance GOT Government of Turkey TL Turkish Agricultural Research PToject IBRD International Bank for Reconstruction and TUBITAK Turkish Lira Development Turkish Scientific and Technology IFAD International Fund for Agricultural Development VAT Research Organization ICARDA International Center for Agricultural WAN Value Added Tax Research in the Dry Areas Wider Area Network ICR Implementation Completion Report IIMI Intemational Irrigation Management Institute IPM Integrated Pest Management IRR Internal Rate of Return m Vice President: Johannes Linn Country Manager/Director: Ajay Chhibber Sector Manager/Director: Marjory-Anne Bromhead Task Team Leader/Task Manager: Cuneyt Okan  FOR OFFICIAL USE ONLY TURKEY TURKEY AGRICULTURAL RESEARCH PROJECT CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 8 5. Major Factors Affecting Implementation and Outcome 18 6. Sustainability 21 7. Bank and Borrower Performance 24 8. Lessons Learned 26 9. Partner Comments 30 10. Additional Information 33 Annex 1. Key Performance Indicators/Log Frame Matrix 34 Annex 2. Project Costs and Financing 35 Annex 3. Economic Costs and Benefits 37 Annex 4. Bank Inputs 38 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 40 Annex 6. Ratings of Bank and Borrower Performance 41 Annex 7. List of Supporting Documents 42 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.   ITXeam Leader: Cuneyt Okan ITIL Uanit: ECt3SS |/CR Tvp~e.Core ICR |Report Date: March 14, 2002 1. Project Data Name: TURKEY AGRICULTURAL RESEARCH L/C/TF Number: CPL-34720; PROJECT SCL-3472A; SCPD-3472S Couint,y:Departnient: TURKEY Region: Europe and Central Asia Region Sector/subsector: AR - Research KEY DATES Original Revised/Actual PCD: 12/08/1987 Effective: 09/16/1992 09/16/1992 Appraisal: 05/31/1991 MTR: 04/30/1995 11/30/1995 Approval: 05/21/1992 Closing: 12/31/1999 04/30/2001 Borrowt er/Imi?pleinentiig Agency: REPUBLIC OF TURKEY Other Partners: Implementing agencies: MARA (GDAR, AER1), MOF, GDRS, TUBITAK STAFF Current At Appraisal Vice President: Johannes Linn W. A. Wapenhans Ctountrv MlaInager: Ajay Chhibber Michael Wiehen Sector Manager: Marjory-Anne Bromhead Ridwan Ali Team Leader at ICR: Cuneyt Okan Henry P. Gassner ICR Primary Au4thor: Cuneyt Okan; Keith Milligan (Consultant) 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcomne: S Sustainability. UN Institutional Development Impact: M Bank Perfbrmance: S Borrower Perfbrmance: S QAG (if available) ICR Quality at Entry: Project at Risk at Any Time: Yes  3. Assessment of Development Objective and Design, and of Quality at Entry 3. 1 Original Objective: The purpose of the Turkey Agricultural Research Project (TARP) was to support the Government of Turkey's (GOTs) efforts to strengthen the research capacity of the Ministry of Agriculture, Forestry and Rural Affairs (MAFRA) to provide technologies for solving Turkish farmers' problems and raising their incomes. Specific project objectives were: I . Developing technologies suitable for the less well-developed regions of Eastern and Southeastern Turkey. 2. Strengthening selected high-priority national research programs because of their relevance beyond the less developed regions. 3. Contributing to institutional building of the agricultural and forestry research services. 4. Establishing an economics and policy research capability for agriculture. 5. Strengthening MAFRA's capacity for testing food products so as to enforce pesticide and hormone residue regulations. The largest component of TARP consisted of investments to strengthen research in the less developed regions of east and southeast Turkey. Pro,ect Designt Context Agricultural growth rates in Turkey from 1962 - 1988 were around 6 percent, but these were largely associated with increases in area of cultivation rather than increased yields. The lack of productivity increases were, among others, thought to be related to inadequate research, weak extension services and inappropriate farming systems. During the 1 980s, Government of Turkey's (GOT's) wide ranging structural reforms were designed to promote increased efficiency and growth through greater reliance on market forces and the private sector, and promote a more outward orientation of the economy. GOT's main agricultural sector development objectives were to: (i) modernize production techniques to raise productivity, yields and farmer income; (ii) maintain food supplies to meet requirements of a growing population; and (iii) promote agricultural exports. These were supported by strategies to improve agricultural productivity through strengthening research and extension services, expanding the irrigated area, improving irrigation efficiency and encouraging the use of improved seeds and crops. The Bank's assistance strategy for Turkey was to support institutional and policy reform, improve extension service delivery, establish a sectoral planning capacity and rationalize the public sector investment program. This strategy was implemented through adjustment lending and a number of technical projects designed to increase production (livestock, horticulture and rural development), strengthen access to credit, improve irrigation and drainage infrastructure and strengthen agriculture extension and applied research. Turkey had a well established (State) agricultural research infrastructure that was centrally managed and carried out a mix of public and private good activities. University based research was negligible but had started to emerge. The output of the State system was minimal and it was foreseen to finance and in depth study of agricultural research under the Agricultural Sector Adjustment Loan (2585-TU. 1985). This - 2 -  study only materialized, after considerable debate, into a much smaller effort completed in 1989. This resulting report recommended further reorganization of the research system, including establishment of a General Directorate for Agricultural Research (GDAR) within MAFRA. It proposed the reorganization and consolidation of research through establishment of 11 regional research institutes and downgrading the remaining institutes to substations. GDAR was to report directly to the Minister of Agriculture, but with autonomy in staff recruitment/management and in research planning. The review suggested that strengthening MAFRA's agricultural research system could be supported by the Bank through a separate Agricultural Research Project. The project was to improve research planning, improve infrastructure and demonstrate different methods of research funding. The World Bank and International Fund for Agricultural Development (IFAD) had already started to support extension and applied research through the A ricultural Extension and Applied Research Project (AEARP-2405-TU; 1984-1993) which, rather than a nationwide extension improvement effort, implemented a phased approach based on agro-ecological zones. It was was subsequently followed by a Second Agricultural Extension and Applied Research Project (AEARP 11-3177-TIJ; 1990-1998) that continued this effort. Although AEARP I benefitted from multi-disciplinary research, AEARP II noted the declining capacity in research, largely due to aging staff, and a need for better extension-research linkages. Both projects were marginally satisfactory, largely related to limited implementation capacity, lack of incentives for researchers and extension staff, and high costs of extension delivery that was insufficiently demand-driven. Project Preparationi was undertaken by the Food and Agriculture Organjization (FAO) in August 1990, mairtly through consulting and collecting information from the Turkish researchers. The preparation team reviewed the recommendations of the Agriculture Sector Adjustment Loan (ASAL) but considered the proposed system reform not feasible as it would have required new legislation, for which political support was lacking at that time. The Bank team then developed the original design further with the Ministry and research institutes. Although the project was to emphasize demand-driven and prioritized research, the process of developing master plans was expected to take time, and in order to keep the research system going, the design team, in collaboration with institute directors, also designed a research program for the early years of the project. This design led to the Staff Appraisal Report (SAR) of April 1992. The project became effective on September 16, 1992. 3.2 Revised Obecrtive: Re.strn cturing Desig,Yn Con text The start up was slow. During a three-year period, significant changes in the Ministry of Agriculture, Forests and Rural Affairs (MAFRA) occurred. MAFRA was split into the Ministry of Agriculture and Rural Affairs (MARA) and Ministry of Forestry (MoF). The General Directorate of Agricultural Research (GDAR) was established within MARA. An independent General Directorate for Rural Services (GDRS) responsible for soil conservation, small-scale irrigation drainage was established. There were also increasing macro-economic and budget problems leading to austerity measures that were applied in 1994. These changes diverted attention from implementation and complicated procurement. Furthermore, initial project resources were focused on physical investment rather than priority research outputs, leading to input rather than output related research. Indeed, the lead-time required to improve institutional ownership - 3 -  of the national programs selected under the project proved longer than expected. Little progress was made with farm systems research and progress in establishment of an Agricultural Economics Research Institute (AERI) was nil. Disbursement lagged, and progress and organization of the project were deemed unsatisfactory. A new task team had doubts about the design, and put greater emphasis on completion of Research Master Plans before disbursing funds for research. A revision of project objectives was proposed during the Mid-Term Review (MTR) in December 1995. The MTR undertaken in 1996, noted various factors that contributed to slow implementation, including the austerity measures of Government, the lack of coordination between the three implementing agencies, the lack of comprehensive research plans at the onset of the project, and weak managerial capacity in the agencies. The MTR recommended placing more emphasis on participatory research planning, on development of a research environment encouraging private sector participation in research, on collaboration between various agencies and institutions involved in agricultural research, and on using competitive research grants to invest in high priority research. Each agency was made fully responsible for the activities under its purview with no single project coordination unit. Other revisions were an amendment of the Technical Assistance and Training contracts, and redesigned monitoring and reporting assignments. Project restructuring was approved by Management and Government as reflected in the amended loan agreement (approved in May 1997). In view of tight budgetary conditions prevailing in the country, the civil works component was substantially reduced and local budgetary allocation to project agencies was increased. The revised project objectives were to: 1. Improve the capacity to deliver high quality, relevant research programs of Ministry of Agriculture and Rural Affairs (MARA); General Directorate of Rural Services (GDRS); and Ministry of Forestry (MoF). 2. Strengthen selected high-priority national programs focusing on less developed regions. 3. Institutionalize a competitive agricultural research program that invests in high priority relevant research. 4. Increase the efficiency and effectiveness of resource use and develop a research environment that encourages private sector support for research and collaboration between agricultural research agencies. 5. Establish an economics and policy research capability for the agricultural sector. The original components of farming systems, crops research, livestock research and integrated pest management were amalgamated into institutional strengthening and a new componeuit for training and applied research. The objective of strengthening MARA's capacity for testing food products, intended to assist in the enforcement of pesticide and hormone residue regulation, was removed in August 1996; during MTR it was concluded that these services would be more appropriately provided by the private sector. 3.3 Original Comnponents: The original components were: 1. Institutional Strengthening (US$ 10.6 million) (a) strengthening research planning, program management and research management information systems; and -4 -  (b) establishing an Agricultural Economics Research Institute (AERI). In addition, a research grant facility was to be established to provide merit awards to high performing research workers. A major contribution to institutional strengthening was to derive from a "training abroad" program with an expected 30 PhD and 110 MS trainees, and over 1000 man-months of short term training. 2. Farming Systems Research (US$ 1.5 million) This new national program would support pilot research programs at three Research Institutes (Rls) in the lesser developed regions i.e. in Eastem and Southeastem regions. 3. Crops (US$ 22.6 million) and Livestock (US$ 9.3 million) The Ministry initially selected 197 priority areas in Crop Research with 154 research projects, implemented at 13 RIs, including cereals, food legumes, cotton, second crops for irrigated areas, fruit, pistachios, grapes, vegetables, and genetic resources; and in Livestock Research with 43 research projects, implemented at 6 livestock and 6 animal health RIs, including large ruminants, small ruminants, fodder and pasture, apiculture, sericulture, and inland fisheries. Many of the selected research topics were a continuation of previous programs where the project was expected to finance a time slice. 4. Soil and Water (US$ 12.1 million) The national program in Erosion Control and Soil C'onservation Research was to be undertaken at 10 RIs including hydrological modeling for small agricultural watersheds; studying strip cropping and stabilization methods in soil and water conservation; the effects of cultivation systems on yields; erosion and moisture conservation. The programs in irrigation/drainage for less developed areas were consolidated with 3 research projects implemented at 2 RIs in Eastem Anatolia, including water requirements of crops, reclamation criteria for saline and boron-affected soils, and water saving irrigation methods adapted to crops. 6. Forestry (US$ 5.6 million) Four research programs in Forestry Research were to be implemented at 2 RIs, and included silviculture, forest management and forest protection; major and minor forest products, community forestry, erosion control, and soil conservation. 7. Integrated Pest Management (US$ 4.6 million) This was a national program to be undertaken by the 8 Plant Protection Institutes. The proposed research included bio-technical methods of pest control, biological control of citrus pests, mass production of natural predators, and Integrated Pest Management (IPM) in crops with high usage of pesticides. 8. Residte Control Research (US$ 2.3 million) The residue control program was a national program to be implemented at 4 Plant Protection Institutes to provide laboratory diagnostic services for monitoring residues in pesticides, fertilizers and growth regulators in crops, heavy metals, animal remedy residues in livestock and residues in food. To respond to the objective of priority for less-developed areas, the components in crops, livestock, irrigation and forestry were to focus on research that was relevant to these areas. 3.4 Revised Comzponenits: In line with the new objectives, the revised components were: - 5 -  1. Agricultural Policy Establishment of the AERI including office equipment, capacity building and delivery of economic and policy research projects. 2. Applied Research Institutionalization and funding of Competitive Research Grants Programs (CRGP) for the high priority research programs of GDAR, GDRS, MoF and Turkish Scientific and Technology Research Organization (TUBITAK). 3. Institutional Strengthening Including improved research delivery capacity building through training and access to international and other national research knowledge information systems, research management capacity building through improved planning, prioritization, and support to research delivery systems and monitoring and evaluation for GDAR, MoF and GDRS. The component also included development and management of a Regional Management Information System (RMIS) incorporating a Library Information System (LIS), and a research project database. Each implementing agency would have a fund that would finance competitive research based on priorities defined in research master plans. The agencies would give highest priority to research proposals with client support, followed by collaborative research proposals, and proposals from individual institutions. The proposed financing was to be 100% in 1996, 80% in 1997, 60% in 1998 and 30% in 1999. The revision included canceling some civil works and vehicle purchases from the Project leading to US$6 million savings (cancelled on May 7, 1997). 'rhe remaining US$ 49 million of pooled currency loan was fixed at US$ 46.85 million, during the July 1997 single-currency conversion process. Closing Date Extension. A GOI' request to extend the closing date by one year to December 31, 2000 was approved by the Bank in October 1999. At that time, budget allocations to the five implementing agencies were revised in line with Supervision Mission recommendations. The AERI component was further extended in December 2000 to April 30, 2001, primarily to assist GOT in the preparation the Agricultural Reform Implementation Project (ARIP) through studies and small research projects. 3.5 Quality at Entry: Original Objectives and Components Quality at entry for the original objectives and components is rated as unsatisfactory in terms of project design and risk assessment for the following reasons: Project Design There were four difficulties with the original project design: I . Short versus long term implementation. The original project design on the one hand promoted development of research master plans and prioritization of research programs and management, on the other it prescribed research programs and targeted certain institutions. This contributed to some confusion on how to proceed with implementation. 2. Limited ownership. The short term research plan was developed by the design team and local institute directors. This, and the fact that the Staff Appraisal Report (SAR) was initially only available in - 6 -  English, limited awareness and ownership by researchers. Ministry reorganization at the start of the project further contributed to lack of ownership. Concepts, such as prioritization and a shift to demand-driven research were only slowly understood and accepted. The establishment of an AERI, independent of MARA, providing impartial policy advice with a portfolio of nongovernmental clients, did not find full ownership within MARA. 3. Reforms. The design did not really address two related and fundamental problems. (a) The system, with mainly commodity-based, research institutes and 1200 programs was unwieldy and not efficient. Drastic consolidation proposed in the ASAL-funded review, however, was not attempted. It would have required legislative action, and the political support for such action was lacking. (b) The institutional strengthening and applied research component (via the competitive grants) did not provide solutions for sustainable funding. The Preparation Mission had already noted that research operational funding was less than 14% of research budget allocations, but no attempt was made to further analyze this problem and develop sustainable solutions. The investment in research planning did not directly encompass funding reform. 4. Priority Research Programs for Lesser Developed Areas The intent of the project objective of focusing on less developed areas was laudable, but implementation proved difficult. There were two difficulties: (a) An unclear definition of less developed areas. It was not clearly defined whether the definition was based on geography; often it was understood that it was targeting Eastern Anatolia. However, less developed areas are dispersed throughout Turkey, and are often more related to topography than geography. (b) The Second AERI included support for research in less developed regions. TARP did not sufficiently incorporate lessons from the AERI projects, which faced many implementation difficulties. Risk Assessment The risk analysis undertaken in the SAR was cursory. The risk of inadequate commitment was recognized but its importance and possible mitigation were not well addressed. Other risks were not mentioned, for example, the Govermment salary and promotion system that did not really provide incentives for good research. It should be noted that in 1991 risk assessment was not commonly a major part of project appraisal. Revised Objectives and Components The revised objectives and components were more realistic and are rated as satisfactory. These project revisions increased the ownership and commitment and supported improved effectiveness of research through new research funding mechanisms. They also helped capacity building. 1. The establishment of five separate implementation agencies and project components (GDAR, GDRS, MoF, AERI and TUBITAK) did increase the complexity and reduced collaboration, but was the only - 7-  practical solution. Separate budget allocations to each of the five implementing agencies, were agreed upon. 2. The revised objectives increased emphasis on research management. However, the objective of focus on less developed areas was diluted; the agenda was now more driven by the researchers and larger political issues such as support for less developed areas were not high on their priority list. 3. The shift to competitive research improved links between research planning and field research. However, the somewhat fragmented implementation complicated the broad overall measurement of the broader project impact. This issue was neither addressed adequately in the original - nor in the restructured - design. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: The overall outcome of the project, as revised following the MTR, is rated satisfactory. Although the monitoring and evaluation was inadequate, three of the four key indicators were largely achieved i.e.: (i) an AERI was established; (ii) management information systems were installed and used to monitor/manage the research system; (iii) the research master plans became indeed the basis of research programs and later of budget requests by MARA (although not adequately funded by the State Planning Organization (SPO). The fourth indicator, farmers adopting new technologies releases by the RIs, cannot yet be fully assessed. The project: I. Contributed to a substantial improvement in technical skills as well as management of research. 2. Contributed to the modemization of the agricultural research system in terms of equipment and technologies used during project implementation. 3. Successfully shifted the emphasis from supply driven to demand driven research. 4. Increased the attention to skills in agricultural economic research, and helped Turkish policy makers to rethink the State's role in supporting agriculture. 5. Helped to build valuable linkages to intemational research and established a basis for further integration in the European agricultural research system and markets. 6. Was not fully successful in consolidating the Turkish agricultural research system into a more efficient system based on multidisciplinary institutes. 7. Was considered positively by the agricultural research community in Turkey. 8. Did not adequately support improved research delivery in less developed regions. 4.1.1. Outconme by objective: Objective 1: Improve the capacity to deliver high quality, relevant research programs of Ministry of Agriculture and Rural Affairs (MARA), General Directorate of Rural Services (GDRS), and Ministrv of Forestry (MoF). The improvement of overall research capacity was satisfactory. Success was achieved in (i) short and long-term training for individual staff; (ii) improvement in access to international research databases, particularly in GDAR and to a lesser extent in GDRS and MoF; (iii) improved English language skills; (iv) improved links with international agencies such as International Maize and Wheat Improvement Center (CIMMYT) and International Center for Agricultural Research in the Dry Areas (ICARDA), the International Irrigation Management Institute (IIMI) Project as well as with research institution and universities in various countries; (v) collaborative research projects with Turkish Universities; (vi) - 8 -  opportunities to contribute to international scientific meetings; and (vii) AERI's collaboration with national and international experts for economic and policy research project delivery. The overall outcome of upgrading the equipment, and especially the technologies and skills to late 20th century standard is difficult to quantify, but has and will have a great impact on agricultural development in the country. Information Management. The investment in information technology and libraries greatly increased the access of researchers to relevant information, thereby lessening duplication of research and improving the design of research proposals. Research outcomes of GDAR include 133 newly registered varieties of high yielding and export oriented field and horticultural crops as well as agronomic packages suitable for different ecological regions. The project also released IPM packages, mainly for horticultural crops. An early warning system for plant protection was further improved through research under the project and is now in use, there by reducing farmers risk as well as the use of pesticides. Various crop rotation systems for rainfed and dryland conditions were tested and recommended to farmers. Improved diagnostic and vaccine-production technology assist producers in reducing the risk of animals disease. new varieties and cultivation of fodder crops have increased use of preserved fodder. Cooperation with the GDRS capacity in GIS is leading to better planning in the use of Eastern Anatolia rangelands. The project also supported development of new methods of (biological) crop protection, use of water saving irrigation technology, better diagnostics of animal diseases, and application of remote sensing in planning and design of rural development projects. Research outcomes of GDRS Studies on soil fertility and micro-nutrient levels of Turkish soils were completed and provide farmers the needed information to fine tune their fertilizer use. Research on the use of water saving irrigation technology and erosion control has led to increased interest of farmers in such technologies to be sold through private suppliers. The improved GIS capability developed under the project has led to greater demand for the application of remote sensing in planning and design of rural development projects. Moreover, the demand for GIS is broadening to applications of the urban sector (for town planning). Research outcomes of MoF New genetics in forestry, especially the introduction of fast growing plantation trees such as Pinus brutia, poplar and Eucalyptus, and new multiplication and planting technologies have attracted the interest of the industry which started to use the research capacity for contract research in plantation forestry. Poplar timber production has doubled since the late eighties to 5 million m. Significant knowledge and experience was developed regarding the determination of the botanical composition, carrying capacity and suitable rehabilitation methods for Central and Eastern Anatolia rangelands by using a GIS center developed under the project. GIS technology was also used to assist in site characterization and landuse planning for forest growing species. The research in aromatic and medicinal plants has led to application now used by producers and exporters. The research in participatory forest management is being applied in various natural resources- and watershed management investments. Objective 2: Strengthen selected high-priority national programs focusing on less developed regions. Implementation progress is described as less than satisfactory in successive Project Status Reports (PSRs). During the first phase there were social and security problems in the southeastern Turkey, which was targeted as a less developed region Staff retention was a problem primarily due to low living standards in these areas and the lack of incentives to attract quality staff. Nevertheless, TUBITAK targeted 25 percent 9..  of its Competitive Research Grant Program (CRGP) funds for research for the South East Region. MoF revised its RMPs to emphasize sustainable forest management and income generation for forest-dependant villagers from non-forest products, all of which are applicable in the less developed regions. Much of GDRS's soil and water research including basin management, erosion modeling, soil and hydrology mapping were also relevant to drier and less developed regions. In hindsight, research based on geographical targeting with the aim of reducing poverty may have been more complex than envisaged. Objective 3: Institutionalize a competitive agricultural research grant program that invests in high priority relevant research. The project was among the first to introduce a Competitive Research Grant Program in the Europe and Central Asia (ECA) region. (The concept was not completely new in Turkey as it had been applied by TUBITAK in its funding of research). 1. The Research Master Plan approach took some time to be understood, but was fully accepted by the technical ministries toward the end of the project when funding of new as well as some on-going projects was fully based on priorities of the RMPs. 2. The concept of competition was introduced. Turkey is a large country with 71 Universities having 21 Agricultural, Animal Health and 5 Forestry faculties. There are 56 GDARs, 11 MoFs and 10 GDRS research institutes funded and managed by government. About 2,000 agricultural researchers (excluding universities) provided a good base for competition. The 2,432 sub-projects that were supported represent over 6,500 proposals submitted; the success rate for proposal submission was about 40 percent. 3. The combination of RMPs and competitive research worked well. Annual meetings to review candidate projects were held at GDAR, GDRS and MoF, primarily to assess project proposals' match with the priorities set in the RMPs. 4. The competitive research system was not the only method in Government-coordinated financing of research. Research financing continued from the central budget. The Government also continued to pay the salaries of the researchers, and provided some basic overhead support to the institutes. The Government did not, however, incrementally increase its funding share for the competitive research during project implementation as planned in the MTR. Competitive grants were not included in the national budget after project completion, leading to questions about sustainability. This may be in part be related to the unusual constraints on SPO and Treasury in 2001, but may also be linked to a mid-term design flaw that appeared to assume that by the end of TARP most research would be funded competitively. Objective 4: Increase the efficiency and effectiveness of resource use and develop a research environment that encourages private sector investment in research and collaboration between agricultural research agencies. 1. The project emphasized improved planning through Research Masters Plans (RMPs). After 1998, the RMPs were modified by using a participatory process that included input from universities, agribusiness, farmers, extension and research staff representatives. 2. Although less than expected at the onset of the project, research did shift towards addressing farming systems and away from single commodities. 3. The proposed concentration of GDAR research in seven Central Research Institutes (CRIs) largely failed, in part because it contradicted the competitive research concept where competition rather than direction guides the flow of funds. The same applies to the concept of "Centers of Excellence" floated during the 1998 supervision. - 10 -  4. While there has been a significant change in the research portfolio, the future impact with respect to efficiency in research is questionable. The number of institutes did barely decrease and most funding went to individuals or individual institutes. Collaboration between institutes, or integrated approaches to address research was limited. 5. The transfer of research outputs to farmers requires a well functioning extension system. In Turkey the extension quality, however, has been variable largely due to budget constraints. In some cases the institutes incorporated extension into the research program; the outcome of TlJBITAK funded research on IPM, for example, included newsletters and leaflets for distribution to farmers. 6. GDAR, GDRS and MoF developed performance indicators relating to the goals defined in their respective RMPs. Each of these agencies has, in their individual agencies Implementation Completion Reports (ICRs), reported achievements against performnance indicators. Overall assessment of achievement against performance indicators is generally satisfactory (Annex 1). 7. In GDAR, the perception of bias in project proposal evaluation and the non-operational status of the Higher Council for Research slowed the rate of change. There was excessive focus on a large number of small projects. Objective 5: Establish an economics and policy research capability for agriculturejSee 4.2.(i) Despite the long delay in its establishment, the AERI has rapidly found its niche in agricultural policy development, a topic that was not well developed in Turkey before 1990. Its publications on crop outlook, as well as broader policy recommendations, are nationally and internationally respected, albeit not always accepted by Government. AERI's use of local consultants has also enhanced the national attention and recognition of agricultural policy research, and impacted Government decisions on agricultural policy. 4.1.2 Implementation Process 1. Project management. The project was implementecd without a formal implementation unit. The project steering committee met once or twice early on in the project but did not play a further role. 2. After restructuring, MoF established the Research and Survey (technical) Division within a new Research and Development Department. GDRS established a Soil and Water Resources Research Section. These small project management units consisting mainly of technical staff, with financial and administrative support, operated their own planning, budgeting and procurement and reported to their respective Planning Departments (APKs). The TUBITAK component was implemented by the Executive Secretariat of that organization and reported to the Agriculture, Forestry and Food Technologies Research Grant Committee of TUBITAK. These teams stayed in place for the last 4 years of the project. 3. While the system worked well, it missed central coordination between the agencies and overall oversight and, to some extend, Ministry ownership. By default the coordination was mainly carried out by the World Bank task manager, resident in Ankara. 4.2 Outputs by) components: 1. Agricultural Policy - Rating is satisfactory. The outputs of this component include: The Agricultural Economic Research Institute (AERI) was established on May 17, 1996. Its value has been recognized through demand for its publications, both from the public and private sector. AERI outputs include 43 research projects contracted at a cost of US$ 1,776,270. The average cost of U$ 41,000 per project included contracted specialist international and national expertise. These projects resulted in research publications including:  * (annual) Situation and Outlook Reports and updates: for wheat (4), cotton (4), feed grains (3), red meat and meat production, dairy and milk products, cotton and oilseeds and vegetable oils; * Contributions to, or publishing, 7 books; * 10 working papers and monographs, 14 research papers presented at international and national conferences; and 23 articles published in a range of national publications. AERI has made contributions to the preparation, and future monitoring, of the US$ 600 million Agricultural Reform Implementation Project (ARIP) approved by the GOT and IBRD in July 2001, a new project that introduced major changes in Turkey's agricultural policy. 2. Applied Research - Rated as satisfactory. The outputs include: (a) A total of 2,432 research projects, proposed by researchers, have been funded and are to be completed by the end of 2001. (See Table for detail) Table 5.1 ber ofp funded Agency # Projects Cost US$ $/Project GDAR 1,400' 4,802,085 3,430 GDRS 600 2,900,000 4,833 MoF1 67 1,224,847 18,281 TUBITAK 365 3,900,000 10,6853 AERI 42 1,025,750 24,500 Total 2,432 12,826,932 5,275 Estimated - 1900 projects approved but some are multi-year funded projects. 2 1996 - 2000 only. All projects are multi-year funded, recognizing the longer-term nature of forestry research and several large national projects account for high cost per project. 3 High relative costs include financing 11 larger multi disciplinary, multi institute projects during the last two selection rounds. Close to 450 GDAR research projects were still on-going at the completion of TARP. Given their long duration, 30 percent of MoF projects will continue beyond 2003 subject to availability of GOT funds. Almost 50 percent of TUBITAK funded projects have been completed. All remaining co-funded TUBITAK projects will be completed by the end of 2002. - 12 -  Review of TUBITAK's Competitive Research Funded by TARP (1997-1999) The material reviewed was a set of abstracts of the research proposals selected in the last round (1999) of the TUBITAK sponsored research. The abstracts were pre-formatted and well organized. In addition a number of institutes were visited, in particular the Ege University that is a large beneficiary of the TLBITAK grants. A total of 290 research grants were awarded to 41 institutions, out of a total of 679 applications. The major recipient institutions were (Cukurova University in Adana (48), Ankara University (41), Ege University in Izmir (29), Uludag University in Bursa, Adeniz University in Antalya and Harran University in Urfa and A. Menderes University in Aydin each with 13. The total funding for these seven institutes was 62% of the total. This seems to indicate that the majority of grants were awarded to older established institutions. With respect to one of the TARP project objectives that was to direct some of the research to underdeveloped regions of the country, the Black Sea and Eastern Anatolian institutions received only 12.5 percent of total funds (although some of the research of other institutions was carried out in, or was relevant to, Eastern Anatolian conditions). Overall, however, it seems that the project objective of improving understanding about less developed areas, and target research to these underdeveloped regions has not been achieved. Rejected proposals were not reviewed. Interviews with a limited number of researchers, that submitted proposals but did not obtain research funding from TUBITAK, indicated concems about the even-handedness of the review committees and council. (b) Progress and final reports for all research carried out at all agencies were requirements for each CRGP project. Progress reports have been completed for the majority of research projects funded under TARP. The Research Management Information System (RMIS) of GDAR now has the capacity to electronically generate research abstracts as part of the research progress and final reporting process. However, there has been no data entry since 1997 reports. The reasons for this are a combination of disinterest at RI and researcher level to provide input, a lack of Directorate support for the RMIS, and an RMIS that is probably overdesigned for the Information Technology (IT) skills base in GDAR and the RIs. The reporting in GDRS is up-to-date. (c) Publications (1993 - 2000). The output of research publications in national and especially in intemational journals is disappointing but may be partly due to a time lapse of at least one year between completion and publishing. Still, it appears that the scientific discipline of publishing is not a strong part of the Turkish research culture. This may be a consequence of conditions such as lack of recognition of the importance or publications as a measure of research output, poor linkages between publication record and salary or rank, or difficulties in meeting editorial standards of international journals. TUBITAK's higher publication record is probably due to career incentives offered in academia. - 13 -  Table 5.2 Publications by Beneficiary Institutes 1993-2000 Research Books Journals Conference Technical Symposia Abstracts/ (Int'l.) Papers Bulletins Organized Completion (international) and Hand Reports books GDAR 6072/1400 19 623 (191) 266 (79) GDRS' 276/600 - 62 (14) MoF ' 108/67 - 7 54 2 AERI 59 7 24 15 26 TUB1TAK 105/365 10 154 _ _12 2 1 1996 -2000 only 2 Estimated. Abstracts of completed projects (450) published in Annual Reports generated through RMIS for 1996 & 1997 only. 'GDRS output included topographical maps, and other digitized geographical images. (d) Interaction with extension. The frequency and quality in the interaction with extension was highly variable, both on the output side, as well as when obtaining field information on research priorities. Some research institutes, as well as TLBITAK, prepared extension bulletins that were distributed either directly or through extension channels, as was information about new seeds etc. This was, however, done on a rather ad-hoc basis and not really formalized. 3. Institutional Development--rated as satisfactory. (a) Research Management Capacity. The outputs include: * re-establishment of the General Directorate of Agricultural Research (GDAR) in the Ministry of Agriculture and Rural Affairs (MARA); * establishment of a research department in the Ministry of Forestry (MoF); * reports on the strategies for the reform of 6 major research institutes of GDAR (see Annex 4); * research Master Plans; separate plans for MoF, GDRS and GDAR, prepared and implemented in 1999. * implementation of the competitive grants system. (b) Technical Capacity Rated as satisfactory. TARP outputs include: * improved technical skills (see under training); * improved technical and analytical capability in selected institution (see under equipment) * improved language skills and access to international research results and data. (c) Training. The output of this component includes: 0 93.3 person years of graduate training of young researchers; * nearly 340 person years of English; * 86 person years of short term training and study tours. Table 5.3 Summary of training (person years) undertaken and supported by TARP. |Agency I PhD I MSc I Short-term I Technical I English I Total I - 14 -  & Study Symposia or Language I Tours Meetings GDAR 22.5 65.8 37.0 2.4 222.5 350.2 GDRS 0 4.0 41.5 3.0 92.5 141.0 AERI 0 0 0 1.7 4 5.7 MoF 0 0 7.7 0.6 23.3 31.6 Total 22.5 69.8 86.2 7.7 342.3 528.5 The training output included a total of 6 person years of support for participation of Turkish researchers in national and international scientific meetings and seminar which apart from the training aspect, provided access to international peers and research. (d) Equipment. The TARP financed major equipment purchases for the national research system such as: * complete mohair and wool testing laboratory (National Livestock Institute in Ankara) for livestock breeding studies; * equipment for a map digitizing center (GDRS Headquarters in Ankara) to develop the capacity to support GDRS mainline of work to develop infrastructure and small irrigation schemes for the rural areas; * Atomic Absorption Spectrophotometers (for GDAR and GDRS RIs) to provide reliable testing for chemicals and chemical residues as well as for soil mineral and contaminants; * seeding drills and combine-harvesters for experimental plots (for 6 GDAR RIs) for crop breeding and agronomy research; * flour mill and malting (test) equipment (GDAR RIs) for crop breeding/ grain quality research and testing; * meteorology stations and soil humidity testing equipment (GDRS RIs) for soil and water research; * management - and Library Information Systems (GDAR, GDRS, and MoF Research Headquarters in Ankara); * complete sets of PCs, servers, printers, and software to support Local Area Network (LAN) and Wider Area Network (WAN) requirements of all RIs ( GDAR, GDRS, MoF and AERI). 4. Access to International Research. TARP's physical outputs include: * an automated library service in GDAR and MoF, including an online and on-request access to an Library Information System (LIS) and Management Information System (MIS) system (e.g. several CD-ROMs) and over 1300 library acquisitions; * expansion of the libraries of GDRS and MoF with additional specialized CD-ROMS, available on an institutional basis; * installation of WANs and LANs linking all GDAR, GDRS and MoF RIs to enable widespread access to information; and * Internet connections to all institutes, which facilitate access to information/databases contained in other national and international research organizations. All implementing agencies acknowledge that improved access to international research databases is an - 15 -  important project outcome. Researcher use of the information system is reported to be steadily increasing and seen by many research workers as a cost effective capacity building tool. 4.3 Net Present Value/Economic rate of return: The SAR did not calculate a project Net Percentage Value (NPV) or Economic Rate of Return (ERR) and therefore no NPV or ERR is calculated for TARP. The only available estimates of internal rates of return (IRR) to research in Turkey are those calculated in the AERI publication of Furtan et al. (1999). While this does not provide an assessment for TARP, it does provide evidence for the IRR for agricultural research in general. The long-term IRR estimates for two of the most important crops in Turkey was 67 percent for wheat research and 143 percent for cotton. Even after yield increases were decreased by 50 percent, taking into account of poor information flow to farmers, the IRRs only fell to 56 percent for wheat, and 126 percent for cotton. 4.4 Financial rate of retun7: The SAR did not calculate financial rates of return. None of the irnplementing agencies, apart from AERI, as noted above, have conducted ex-post or ex-ante analyses of research impact. This would appear to be an area where collaboration between AERI and other implementing agencies would pay dividends in promoting the benefits of investment in research. 4.5 Institutional development impact: See also under outcome. 1. The first important TARP achievement was the development of a new research "culture", focusing on priority research programs and research outputs and outcomes. 2. The second achievement was the increased focus on (agricultural) economics whether by AERI, or indirectly through AERI and its participation in various research planning activities. While this still needs improvement, the project helped in the important first step. 3. Thirdly, the investment in equipment, in skills and in language training helped the institutions to move out of its isolation, join the international research community, increasingly be able to participate in European agricultural research, and bring back new technologies that were either tested under local condition or could be applied immediately. 4. Finally, the improved research capacity, increased skills and changing mentality enabled selected institutions to tap into other sources of funding, including private sector support. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: The project was implemented in a problematic environment where substantial reforms were being implemented by Govenmment. Among the broader factors which influenced performance are: 1. New universities were established during the period that the project was implemented. Employment opportunities for researchers outside government agencies, that were limited at the time of the SAR preparation, expanded significantly. This increased demand for staff outside the government caused somewhat of a brain drain from the research institutes. 2. The presence (CIMMYT) and collaboration of (IRRI) international centers facilitated modernization of - 16 -  research programs in respectively, wheat selection and irrigation. 3. The lack of social incentives (salary, housing, etc.) to work in some parts of Turkey, particularly in the eastern regions impeded implementation that would target the lesser developed areas. 4. The devaluation of the Turkish Lira from 5,700 per US$ in 1992 to 526,460 per US$ in December 1999, to over 1,000,000 per US$ at project closing had considerable effect on the project, especially on counterpart funding which, when expressed in US$, was far less than estimated at appraisal. Apart from enforcement of austerity budgets, it often forced Government to make mid-year budget adjustments which further increased budgetary instability at the implementing agencies. 5. The devastating earthquake in August 1999, had a major impact on the MoF RIs in the provinces of Izmit and Bolu, subsequently a special case for the use of MoF TARP funds (US$ 1 .06m) for rehabilitation. 6. The occasionally ambivalent actions of the Bank task teams that, while trying to improve implementation, may have given different and somewhat confusing messages to the implementing agencies and contributed to some of the delays. 5.2 Factors generally subject to government control: Factors generally subject to government control include: 1. Implementation. (a) Major structural changes within government and consequently MAFRA had profound effects particularly during first three years and slowed implementation. (b) Austerity measures adversely affected the flow of funds. Government financing was unable to cover duties and taxes. Consequently, some equipment was not purchased and training was not implemented as planned. There were no problems that related to government's ability to pay staff salaries. (c) Limited procurement experience in the early phases of the project delayed implementation as did the RI managers' slow adjustment to the rules of the Bank's planning and competitive tendering system Procurement training was provided to all implementing agencies, but staff changes reduced effectiveness. 2. Research and Research Management Capacity was negatively affected by the difficulties to retain staff. because of low salaries, and lack of a merit-based promotion system. 3. The TARP did not address the oversized research infrastructure and overcapitalization in research facilities, evident at the time of the SAR. There appears to be little political will in government or the implementing agencies to rationalize research facilities. A high proportion of available GOT funds continue to be used for RI overheads costs. 4. Insufficient research funding and funding mechanisms by Government. The funding to research institutes, provided through the central budget, was very low. During project implementation, this was compensated by TARP funds where CRGPs filled the gap in operational funding. However, the SPO has maintained a policy of allocating funds directly to RIs in line with the requests of the related ministries and/or general directorates. 5. Weak link to farmers and extension. There are substantial variations in performance of research and extension services among provinces. In Izmir for example, where research management was stable, meager funds were used effectively and with the help of an enthusiastic staff the links are strong. In - 17 -  other regions the link between the research and extension was weaker. An extension infrastructure has been established during two Bank-funded extension projects. The effectiveness, however, was negatively affected by some of the same issues that affected the Turkey Agriculture Research Project (TARP) i.e.: (i) staff hiring moratorium starting in the early '90's; (ii) low salaries; (iii) low operational budgets caused by the constraints in public finance; (iv) frequent staff turnover in the provinces; and (v) variable competence of extension managers in the field. 5.3 Factors generally subject to irnpleinenting agencv control: I . The lack of full beneficiary participation at project preparation delayed project implementation. Even a year after the start of the project, there was insufficient awareness about the project and its objectives among the various research institutes' directors and staff. 2. Although the TA for master plan development worked well, consultancies for the training component were not well planned and training plans were beyond the absorptive capacity of implementing agencies. Towards project completion, implementing agencies had improved and they were better able to manage their training programs. 3. The lack of a central implementing agency (or project coordination unit) led to poor recognition of the weak linkages between research and extension. These linkages (e.g. monthly and bimonthly workshops, diagnostic surveys, etc.) had been developed by the first and second Agricultural Extension and Applied Research Projects (AEARPs). Unfortunately, experience from these projects was not linked to TARP. This linkage is now being revitalized. 5.4 Costs and financing: The final project cost is estimated at US$ 59.5 million (about US$ 45.2 million from the Loan) compared with the SAR estimate of US$ 77.6 million. Project revision in May 1997 resulted in a US$6 million Loan cancellation. The remaining funds from the Loan were fixed following the July 1997 single currency, conversion process. The unused portion of US$ 4.2 million remaining from the Loan is-cancelled. Possible problems due to lack of counterpart funding related to austerity measures had been recognized before the Loan was signed. In order to avoid major disbursement problems early in the project, Loan funding levels of all Categories to 100 percent was accepted at Loan signing, with the exception of Civil Works and Incremental Operating Costs. Throughout the project, regulations regarding the liability of the agencies to pay Value Added Tax (VAT) and customs duty changed. While exempt until 1999, they became liable both for 1999 and 2000. As this had not been calculated for, it resulted in counterpart funds shortages slowing disbursement, as most of the laboratory equipment was imported. 6. Sustainability 6.1 Rationale for sustainability rating: Sustainability is rated as questionable. The main reasons for this assessment are: (i) lower levels of after-project GOT funding for research operations ; (ii) inability to retain income generated for maintenance of staff and activities, particularly at AERI; and, (iii) continued erosion of staff numbers and technical capacity. The following is an assessment by objective: 6.1.1 Improve the capacith to deliver high qualitn, relevant research proerams of Ministry of Agriculture and Rural Atfairs (MARA). General Dir-ectorate of Ryral Services (GDRS). and Ministry of Forestry (MoF). The use of research equipment procured under TARP, and the manpower training provided, will assist sustainability of research output, at least in the short to medium term, provided staffing capacity is - 18-  retained. The improvement in language skills and computer literacy resulting in subsequent intemational contact was paramount in modemizing research. Even, if staff can not be retained by Government institutions, their work in academia or private sector will contribute to the delivery of relevant research. 6.1.2 Stiengthen selected hizh-prioritv nationtal prograrns focusing on less deieloped r egions. Although for a variety of reasons ( see 5.1) the research capacity in the less developed region has not significantly improved ( and may have even declined due to staff attrition) institutes outside of the less developed regions did pay more attention to these regions and their problems. Given the skills developed under TARP, research in fields that are relevant to the less developed regions will continue. These fields include innovation in water-efficient agriculture, in new and improved crops, and in cultivation techniques that are adopted to these areas. 6.1.3 Institutionalize a conipetitive agricultural r esearch proeram that invests in hijh piriority r elevanit The competitive research component has led to a change in the research system where research is more demand driven, with greater input of the individual researchers as well as the beneficiaries. In addition, researchers obtained access to international standards and tools in research and to more peer reviews that are part of the international research system. This change is likely to be sustainable. The sliding scale whereby the project would finance 100% of competitive research in 1996 with an increasing contribution of the national budget to the competitive research fund until 1999 did not materialize. For GDAR, the largest of the implementing agencies, the 2001 budget request was 6.2 trillion TL (US$ 4.4 million). Even after allowing for capital, repairs and maintenance, fuel and travel costs, about US$ 2 million remains for research funding. However, SPO reduced this by more than 50 percent and then reallocated the budgets on the basis of research programs that bear little relationship to RMP research priorities and more to research programs prior to TARP. So, in spite of the positive experiences under the CRGPs, and full acceptance by the line ministries, it has not been implemented by SPO (at least not for the very constrained 2001 budget). 6.1.4 C7owtributini' to institutional buileling of A RA 's anQd MoF's a!Zicultural and tbrestr1 research se-rvices. About 30 percent of the TARP budget was spent on training which greatly improved the skills (both language and technical) in research and allowed Turkish researchers' access to international research and consequent connections. The outcome of this effort was somewhat tempered by: (i) continuing high staff turnover rates, particularly in GDAR; (ii) the continued moratorium on staff hiring; (iii) the absence of merit criteria in appointment decisions; and (iv) a relatively low number of years of service before retirement (25 years of service for males, 20 for females) together with generally unfavorable conditions of employment leading to an increased rate of early retirement. Still, although affected staff may have left the particular MARA institutions, they often moved to universities or private sector where the acquired skills contributed to research or to the Turkish economy in general. 6. 1. 5 Establishinz an economics and policy researc/h cavpability fir agricultuwe. AERI was established as an institute with a relatively low in-house capacity to undertake agricultural economics and policy research. TARP funds were used initially to contract a small number of institute staff and some short-term research project staff. This contracting facilitated implementation and guaranteed AERI's financial and scientific independence but may have been at some costs to local (MARA's) ownership and integration of economic issues in the other (technical) institutes. Turkey's economic research capacity was also enhanced by contracting, initially a mix of foreign and local and later on mainly local skills and institutions, for example in the Middle East Technical University (METU). As such, the project contributed significantly to establish and enhance agricultural policy research. * 19-  Unfortunately, AERI as a possible sustainable institution came together only in the final two years of the Project, when a local acting director was confirmed and when it had built a track record of reports that attracted national attention, and generated some revenue. Efforts to obtain authorization to operate a flexible revolving fund and retain some of its revenue have, so far, failed. Currently the AERI's status is that of a department of MARA with staffing and central budget links to GDAR. Given its performance in economic and policy research and the reputation it rapidly established, both in the public and private sector, an AERI with a limited opportunity to contribute would be a disappointing outcome. 6.2 Transition arrantgement to regiular operations: The RMPs developed in TARP have been updated and remain as the basis for research prioritization and budget preparation. However, the "output" based approach to research funding supported under the project has been revised. Ongoing austerity measures in 2001 may temporarily delay full absorption/implementation of the concepts and investments promoted by TARP. Following an almost 100 percent devaluation of the Turkish Lira (TL) in February 2001, and a mandated maximum of 40% budget increase of the institutions most likely means a 30% fall in real terms in research budgets. 7. Bank and Borrower Performance Bank 7.1 Lending: While design and quality at entry were deemed unsatisfactory, the Bank's overall performance was satisfactory. Various changes in the structure of the loan as well as in the developmnent objectives and technical implementation were discussed with Government and successfully applied. Increasing the research grant component was instrumental in overcoming difficulties with counterpart funding and facilitated disbursements following project restructuring. 7.2 Supervision: Sixteen supervision missions involving 14 staff were completed with only one member common on more than 10 missions. Changing staff, both within Bank supervision missions teams and within imnplementing agencies appeared to result in inconsistent messages. For example after the MTR, the GDAR was encouraged to rationalize its use of research resources through strengthening 7 Central Research Institutes (CRIs). Plans were prepared and a "no objection" given for the development of 2 CRIs and approval in principle for others subject to acceptable plans. A later Supervision mission rejected the concept of CRIs and suggested instead up to 27 Centers of Excellence. In the meantime, GDAR on the basis of the "no objection", went ahead with development at 7 CRIs. During the first 4 years, the emphasis was on sound planning before authorizing disbursements. This on occasion led to inconsistent performance ratings. For example over a five-month period from January 1997 the performance of MoF changed from "satisfactory, making good progress" to "well behind schedule" with a prospect of cancellation. The initial assessment was based on reviewing total disbursement without regard to quality. The Bank supervision focused largely on implementation issues and did not pursue solutions to issues of government ownership, monitoring and evaluation, and improvements in financial management. The final outcome was increased disbursement for goods and competitive research, but little debate about or progress on the more fundamental issues facing the agricultural research in Turkey. -20-  The change from headquarter-based supervision to Resident Mission (RM) based supervision helped to improve the performance and the relations between the implementing agencies. However, it greatly increased the Bank's direct involvement in project operations leading to micro-management, with a risk of too great dependency of the Borrower on Bank supervision as a monitoring and evaluation tool. 7.2.] Task managemeni. The TARP was supervised by four different Bank Task Managers. Overall supervision was satisfactory. The Bank's recognition of the need for improved and more constant relationships with the implementing agencies resulted in appointment of an Ankara based project management. This decision, in mid-1998, which resulted in more consistent support from the Bank, was welcomed by implementing agencies. 7.3 Overall Bank performance: The Bank's overall performance was satisfactory, albeit not always consistent. While this led to a project design that was over-ambitious, the Bank's response to the issues identified in the MTR was flexible and sought to improve implementation of a complex project that involved five separate implementing agencies. Borrower 7.4 Preparation: GOT Commitment to TARP was marginally satisfactory. The preparation process was prolonged, in part because of changes made in Government during the later stages of preparation. The SAR accepted these changes without modifying the project organization, especially by leaving GDAR as the main coordinating agency. This consequently led to noncooperative relationships between GDAR, GDRS and MoF during the first 3 years of project implementation. Establishment of separate implementing agencies resolved that issue to some extent, but raised other issues of research collaboration and duplication. 7.5 Government implementation peu]br,nance: The main difficulty brought about through the relatively poor economic climate, was the lack of sufficient counterpart fumds and changing requirements for financing of taxes and duties. This affected project implementation and disbursement. TARP was exempted from these taxes from 1997 - 98 (when over 40 percent of total TARP funds were used). Re-application of taxes in 1998 again constrained disbursement, particularly in GDRS and MoF. Turkish Treasury's auditors, as agreed with the Bank, carried out financial auditing. These audits reports were, initially: (i) late and weak; (ii) did not report on total project funds to be consolidated with IBRD records; and (iii) did not require a follow-up in case of discrepancies. The delays in audits were reaching 2 years or more, which was pointed out during supervision in 1999, and in a letter from the Country Director to Treasury in November 1999. The Treasury, assisted by the Bank, took action through training of auditors, resulting in improvements in the delivery and quality of audits and their follow-up, if necessary. 7.6 Implementing Agency: The performance of GDAR as a single implementing agency of the project was deemed unsatisfactory, and after 1994 each agency in the project became an independent implementing agency. These agencies performed well as such, but fragmented the project management, as there was no single agency responsible for the entire project. 7.7 Overall Borroiver per/ormance: Overall Borrower performance was marginally satisfactory. Despite extemal uncertainties such as weak public finance, unstable political environment, frequent changes at the ministerial level, and high turnover of management and staff, nevertheless, great progress was made in changing the approach to research and in achieving development objectives. - 21 -  8. Lessons Learned 8.1. The lessons from previous or ongoing projects at the time of preparation, in particular the first and second agricultural extension and applied research projects, appear insufficiently incorporated into the project design. Bank projects should incorporate more fully lessons, positive and negative from ongoing operations in the country. 8.2. Although the structural problems of an oversized, fragmented research structure and inflexible personnel management policy in MARA was known, they were not addressed. Future operations should acknowledge the risks of proceeding with investment lending in the absence of broader structural and civil service reform. 8.3. There was insufficient local ownership within the research community or MARA at the outset. Preparation of future operations should devote more time to broad stakeholder involvement and commitment to project design. 8.4. The method of research funding through a competitive grant system worked reasonably well in Turkey. The concept was not completely new in Turkey as it had been applied, in a less rigorous manner, for some time by TJBITAK in its funding of university based research. (A number lessons emerge which could be useful also for other ECA countries): 1. Research funding through competitive grants need to be complemented by funding for longer-term research. 2. The "farmer-responsive" competitive grants process, needs to be carefully explained to the research and extension community. 3. Financial, procurement and other fiduciary arrangements need to be carefully worked out. 4. Benefits and results need to be carefully monitored in terms of "outcomes" (fanner adoption rates, higher productivity) as well as "outputs" (reports and extension messages). 8.5 The technical assistance worked quite well. The lesson is that long-term TA, with strong commitments both from the client and the provider, can help to catalyze change. 9. Partner Comments (a) Borroiver/inplementing agency: 9.1 Letter from Treasury. The borrower's letter (11. 1.01 from the Director General, Foreign Economic Relations of the Treasury) included the following evaluation: 1. Government recognized the changes in the project from being managed by a single government agency to five implementing agencies. 2. It considers the disbursement patterns as a good example of a project affected by a number of features that evolved continuously during the lifetime of the project and were actually based on improvements made during implementation. 3. It also recognized that the currency conversion and cancellation lead to minor confusion with respect to budgetary allocation among the implementing agencies. However implementing agencies adapted well to the changing conditions and issues. 4. In general, the project is evaluated as "an extremely enriching experience" in agricultural and forestry research. 5. In addition to encouragement and enhancement of research in public institutions, the project funds - 22 -  were timely and valuable with respect to the creation of AERI and in supporting essential economic policy research and development work to support sectoral reform. AERI was considered to have filled a niche in research needed in agricultural policy making. 6. It is considered worth noting that visible improvements are noticeable in all agencies, and even among their private partners, in project selection performance criteria, research training and funding of agricultural research. The Treasury considers borrower performance satisfactory as occasional difficulties were overcome through consultations and were never permanently affecting the project. 9.2. Summarv of comments of implementing agency in Borrower Implementation Completion Report Prepared by the Task Manager and is based on the comments of the Turkish Treasury and implementing agencies of the TARP-November, 2001. General comments Treasury. With its highs and lows, the Project has been an extremely enriching experience in the agricultural and forestry research area in Turkey, in the government bodies and also with the private partners that joined in the implementation of some components of the research especially during the recent years. Project funds supported timely research and the AERI contributed to sectoral reforms. TUBITAK. The project helped TUBITAK to gain very valuable experience in defining research areas, evaluating/screening project proposals, implementing and monitoring and publishing the results. The agency adopted the CRGP concept fully and developed it further and is committed to use it in their future programs. There has been significant ownership in TUBITAK for the project. The agency allocated considerable amounts of funds from its budget to cover taxes and management expenditures for smooth implementation of the component. The project provided an opportunity to better understand and further develop the CRGP. The experience gained earlier in the project provided a valuable basis in the preparation and supervision of the "Guided Projects" which was introduced in the last year of the project. Meetings organized with the producers provided important inputs to develop demand driven projects. The program also helped TUBITAK to seek and improve the collaboration among research institutions, industry and producers. MoF. There is no doubt that the forestry research system highly benefitted from the project as a result of support provided to the institutes and the researchers, GDAR. The project introduced a system and discipline to the research environment. It also helped to direct the resources towards the priority areas. GDRS. The project provided important contributions to the rejuvenation and development of the research system. Project preparation Treasurv. The Project was designed for a single government body, the Ministry of Agriculture, Forestry and Rural Affairs (MAFRA) but was implemented by five agencies since MAFRA was split into two - 23 -  twice, the first being in 1991 and the second in 1993. In addition, an institute (AERI) was established in 1996 and TUBITAK was included in mid-1997. The problems due to the over-ambitious project design in relation to the economic, administrative and research management capacity in Turkey at the time of the preparation of SAR have at times negatively affected the project implementation. GDRS. Since GDRS was part of the MAFRA at the time of preparation, it was not directly involved in preparation. Bank and Borrower Performance Comments of the agencies are summarized below: Treasur,. During the implementation period, the establishment of AERI in 1996 and inclusion of TUBITAK as an additional implementing agency had a positive effect on the overall performance of the project. Since AERI has filled up an important niche in research needed in agricultural policy making of the govenmment, it is important that it be supported financially to continue performing its activities. Although the restructuring of budget allocation process for research activities was not envisaged during design of the project and in the revised objectives, the implementing agencies have adapted their procedures and fund flows quite successfully to overcome budgetary difficulties. Overall borrower performance should be rated as "satisfactory" since occasional hardships in specific implementation or disbursement related matters were overcome through consultations among the agency Project Implementation Unit (PIU), Treasury and the World Bank project task teams. Instead, it is worth praising the capability of the agencies in adapting their organizational and financial structures to the activity plans and procedures regarding administrative, technical and technological issues over the project life, and their adoption of the fundamental objectives of the project in conducting their research activities in a way that seems to be sure to continue after the project is closed. The main indicator of adoption of the objectives of the project is the importance all agencies attach to implementation based on the Research Master Plans developed under the project and revised and improved periodically. Visible improvements have occurred in the approaches of all agencies, and even their private partners, to project selection, performance criteria, researcher training and funding in basic agricultural research. TUBITAK. TUBITAK treated the Research Master Plans prepared by MARA, GDRS and MoF as basics documents for the project and used them as a basis in screening the project proposals submitted by universities and other research institutions. The agency made special efforts to ensure effective and efficient use of resources and in order to prevent duplications, a website was developed which listed supported projects. AERI. There was a general lack of will in the agencies to cooperate and proceed in a coordinated manner. There was an effort to organize and re-formulate Turkish agricultural research organized by TUBITAK in 1998. However, no consensus was reached in the Working Group due to the agencies' being too turf-conscious. The proposed Higher Council for Research and Development is strongly needed to reconcile the collaboration and duplication issues. The CRGP and priority setting need to be mainstreamed in research programs of all research agencies. However, this is not the case. The system's performance was marginally satisfactory. - 24 -  GDAR. In the earlier phase of the project, the project performance was affected by: (i) insufficient counterparts; (ii) lack of coordination among agencies; (iii) heavy bureaucratic procedures; and (iv) insufficient number of staff. This called for effective coordination among SPO, Treasury, Ministry of Finance, and implementing agencies but the efforts were insufficient. However, it should be noted that the Treasury and the World Bank's efforts are regarded as useful. After 1995, no problems were encountered regarding the domestic budget allocations and overall coordination. Therefore, project implementation was accelerated. The Government's decision to exempt imported scientific goods from customs duties and taxes was very helpful for the project. The Project highly benefitted from this for the library information system, field and laboratory equipment. However, Government's austerity measures limited the project activities. If the externally financed projects had been exempted from these measures, it would have enhanced the project implementation. MoF. Although the Loan funds were available, insufficiency of domestic budget allocations did not allow their timely use. There was a need for a coordinated effort from Treasury, Ministry of Finance and the Central Bank sides. The government's austerity measures did not allow the agency to proceed with the graduate program for researchers. The project was not promoted within the Ministry during the initial years of the project. Only after the first RMP, did any ownership materialize. Quality research staff could not be maintained due to the government's personnel policies such as low salaries and low status. GDRS. The agency did not experience any difficulty in provision of the counterpart funds. The agency agrees with the evaluation and ratings in the ICR (b) Cofinanciers: N/A (c) Other partners (NGOs/private sector): Summary of comrnents of randomly selected beneficiary institutes (2000) Institution Benefits Concerns Livestock Institute With TARP investments, the Institute is Institute has not evaluated the negatives (Konya) now better equipped to do research; i.e. (if any). better skills, better equipment. Overseas training and visit .opened our eyes" and provided new views. Winter Cereals TARP increased flexibility, and secured Lack of extension, lack of economics. Institute (Konya) longer term research funding. Better collaboration with international institutes. GDRS (Konya) Better equipment, office technology TARP is ending, and not sure the MARA will compensate for loss of funding. Overseas training was beneficial. E-mail and new journals/books provide access to worldwide research results. - 25 -  "TARP opened eyes and views". Forestry Dept. TARP allowed contact with the outside Retention of quality staff is still a (Izmir) world, resulting in introduction of new problem. species, and international collaboration. Process of Masterplan brought discipline and priority in selection of research topics. TARP provided better motivation for staff. Ability of draft proposals for international support has improved. GDRS Research TARP brought new equipment, and enabled (Izmir) the institute to provide better research and services. Enabled to provide demonstration of modem tools and methods (for farmers) Ege University Better streamlining of research awards is Dissemination of results is still (Bornova-lzmir) suggested. inadequate, although University has its own extension section. [Note: university TARP helped many young researchers, and Bias in TUBITAK's review and reward research was only contributed to MS and PhD training at the system (in favor of crops, and-within supported through University.. livestock- favoring animal health). TJBITAK] Researchers are now better able to approach TUBITAK will not support economic other funding agencies and private sector. research, as this is considered social science. Observation: TUBITAK gets all rights to new inventions, even if it only paid part of the research (and not for salaries). Still a lack of collaboration between researchers (on same topic), and risk of duplication. Olive Research Master plan is helping to prioritize research. Funding"after TARP" is insecure. Institute (Izmir) TARP helped to maintain research when Limited application of research in the Turkey withdrew from International Olive field. Council 10. Additional Information - 26 -  Annex 1. Key Performance Indicators/Log Frame Matrix Outcome / Impact Indicators: Indicator/Matrix Projected in last PSR ActuallLatest Estimate 1. Farmers adopting new technologies The time lag between the carrying out of Although in most cases it is still too early to released from GDAR, GDRS or MoF research and the time it reaches the farmers measure this impact, some technologies (in research. is too long to be evaluated in such IPM for example) are being transferred and projections. adopted. 2. Research Master Plans are basis for Although RMP were used by the Ministry of budget allocations based on program Agriculture in its budgets planning, the priorities. concept was not (yet) accepted by the State Planning 3. Management informaton systems Information system is installed, functional installed and used to monitor and manage and respected. research system. 4. AERI established and carrying out AERI established and filling a role in research and analysis of priority public policy agricultural policy (see Borrower comment) issues in agriculture. Output Indicators: Indicator/Matrix Projected in last PSR Actual/Latest Estimate 1. Contestable Research Grant Programs 1. a. RMPs defining research priorities a. Satisfactory b. RMPs revised b. Completed and Highly Satisfactory c. Funds allocated to priority areas c. Satisfactory d. Project proposal and evaluation criteria in d. Satisfactory place e. Research contracts and funds transferred e. Satisfactory 2. Research capacity to deliver priority 2. research a. Training needs analysis relating to priority a. Marginally Satisfactory programs b. b. Training delivered Marginally Satisfactory c. Economics and Policy capacity developed c. Highly Satisfactory d. Research coordination and collaboration d. Unsatisfactory e. Research Projects completed e. Satisfactory f. Multi-disciplinary research programs f. Unsatisfactory g. Research applicable to less developed g. Marginally Satisfactory areas h. Research Publications h. Marginally Satisfactory 3. Research Management Development 3. a. On-the-job and formal training delivered a. Satisfactory b. Job Descriptions b. Unsatisfactory c. Performance management systems c. Unsatisfactory (beyond Project control) d. Research Institute rationalized d. Unsatisfactory e. Nongovemment funding for research e. Marginally Satisfactory f. NARS sustainability f. Unsatisfactory 4. Research Management Information 4. System a. RMIS defined a. Satisfactory b. LANs and WANs installed b. Satisfactory c. RMIS installed and operational c. Satisfactory d. LIS and central library operational d. Satisfactory End of project Note: The SAR did not provide a logical framework or define any performance indicators. Such indicators were developed during the first supervision mission (May 1993). The Mid-Term Review developed a logical framework and GDAR, GDRS, MoF and AERI MoF defined indicators and means of verification in 1997. - 27 -  Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) Appraisal Actual/Latest Percentage of Estimate Estimate Appraisal Project Cost By Component US$ million US$ million Agricultural Policy 0.00 6.00 Applied Research 0.00 7.30 Institutional Strengthening 10.60 23.80 220 Soil and Water 12.10 5.50 45 Forestry 5.60 3.80 68 Crops 22.60 9.60 42 Livestock 9.30 1.50 16 Farming Systems 1.50 0.80 53 Integrated Pest Management 4.50 0.80 18 Residue Control 2.30 0.00 0 Total Baseline Cost 68.50 59.10 Physical Contingencies 3.70 0.00 0 Price Contingencies 3.80 0.00 0 Total Project Costs 76.00 59.10 Total Financing Required 76.00 59.10 _ - Note: All components after the restructuring in 1997 have been taken into consideration. Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) Expenditure Category ICB Procurement Method N.B.F. Total Cost NCB Other2 1. Works 10.00 11.90 0.00 0.00 21.90 (3.50) (4.20) (0.00) (0.00) (7.70) 2. Goods 14.60 0.00 3.20 0.00 17.80 (12.70) (0.00) (2.60) (0.00) (15.30) 3. Services 0.00 0.00 28.10 0.00 28.10 (0.00) (0.00) (25.10) (0.00) (25.10) 4. Miscellaneous 0.00 0.00 4.00 0.00 4.00 Research Grants (0.00) (0.00) (4.00) (0.00) (4.00) 5. Miscellaneous 0.00 0.00 0.00 5.80 5.80 Incremental OC (0.00) (0.00) (0.00) (2.90) (2.90) 6. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 24.60 11.90 35.30 5.80 77.60 (16.20) (4.20) (31.70) (2.90) (55.00) Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) Expenditure Category ICB Procurement Method N.B.F. Total Cost NCB Other -28 -  1. Works 0.00 1.38 0.00 0.00 1.38 (0.00) (0.69) (0.00) (0.00) (0.69) 2. Goods 4.57 0.00 6.96 0.00 11.53 (3.94) (0.00) (5.80) (0.00) (9.74) 3. Services 0.00 0.00 19.30 0.00 19.30 (0.00) (0.00) (17.37) (0.00) (17.37) 4. Miscellaneous 0.00 0.00 16.67 0.00 16.67 Research Grants (0.00) (0.00) (15.64) (0.01) (15.65) 5. Miscellaneous 0.00 0.00 0.00 0.01 0.01 Incremental OC (0.00) (0.00) (0.00) (0.00) (0.00) 6. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 4.57 1.38 42.93 0.01 48.89 (3.94) (0.69) (38.81) (0.01) (43.45) Notes: 1. National shopping was disaggregated to below limits of NCB for expediency reason. 2. Throughout the project the regulations regarding payment of VAT and customs were variously exempt or applied. On average VAT and Customs duty is assumed to be 15 precent each, with the cumnulative effect being 32 percent. ' Figures in parenthesis are the amounts to be financed by the Bank Loan. All costs include contingencies. 'Includes civil works and goods to be procured through national shopping, consulting services, services of contractcd staff of the project mranagemcnt office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. Project Financing by Component (in US$ million equivalent) r | | Perce~~~~~~~~~~~~~~ntage of Appraisal| Component L Appraisal Eslimate ActualfLatesl Estimale Bank Goil. CoF. Bank Go% t. CoF. Bank Govt. CoF. Institutional strengthening 7.90 2.70 18.00 5.80 227.8 214.8 Farming systems 1.30 0.20 0.60 0.20 46.2 100.0 Crops 12.50 10.10 7.30 2.30 58.4 22.8 Livestock 6.70 2.60 1.20 0.30 17.9 11.5 Soil & water 10.20 1.90 4.20 1.10 41.2 57.9 Forestry 5.00 0.60 2.90 0.90 58.0 150.0 Integrated pest 3.50 1.00 0.60 0.20 17.1 20.0 management Residue control 1.90 0.40 0.0 0.0 Additional components after restructuring: Applied research 5.60 1.70 Agricultural policy 4.60 2.60 [Cancellation] 6.00 Agricultural Policy, Training, Applied Research were new components added after midterm review. The original components Crops, Livestock, Integrated pest Management were amalgamated in Applied Research and Institutional Strengthening. Residue Control was cancelled. - 29 -  Annex 3. Economic Costs and Benefits N/A - 30 -  Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, I FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 06/90 6 Crop, Economist, Soils Irrigation, Forestry, Livestock, Farming Systems Research Appraisal/Negotiation Preappraisal 11 Team leader, Consultant, 04/91 Farming Systems Appraisal 06/91 -- -- Post Appraisal Team leader 01/92 Supervision 6/92 3 Team leader, agriculturist, operation specialist (consultant) Startup mission 05/93 2 Team leader, agriculturist, U U operations specialist (consultant) 10/93 03/94 3 Team leader, agriculturist, U U Farming Systems specialist 11/94 4/95 3 Team leader, farming systems U U specialist (consultant) 11/95 Tearn leader U U 3/96 3 Team leader, economist, U U agriculturist 8/96 2 Team leader, economist, U U agriculturist 12/96 2 Team leader, economist U U 5/97 2 Team leader, project officer S S 02/98 5 Tearn leader, sector leader, S S agriculturist, forester, land and water development specialist (consultant) 10/98 3 Team leader, agriculturist, sector S S leader 3/99 Teamn leader, financial specialist S S 09/99 6 Team leader, economist livestock S S specialist, agriculturist, financial management specialist, procurement specialist (consultant) - 31 -  6/00 3 Team leader, livestock/research S S mgt. specialist 10/00 3 Team leader, sector leader, S S livestock/ research mgt specialist ICR 2 Team leader, research management specialist (consultant) (b) Staff Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation 17.1 279,954 Appraisal/Negotiation 29.1 included in above Supervision 67.0 579,667 ICR 8.6 21.2 Total - 32 -  Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU-Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating OMacro policies O H OSUOM * N O NA O Sector Policies O H O SU O M * N O NA O Physical O H O SU * M O N OVA O Financial O H OSUOM ON * NA O Institutional Development 0 H 0 SU * M 0 N O NA O Environmental O H O SU * M O N O NA Social 3I Poverty Reduction O H OSUOM O N * NA L Gender O H O SU O M O N * NA L Other (Please specify) O H OSUOM O N * NA L Private sector development 0 H 0 SU 0 M 0 N * NA O Public sector management 0 H O SU * M 0 N 0 NA OOther (Please specify) OH OSUOM O N * NA - 33 -  Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating 0 Lending OHS OS *U OHU CJ Supervision OHS OS OU O HU C Overall OHS OS OU O HU 6.2 Borrowerperformance Rating O Preparation OHS OS * U O HU O Government implementation performance O HS O S 0 U 0 HU O Implementation agency performance O HS OS 0 U 0 HU O Overall OHS OS O u O HU - 34 -  Annex 7. List of Supporting Documents I . Staff Appraisal Report 2. Loan Agreement 3.. Aide Memoires 4. Project Status Reports (Form 590s) 5. Project Files -correspondence with govemment 6.. Research Master Plans - GDAR, GDRS, MoF (first and revised) 7. Mid-Term Review 8. Strategic plans: (a) Strategic plan to improve research relevancy And efficiency of the Central Research Institute for Field Crops; 1997. (b) The plan for improvement of the Trabzon Fisheries Research Institute, 1997. (c) Strategic plan for improvement of the Central Research Institute of Food Technology; 1997. (d) The plan for improvement of the Ataturk Horticultural Central Research Institute; 1997. (e) The plan for improvement of the Lalahan Livestock Central Research Institute; 1997. (f) The plan for improvement of the Ankara Plant Protection Research Institute; 1997. (g) The plan for improvement of the Etlik Central Veterinary Control and Research Institute, 1997. 9. GDAR Implementation Completion Report. 10. MoF Implementation Completion Report. II. GDRS Implementation Completion Report. 12. TUBITAK Research Grant Committee Status Report. 13. AERI Research projects Report. 14. TARP Independent Auditors Report. 15. Furtan, W. et al., 1999 An Examination of Agricultural Productivity and Returns to Agricultural Research in Turkey, Ankara, AERI May 1999. 16. Letter Treasury Department 11/01/01 on implementation evaluation. - 35 -  -36 -