Ghana - Structural Adjustment Institutional Support Project

Document of The World Bank FOR OFMCUL USE ONLY Report No. 13262 PERFORMANCE AUDIT REPORT GHANA STRUCTURAL ADJUSTMENT INSTITUTIONAL SUPPORT PROJECT (CREDIT 1778-GB) JUNE 30, 1994 Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS Currency Unit: Cedil' 1987 1988 1989 1990 1991 1992 Cedis per US$1.00 153.7 202.4 270.0 326.3 362.8 431.9 ABBREVIATIONS AND ACRONYMS ASYCUDA Automatic System of Customs Data Entry, Control and Management CB Capacity building ELU Economic Liaison Unit EMS Economic Management Support Project EPU Economic Policy Unit ERP Economic Recovery Program GDP Gross Domestic Product GLSS Ghana Living Standards Measurement Survey GSS Ghana Statistical Service IBRD International Bank for Reconstruction and Development ID Institutional Development IERD International Economic Relations Division IPAD Investment and Project Analysis Division IPPD Integrated Personnel and Payroll Database IRS Internal Revenue Service LSMS Living Standards Measurement Study MFEP Ministry of Finance and Economic Planning MSD Management Services Division NRS National Revenue Secretariat ODA Overseas Development Administration OHCS Office of the Head of the Civil Service PAD Policy Analysis Division PAR Performance Audit Report PCR Project Completion Report PIP Public Investment Program PMU Project Management Unit PPF Project Preparation Facility PRD Planning and Research Division SAIS Structural Adjustment Institutional Support SAP Structural Adjustment Program SAPSEC Structural Adjustment Program Secretariat SAR Staff Appraisal Report SMS Skills Mobilization Scheme UNDP United Nations Development Programme FISCAL YEAR January 1 - December 31 1' Period average. FOR OFFICIAL USE ONLY THE WORLD BANK Washington, D.C. 20433 U.S.A. Office of Director-General Operations Evaluation June 30, 1994 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Performance Audit Report on Ghana - Structural Adjustment Institutional Support Project (Credit 1778-GH) Attached is the Performance Audit Report on Ghana - Structural Adjustment Institutional Support Project (Credit 1778-GH), prepared by the Operations Evaluation Department. This project, approved in 1987 and closed in 1993, accompanied the first Structural Adjustment Credit to Ghana. Its twin goals were to facilitate implementation of the economic reform program and to rebuild the Government's capacity to efficiently manage the economy. The findings of the audit differ from those of the Project Completion Report (PCR) in several respects. The PCR found that the project was implemented for the most part as designed. The audit, however, considers it a major departure from design that equipment costs--projected in the Appraisal Report at about 30 percent of total project costs--accounted for 70 percent of actual total cost. The problem was not the departure from design per se. Indeed, it was appropriate that the project was designed in accordance with a "process" (as opposed to "blueprint") approach. The main weakness was rather that the process itself was underdesigned and undermanaged. In contrast to the PCR, therefore, the audit rates the project outcome as unsatisfactory. The PCR and audit agree that the project's institutional development (ID) impact was partial/modest while the audit rates sustainability as uncertain, in contrast to the previous rating of likely, based on PCR findings. One of the main lessons identified in the audit is that "externalities matter" but are too often neglected in the design of technical assistance projects such as this one. Such externalities may include distortions in incentive structures that lead to rent-seeking and other behavior not conducive to viable ID. Robert Picciotto by H. Eberhard K6pp Attachment This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.  FOR OFFICIAL USE ONLY PERFORMANCE AUDIT REPORT GHANA STRUCTURAL ADJUSTMENT INSTITUTIONAL SUPPORT PROJECT (Credit 1778-GH) TABLE OF CONTENTS GHANA AT A GLANCE Preface ................................................................. Basic D ata Sheet ........................................................ iii Evaluation Summary .................................................. . vii I. INTRODUCTION/BACKGROUND................................. 1 Background .. ................................................... 1 Origins of the Project ............................................. 2 How the PAR Differs from the PCR in its Coverage ...................... 2 How the PAR Differs from the PCR in its Conclusions .................... 3 II. PROJECT OBJECTIVES AND DESIGN..............................4 Project Objectives...............................................4 Cofinancing by ODA .............................................5 Salient Project Characteristics.......................................5 Expertise and Experience of the Appraisal Team ........................6 Relevance of the Objectives: Evaluation ..............................6 Timeliness of the Project: Evaluation .................................7 Participation and Ownership: Evaluation...............................7 Risk Assessment at Appraisal .......................................8 Annual Work Programs and the "Process Approach"....................... 9 III. PROJECT IMPLEMENTATION................................... 10 Rating Methodology............................................ 10 Lack of Data/Indicators on Project Outcomes........................... 11 Project Outcomes and Efficacy..................................... 11 Main Points of PCR/PAR Agreement................................ 15 Main Points of PCR/PAR Disagreement............................... 15 The Shift to Equipment Purchases................................... 17 This report was prepared by Robert Armstrong (Task Manager) who audited the project in October 1993. Norma Namisato provided word processing assistance. This document has a restricted distribution and may be used by recipients only in the performance of their ain Its contents may not otherwise be disclosed without World Bank authorization. Table of Contents (cont'd) IV. EVALUATION AND SUSTAINABILITY ........................... 22 Outcome and ID Impact .......................................... 22 What Were the "Side Effects" on the "Institutional Environment"? .......... ..22 SAIS Project Externalities: Positive ................................. 22 SAIS Project Externalities: Negative ................................ 23 Conclusion .................................................... 24 Overview of Project Strengths and Shortcomings ........................ 25 Outcome of the ODA-Managed Components--Civil Service Reform Program . . 25 Sustainability .................................................. 25 Borrower Performance and Compliance with Covenants .................. 27 Bank Performance .............................................. 28 The Follow-up Project ........................................... 30 V. LESSONS OF EXPERIENCE ..................................... 32 BOXES 1. Main Project Strengths and Weaknesses ................................ x 2. The National Revenue Service: A Reform Enclave? ...................... 16 3. Need for a Country-wide and Multi-donor TA Review .................... 24 4. Excerpts from the ODA Evaluation Report ............................ 26 TABLES 1. Project Results According to the PCR ................................ 12 2. Efficacy Ratings by Main Components ................................ 14 3. SAIS: Planned and Actual Allocations ................................ 18 4. Main Components of Equipment Purchased Under SAIS Project ............ 20 5. Cost Effectiveness Ratings by End-use ................................ 20 CHART 1. SAIS: Planned and Actual Expenditure by End-use ...................... 19 GHANA AT A GLANCE Ghana is a low-income country, with a &mank Dffdopuxb population of about 16 million, located on the west coast of Africa. Its land area is about the same as Before the Economic Recovery Program (ERP) that of the United Kingdom. Uganda and the Lao was adopted in 1983, Ghana's economy had PDR. Per capita income in 1992 was US$450 collapsed. The policy framework, physical (World Bank Atlas method). Average life infrastructure and institutions were all in disarray. expectancy is about 55 years and the literacy rate The ERP was therefore launched at a time when a is about 60 percent. broad consensus for change prevailed. Initial reform efforts focussed on stabilization and PoMmya Denlopwials liberalization and were in general successful unWi a weakening of fiscal dicipline in 12 resulted in Ghana achieved its independence in 1957 and a resurgence in inflation and in a rapid depreciation was ruled by a succession of civilian and military of the now market-determined exchange rate. regimes until 1992, when a new constitution was approved by referendum and multi-party Average annual GDP growth over the decade presidential and parliamentary elections were held. following the introduction of the ERP was nearly The current President of the elected civilian regime 5 percent, with services being the fastest-growing is Flight Lieutenant (retired) Jerry Rawlings. The sector. Agricultural sector growth over this decade next elections, both presidential and parliamentary, was 2.3 percent overall, with 3.1 percent growth in will be in 1996. cocoa production but only 2.2 percent growth in food production. The Ecoomic SucwT Adjustment measures were implemented with Ghana is well endowed with a broad range of varying degrees of intensity and success. Good natural resources such as arable land, forests, and progress was made, for example in exchange rate, sizeable deposits of gold, diamonds, bauxite, and pricing and trade policy reform. Ther remains, manganese, as well as a considerable capacity of however, a substantial agenda of reform to be hydroelectric power. The economy has traditionally undertaken in public sector management and civil depended to a high degree on primary (agricultural service reform, public enterprise divestiture, and in as well as mineral) production and exports. Exports creating an enabling environment for private sector of gold, cocoa, and timber still account for the developmen. Low private savings and investment bulk of total merchandise exports, with respective rates need to increase substantially to sustain shares of 29 percent, 31 percent, and 13 percent. growth in the medium-term while issues of high population growth, dectining agricultural The agricultural sector, inclusive of forestry productivity and environmental problems need to and fishing, remains the dominant sector, be addressed if growth is to be sustained over a employing some two-thirds of the labor force and longer run. accounting for nearly half of total GDP. Agricultural production, which is primarily small scale, is concentrated in cocoa and staple food A Few Key b&mtorw (khmi) crops. Ghana is currently ranked as the second largest (after CMte dlvoire) producer and exporter % of of cocoa. 9M I Private invearrient 4 Population Ov1. inveutmotl 10 (ammual rwwth) 3.1 Services comprise the second largest sector in Cospon 102 Enollmad ratio the economy, accounting for an increasing share in National saviogs I Prinwry 75 GDP (37 percent in 1993), while the industrial Broad soney 17 Secondary 39 sector accounts for the remaining 16 percent. Forei8n finasing 10 Inlation 24  PERFORMANCE AUDIT REPORT GHANA STRUCTURAL ADJUSTMENT INSTITUTIONAL SUPPORT PROJECT (Credit 1778-GH) PREFACE 1. This is a Performance Audit Report (PAR) on Ghana - Structural Adjustment Institutional Support (SAIS) Project (Credit 1778-GH). The Credit, in the amount of US$10.8 million, was approved by the Board on April 14, 1987. The Credit was fully disbursed and was closed, on schedule, on June 30, 1993. 2. The Overseas Development Administration (ODA) of the U.K. provided parallel financing in support of components supporting a wider Civil Service Reform Program (CSRP). This financing, originally projected at the level of US$1.1 million, in the event amounted to £3.5 million over the period of the SAIS project. ODA prepared its own Evaluation Report on its assistance to the CSRP. The main findings and conclusions of that report are cited in this PAR. 3. This PAR is based on the PCR,V the President's and Staff Appraisal Reports, credit documents, credit and country files, Bank and IMF economic reports on Ghana, other relevant documents, and discussions with Bank staff, present and former Ghanaian officials, and ODA staff who were associated with the project. Their kind cooperation and valuable assistance is gratefully acknowledged. 4. The coverage, findings and ratings of this PAR differ from those of the PCR in several respects. The PCR is quite informative, and provides an adequate description of the objectives and outcomes of the main project components. It does however beg some important questions, e.g. the extent of Borrower ownership, the relevance and appropriateness of project objectives and conditionalities, and why the composition of actual project expenditures was so different from the composition of planned expenditures. The PAR explores these issues in some detail and comes to the conclusion, unlike the PCR, that the project outcome was on the whole unsatisfactory. 5. The draft PAR was sent to the Borrower and cofinancing agency for comments but none were received. ' Report No. 12502, dated November 15, 1993.  kq :1 IJ 1~~ 0e 0< 1 - Iv - PROJECT DATES Date Planned Actual Date Identification 03/86 05/86 Preparation 06/86 07/86 Appraisal Mission 10/86 10/86 Credit Negotiation 02/87 02-03/87 Board Approval 02/87 04/87 Credit Signature 03/87 05/87 Credit Effectiveness 06/87 07/87 Project Completion 12/92 06/91 Credit Closing 06/93 06/93 STAFF INPUTS (staffweeks) FY87 FY88 FY89 FY90 FY91 FY92 FY93 Total Preappraisal 14.6 14.6 Appraisal 44.8 44.8 Negotiations 8.2 8.2 Supervision 4.0 29.2 34.9 15.9 11.5 8.2 0.4 104.1 Total 71.6 29.2 34.9 15.9 11.5 8.2 0.4 171.7 MISSION DATA Mission Date No. of Persons Appraisal 10/86 4 Supervision 1 10/87 3 Supervision 2 02/88 2 Supervision 3 06-07/88 2 Supervision 4 03-04/89 4 Supervision 5 09/89 5 Supervision 6 03/90 5 Supervision 7 10-11/90 2 Supervision 8 01/91 2 OTHER PROJECT DATA RELATED BANK CREDITS Borrower: Republic of Ghana hRt Crdt No. Amemat Bwd Public Enterprise 1847-GH US$10.50 million 10/13/87 Export Rehabiliation T.A. 1436-OH US$17.10 million 01/03/84 Economic Management 2224-GH US$15.00 million 03/28/91 Support Structural Adjustment 1777-GH/AO25-0-GH/AO25-1-GH US$130.00 million 04/14/87 Second Structural Adjustment 2005-OH/2005-1-GH/2005-2-OH US$134.00 million 04/18/89 SAIS: 590 SUPERVISION REPORT RATINGS Eval. Year Overall Dev. Obj. Leg. Cov. Mgt. Av. ftnds Peif. 1987 2 2 3 1 1988 2 2 2 1 1989 2 2 1 3 1 1990 2 1 2 3 1 1991 2 1 2 3 1 1992 2 2 2 3 1  - Vii - PERFORMANCE AUDIT REPORT GHANA STRUCTURAL ADJUSTMENT INSTITUTIONAL SUPPORT PROJECT (Credit 1778-GH) EVALUATION SUMMARY Introduction Project Objectives and Design 1. The SAIS project was prepared in 1987, 4. The project's immediate goal was to when the Government of Ghana was into the facilitate implementation of the ERP/SAP; its fourth year of its ambitious Economic Recovery longer-term goal was to rebuild the Program (ERP). The ERP had been launched Government's capacity to efficiently manage the in 1983, following two decades of political economy. The project had the following seven instability and economic decline. The economy specific objectives/components: at that time had virtually collapsed. Neither the country's infrastructure nor its institutions were * establishing a policy advisory staff under functional. the (then Secretary, now Minister) of MFEP; the unit was initially called the 2. During the mid-1980s the Government Economic Policy Unit; its name has since and Bank became increasingly aware that been changed to Policy Analysis Division Ghana's weak economic management (PAD); institutions, its insufficient policy analysis capabilities and its shortages of qualified 0 creating an Economic Liaison Unit manpower (the last partly owing to a substantial (ELU), within the office serving the brain drain) were serious constraints to the head of state and government, to process Government's capacity to implement its economic policy decisions and aid ambitious and far-reaching reform program. agreements; 3. Against this background, the G strengthening MFEP's capacity to Government and the Bank jointly undertook to manage public investment programming, make public sector management reform a key budgeting, expenditure forecasting and goal in principle if not necessarily in practice. control, revenue collection, debt The Structural Adjustment Institutional Support management, and aid coordination; (SAIS) Project was, as its name implies, a technical assistance (TA) project that executing staffing and functional reviews accompanied and complemented the Bank's first of the civil service and a related salary Structural Adjustment Credit (SAC I). Both policy review, and establishment of a projects were presented to the Board on the sound administration and personnel same day: April 14, 1987. management system; - vii - * design of a redeployment program and a participation by Ghanaians in its identification framework for co-financing and preparation, and consequently a low degree redeployment activities; of "ownership" of the project by its implementing agencies. * studies and advisory services in support of critical SAP-related tasks, where 9. In contrast to the policy reforms of the possible by mobilizing skilled Ghanaians ERP, where there was a high degree of from outside the public service (this ownership by the political leadership and top- component became known as the Skills level technocrats, the SAIS project did not have Mobilization Scheme, or SMS); and the same kinds of supportive "champions" and constituencies that made for substantial * implementing a living standards commitment to the project's objectives. This measurement survey. was partly because of a heritage of mutual mistrust between the political leadership and the 5. Although the project was perhaps not as civil service, and partly because of the timely as it might have been, the PAR finds that management styles of those overseeing the its objectives were highly relevant and project. appropriate to the country's circumstances as of the mid-1980s, when the project was identified, 10. Process Approach. The project's designers prepared, and advanced. were realistic enough to appreciate that (i) as mentioned, some project components were 6. ODA Cofinancing. The two components "under-designed"; and (ii) that they could not involving civil service reform were financed and pretend to predict the form or timing of various managed by ODA in a parallel-financing contingencies that might arise that would call for arrangement. ODA conducted its own either coping mechanisms or ongoing project re- evaluation of its support to the Ghana Civil design. They therefore adopted, quite Service Programme. The main findings and appropriately, as a main modality for risk conclusions of that evaluation are reported in management (as well as for project management Box 4 of the PAR. The ODA evaluation found in general) the Annual Work Program "as a performance to be mixed: it was adequate on planning, implementation and monitoring tool the diagnostic side but disappointingly slow on [to] provide the necessary flexibility to ensure the implementation side. adjustment of project activities to current circumstances." In retrospect it may be seen that 7. Project Characteristics. The project was more attention should have been given to ambitious, complex, demanding, multi-faceted, specifying the instruments needed to make the and in some areas politically sensitive. Its design "process approach" work well. incorporated both "blueprint" and "process approach" elements. Some elements were quite Implementation and Outcome innovative, e.g. the SMS, which inter alia was intended to maximize the use of trained and 11. The PCR found the Project to have been experienced Ghanaians at home and abroad--and implemented "for the most part as designed." by so doing minimize the need to rely on The PAR finds instead that some key expatriate advisers. Partly owing to the rushed components of the project were not preparation of the project, however, this and implemented and that the actual instrument mix other components were still "under-designed" at was very different from the one lanned. the time of Board approval. The risk assessment at appraisal was also inadequate. 12. The PCR and PAR both provide explicit ratings of the "efficacy" of the project's main 8. Ownership. One of the main shortcomings components, where efficacy is defined on the of the SAIS project was that there was little extent to which project objectives were achieved. Unlike the PCR, the PAR calculates its overall other aspects of the "institutional environment" rating as a weighted average and provides also a or the "institutional culture." Among the (weighted) rating of the project's cost- characteristics of this environment are non- effectiveness. transparent patronage relationships and varieties of donor-financed salary supplements that have 13. With regard to the components, the Skills created resentments, rent-seeking and other Mobilization Scheme and Economic Liaison Unit behaviors not conducive to long-term ID. were non-starters that were soon abandoned, while the poverty monitoring function (under 17. Taking all these considerations together-- the GSS) was soon shifted to the Social relevance, efficacy, cost-effectiveness and Dimensions of Adjustment project. exteralities--the conclusion is drawn that the Implementation of the component to strengthen project's outcome was on-the-whole the Economic Policy Unit (subsequently called unsatisfactory. The fairly low efficacy and cost- the Policy Analysis Division) proceeded effectiveness assessments alone support this haltingly. conclusion, with the "externality" considerations being additional. Since the project objectives 14. Mainly owing to the non-performance of themselves gave considerable weight to the ID the above components, the monies thereby objectives, it follows that the ID impact was at "freed up" were spent largely on equipment for best modest. the remaining components. The story of what happened to the SAIS project funds is 18. The sustainability of benefits from the SAIS summarized by the fact that planned equipment project will require: (i) greater ownership of the costs comprised 31 percent of the total ID and capacity building objectives; (ii) (including prorated unallocated expenditures) developin and implementing a coherent strategy while actual equipment expenditures comprised for ID/CB; (iii) accelerating the implementation 70 percent. of civil service reform, including not only retrenchment but also (and especially) measures Evaluation and Sustainabiity to deal with systemic problems that adversely affect civil service morale and productivity; and 15. The project did contribute to SAC (iv) accelerating private sector growth upon implementation. Yet there were major which the financial and foreign exchange shortcomings in both the extent to which original resources of the government ultimately depend. project objectives were achieved (efficacy) and Some progress is being made, but there is a need in cost-effectiveness (efficiency). During the for accelerated actions on all fronts to ensure course of project implementation, some of the the sustainability of the growth process in Ghana most important ID and capacity building (CB) in general, and the ID process in particular. objectives were dropped, shelved, or in any The sustainability rating is therefore "uncertain.* event not implemented satisfactorily. The PAR concludes also that the "externalities" of the Borrower and Bank Performance project (i.e., project impacts, in terms of costs and benefits, not captured in the efficacy and 19. Borrower and Bank performance were both cost-effectiveness assessments) were on balance found to be wanting in several respects. The negative. main strengths and weaknesses of both the Borrower and the Bank's performance are 16. The negative externalities include distorting described in Box 1. Te main problems on the effects that the SAIS project--taken together Borrower side were weak ownership, loose with other TA projects of the Bank and other management (which gave rein to the pressures donors--have had on priority setting (budgetary to use the project funds mainly for equipment) and otherwise), on incentive structures in the and marginal compliance with some covenants, labor market for trained Ghanaians, and on as reflected in problems with procurement and transfer project monies from true ID purposes to record-keeping. equipment purchases. Because this happened in such piecemeal fashion and was not flagged by 20. The main problem on the Bank's side was supervision reports as representing the project's that the Bank did not establish strong discipline "going off track," the project did not become a in terms of process, in order to ensure constant "problem project," as it should have done. There reassessment of progress towards the most should have been a stock-taking and a more important long-term objectives. A proper choice thorough project re-design via a mid-term review of performance indicators could have gone a or the equivalent. long way towards this end. There was adequate compliance with OMS 4.00 (the OMS on TA Sunmary of Projed Strengths and Weaknesses that was in effect between 1984 and 1992), but this was not reflected in the adequacy of risk 21. Box 1 provides an overview of the main management. In the event, the Bank was strengths and weaknesses of the SAI project. excessively acquiescent to Borrower pressures to Box 1: Main Project Strengths and Weaknesses spriinrprsaersMain Strengths and Main Wenknet hpser Desi"g Appropriate/relevant objectives Lack of Government *Government commitment to overall participation/ownership/commitment to ERP/SAP project objectives Good Bank/Government overall Late and rushed preparation relations Lack of country knowledge and Strong technical qualifications of operational experience by appraisal appraisal team team Some innovative features of design g Process approach was appropriate but "Process" (versus blueprint) approach the process was under-designed BiLittle reliance on expatriate long-term Scattershot approach to ID; no ID advisers strategy Considerable use of local consultants .Efficiency aspects of civil service reform neglected by Bank Lack of attention to political, social, cultural parameters GoWeak risk assessment tNeglect of institutional eexternalities and "environmental" aspects; incentive myopia Lack of monitorable performance indicators - xi - Main Strengths and Mair Weaknesses Outcome National Revenue Secretariat ID/CB objectives were largely strengthened abandoned Design of civil service reform was Most project funds went for equipment: advanced a "craze for equipment" Some civil service management/review Little progress in strengthening key functions were strengthened MFEP functions Cocoa pricing study was useful Project contributed to distortions of Equipment inputs had some incentive structure, including through productivity/morale benefits rent-seeking Annual work programs were widely Too much use of ad hoc groups, quick adopted fixes *Adequate Bank/ODA coordination Short-term "output" objectives, often driven by donor objectives, were sometimes achieved at expense of long- term ID objectives Management EMS project had mid-term review Loose project management by the Slwerrision National Capacity Building Initiative is Borrower FoUlow-up underway Too-permissive and too-passive supervision by the Bank *No mid-term or other high-level reviews by Bank or Government *Lack of donor coordination in TA; NATCAP process ineffective *Weaknesses in procurement, disbursement, monitoring, reporting n"Halo effect" may have led Bank to underestimate SAIS shortcomings Lessons of experience not applied to the design of EMS project Capacity building study lacks priorities; not a strategy Lessons Corollary: The "looser" and more process-oriented is a TA project's design, 22. This project confirms several lessons of the more the need to build an efficient experience recently codified in the TA monitoring process into the design itself. Handbook and provides a few others not yet entered into the lexicon of TA best practice. 0 Externalities matter. And they may These lessons are as follows: matter a lot. Much more explicit attention needs to be given to the * The central challenge for operations like "1environmental" aspects of TA projects the SAIS project is to establish strong (where the environment is here defined discipline in terms of process, in order to as the "institutional environment"). ensure a constant reassessment of issuesG and progress toward lonp-term Corollaiy: Consideration should be given objectives. to doing a form of "institutional - xi - environment impact assessment," focussing upon where it belongs, rewarding those who "cultural" and behavioral dimensions of ID and take the "right" risks (including the risk systemic issues such as how TA gives rise to new of Borrower displeasure and criticism forms of rent-seeking through patronage when the Bank "just says no"). relationships when the incentive system is distorted. * TA aid coordination (i.e., the lack thereof) is a problem in Ghana that * There are no substitutes for ownership needs to be recognized and addressed by on the Borrower's side and for in-depth new means. The variant donor policies country knowledge on the Bank side. and practices regarding salary Bank staff who appraise and design TA supplements, for example, require projects need to know how local rationalization as a matter of urgency. institutions really work and how political, The NATCAP objectives are important, cultural and personal factors must be but the NATCAP process is not "factored in" to project design and working. The Bank should help the supervision. Government to take the lead in getting donors to jointly pursue policies and Corollary: Borrower ownership of ID practices that will minimize the negative objectives must be established before externalities and improve the appraisal, and only mature, experienced, developmental effectiveness of TA. It is and reasonably senior staff should be recommended that a local aid assigned to appraise and supervise IDTA coordination group for TA be organized projects, especially in situations where and launch a study on how to improve, rent-seeking is rife. collectively, the developmental effectiveness of TA in Ghana. * The importance of having local "champions" should not be * Improve the performance indicators and underestimated. Participation and focus them more on quality, process, and ownership by "steering committees" and behaviors. Disbursement rates may be advisory counterpart groups are no highly misleading indicators of project substitutes for individual performance and absorptive capacity. leaders/managers who know what TA More attention needs to be paid to they want and know how to use it. A monitoring the quality of TA outputs of risk that TA may be too ad hominem all types. Most importantly, the may be a risk worth taking. indicators should not measure just short- term outputs of studies, documents, * Taking a long-term, strategic approach meetings, etc. They should aim at to TA implies not letting the Bank/donor monitoring whether processes and agendas, work programs, and incentives learning and other desired behaviors are "drive" project identification, design and changing in the desired directions, and implementation. It also implies the are improving in quality. Supervision needs to be patient, to help the reports should explain clearly and Borrower learn by doing and by being succinctly, up front, whether the project accountable, to accept delays and some is on track to have its intended flops as costs of keeping accountability developmental impact. PERFORMANCE AUDIT REPORT GHANA STRUCTURAL ADJUSTMENT INSTITUTIONAL SUPPORT (SAIS) PROJECT (Credit 1778-GH) I. INTRODUCTION/BACKGROUND 1. This introduction describes briefly (i) the project's background and origins; and (ii) how the coverage and some findings of this PAR are in several respects different from those of the PCR. Background 2. The SAIS project was prepared in 1987, when the Government of Ghana was into the fourth year of its ambitious Economic Recovery Program (ERP). The ERP had been launched in 1983, following two decades of political instability and economic decline. The economy at that time had virtually collapsed. Neither the country's infrastructure nor its institutions were functional. 3. During the initial years of the reform program, emphasis was on "getting the prices right," on reducing severe imbalances in government finances, on restraining credit expansion and on rehabilitating the rundown infrastructure--to get the economy moving again. 4. The policy package included successive devaluations of the exchange rate, the lifting of domestic price controls, sizable increases in real producer prices for cocoa and other agricultural export crops, the mobilization of government revenue through a broadening of the tax bases and a strengthening of tax administration, and tightened controls over govt expenditure. The IMF took the lead in setting the "hard" conditionalities. 5. The Bank's lending program during the mid-1980s consisted mainly of a series of low- conditionality, rehabilitation-oriented operations. One factor underlying this strategy was the recognition that there was a low absorptive capacity on the Ghanaian side for managing a wide(r)- ranging policy reform. There was also a demonstrated lack of confidence by the political leadership in the civil service. 6. During the years 1986-87, the ERP/Structural Adjustment Program (SAP)Y was passing into a second stage of broadened reforms including a comprehensive liberalization of the exchange and trade system, a divestiture program for the state enterprise sector, civil service reform to reduce the size of the civil service and increase remuneration levels, further tax reform, institutional and financial reforms to strengthen the domestic banking system, and substantial increases in government investment outlays for rehabilitation of the economic and social infrastructure. This second stage, it will be seen, called for considerable more institutional capacity than the first stage. 1' It is perhaps significant that Ghanaians continue to refer to the reform program as "their ERP," as distinguished from a SAP that connotes origins and ownership more in the Bank and IMF. -2- Origins of the Project 7. Thus, by the mid-1980s, the Government and Bank were both aware that weak economic management institutions, insufficient policy analysis capabilities and shortages of qualified manpower (the last partly owing to a substantial brain drain) had contributed to past inadequate performance, and would limit the Government's capacity to implement the ambitious and far-reaching reform program described above. 8. Neither the Bank nor the Government had articulated an institutional development (ID) strategy or action plan by that time (or since), but some Bank reports and the ensuing policy dialogue had begun to focus on the needs for rehabilitating institutions and building capacity. 9. The Bank's 1985 Country Economic Memorandum, for example, noted that "in the short term, the highest priority must be strengthening the capability of the Ministry of Finance and Economic Planning (MFEP) to fulfill its crucial role as initiator and overseer of the development process. And the Bank's 1986 Country Program Paper (CPP) gave prominence to the urgent need for the Bank to help the Government "tackle the severe administrative and technical manpower constraints, initially through a focus on high-priority economic management and planning functions." That CPP went on to propose actions on most of the fronts that the SAIS project would subsequently focus upon. This was how and where the project was first identified. 10. Against this background, the Government and the Bank jointly undertook to make public sector management reform a key goal in principle if not necessarily in practice. The Structural Adjustment Institutional Support (SAIS) Project was, as its name implies, a technical assistance (TA) project that accompanied and complemented the Bank's first Structural Adjustment Credit (SAC 1). Both projects were presented to the Board on the same day: April 14, 1987. How the PAR Differs from the PCR in its Coverage 11. The coverage as well as some findings and ratings reported in this PAR differ significantly from those in the PCR. There is however considerable agreement between the PCR and PAR regarding the outcomes of most components of the project, and hence regarding the extent to which project objectives were achieved. 12. The DGO's Memorandum of Transmittal to the Board noted that the PCR provided an informative and seemingly accurate description of project objectives and of the outcomes. But it was also stated that the PCR was not very "evaluative," and that some of its lessons (e.g., the need for a "clearer understanding of the socio-political framework" on the project designers and managers), were not sufficiently explained and/or supported by analysis. 13. This PAR therefore seeks both to be more evaluative and to deal with issues not much touched upon in the PCR. For example, the PAR addresses more explicitly than the PCR, such questions as: How relevant/important were the project objectives? To what extent was the project "owned" by the beneficiaries and stakeholders? How adequate was the risk assessment and management? How cost-effective was the implementation? And what were the key systemic problems (for example in the government and donors' incentive structures) that led to unanticipated (and in some cases counterproductive) outcomes? -3- 14. In a widely quoted speech given last year, Mr. E.V.K. (Kim) Jaycox stated that TA in Africa sometimes undermines rather than supports long-term institutional development (ID) and capacity building (CB). This PAR addresses the question of whether that criticism applies to the SAIS project. 15. Other questions addressed in the PAR (that were not much treated in the PCR) are: * What were the main strengths and successes of SAIS upon which future operations can build? And what were the main weaknesses and shortcomings that need to be avoided or remedied in future? * What is the status of Ghana's ID strategy? Is there in place, or about to be in place, a well-prioritized ID strategy to guide the TA activities of the Bank and other donors? * Are there problems of aid coordination that hinder the effectiveness of TA in general in Ghana (e.g., variant donor policies regarding the payment of salary supplement to public servants)? * Is TA fostering a dependency syndrome? Rent-seeking? * Were the lessons of experience derived from the SAIS project applied to the design of the follow-up project (i.e., the Economic Management Support Project)? How the PAR Differs from the PCR in its Conclusions 16. The PCR found that "the project was implemented for the most part as designed." The PAR finds otherwise. The PCR found the "overall results" of the project to be satisfactory. The audit finds the outcome to have been, on balance, unsatisfactory. The PCR and PAR agree that the ID impact was partial/modest. The PCR was unclear regarding the sustainability of the project's benefits, although the overall impression given was that sustainability is likely. The PAR finds the sustainability to be uncertain. Finally, with regard to implementation performance, the PCR found Borrower performance on project management to be unsatisfactory while Bank supervision performance was found to be satisfactory. The PAR finds both to have been unsatisfactory. The reasons for these differences are explained throughout the course of this report. -4- II. PROJECT OBJECTIVES AND DESIGN 17. This section provides a largely descriptive overview of the project's objectives and design. The coverage is selective, with emphasis on information deemed necessary as background to the evaluation. The more evaluative parts relate mainly to the relevance and appropriateness of the objectives, the project's timeliness, and the extent to which there was ownership by the main stakeholders and implementing agencies. (The PCR's evaluation of these particular attributes was minimal). 18. The PAR also evaluates other aspects of the project's design, for example: the appropriateness of the "process" approach vis-2-vis a "blueprint" approach; the adequacy of the risk assessment, and the Borrower's and Bank's performance in the upstream stages of the project cycle: identification, preparation, and appraisal, all with a view to assessing the project's "quality at entry." The evaluation of these aspects of project design is presented following the presentation of the project's outcomes (provided in the next section); this facilitates the attribution of success and failure as between design and implementation factors. Project Objectives 19. The project's origins and relation to the country strategy were described briefly in the Introduction. The project's immediate goal was to facilitate implementation of the ERP/SAP; its long-term goal was to rebuild the Government's capacity to efficiently manage the economy. 20. The project was planned as a seven-year undertaking costing a total of US$14.1 million, of which the Bank committed US$10.8 million, the Government US$2.2 million, and the Overseas Development Administration (ODA) of the United Kingdom, US$1.1 million. 21. The project had the following seven specific objectives: (i) establishing a policy advisory staff under the (then Secretary, now Minister) of MFEP; the unit was initially called the Economic Policy Unit; its name has since been changed to Policy Analysis Division (PAD); (ii) creating an Economic Liaison Unit (ELU), within the office serving the head of state and government, to process economic policy decisions and aid agreements; (iii) strengthening MFEP's capacity to manage public investment programming, budgeting, expenditure forecasting and control, revenue collection, debt management, and aid coordination; (iv) executing staffing and functional reviews of the civil service and a related salary policy review, and establishment of a sound administration and personnel management system; (v) design of a redeployment program and a framework for co-financing redeployment activities; -5- (vi) studies and advisory services in support of critical SAP-related tasks, where possible by mobilizing skilled Ghanaians from outside the public service (this component became known as the Skills Mobilization Scheme, or SMS); and (vii) implementing a living standards measurement survey. 22. These goals/objectives were not explicitly revised over the project's life. Some of the means were changed substantially, but the ends remained substantially unchanged through the project's life. The follow-up operation, the Economic Management Support (EMS) Proiect, approved in 1991, retained essentially the same objectives as the SAIS project. 23. The seven main objectives of the SAIS project were translated into many more than seven components, because some of the above-listed activities were multi-faceted and involved separate implementing units. Cofinancing by ODA 24. Components (iv) and (v) as listed in para. 21 were financed and managed by ODA in a parallel-financing arrangement. ODA provided its own consultants and its own project managers to administer activities relating to the wider Civil Service Reform Program. The SAIS project constituted an "umbrella" for the ODA project, which had additional components not under the SAIS umbrella. Given that the ODA components were administered virtually as a bilateral project, the evaluation will distinguish between the Bank's performance and ODA's performance in the evaluation section. 25. ODA has in fact already conducted its own evaluation of the ODA Project in Support of the Ghana Civil Service Programme. The main findings and conclusions of that ODA evaluation are reported in Box 4. Salient Project Characteristics 26. The project was ambitious, complex, demanding, multi-faceted, and in some areas politically sensitive. Its design incorporated both "blueprint" and "process approach" elements. Some elements were quite innovative, e.g. the SMS, which inter alia was intended maximize the use of trained and experienced Ghanaians at home and abroad--and by so doing minimize the need to rely on expatriate advisers. 27. The project was geared to be implemented on the basis of annual work programs, prepared by each implementing agency, coordinated and reviewed by the Project Management Unit (PMU), and approved by the SAP Team (of high level Government officials) and IDA. These annual work programs were intended to serve as combination "planning, implementation and monitoring tools that would provide the necessary flexibility to ensure adjustment of project activities to current circumstances." The modalities included all the familiar instruments of Bank TA: short-term consultants and long-term advisers, a study fund, equipment and 0 & M allocations, and training components. 28. The PMU was an agency called the Structural Adjustment Program Secretariat (SAPSEC), whose role was to manage and monitor the activities of the main implementing units in -6- MFEP, the National Revenue Secretariat (NRS), the Office of the Head of the Civil Service (OHCS), the Ghana Statistical Service, and the Central Committee of Secretaries (in effect the cabinet). The PMU had a complex history, elements of which will be mentioned in the evaluation of Borrower and Bank performance. ' Expertise and Experience of the Appraisal Team 29. The Bank team that appraised the project included several Bank staff members widely acknowledged to have exceptional expertise in ID. They were regarded by their colleagues in the Bank as among the "brightest and the best" ID experts; they were reportedly well regarded also by their Ghanaian counterparts. The SAIS Staff Appraisal Report (SAR) was regarded as an impressive document (including by several speakers at the Board meeting that approved the credit), and it became for a time a model emulated by those preparing TA projects in other countries, not only in Africa. 30. One surprising fact about the appraisal team is that none had prior knowledge of or experience in Ghana, nor did any of the team have operational field experience in the actual design and implementation of TA operations. Thus, the appraisal team was strong on ID expertise but weak on country and operational experience. Relevance of the Objectives: Evaluation 31. It is the judgment of this PAR that the project's objectives, as perceived and defined in the SAR--and as summarized above--were highly relevant and appropriate to the country's circumstances as of the mid-1980s when the project was identified, prepared and appraised. 32. As has been mentioned, the country was then in a stabilization, recovery, and rehabilitation mode economywide. The institutional, as well as the physical infrastructure, was in disarray and disrepair, and in need of urgent measures to make it functional. Nowhere was this need more critical than in the core economic agencies (especially MFEP) that were most responsible for formulating and implementing economic policies in pursuit of the stabilization and adjustment objectives. Neither the typewriters nor the budget were functional. Both needed to be fixed, and with some urgency. 33. In this context, it was necessary and appropriate that some of the project's key components were directed at meeting short-term needs to improve the management of the budget, the debt, the public investment program, the tax administration, and so forth. Some office equipment was also clearly needed. But it was equally important to recognize--as was done in the SAIS project-- that ID is a long-haul objective requiring that short-term measures to raise capacity utilization be accompanied by measures geared to build capacity for the longer-run. -' A separate study should be made of the history and experience of PMUs (including those managing projects of other donors) in Ghana over the past decade. The lessons to be learned from such an evaluation could have an important bearing on the ID/CB strategy currently being formulated under the direction of a National Capacity Building Steering Committee, on which committee the Bank is represented. -7- 34. This second objective entailed a recognition of the need to restructure the civil service and other public sector institutions, to make them more relevant to, and supportive of, the new, more market-friendly, strategy for national development. The project objective to assess the social impact of the reforms, with the living standards measurement survey (LSMS) as a principal tool, was also extremely relevant, as this would facilitate the design of measures to alleviate social costs of adjustment in the short term, and the formulation and implementation of a more poverty-oriented development strategy in the longer term. TYmeliness of the Project: Evaluation 35. The relevance of a project's objectives and the timeliness of its undertaking are closely related. In the case of the SAIS project, it would seem that the project could and should have been mounted earlier than in 1987 well into the fourth year of Ghana's implementation of its ERP. 36. This is not to say that it was untimely in 1987, or that opportunities were lost owing to the passage of time. Yet it was evident all along that Ghana's public administration in the early- and mid-1980s was non-functional (if not dysfunctional), and that major institutional reforms and efforts to build capacity and improve capacity utilization would be required if the overall reform program were to be deepened and sustained over time.2' 37. The conventional wisdom (best practice) about the sequencing of reform was the same in the mid-1980s as it is today with regard to when to undertake ID, including through TA operations: start from the beginning, and continue throughout. 38. What determined the timing of the SAIS project was mainly the practice (common in the mid-1980s) of linking lending for ID in public sector management to Bank adjustment operations, mainly SAls/SACs. Thus it happened that, because the Bank's first SAC did not occur until 1987, the SAIS project was delayed until 1987. Indeed, the SAIS project projections had to be rushed in late 1986 and early 1987 in order to go to the Board with, rather than after, SAC !y Participation and Ownership: Evaluation 39. One of the main shortcomings of the SAIS project was that there was little participation by Ghanaians in its identification and preparation, and consequently a low degree of "ownership" of the project by its implementing agencies. 1' The 1981 CPP remarked the disarray and demoralization of the Government administration and public sector bureaucracy that prevailed in 1980, a situation that was inherited by the new government team that launched the ERP in 1983. In fact, a US$5 million public administration (TA) credit was appraised in 1980 but did not go forward owing to the general failure of the (then) government to pursue reform. But this indicates that some "thinking was already on the shelf" regarding what needed to be done to help the government with TA for public sector management once the Bank was persuaded that sufficient commitment to reform was present. Y Nunberg found this to be a common phenomenon, i.e. that the schedules of TA projects were often driven by SAL timing requirements, leading to inadequate preparation of TA projects. Nunberg also identifies a "dissonance" between SAL schedules and the longer-term horizons required for ID that constrains reform efforts in many countries. As will be seen in the evaluation section, this was very much a problem in Ghana. See B. Nunberg, "Public Sector Management Issues in Structural Adjustment Lending," WB Discussion Paper No. 99, 1990. -8- 40. In contrast to the policy reforms of the ERP, where it is generally acknowledged that there was a high degree of ownership by the political leadership and top-level technocrats, the SAIS project did not have the same kinds of supportive "champions" and constituencies that made for substantial ownership and commitment to the project's objectives. This was partly because of a heritage of mutual mistrust between the political leadership and the civil service, and partly because of the personalities and management styles of those overseeing the project. 41. To be sure, some components were identified by the Government (e.g., the needs for equipment, housing, and salary supplement and other elements of the SMS). But the perceived and felt needs for other important components--such as the Economic Liaison Unit, the Policy Analysis Division and the study on cocoa pricing--were much more the Bank's than the Government's, and indeed the Government accepted some of these components reluctantly, with little conviction of their necessity and utility. 42. This PAR's assessment of the ownership of the SAIS project (applying criteria developed initially in OED to evaluate borrower ownership of adjustment operations) rates the SAIS project as "3" (on a scale of 1-4, with 1 being highest and 4 lowest) in terms of the following "dimensions" of ownership: * "locus of initiative" (largely in the Bank); * "intellectual commitment" (tactical rather than strategic); * "consensus building" (there was considerable disagreement amongst stakeholders and implementing agencies about what they wanted the project to accomplish, and how, and where). Risk Assessment at Appraisal 43. The President's Report and SAR identified three project risks, namely: (i) dependence on the few Government officials who constitute(d) the core economic team; (ii) the (then-proposed) split of MFEP into separate ministries of Finance and of Planning; and (iii) political difficulties in carrying out the civil service reform. 44. Neither the SAR nor the project files indicate that much attention was given to the "over-dependence on a few" risk; this seems to be have taken as a "given," with a presumption that the Bank couldn't or shouldn't attempt to do much to broaden ownership. 45. The risk that the MFEP might be split (as recommended by a UNDP-financed team from Hungary, giving rise to disagreements between the Bank and UNDP) was proposed to be minimized by requiring (in a covenant) that IDA be "consulted" before this proposed change were made. The risk of political resistance to civil service reform was intended to be managed by having the Government "set aside" budgetary resources to fund the initial retrenchment and by committing IDA to help raise funds from other donors to fund the retrenchment. 46. Finally, it was asserted that "the government's management weakness would be mitigated by the streamlined project organization that has been established [sic] and by providing external assistance." -9- 47. The project files reveal that the nature and extent of these risks was given more attention than is suggested by the perfunctory treatment in the SAR, but not much more. The adequacy of this risk assessment and the efficacy of risk management during implementation will also be revisited in the evaluation. Annual Work Programs and the "Process Approach" 48. Although the SAR did not explicitly identify as instruments of risk management either the "process approach" allowed for in project design or the unit-specific annual work program, these were de facto the main instruments--more so than the specific project conditionalities. 49. The project designers were realistic enough to appreciate that (i) some project components were still "under-designed" at the time of appraisal; and (ii) that they could not pretend to predict the form or timing of various contingencies that might arise that would call for either coping mechanisms or ongoing project re-design. They therefore adopted, quite appropriately, as a main modality for risk management (as well as for project resource management in general) the Annual Work Proaram "as a planning, implementation and monitoring tool [to] provide the necessary flexibility to ensure adjustment of project activities to current circumstances!' (underlining added). F SAR, page 22, para. 49. - 10 - III. PROJECT IMPLEMENTATION 50. The project became effective in July 1987, two-and-a-half months after signing. It was closed as scheduled on June 30, 1987, having been fully disbursed. In fact, disbursements proceeded throughout the project period more quickly than was projected at appraisal. 51. As mentioned in the Introduction, the PCR found the project to have been implemented "for the most part as designed." The PAR finds instead that some key components of the project were not implemented and that the actual instrument mix was very different from the one planned. The PAR agrees with most PCR findings regarding the implementation of individual project components. The reasons for this seeming anomaly are explained below. This section (i) reviews the main project outcomes and compares those actual outcomes to the project's original objectives and targets to arrive at an "efficacy rating" and (ii) assesses the project's cost-effectiveness, also to arrive at a rating. Rating Methodology 52. Given the multiplicity of project components, with different sizes and objectives, the framework for the efficacy rating is built upon a matrix included in the PCR as an Annex. This Annex, reproduced here as Table 1, compares systematically, component by component, project objectives with results, and provides a quantitative rating for the "efficacy" of each component. 53. The PAR builds upon this matrix in two principal ways: first, by adding a column showing the PAR's own ratings, with explanations in the text of why the PAR comes to different ratings, and second, by assigning explicit weights to the components and their respective objectives. This permits also the calculation of weighted average performance (efficacy) by each major component group. The results of this exercise are shown in Table 2. This approach is preferable to one that allows small and/or insignificant components to count for as much as the largest and/or most important components. 54. There is no established methodology for assigning the weights. One approach, and the one adopted here, is to weight the components according to the share of expenditure intended to be used for that component at the time of appraisal. This is a seemingly arbitrary and simplistic approach. But it makes particular sense in the case of the SAIS project because: (i) the initial project objectives were all deemed to be highly relevant (i.e. to correspond to country developmental priorities); (ii) the country conditions and developmental priorities did not themselves change significantly over the project period; (iii) the project objectives and components were not explicitly revised; and (iv) there seems to be no alternative impelling logic for weighting the components otherwise, whether by an objective or more subjective standard. Moreover, it is to emphasized that the final project ratings are "informed" rather than "determined" by the mechanical system, given that the system does not purport to capture all of the project's attributes or outcomes. 55. The cost-effectiveness rating methodology is problematic in the case of TA operations, given that a cost-effectiveness rating should measure the "developmental achievements" (benefits) expected from the operation in relation not only to the resources used up to closing, but including those expected to be used over the life of the project. - 11 - 56. Given the inherent measurement problems involved in doing this for an ID operation, the approach taken here was to make an (admittedly largely subjective) assessment of the "developmental impact" of the main types of expenditure (equipment, training, etc.) relative to their actual costs. The ratings derived from that assessment were then weighted by the share of the end- use categories in total project expenditure. (There was no PCR assessment of cost-effectiveness to build upon here). Lack of Data/Indicators on Project Outcomes 57. A major defect of the SAIS project was its lack of monitorable performance indicators to facilitate the assessment of project performance. The project management was also remiss in not keeping proper accounts and was lax in its monitoring project progress even on an ad hoc basis. For example, only one perfunctory small study was made, over the seven years of project implementation, to assess to efficacy/efficiency of training, and the lack of adequate monitoring of the huge outlays on equipment also bespoke an insufficient concern for ensuring the utility and efficiency of these outlays. 58. Moreover, when the OED evaluation mission visited Ghana, it was told that virtually all project documents had disappeared while the former head of the project's PMU (who is now in the private sector) refused repeatedly to meet the mission. Needless to say, these factors were handicaps to the OED evaluation, which therefore had to rely excessively upon Bank supervision reports, other Bank documents, and upon the recollections of those interviewed. 59. The OED mission did a few "spot checks," e.g. of turnover in the debt management division, of retention of trainees, and of utilization of vehicles and computers. But these fell far short of systematic surveys and gave rise more to impressions than to hard facts and numbers. Project Outcomes and Efficacy 60. Table 1 presents the project results and ratings as assessed by the PCR. Table 2 compresses the framework of the previous table and, as described above, assigns weights and PAR ratings that can be compared with the PCR ratings. 61. The PCR table (Table 1) arrived at an overall project rating of 3.5. This is in effect an "efficacy rating" in that it is a measure of the extent to which project objectives were achieved. It is unclear, however, how the overall rating was arrived at, since a simple unweighted average of the component ratings comes to 2.9 rather than 3.5 (see last row of Table 2). Table 2 shows that the PAR arrives at somewhat different ratings for several components, while its overall efficacy rating of 2.2 is a weighted average. 62. All of these measures need to be interpreted with caution. In the first place, they are numerical representations of largely qualitative judgments about project performance. Moreover, there is no objectively-defined "cutoff" rating above or below which the project's efficacy can be said to be satisfactory or not. What the PAR rating captures is the judgment that the weighted average outcome was closer to poor than to very good, and was therefore unsatisfactory. - 12 - Table 1 PROJECT RESULTS ACCORDING TO THE PCR Compoebt Objecdves Resuts Scorc Economic Policy Management EPU (MFEP) Rebuild MFEP's capacity to analyze The Unit was understaffed and did 1 policy issues and advise the Government not fully comply with the terms of on the appropriateness of policies. reference, mainly because the Government had no interest in the Unit. ELU (Prime Minister's Improve policy decision-making process. ELU did not comply with the terms 1 Office) of reference and had a short life. Budget Division, IPAD Establish systematic budget preparation Partial computerization of the budget 4 and PRD (MFEP) and monitoring procedures in the Budget process and improvements in budget Division. IPAD and PRD to assume full preparation permitted early responsibility for drawing up a rolling publication of the budget. PRD was three-year PIP. eliminated and IPAD strengthened. Rolling three-year PIP started in 1988. IERD (MFEP) Assume responsibility for debt CFTC consultants installed the 3 management. Maintain a data system for DRMS system in 1989. Link was project aid and disbursement information not done. and link it to debt management. NRS NRS was to return to MFEP. Improve ASYCUDA system was installed. A 4 the information base for income, customs local consulting firm developed a and excise taxes through computerization system for the sales tax and linked it and training of staff. with ASYCUDA. Public Service Productivity OHCS Strengthen the staff inspection function A requirements study for the 3 and personnel management information installation of the IPPD was systems. Join the administration of prepared by ODA-financed personnel with that of the payroll to consultants. In the meantime, an reinforce internal checks on the payroll. information system on manpower levels was started which helped keep recruitment under control. OHCS and Ministry of Carry out a staffing and functional review MSD of OHCS was strengthened 4 Mobilization and and establish a coherent redeployment and job inspections have contributed Productivity program. Reduce the size of the civil to a more productive civil service. service. Civil service was reduced according to targets, but little retraining of redeployees took place. - 13 - Componexs Objectves Resuf Scorvp Studies on salary Help formulate salary policy. A Pay and Grading report was 4 policy (OHCS PIB) produced by ODA-financed consultants in conjunction with PIB which was the basis for initiating the decompression of the salary structure and a regrading exercise. Structural Adjustment Studies and Advisory Services Studies Execute studies in support of the SAP. The Cocoa Pricing Policy study 5 induced the Government to take policy actions toward improving producer incomes through price incentives. Advisory Services Hire qualified consultants to work in The SMS was the scheme to be used 1 tasks related to the SAP. for this purpose, however it never really worked. The SMS facility was used only once. Poverty Monitoring Monitor the socio-economic impact of the Various impact studies were done. 3 SAP at the household level. The Statistical Abstract was used to determine minimum wages. The Statistical Service was reinforced. Training Strengthen economic institutions. Substantial overseas training took 2 place, mostly benefiting NRS. Some 17 training courses were offered locally, 12 of which in computer literacy. Except for a few, trainees were not effectively utilized. Overall Project 3.5 Rating:Z' Relative success of attaining objectives: 1 = poor; 2= mediocre; 3= average; 4= good; 5= very good. It is not clear how the PCR's "overall" rating was calculated, since no weights are indicated in the PCR, while an unweighted average of the above scores comes to 2.9 rather than 3.5. - 14 - Table 2 EFFICACY RATINGS BY MAIN COMPONENTS Efficacy Rating Weighted Score Weight PCR PAR PAR Max A. Economic Policy/Management Economic Policy Unit 16 1 1 16 80 Economic Liaison Unit 2 1 1 2 10 Public Invest. Planning 9 4 3 39 65 Budget/Expend. Control 4 Aid Management 1 3 3 9 15 Debt Management 2 Revenue Mobilization 10 4 4 40 50 Training 2 2 3 6 10 112 240 B. Public Sector Productivity Civil Service Mgmt. (OCHS) 6 3 3 18 30 Redeployment/Salary rev. 11 4 3 33 55 Salary Policy 1 4 3 3 5 54 90 C. SAC Studies and Advisory Services Studies 18 5 2 27 90 Advisory Services (SMS) 1 1 D. Poverty Monitoring!v 12 3-V 2-' 24 60 51 150 F. Proiect Mananement 6 n.a. 1 6 30 100 223 500 Weighted Average Rating - 2.2 Unweighted Average Rating 2.9 - In principle, this component should be "unrated", since (as is explained in para. 65 of the text) it was continued under another project and its outcome is therefore better evaluated in the context of that project. The ratings shown above, since they are virtually the same as the overall ratings, do not affect the overall ratings. - 15 - 63. The PCR itself had as a main conclusion that "little use was made of the project outputs," a finding with which the PAR concurs. But it is hard to reconcile this finding in particular with the PCR's overall conclusion that the project outcome was satisfactory. Main Points of PCR/PAR Agreement 64. Among the main findings upon which the PAR and PCR agree are that: * the Economic Policy Unit (now the PAD), the Economic Liaison Unit, and the SMS were non-starters, and remain so; * the TA (mainly training and computerization) in aid of strengthening the National Revenue Secretariat was efficacious (see Box 2); * TA in aid of the Civil Service Reform Program had mixed results; and * project management was poor. Main Points of PCR/PAR Disagreement 65. Table 2 shows that the PAR arrived at different findings and/or different conclusions about the efficacy of the SAIS project in several areas. The reasons for the main differences are as follows: Public Investment Planning and Budget/Expenditure Control. The PCR refers to the role of the project in "restoring credibility to the budget process" and it cites achievements in terms of studies, computerization, manuals, and accelerated publication of the budget. The PAR finds, however, that little progress was made in key areas (flagged also in the SAR) such as the relationship between the budgeting and expenditure control functions, and in expenditure monitoring and management generally. Many Bank staff and donors have cited their disappointment at the lack of progress on these important fronts. Training. The main reasons cited by the PCR for giving a low ("2')rating to the efficacy of training were that an excessive amount of overseas training took place and that most trainees were not effectively utilized. The PAR agrees with that assessment but observes that a substantial proportion of the overseas training, even though (too) expensive and often provided for the wrong reasons (e.g. as rewards rather than as investment in human capital), nevertheless had some positive effects on motivations and on private sector development. The PCR in effect merged a cost- effectiveness rating with its efficacy rating. The PAR gives a slightly higher efficacy rating to training than the PCR, but gives a low rating to the cost-effectiveness of the training program. The PCR and PAR agree that the training programs, whether for computer usage or debt management, were poorly managed and monitored. - 16 - Box 2 THE NATIONAL REVENUE SERVICE: A REFORM ENCLAVE? One development that was unplanned and unanticipated in the design of the SAIS project was that SAPSEC, the PMU for the project, would come under the supervision of the head of the National Revenue Service (NRS), the agency responsive for customs and internal revenue assessments and collection. This led, not surprisingly, to more attention and resources, especially for training, being devoted to the NRS than had been planned at appraisal. Although the training awards (mainly for training abroad) to NRS staff were made with insufficient objectivity, it does appear that the NRS became a relative success case. The PCR and PAR agree that while only a small part of the improvement in Ghana's tax administration can be credited to the SAIS project, this component was one of the more successful components of the project. Organizational changes, including improved remuneration and job conditions, lead to notable improvements in efficiency within NRS. This improvement is said to be explained by the following factors: Good management--an outstanding leader at the top; Management autonomy, notably in salary determination and in the right to use a share of tax collections for agency spending; A better incentive framework for employees. Base salaries were comparable to those in the civil service, but allowances (in cash and in kind) were much higher. External training opportunities were made available in abundance, as an incentive device; * Because of its budgetary autonomy, the National Revenue Service could offer much better working conditions than available elsewhere--supplies, office space, and the like; * Early in the program, the agency fired many excess workers and recruited better-skilled and better-motivated workers; and * The agency developed effective information-processing technologies. One lesson to be derived from this experience is that government workers in Ghana do respond with much-improved performance to a combination of good leadership, better pay, better working conditions, and satisfying work. A lesson drawn by Elliot Berg from the NRS experience is that it shows the workability of piecemeal reform, focussed on a few critical functions or agencies (such as the NRS), through the creation of "reform islands or enclaves." Such piecemeal reform, it is argued, is appropriate when general administrative reform and even comprehensive civil service reform face so many pitfalls that alternative tactics and strategies, based on more partial approaches, should receive greater attention. The SAIS audit did not investigate this issue sufficiently to take a position on it. - 17 - Civil Service Reform Proaram. The PAR rating for these components relies mainly on findings of a four-man ODA evaluation team (see Box 4 on page 26). The project supported sound technical work in several areas related to the civil service reform program, but progress made on the design side was not matched by progress in implementation. Studies. The PCR gives a rating of "5"to the efficacy of studies financed by the SAIS project, solely on the basis of the Cocoa Pricing Study. The PAR agrees that this study was efficacious (and cost-effective). But since the SAR clearly intended that a much more substantial program of studies would be financed by the project than was implemented, the PAR rates this component much lower. Poverty Monitoring. The main purpose of this component was to finance the Ghana Household Survey Program, the first phase of which comprised the Ghana Household Living Standards Measurement Survey. As the PCR acknowledges, however, the Social Dimensions of Adjustment (SDA) Project took over the financing and supervision of these components from the SAIS project at just about the time the SAIS project became effective. As described in the footnote to Table 2, this component should not therefore be rated. It will however be relevant to consider how well spent were the resources freed up by dropping this component from the SAIS project. 66. The preceding paragraphs and tables reviewed project outcomes by main component against the objectives as defined at appraisal. If an outcome rating were to assign only the basis of this review, it would have to be "unsatisfactory." But the efficacy assessment should not stop here, since the project was fully disbursed even though far less was spent on some of the original components than had been planned. And while the project objectives did not change significantly, the modalities changed a great deal during implementation. 67. The audit turns now, therefore, to the questions: Where did the money go? By what criteria was it redirected? Was it well spent? In other words, how should cost-effectiveness be rated? The Shift to Equipment Purchases 68. The answer to the first question is that the money went mainly for equipment: vehicles above all, but also for air-conditioning, furniture, computers, and office equipment such as typewriters and filing cabinets. Table 3 and Chart 1 tell a large part of the story, as they show the extent to which funds were redirected to equipment purchases. They show also a much larger share spent on training than was planned, reflecting in large part that a much higher share of training took place overseas than had been planned, at much higher unit costs. 69. The planned outlay on equipment was equivalent to 31 percent (if unallocated funds are prorated); the actual came to about 70 percent. Related to this was an increased share of spending on O&M costs. Conversely, far less was spent on consultant services (whether for Ghanaian nationals or expatriates) and studies than was originally intended. - 18 - Table 3 SAIS: PLANNED AND ACTUAL ALLOCATIONS (in SDR millions) Category Planned Actual Amount % Amount % (1) Consultants' Services and Studies 3.7 51 0.7 8 (a) for PMU (0.1) (2) (0.1) (1) (b) under SMS (0.8) (11) (0.0) (0) (c) other (2.8) (38) (0.5) (6) (2) Training 0.3 4 0.9 10 (3) Equipment, materials, supplies, housing 2.3 31 5.9 70 materials and vehicles (a) housing materials (0.2) (3) (0.2) (3) (b) other (2.0) (28) (5.7) (67) (4) Operating and maintenance costs of 0.4 6 0.7 8 vehicles and computers (5) Refund of PPF advance 0.6 8 0.3 3 Subtotal (not incl. unallocated) 7.3 100 8.6 100 (6) Unallocated 1.3 TOTAL 8.6 100 8.6 100 Note: Figures are rounded. 70. How and why did this happen? The answer is that the lack of progress and related spending on the PAD, ELU, and SMS components and the shift to SDA-financing of the poverty monitoring component "freed up" substantial funds, which then became available for spending on other components and for other uses. The project design allowed for considerable flexibility "to ensure adjustment of project activities to current circumstances," with the annual work program of each implementing unit being the main instrument by which the Government and Bank would negotiate which specific expenditures--whether on equipment or consultant services or whatever- would be financed by the SAIS project. 71. A covenant to the project required each executing unit's draft work program, prepared an annual, rolling basis, to include detailed investment and operating budgets, detailed staffing, training and other TA plans, and financing and procurement plans. The Bank supervision missions then had great freedom, within the limits of Bank regulations on procurement and such matters, to negotiate with the PMU and unit heads the uses to which SAIS funds would be put. - 19 - CHART I SAIS: PLANNED AND ACTUAL EXPENDITURE BY END-USE C%D Equipment PLANNED 51.0%) Ceo ACTUAL Consutants Services ACTUALTraining Equipment 30%) Operating and ma intenance ( 70. 7% Refund of PPF advances Equipment 72. It transpired that what the implementing agencies mostly wanted was equipment, and this is what they mostly got. In some cases vehicles and other equipment was provided to units not included in the SAIS project. In the absence of explicit de jure criteria regarding which expenditures could be justified, the de facto criterion was to authorize "whatever expenditures seemed needed to get reforms." In principle and in practice, this was a vague and general principle that permitted Bank staff to exercise a wide degree of discretion. It appears that, in practice, the Bank made rather generous interpretations of what requests for equipment (or external training) would qualify for SAIS support, especially when such requests were made, or endorsed, by high-ranking Government officials. But as several participants on both the Bank and Government sides observed, "the pressures were enormous." This question of whether the Bank was appropriately permissive in this matter will be taken up again in the evaluation. 73. Was the money well spent? This critical question is not easy to answer because detailed data and hard evidence on where the money went, for what purposes, and with what cst- effectiveness, are not available. The Government produced a table (Table 4) on the volume of the main components of equipment purchased, but this does not reveal anything about the benefits derived, the cost distribution, unit costs, end-users, or end-uses. Nor is it clear how these units accounted for the bulk of the US$6 million. 74. Given this lack of quantitative materials, the OED mission had to rely mainly upon information and impressions obtained from interviews and correspondence with those involved with the project on both the Government and Bank sides. This information was used to prepare cost- effectiveness ratings for each of the four main end-uses of the SAIS funds shown in Table 5. The ratings scale is the same as in Tables 1 and 2. - 20 - Table 4 MAIN COMPONENTS OF EQUIPMENT PURCHASED UNDER SAIS PROJECT Total 1987 1988 1989 1990 1991 Vehicles 102 36 5 48 13 0 Computers 100 37 18 3 13 29 Typewriters 345 36 10 43 256 0 Air conditioners 86 22 0 19 34 11 UPS and stabilizer 137 0 15 28 7 87 Duplicating machines 59 0 0 9 50 0 Source: Part II of PCR, based on Price Waterhouse report. Table 5 COST EFFECTIVENESS RATINGS BY END-USE Cost Effectiveness Rating Weight'-' Ratinge PAR Max Consultants Services and Studies 9 4 36 45 Training 11 2 22 55 Equipment 71 2 142 355 O&M Costs 9 3 27 45 227 500 Weighted Average Cost- 2.3 Effectiveness Rating Weight is share of total actual expenditure. Rating scale is same as in Tables I and 2. - 21 - 75. It appears that the cost-effectiveness of the consultants' services was high; that of the O&M costs, "average,"and of the training and equipment, "mediocre." The efficacy rating for training is higher than the cost-effectiveness rating. More training could have been provided locally, at much lower unit costs. As for the equipment, clearly much of it was needed and did serve to raise productivity. But a substantial proportion of equipment purchased also seems to be underutilized (especially computers) or only loosely related to project objectives (especially vehicles). Overall, the efficacy rating of 2.2 and the cost-effectiveness rating of 2.3 are close to each other. - 22 - IV. EVALUATION AND SUSTAINABILITY Outcome and ID Impact 76. OED's outcome ratings are based upon a composite of judgments about a project's relevance, efficacy and cost-effectiveness. The SAIS project was found to have had highly relevant and appropriate objectives, both with regard to supporting the implementation of the ERP and Bank adjustment operations and to fostering institutional development (ID) and capacity building (CB). 77. The review of project implementation found that there were major shortcomings in both the extent to which original project objectives were achieved (efficacy) and also in cost-effectiveness (efficiency). During the course of project implementation, some of the most important ID/CB objectives were dropped, shelved, or in any event not implemented. Especially disappointing were the failures to build a policy analysis capability within MFEP and to strengthen adequately several of MFEP's other key divisions. In the event, many of MFEP's key functions (e.g. preparing the Public Investment Plan) were performed by ad.ho task forces and working groups. This "task force approach" had its advantages but also disadvantages that were neither anticipated in the project design nor captured in the PCR evaluation. What Were the "Side Effects" on the "Institutional Environment"' 78. This last point raises the question--not previously raised in either the SAR or the PCR or previously in this PAR--of what indirect or "side effects", i.e. "externalities," were associated with the SAIS project. 79. Externalities are defined here in the conventional way,i.e.as project effects not captured and measured in project appraisals and evaluations of a project's net present value, rate of return, cost/benefit ratio or whatever measures of project impact or effectiveness is applied. In the case of the SAIS project, its externalities are defined to be those project effects not included in the assessment of outcomes, efficacy and cost-effectiveness as described in the preceding section on project implementation. 80. The external effects may also usefully be thought of as "environmental effects", where the relevant environment is not physical or ecological but rather an institutional, cultural and behavioral environment inhabited by a variety of political appointees, technocrats, bureaucrats, patrons, rent-seekers, academics, consultants, and donor agencies with their own cultures, agendas, institutional imperatives, and incentive structures. 81. In this context, then, it may be asked what effects, intended or unintended, direct or indirect, the SAIS project and others like it are having on this institutional environment, e.g. on the motivations, attitudes and behaviors of the various species inhabiting the environment, including those outside the units and agencies directly involved in the project. SAIS Project Externalities: Positive 82. In the SAIS project, it was found that introducing of annual work programs into implementing units had a demonstration effect that led to adoption of an "annual work program - 23 - culture" in other parts of the government. This was clearly a positive externality. So was the benefit of training (e.g. in computer skills and debt management) provided to civil servants who subsequently left government for the private sector. A third positive externality was whatever enhancement of morale and productivity (including private sector productivity) can be attributed to the equipment (vehicles, air conditioners, computers, etc.) distributed to agents or used for purposes outside the scope of the project. SAIS Project Externalities: Negative 83. The last example serves to highlight an import aspect of externalities associated with TA projects, namely that the same action (e.g. the distribution of equipment beyond project boundaries) may have at the same time both positive and negative external effects. For as was seen from in previous section, the story of SAIS was very much as story of how the "craze for equipment" in effect drove out the project's ID objectives. 84. The most serious effect of the change in project components--an effect "undermining" the public administration generally--was the impetus given to rent-seeking behavior, to behavior that finds rewards through patronage relationships. 85. The SAIS project, for example, helped to foster a system in which a substantial part of the regular work of government, especially in MFEP, whether preparing the public investment program or for a Consultative Group meeting, came to be undertaken by ad hoc task forces and short-term consultants rather than by the institutional units charged with these tasks. The task forces did facilitate getting jobs done (e.g. preparation of the public investment program) in more timely and efficient ways than would otherwise have been possible. At the same time, however, this de facto privatization may serve to postpone if not undermine the capacity of the institution to do its assigned tasks. And since members of the task forces, steering committees and advisory groups all get supplements of one kind or another, it is no wonder that this approach often prevails. And so it happened, as many donors and Ghanaians observe, that work on donor-financed projects--which in Ghana means most projects--does not get done without the payment of supplements of some sortY 86. The behaviors induced by these distortions are sometimes disabling, as in the case of public servants who became resentful and de-motivated because they cannot share in TA-financed benefits received by a favored few. The SMS clearly had this kind of adverse external effect. Yet when the value of a vehicle is worth several years of a Ghanaian civil servant's salary, and when a day's consulting fee may approximate a month's salary, it is no wonder that so much time and effort are spent in pursuit of fees and favors dispensed through the interlocking Government/donor patronage systems.2' This has opportunity costs, largely unmeasured and not reflected as project effects, e.g. in the form of lowered teaching time by university professors. Y Regional staff indicate that the SAIS project itself "provided no fees to any civil servant and any per diems, etc. were kept within Government guidelines (in spite of pressure to do otherwise)." 2' Salaries, however, may constitute only a small proportion of total remuneration, especially for senior civil servants eligible for housing and other allowances. - 24 - Conclusion 87. Taking all of the above-mentioned considerations together--relevance, efficacy, cost- effectiveness and externalities--the conclusion is drawn that the project's outcome was on-the-whole unsatisfactory. The fairly low efficacy and cost-effectiveness assessments alone support this conclusion, with the "externality" considerations being additional. Box 3 NEED FOR A COUNTRY-WIDE AND MULTI-DONOR TA REVIEW During the past few years, increasing attention has been given to the shortcomings of TA in general. A number of recent documents have identified a variety of systemic effects of TA which too often seem to underming sustainable development and aid effectiveness.y The PAR found that the SAIS project was g characterized by much reliance on resident expatriate TA personnel--one of the main criticisms of TA. But it did find that the project--along with many other TA projects and related donor initiatives-was contributing to distorting effects (negative externalities) on priority setting (donor-driven ad hoc budgetary and work program priorities), on incentive structures in the labor market for trained Ghanaians, and on other aspects of the "institutional environment.' These issues raised above deserve to be considered in some depth, for they go to the heart of broader questions about the developmental effectiveness of the Bank's overall assistance for ID in Ghana, and about the role of the donors in general in ID and capacity building. This is beyond the scope of this audit. The more important systemic and strategic issues will be addressed in the OED Country Assistance Review of Ghana, currently under preparation. Unfortunately, a major UNDP/Government study on capacity building, intended to be the basis for a national action plan, deals hardly at all with the systemic issues. Nor does the NATCAP process seem to be an effective tool for improving the developmental impact of TA. Because the adverse systemic effects have come about through the collectiv actions of Government and donors, it follows that they need to be addressed collectively, on a country-wide and multi- donor basis. It is therefore recommended that a local aid-coordination group be formed with a view to launching a study, with participation by the Government and the major donors of TA, that would identify ways and means to improve the developmental effectiveness of TA in Ghana. V' Cf. DAC principles on technical cooperation (1991), World Bank TA Handbook (1993), UNDP/DAI book Rethinking Technical Cooperation (1993) and Address by E.V.K. Jaycox to the African-American Institute (1993). 88. Insofar as the project's ID-oriented components were among the ones where performance was least satisfactory, it follows that the project's overall ID impact was modest at best, as measured against objectives and in consideration of what might have been achieved with an outlay of almost US$9 million in alternative uses. - 25 - Overview of Project Strengths and Shortcomings 89. It will be useful at this point in the evaluation to review the project's main strengths and shortcomings as summarized in Box 1 in the Evaluation Summary. Outcome of the ODA-Managed Components-Civil Service Reform Program 90. The rating of the project's outcome has not so far differentiated between components managed by the Bank and those managed by ODA. It may be recalled that the PCR and PAR both found the efficacy of the ODA-managed components, namely those having to do with the Civil Service Reform Program--to have fared better on average that the Bank-supervised ones. 91. As mentioned earlier, ODA conducted its own evaluation of its assistance to Ghana's Civil Service Reform Program--including items carried out outside the SAIS "umbrella." Box 4 below presents some of the salient points excerpted from the ODA evaluation. Sustainability 92. The PCR's conclusions on the matter of the sustainability of the benefits emanating from the project are ambiguous. The main points made by the PCR on this matter are that: (i) institutional capacity in several agencies was enhanced through the SAIS project; (ii) insufficient use is made of the project's outputs; (iii) the EMS project (the follow-up project to the SAIS project) takes a "more focussed approach" to capacity building; (iv) "self-sustainability" of the civil service reform was enhanced by the activities of the ODA project; and (v) the input of local consultants and trainers represents an important contribution to sustainability. 93. The PCR does not take a position on whether sustainability should be rated as likely, uncertain or unlikely, but the overall impression given is that sustainability is likely. Not all of the above points are clearly germane to that evaluation. Moreover, the PCR did not consider financial sustainability, i.e.whether the replacement and maintenance of all the equipment imported under the SAIS project could in time be paid for by the government, or would require an indefinite provision of external assistance. 94. One argument made by the PCR is that the EMS follow-up project will help keep the benefits from the SAIS project flowing. That is true, up to a point. Since the EMS follow-up project is nearly identical, it would seem that at least the near-term flow of benefits (e.g. the strengthening of debt management, public investment programming, job inspections, etc.) is more likely to be maintained with the support of the EMS project than without. If the SAIS-provided equipment needs replacement, for example, the EMS project is there to finance the replacement. 95. The next question is whether the Government is on track to become more self-sufficient in financing the expenditures financed in the past by the SAIS project and more recently financed by the EMS project. The matter of equipment replacement and maintenance is essentially one of foreign-exchange earning prospects. But human capital replacement and maintenance is not just a matter of revenue prospects. The "maintenance" and sustained improvements in productivity of highly-qualified public servants will call also for fundamental reforms in the civil service-not just eventually, but before the expected life of the SAIS benefits expires. - 26 - Box 4 EXCERPTS FROM THE ODA EVALUATION REPORT The United Kingdom's ODA was a parallel financier of the SAIS project. It financed and managed components (excepting equipment, financed by the Bank) related to the Ghana Civil Service Reform Programme (CSRP). The ODA project in support of CSRP was however broader than the components nominally under the SAIS project. The following are some salient excerpts from ODA's own evaluation. Overall Conclusion: "The project has been partially successful. The achievements ... have been significantly less than were expected at appraisal. Many of the components have taken considerably longer to achieve than planned and many reforms have not yet been implemented." On Ownership and Commitment: "The fact that the reforms were essentially externally driven and lacked the full commitment of some senior member of the Ghana Civil Service is an important factor in explaining the level of success achieved.... The design of the project paid little attention to the history of the Civil Service, and Ghana's political and socio-cultural context." On Key Comoonents: The more successful components include the diagnostic/design work; progress with implementation has been slow. The Job Inspection Programme has been successful while the pay and grading work is as yet unsuccessful; there has been progress in decompressing salaries but not in introducing the new grading structure. The collapse of real pay levels since the 1970s has only begun to be reversed. On Gross and Net Retrenchment: "Given a theoretical recruitment ban, retrenchment, retirement and natural wastage should have led to a rapid reduction in the size of the civil service.' The substantial gross reductions (involving payments of compensation) have been largely offset by new recruitment, as the freeze on recruitment has clearly been ineffective. Selected Lessons: * A more strategic and long-term approach to civil service reform should be taken. * World Bank conditionalities can strengthen CSRP's prospects. * Conditionalities should set limits on the civil service size and the wage bill. * Project design should take into account important social parameters such as the role of traditional authority structures, family loyalties and gender. * Precise and quantified targets should be set and monitored. Y From Evaluation Report EV 546, "Evaluation of ODA Project in Support of Ghana Civil Service Reform Programme," in two volumes, ODA, September 1993. - 27 - 96. True sustainability of the SAIS benefits would therefore seem to call for (i) greater ownership of the ID and capacity building objectives; (ii) developing and implementing a coherent strategy for ID/CB; (iii) accelerating the implementation of civil service reform, including not only retrenchment but also (and especially) measures to deal with systemic problems that adversely affect civil service morale and productivity; and (iv) accelerating private sector growth upon which the financial and foreign exchange resources of the government ultimately, depend. 97. Some progress is being made on these fronts. But there is a need for accelerated actions on all fronts to ensure the sustainability of the growth process in Ghana in general, and the ID process in particular. 98. A major capacity building initiative is underway, but it is not clear that this involves the degree of government ownership and commitment required to implement a successful program. The initiative itself is also still much in the nature of a list of wants and needs for capacity building, as distinguished from a true strategy with well-defined priorities and sequencing.Y What is needed is an action program that deals with systemic issues and with demand side constraints to effective capacity utilization, as well as with supply side constraints, i.e. the insufficiency of capacity. 99. In civil service reform, government performance has been weak. The signals are not yet loud and clear that action on this front will be accelerated to the extent necessary. Finally, at the macroeconomic level, Ghana does not yet have in place a strategy for raising significantly the very low levels of savings and private investment. But future growth prospects--and financial sustainability- -ultimately depend on raising those rates. For all of these reasons, therefore, the PAR finds the sustainability of the SAIS project to be "uncertain."2' Borrower Performance and Compliance with Covenants 100. The PAR agrees with the PCR assessment that "overall performance of the project management was less than satisfactory". The PCR describes serious problems in the management of the Special Account, procurement, accounting, and record keeping for the PPF account. It found, on the other hand, that the Borrower complied with the essential legal and financial conditions and covenants of the credit agreement, and also that the release of counterpart funds was timely. Finally, the PCR found the Government to have had a lack of interest in making the EPU and ELU operational, whereas "the implementing agencies showed special interest in the project especially because it provided the greatly needed logistics to allow them to perform their functions." The PAR agrees with these findings, with a couple of qualifications. 101. With regard to the covenants, the Government did not comply fully with sections of Article III of the Development Credit Agreement that called for the Borrower to maintain its SAP team, PMU, EPU or ELU all "in forms and with functions and staffing satisfactory to the ' Over the past year, preparations have been underway, under the guidance of a National Capacity Building Steering Committee (on which the Bank is represented) to launch a major capacity building program. A report entitled "Ghana: Next Steps in Capacity Development and Utilization for Accelerated Growth" (UNDP and Government of Ghana, January 1993) will be discussed at a Workshop in May 1994 with a view to preparing a subsequent Strategic Action Plan. V Most of the issues relating to the sustainability of Ghana's growth in general and its ID in particular will be addressed in more detail in OED's forthcoming Country Assistance Review on Ghana. - 28 - Association." Nor was compliance very satisfactory with regard to monitoring (e.g. the submission of timely annual progress reports from each implementing unit) or the financial covenant on record keeping. 102. In interpreting the adequacy of compliance, however, it is important to distinguish between covenants that deal with, say, the probity of financial management and those that concern particular forms by which administrative reform is to be pursued. The Bank's own best practice regarding which kinds of conditionalities are appropriate in this kind of project is still not well- defined. One school of thought is represented by the following quotation from Berg: "The prospects for success in administrative reform would be greater if it were distanced from externally imposed conditionalities. Without local ownership, the best administrative reforms are easily undone and conditionality undermines the growth of local ownership. Besides, the 'process' type of conditionality that is typical in administrative reform programs is rarely effective because assessment of performance is too subjective."Lo' 103. The experience of the SAIS project would seem to lend weight to the view expressed in the last sentence in particular. But it is also true that the basic problem in the SAIS project was the lack of Government ownership of the institutional objectives. Where ownership was present and there was a "champion" of certain reforms/improvements in place, as in the case of the NRS, the project was efficacious. When these conditions were not present, covenants could not force effective implementation. And it would have been counterproductive to try too hard to force compliance. 104. One area of mixed performance on the Government side was in the use of consultants. On the one hand, it was desirable that the Government was in general resistent to the use of expatriate advisers, especially long-term resident advisers, and sought to maximize instead the use of Ghanaian consultants. This approach involved some trade-offs as between near-term efficiency objectives and learning-by-doing. But it has the advantage of fostering long-term ID and minimizing dependency on long-term expatriate advisers. The Region's judgment is that the SAIS project's financing of local consultancy services had generally good results. On the other hand, some believe that the interests of both short-term efficiency promotion and long-term ID could be served better if the government took a more flexible approach to the use of expatriates, whether short-term or long-term. The PAR finds truth in both arguments. Bank Performance 105. Most of the main strengths and weaknesses of Bank performance have already been cited. On the "strength" side, in the upstream stages of the project cycle (identification, preparation, appraisal, negotiation), were the choice of highly relevant objectives; the choice of the "process approach," which would allow considerable flexibility to adjust to changing circumstances; good coordination with ODA; the introduction of the annual work program as a central instrument of project implementation; and diplomatic skills that served to promote good country relations. 12 Elliot Berg & UNDP, on. cit. - 29 - 106. The weaker performance areas in the upstream stages were: the decision to go ahead with the project in the absence of stronger government ownership; the rushed preparation and appraisal, with the consequence that some components were underdesigned; the lack of "insider" understandings of some of Ghana's political, social, personal and institutional realities, that might have led to different, and possibly more productive, approaches to such components as the ELU and PAD; inclusion of the housing component; inadequate risk analysis; and insufficient attention to design of "the process" by which the negative externalities and Lg[ risks (e.g. that the "craze for equipment" would in effect crowd out other objectives) would be minimized and managed appropriately. 107. The project design did not deal explicitly with the systemic and strategic issues underlying the poor morale, motivation, and productivity of the civil service. Nor did it take into account the possible negative externalities, the need for better aid coordination in TA, or the matter of sustainability (even at the narrow project level). There was no Government or Bank ID strategy, and the project was four years after the ERP was adopted, essentially a rehabilitation operation rather than an integrated component of an adjustment strategy. 108. These last omissions need to assessed in the context of what was "best practice" at the time of appraisal, and also in recognition that the "process approach" itself presumed that the project would be supervised and revised, on a rolling basis, in such a way as to deal with issues-including those neglected upstream-in a "best practice" way. In light of the these considerations, the conclusion emerges that the Bank's performance was more satisfactory in the upstream than in the downstream stages. The main problem on the Bank's side was that it did not establish strong discipline in terms of process. in order to ensure constant reassessment of progress towards the most important long-term objectives, A proper choice of performance indicators and a mid-term review could have gone a long way towards this end. 109. The PCR states that the project allowed "sufficient flexibility in the allocation of funds," without however raising the question whether the agl&d allocations that ensued (largely for equipment) represented the best possible use of the available resources. On this point, the PAR's judgment is that Bank managers and staff should have made more effort to take stock of, and subsequently to re-design the project, with more emphasis on ID and less on equipment purchases. It is also the PAR's conclusion that whereas the Borrower's management of the project was too loose, the Bank's supervision was too permissive. 110. This is a judgment of the practice rather than of the probity of Bank supervision. It is not suggested that there was a lack of compliance with the letter of Bank policies and procedures. It is a vexing issue in many TA projects, including the SAIS project, how far staff may and should go in terms of accommodating requests for vehicles, trips and training abroad, etc., including requests from agencies other than those directly participating in the project. Criteria are often not clear and decisions are often not transparent. And responsibility and accountability are in turn often not sufficiently clear. 111. In the SAIS project choices had to be made as to how to respond to political-level pressures to authorize procurement of questionable direct relevance to project objectives. Some cases were resolved in the interest of "keeping the adjustment program on track" and "fostering good country relations." This is tricky territory, where there needs to be consistency regarding "how lines - 30 - are drawn." This is however a Bank-wide issue to which relevant parts of the Bank may wish to give further attention, to ensure clarity and consistency of Bank practice. 112. During the course of the SAIS project, OMS 4.00 was the effective Bank directive on TA. The Bank's management of the various stages of the project cycle was found to be in compliance with this OMS. 113. The quantity of Bank supervision provided was substantial. During the three-year period FY88-90, when about two-thirds of total project funds were disbursed, there were semi-annual supervision missions averaging 3.5 persons each. During those years, according to TRS data, 80 staff weeks (SW) were spent on supervision. And in FY91, when the project became virtually fully disbursed, there were another two supervisions of two persons each. Over the project's life, a total of 104SW were reported to have been spent on supervision. 114. The quantity and apparent intensity of supervision is reflected in many long and detailed Aide-Memoires and Back-to-Office Reports prepared by the supervision missions. Reading these reports one by one, the reader is impressed by the in-depth familiarity with many project issues, by the clear and lucid writing, and by the sensible recommendations. 115. At the same time, the very degree of detail provided on administrative and logistical matters may have deflected attention away from the most basic question of whether the project was on track to have a satisfactory developmental impact. Neither the Aide-Memoires nor the BTO reports contained overview or summary sections, nor did they deal much with systemic or cross-cutting issues, or with the cumulative effects of deviations from project objectives. Thus, with the re- allocations taking place piecemeal, and in the absence of a major stocktaking or mid-term review type of exercise, and with no disbursement rate problems flagging this as a problem project, it is not hard to see how the project became transformed over time more than was probably realized by most managers. 116. The supervision ratings were "2" overall for every year between 1987 and 1992 even though the management performance rating was "3" in every year after 1988. What seems most surprising, however, is that the rating for achieving developmental objectives was "1" in 1990 and 1991. This suggests the hypothesis that there was a "halo" effect emanating from Ghana's widely hailed "successful" adjustment program that may perhaps have led the SAIS project to be seen as more of a success than it now appears to have been in the light of comparing its achievements against its own objectives. The Follow-up Project 117. A follow-up project to the SAIS project, the Economic Management Support Project, was approved by the Board in March 1991. This US$23 million project (of which IDA is financing US$15 million; ODA, US$4 million; and the Government, US$4 million) is truly a "son of SAIS" in that its objectives and design are nearly identical to those of the SAIS project, with even more emphasis given to ID and capacity building. The PCR cites as a strength of the EMS project that the continuity of staff involved "allowed for full understanding of the problems involved in implementing TA and very open communication between the Government and IDA." - 31 - 118. The PAR finds, however, that the EMS project has a number of the same design and implementation problems as the SAIS project. More attention should have been given to identifying lessons of the SAIS project before the EMS project was appraised. In this case, moreover, an arm's length assessment of the lessons might have been most appropriate. - 32 - V. LESSONS OF EXPERIENCE 119. This project confirms several lessons of experience recently codified in the TA Handbook and provides a few others that have not yet entered the lexicon of TA best practice. These lessons are as follows: * The central challenge for operations like the SAIS proiect is to establish strong discipline in terms of process. in order to ensure a constant reassessment of issues. and progress toward long-term objectives. Corollary: The "looser" and more process-oriented is a TA project's design, the more the need to build an efficient monitoring process into the design itself. * Externalities matter. And they may matter a lot, Much more explicit attention needs to be given to the "environmental" aspects of TA projects (where the environment is here defined as the "institutional environment.") Corollary: Consideration should be given to doing a form of "institutional environment impact assessment," focussing upon "cultural" and behavioral dimensions of ID and systemic issues such as how TA gives rise to new forms of rent-seeking through patronage relationships when the incentive system is distorted. * There are no substitutes for ownership on the Borrower's side and for in-depth country knowledge on the Bank side. Bank staff who appraise and design TA projects need to know how local institutions relly work and how political, cultural and personal factors must be "factored in" to project design and supervision. Corollary: Borrower ownership of ID objectives must be established hfore appraisal, and only mature, experienced, and reasonably senior staff should be assigned to appraise and supervise IDTA projects, especially in situations where rent-seeking is rife. * The importance of having local "champions" should not be underestimated. Participation and ownership by "steering committees" and advisory counterpart groups are no substitutes for individual leaders/managers who know what TA they want and know how to use it. A risk that TA may be too ad hominem may be a risk worth taking. * Taking a long-term. strategic approach to TA implies not letting the Bank/donor agendas. work programs. and incentives "drive"proiect identification. design and implementation. It also implies the needs to be patient, to help the Borrower learn by doing and by being accountable, to accept delays and some flops as costs of keeping accountability where it belongs, rewarding those who take the "right" risks (including the risk of Borrower displeasure and criticism when the Bank "just says no"). - 33 - * TA aid coordination (i.e.. the lack thereof) is a problem in Ghana that needs to be recognized and addressed by new means. The variant donor policies and practices regarding salary supplements, for example, require rationalization as a matter of urgency. The NATCAP objectives are important, but the NATCAP process is not working. The Bank should help the Government to take the lead in getting donors to jointly pursue policies and practices that will minimize the negative externalities and improve the developmental effectiveness of TA. It is recommended that a local aid coordination group for TA be organized and launch a study on how to improve, collectively, the developmental effectiveness of TA in Ghana. * Improve the performance indicators and focus them more on quality. process, and behaviors. Disbursement rates may be highly misleading indicators of project performance and absorptive capacity. More attention needs to be paid to monitoring the guality of TA outputs of all types. Most importantly, the indicators should not measure just short-term outputs of studies, documents, meetings, etc. They should aim at monitoring whether processes and learning and other desired behaviors are changing in the desired directions, and are improving in guality. Supervision reports should explain clearly and succinctly, up front, whether the project is on track to have its intended developmental impact.