Uganda - First Health Project

Document of The World Bank FOR OFFICIAL USE ONLY Report No. 19536 PERFORMANCE AUDIT REPORT UGANDA FIRST HEALTH PROJECT (LOAN 1934-UG) June 28, 1999 Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents (annual averages) Currency Unit = New Uganda Shilling (NUsh) 1988 US$1.00 Ush 60 1989 US$1.00 Ush 370 1990 US$1.00 Ush 540 1991 US$1.00 Ush 915 1992 US$1.00 Ush 1,214 1993 US$1.00 Ush 1,130 1994 US$1.00 NUsh 927 1995 US$1.00 NUsh 1,009 1996 US$1.00 NUsh 1,029 Abbreviations and Acronyms AIDS Acquired immune deficiency syndrome DANIDA Danish International Development Agency EVM Evaluation memorandum FHP First Health Project GDP Gross Domestic Product HIV Human immune-deficiency virus ICR Implementation Completion Report IDA International Development Agency MOH Ministry of Health NGO Non-governmental organization PHC Primary health care PIU Project implementation unit SDR Special Drawing Rights SIDA Swedish International Development Agency WHO World Health Organization Fiscal Year Uganda: July 1 - June 30 Director-General, Operations Evaluation Mr. Robert Picciotto Director, Operations Evaluation Department Ms. Elizabeth McAllister Manager, Sector and Thematic Evaluations Group Mr. Gregory K. Ingram Task Manager : Ms. Caroline Cederlof The World Bank Washington, D.C. 20433 U.S.A. ROBERT PICCIOTrO FOR OFFICIAL USE ONLY Director-General Operations Evaluation June 29, 1999 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Uganda-First Health Project (Loan 1934-UG) Performance Audit Report Attached is the Performance Audit Report (PAR) on Uganda: First Health Project (Loan 1934-UG, approved in FY88) prepared by the Operations Evaluation Department (OED). The objectives of the First Health Project (FHP) were to rehabilitate and equip selected health- care facilities, build a hospital in Rakai District, and strengthen preventive health programs through health education and community activities. The project became effective in January 1989 and was closed in March 1996. The project disbursed US$42.5 million equivalent. The governments of Austria and Sweden provided cofinancing totaling US$16.5 million. The project contributed to building necessary health infrastructure and rehabilitating dilapidated infrastructure in a country that had suffered many years of civil strife. It addressed key issues at the time and filled gaps where other donors were reluctant to step in. The project helped increase the range of health services provided and to bring services closer to the population. Utilization rates increased in many of the health facilities rehabilitated by the project. For example, the number of new cases in the outpatient departments of rehabilitated district hospitals increased by 16 to 48 percent. However, the project was poorly designed and slowly implemented. This resulted in increased costs and severe delays. Project design was "quick and dirty" and the Bank failed to involve key implementing agencies and to properly address such issues as the Ministry of Health's underpaid and demotivated staff. In addition, the project design failed to anticipate the large recurrent cost implications of the project, particularly given the low level of government allocations to the health sector in general and to maintenance in particular. The project accomplished little before the midterm review in 1992. The delays, in many cases, were the consequence of procurement difficulties and cost underestimates. Implementation problems also stemmed from the lack of experience with Bank projects in the health sector and the aforementioned failures in project design, which led to confusion about roles and responsibilities and contributed to the decision to drop some activities. In addition, harsh economic realities, such as decreasing coffee prices and high inflation, contributed to the failure of the borrower to provide sufficient counterpart funding-only US$3.5 million was provided out of the agreed US$6.5 million. The project was redesigned after the midterm review to address severe cost overruns and the failure to implement some of the project activities, as well as to increase the focus on primary health care. Some activities were dropped to allow a redistribution of funds to priority areas and to cover cost overruns. Some of these activities were taken over by other donors with more experience in the field. The project would have benefited from more technical assistance and training in Bank rules and procedures, especially at the beginning of project implementation. In addition, MOH participants This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. 2 in a workshop conducted for this audit expressed strong concerns about problems with Bank bureaucracy and complex procedures, which further delayed project implementation. In all fairness, however, it should be stressed that after the midterm review the Bank showed increased flexibility, built MOH capacity, and simplified procedures. The activities defined after the project redesign were achieved according to plan. The sustainability of the project is threatened by failures to properly address major risks, including the demoralized health care workers and recurrent cost implications. Sustainability of both the project and the health reforms are also threatened by factors largely beyond the control of the project. These problems are linked to poor salary structures and deficient payment procedures, leading to corruptive activities, and the Civil Service Reform (which the Bank and the GOU are encouraging as part of macro-dialogue), which has contributed to job insecurity. The participatory workshops conducted as part of this audit also revealed a lack of trust and open communication among different levels of the MOH staff which can also be related to job insecurity. The Control Self-Assessment (CSA) methodology was adapted for use in this audit and was employed in workshop settings to elicit stakeholder views. The workshops provided evidence of strong commitment to the decentralization process and to health reforms at both MOH and district levels. According to the self-assessment, districts were strong on issues related to their objective setting, planning, learning from analysis and empowerment, and clarity of roles. This is an asset for the implementation of health reforms and the Bank follow-up projects. While district officials showed high confidence regarding their roles and responsibilities under the decentralized system, central MOH officials were constrained by lack of training, unclear roles and responsibilities, and instability in decision making, which undermines their ability to carry out the work efficiently. These findings may explain some of the difficulties encountered in achieving the goals of the FHP and should be considered in estimating the feasibility and scope of further sector reform efforts in Uganda. This audit shows that using the CSA methodology in an audit provides value added in comparison to more traditional audits. It was an effective tool for addressing institutional issues affecting project implementation as well as facilitating discussion around sensitive issues. It can also be effectively used to build capacity for evaluation within the borrower country. The audit agrees with the ICR in rating sustainability as uncertain, institutional development as modest and Borrower performance as satisfactory. The audit downgrades project outcome to marginally satisfactory, mainly due to the delays, cost overruns and failure to address key issues, which meant that outcomes fell below expectations. The audit also downgrades the rating of Bank performance to unsatisfactory because of the insufficient provision of training and technical assistance in project design, the slow reaction to making necessary adjustments and failure to properly address key issues affecting project outcome. Several lessons can be drawn from this project. The first is the necessity of timely, flexible and simple project design including a training component on Bank procurement rules early on in project implementation, especially in post-conflict projects. A second lesson is the necessity to maintain a holistic view of the sector and address fundamental issues likely to affect project outcomes such as human resource issues. A third lesson is that the Bank needs to be sensitive and recognize the complexity of and the difficulties involved in introducing cost-sharing mechanisms; it should emphasize the need for preparatory studies such as studies on the willingness and ability to pay for health care services; and, Bank financed credits should emphasize short-term technical assistance for the development and implementation of cost-sharing schemes. Attachment Contents Pincipal Ratings & Key Staff Responsible ................n............... Preface....................................................... i 1. Introduction.................................................1 Organization of Health Services ..............2....... ...........2 Financing ofHealth Services....................................2 First Health Project ............3............. ...............3 2. Project Preparation and Implementation.............................5 Preparation Issues...............................................5 Participation............................................ Risk Analysis............................................6 Monitoring and Evaluation..................................6 Finance ................................................6 Implementation Issues............................................7 Delays .................................................7 Design Adjustments........................................8 Financial Covenants........................................8 3. Outcomes And Ratings..........................................9 Physical Rehabilitation and Equipping ofHealth Facilities.................9 Building a Hospital in Rakai District..............................10 Health Education and Community Activities................................ Strengthening the Delivery of Health Care Services.....................12 Self-Assessment ofInstitutional Strengths and Weaknesses.................12 Ratings. ..................................................13 Borrower Performance........................................13 Bank Performance ..............................................14 4. Conclusion and Lessons ............................... .........15 Annex A: Basic Data Sheet e.............................. .........16 Annex B: The CSA Methodology ........................... ........18 AnnexC: Data From Voting on Project Evaluation Assertions, MOH, Districts.... 19 Annex D: Assertions Related to Self-Assessment Used in CSA WorkshopsWeaneses.....21 This report was prepared by Caroline Cederlof (Task Manager), who audited the project in October-November 1998. Mr. William B. Hurbut edited the report and Marcia Bailey provided administrative support. 11 Principal Ratings ICR EVM Audit Outcome Satisfactory Satisfactory Marginally Satisfactory Sustainability Uncertain Uncertain Uncertain Institutional Development Modest Modest Modest Borrower Performance Satisfactory Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Unsatisfactory Key Staff Responsible Task Manager Division Chief Country Director Appraisal V. Jagdish F. Lethem C. Madavo Midterm V. Jagdish D. Makar F. Colaco Completion M. Mulusa J. Maas J. Adams 111 Preface This is a Performance Audit Report (PAR) on the First Health Project (FHP) in Uganda for which a credit of SDR 30.8 million (US$42.5 million) was approved on June 23, 1988. The credit (Credit 1934-UG) closed on March 31, 1996. An undisbursed balance of SDR 1.9 million (US$2.76 million) was cancelled. The project was cofinanced by the Swedish International Development Agency (US$6.5 million) and the Government of Austria (US$10 million). Operations Evaluation Department (OED) staff prepared this report based on a review of the Staff Appraisal Report (SAR), Implementation Completion Report (Report No. 16259, published January 28, 1997), project correspondence files, Bank documents on other health projects and other projects in Uganda, and other Bank material and reports. In November 1998, an OED mission traveled to Uganda where it held discussions and workshops with official representatives of the appropriate ministries, district teams, private (NGO) and civil entities, other donor agencies, and communities. The audit piloted the control self-assessment (CSA, a participatory methodology) for the first time in an OED performance audit. The CSA has been used extensively within the Bank to improve effectiveness and efficiency. Three workshops were carried out using this methodology including key government officials from the Ministry of Health and the Ministry of Information and Broadcasting, representatives from eight districts and political, religious, and traditional leaders from a community. The CSA methodology allowed the team to address institutional issues as well as sensitive issues (otherwise not easily discussed openly) both of which are likely to affect project implementation and hence the success of any project. OED also reviewed the project with Bank staff and representatives from the project implementation unit (PIU). The PAR had five objectives: (a) to assess project preparation, including borrower participation; (b) to assess project implementation; (c) to evaluate project outcomes, institutional development, and sustainability; (d) to draw lessons for the future; and (e) to pilot the use of the CSA methodology in an audit setting. Following standard OED procedures, copies of the draft PAR was sent to relevant government officials for review and comments. No comments were received. 1. Introduction 1.1 During the civil unrest that prevailed in the 1970s and 1980s, Uganda suffered a severe decline in health services. By the late 1980s the country's health sector was in disarray and health facilities at all levels, looted of their equipment and supplies, were in disrepair. Uganda's health indicators were among the worst in the world (Table 1.1). The level of mortality was significantly higher than in Kenya, Tanzania, or Zimbabwe and treatable diseases-measles, respiratory tract infections, gastroenteritis, anemia, and malaria-accounted for 52 percent of that mortality.1 1.2 Women and children were particularly at risk in this situation. Most maternal and child health programs had been abandoned, and immunization rates had declined precipitously. Coverage rates for tuberculosis immunizations (BCG), at 70 percent in 1971, had fallen to less than 5 percent by 1984. Coverage for DPT (diphtheria, pertussis, and tetanus) and polio was less than 3 percent of the target population in 1984. The major reasons for these low rates were lack of transport and a complete breakdown of the delivery and storage system for vaccines. 1.3 At the same time, Uganda had a growing HIV/AIDS epidemic (one of the worst in the world). An estimated 25 percent of pregnant women attending Kampala hospitals tested positive for HIV and 15 to 25 percent of blood transfusion recipients contracted HIV because of inadequate screening of the blood supply Largely because of the heavy disease burden, life expectancy in the late 1990s has actually decreased in comparison to the late 1980s and the crude death rate has increased during the same period. Table 1.1. Selected Indicators of Demographic and Health Status 1986/87 1996/97 Sub-Saharan Uganda Sub-Saharan Uganda Africa Africa Crude Death Rate/1000 15 18 14 19 Crude Birth Rate/1000 45 51 41 49 Life Expectancy (M/F) 45/48 48/49 51/54 43/43 Infant Mortality Rate/1000 104 116* 91 99 Total Fertility Rate 6.4 7.2 5.6 6.7 GDPI capita (US$) 462 271 532 310 Source: World Bank Regional Database * Some reports referred to in the SAR suggest that infant mortality rates were between 120-200/1,000 live births around 1988. 1. World Health Organization, 1981 cited in the "Staff Appraisal Report, The Republic of Uganda, First Health Project", June 24, 1988. Report No. 7203-UG. The World Bank. 2. Uganda SAR on First Health Project, June 24, 1988. 2 Organization of Health Services 1.4 The organization of the health sector in Uganda changed during the First Health Project under an economy-wide decentralization program that started in the early 1990s. Before the reforms, the Ministry of Health (MOH) was responsible for health service provision by district hospitals and above, while the Ministry of Local Government was responsible for school health services, maternal and child health services, rural ambulance service, health education, and control of communicable diseases. Since decentralization, the role of the central government has been limited to policy formulation, planning, inspection, and management of national programs. The MOH still supervises district and referral hospitals and now also provides the districts with technical support and guidance, and monitors and supervises performance. District Health Committees provide policy guidance at the district level and oversee implementation of health services. Health Committees carry out the same tasks for the health units in each sub-county. Health Unit Management Committees provide operational guidelines for individual health units. The district director of health services (DDHS) and the district health team (DHT) perform executive functions such as planning, organizing the distribution of drugs and vaccines, assessing manpower and training needs, and monitoring and supervising health activities. The economy- wide Civil Service Reform, aiming at reducing the number of staff in the Civil Service in order to improve its efficiency, has put a ban on hiring staff at all levels. This has had a negative impact on performance in the health sector (discussed further in chapter three). 1.5 NGOs (mainly churches) have a large presence in Uganda's health sector. During the civil unrest, their health facilities were sometimes the only ones able to provide care. In 1994, these organizations accounted for 38 percent of all hospital beds and 54 percent of total bed days' The NGOs, not surprisingly, are well organized (the Catholic and Protestant missions even have a joint service to procure, store, and distribute drugs) and very experienced. Financing of Health Services 1.6 Financial allocations to the health sector in Uganda are low by the standards of sub- Saharan Africa (1.6 percent of GDP in 1990, compared to an average of 2.5 percent in sub- Saharan Africa). In 1986/87, when the First Health Project was under preparation, actual expenditure on health was less than 4 percent of government budget. In addition, until 1990/91, only 10 percent of government spending on health was allocated to primary health care. The continued underfunding of government recurrent expenditure for health was noted in the SAR. The underfunding meant that public sector health staff had suffered from low remuneration-a situation also noted in the SAR-and drug shortages were a serious problem in the country. When the project was under preparation, the total drug requirement for the country was estimated at US$13 million per year. Only 10 to 20 percent of that requirement had been made available from government sources. The SAR noted that since 1985 DANIDA, together with the Danish Red Cross, had provided additional drugs worth US$6 million. The health sector is declared a priority sector in the recent Poverty Eradication Plan developed by the GOU. 3. World Bank, 1994. "Staff Appraisal Report, Republic of Uganda, District Health Services, Pilot and Demonstration Project." Report No. 13515-UG. 3 1.7 While medical care was officially free, unofficial user fees were charged at all facilities, including those of the government.' Since decentralization and the accompanying formalization of user fee policies in 1993, district health centers have come to rely on revenue from user fees for their non-salary and non-drug expenses. 1.8 Other donors involved in the health sector at the time of project preparation included the Canadian International Development Agency (funding health training through the African Medical Research Foundation), World Health Organization (providing funding for HIV/AIDS control) and UNICEF (supporting primary health care and child nutrition) as well as various support from Netherlands Ministry of Development Cooperation. First Health Project 1.9 The design of the First Health Project attempted to address both the deterioration of Uganda's health system and the financing problems that led to that decline. The project was expected to improve effectiveness and efficiency in the health care system through a tripartite strategy. First, it would rebuild health staff morale by rehabilitating facilities and providing essential equipment and supplies. Second, it would strengthen preventive health programs. Third, it would address planning, management, and financial issues. The strategy was also expected to help slow the spread of HIV/AIDS.' The project is summarized in the table that follows. 4. Pieter van Dijk, 1988. "Health Financing and the World Bank's Health Project in Uganda." Consultant mission report. 5. Ibid. 4 Table 1.2: At-a-Glance- First Health Project Credit No.: 1934-UG Approved: June 23, 1988 Total cost: US$60.88 million Effective: January 11, 1989 World Bank credit: US$40.48 million Closed: March 31, 1996 Cofinancing: US$16.9 million (Sweden and Disbursement record: US$2.76 million canceled Austria) at close Objectives: * to carry out urgent rehabilitation and equipping of selected health care facilities * to build a new hospital in Rakai District * to strengthen preventive health programs through health education and community activities * to improve long-term effectiveness of the health care system through institutional development and improved internal efficiency Components: * Rehabilitate key areas of Mulago central hospital, re-equip the Blood Transfusion Center, rehabilitate eight district hospitals and 30 rural health centers, and construct a 100-bed hospital in Rakai district. * Promote health status by creating awareness through health education programs, developing a community-based distribution system for drugs and supplies, and developing a program for counseling and patient management for people infected with HIV/AIDS. * Strengthen the Health Planning Unit, the management capability in the Ministry of Health, hospitals, and rural health facilities, and develop alternative ways of ensuring adequate financing of the health sector. 1.10 The Development Credit Agreement was amended after the mid-term review, in October 1994. Funds were reallocated from the proposed construction of a 100 bed hospital in Rakai district to construction of a new health center, rehabilitation of five health centers and an office for the District Medical Officer. 5 2. Project Preparation and Implementation 2.1 The project objectives were arrived at after consultation with the government of Uganda, the MOH, and other donors and NGOs. The audit concurs with the ICR' s judgment that project objectives - in terms of prioritized areas for intervention - were in line with requirements and priorities at the time of project preparation (although aspects of project design were poor as noted below). The audit also concurs with the view that the project filled gaps other donors were reluctant to fill. This was supported by the MOH and the districts in the control self-assessment (CSA) workshops,6 where participants rated project "relevance" as high. The project was later modified to accommodate severe cost overruns and an increased focus on primary care and to allow for the preparation of follow-up projects.' The changes in the project objectives were appropriate adaptations to changing circumstances, but came late-only after more than four years of implementation. Preparation Issues 2.2 The urgency of Uganda's health sector needs in the aftermath of its conflict was the proximate cause for a compressed preparation period (six months) that may have contributed to weaknesses in the design of the First Health Project. Participation 2.3 In one CSA workshop, MOH officials, especially those involved in implementation, indicated that they were not prepared when the project started. Indeed, OED has suggested that the project may have been premature, largely because of the weakened state of the institutions involved.' This may explain why the MOH rated design quality as poor. Not only were the implementing agencies not involved in preparation, the districts-the principal target of the facility reconstruction and rehabilitation-also were not substantially involved at the beginning of the project. 6. The Control Self-Assessment (CSA) methodology is a participatory methodology used in workshop settings to let the participants assess their own organizational strengths and weaknesses. It has been used extensively within the Bank. This audit used the methodology for the first time in an OED evaluation. Workshops were conducted with MOH officials, districts, and one community. The questions were adjusted to better reflect the audit requirements. (For more on the methodology and the workshops, see Chapter 4 and Annex B and C) 7. Two follow-up projects are ongoing: Sexually Transmitted Infections (Credit 2603-UG, effective July 22, 1994) and District Health Services Pilot and Demonstration Project (Credit 2679-UG, effective July 17, 1995). 8. "The World Bank's Experience with Post-Conflict Reconstruction, Volume IV: Uganda Case Study," Report No. 17769, May 4, 1998, p. 27. 6 Risk Analysis 2.4 The project preparation team identified three risks at the beginning of the project: (i) weak institutional capacity, (ii) uncertain willingness and ability of health authorities to develop alternative methods of financing, and (iii) demoralized and underpaid staff. To avoid the first risk, project implementation was to be structured in a way that avoided overextending MOH staff. The second risk was to be addressed by establishing independent hospital and health center management boards who would build on NGO experience with user fees in the country. However, the design documents did not recognize the complexity of introducing user fee systems. They also did not include plans to analyze the existing informal user fee system or build capacity or consensus within the government. Nor did they plan to study the willingness and ability to pay or provide technical assistance in designing and implementing a sustainable user fee system. The third risk was to be addressed by rehabilitating health facilities, improving the working environment, and increasing community involvement (later dropped). Yet no analysis was undertaken to examine what factors influenced staff morale. In retrospect, it is evident that morale and productivity remain low today, so whatever factors contribute to that problem must still be present. Monitoring and Evaluation 2.5 The SAR did not define any monitoring indicators or system for monitoring project performance or impact. This makes it impossible to evaluate whether the project achieved any impact on, for example, immunization coverage, child health, or service utilization. The only indicators defined related to accomplishing various implementation steps, such as "a work plan developed for rehabilitation" or "number of health educators trained" or "number of seminars held." Finance 2.6 Insufficient attention was paid during preparation to the ability of the government to provide counterpart funding. The amount required (US$6.5 million) was unrealistic at a time when the country was faced with rebuilding a war-torn economy. Moreover, given the project's implications for recurrent costs, the question of sustainability should have been raised early in preparation. It was simply expected that the government would assume these costs, despite the lack of available government funding (Uganda's low level of domestic tax revenue by sub- Saharan Africa standards remains a problem) and the low expenditure on maintenance. Out of the total recurrent expenditure on health care in Uganda, the government accounted for about 25 percent in 1996/97, while donors accounted for about 17 percent and private spending accounted for about 58 percent.' It is encouraging, though, that the total share of government budget allocated to health has increased from 4 percent in the 1986/87 budget to 7.1 percent in the 1996/97 budget (still low by sub-Saharan Africa standards), and that the government continues to articulate a commitment to increasing allocations to priority social sectors such as health and education. The failure to properly address the recurrent cost implications was confirmed in the CSA workshop by both the districts and the MOH officials. 2.7 The hasty preparation of the project also affected implementation. It resulted in confusion regarding roles and responsibilities during implementation (in part because key implementing 9. World Bank, 1995. "Uganda: Public Expenditure Review June 1995." Draft. 7 agencies had not been involved in the design phase), led to unnecessary delays in project implementation and cost overruns, and resulted in changes in project objectives after the midterm review. Implementation Issues 2.8 The project was managed by a project implementation unit (PIU) in the Planning Department of the MOH. The Permanent Secretary of the MOH had overall responsibility for project coordination. Delays 2.9 The project was slow to start and remained slow in implementation. Little had been accomplished by the midterm review in 1992. The ICR attributes the difficulties and delays experienced in implementation to shortage of counterpart funding, lack of capacity in the PIU, and too little time devoted to preparing implementing agencies. The audit agrees with this analysis, but identified three other shortcomings in project implementation: * lack of clarity in implementation roles and responsibilities; * failure to train implementing staff in Bank procedures and rules at an early stage; and * burdensome bureaucracy 2.10 It is clear from the CSA workshops that the failure to involve key implementers in the design of the project led to lack of clarity in roles and responsibilities. This contributed to the delays in project implementation and the decision to drop some activities and redefine others. 2.11 According to the CSA workshops, the project benefited from a conducive environment with strong political commitment and leadership and an enthusiastic MOH eager to show results. The enthusiasm and commitment notwithstanding, the MOH and the PIU had insufficient capacity to carry out the demanding tasks specified in the project design. They had no previous experience with Bank projects. Capacity improved over time, especially after the midterm review, when more qualified staff were hired, the project coordinator was replaced, and technical assistance was brought in to strengthen the PIU and the MOH. While this helped improve performance, the project could have benefited from training (especially in procurement procedures) and technical assistance at the beginning of the project. 2.12 The CSA workshops provided examples of the bureaucracy and "red tape" that contributed to implementation delays (not recognized in the ICR): "no-objections" took too long to get from Washington and accessing funds was complicated and time-consuming."o The project had to go through the Central Bank of Uganda to access funds, which further delayed access. This was changed after the midterm review and a special account was opened. This could have been done earlier in the project as the problems were evident. Participants in the workshops also criticized the Bank for being slow in informing the borrower about procedural changes. 10. One example, from the Health Information Unit, regarded the trouble involved in receiving funding for a workshop in northern Uganda. Somebody from the MOH had to travel to the site (it takes several days) to get an invoice from the hotel, then travel back to Kampala to send it to Washington for approval. Design Adjustments 2.13 The project agreement was amended in October 1994 because of cost overruns and to avoid duplication with other donors and provide funds for the preparation of follow-up projects. Funds were reallocated from the proposed construction of a new 100-bed hospital in Rakai district, procurement of some equipment and supplies, and community activities. Funds were also allocated to pilot sector reforms. The reallocations and the redefinition of project objectives were appropriate under the circumstances. Adjustments in the project design to better reflect the changing circumstances, however, were slow and corrective action was not taken until after the midterm review. In retrospect, it should have been clear from earlier supervision missions that the project design needed adjustment. It is somewhat surprising that project implementation was rated as "highly satisfactory" in all supervision mission reports up to the beginning of 1991, despite severe delays in many activities and high cost overruns. It is also surprising that the high number of supervision missions (22 in comparison to an average of 19) did not result in an early detection of the problems. 2.14 The PIU has been built up gradually during the FHP and the follow-up projects, and is benefiting from both technical and financial support. PIU staff members are paid market rate salaries and the unit has therefore been able to attract some of the best people in Uganda!' The unit efficiently runs the ongoing projects, and develops and implements many health reforms. One of its remaining problems is its lack of integration into the MOH, making its sustainability questionable. Financial covenants 2.15 The financial covenants were largely met. A financial review was carried out in 1992 and a proposal for cost sharing was developed but implemented only on a limited scale. Community participation was achieved through the Government's decentralization policy, whereas cost sharing was mainly implemented informally. Later, the constitution gave the districts mandate to raise revenues. Revolving funds to buy drugs were not given to Mulago and to district hospitals due to revenue shortfalls, although some health units have now started to provide such funds. 11. This has clearly led to tension between the PIU and the rest of the MOH and has contributed to unclear division of roles and responsibilities for policy and planning between the PIU and the Planning Unit. 9 3. Outcomes And Ratings Physical Rehabilitation and Equipping of Health Facilities 3.1 Physical rehabilitation and equipping of eight district hospitals and 30 health centers accounted for the major part of the project costs (61 percent). The physical objectives of this component were largely achieved after initial delays (by an average of 7 months), cost overruns (77 percent, or US$8 million, at midterm review), and shortfalls in counterpart funding. As discussed earlier, lack of experience in Bank procurement procedures was a significant factor causing delays. After midterm review, commitments were scaled down and the savings were used to finance cost overruns of about US$2.5 million in rehabilitation of district hospitals and health centers. 3.2 The range of services provided in the rehabilitated facilities improved, and access to health care increased, as some areas previously lacked access to health services (that is, either had no health facility or had an existing health facility that was so dilapidated that it could not provide services). From discussions and the CSA workshops, it is clear that district and community representatives perceive that services improved through the project. An example of the increased access was given in one of the districts visited: people previously had to walk for days to reach the nearest functioning health facility, by which time the sick person would often be dead, and would have to be carried all the way back to the village on the back of a bicycle. One positive outcome clearly attributed to the project is the increased demand for health services at the rehabilitated health facilities in comparison to those not rehabilitated. 3.3 Communities and districts reported that rehabilitated facilities were clean and well-kept. Maintenance was initially a problem as the budget for maintenance was low (around 2 percent of recurrent budget in 1994) and rarely disbursed. This has improved through the decentralization, as districts now have increased power and control over resources and, hence, can budget for maintenance within the district. In addition, the revenues collected from user fees car also be partly used for maintenance. Under the IDA credit, a central maintenance workshop was constructed to carry out equipment repairs, and 10 technicians were trained. This unit now depends on DANIDA for 85 percent of its operating costs. 3.4 New equipment-financed by an Austrian grant (US$10 million) as well as by IDA- helped to increase the range of services that health facilities could provide. But the Ugandans were unaccustomed to the equipment (imported from Austria) and were therefore unfamiliar with its maintenance. The audit team found that few people in the hospitals had been trained in maintaining this equipment. Furthermore, spare parts had to be ordered from Austria, which was excessively expensive-an issue not addressed in the ICR. Some of the equipment procured under this grant was reported in the CSA workshops to have broken down quickly and to now be unusable. 10 3.5 Although the FlP only scratched the surface of the need for new and rehabilitated health facilities in Uganda," the project greatly improved the facilities at Mulago hospital, eight district hospitals, and 40 health centers. It also provided essential equipment and improved working and living conditions for staff. 3.6 Even so, the rehabilitation of health facilities failed to make a major difference in the morale of health care workers as anticipated in the SAR. Interviews and district workshops revealed that motivation among some staff had been increased by the provision of a clean working environment, accommodations, and water. Among others, however, low salaries and failure to pay salaries-sometimes for as much as several years-had a highly negative impact on motivation that outweighed the other positive impacts, something the ICR did not address. This problem is not confined to the health sector, but also affects other sectors in the country. It was clear from the CSA workshops that to make up for the salary difficulties some staff engage in corruptive activities such as collecting supplementary "fees" for service or selling pharmaceuticals privately and staff not being present during regular working hours. One effect of these corruptive activities is that the poor rarely receive free services and an estimated 40 to 94 percent of the public drug supply is sold through informal channels.3 Without both drugs and health care staff, it is difficult to provide any health care to the communities. 3.7 Lack of training among lower-level health care workers and job insecurity resulting from civil service reform are also a problem. According to anecdotal evidence, 80 percent of the health care workers in rural health facilities lack any formal training. At the same time, unemployment is high among newly certified doctors and other trained health care workers. The situation is aggravated by a ban on hiring new staff that is part of the civil service reform. The Bank could play an important role in negotiations with the MOF to lift the ban on hiring new, qualified staff at the lower levels in the health care system. The project failed to recognize these problems, despite their impact on sustainability of the project as well as on the overall health reforms. 3.8 The project's hypothesis that staff morale would increase through rehabilitation of health facilities proved not only naYve, but also insensitive to people's needs and difficulties. While renovating and building of health facilities was undoubtedly essential to reestablishing health care in Uganda, the staff for those facilities was neglected. The project design could have done more to evaluate staff incentives. Perhaps then the importance of salaries that are both sufficient and paid on time would have been clear. Demoralized staff continues to be a problem overshadowing positive outcomes of the FITP but also of the overall health care reforms. Building a Hospital in Rakai District 3.9 Under the restructured project, the plans to build a 100-bed hospital in Rakai were dropped. In its stead, a new health center was built in Rakai and five additional health centers were rehabilitated. The health center in Rakai was finished in 1996 and is referred to by patients and the DMO as the district hospital. The redirection of funds was justifiable because a nearby NGO hospital was expanding to a 100-bed capacity. This decision was made in consultation with 12. Geographical access to health care is still severely limited, with only 49 percent of the population living within 5 kilometers of any type of health service unit, according to the "Health Facilities Inventory 1992" produced by the MOH. The national coverage, as estimated by the MOH in 1997, is 798 persons per bed and 12,500 persons per health facility. The MOH estimates that an additional 440 health centers (level III and IV), dispensaries, and sub-dispensaries are being upgraded to level III health centers under various bilateral and multilateral programs. 13. Asiimwe D, Mwesigye F, McPake B, Striefland P, 1997. "Informal Markets and Formal Health Financing Policy." Mimeograph. 11 the district and seems to have been in line with district policy. It showed some flexibility on the Bank's part, but again could have been done earlier in the project. Health Education and Community Activities 3.10 The major objective of these activities was to promote improved health status through health education programs, development of a community-based distribution system for drugs and supplies, and development of a program for counseling and patient management for people infected with HIV/AIDS. While the health education component was accomplished, none of the community activities were carried out. The cost overruns in other parts of the project made it necessary to reprioritize. Some of the dropped activities, such as the drug component, were taken over by other donors with more experience in the field than the Bank. 3.11 The project helped strengthen the Health Information Unit at the MOH, mainly through training. The originally planned activities were altered along the way, but the Health Information Unit strongly felt that this was an improvement over the original project design, in which the implementers had not been involved. It was evident from discussions with MOH and the CSA workshops that the quality and creativity of the work undertaken in the Health Information Unit had substantially improved. For example, the unit worked closely with military, traditional, and religious leaders in its effort to combat the HIV/AIDS pandemic. The unit communicates innovative health education messages through drama, radio, television, mobile film vans, newspapers in local languages, and other media. The unit is also doing a better job today at pre- testing the promotional materials and methods used. 3.12 In terms of impact, condom use has increased and sexual debut has been delayed. Moreover, the proportion of males age 15-19 who reported that they had never had sex rose from 31 percent to 56 percent between 1989 and 1995. For females, it increased from 26 percent to 46 percent in the same period. Uganda is also the only country in sub-Saharan Africa that has achieved a decline in HIV prevalence. It is difficult to fully attribute these improvements to the FHP, but the financial support, capacity building, and overseas training of seven senior and mid- level staff-including some from the Health Information Unit-is likely to have contributed to the quality of the health messages carried out by the Health Information Unit. According to people interviewed, the project helped mobilize interest in health promotion in schools and some NGOs became more involved and actually took over some activities. The effort to combat the HIV/AIDS pandemic has continued in the Sexually Transmitted Infections Project. During the audit mission, however, the Health Information Unit raised concerns about the lack of health education specialists in the supervision teams of the follow-up projects. 3.13 One activity of this component sought to procure radio transmitters to help get health messages to more remote areas of the country. Little progress was made with this activity in part due to failure to agree on specifications and lack of training in procurement procedures (affecting procurement in all parts of the project). Only at the end of the project provided a consultant to assist. Financing procurement of radio transmitters is inevitably a highly political issue in a country like Uganda, but that makes it all the more important that project have the best possible procurement support at the start. This component has been taken over by the follow-up projects, but to date no radio transmitters have been procured. 3.14 This component also conducted a knowledge, attitudes, and practices study in three pilot districts in 1991/92. Senior health officials reported that the results of this study contributed to policy development and implementation. 12 Strengthening the Delivery of Health Care Services 3.15 The project supported technical assistance to develop an MOH white paper on health policy in 1993 and a complementary three-year health plan (1993/94). On that basis, the audit agrees with the ICR that sectoral policies were improved through the project and that the planning function was strengthened. But this support may not have been optimal. Some MOH officials complained to the audit mission that the Bank-financed consultant for this activity prepared the white paper mostly on his own, thereby failing to maximize capacity building. 3.16 An effort to improve capacity at the MOH through training also seems to have fallen short. The project trained several staff in the ministry's Health Planning Unit. Retaining the trained staff members proved difficult, however, as their standard civil service salaries were not competitive enough to keep them from finding other, better paying jobs. 3.17 A budget framework to monitor trends in health expenditure was introduced with project support, and trends showing allocations among primary health care and hospitals are now produced regularly. Activities to strengthen planning and budgeting in the districts were also initiated with the Bank's project assistance. This contributed to laying foundations for the decentralization process and has been taken over and further developed under the follow-up District Health Services Pilot and Demonstration Project. 3.18 The development of alternative health care financing schemes fell below expectations according to the ICR. But work done under this component set up an inter-ministerial task force in 1990 to address health care financing policy issues. The task force developed a proposal to introduce cost-sharing mechanisms but the National Resistance Council rejected it. Meanwhile, cost sharing was widely practiced but without formal authorization. In 1993, a local government statute permitted fee-for-service and the MOH distributed basic guidelines for the collection and use of revenue. But without a formal health care financing policy endorsed by the MOH, the practice and implementation of user-fee schemes was (and still is) poorly monitored. The successful introduction of user fees is a complex issue that requires political will and commitment, marketing and the development of proper accounting and Monitoring and Evaluation systems (M&E) is essential. All of which were clearly lacking. 3.19 The CSA community workshop found that uncertainty surrounding the fee policy has contributed to confusion among patients. People are unclear about which fees are legitimate and which are not. The audit mission also found a lack of clarity about the objective of introducing user fees. Was it purely to increase revenue (which would have been difficult given Uganda's economic environment with a large proportion of poor), or was it also to improve the referral system or make people more responsible for their own health? These objectives have different implications for the levels and types of fees required. The Bank failed to recognize the difficulties in implementing a sustainable user fee system and the need for technical assistance and advice on such a complicated issue. Self-Assessment of Institutional Strengths and Weaknesses 3.20 The self-assessments of the MOH and the districts (using the CSA methodology) addressed values and beliefs, team environment, communication and relations, planning and risk assessment, availability and reliability of information, level of skills and knowledge, team environment and responsibility, monitoring and learning, resources, and control mechanisms. 13 3.21 These discussions yielded evidence of a strong commitment to the decentralization process and the health reforms at both the central and district levels. The organizational structure at both levels was felt to be effective and supportive. 3.22 The scores in the district workshop were extremely high on issues related to objective setting, planning, learning from analysis, controls, and adherence to policy. Interest in learning from analysis was strong and they were eager to improve performance. The districts felt empowered to carry out their responsibilities and were clear on their roles and responsibilities. It is evident that the decentralization process - in which the Bank has played an important role - has built capacity and empowered district staff, which is an asset for implementing the health reforms, although it was felt that more training was needed at both district and central levels. 3.23 The central level, on the other hand, faces unclear roles and responsibilities, little or no capacity building, and instability in decision making, which the participants reported to be related to the constant restructuring of the MOH, leading to ad hoc assignments and job insecurity. This has led to an unclear structure at the central level. On the positive side, the work process was considered to be efficient and effective, and the work was still considered enjoyable and challenging and the participants felt involved in setting their own objectives. 3.24 The major problems highlighted in all the workshops were human resources issues not addressed in the ICR: low salaries, delays in salary payments, lack of qualified staff at lower levels, and the ban on hiring new staff. This is a demotivating factor and puts a strain on proper use of resources. Cultural factors, combined with the restructuring and job insecurity, put a strain on openness and trust. For example, some workshop participants felt that bad news could not easily be discussed with the next level up and that there were no clear mechanisms for solving conflicts between different ministries or for coordinating efforts between them. 3.25 These problems identified in the CSA workshops are likely to affect implementation of any project or policy in the MOH. Human resource problems-by demotivating and demoralizing staff in the health institutions-have clearly overshadowed gains made in the FHP on improving the availability of equipment and improving the infrastructure. Ratings 3.26 Although many of the initial objectives defined in the SAR were not accomplished, the audit rates outcome as marginally satisfactory mainly because the objectives were relevant to country and sector development and the goals as defined after the amendment of the project were achieved, although the revision came late. Sustainability is rated as uncertain, mainly because of the high risks involved in the unstable workforce situation and the corruptive activities at all levels. Despite efforts to improve institutional development in the project design, the audit rates it as modest because many of these activities failed during implementation and although the project built capacity in specific units it did not contribute to any major institutional improvements in terms of policy or organizational development. Borrower Performance 3.27 Despite the lack of sufficient counterpart funding and delays on the borrower's side, borrower performance is judged to be satisfactory based on the borrower's efforts and willingness to change and improve. Given the low capacity in the MOH at the start of the project and its unfamiliarity with Bank projects, it would be unrealistic to expect more. 14 Bank Performance 3.28 Bank performance is rated as unsatisfactory mainly because of the failures in project design; failure to provide sufficient assistance and training, especially at the beginning of the project; and the slow reaction to making necessary adjustments during project implementation and failure to properly address issues affecting the project such as human resources. It is particularly surprising that the supervision missions rated implementation as highly satisfactory until mid 1991 despite the delays and other difficulties encountered. In addition, during this period only 30% of expected disbursement had been made. It should however be recognized that major improvements took place following the mid-term review. 14. Operations staff noted that extra time is required and expected for implementation of any project in a country that is just coming out of a conflict situation. They also wanted to recognize that, on top of the supervision missions, day-to- day correspondence was taking place between the Bank and the MOH, in which issues arising were regularly followed up. Operational staff also noted that the human resource issue was outside the capacity of the project, and that some of the issues have only recently become clear to the Bank and the borrower, such as the large outstanding salary payments. 15 4. Conclusion And Lessons 4.1 Projects require timely and flexible project design with flexibility, simplicity, and training in Bank procurement rules, especially in a post-conflict situation. The Bank's procurement and disbursement processes and the initial lack of flexibility in the FHP contributed to severe delays in project start-up, similar to many other post-conflict projects. The Bank should adopt a more flexible approach to the design and implementation of projects, especially in post-conflict areas. It should also be flexible in enforcing its procurement rules and should invest in capacity building and training in procurement procedures early in project implementation. Expectations for progress need to be adjusted to the realities of post-conflict situations and Bank-financed projects should endeavor to restore and build capacity at the same time that they make infrastructure repairs. The setting of reasonable expectations also extends to project complexity: the Bank should be more cautious about financing projects that try to address everything, especially in post-conflict situations. 4.2 The Bank must employ a holistic view over the sector and be coherent to and address fundamental constraints to effective delivery such as human resource issues. The Bank must take into account and seek ways to address outside factors such as human resource issues affecting project outcomes as well as sectoral performance. The Bank should seek appropriate instruments (including policy dialogue, sector studies, lending) to begin to address these challenges. Without a motivated workforce for example, the achievements made will not have the expected impact and will reduce the benefits especially for the poor. 4.3 The Bank needs to recognize and be sensitive to the complexities involved in introducing cost-sharing mechanisms. Bank-financed projects should include provision of technical assistance in the development and implementation of cost-sharing schemes. Specific topics that would benefit from such assistance include ensuring that the objectives of introducing such a scheme are clear from the beginning, developing knowledge of the market in which the cost- sharing system is to be introduced-including analysis of willingness and ability to pay, conducting proper analysis of different types of user-fee schemes and their respective implications on incentives, and the design and implementation of accounting and M&E systems, as well as exemption mechanisms to safeguard access for the poor. 4.4 Using the CSA methodology in audits provided value-added in comparison to traditional audits. The participatory methodology employed in this audit was an effective tool for addressing, in a structured way, institutional issues that affect the successful implementation of any project or government policy. It also facilitated the discussion of sensitive issues affecting implementation. This audit demonstrates that available tools (such as CSA) allow the collection of client and beneficiaries' views in a relatively short period. The methodology complements traditional interviews and review of documents. Feedback from the client representatives on the methodology was very positive and the client asked for the material used in the workshops to use it for monitoring and self-assessment in the MOH. Dissemination of this methodology may provide an opportunity for the Bank to build capacity and to provide value-added in M&E for our client countries. 16 Annex A BASIC DATA SHEETS FIRST HEALTH PROJECT (CREDIT 1934-UG) Key Project Data (amounts in US$ million) Appraisal Actual or Actual as percent of estimate current estimate appraisal estimate Total project costs 65.50 60.88 92.9 Credit amount 42.50 40.48 95.2 Co-financing (SIDA and Austria) 16.50 16.90 102.4 Domestic contribution 6.50 3.50 53.8 Cancellation - 5.02 - Date physical components completed 6/95 3/96 Economic rate of return N/A. N/A. N/A. Cumulative Estimated and Actual Disbursements FY89 FY90 FY91 FY92 FY93 FY94 FY95 FY96 Appraisal estimate (US$M) 10.60 22.10 30.50 37.4 39.10 40.50 42.50 Actual (US$M) 3.39 6.68 11.40 21.28 27.81 31.96 37.36 40.47 Actual as percent of appraisal 32 30 37 57 71 79 88 Date of final disbursement: July 31, 1996. An exception was made for one last payment on a civil works contract for work completed by the project closing date Project Dates Original Actual Initiating memorandum January-February, 1988 January-February, 1988 Negotiations May, 1988 May 12-18, 1988 Board approval July, 1988 June 23, 1988 Signing July, 1988 July 11, 1988 Effectiveness September, 1988 January 11, 1989 Closing date March 31, 1996 March 31, 1996 Staff Inputs (staff weeks) Stage of Project Cycle Weeks Preparation to Appraisal 77.4 Negotiations through Board Approval 14.0 Supervision 154.5 Completion 6.1 Total 252.0 17 Mission Data Stage of project Date No. of Staff days Specialized staff skills Implemen- Develop- cycle (monthlyear) persons in field represented tation status ment Types of problems impact Identification/Prepara 2/1988 14 21 Public Health Specialists, IEC tion Specialist, Operations Officers, Ianagement Specialists, krchitect Board to 9/1988 10 14 Public Health Specialists, IEC Critical PIU staff not yet appointed. Work Effectiveness pecialists, NGO Specialist, programs not developed. Delays in rchitect, Management opening special account and signing Specialist, Medical Equipment agreements with the Swedish Expert International Development Agency Lack of coordination among implementing I I_ agencies Supervision 2-3/1989 4 12 Operations Officers, Public HS HS PlU still weak. Work programs not yet Health Specialists, JEC eveloped. Project Coordination Consultant Architect,omitentytnplc Management Specialist ommittee not yet in place 4/1990 7 11 Public Health Specialist, HS HS Delays in plumbing and electrical Economist IEC Specialist orks at Mulago. Delays release of frhiect, Medical Equipment special account funds. Expert 8/1990 4 5 Public Health Specialist, TEC HS HS Delays in procurement of radio Specialist, Architect, Operations transmitters and in preparing the Officer design for Rakai hospital. A number of health education items found not viable and dropped from project 2/1991 3 12 Public Health Specialist, HS HS Major problem - design of Rakai Architect, Operations Officer hospital found to be unacceptable and audits overdue. 9/1991 3 12 Public Health Specialist, US S Project management unsatisfactory, Architect, Operations Officer project coordinator to be replaced. Actions from previous mission not addressed. Audit report incomplete and qualified. TA not effectively used. 311992 3 12 Public Health Specialist, S HS Major cost overruns on civil works as Architect, Operations Officer, result a number of institutional Management Specialist houses at Mulago and for the PIU to be dropped 2-3, 1993 3 24 Public Health Specialist, US S Shortfall in counterpart funds. Need to Architect, Operations Officer reduce scope of work at Rakai and Mulago. Delay on Borrower action on radio transmitters. 4/1993 1 4 Public Health Specialist S S Rakai and Mufago civil works - scaling down. Procurement of radio ransmitters still unresolved. 6/1994 3 14 Public Health Specialist, S S Rakai and Mulago civil works - Management Specialist, scaling down. Procurement of radio Implementation specialist transmitters still unresolved. 2/1995 2 3 Projects Officer, Operations S S Quality of construction at Buhazi clinic Officer unsatisfactory - works scaled down, audits qualified, inadequate maintenance of rehabilitated facilities. 10/1995 5 10 Public Health Specialist, S S Procurement delays - Joint Clinical Architect, Operations Officer, Research Center. Qualified Audits. Management Specialist Earlier counterpart funding shortfall unlikely to be met. 2/1996 2 7 Architect, Operations Officer S HS Counterpart funds backlog not settled. Somea adtqualfications not yet addressed. Some civil works on Rakai and JCRC not yet completed. Completion 8/1996 5 4 Operations Officer, Clinical S S Project activities completed. o___ _ Specialist, Architect _ Other Project Data FOLLOW-ON OPERATIONS Operation Credit no. Amount Board date (US$ million) Sexually Transmitted Infections Project 12630-UG 50.0 April 12, 1994 District Health Services Pilot and Demonstration 13515-UG 45.0 February 7, 1995 Project 18 Annex B The CSA Methodology 1. This audit piloted the use of a participatory approach called the CSA (Control Self- Assessment) methodology for evaluation. CSA has been used extensively within the Bank to assess the strengths and weaknesses in various units and departments. The methodology uses a workshop format wherein participant views are sought using facilitated discussion centering on business/project objectives. Participant votes are then registered on a standard set of assertions. The assertions traditionally have focused on control and other organizational issues. These standard assertions were adjusted to audit-specific needs prior to the Uganda mission. Care was taken in this adjustment not to make the assertions so specific as to preclude benchmarking if this methodology is applied to other Bank projects in the future. OED did this in collaboration with the Bank's controller's office and the Africa region, which has been working internally with this methodology. See Annex C for the assertions that were used. 2. The team conducted three workshops in Uganda: for members of the Ministry of Health, district representatives, and community members in a project district. The method applied consisted of two parts. The first part addressed project-specific issues. The second part addressed organizational issues related to the MOH organization. The method allows members of a group or organization to vote anonymously on whether they agree or disagree with a series of assertions on scale of I to 7, where I stands for "strongly disagree with the statement" and 7 stands for "strongly agree with the statement." The voting was followed by group discussion. In the workshops in Kampala, electric voting devices were used, and the results were immediately shown on a screen as a basis for discussion. In the community workshop, only pens, self-adhesive notes, and a flip chart were used. 3. The CSA approach yielded insights beyond the standard interviews typical of audit missions, and the participants found the process both enjoyable and insightful. The outcome is a more in-depth analysis of the successes and failures of the project and helps to clarify the institutional and organizational (both formal and informal) dynamics that influence implementation of any project. These are areas that, according to OED's review of World Bank HNP lending, receive insufficient attention in Bank projects, but have a major impact on project success or failure. 19 Annex C DATA FROM THE VOTING ON PROJECT EVALUATION ASSERTIONS, MOH AND DISTRICTS MOH Districts Project Objectives - Average results: 4.1 4.5 The objectives of the project were clear and relevant at the time 6.0 5.7 The project (and its components) was well designed 3.8 4.9 The project objectives are in alignment with the overall objectives of and strategies of the sector 4.4 5.3 The issue of demoralized and underpaid health workers was sufficiently addressed in the project 2.4 Na The project provided critical support to strengthening management capacity at all levels 3,8 2.2 Outcomes - Average results: 4.7 5.5 The project satisfactorily met its objectives/the project components were efficiently and effectively completed 4.0 4.9 The project contributed to increasing motivation of health staff in government facilities 2.1 3.5 The project was flexible enough to adapt to outside changes and to suggestions brought forward 4.8 5.1 The studies carried out within the project have been useful and used in policy making 4.1 Na Decentralization is bringing more power and resources to the districts 6.9 7.0 The project 's health education component raised awareness and contributed to behavioral changes 5.5 Na The project contributed to improve health outcomes 4.5 5.5 The project has contributed to improving effectiveness and efficiency of the health care system 5.4 5.5 Demand has increased at the rehabilitated health facilities 4.9 6.8 Sustainability - Average results: 4.1 3.8 The projects contributions are sustained 4.6 5.1 It is clear to everyone including health personnel as well as patients how the fee system works 2.8 4.4 Health staff and government facilities do not feel they need to charge extra 2.3 3.9 Corruption is not a problem in the health care system 2.1 2.4 We are in strong favor of the decentralization process 6.1 Na The AIDS pandemic is being tackled in an efficient way 5.6 Na I feel I had a personal stake in the project 5.4 4.0 The project contributed to the development of a functioning budget system in the MOH 4.8 Na The project improved health information system and the ability to monitor system performance 3.6 Na There are sufficient funds available to maintain the rehabilitated facilities Na 3.3 Institutional Development - Average results: 3.7 6.1 The adoption of Bank's procurement rules has contributed to improving efficiency of government 2.6 Na procurement The project encouraged team approach at the district and regional levels Na 6.7 The project enhanced our capacity to plan, manage and evaluate health program 4.6 Na We have befitted from capacity building funded by the project 4.9 5.5 The project is well integrated with MOH management structures 2.6 Na Borrower's Performance - Average results: 5.2 5.2 We were clear on our role(s) and responsibilities in the implementation of the project 4.6 6.0 I am confident that the project funds are applied in a manner consistent with their intended use 4.9 5.6 There is close collaboration and coordination between the MOH and its partners 5.5 Na Government made sufficient effort to provide counterpart funding 6.0 Na Project implementation was well managed by the PIU/PCO Na 5.0 20 1 feel confident that the government funds are applied in a manner consistent with its intended use Na 4.0 Bank's Performance - Average results: 4.3 5.4 There was sufficient preparation of the project 2.9 5.5 We had sufficient involvement in setting the project objectives 3.1 4.5 The Bank/MOH adequately assessed the recurrent costs implications 3.0 4.3 We were regularly consulted during project implementation 3.9 5.2 Effective and regular supervision was carried out by the Bank 5.6 5-6 The MOH had strong ownership over the project 4.8 6.8 The Bank provided sufficient support to the PIU 5.0 Na The Bank provided important support to policy development 5.4 Na Bank procurement procedures are not an excessive burden on local capacity 3.1 Na The Bank demonstrates commitment to alleviating poverty 5.4 6.2 The Bank plays a major role in coordinating donors 5.6 Na Note: The voting was done on a scale from 1 to 7, where 1=strongly disagree, 4=nor agree nor disagree, 7=strongly agree. 21 Annex D ASSERTIONS RELATED TO THE SELF-ASSESSMENT USED IN THE CSA WORKSHOPS, MOH AND DISTRICTS MOH Districts Team Environment and Responsibility - Average results: 5.0 6.8 A. We have an effective management team 4.3 6.9 B. I am empowered to carry out my job responsibilities 5.1 6.9 C. Our team constantly seeks out ways to improve its performance 4.7 6.9 D. My role and responsibility in the team is clearly understood by all members. 4.6 6.9 E. I enjoy my work 6.5 6.3 Setting Team Objectives - Average results: 4.7 NA A. I have sufficient involvement in setting the team's objectives 4.9 NA B. Our objectives are congruent with the objectives of other departments 4.5 NA Risk Assessment - Average results: 4.6 6.7 A. Our team works collaboratively to achieve common objectives NA 7.0 B. We have an effective method to resolve conflicting objectives between ourselves & other 3.7 6.7 Dept. C. Our team objectives are in alignment with the MOH's objectives and strategies NA 7.0 D. I feel I have a personal stake in our corporate vision 6.1 6.4 E. I have a clear understanding of the risks inherent to our Dept. work/health reforms 3.9 6.6 F. We are effective at periodically reviewing and assessing risks to achieve our objectives NA 6.4 G. Our team's operating plan provides targets against which the team can be measured 4.8 7.0 H. I understand the level of risk acceptable to the department when setting objectives 4.7 NA Communication and Relations - Average results: 4.2 5.3 A. Bad news can be discussed as easily as good news with all members of our team 4.3 4.5 B. Our team can discuss bad news as easily as good news with the next level up 3.7 4.6 C. Our team discusses openly those matters which need to be discussed 3.9 6.9 D. I encourage staff in my unit to openly discuss potential problems 4.9 NA Resources - Average results: 4.7 5.6 A. Our planning process includes estimates of the resources required to meet our objectives 6.0 7.0 A, Resources available to my team are sufficient to ensure high quality output NA 3.3 B. The resources available to my unit will allow us to achieve our planned objectives 3.8 NA C. The organizational structure is effective in providing the support required to meet our 4.4 6.5 objectives Information - Average results: 5.0 5.4 A. The informatipn I need to do my job is easily accessible 5.1 4.9 B. The information systems I used are helpful to me 5.0 NA C. I have confidence in the accuracy of the information that I receive 4.8 6.0 Skills and knowledge - Average results: 5.5 6.5 A. I have adequate skills to do my job 6.3 6.1 B. I understand how my job interrelates to other partners 6.0 NA C. I understand how my job interrelates with other levels of the health care system NA 7.0 C. The overall capability level of our team is sufficient to meet our objectives 4.8 6.4 D. Our team major work processes are efficient and effective 5.0 6.5 Control activities - Average results: 4.9 5.4 A. Our team adheres to policies and procedures NA 6.8 B. Our team has adequate controls to prevent major embarrassment 4.9 6.8 C. I know the standards against which my performance will be measured NA 6.5 D. I receive adequate compensation for my performance NA 1.6 Monitoring, learning and Evaluation - Average results: 3.9 6.0 A. Our team measures the effectiveness of its own performance 3.9 5.7 B. Our team periodically challenges the assumptions upon which our objectives are based 3.3 6.6 C. Our team makes effective changes based on what we learn from our analyses 4.5 6.5 D. Our team does a good job of analyzing our results NA 5.4