Madagascar - Social Fund Iii

 ICRR 11684 Report Number : ICRR11684 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 03/18/2004 PROJ ID : P064305 Appraisal Actual Project Name : Social Fund Iii Project Costs 17.7 31.7 US$M ) (US$M) Country : Madagascar Loan/ US$M ) 15 Loan /Credit (US$M) 28.2 Sector (s): Board: SP - Other social Cofinancing 0 0 services (24%), General US$M ) (US$M) education sector (20%), Water supply (20%), General transportation sector (20%), Central government administration (16%) L/C Number : C3180 Board Approval 99 FY ) (FY) Partners involved : Closing Date 12/31/2001 06/30/2003 Prepared by : Reviewed by : Group Manager : Group : Soniya Carvalho Nalini B. Kumar Alain A. Barbu OEDST 2. Project Objectives and Components a. Objectives Project development objectives were : (i) improved access of poor rural population to social and economic infrastructure; (ii) on a pilot basis, empowerment of poor rural communities and /or communes to identify, organize, manage funds and implement subprojects responsive to community needs in order to achieve decentralized social and economic development; (iii) employment creation; and (iv) improved capacity of the private sector : local small contractors and consultants, artisans, skilled labor, and NGOs . In response to a series of cyclones in Madagascar, the Bank approved a supplemental credit of US$ 18million in July 2000. There was no change in the project's objectives or components. The project implementing agency, the Social Development Fund (FID), was to continue its normal activities, however, targeting communities in areas stricken by the cyclones . b. Components The project consisted of three components : (i) Rehabilitation/Construction of Communal Infrastructure (total appraisal project cost including supplemental credit US$ 35.5million, actual project cost US$ 27.1million); (ii) Capacity Building (total appraisal project cost including supplemental credit US$ 1.5million, actual project cost US$ 1.2million); and (iii) and FID Management and Operations (total appraisal project cost including supplemental credit US$ 5.2million, actual project cost US$ 1.5million). c. Comments on Project Cost, Financing and Dates The initial cost of the project was about US$ 17.7 million and IDA credit was US$15.0million. A supplemental IDA credit of US$18million was approved to finance cyclone rehabilitation activities, and the total project cost for the supplemental was US$24.0 million. With the original and supplemental credits, the total IDA contribution to the project was US$33million and the total project cost was US$ 41.7million. At project closure, IDA disbursement for both credits amounted to US$31.7million (US$14.3 million for the initial credit and US$17.4 million for the supplemental credit). The project closed one and a half years later than originally planned . 3. Achievement of Relevant Objectives: Under the supplemental credit (which comprised over half of the total IDA disbursement ), FID was supposed to target "communities in areas stricken by the three cyclones " (ICR page 4). In actual practice, "the supplemental credit was distributed across the entire country ..." (ICRpage 10). The MOP had estimated the total infrastructure and asset damage at US$137million of which US$61million was public and US$76million was private (MOP page 2). By spreading IDA's US$18million (which was less than a third of the total requirement for addressing the damage in public infrastructure ) across the entire country rather than focusing it on cyclone -affected areas, the impact of the supplemental credit was diluted and the extent to which the supplemental credit was used for the purposes for which it was approved by the World Bank's Board of Executive Directors is clearly questionable.  The project's target groups were defined broadly as "poor rural population" or "poor rural communities/communes" (PAD page 3) despite the fact that poverty targeting was recognized as an issue in the predecessor Social Fund II project and "there was a decision to spread the program to the poorest, most isolated regions" at the mid term review of that project (Social Fund II project ICR page 4). This raises questions about the relevance of the poverty targeting objectives of the Third Social Fund project . In any case, the ICR fails to provide evidence that the project reached the "poor rural population" or "poor rural communities/communes" and notes "allocations to poorest districts decreased from 21.5 percent to 13.5 percent of total investments during the period of 1997 to 2001. Close to 45 percent of the funds were allocated to the two richest quintiles of districts...FID could have done more to ensure that more objective and transparent criteria (such as accessibility or density of existing services ) was used or that subprojects were distributed more equitably across districts and communes " (ICR page 10). While the objective of improving access to social and economic infrastructure appears to have been largely met with output targets more or less achieved and before -after analysis of outcomes pointing in a positive direction (see section 4), it is not clear how much of the improvement in outcomes can be attributed to the project . The ICR notes "It is more difficult to assess FID's interventions in terms of impacts given the lack of reliable baseline data and comparison group data " (ICR page 5). With regard to the objective of local capacity building, while the ICR reports that this objective was met, no analysis is provided about the extent or nature of empowerment or the degree of the project's responsiveness to community needs. Transparency and objectivity are key requirements of a process of empowerment, but given that "sub-project selection, carried out by provincial steering committees, often followed subjective criteria " (ICR page 10), it is questionable how well the empowerment objective was actually achieved . The ICR notes that the pilot transferred procurement responsibilities for subprojects below US$ 7,500 to local associations and communities and assisted them in dealing with companies and contractors, and a sense of ownership was created in the community (ICR page 6). Regarding the employment generation objective, the project fell short of its temporary employment generation estimate by over 1 million person days. The estimate of 1.9 million person days under the supplemental credit was made in 2000 when only 35 percent of the estimate under the initial credit had been achieved . The claimed reduction in the need for employment generation because the "affected rural population needed to return to work in the fields" and the trade-off between employment generation and technical quality should have been foreseen given that the Bank had previous experience in Madagascar with cyclone and social fund projects . The objective of increasing the capacity of private sector contractors and consultants was largely achieved . 4. Significant Outcomes/Impacts: 1. While no with/without analysis was conducted and the observed outcomes cannot be attributed to the project, before/after analysis for six types of infrastructure projects in the six project provinces showed improvements in student enrollments, average rate of exam success in participant primary and secondary schools, total number of consultations per year in beneficiary health posts, total number of children vaccinated, use of the irrigation systems and yields per year, and total number of vendors in markets installed (ICR pages 5-6). 2. The project contributed to strengthening the managerial and execution skills of local contractors and other private sector partners (ICR page 6). 5. Significant Shortcomings (including non-compliance with safeguard policies): 1. Under the supplemental credit (which comprised over half of the total IDA disbursement ), FID was supposed to target "communities in areas stricken by the three cyclones " (ICR page 4). In actual practice, "the supplemental credit was distributed across the entire country ..." (ICRpage 10). The MOP had estimated the total infrastructure and asset damage at US$137 million of which US$61 million was public (MOP page 2). By spreading IDA's US$18 million (which was less than a third of the total requirement for addressing the damage in public infrastructure ) across the entire country rather than focusing it on cyclone -affected areas, the impact of the supplemental credit was diluted and the extent to which the supplemental credit was used for the purposes for which it was approved by the World Bank's Board of Executive Directors is clearly questionable . 2. The project's target groups were defined broadly as "poor rural population" or "poor rural communities/communes" (PAD page 3) despite the fact that poverty targeting was recognized as an issue in the predecessor Social Fund II project and "there was a decision to spread the program to the poorest, most isolated regions " at the mid term review of that project (Social Fund II project ICR page 4). This raises questions about the relevance of the poverty targeting objectives of the Third Social Fund project . In any case, the ICR fails to provide evidence that the project reached the "poor rural population" or "poor rural communities/communes" and notes "allocations to poorest districts decreased from 21.5 percent to 13.5 percent of total investments during the period of 1997 to 2001. Close to 45 percent of the funds were allocated to the two richest quintiles of districts ...FID could have done more to ensure that more objective and transparent criteria (such as accessibility or density of existing services ) was used or that subprojects were distributed more equitably across districts and communes " (ICR page 10). 3. Methodological weaknesses in the monitoring and evaluation system reduced the project's ability to learn adequately from a long standing project experience . 4. "...there was no cost-benefit analysis was performed during the preparation although FID had been operating since 1993 and data had been collected earlier " (ICR page 5). The discussion of costs on page 7 of the ICR refers, for the most part, to the earlier Credit, and in any case provides a mixed and ambiguous picture of efficiency . 5. Despite being the third in a series of projects, the design of this project did not adequately reflect the multi -donor environment in the country nor the lack of a clear decentralization strategy both of which raise issues for relevance . In addition, there is some suggestion in the ICR of an adverse impact on project outcome "..FID's progressive move towards community driven and local government executed projects was not "framed" by a clear and coherent national strategy on decentralization . Despite efforts to coordinate implementation procedures at the technical level, this contributed to the fragmentation of local development programs in Madagascar " (ICR page 9), and "[The lack of donor coordination] may have contributed to the overall fragmentation of community driven development (CDD) initiatives that can be observed in Madagascar today " (ICR page 5). 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately OED rates a project moderately Unsatisfactory unsatisfactory when the project meets its major relevant objectives with major shortcomings. See sections 3 and 5. Institutional Dev .: Substantial Modest While the project facilitated the emergence of a large number of private contractors and provided communities an opportunity for self-help, weak donor coordination and inadequate attention to the institutional context (which lacked a coherent national strategy on decentralization) contributed to the fragmentation of local development programs in the country (ICR pages 5, 9, 15). There were also coordination problems between the FID and line ministries (ICR page 9). Sustainability : Likely Non-evaluable There is lack of sufficient information to determine subproject sustainability . The ICR concluded that "While FID collected abundant information on project implementation, the lack of counterfactual and baseline data presents a number of challenges for analyzing the impact ....The sustainability issue cannot be properly addressed without empirical evidence . Surveys on the operation and maintenance of the completed subprojects should be performed regularly and remedial actions should be taken in a timely manner" (ICR page 14). Bank Performance : Satisfactory Unsatisfactory Both quality-at-entry and quality-of supervision were deficient: "no cost-benefit analysis performed during preparation although FID had been operating since 1993 and data had been collected earlier" (ICR page 5), weak targeting to cyclone-affected areas and poorer districts (ICR page 10); lack of follow-up to the framework for coordination of the FID with other World Bank-financed projects (ICR page 12); supervision teams inadequately multisectoral (ICR page 12); questionable supervision of FID contracts allowing contracts for senior staff at central and regional levels to be extended with exponential increase in salaries completely out of line with inflation and comparator salaries (ICR page 13); a methodologically deficient M&E (ICR page  10); and weak coordination with other donors working at the community level (ICR page 9). These weaknesses are all the more significant considering that this project was the third in a series which provided an opportunity to learn from experience and avoid past mistakes . Borrower Perf .: Satisfactory Unsatisfactory Weak targeting (ICR page 10), subjectivity in subproject selection by provincial steering committees (ICR page 10), dubious financial policies with an exponential increase in senior central and regional staff salaries completely out of line with inflation and comparator salaries (ICR page 13), methodologically deficient M&E (ICR page 10), and weak coordination with other donors working at the community level (ICR page 9). Quality of ICR : Satisfactory NOTE: NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. 7. Lessons of Broad Applicability: Drawing on the ICR, some lessons are : 1. To be a useful management tool, and identify areas of weakness /strength for remedial action/replication, the M&E system has to be based on a sound methodology . 2. Creating maintenance committees is not sufficient to ensure sustainability of community subprojects . Measures to ensure that these committees continue to function after subproject completion should be taken . 3. Cross-sector coordination is critical among Bank sectoral staff, among country sectoral staff, and between Bank and country sectoral staff in multisector operations . 4. Coordination is also needed among the various donors working at the local level with respect to harmonizing their project requirements (e.g., for beneficiary contributions, subproject cycles etc ). 8. Assessment Recommended? Yes No Why? To verify the ratings. 9. Comments on Quality of ICR: Satisfactory overall but greater analysis of the extent of empowerment resulting from the pilot and of the degree of the project's responsiveness to community needs would have been useful .There are discrepancies in project cost figures in Annex 2 and section 5.4 of the text and also in the discussion of coordination issues .