St Lucia - Hurricane Tomas Emergency Recovery Project

ICRR 14926 Report Number : ICRR14926 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/23/2016 Country : St. Lucia Project ID : P125205 Appraisal Actual Project Name : Saint Lucia Hurricane US$M): Project Costs (US$ 15 13.4 Tomas Emergency Recovery Loan L/C Number : Loan/ US$M): Loan /Credit (US$ 15 13.4 Sector Board : Urban Development US$M): Cofinancing (US$ Cofinanciers : Board Approval Date : 03/10/2011 Closing Date : 03/31/2014 09/30/2014 Sector (s): Rural and Inter-Urban Roads and Highways (40%); General education sector (20%); Health (15%); General water; sanitation and flood protection sector (15%); Central government administration (10%) Theme (s): Natural disaster management (80%); Other urban development (20%) Prepared by : Reviewed by : ICR Review Group : Coordinator : Stephen Hutton George T. K. Pitman Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The objectives as stated in the Project Paper (p. 6) and the Financing Agreement (p. 4) are identical: "to support the recovery and reconstruction of the Recipient in the aftermath of Hurricane Tomas ." The objectives were not revised . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets ? Yes Date of Board Approval: 10/10/2013 c. Components: The project had four components : 1: Support for Early Recovery (appraisal estimate US$1.50 million, actual cost US$1.50 million). This supported retroactive finance of goods , technical advisory services , and emergency operating costs such as additional transport costs and fuel, electricity bills, staff overtime, and equipment rental. 2: Institutional Strengthening and Hazard and Risk Analysis (appraisal estimate US$1.50 million, actual cost $0.85 million). This supported institutional capacity building of key government ministries and agencies to improve their ability to carry out risk analysis , through integrating hazard and risk reduction information into national development policy and planning processes , carrying out risk assessment studies , and reviewing the national disaster management plan and improving disaster management capacity . 3: Reconstruction and Rehabilitation of Damaged Critical Public Infrastructure (appraisal estimate US$11 million, actual cost $10.0 million). This financed civil works and technical advisory services for reconstruction and rehabilitation of damaged infrastructure in transport , health, and education sectors. 4: Project Management and Monitoring Support (appraisal estimate US$1.00 million, actual cost $1.00 million). This supported goods, technical advisory services , and operating costs for a central Project Coordination Unit . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Cost : Actual costs differed from original cost estimates because the actual of cost of some activities was higher Cost: than originally expected and because some activities were not completed . The cost estimates at appraisal were highly uncertain as they were made on the basis of the limited information that was available in the wake of the hurricane. During implementation, it became apparent that some works would be more expensive than expected : damage to works from flooding in 2012 and 2013 increased the cost of some works in part because this meant that more expensive bridge options would be needed in order to be sufficiently resilient , and decisions were made during implementation that school and hospital /clinic upgrades should be more extensive in order to be sufficiently resilient . However, the total project cost was slightly lower than planned as not all activities were completed . Some activities were dropped in restructurings (and a small number of additional activities were added ). But even following restructuring not enough subprojects were prepared to utilize all available funds due to capacity limits in some government departments Less was spent on institutional strengthening than expected in part because a planned risk assessment study was not carried out . Less was spent on infrastructure than expected as some school upgrades were not carried out due to implementation delays . Financing : The project was financed by an SDR 9.7 million ($US 15 million) IDA credit. At the time of approval this was the largest project ever financed by the World Bank in St Lucia . At closure, SDR 1.17 million ($US 1.6 million) was cancelled from the loan. Borrower : No contribution was expected due to the tight fiscal constraints on the government , and none was provided. However, some of the delayed infrastructure was later completed at the government cost . Dates : The project was restructured twice (in October 2013 and March 2014) to adjust outcome targets due to cost overruns and implementation delays . The project was extended to close 6 months later than originally expected in order to allow some activities to be completed . The Bank decided not to extend the project a second time . Activities that were dropped from this project were picked up by the Bank -financed Disaster Vulnerability Reduction Project (P127226). 3. Relevance of Objectives & Design: a. Relevance of Objectives: The broad objectives being pursued were very relevant to St Lucia 's national development strategy , its disaster recovery strategy, and the Organization of Eastern Caribbean States (OECS) Regional Partnership Strategy with the Bank. The regional strategy includes Building Resilience as one of its two main pillars , with a specific subpillar on strengthening climate resilience . Hurricane Tomas was a severe event , causing an estimated $336.2 million in damage (43.4% of GDP), greater than any previous disaster in St Lucia . Infrastructure was the most severely affected sector , representing 43% of the total damage. Landslides blocked roads and damaged other key economic assets , so a rapid response to restore service was a critical part of reducing the overall negative economic impact of the disaster . The Bank had a long and ongoing program of disaster risk management in St Lucia and the Organization of Eastern Caribbean states and active relationships with the key government agencies , meaning that the Bank had a comparative advantage in providing support in reconstruction . The broad objective remained relevant at closure , as St Lucia continues to face serious threat from natural disasters , as shown for example by severe flooding in December 2013. However, the specific project objectives could have been framed more usefully . The objectives could have been highly relevant if they had been stated in terms of actual objectives being pursued : to restore economic sectors following Hurricane Tomas, to improve disaster management capacity , and to reduce the vulnerability to future disasters. The relevance of objectives are rated Substantial b. Relevance of Design: The design was relevant. Support for short-term disaster response and recovery was balanced with infrastructure reconstruction that would reduce long term risks and activities to build capacity for disaster risk management and improve national disaster response coordination and planning . Retroactive financing of emergency supplies was relevant . It allowed for rapid disbursement of some funds , assisting the government with short-term emergency needs and providing much -needed liquidity in the wake of the disaster . Repairing and reconstructing key infrastructure would help to reduce the impact of the disaster , by building back at higher design standards would contribute to long term risk reduction . Replacing roads and bridges would help to keep critical transit routes open in future storm events ; schools function as emergency shelters and so rehabilitation of schools to reduce disaster risk helps to make communities more resilient to disaster events , and hospital and clinic upgrades help to protect highly vulnerable patients . Reviewing the disaster risk management plan would help to incorporate lessons learned from the response to Hurricane Tomas , and risk assessment and institutional capacity building could help to ensure that disaster risks would be considered in a broad range of development . One weakness in the design is that it did not incorporate a plan for maintenance of the constructed assets . Maintenance was known at the time of approval to be a serious problem in St Lucia due to funding and capacity limitations, and the lack of a plan for maintenance presents a sustainability risk . The relevance of design is rated Substantial. 4. Achievement of Objectives (Efficacy): The objective of the project here was : "to support the recovery and reconstruction of the Recipient in the aftermath of Hurricane Tomas." Though component one of the project was focused on the provision of early support for Hurricane recovery, it is incorporated into the broader objective here as the majority of the work was built on a recovery strategy and reconstruction efforts . Outputs : Reimbursement to the government for $1.5 million of emergency supplies (fuel, material for emergency works, equipment rental, staff overtime) freeing up government resources for other disaster recovery needs . A number of studies and assessments were carried out : Three hydraulic studies (studies of river and surface runoff) were produced in areas facing high risks of landslides and flooding . The national emergency management plan was reviewed and updated (based on the experience of the disaster ) for the first time since its creation 12 years prior. A geodetic survey was carried out . A planned risk assessment study was not carried out. Eighteen staff were trained in Geographic Information Systems (GIS) techniques. Communication equipment was purchased, increasing communication capacity in the wake of a disaster . This was important as communication systems had failed following Hurricane Tomas . Two bridges that had sustained damage in the storm were rehabilitated (though the second was finished after project closure). 11 schools were retrofitted to be more resilient to storms with improved roofs , windows, drainage, and water supply. The original target of 27 was based on unrealistic cost estimates and was revised downwards to 10 in October 2013. Four health facilities were retrofitted to be more resilient to storms with improved roofs, windows, and other facilities. The original target of 5 was based on unrealistic cost estimates and was revised downwards to 4 in October 2013. Some school rehabilitation works that were dropped from the project were financed separately by the government using its own resources . Outcomes : The ICR argues (page 19) that improvements to the emergency management plan mean that the response for future emergencies will be improved, that geodetic surveys will provide information for siting of buildings based on disaster vulnerability, and that flood risk studies will reduce disaster exposure . It also notes that as the studies were completed shortly before project closure , that they were not yet being used in new government policies , but that there were some reports that they were being used in decision -making for new construction. The ICR also argues that communication equipment has contributed to empowerment of district disaster committees and to the strengthening of collaboration between users . The project also likely contributed to increasing attention to disaster risk management in a number of government institutions . Unfortunately, both of these claims are not adequately supported by evidence making it difficult to gauge how much influence the inputs really had on reconstruction efforts and strategy . The infrastructure work, planning and implementation was far more important to identifying real progress on the hurricane recovery. School retrofits meant that 6,080 additional students had access to disaster resistant buildings . The schools are also used as emergency shelters in the wake of a disaster , so this understates the number of beneficiaries. Hospital retrofits meant that facilities with 145 hospital beds had access to disaster resistant buildings . Construction of temporary bridges and then a permanent bridge meant that transport was restored for 28,916 daily users (out of a national population of roughly 175,000). Significantly, these infrastructure reforms were instituted following guidance from the Bank and were informed by good practice disaster resilience practices . Works that had been completed survived severe floods in 2013 without damage, demonstrating resilience to heavy rainfall (though the buildings have not yet faced severe winds ). Some works that were not complete suffered some damage or had to be redesigned due to changed risk exposure following these floods (as river beds had changed ). However, the floods also demonstrated that some schools were located in flood -prone areas and so would remain vulnerable to future disasters . Bank support and guidance with regards to this exposure was essential to a more appropriate (though also more expensive ) response to disaster risk mitigation , though tracking how much influence the planning and system reform issues would have been useful to understand their impact on longer term planning . Efficacy is rated Substantial . 5. Efficiency: Economic and Financial Efficiency : No economic analysis was prepared at appraisal , as the project was approved early in the response and reconstruction process and before the detailed designs of projects had been approved. An economic analysis carried out at closure considered the returns on bridge and school rehabilitation and reconstruction. For bridges, the analysis considered the value of rehabilitated work equivalent to the avoided costs (in terms of fuel cost and travel time of detours ). For school facilities the benefit was assumed to come from the extended life of the buildings and deferred replacement cost . These estimates do not include other benefits such as the reduction in the impact of disasters from having more secure schools to serve as evacuation centers . This led to an estimated rate of return of 30.0% for bridge works and 10.4% for school works, for a weighted average of 21.7%. However, while cost figures were clear the methodology for calculating benefits was not clear . For other elements of the project economic returns were not calculated . Cost comparisons to similar works elsewhere in the region were not reported. According to the US Army Corps of Engineers expert who reviewed the design , for the most expensive asset (a bridge costing $US 4.1 million) costs were at the high end of what similar bridges might cost elsewhere , but this was justifiable given generally high costs of construction in remote islands , where materials have to be imported (ICR page 20). Institutional and Administrative Efficiency : The overall number of works constructed was less than initially expected as cost overruns on some items meant that not all activities could be completed , and that there was insufficient capacity in some line ministries to prepare enough subprojects to fully utilize all project funds . However, these were not necessarily indications of weak efficiency . In some cases, the initial cost estimates were done on the basis of replacement cost of the existing infrastructure , but it turned out during implementation that a more advanced but expensive structure would be required in order to be sufficiently resilient to future severe storm events . In other cases, additional damage to in flood events in 2012 and 2013 increased the cost of works . Design of a key bridge had higher than costs than initially expected but was expected to have lower long term maintenance costs . School upgrades were redesigned to focus on more extensive improvements to a smaller number of schools . However, there were significant implementation delays in some outputs . Some implementation delays were due to additional damage caused by the 2013 storms. Other delays followed ministerial reorganizations following elections in 2011 and a subsequent lack of clearly delineated responsibilities . Weak technical capacity for works in some line ministries further contributed to delays . The redesign of a major bridge also meant that it could not be completed as of project closure, though it was completed two months later . The ICR also notes that there was a moderate delay between approval and effectiveness due mostly to country requirements for approving funding . Government officials interviewed by IEG during a mission to St Lucia noted that one cause of delays had been that staff within the infrastructure ministry were initially overwhelmed by the volume of works to be supported , which were collectively larger in scope than for previous projects . Efficiency is rated Modest. ERR )/Financial Rate of Return (FRR) a. If available , enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 21.7% 63.5% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: With substantially relevant objectives and design , substantial efficacy, and modest efficiency, outcome is rated Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The main threat to the achievements of the project is that of insufficient maintenance funding . The ICR notes (page 12) the lack of a government maintenance strategy and (page 24) the financial and capacity constraints for maintenance in line ministries. It notes that this is a common problem in the Caribbean , and is not specific to this project. Many community facilities rely on fund -raising activities and community volunteers for upkeep . The ICR also noted that the project's efforts for improved disaster risk management are challenged by weak coordination between line ministries. The Bank is providing continued support on disaster risk management through the $68 million disaster vulnerability reduction project, which will focus on scaling up the impact of the Hurricane Tomas project . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The Bank responded rapidly to the country 's request for assistance . Hurricane Tomas hit St Lucia on October 31 2010; the Bank conducted an initial damage assessment mission in early November followed by an emergency project appraisal mission in December , project approval in March 2011 and effective in August 2011. The Bank worked rapidly with the client to identify priority infrastructure needs and key areas where capacity building would be needed. Because of the limited preparation time and the priority on a rapid response , some detailed aspects of the design had not been determined at appraisal . The Bank team recognized that the rapid response would lead to some incompleteness in design, and made the deliberate decision to move ahead with preparation based on preliminary cost estimates rather than delay approval until a wider set of information was available , judging that rapid support was more important than a more complete design . It was expected that the Bank would support the government in finalizing many aspects of detailed project design during implementation . Targets for some indicators were not included as detailed information on damage and costs was not yet available (as detailed damage assessments had not been completed ). Initial cost projections for works were made based on limited general information. Nevertheless, these were arguably acceptable deficiencies given the need for rapid response and the limited information available at the time of approval . However, the original project cost estimates did not include sufficient physical or price contingencies , so cost overruns meant that activities had to be dropped from the original project design . The Bank may have also overestimated the capacity of the education ministry to implement a large number of school rehabilitations within the time scale of an emergency operation . These constitute a moderate weakness in quality at entry . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The ICR reports that the Bank carried out frequent missions and regular support to clients in the Project Coordination Unit (PCU) and implementing agencies. The Bank facilitated technical support from the US Army Corps of Engineers. The ICR notes that the Bank carried out regular supervision to ensure compliance with national building codes . The Bank worked with the client to restructure the project and adjust targets when it became clear that the original target values of indicators could not be reached due to cost overruns . The ICR notes a missed opportunity to improve the results framework during the first restructuring ; the framework could also have been improved earlier, prior to midterm review, to make up for the drawbacks of the rapid project preparation process , but this did not occur.This does detract from the positive aspects of the project given the acknowledged rush to approval in the design phase and the ill -informed and overly ambitious targets set at the beginning . The Bank provided a range of support to the PCU and others . It conducted training to strengthen safeguards management capacity, it provided support on increasing the technical engineering capacity on the PCU , it carried out an implementation support workshop to address challenges faced by implementing agencies and the private sector, it conducted a procurement workshop for local suppliers and contractors , and it conducted a range of capacity building activities on financial management including a workshop for technical staff . The Bank team worked with the implementing agencies to reduce the risk of procurement issues , which faced challenges in attracting a sufficient number of bidders for procurement requirements . In April 2013 the Bank adopted a simplified procurement procedure , which eased procurement requirements (thresholds, bidder number requirements, enabled registered individuals to bid as contracting firms , etc.) and so enabled faster project implementation for the last year of the project . However, there were a number of important shortcomings to supervision . First, the transition over time from an emergency response operation to a recovery project was not well articulated and the emphasis on cost heavy reconstruction of the two bridges is not adequately outlined with regards to strategic direction and prioritization . Second, the adjustment to key indicators and to the scope of works is not well articulated in the ICR . While the point is made that costing was problematic due to the speed of design and approval , it is disappointing that the Bank team did not do more to explain and record the necessary adjustments to costs and the design elements necessary to resilient infrastructure improvements . More work in the articulation and tracking of these issues would provide useful lessons for subsequent projects in the region . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR provides relatively little information on performance of the government in supporting the project (as opposed to the PCU and the implementing agencies ). It notes that the government was actively involved in the identification, prioritization, and preparation of project activities , and remained committed to the project activities throughout the project. It also notes (page 11) that the government could have been more proactive in supporting collaboration and information sharing across implementing agencies , and that government restructuring in 2012 contributed to implementation delays . The ministerial reorganization and lack of attention to clarifying responsibilities of key agencies led to coordination and implementation problems and there are remaining questions with regards to the institution of planning reforms in the longer term . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The project was implemented through a central Project Coordination Unit (PCU) in the finance ministry which handled reporting, procurement, financial management, and other responsibilities, and supervised implementation through line ministries and agencies (ministries of infrastructure, education, health, and physical development, along with the emergency management organization ) that provided technical support . Each line ministry also had its own designated focal point . The existence of a central PCU with adequate staffing helped to ensure efficient and effective implementation , and to reduce the burden on line ministries for complying with Bank procedures and policies, while focal points helped to support buy -in from the line ministries. There were a number of implementation delays , as described above, due in part due to lack of capacity in the line ministries to handle what for the country was a very large project , and in part to weaknesses in collaboration between implementing agencies. In particular, a lack of technical engineering capacity in the education ministry contributed to delayed implementation of school rehabilitation projects . The PCU attempted to compensate for this by hiring consultants to fill this gap , though they were not fully successful . Implementing Agency Performance Rating : Moderately Unsatisfactory  Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E system was very simple, with a total of 8 indicators, nearly all tracking production of outputs . This was in part the result of the emergency situation and rapid design project , and was designed to avoid overwhelming limited M&E capacity. One indicator usefully assessed the number of people benefiting from safe structures . In future, indicators could be more useful in assessing the impact of capacity building and institutional works ; rather than measuring the number of studies being produced or people trained , indicators could track how these were contributing to planning processes . The ICR provides a number of other useful suggestions of how indicators could have been improved (page 13). Some targets were established at midterm review rather than at appraisal , as it was not possible to immediately estimate the number of beneficiaries that would be assisted . Targets for the numbers of facilities to be upgraded were set at appraisal, but these targets were revised as initial cost estimates underestimated actual costs . Baseline figures for some indicators were mis -stated in the original M&E design (and should have been zero ). b. M&E Implementation: Indicators were used primarily to track progress on the production of outputs and did not usefully institute good practice on tracking progress against planning and policy reforms , nor in identifying what transformations might be expected for those trained and applying new practices to their work . There is little information on how the implementation and structure of the M &E system worked and nothing on what kind of adjustments were considered and useful to the project over time . c. M&E Utilization: M&E was utilized to guide support towards areas where implementation progress was lagging , but it is unclear how the M&E was used to inform strategic decision making or in how the restructure could better respond to the gaps in recovery efforts. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was assigned category B under OP 4.01 on Environmental Assessment due to the possibility of modest environmental impacts from small scale infrastructure works . Though land acquisition was not originally envisioned , the safeguard on Involuntary Resettlement (OP 4.12) and Natural Habitats (OP 4.04) were triggered as a precautionary measure. The ICR notes that Bank supervision missions worked to ensure safeguard compliance . An environmental and social screening assessment framework was produced , an environmental management plan created for a previous disaster risk management project supporting similar outputs was used . The ICR states that "no social and environmental safeguards issue were highlighted ", but does not otherwise comment on safeguards implementation except to note adequate management of asbestos issues at two sites . The safeguard on Physical Cultural Resources (OP 4.11) was also triggered. The ICR does not address how this safeguard were handled. b. Fiduciary Compliance: Financial management . The ICR states that there were no major financial management issues during implementation. It does not state whether or not audits were on time or unqualified . Procurement . There were a number of challenges with procurement in some implementing agencies , due to a lack of capacity in those agencies and the burden of Bank procedures . These were eased after the Bank 's adoption of simplified procurement procedures for small states . Specialized technical assistance on procurement c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Satisfactory Moderately There were issues with regards to the Satisfactory costings and capacity in design and in regards to appropriate project adjustments in supervision. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR includes a number of useful lessons , including: It is difficult to carry out major infrastructure works within the standard three -year timescale of an emergency response project. This project supported assets such as a major bridge , which was not completed during as of project closure. Initial cost estimates of rehab/retrofitting work need to be made conservatively to avoid a scenario where the scope of the project needs to be dramatically reduced . In this project, an insufficient level of contingencies meant that some planned activities had to be dropped when it was determined that other works would be more expensive than planned. Maintenance planning and budget needs to be part of rehabilitation work . In this project, sustainability is at risk because of a lack of sufficient maintenance budgets given tight fiscal constraints . World Bank procurement procedures need to take into account the limitations and realities of the small island state context. Expectations that all bidders will be companies or that it will be possible to generate 3 or more bids for works may not be realistic in a small state context . In this project, simplification of procurement procedures helped to enable faster implementation . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: Overall the ICR provided a very good description of project implementation including a candid description of challenges and efforts taken to overcome them . It acknowledges the weaknesses of the results framework and indicators, and makes an effort in the text to provide some evidence on impact beyond that of the results framework . It provides useful lessons that are relevant for other projects . The ICR included the Borrower's ICR which was useful in providing the Borrower 's perspective of the project . The ICR also included a map showing the location of project sites and a number of photos of specific works constructed under the project. This helped the reader to gain a better understanding of the project 's outputs. No beneficiary assessment was carried out , which could have been a useful addition in assessing the perspective of beneficiaries on improved infrastructure , rather than focusing just on the government perspective . a.Quality of ICR Rating : Satisfactory