China - Hebei Urban Environment Project

 ICRR 13076 Report Number : ICRR13076 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/26/2009 PROJ ID : P045910 Appraisal Actual Project Name : Hebei Urban US$M ): Project Costs (US$M): 293.3 290.4 Environment Project Country : China Loan /Credit (US$M): Loan/ US$M ): 150.0 150.0 Sector Board : WAT US$M): Cofinancing (US$M ): Sector (s): Sewerage (66%) Water supply (33%) Sub-national government administration (1%) Theme (s): Water resource management (25% - P) Pollution management and environmental health (25% - P) Environmental policies and institutions (25% - P) Access to urban services and housing (25% - P) L/C Number : L4569 Board Approval Date : 06/27/2000 Partners involved : Closing Date : 06/30/2007 06/30/2008 Evaluator : Panel Reviewer : Group Manager : Group : Roy Gilbert Kris Hallberg Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: To assist Hebei Province in its long -term urban environmental services improvement program : (a) to recover from environmental degradation of its water and land resources; (b) to provide an adequate supply of safe water to its growing urban population and industries; (c) to strengthen the institutional and financial capacities of utilities . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) Water supply Tangshan, incl : (i) raw water intake pumping station and treatment plant; (ii) 19 kms of raw water transmission lines and 10 kms of distribution mains (appraisal cost US$35.8m.; actual cost US$37.9m.) (b) Water supply Handan, incl: (i) raw water intakes and 57 kms of raw water transmission lines; (ii) expansion of Tiexi treatment plant; (iii) rehabilitation of about 54 kms of distribution network in Handan (appraisal costs US$33.5m.; actual costs US$48.4m.). (c) Wastewater management Shijiazhuang, incl : (i) construction of treatment plant; (ii) construction of about 40 kms of trunk mains and secondary wastewater collection network (appraisal costs US$88.3m.; actual costs US$123.2m.). (d) Wastewater management Tangshan, incl : (i) construction of treatment plant; (ii) construction of secondary wastewater collection network of 20kms (appraisal cost US$23.8m.; actual cost US$31.3m.). (e) Wastewater management Handan, incl : (i) construction of wastewater treatment plant and secondary network of 40km; (ii) rehabilitation and expansion of North Lift sewage pumping station; (iii) expansion of sludge treatment facility at East Wastewater Treatment Plant (appraisal cost US$31.5m.; actual cost US$30.7m.). (f) Environmental pollution control, incl : program of actions to reduce pollution loads from selected industries (appraisal cost US$0.7m.; actual cost US$0.0m.). (g) Institutional strengthening, incl : strengthening the capacity of utility companies in (i) project and financial management and utility operation and (ii) construction management and supervision (appraisal cost US$7.7m.; actual cost US$7.2m.). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Final project cost were very close to those estimated at appraisal and the Bank loan was fully disbursed . A mid-term review was carried out in November 2004 as planned. Although it found the project largely to be on -track, it did lead to more resources being allocated for additional rehabilitation works in the same municipalities . The preparation and implementation of them was a time-consuming process (the ICR does not explain why) that required an extension of the project closing . Necessary reallocations were approved as the RVP level . Since the project objective did not change, nor a fundamental restructuring of the project design, a legal amendment for approval by the Board was not considered necessary. 3. Relevance of Objectives & Design: Substantially relevant : Project objectives were substantially relevant to China's priorities of cleaning up river stretches in and near urban areas that had become highly polluted through untreated sewage of rapidly growing urban agglomerations. Managing water pollution is a key component of China's 11th Five Year Plan (2006-2010). The corresponding 2006 Bank Country Partnership Strategy (CPS) also highlights the priority of curtailing water pollution. There were serious deficits in environmental services in Hebei, whose capital city, Shijiazhuang, treated only 16 percent of its wastewater. The project's focus upon pricing water was also substantially relevant to the priority given to the pricing of public services more generally by the 2006 CPS. The project design was also substantially relevant for including components, particularly wastewater management and environmental pollution control that directly curb pollution . Relevance would have been higher had it been made clearer in the design that tariff increases were sufficient to ensure financial sustainability . 4. Achievement of Objectives (Efficacy): The project helped Hebei Province strengthen its long -term urban environmental services, but the thinness of data to measure achievements undermines the performance ratings . (a) Recovery of environmental degradation modestly achieved : wastewater treatment plants in Handan, Shijiazhuang and Tansgshan are now fully operational and the ICR reports that they reduce the discharge of contaminants into local water courses by 121,990 tons of COD; 54,100 tons of BOD; 38,700 tons of suspended solids and 10,610 of ammonia nitrate. Additional post MTR investments have contributed further reductions . With the lack of baseline data or dates in the ICR it is not possible to know what the percentage reduction was, or over what period. (b) Adequate supply of safe water modestly achieved : Although the ICR states that the project's water supply components are operational and supplying safe water, it does not provide any supporting data or evidence for this . (c) Institutional and financial strengthening of utilities modestly achieved : While most of the implementing utilities met or exceeded their financial covenants on cost recovery and debt service coverage, actual financial data is thin and the collection of data on key performance indicators only begun in 2007. Being unable to compare the known recent performance with that prior to the project means that evidence for the institutional and financial strengthening of the utilities is weak. Companies targeted for raising tariffs had little autonomy to execute it, since most responsibilities for setting tariffs remain with the municipalities .. 5. Efficiency (not applicable to DPLs): ERRs point to substantial efficiency, at least for the few project investments for which they were estimated . The economic rates of return at completion were : 18.6% at Tangshan and 12.3% at Handan. These compare with estimates of 14.4% and 12.9% at appraisal. These results demonstrate that these particular investments were worthwhile. Financial rates of return were estimated at completion to be in the 7.2%-12.6% range. While not a bad result for utilities that previously had not sought financial returns, there is clearly some way to go for full cost recovery. (The ERRs noted below are the simple averages of the two rates noted above ). ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the  re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 13.7% 23.2% ICR estimate Yes 15.5% 29.7% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project appeared to achieve its relevant objectives only modestly, and there are moderate shortcomings deriving from the little evidence available regarding achievements of water supply quantity and quality and the financial performance of utilities. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The outlook is positive for government commitment to the project and the utilities, not least of all in the willingness and ability to subsidize those utilities that have still be unable to fully recover costs . In the meantime, municipalities remain committed to the project's financial targets . With the continued operation of the wastewater treatment plants built under the project, the environmental benefits of reduced pollution can be expected to continue . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: The Bank brought considerable international experience to help the provincial government with sector policy formulation and investment program design . The Bank also provided inputs to improving engineering designs . A weakness of preparation was the inadequate M&E and little attention to documenting the start up conditions of the project through simple baseline data . Supervision was undertaken dilligently by individual team members, but it was interrupted and continuity lost by the frequent turnover of Bank staff on the project team . One of the consequences of this was that the final revision of the project to incorporate additional investments was processed too close to the original completion date, giving insufficient time for the Bank and the Borrower to agree on the targets of the new operations . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The (Hebei Provincial) Government demonstrated adequate commitment to the project, especially through tariff adjustments and the involvement of its high level staff . But the project management office was very active in disseminating the project experience across the province, as well as presenting a paper at the 2006 World Water Association conference in Beijing . But municipal authorities did not always separate utility regulation from utility operation. But most utilities, as executing agencies, were compliant with the legal covenants requiring cost recovery and debt service coverage . The only exception was the Shijiazhuang Drainage Company that continued to be loss-making despite the tariff increase authorized by the Government . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: There were several shortcomings . The design did not include indicators that could be monitored to verify progress toward meeting the objectives. M&E design was further weakened by the appraisal's lack of assembly of baseline data or of setting targets to be achieved . After mid-term, a new set of performance indicators was proposed by the Bank, allowing some M&E implementation for the first time, but the continuing lack of explicit targets made effective utilization of M&E impossible. a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The ICR reports satisfactory compliance with safeguards for this environmental category B project . Environmental Management Plans and Resettlement Action Plans were prepared and implemented satisfactorily according to the ICR, although the report provided not detail on these . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Evidence of achievements, especially Satisfactory in water supply operations and financial performance of operators is thin . Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Moderately Insufficient attention by the Bank to Satisfactory M&E and disruptions to supervision continuity by Bank staff turnover downgrade an otherwise satisfactory performance. Borrower Performance : Highly Satisfactory Satisfactory Both government and implementing agencies performed well, but the continued loss-making of a key utility precludes the top rating overall for the implementing agencies. Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: High caliber management capacity in central project management offices can help ensure that policy changes are executed. Knowledge sharing can be successful and an important product of the project when it has strong central government backing. Solid technical inputs, particularly in engineering, are essential for a project like this to succeed . Analysis of water pollutants should be based upon the sampled collection of empirical data . Theoretical parameters, as used commonly by design institutes in China, do not provide the correct inputs for the analysis that is needed. Financial covenants regarding the performance of project implementation agencies should be realistic and only cover those aspects of financial performance that are relevant to the project outcomes . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: This ICR is rated unsatisfactory because it continues to draw conclusions about performance even when it recognizes that there is very little evidence . Examples include: (i) the ICR provides no actual values achieved of the three PDO indicators or the Intermediate Outcome Indicator (pp. iii-iv); (ii) the ICR claims that the project achieved greater pollution control than planned thanks to the Tangshan plant, but provides no data or evidence of pollution control originally intended or that actually achieved; (iii) the ICR claims that through its implementation, the project "contributes substantially to the promote economic growth in the three cities, increase development opportunities and decrease poverty" without producing evidence either of these outcomes themselves or how they could have been the result of the project (p. 17); (iv) it concludes that the project caused significant beneficial institutional changes, referring generically to innovations promoted and introduced in "specific" functional areas like institutional development, business planning, financial management etc . without being truly specific about what the innovations were and how they could have been achieved through the project (p. 17); (v) the ICR states that the project PDOs remained relevant "to current China objectives" without explaining what these objectives are or what they refer to (p.17); (vi) the ICR states that construction and support services during implementation "contributed to create job opportunities" without citing any evidence or data (p. 17); (vii) referring to "several examples of notable achievements" in institutional strengthening, the ICR provides only generic list of types of improvements that could be made without specifying precisely what actions were engendered by the project, with no data on their frequency or location (p. 18); (viii) the ICR refers to the "remarkable" and "excellent management and dissemination of project results" without explaining exactly what was disseminated on what scale and where, who did the dissemination and who benefited from it (p. 19). The candid recognition at the beginning of the ICR of the data limitations this project faced is commendable, but it should require that ICR conclusions be drawn cautiously and with significant caveats. Thus, while the report acknowledges the lack of baseline and target data relating to project performance, it does not recognize how the weak evidence base precludes a satisfactory assessment of project performance . Thus, when an ICR presents a table ostensibly showing "compliance status of financial covenants " containing mostly 2008-2012 projections, and missing actual 2001-2006 performance data, the credibility of its evaluation is undermined . Instead, a self-evaluation exercise such as an ICR could be expected to provide a critique of such material, something that is absent from the present report . In general, where evaluation evidence is weak or nonexistent, the ICR could be a tool to explain to the Bank and Borrower what is needed for the performance of a project to be adequately monitored and evaluated. The thinness of evidence on water supply and the financial performance of the operators is particularly striking. There are important inconsistencies of some assessments in the ICR . For instance, while section 3.2 reports the weak financial performance of the companies, their performance in section 5.2 is rated highly satisfactory. The report would have benefited from editing to improve consistency and bring out the main points more clearly. The Borrower ICR describes well how the project was prepared and implemented, but gives little attention to project performance from the perspective of the outcomes of the project a.Quality of ICR Rating : Unsatisfactory