Mexico - High-efficiency Lighting Pilot Trust Fund Project

Document of The World Bank FOR OFFICIAL USE ONLY Report No. 20425 PERFORMANCE AUDIT REPORT MEXICO HIGH-EFFICIENCY LIGHTING PILOT TRUST FUND (GEF GRANT NO. GE-7492) April 28, 2000 Operations Evaluation Department Sector and Thematic Evaluations Group This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents Currency Unit= New Mexican Peso (N$) US$1.00 = N$ 6.42 (1995 average) US$1.00 = N$ 7.60 (1996 average) US$1.00 = N$ 7.92 (1997 average) US$1.00 = N$ 9.07 (1998 average) Special Drawing Rights: US$1.00 = 0.73 Norwegian Krone: US$1.00 = 7.55 Weights and Measures 1 Metric ton (mt or ton) = 1,000 kg. 1 MW (Megawatt) = 1,000 kW 1 GW (Gigawatt) = 1,000 MW I TW (Terawatt) = 1,000 GW 1 Tt (Teraton) 1,000,000,000 tons Abbreviations and Acronyms BANOBRAS Banco Nacional de Obras y Servicios Pblicos (National Bank of Public Works and Services) CFE Comisi6n Federal de Electricidad (Federal Electricity Commission) CETES Certificados de Tesoreria (Government of Mexico Treasury Bills) C02 Carbon dioxide CONAE Consejo Nacional de Energia (National Energy Council) DSM Demand-side management UNFCCC United Nations Framework on Climate Change FIDE Fideicomiso de Apoyo al Programa de Ahorro de Energia del Sector E16ctrico (Trust Fund for Energy Conservation) FL Fluorescent light bulb GEF Global Environmental Facility GET Global Environmental Trust GOM Government of Mexico ILUMEX Iluminaci6n de Mexico (Mexico High-Efficiency Lighting Project) IU Implementation unit IRR Internal rate of return LRMC Long-run marginal cost NOx Nitrogen oxides PIU Project implementation unit STAP Scientific and Technical Advisory Panel (Panel de Asesores de Ciencia y Tecnologfa) S02 Sulfur dioxide Fiscal Year Government: January I - December 31 Director-General, Operations Evaluation : Mr. Robert Picciotto Director, Operations Evaluation Department : Ms. Elizabeth McAllister Manager, Sector and Thematic Evaluations Group : Mr. Gregory Ingram Task Manager : Mr. Andres Liebenthal FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation April 28, 2000 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Performance Audit Report on Mexico High-Efficiency Lighting Pilot Trust Fund (GEF Grant GE-7492) Attached is the Performance Audit Report prepared by the Operations Evaluation Department on the Mexico High-Efficiency Lighting Pilot Trust Fund for which a GEF Grant (GE-7492) in the amount of SDR 7.3 (approximately US$10.00 million) was approved on March 14, 1994. The project was cofinanced by a grant from the Kingdom of Norway for NKr. 20.25 million (approximately US$3 million). The project was made effective on February 10, 1995, and the final disbursement took place in March 1997. The Grant closed December 31, 1997, and the project was completed by December 31, 1998. Both dates were about one year behind schedule. The project objectives were: (a) to demonstrate the technical and financial feasibility of reducing emissions of greenhouse gasses and simultaneously reduce local environmental contamination through the widespread installation of high efficiency lighting; (b) to build the institutional capacity for technological change and energy conservation; (c) to provide a replicable model for energy demand side management; and (d) to strengthen CFE's (Mexico's utility) capacity to practice demand side management on a sustainable basis. The main project component was the promotional sale of approximately 1.7 million fluorescent light bulbs (Fls) among residential users in the cities of Guadalajara (Jalisco State) and Monterrey (Nueva Leon State), at prices 60% below cost. Since Fls consume 75% less energy than equivalent standard incandescent bulbs and last 15 times longer, significant energy savings and environmental benefits were envisaged. Regarding the project outcomes, by the Grant closing date the project slightly surpassed the original target of 1.7 million Fls sold. Furthermore, CFE continued selling Fls through January 31, 1999, to attain a total of 2.6 million. Project costs remained remarkably close to original estimates, and Fls sale prices went up, so that the average subsidy was actually 49% of the bulb's cost. Also, the energy savings and reduced CO2 emissions were in line with initial estimations. On the other hand, power capacity savings (reduced power consumption delays the need to invest in new capacity) were only one-third the initial estimates. The project lost its original focus on poor consumers due to slow sales among low- income/low electricity consumers, and the program did not fulfill its free driver expectations of promoting Fls among the public at large. Based solely on energy savings the project had a positive and acceptable economic internal rate of return, albeit significantly lower than the one put forward at appraisal, due to the initial overestimation of capacity savings. On the other hand, if the local and global environmental benefits were factored into This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. 2 the economic analysis, the rate of return could increase substantially. The distribution analysis (without environmental benefits) shows that: (a) all gains went to participant households and they exceeded the appraisal estimate because marginal electricity tariffs were two to three times higher than the utility's marginal costs; (b) CFE lost profits, mostly in the form of forgone electricity sales; but (c) CFE financial losses were lower than the ICR estimates since part of the project costs was covered by the grants from GEF and the Kingdom of Norway. Internationally, the project compares well with similar initiatives around the world, in terms of number of Fls distributed, unit cost, and level of subsidy. However the project design and implementation never addressed the issue of sustainability. What will happen with the use of Fls in the participating cities after the project wanes, remains unknown. Comparing goals and outcomes the audit concludes that the project succeeded in demonstrating the technical feasibility of reducing emissions of greenhouse gases and simultaneously reducing local contamination through the widespread installation of high efficiency lighting, but it failed to demonstrate the financial feasibility of such endeavor and it contributed modestly towards building the institutional capacity for technological change, energy conservation and demand-side management at CFE, the implementing utility. The audit ratings of the project are as follows: Outcome is rated marginally satisfactory (the audit downgrades the ICR rating of satisfactory). Institutional development impact is rated modest. (the audit downgrades the ICR rating of substantial. Sustainability is rated uncertain (the audit downgrades the ICR rating of likely). Bank performance is rated unsatisfactory (the audit agrees with the ICR rating) The project offers several lessons: * Replicability and sustainability are essential elements in demonstration projects and deserve close consideration. They need to be explicitly incorporated in the project design, which should spell out the strategy, the activities, the resources and the monitoring that will be committed to these purposes. * Well-planned implementation schemes are central to the project performance. In this case the implementation scheme, based in decentralized units and the establishment of independent trust funds with BANOBRAS was very successful. It gave CFE (Mexico's utility) overall control of the project, while assuring a smooth flow of funds to the local implementation units. * Bulk procurement can be an important leverage. The project proved that bulk fluorescent lamps (Fis) procurement was an effective way to obtain technical improvements and significant price rebates from manufacturers. Attachment 1 Contents Preface ............................................................................................................................................. v 1. Project Background ......................................................................................................... 1 2. Objectives, Components, and Design ................................................................................. 2 Objectives...................................................... 2 Components and Design ....................2....... ................2 3. Im plem entation ........................................................................................................................ 3 4. Outcomes and Follow-up Activities .................................................................................. 4 Outcomes...................................................... 4 Follow-up Activities .................................................... 7 5. Conclusions............................................................................................................................... 8 Findings ....................................................... 8 Ratings ........................................................ 10 6. Lessons .................................................................................................................................... 10 Annex A: Basic Data Sheet ...................................................................................................... 11 Annex B: Distribution Analysis . ............................................................................................ 10 This report was prepared by Mr. Pablo Gutman (Consultant) and Mr. Andres Liebenthal (Task Manager), who audited the project in June-July 1999. Mr. William Hurlbut edited the report. Ms. Soon-Won Pak provided administrative support. Ill Principal Ratings ICR Audit Outcome Satisfactory Marginally Satisfactory Sustainability Likely Uncertain Institutional Development Substantial Modest Borrower Performance Satisfactory Satisfactory Bank Performance Deficient Unsatisfactory Key Staff Responsible Task Manager Division Chief Country Director Appraisal Luis Luzuriaga Martin Staab Edilberto Segura Mid-Term Review Luis Cosenza Richard Clifford Olivier Lafourcade Completion Enrique Vanegas Daniel Leipzgeir Olivier Lafourcade V Preface This Operations Evaluation Department (OED) Performance Audit Report (PAR) is on the High-Efficiency Lighting Pilot Trust Fund (known in Mexico as ILUMEX), for which a GEF Grant (No. GE-7492) of SDR 7.3 (approximately US$10.00 million) was approved on March 14, 1994. The project was cofinanced by a Nkr. 20.25 million (approximately US$3 million) grant from the Kingdom of Norway. The project was made effective on February 10, 1995, and the final disbursement took place in March 1997. The grants were closed December 31, 1997, and the project was completed by January 31, 1998, both dates are about one year behind schedule. The PAR is based on the Implementation Completion Report (ICR), the Staff Appraisal Report (SAR), loan documents, project files, and discussions with Bank staff. An OED mission visited Mexico in June-July 1999 for discussions with the government, the project implementing agencies, the Bank's Mexico office, and other project stakeholders. Their cooperation and assistance are gratefully acknowledged. The High-Efficiency Lighting Pilot Trust Fund was the first project of its kind in the GEF portfolio, and assessing its outcome in the ICR resulted in a lively discussion among the Bank's Country Unit, Sector Unit, Environmental Department, OED, and GEF, regarding how to evaluate its outcome and sustainability. While this audit follows the standard PAR procedures, it also draws upon experiences from similar projects elsewhere to help evaluate the project. The audit substantiates findings of the OED review of the ICR. It presents a more thorough consideration of the lessons to be learned from a project that, despite its shortcomings, pioneered a new and increasingly important field for the borrower, the Bank, the GEF, and the development community at large. Following standard OED procedures, this draft PAR was sent to the borrower for comments, but none were received.  1 1. Project Background 1.1 Mexico's energy consumption ratios compare favorably with OECD averages. Nevertheless, the country's size, rapid population and economic growth, and the fact that energy consumption has been growing faster than the GDP made it the 13 largest energy consumer and the 12t largest energy-related C02 emitter in the world by the mid-1990s.' 1.2 The 1998 electricity generation of 137 Terawatt hours (TWh) amounted to approximately 10 percent of Mexico's total final energy consumption. Electricity is the fastest growing energy component, and it is estimated that for 1996-04 an additional I lGigawatts (GW) of generation capacity will be required, with investment costs of US$3 billion per year. High investment needs are not the only sectoral concern. Since some 73 percent of power generation uses fossil fuels, utilities are a major source of local pollutants (SOx and NOx) and a significant source of C02, the main global greenhouse gas. Therefore, since the late 1980s Mexico's energy authorities have been promoting energy efficiency and pollution reduction alternatives among producers and consumers alike. For instance: * In 1989, with the creation of the Trust Fund for Energy Conservation (FIDE), Mexico's utility company, started its Energy Conservation in the Electricity Sector program. During 1990-94 FIDE approved more than 500 loans at zero interest for a total of some US$17 million to finance energy efficient projects that by 1994 have saved an estimated ITWh of energy. FIDE also undertakes training, energy efficiency certification, and demonstration projects. * Eleven national energy efficiency standards were approved by 1996 (e.g., for water pumps, water heaters, etc.) as part of a program that aims to achieve, by the year 2005, a 10 percent energy saving (based on 1995 power consumption figures). * The national energy council, CONAE, conducts a program of sector and industry energy audits, and promotes training, demonstration projects, and education campaigns on energy efficiency. * CFE has a program for replacing natural gas for oil in power generation plants, particularly in and around Mexico City. 1.3 As part of these initiatives CFE explored the potential of demand-side management (DSM). Broadly speaking DSM, is any initiative that aims to increase the use of energy saving devices. The rationale for DSM rests in what has been labeled the "energy conservation paradox." That is, households and other small energy consumers fail to take advantage of available energy- efficient technologies, even when they would clearly reap monetary benefits. This market (or behavioral) failure has been linked, among other things, to lack of information, high up-front costs, and high household rates of discount. Many studies have found it to be pervasive among households and small energy consumers, in both high- and low-income countries.' 1.4 Mexico has high expectations for DSM. So high, in fact, that the current national electric plan put forth a target of saving 50 TWh by the year 2004 through DSM, with related reductions in the need for additional capacity of more than 5 GW. When the Bank approached CFE in the identification and preparation phase of this project, CFE was already surveying the potential market for a high-efficiency fluorescent lights (FLs) project in Guadalajara and Monterrey (the 1. OECD (1998) " Mexico Environmental Performance Review" OECD, Paris; and Telnes, E. and Uzzell,G.(1999), Illumex Lessons Learned" Technical Report, DNV Development, Oslo 2. See Shipper, L. (1993) "Energy Efficiency and Human activities" World Bank 1993 Development Conference 2 second and third largest cities in the country), based on its experiences with several small DSM activities, including: * FIDE's financing of improvements in home insulation, * A program to equip public lighting with sodium vapor lights, * The replacement of some 90,400 household incandescent bulbs with high-efficiency fluorescent light bulbs, mostly through sales at discounted prices. 1.5 Mexico's interest in a sizable FLs project was shared by the Global Environment Fund (GEF), the Government of Norway, and the World Bank. GEF was interested in exploring the potential of high-efficiency lighting projects to reduce greenhouse gases. An interesting point is that these types of projects are economically viable on the basis of domestic benefits (energy cost savings) and therefore would not usually merit GEF support. Yet experience worldwide shows that commercial diffusion of FLs among households is low without a major promotional boost. With that understanding, GEF participated in the financing of the pioneering ELUMEX project, which has since been followed by similar projects in Poland, Jamaica, and Thailand. 1.6 Further to its contribution to Mexico's sustainable development, the Government of Norway looked upon the project as an opportunity to learn about activities implemented jointly as envisaged in the UN Framework Convention on Climate Change (UNFCCC). This interest was shared by the Bank's Environmental Department, which has devoted significant efforts to develop the Bank's know-how to act as a clearinghouse in a potential worldwide carbon offsets market. To this end, the Bank's Climate Change Unit gave technical support to the project, commissioned a carbon abatement audit of it, and used it as a case study to develop carbon abatement monitoring and verification methodologies.' 2. Objectives, Components, and Design Objectives 2.1 As stated both in the Project Document, and the Grant Agreement, the project objectives were: (a) to demonstrate the techmical and financial feasibility of reducing emissions of greenhouse gasses and simultaneously reduce local environmental contamination through the widespread installation of high-efficiency lighting; (b) to build the institutional capacity for technological change and energy conservation; (c) to provide a replicable model for energy demand-side management; and (d) to strengthen CFE's capacity to practice demand-side management on a sustainable basis. Components and Design 2.2 The main project component was the replacement of approximately 1.7 million incandescent bulbs with fluorescent light bulbs (FLs) among residential users in the cities of Guadalajara (Jalisco State) and Monterrey (Nueva Leon State), thus reaching approximately 10 to 15 percent of these cities' households. 2.3 Since FLs consume 75 percent less energy than equivalent standard lamps and last 15 times longer, significant energy savings and environmental benefits were expected from this 3.Telnes, E. and Uzzell,G.(1999), Illumex Lessons Learned Technical Report, DNV Development, Oslo 3 project. High rates of return were also estimated, but varied widely, depending on whose costs and benefits were measured, whether it was all the stakeholders (true economic costs and benefits), participant households, the utility company, or the global commons. 2.4 Seventy seven percent of the project's total costs went to the acquisition of FLs, with the rest paying for project offices, marketing activities, and consultants for testing, surveying, and monitoring. Some important aspects of the project design were: * The project organization was similar to the one CFE already had in place for its home thermal insulation improvement project: two project implementing units (PIU), one in each city, financed through local trust accounts. FLs would be sold directly by the PIUs, either in existing CFE locals, stands in shopping malls, or through door-to-door campaigns. * Initial preference would be given to low-income, low-electric-consumer households, with the purpose, in the short run, of reducing the below-cost sales to this market segment, and in the medium run to help offset the impact of future tariff increases among low-income dwellers. * Targeting low electricity consumers and distributing up to six lamps per households was expected to increase the use of the FLs during peak electricity demand time, therefore increasing capacity savings. * The project's promotional character: FLs would be sold at approximately one third of retail prices for equal quality FLs. Furthermore, buyers had the option of paying either by cash or with a two-year financing plan at an interest rate equivalent to the government treasury bills, to be charged to their electricity bills. Also the project gave a two-year replacement guarantee for defective bulbs. * The project design incorporated surveys and monitoring activities, directed to: (a) check the pace of sales, (b) gather information regarding the use of FLs and their impact on savings in energy and capacity, and in emission reductions, and (c) to estimate the "free-driver effect" (the induced increase in non-subsidized retail sales of FLs that could be attributed to the project's demonstration effect). * The Climate Change Unit of the Bank's Environmental Department selected the project as a pilot for testing verification of carbon savings and to this end gave support and follow-up to the project design and implementation. * The project document stated that no policy or institutional reforms were needed to ensure effective project implementation. * The only risks considered were low sales and technical problems with the design or operation of the FLs. * Remarkably, the sustainability of this project does not seem to have been considered at the design stage. The GEF Project Document states that the project would last until the original funds were exhausted without further discussion of sustainability beyond that. 3. Implementation 3.1 Effectiveness, planned for September 1994, suffered almost a year's delay due to complications in the establishment of the trust funds for the two PlUs. Although the project developed thereafter without major complications, it was not able to catch up from the delay. No disbursements were made in fiscal 1994, and it proceeded slowly during fiscal 1995. It gained momentum in fiscal 1996, so that 84 percent of the grant was actually disbursed by the end of that fiscal year. When the project officially closed, in December 1997, it had used all of the original grant and local counterpart moneys (although money was still coming into the project as households repaid financed FL purchases). Some implementation issues worth noting are: 4 * The final project budget was US$25.95 million-some 13 percent larger than originally estimated-due to increased government contributions to the project; reinvestment of reflows from FLs sales, and exchange rate appreciation of the external funding. * According to appraisal, all grant funds (US$13.85 million) were to be transferred to participant households through rebates in the price of FLs sold. By the grant's closing date (December 31, 1997), US$11.7 million had been distributed to FL buyers. By the sales closing date (December 1, 1999) the project distributed 2.6 million of FLs and buyers have benefited from US$17.7 million in rebates. * CFE had originally targeted low-income/low power consumers, yet sales to this sector were slow, and sales were redirected to medium and high-income consumers. Also sale concerns justified increasing to 10 the original 6 FLs per household limit, and expanding the marketing efforts to the cities' entire metropolitan areas. * Through bulk procurement, the project received significant price discounts and, more important, was able to demand changes in the specifications of the FLs to overcome quality problems. * While keeping sales up was an overarching concern, other activities were given much less attention. The surveys scheduled to inform the midterm reviewers were late and their results were made available only after the project had closed. Surveys to estimate free-driver effects were not commissioned. Also CFE and the PIUs failed to comply with the Bank's auditing and accounting requirements, a situation that persisted through project completion. 4. Outcomes and Follow-up Activities Outcomes 4.1 FL distributed: By the grant's closing date the project had slightly surpassed the original target of 1.7 million FLs. Furthermore, CFE continued selling FLs through January 31, 1999, to attain a total of 2.6 million FLs sold, 53 percent above the original project's target. 4.2 FL costs and subsidies: Average FLs dollar costs remained remarkably close to original estimations (13.93 dollars against 13.53 dollars) Yet CFE operational practices regarding adjustments on its sale prices, to account for exchange devaluations, resulted in significant increases of FL prices in Mexican pesos, and varying percentage of the subsidy per FL through the project life. At closing, the actual average price subsidy per FL was 49 percent of CFE's costs, down from the 63 percent expected in the GEF Project Document. 4.3 Poverty targeting: The original focus on low-income/low electricity consumers was set aside due to slow sales in that market segment. Overall, 9.6 percent of FL sales went to low electricity consumers, 31.3 percent to intermediate, and 59.2 percent to high electricity consumer households. 4.4 Energy and environmental savings: These were in line with initial estimates of energy savings (approximately 1,000 GWh) and C02 emissions avoided (approximately 764 thousand tons). Reductions in local pollutants were smaller than envisaged because the utilities moved to gas and lighter fuels during the project life. Savings gained from postponing the need to increase generation capacity were only one-third of the initial estimates (see Table 1). 5 Table 1. Savin s Achieved Envisaged at appraisal At closing (12131/97) Comments or/and follow-up activities rgysavings (totals through the project Ie) In capacity needs 100 MW In capacity needs 33.7 MW The change in capacity reflects an In energy produced 914 GWh In energy produced 1,014 GWh overestimation at appraisal. Furthermore, the savings are temporary, lasting only as long as the FLs are in use. fnnmntal savig (al thogroetIo Of global concern: Of global concern: Differences from appraisal stem from Avoided C02 709,800 ton Avoided C02 763,692 ton CFE increasing use of gas and lighter fuels, that resulted in a reduction in Of local concern: Of local concern: emissions per unit of energy produced. Avoided SOx 25,675 ton Avoided SOx 10,920 ton Avoided NOx 1,754 ton Avoided NOx 2,098 ton Avoided Particulates 5,345 ton Water savings 3.5 m3 millions 4.5 Economic analysis: The ICR economic analysis, based solely on energy savings shows that the project had a positive and acceptable rate of return, albeit significantly lower than the one put forward at appraisal, owing to the initial over-estimation of capacity savings. On the other hand, if the environmental benefits, both local and global, were factored into the economic analysis the rate of return could increase substantially. 4.6 The audit economic analysis explored several other alternatives. If the country or the utility face severe short-term limitations to capacity expansion, this can add a premium to the benefit of postponed capacity expansion, increasing the project and CFE benefits. Although this does not seem to be the case for Mexico, it has been the main rationale for some FL distribution programs elsewhere. Also, since all parties considered this project a greenhouse gas reduction initiative, C02 abatement benefits should have been factored into the economic or financial analysis. Doing so would bolster the project benefits. Suppose a third party is interested in buying a product with a C02 emission reduction label. If the project's subsidy component (49 percent of its costs) is considered to be the price of the 763,692 tons of C02 avoided, then the unit price would be approximately US$18 per ton of C02. Alternatively, if all the project costs were charged to the avoided emissions, the unit price would climb to US$34 per ton. Finally, if the hypothetical C02 buyers would have to compensate CFE for its financial losses (due to reduced electricity sales), it would increase unit price to approximately US$40 per ton. There is no international consensus on what would be a fair price for reducing a ton of C02 emissions, but these figures are in the high range. In Table 2, the audit uses US$10 per ton to estimate a project economic rate of return, including C02 abatement benefits. Table 2. Project Economic Analysis Net economic benefits to all parties, (including GEF and Kingdom of At appraisal At ICR This audit Norway) RR without considering pollution and C02 abatement benefits 56% 29% 29% Including C02 abatement benefits at US$10 per ton na na 34% 4.7 Distribution analysis: Further to the economic analysis, the appraisal document and the ICR presented estimates of returns to households and to CFE (the power company). These are not economic but distribution (or financial) figures and the project effort should be commended since few Bank projects attempt a distribution analysis. Then again, the appraisal and ICR exercise is 6 incomplete and flawed in several ways (see Annex B table for details). When these distortions are corrected the audits distribution analysis shows that: * As expected, most project payoffs go to participant households. They are higher than estimated at appraisal simply because marginal electricity tariffs were two to three times higher than the utilities' marginal costs. This was due both to power price increases and to the project reorientation of FL sales from low to high electricity consumer households. * CFE loses money mostly as forgone electricity sales. A rather obvious outcome since, barring sales at prices below cost, there are few situations where any business would make a profit by selling less.' CFE financial losses are lower than the ICR estimates because some of the project costs were paid with the grants from GEF and the Kingdom of Norway. 4.8 The project compared: Regarding energy saved and pollution avoided by the project, outcomes are modest but meaningful at participating cities level (around 5 percent of energy- related CO2 emissions for both in the project's best year) and irrelevant at the country scale (less than 1/20'h of one percent). The project compares well with similar initiatives around the world, in terms of number of FLs distributed and unit cost. Regarding the distribution of FL costs between consumers and the project, ILUMEX is in an intermediate position. Table 3. The Project in the Context of Mexico's Power Sector Power reduction In the best project year, 1998 135.1 GWh (100%) * Compared with residential power consumption in participating states: 4,193 GWh (3.2%) * Compared with total power consumption in participating states: 20,122 GWh (0.7%) * Compared with residential power consumption, country: 31,690 GWh (0.4%) * Compared with total power consumplion, country: 137,286 GWh (0.1%) Annual sales of all types of light bulbs in participating cities,1997: 15.8 million (100%) * FLs sold by Grant close are equivalent to: 20.4 million (129%) * FLs sold by program end are equivalent to: 31.2 million (197%) CO2 avoided (in thousand tons per year, Tty) in the project best year, 1998 94.75 Tty (100%) * Compared with energy related CO2 erissions, participating cities (1 995/OECD): 1,806 Tty (5.2%) * Compared with energy related CO emissions, country (1995/OECD): 327,600 Tty (0.03%) 4. This was part of the project's original design: that CFE would have financial gains by reducing below cost sales to low electricity consumers, and increasing sales at average rates. But as already mentioned, most sales went to high electricity consumers that pay high marginal rates. 7 Table 4. The Project Compared with Other Similar Initiatives CountrylProject Project Design Number of Cost per FLs (US$) origin vered For the buyer For the project Total Mexico (GEF) Subsidy to consumers 1,712,361 7.1 6.8 13.1 Poland (GEF) Subsidy to 1,600,000 9.0 3.2 12.2 manufacturers Jamaica (GEF) Subsidy to consumers 85,000 5.8 6.8 12.6 Thailand (GEF) Bulk purchases 1,500,000 9.0 2.9 11.9 Brazil Give away 89,000 - 8.3 8.3 Denmark Small subsidy plus quality 1,000,000 11.0 4.4 14.4 control programs UK 1 Subsidy to consumers 3,000,000 12.4 1.8 14.2 UK2 Give away 814,000 - 11.3 11.3 Peru Publicity campaign, no 400,000 20.0 16.8 36.8 subsidies USA (several) Subsidy to consumers Rebates Usually none. between 20% Paid out of a and 50% of surcharge on retail prices electricity bills. Source: Martinot E. and Borg. N. (1998) 'Energy-efficient lightning programs. Experience and lessons 1rorn eight countries" 4.9 The participants' reactions: The Mexican implementing agency considered this project as delivering valuable management experience (e.g., in the organization of the PIUs, the procurement of FLs, and their technical testing, marketing, etc.). CFE plans to use these experiences in other ongoing and future energy efficiency projects (see below). Surveys conducted during 1997 in the two participant cities showed high levels of satisfaction among participant homes; high visibility of the project city-wide; and mixed results regarding households disposition to buy FLs at full price. On one hand, 80 percent of the participant households said they would be willing to pay full price to replace their LUMEX FLs when they wear out. On the other hand, none of these households have yet attempted to increase their use of FLs with purchases at non-subsidized retail prices. 4.10 Free-driver effects: The project did not collect systematic information to estimate its free-driver impact (the induced increase in retail sales of FLs that could be attributed to the project demonstration effect). The PIU staff observations suggest that: (i) during the project life, retail sales to households actually went down due to the competition from the project; (ii) there is anecdotal evidence that some people transferred their home experience into their workplace, increasing demand of FLs among services, commerce, and industry; (iii) there were no retail price reductions; and (iv) by project end more retailers stocked and displayed FLs. Follow-up Activities 4.11 The project follow-up activities: CFE continued with the project's sale program beyond the grant closing date, and sold 2.6 million FLs by January 31, 1999. The PIUs have since stopped sales but will continue to maintain a presence in both cities through the year 2000 to collect payments and service clients, including replacement of defective FLs. ILUMEX findings were presented on March 24, 1999, at Vallarta, to an audience of 100 persons, including 22 participants from foreign countries. 4.12 Final destiny of FLs sales moneys: According to the project documents, if the project was successful, the PlUs would continue selling FLs until the balance of reflow funds (estimated to be approximately US$9 million) was completely exhausted. CFE has already re-invested part 8 of the reflows, but, with sales ending January 31, 1999, additional reflows to CFE through the year 2000 would amount to approximately US$11.6 million. CFE plans to invest these moneys in several energy conservation programs elsewhere in the country. 4.13 FLs follow-up activities: In 1998, FIDE (the National Trust Fund for Energy Conservation) began a country-wide FLs project to sell 6.1 million lamps. FLs would be sold at full direct costs, but price would not take into account the project operation and financial costs, (what amounts to a subsidy of approximately 24 percent of the FL total cost). During 1998 and 1999, FIDE sold FLs in the cities of Queretaro, Michoacan, Puebla, Veracruz, and Morelos. 5. Conclusions Findings 5.1 Assessing outcomes against the project objectives, this audit concludes that: * The project did "demonstrate the technical ... feasibility of reducing emissions of greenhouse gasses and simultaneously reduce local contamination through the widespread installation of high-efficiency lighting." Although the impact of the project was minimal at the country level, it was relevant at the local level, with reductions in greenhouse gas emissions in the 5 percent range. It also compares well internationally, both in size and cost per FLs lamp. This speaks well of the relevance, efficacy, and efficiency in attaining the project physical goals: installing 1.7 million FLs. * There were also improvements in "building the institutional capacity for technological change and energy conservation" and in "strengthening the CFE's capacity to practice demand-side management"-the institutional development impact. Although improvements look modest. They are the result of a scale effect, rather than of an innovative approach, since the project copied the institutional organization that CFE had already in place in other energy efficiency program and the technical change achieved were limited to increasing the quality of the lamps procured. * The project clearly failed to "demonstrate the ... financial feasibility of reducing emissions of greenhouse gasses and simultaneously reduce local contamination through the widespread installation of high-efficiency lighting." Although this goal appears prominently among the project objectives, nothing in the project design or implementation worked in that direction. * As for its replicability, the project did "provide a replicable model of energy demand-side management," in its technical and management design, but failed to test or even discuss a replicable financial design. * A project can be non-replicable, yet its outcomes could be sustainable through other means. Regarding the two cities involved, there is little evidence to believe that thanks to the project's impact, a similar or larger amount of fluorescent light bulbs will be bought at full market price when the ones distributed by the project are worn out. On a broader scope, Mexico has reaffirmed its interest in energy conservation, and CFE (Mexico's utility) plans to increase its demand side management programs. Then again, what this project has contributed to CFE commitment to demand side management is mostly on the technical side. 9 Box 1. Assessing sustainability and replicability in a GEF pilot demonstration project Since the ELUMEX project is the first of its kind completed in the Bank and GEF portfolio, assessing its outcomes has fostered a lively discussion among Bank's staff. While no one disputes the project achievement in terms of FLs distributed, there are contrasting views regarding the project sustainability and replicability and the lessons that can be learned from it: One position argues that any project with a large subsidy component would be unsustainable and difficult to replicate. Actually some activities-like elementary school and primary health care-are regularly delivered with a large subsidy component, that few will contest. Similarly, in the presence of positive environmental externalities, subsidies could be a way to pay for them (e.g., payments for C02 abatement). The problem regarding ILUMEX's sustainability and replicability is not the presence of a subsidy, but the fact that, aside from some comments on the desirability of free-driver effects, neither the project design, its implementation, nor the ICR addressed issues like: Why is this level of subsidy necessary? or, How will FLs be distributed in the future? For example, when the actual subsidy went from 60% to 49%, thee project did not open a discussion of what was then the appropriate level of subsidy: 60%, 49%, or 24% as in the current follow-up FIDE program? Similarly, among the many studies required by the Bank, not a single one asked for the analysis of alternative sources of financing FL distribution in the future (a surcharge in electricity price, selling CO2 abatement, an environmental investment fund, term sales at full price, etc.). Another position put forward is that sustainability was outside the reach of the project because it depended on Mexico's willingness to increase electricity rates. There is no basis to support this argument. Between 1995 and 1997 the relevant marginal electricity price per kilowatt hour, the one paid by participants households, was two to three times the utility's marginal cost, and growing. Actually others in the Bank argue the project sustainability exactly the opposite way. Since households will save kilowatt hours priced at such high rates, they will make handsome gains from their energy savings and therefore-the argument goes-the project is highly beneficial and it will be easy in the future to make households pay the full price for their FLs. It should be noted that high financial gains to households should not be constructed as project economic gains, since most of them are mere transfer payments from CFE and the donors. As for the argument that households will pay full price in the future, this might be so, but the project design is rooted in an assumption that households are reluctant to pay full price for their FLs. The results of the project do not provide any clues to challenge that assumption. Still others would defend the project with the pilot project justification-this being the first project in a new area where there is so much to learn-yet this is precisely where the shortcomings of the ILUMEX project are evident: It had a good learning payback in terms of technical designs (procurement, lamp specifications, opportunities for greenhouse gas verification, and AI design, etc.) but by and large, it failed to deliver lessons for the economic and financial design of future DSM projects. * This brings into focus the efficiency question. If the project's main objective was to show how a large amount of FLs could be successfully distributed, it did it efficiently and at reasonable per unit prices compared with similar projects (see Table 4). If the broader objectives were to develop new approaches, learn, and provide replicable and financially sustainable models for demand side management, the project efficiency was low. 5.2 In sum, the project was well crafted and managed to deliver over 2.6 million FLs (by January 1999), not a minor achievement for one of the first projects of this size in Mexico and in the GEF portfolio. But it was poorly designed as a pilot demonstration project in that it did not address the sustainability and replicability questions that should be at the core of any project in general but especially of a demonstrative one. 10 Ratings 5.3 Based in all the above, this audit rates the project as follows: * Outcome: Marginally Satisfactory. The audit downgrades the ICR rating (Satisfactory) to reflect the fact that while ILUMEX had surpassed its physical targets, it failed to fulfill expectations as a pilot demonstration project. * Institutional Development: Modest. The audit downgrades the ICR rating (Substantial) based in the fact that the project added little to the existing CFE institutional arrangements. * Sustainability: Uncertain. The audit downgrades the ICR rating (Likely), because the project design and implementation never addressed the issue of sustainability, and what will happen with the use of FLs in the participating cities. * Bank Performance: Unsatisfactory. The audit agrees with the ICR rating (Deficient). As pointed out in the project's ICR, the project had three task managers in three years, and this high turnover was partially responsible for the Bank failure to demand and achieve a timely performance of the monitoring activities and compliance with audit and accounting requirements. While the ICR. noted the poor supervision, this audit is more concerned with the project design's lack of attention to sustainability and replicability concerns, an omission that extends to the ICR. * Borrower Performance: Satisfactory. The audit agrees with ICR rating, in that CFE staff, being an utility entity, was focused on what they knew best: the technical aspects of delivering the FLs, which they did very efficiently. 6. Lessons * Replicability and sustainability are essential design elements in demonstration projects and deserve close consideration. For any pilot demonstration project, and more so for a GEF grant, addressing project replicability, the sustainability of outcomes, and the learning from the project are central issues. They need to be explicitly incorporated in project design, which should spell out the strategy, the activities, the resources, and the monitoring that will be committed to these purposes. Sticking to a physical target and delivering it is important but is not enough since, by definition, a pilot project is a small-scale endeavor that is not expected to make a large difference on the ground, except through its replication and the application of lessons learned. * Continuity of Bank staff might help mitigate the effects of design shortcomings. While it is difficult to assert the chances that project's design shortcomings will be detected and corrected during implementation, the high turnover of the Bank staff assigned to the project surely did not help. * Well-planned implementation schemes are central to project performance. The project implementation scheme, based on decentralized implementation units (Plus) and establishment of independent trust funds with BANOBRAS, was very successful: It gave CFE oversight in the project while ensuring a smooth flow of moneys to the PIUs. * Bulk procurement can be an important leverage. As already has been proven in similar projects, bulk FL procurement was an effective way to obtain technical improvements and significant price rebates from FL manufacturers. 11 Annex A. Basic Data Sheet Key Project Data (in US$ million) Appraisal estimate Actual or current Actual as % of estimate appraisal estimate Total project costs 23 25.95 113 GEF grant amount 10 10.7* 107 Kingdom of Norway grant amount 3 3.15* 105 OFE (Mexico) 10 12.1* 121 Date physical components completed December 31, 1997 December 31, 1998 Economic rate of return *.* Source: ICR * The actual amount disbursed exceeded the appraisal estimate due to the variation of exchange rates (SDR and NKr) vis-a-vis the US$. This figure includes US$973,000 of reinvestment of CFLs sale proceeds. ** Different rates of return -for the country, the power utility and the households- have been estimated, see main text. Cumulative Estimated and Actual Disbursements, GEF Grant GE-7492 (in US$ million) FY94 FY95 FY96 FY97 Appraisal estimate .65 6.5 5.85 Actual 0 2.2 9.4 2.2 Actual as % of appraisal estimate 0 34 161 Source: ICR Project Dates Original Actual Identification October 1991 Preparation June 1992 Appraisal August 1993 October 1993 Negotiations December 1993 February 1994 Board presentation March 1994 March 14, 1994 Signing June 1994 June 1994 Effectiveness September 1994 February 10, 1995 Midterm review September 1995 June/July 1996 Project completion December 31, 1997 December 31, 1998 Loan/Grant closing December 31, 1996 December 31, 1997 Final disbursement March 1996 March 1997 Source: ICR 12 Staff Inputs, GEF Grant GE-7492 Stage of project cycle Time-frame Staff weeks identification-appraisal Oct. 91 - Oct. 93 59 Negotiations -effectiveness Feb. 94 - Feb. 95 10.4 Supervision Jun. 95 - Dec. 97 24.9 Completion Jan. 93 - Dec. 98 8 Total 102.3 Source: ICR Mission Data Date Staff Staff days Specializations Perform Weeks in the field represented ance rating 1992 7..3 Through Appraisal August 1993 39 5..3 Operations officer, Environmental spec., Power engineer Appraisal -Board July 1994 20 3.3 Operations officer Approval 1995 5 Supervision April 1996 24.9 3.1 Power engineer, Economist 1997 2.4 Completion 1998 8 2 Power engineer Source: ICR (The mission data sometimes provide rating trends in the ICR project tables also, but were not available for this project.) Related Projects Borrower /Executing Agency: Loan/Credit title Credit/Grant Amount ($US million) Board date Status (as of June 99) Promotion of electric efficiency (GEF- n/a 14.0 1994 Ongoing Thailand) Efficiency lighting (GEF - Poland) n/a 5.0 1996 Completed Demand side management (GEF - Jamaica) n/a n/a 1995 Ongoing Source: ESSD core database/ ICR 13 Annex B. Distribution Analysis (without considering pollution and CO abatement benefits) * At appraisal At ICR This audit Distribution between participating households and all others a) Net financial gains (+) to participating households, 151% IRR 277% IRR 277% IRR including saved taxes NPV: na NPV: na NPV: + $ 30.0 M b) Net financial gains (or losses -) to all other parties (CFE 32% IRR negative IRR negative IRR + Mexico's treasury + GEF + Kingdom of Norway) NPV: na NPV: na NPV: - $18.9 M NPV (a+b)= Net economic benefits 56% IRR 29% IRR 29% IRR NPV:na NPV:na NPV: +$11.1M Distribution between Mexico and Donors a) Net financial gains to Mexico na na 96% IRR (participating households + CFE + Mexico's treasury) NPV: + $ 22,6 M b) Net financial gains (or losses -) to GEF + Kingdom of na na Negative IRR Norway NPV: -11.5 M$ NPV (a+b)= Net economic benefits 56% IRR 29% IRR 29% IRR NPV: na NPV: na NPV: + $ 11.1 M NPV figures are estimated at a 12 percent discount rate, and August 1998 prices * The distribution analysis performed at appraisal and at the ICR are incomplete and flawed in several ways (a) they mistake worldwide gains for CFE gains. Yet CFE's (or Mexico's gains) are larger, since part of the costs were paid with grants from the GEF and the Kingdom of Norway; (b) For a distribution analysis to be correct, summing the gains and losses of all parties must render the project's net economic benefits, since the only costs and benefits that can be distributed are the project's economic costs and benefits, plus transfer payments, and transfer payments if fully reported will cancel each other. Yet the appraisal and the ICR account for some transfer payments only one way, that is, they include avoided taxes and avoided electricity payments as part of the financial gains of the participant households, but they fail to allocate symmetric financial loss to the treasury and to the electric company.; (c) Finally, a distribution analysis should use net present value figures, as it is almost meaningless in rate of return. Consider that 500 percent of financial gains on $2 invested by participant A, can be exactly the same as 10 percent of financial losses on $100 invested by participant B. Both the appraisal and ICR fail here, praising the high IRR accruing to households, without noticing that this is merely because others are paying part of the bill.