Grenada - Oecs Education Development Project

 ICRR 13861 Report Number : ICRR13861 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/24/2012 Country : Grenada Project ID : P077759 Appraisal Actual Project Name : Grenada: Oecs US$M ): Project Costs (US$M): 11.3 11.4 Education Development Project L/C Number : C3809; C4542; L7187 Loan/ US$M): Loan /Credit (US$M): 8.0 9.56 Sector Board : Education Cofinancing (US$M): US$M ): 0.8 0.8 Cofinanciers : DFID Board Approval Date : 06/27/2003 Closing Date : 09/30/2007 06/30/2011 Sector (s): Secondary education (100%) Theme (s): Education for the knowledge economy (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Robert Mark Lacey Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: Original Objectives According to the Project Appraisal Document (PAD, page 4), the objectives of the project were : (i) to increase equitable access to secondary education; ii) to improve the quality of the teaching and learning process; and (ii) iii ) to strengthen management of the sector and governance of schools (iii) . The project was the second part of a "horizontal" Adaptable Program Loan (APL), approved in 2002 for sub-regional support to the education sector . The program supported countries of the Organization of Eastern Caribbean States (OECS), including Grenada. According to the PAD (page 4), the overall objective of the APL program was "to build human capital in the OECS which, in turn, will contribute to the diversification of their economy and more sustainable growth." The Development Credit Agreement (DCA, page 15) articulates the project objectives similarly to the PAD : (i) to increase equitable access to Secondary Education; (ii) to improve quality and efficiency of Secondary Education; and (iii) to strengthen the institutional capacity of the educational sector . As the PAD statement is slightly more specific (i.e. quality of "the teaching and learning process " rather than "secondary education"), it is used for the purposes of this review, while the additional dimension of improved efficiency in DCA objective (ii) is also included in the review. Revised Objectives Following Hurricane Ivan in September 2004, the objectives were formally revised * such that objective (i) was as follows: to increase equitable access to secondary education, including replacing and rehabilitating secondary schools damaged by the passage of Hurricane Ivan on September 7, 2004. 2004 . Objectives (ii) and (iii) were unchanged. With the completion of emergency activities related to the hurricane, the project was formally revised again in November 2008, reverting back to the original wording for all three project objectives . For the purposes of this review, the term "original project" refers to the two time periods in which the project was governed by the original objectives. These two periods were: from project effectiveness (January 2004) to the first revision (November 2004); and then second revision (November 2008) to project closing (June 2011). (Note: Only US$0.29 million, or less than 3%, of the loan had disbursed between effectiveness and the first revision ). The term "revised project" refers to the time period in which the project was governed by the revised objectives: from first revision (November 2004) to second revision (November 2008). * A Project Paper documenting the formal project revision was not located by IEG . According to the project team, such documentation was not required at the time; however, OPCS guidelines (B.P. 13.05), dated 2004, state otherwise. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 11/18/2004 c. Components: Original Project (Note: Appraisal estimates are provided in two amounts : original estimate and revised estimate (based on 2004 restructuring + 2008 additional financing)). 1: Increasing Equitable Access (Appraisal: US$1.17 million and US$6.3 million; Actual: US$6.67 million): This component aimed to increase access to secondary schools by expanding and rehabilitating three existing schools in the underserved parishes of St . George, St. Andrew, and St. David. Activities included construction and equipping of new classrooms in each school to relieve overcrowding and better meet the demand for secondary education . 2: Improving Quality (Appraisal: US$7.78 million and US$3.47; Actual: US$2.77 million): This component aimed to reform curriculum, improve teacher skills, improve the quality of student assessment, and improve student support services. Activities included developing curriculum emphasizing literacy and numeracy skills; training for education staff; creation of learning resource centers (LRCs); upgrading of science labs; developing a system for student assessment, including training on continuous assessment; and developing mechanisms to enhance students' ability to successfully participate in learning such as guidance counseling, peer counseling, and extra -curricular activities. 3: Strengthening Governance and Management (Appraisal: US$1.57 million and US$0.86 million; Actual: US$0.83 million): This component aimed to improve the level and efficiency of supervision at the school level and improve communication about and participation in the reform process . Activities included: establishing District Education Offices for supervision teams; training in school management and the use of school development plans; upgrading of the educational management information system (EMIS); and dissemination of information on the education reform process. 4: Project Management: (Appraisal: US$0.74 million and US$1.03 million; Actual: US$1.12 million): This component supported management of the project activities in the Ministry of Education (MOE). While the project management unit (PMU) was originally set up in the MOE, it was merged in 2005 with a newly established Project Coordination Unit (PCU) under the Ministry of Finance, which was to oversee all donor -funded projects. Revised Project (s) Although the four project components remained the same throughout the project period, the scope and financing of activities within each component were adjusted during the project restructurings . November 2004: Due to the passage of Hurricane Ivan, there was a need for emergency civil works . The scope of Components 2 and 3 were significantly reduced (activities related to literacy and numeracy, educational materials, and teacher training were scaled down ) while Component 1 was greatly expanded. Civil works related activities under Component 2 (learning resource centers and science labs ) were moved to Component 1, although the number of labs was reduced from 4 to 1, and the number of LRCs from 11 to 3. November 2008: Through the approval of Additional Financing, US$ 1.67 million was provided to Component 2, and US$0.46 million to Component 3, reinstating a number of the original activities . However, some activities such as the expansion of the third school (Grenville Secondary School), summer school programs and expansion of the EMIS were dropped . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost : The final project cost was US$11.4 million, compared to the estimated US$11.3 million. Financing : DFID provided co-financing of US$0.8 million. The first project revision in November 2004 increased Bank funds for Component 1 from US$1.17 million to US$6.3 million. Financing percentages were also increased from 75-80% to 100%. The second project revision in November 2008 provided additional Bank funds of US$ 2.36 million to implement activities under Components 2 and 3, as well as increased financing percentages to 100%. US$1.078 million of the loan was cancelled due to misprocurement (See Section 11). Borrower contribution : There were significant revisions to financing percentages, which decreased the total Borrower contribution from US$2.5 million to US$2.0 million. The counterpart funds were provided with no major difficulties reported . Dates : In November 2004, the project closing date was extended from September 2007, to June 2008, in order to complete civil works. In August 2006, the closing date was extended a second time, to December 2008, again to complete civil works. In November 2008, the closing date was extended a third time, to June 2011, to implement activities with the additional financing. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Original Project Substantial . The 2001 OECS Education Reform Strategy, which was adopted by the OECS Ministers of Education, supported reforms in secondary education, with a focus on access, relevance, quality, youth at risk, and institutional capacity. The accompanying Strategic Plan for Education and Development in Grenada (2001-2010) addressed the above issues and aimed at achieving universal secondary coverage, with the revised Strategic Plan (2006-2015) continuing the focus on these issues . At project closing, the objectives were also consistent with the Bank's Education Strategy 2020 and the OECS Partnership Strategy (FY10-14), which both emphasize the improvement of the quality and relevance of education, as well as the efficiency of spending in the sector . Revised Project Substantial . The first objective was revised to include replacing and rehabilitating secondary schools damaged by the passage of Hurricane Ivan . While the revision was not articulated as an outcome-based objective, it was nevertheless, substantially relevant given the emergency context in which over 85% of the country's school infrastructure was damaged. b. Relevance of Design: Original Project Substantial . The project's design was based on three country -specific objectives which would contribute to the overall regional APL objective to build human capital . The planned activities were likely to contribute to achieving the intended outcomes within the time frame of the project; however, the objectives to improve the quality and efficiency of the teaching and learning process and to improve management could have been articulated with more specificity so that clear, attributable outcomes could be better identified . Revised Project Modest . The revised design concentrated project resources on achieving the first objective, which was clearly the primary objective given the emergency context . While the second and third objectives were not changed, the related activities were significantly scaled back and therefore, it was unlikely that the remaining activities and resources would have been sufficient to achieve these secondary objectives . 4. Achievement of Objectives (Efficacy): Original Objectives : To increase equitable access to secondary education . Substantial , due to evidence of increased access in the project areas, which targeted underserved parishes . Outputs Creation of additional 420 places in two secondary schools, falling short of the target of 520 places. Overcrowding was reduced at those two schools : Grenada Boys' Secondary School and Westerhall Secondary School. Construction at the third school, Grenville Secondary School, did not take place due to a procurement conflict (See Section 11 below). However, the ICR reports (page 20) that when the planned expansion could no longer be carried out under the project, the government identified alternative facilities to accommodate students and utilized government resources for construction of permanent structures . Outcomes The transition rate to secondary education in the project areas (the underserved parishes of St . Andrew, St. David, and St. George), as a proportion of all students in those parishes sitting for the Common Entrance Exam, increased from 51% in 2002 to 72.3% in 2010, surpassing the target of 68%. (The project team confirmed that the rehabilitation of Grenville took place during the project period and thus contributed to the increase in transition rate reported, although it is more attributable to government, rather than to Bank, support ). To improve the quality and efficiency of the teaching and learning process . Modest , due to limited evidence of outcomes. Although exam results improved, the ICR (page 12) notes that other reform initiatives supported by the Government and other donors (including curriculum review, teacher training, literacy and numeracy programs ) were also taking place in the sector which could have impacted these results . Outputs Development of literacy and numeracy policies, which were incorporated into the national curriculum policy . Other subject policies developed included modern languages, creative and performing arts and dance, and information technology. Training of 84 teachers in literacy and numeracy at degree level, surpassing the target of 60. 158 teachers were also trained in literacy and numeracy through in -country training workshops . Training of 31 technical education teachers to upgrade their skills including pedagogy and the delivery of subject matter, achieving the target of 30. Training for teacher educators at T .A. Marryshow Community College (TAMCC), the local teacher training college. Strengthening of a Minimum Competency Test, now conducted annually for grades 2, 4, and 5. The ICR reports (page 13) that results are analyzed and used to tailor support to ensure that students enter secondary education with stronger skills. Development of handbook for continuous assessment, with 361 professionals trained in the use of continuous assessment. Expansion of textbook rental scheme to support 3,000 needy students. Training of 34 guidance counselors at degree level, all of whom are now placed in all 19 secondary schools. Training of 350 students as peer counselors . Implementation of student extra-curricular activities in 25 schools, surpassing the target of 19 schools. Construction of a Learning Resource Center and two science laboratories in GBSS . Outcomes The proportion of students passing at least five Caribbean Examination Council (CXC) Exams (including math and English) increased from 13% in 2000 to 19% in 2010, falling slightly short of the target of 20%. However, interventions to improve quality - curriculum reform and teaching methods - would have been unlikely to have an impact on exam results in such a short time frame . The ICR (page 12) reports the following observations from focus group discussions : The new curriculum is "better aligned with sub-regional CXC curriculum, better reflects the changing demands of the labor market, and meets diversified learning needs of students "; "The new subjects serve to enhance creativity of students, and provide opportunities for disadvantaged students "; The trained teachers have "greatly enhanced capacity to deal with students with a wider range of learning needs and abilities "; Student leaders confirm "usefulness of counselors, including reduction in incidences of student fights in school due to the preventive role played by counselors." The ICR (page 13) reports that external assessors "have provided positive feedback to TAMCC on the quality of its current graduates in the areas where trained tutors were deployed ." However, according to the ICR (page 18), teacher turnover is relatively high, particularly at the secondary level . However, The gross completion rate in secondary schools decreased from 82% in 2002 to 62.2% in 2011. The ICR (page 7) notes that dropouts from secondary schools may have increased as students needed to work to supplement  family income or took advantage of the increased opportunities to enter the labor market as a result of the hurricane. To strengthen the management of the sector and governance of schools . Modest , due to limited evidence of outcomes . Outputs Establishment of 7 District Education Offices, staffed with trained district education officers, curriculum development officers, guidance counselors, and early childhood education officers . Training of stakeholders in school management skills, including establishing school improvement teams and developing school development plans, Activities to expand the Education Management Information System (EMIS) were dropped. Outcomes The ICR (page 14) reports the following observations from focus group discussions : "The district offices have proved to be a useful link between the MOE and schools, providing a support network for schools in each district, facilitating knowledge sharing, and providing on -the-job training for school principals and teachers "; "The planning and self-evaluation process has helped them [school principals] to undertake more systematic planning to focus on weak areas ." The proportion of non-salary recurrent expenditure out of total recurrent expenditure increased from 13% in 2001 to 15.6% in 2011. However, the attribution of project activities to this outcome is unclear . The pupil-teacher ratio decreased from 23 in 2002 to 15 in 2001, falling far short of the targeted increase to 24. Revised Objectives : To increase equitable access to secondary education, including replacing and rehabilitating secondary schools damaged by the passage of Hurricane Ivan . Substantial , due to implementation of reconstruction activities in a context in which over 85% of the country's school infrastructure was damaged. Outputs Rehabilitation of two secondary schools (Grenada Boys and Westerhall). Rebuilding of eight secondary schools . Outcomes The ICR (page 11) states that project support led to reopening of schools earlier than would have occurred without project support, benefiting approximately 4,300 students (surpassing the target of 2,000). The transition rate to secondary education nationwide, as a proportion of all Grade 6 students, increased from 80% in 2002 to 89% in 2011, although falling short of the target of 94%. The transition rate to secondary education, as a proportion of all students sitting for the Common Entrance Exam, increased from 55% in 2002 to 97.8% in 2010, surpassing the target of 89%. However, The net enrollment rate at the secondary level decreased from 70% in 2002 to 61.4% in 2011. As noted previously, dropouts from secondary school may have increased as students needed to work to supplement family income or took advantage of the increased opportunities to enter the labor market as a result of the hurricane. To improve the quality and efficiency of the teaching and learning pr ocess . Modest See outputs and outcomes reported above . To strengthen the management of the sector and governance of schools . Modest See outputs and outcomes reported above . 5. Efficiency: The PAD presented an economic analysis with the assumption that the project would increase both education achievement and education quality, which would increase wages and lifetime income . The analysis calculated an economic rate of return of 30 percent and net present value of US$ 17.2 million. Original Project Modest , due to lack of information. The analysis and discussion in the ICR are largely centered on the revised project activities . Revised Project Substantial . The ICR in Annex 3 provides an ex-post economic analysis for Components 1 and 2, whose combined costs made up 83% of the total project cost. The analysis showed an economic rate of return of 15% and net present value of US$3.68 million. The reconstruction efforts allowed students to return to school earlier than otherwise; and although there was a delay in the implementation of quality improvement activities (Components 2 and 3), the benefits associated with quality would still be realized, albeit later in the project cycle . The ICR also reports that the project utilized online training instead of overseas training, an arrangement which was found to be cost-effective (US$6,900 unit cost, compared to US$23,600) and allowed an increase in the total number of people trained. There was a significant increase in the cost of building materials and labor, mainly due to a scarcity of building materials in the region as a result of the hurricane . This contributed to higher than planned costs of civil works for the targeted schools (the actual cost for Grenada Boys' was US$ 3.7 million, compared to estimated US$1.6 million; the actual cost for Westerhall was US$ 0.97 million, compared to US$0.37 million); the project team confirmed that the estimates were calculated before the hurricane. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 30% 100% ICR estimate Yes 15% 83% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The outcome of the original project is rated Moderately Satisfactory , based on substantial relevance of the objectives and design, and substantial achievement of the objective to increase equitable access to secondary education However, there was modest achievement of the objectives to improve the quality and efficiency of the teaching and learning process and to strengthen management of the sector and governance of schools due to limited evidence of outcomes. There was limited evidence of efficient use of project resources . The outcome of the revised project is rated Moderately Satisfactory , based on substantial relevance of the objectives and substantial achievement of the primary objective to increase equitable access to secondary education including replacing and rehabilitating schools damaged by the hurricane (over 85% of the country's school infrastructure). However, relevance of project design was rated modest as the activities related to the more secondary objectives were significantly scaled back and therefore, it was unlikely that the remaining activities and resources would have been sufficient to achieve outcomes . Achievement of the secondary objectives to improve the quality and efficiency of the teaching and learning process and to strengthen management of the sector and governance of schools were also rated modest due to limited evidence of outcomes . Efficiency was rated Substantial. Therefore, the combined outcome rating is Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The Government's commitment to the sector reforms is reflected in the updated sector strategy and the institutionalization of numerous project activities within the MOE . Focal point technical staff increased skills and competencies, although teacher turnover remains relatively high . A framework for maintenance of school infrastructure has been put in place and newly rehabilitated buildings were built according to hurricane -proof guidelines. The global financial crisis since 2008 continues to present some risk to the financial sustainability of reforms, as well as the ability of schools to raise funds for school improvement plans .  a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Preparation studies (including in the areas of teacher training, learning spaces, youth at risk, curriculum, and school mapping) contributed significantly to the project design . The project's design reflected wide participation of its stakeholders, including through community meetings and focus group discussions with teachers, students, principals, and parents. A project preparation committee, consisting of MOE staff, education officers, teachers, principals, teacher union representatives, NGO members, and other Ministry staff, helped to enhance ownership and support for reforms. However, there were significant shortcomings in the M&E design including the choice of key indicators (See Section 10). at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank team was responsive to significantly changing needs, as evidenced by the multiple restructurings, financing, and procurement arrangement changes . Funds were shifted to high priorities in a timely manner, with adjustments made to financing and procurement arrangements accordingly . The Borrower's ICR (reported on ICR, page 44) reports that "the successes achieved under the project can also be attributed to the quality of supervision received by the Bank " and that the Aide-Memoires were "extremely helpful in identifying activities completed in an effort to facilitate a smooth project implementation ." However, the ICR (page 20) notes that the Bank team "could have provided leadership to revise the project's results framework during the project to better align PDO indicators with project activities ." Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government demonstrated commitment to the policy reforms and activities of the project, as reflected by the establishment of supporting committees and provision of counterpart funds . The focus on development objectives, such as the improved quality of teaching and learning, resumed in the aftermath of the hurricane . However, procurement conflicts with the National Public Central Tenders Board (disagreement within the Government over the lowest bidder's qualifications which prolonged the approval process beyond the bid validity period) led to the dropping of planned construction, declaration of misprocurement, and cancellation of partial loan proceeds (See Section 11). The government also contracted only one firm to assess damages from the hurricane in all secondary schools and to design all school buildings prior to bidding for construction . This led to significant delays in the rehabilitation of target schools . Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The Ministry of Education (MOE) was the primary implementing agency, although the PCU was housed in the Ministry of Finance. Committees consisting of key technical staff were established to elaborating the various project activities during project preparation - the ICR (page 6) reports that these were "highly effective in increasing and maintaining the government's ownership through the project period . It also enhanced sustainability of project activities as many project activities were institutionalized within the MOE ." Clear division of work between the MOE and PCU allowed effective coordination, and continuity of key project members ensured smooth implementation. There were some delays in fiduciary processes (See Section 11) which led to implementation delays. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating :  Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E design had significant shortcomings . A number of the key indicators, particularly those intended to measure quality and management objectives, were not clearly defined nor linked to the specific project outputs . As the ICR notes (page 8), interventions to improve quality - curriculum reform and teaching methods - were not likely to impact exam results in the short time frame of the project . In addition, also noted in the ICR (page 8), the project relied on data to be provided by the Government, although capacity for data collection and analysis in the MOE (which was responsible for project M&E ) was found to be limited before the project start . b. M&E Implementation: The ICR (page 9) notes that there were data inaccuracies in outcome data (such as net enrollment rates), due to inaccurate population projections made in 2001. Targets were revised at the third restructuring in November 2008, taking into account the additional financing, although the indicators selected were still maintained . The EMIS activities, which would have addressed M&E capacity issues, were dropped from the project . c. M&E Utilization: Monitoring data was utilized for updating the sector strategy and annual budget plans . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: As the project was considered an Environmental Category "C" project, no safeguard policies were triggered . The ICR reports (page 9) that the Bank's regional environmental guidelines for education projects were utilized for civil works bidding and activities to ensure no negative impacts on the environment . b. Fiduciary Compliance: Financial management: There were initial problems of timeliness and quality in the financial monitoring reports and occasional delays in the submission of withdrawal applications to the Bank; however the ICR reports (page 9) that overall financial performance was adequate . There is very limited information provided in the ICR on financial management performance, although the project team reported that external audits were unqualified with no major issues reported. Procurement: In November 2004, more flexible procurement arrangements were established to support the emergency reconstruction efforts . However, planned construction at Grenville did not take place due to a procurement conflict with the National Public Central Tenders Board . The disagreement within the Government over the lowest bidder's qualifications prolonged the approval process beyond the bid validity period and annulled the initial bidding. Misprocurement was declared and led to cancellation of US$ 1.078 million of the loan (the ICR does not state when the misprocurement was declared and whether there was discussion of reallocating the funds ). Overall, procurement practices were generally in accordance with Bank guidelines although there were some staff capacity issues and delays . c. Unintended Impacts (positive or negative): Positive: According to the ICR (page 17), the online program for literacy and numeracy developed by the project is now being offered at the University of the West Indies, thus benefiting other countries in the Caribbean . d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments  Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn from IEG findings: When the means to achieve objectives are no longer adequate, the objectives should be formally dropped or reduced in scope to match the resources available . In the case of this project, the formal restructuring was not used as an opportunity to revised objectives as necessary . M&E indicators should clearly link project outputs to expected outcomes within the project time period . If this is not the case for the original M&E design, revisions should be undertaken during the project period . 14. Assessment Recommended? Yes No Why? To verify outcomes and document lessons learned, particularly as part of a cluster evaluation of secondary education projects in the sub -region. 15. Comments on Quality of ICR: The ICR quality is rated Satisfactory , but marginally so. Although the ICR is candid about shortcomings in the project results monitoring (and attempts to draw upon other sources of data and outcomes to assess results ), the links between project activities and outcomes are not always clearly articulated . The ICR does not adhere to guidelines in rating formally restructured projects, as separate outcome ratings should be determined for original vs . revised projects, based on assessments of relevance, efficacy, and efficiency (the ICR only provides separate assessments of efficacy ). There is very limited information provided on financial management performance . a.Quality of ICR Rating : Satisfactory