Grenada - OECS Skills For Inclusive Growth

 ICRR 14346 Report Number : ICRR14346 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/05/2014 Country : Grenada Project ID : P095681 Appraisal Actual Project Name : Oecs (grenada) Skills US$M ): Project Costs (US$M): 4.48 3.39 For Inclusive Growth L/C Number : C4534 Loan/ US$M ): Loan /Credit (US$M): 3.00 3.06 Sector Board : Education US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 01/14/2009 Closing Date : 09/01/2013 09/01/2013 Sector (s): Public administration- Education (50%); Vocational training (40%); Other social services (10%) Theme (s): Education for the knowledge economy (50% - P); Education for all (17% - P); Social safety nets (17% - P); Improving labor markets (16% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Pia Schneider Denise A. Vaillancourt Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Financing Agreement (March 2009) the project development objective (PDO) was to assist the Recipient to increase the employability of youth through public /private sector partnerships for technical and life skills training that is demand driven . The PDO in the Project Appraisal Document (PAD) is identical. The PDO remained unchanged. However, in March 2013 during the Level-2 Restructuring, one PDO indicator was dropped (youth employed after training) and the target value of the other PDO indicator (number of youth certified at Level 1 or above) was reduced. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components: The three components supported under the project are presented below, showing costs estimated at appraisal (including contingencies) and actual costs: 1. Skills training for unemployed youth . (Appraisal: $2.09 million. Actual: $1.22 million (59% of Appraisal)). This component was to finance training of unemployed youth through a competitive, demand -driven training scheme  to provide quality, competency -based skills and life skills training to vulnerable youth, and to ensure that the training corresponds to labor market needs . The National Training Agency received training proposals based on employers' training needs and matched enrolled unemployed youth with winning proposals . 2. Establishing an occupational standards framework to improve the quality and value of training, (Appraisal: $0.43 million. Actual: $0.25 million (63% of Appraisal). This component was to finance activities related to the adoption of an occupational standards framework that is regionally recognized and adapted to the local context . It included the introduction of occupational standards, their vetting by local industries, and the establishment of quality assurance processes . 3. Institutional strengthening and project management . (Appraisal: $1.96 million. Actual: $1.95 million). This component included three parts . Institutional Strengthening was to support the Grenada Council for Technical and Vocational Education and Training (GCTVET) and the associated National Training Agency (NTA) in their role of establishing TVET policy and promoting the adoption of a regional, standards -based occupational framework. Project Management was to support the implementing agency and Project Coordination Unit (PCU) to closely monitor and evaluate the implementation of the project . Communications Campaign will support a targeted communications campaign focusing on the three main stakeholders : youth, training providers, and employers. The first and third components were revised in March 2013 as follows: 1. Skills training for employed and unemployed youth . The revised component was to finance skills training for employed and unemployed youth with private sector employers and training providers . Changes included (i) a reduction in training with an increased focus on Level II training in response to less demand for Level 1 skills; (ii) inclusion of unemployed youth and adults ages 16-40 regardless of educational background, and labor skill certification for employed youth; and (iii) an increase in the trainee stipend . 3. Institutional strengthening and project managemen t support was extended to allow financing of NTA operational costs and Management Information Systems . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : The actual project cost was US$ 3.39 million compared to the original appraised cost of $ 4.48 million (PAD p. 50) (76% of appraised). Financing : The project was part of an Adaptable Program Loan (APL) to the Organization of the Eastern Caribbean States (OECS) to undertake reforms and improve labor skills . This "horizontal" APL was applied sequentially across interested and eligible OECS countries to allow governments to join the program when they were ready and met the eligibility criteria. Phase 1 of the Credit went to St. Lucia (P097141) and Phase 2 to Grenada. The project did not disburse fully . The original loan amount was XDR 2.10 million (equal to US$ 3.00 million), of which XDR 1.99 million (or 95 percent) was disbursed. The balance of XDR 107,309 was cancelled. In US$ the disbursed amount was higher than the appraised amount due to currency gains against the XDR . Under the original design, US$ 0.60 million was to be raised from local private sector sources . The financing plan was revised in March 2013 to eliminate contributions by the private sector under Component 1. Actual contributions from private sources amounted to US$ 0.01 million (2% of appraisal). Borrower Contribution : The borrower was to contribute US$ 0.90 million. The actual contribution from the borrower was $ 0.32 million (36% of appraisal). Dates: Dates The project was restructured : April 2011: The Financing Agreement was amended to include "non-consultant services" as a procurement category to cover logistic al services, and to be in line with the Project Appraisal Document . March 2013: A Level Two Project Restructuring changed the scope of the first component which was amended to include employed and unemployed youth and to increase the value of trainee stipend paid to youth . The third component (project management) was changed to allow financing of operational and staff costs and of the design of a management information system with labor market data . The indicator targets were adjusted . Project costs and financing from the borrower and private sector were reduced in response to the economic downturn that had affected activities i n the key sectors of the economy and both the financial capacity of the Government and private sector . One project development objective indicator has been dropped, and indicator  target values were adjusted as shown below (Section 4). At the time of this restructuring, which was 5 months before closure, the loan had disbursed US$ 1.63 million (53% of actual loan amount). 3. Relevance of Objectives & Design: a. Relevance of Objectives: High The objective of improved employability of young people is highly relevant in the country's labor market context with high youth unemployment and shortages for specific skills among youth . The unemployment rate doubled between 2008 and 2012 when the global economic crisis negatively affected the tourism sector (GDP growth -1.5% in Grenada). The objective is aligned with the strategic objective of enhancing competitiveness and stimulating growth as identified in the Bank's Regional Partnership Strategy for 2009 - 2014. The objective is supporting government's efforts to increase the pool of skilled persons required to meet the demands of the labor market as identified in the New National Party's 2013 Manifesto. It is also in line with the OECS Education Sector Strategy which aims to provide opportunities to vocational training students to develop their skills and knowledge. The borrower's ICR reports that the shortage of skilled labor was cited by businesses as the most important constraint to increasing competitiveness . It is aligned with the World Bank Education Strategy 2020 which emphasizes the need to give young people appropriate opportunities to consolidate their basic knowledge and competencies, and equip them with technical or vocational skills that promote employment and entrepreneurship. b. Relevance of Design: Original : Modest Revised : Modest The original and revised design are both in line with the Bank strategy of a sub -regional/multi-country approach to lending. The design was informed by the St . Lucia project experience, the 2007 World Development Report on Youth and several analytical reports . The design was informed by a skills needs assessment in the hotel and tourism industry, and responded to an anticipated increased demand for trained staff in tourism high -margin and niche markets. The three components (skills training, training standards, and institutional strengthening ) were plausibly linked to the objective of improved employability . The focus was on skill-building with private sector participation. Demand-driven training and outcome-based payment schemes (based on the number of students certified) to training providers are relevant to better respond to labor market needs . However, there were several weaknesses . First, the costing of training was considered as unattractive to training providers. Second, the original design did not take into account the impact of the 2008 global economic crisis, and having employers pay for internships turned out to be too ambitious . Also, the original design did not consider financial implications for the case that employers will not pay a stipend to youths in training . Third, the ICR remarks that the design may have been over -ambitious because it included competency -based training (p. 5). Fourth, the original focus was on training youth for the tourism industry which was hardest hit following the 2008 crisis. Also, the design had a supply -side approach and did not comprehensively examine the current and future demand for trained youth and possible multi -sector collaboration to increase this demand . Only shortly before project closure - and two years into the economic crisis - was the design adjusted to address some of these weaknesses . The revised design added to the target group already employed youth, better meeting this particular demand of the private sector, and it expanded to other sectors less affected by the economic crisis. But the dropping of businesses' participation in the financing of interns /trainees working within their own institutions weakened the results chain making them less invested in their outcome . 4. Achievement of Objectives (Efficacy): Objective : Increase the employability of youth through public /private sector partnerships  Substantial for both the original and restructured project PDO and intermediate outcome indicator targets were scaled down under the March 2013 restructuring of the first and third component (see above) to reflect the economic impact in the aftermath of the 2008 global crisis which affected, among other things, employers ’ ability to offer paid internships to trainees . Output Out put National Qualifications register in operation 240 private sector companies took on trainees from the project exceeding both the original (45) and revised target (90). 220 youth enrolled in the training scheme at Level 2 exceeding both the original (40) and revised (80) target. 63 assessors were certified surpassing both the original (39) and revised target (58). 832 youth enrolled in the training scheme at Level 1 surpassing the revised target of 800 but not meeting the original target of 1,160 886 youth successfully completed a training program (including graduating from technical and life skills modules and completion of a traineeship in a public /private company) surpassing the revised target of 500 but not meeting the original target of 969 youth 55% of trainees in sectors other than tourism exceeding both the original (30%) and revised target (48%) The following output indicators were added during the March 2013 restructuring which added employed youth to the life skills training : National Training Agency (NTA) was established under the Vocational Education and Training Act of 2009, and is in charge for national coordination for technical and vocational education and training . However, the NTA was an effectiveness condition for this IDA credit; thus it is not clear why this indicator has been added five months before project closure . NTA was accredited (Caribbean Vocational Qualifications - CVQ status) by CARICOM, which is an independent endorsement of training standards . 100% employed youth included in the program were trained only in life skills exceeding the target of 75%. 35 employed youth assessed and certified at Level I or II (i.e., Prior Learning) not meeting the target of 50. Outcome 918 youth were certified (NVQ/CVQ) at Level I or above surpassing the original target of 814 and the revised target of 500 youth. This assessment is competency based and includes the perspective of employers on trainees’ skills and behavior during a trainee ’s internship program. Trainees only pass if their internship performance is rated satisfactory by employers . The percentage of youth enrolled in the training scheme who are employed fifteen months after the start of their training was dropped in March 2013 and not measured. However, the Grenada National Training Agency reports that 78% of individuals who graduated in October 2012 were employed by May 2014; and 58% of individuals who graduated in July 2013 were employed by May 2014. On average 63% are employed in May 2014, almost meeting the original target of 65% employed15 months after training. 5. Efficiency: Modest The Project Appraisal Document expected the youth training program to affect employment and average wages for youth. The economic analysis assumed that 40% of unemployed youth would find employment within 12 months after they became unemployed . Wages among the target group in Grenada were assumed to be ECD 830. Different scenarios are presented using discount rates from 5% to 10%. It was estimated that an increase in average wage income of 20% is needed to ensure a positive rate of return with a discount rate of 5% over ten years, while an increase in average wage income of 25% is needed to ensure a positive rate of return with a discount rate of 10%. The corresponding net present value is 700 at 5% discount rate and 667 at 10%. The corresponding Rate of Return is 12.17% (at 5% discount rate) and 11.60% (at 20% discount rate) respectively. The ICR estimates an Internal Rate of Return of 13.2% based on costs from Component 1, and assuming that over the next ten years, youth who have attended training will have a 10% increase in the probability of employment and 15% to 22.5% higher wages. However, the ICR does not provide actual data to show whether the assumptions used at appraisal and in the ICR (employment rate and wage among trained youth ) have been realized with the support provided under this project . The ICR does not explain how training has increased productivity based on which employers are willing to pay higher wages . The unit costs per youth graduated amounts to US$ 3,693 (Actual Project Costs divided by 918 youths).  The ICR reports efficient use of project funds because the training was restructured and shortened to respond to employers' needs. This resulted in 23% lower unit costs per trainee compared to the amount estimated at appraisal. However, this restructuring happened only late into the project, five months before closing . Forty-two percent of the total project cost of US$ 1.45 million was spent on project management, or 138 percent of the amount originally planned. The fact that half of the project funds were disbursed in the five last months of the projects suggests that there were bottlenecks that could have been addressed earlier during a restructuring, and that had contributed to inefficient use of project funds . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 12.17% 100% ICR estimate Yes 13.2% 36% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Project under the original and revised PDO targets : Moderately Satisfactory The sub-ratings are identical under the original and restructured project . Relevance of the project objective is High while the relevance of the original and revised design is Modest . Achievement of the objective is Substantial under the original and revised PDO indicator targets . Efficiency is rated Modest. The Outcome Rating is therefore Moderately Satisfactory . According to OPCS/IEG guidelines for restructured projects, the final outcome rating is determined according to the percentage of the credit that disbursed before and after project restructuring . This IDA credit had disbursed US$ 1.63 million (53% of actual loan amount) at the time of restructuring. But weighting is not needed in this case, because the ratings are identical before and after the restructuring . The combined outcome rating is Moderately Satisfactory. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR states that the Government ensures the continuation of the certification based training program and the functioning of the NTA. However there are several challenges that pose significant risks to development outcomes. Almost half of project funds were used to finance stipends of trainees, operational and staff costs at the NTA and the design of a management information system with labor market data . To ensure the sustainability of the vocational qualification framework, it is vital to ensure the financial sustainability of the NTA, which is granted by current legislation . The current budget allocation covers mainly salaries but not operational expenses. The fact that the private sector did not invest in these trainings /internships raises concerns about the future demand for training participants . The ICR (p. 14) also mentions that changes to the competency -based training program would make it less effective and less efficient because the intensity of training has been decreased, expanding it over a longer time period . This change is likely to increase drop -out rates and discourage employers from hiring permanently . Additional information provided by the Region suggests that future employment for certified trainees is positively influenced by strong economic growth projections and the recent construction of the Sandals La Source Hotel, which is expected to create about 1,500 jobs. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: During preparation the Bank team provided technical assistance to establish the NTA which was a condition for effectiveness. The team prepared an operation that was strategically relevant to the labor market context and included private sector training providers and employers . The team also worked closely with teams in St . Lucia and elsewhere to inform the design, and with the local stakeholders . However, the Bank team did not recognize that the global economic crisis in 2008 may negatively affect economic growth including in the tourism sector. The design was overly ambitious and the assumption that businesses would pay for the training was a leap, especially when the PAD (p. 2) notes that businesses (according to a baseline survey ) were not planning to expand very much during the project life . The results framework had weaknesses and did not include the relevant indicators to measure the project's objective of employability . The Bank's Education and the Social Protection and Labor team could have worked more closely to identify and address issues in labor market entrance for youth and project M&E, especially since youth employment is a multi-sector topic, currently addressed by both sectors . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: The Bank conducted regular supervision missions . However, the Bank team did not show flexibility during the first three project years and the mid -term review occurred late (in July 2012) (ICR p. 35). The team did not take action in 2010 when it became apparent that employers were not willing to offer paid internships to trainees. This resulted in delayed payments of stipends to trainees . By March 2013, five months before closure, about half of the loan had disbursed . Only then did the Bank team become proactive and restructured the project. Training was expanded to include other sectors not as hard hit by the economic crisis as well as educated and employed youth in response to employer demand . It restructured the constraining conditions in the project including the employer -based internship program for unemployed youth, which was subsequently financed by IDA through the Ministry of Youth . As a result, half of the project funds were disbursed in the last five months . Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The project received sustained political support from top government authorities under three different administrations. Legislation was passed to create the NTA; however, its operational costs were not accurately budgeted for by the government . Implementation was severely affected by the economic crisis and the resulting fiscal constraints negatively affected the borrower contribution to the project . The Ministry of Education partnered with the Ministry of Youth to coordinate youth employment programs and in 2012 the Ministry of Youth agreed to pay the stipends (instead of employers) until the project was restructured . Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance:  The project management unit was proactive in the restructuring of the project, and successfully increased the number of participating employers in the program . Training was revised and shortened to reduce training costs. The team worked also closely with the St Lucia team and learned from the experience including the joint development of the information monitoring system . Collaboration with other Caribbean countries informed the development of labor certification standards and certification of key assurance staff . The project coordination unit provided effective support to the NTA and helped coordinate with Ministries . Stability in the unit's management brought much needed continuity . The unit also shared implementation experience with other countries . There were temporary weaknesses in procurement in mid -2010. However these were identified and addressed . Financial management was strong. Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The original M&E design had weaknesses and these were not addressed during the March 2013 restructuring. The results framework included several process indicators, but it lacked an explicit indicator to assess "employability" and the extent to which training services and their outcomes were demand -driven (PDO specifies that training/services were to be demand-driven). The design at appraisal included one employment outcome indicator which was not measured by the project and dropped (percentage of youth enrolled in the training scheme who are employed fifteen months after the start of their training ) during the restructuring in March 2013. To measure employability the design could have included an indicator on the % of youth with internships who were offered a job. The M&E design did not involve the private sector either in the design or in the implementation. The design did also not include any provision to analyze “outcome-based payment schemes