St. Lucia - Oecs (lc) Skills For Inclusive Growth

 ICRR 14347 Report Number : ICRR14347 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/05/2014 Country : St. Lucia Project ID : P097141 Appraisal Actual Project Name : Oecs (lc) Skills For US$M ): Project Costs (US$M): 5.32 5.52 Inclusive Growth L/C Number : C4300 Loan/ US$M ): Loan /Credit (US$M): 3.50 3.69 Sector Board : Education US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 05/08/2007 Closing Date : 03/15/2012 09/15/2013 Sector (s): Vocational training (51%); Central government administration (24%); Other social services (21%); General public administration sector (4%) Theme (s): Other social development (25% - P); Improving labor markets (25% - P); Education for the knowledge economy (24% - P); Vulnerability assessment and monitoring (13% - S); Other financial and private sector development (13% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Pia Schneider Denise A. Vaillancourt Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: The Project Development Objective (PDO) is to assist the Recipient to increase the employability of youth through private-sector driven training (Financing Agreement 2007). The Project Appraisal Document has an identical definition of objectives . The project objective remained unchanged . However, the target value for one PDO indicator (number of youth certified at Level 1 or above) was reduced in December 2011 and again in April 2013. The other PDO indicator (percent of enrolled youth employed fifteen months after the start of the training ) was dropped in April 2013. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components: The three components supported under the project are presented below, showing costs estimated at appraisal (including contingencies) and actual costs: 1. Skills training for unemployed youth (Appraisal US$ 3.55 million. Actual $ 3.05 million (87% of Appraisal). This component was to finance the operation and expansion of a training scheme to train approximately 1,850  unemployed youth who would subsequently be placed in an internship with private firms to gain on -the-job experience. Private firms were to pay a stipend to trainees . The National Skills Development Centre (NSDC) at the Ministry of Education was to implement this competitive training scheme (including selection of private sector training provision). Initially, the Scheme was to fund training within the hospitality sector which had the most employment vacancies. Thereafter, training was to be expanded to other priority sectors . Training providers were to be paid an amount of $907 per trainee to cover the four months of training; 30% of this amount was paid upon successful completion of training . The component also financed public awareness campaigns . 2. Improving the policy framework for training (Appraisal US$ 0.84 million. Actual $ 0.56 million (69% of Appraisal). This component was to finance activities to improve the policy framework for delivering training by enhancing collaboration for training among the Caribbean States to reap benefits of economies of scale, introducing occupational standards to increase the quality and value of training and transferability of skills . Activities to be financed included consultancies, employer ’s surveys, minor equipment, and travel expenses . It also was to finance the development of policy actions plans in two areas (at-risk youth and job-training) where there were several uncoordinated programs financed by different agencies . 3. Project management and institutional strengthening (Appraisal US$ 0.92 million . Actual $ 1.92 million (208% of Appraisal). This component was to finance project management including activities and term staff to implement the project, monitor and evaluate the project ’s impact, and strengthen the capacity of participating institutions to better implement, monitor and plan . Revisions to project components : During the December 2011 restructuring, changes were introduced to Component 1 including a reduction in the number of trainees and expansion of age range up to 40 years, the inclusion of already employed youth, and elimination of the monetary contribution of employers to the training scheme . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost, Financing and Borrower Contribution The appraised project cost was US$ 5.32 million and actual project cost US$ 5.52 million (ICR p.20). The project was part of an Adaptable Program Loan (APL) to the Organization of the Eastern Caribbean States (OECS) to undertake reforms and improve labor skills . This "horizontal" APL was applied sequentially across interested and eligible OECS countries to allow governments to join the program when they were ready and met the eligibility criteria . This first phase of the Credit went to St . Lucia, which was followed by Phase 2 to Grenada (P095681). The project was financed by an IDA Credit in the amount of XDR 2.40 million (US$ 3.50 million equivalent) of which XDR 2.39 million had disbursed and XDR 67.45 were cancelled. The disbursed amount in US$ 3.69 million was higher than the appraised amount due to currency gains against XDR . An additional $ 0.95 million were to be raised from the private sector in form of contributions to training cost . The actual amount contributed was $ 0.19 million (20% of Appraisal). The financing plan was revised to eliminate private sector contributions during the first restructuring in December 2011. The government was to contribute $ 0.87 million. The actual contribution was $ 1.44 million (165% of Appraisal), Dates There were two Level-2 restructuring: December 2011: This restructuring was to adjust the design to the changing circumstances caused by the global economic crisis. Revisions included (a) changes to the scope of activities under the first component (see above); (b) revisions to project financing including the reallocation of loan amounts across categories to compensate for the elimination of private sector financing; (c) update of the Results Framework including revision of indicator targets to reflect the changes in scope of activities; and (d) an extension of the closing date by 18 months from March 15, 2012 to September 15, 2013 to allow sufficient time for additional cohorts to complete training, establish the quality assurance framework and disseminate results . At the time of this first restructuring, the IDA Credit had disbursed $ 1.99 million (or 54% of the actual amount). April 2013: The PDO indicator and targets were revised . The IDA Credit had $2.99 million disbursed (81% of actual amount). 3. Relevance of Objectives & Design: a. Relevance of Objectives: High The objective of improved employability of young people is highly relevant in the country's labor market context with high youth unemployment of 39% in 2005 and shortages of specific skills among youth . Employees in the tourism industry lack the necessary skills, and technical and vocational education (TVET) is still a priority for the country given the high drop out from secondary schools and poor achievement among students (ICR p. 29). The objective is aligned with the strategic objective of enhancing competitiveness and stimulating growth as identified in the Bank's Regional Partnership Strategy for 2009 - 2014. The objective is supporting government efforts (as outlined in the 2013 Budget Speech) to increase the pool of skilled persons to enhance quality and productivity while tackling stubbornly high unemployment in order to ignite the country's growth potential . It is also in line with the OECS Education Sector Strategy which aims to provide opportunities to vocational training students to develop their skills and knowledge, and the government's TVET strategy for 2014 - 2019, which highlights the shortage of skilled labor . Training of unemployed youth is also a priority in the Ministry of Education Development Plan (2009-2013). b. Relevance of Design: Original : Modest Revised : Modest The project design with its components remained essentially the same for the original and restructured project . The design responded to the need for better skilled youth to enter the labor market and it was informed by analytical work including a labor market assessment . Training included relevant features focusing particularly on unemployed youth, demand-driven training, incorporation of soft skills training, and output based payment to training providers subject to trainees successfully obtaining certification . However, skills training became less relevant when the global economic downturn negatively affected the labor market absorption capacity and the demand by employers for employer -based internships. Also, the ICR mentions that the original design was a template for larger Latin American countries, which needed to be restructured to be more responsive to the realities of a smaller island state, such as St Lucia (p. 29-30). The design was revised in 2011 and 2013 mainly by reducing the targets for the number of trainees, and by expanding training to sectors other than tourism , not as hard hit by the global economic crisis . These targets were more achievable and realistic . In addition, the design was changed to move from employer -paid internships to publicly paid internship in sectors . The stipend amount was increased and financed by IDA and the government as employers did not make their planned contributions . The design was over-ambitious as it included the simultaneous development of a vocational qualification framework and the establishment of a demand -driven, competency-based training mechanism, although the latter depended on the framework . The ICR (p. 29) also states that the design should have catered to those already employed, whose knowledge needed to be assessed and certified . In addition, it did not adequately address the situation of trainees with social vulnerabilities . However, the government in its subsequent response highlighted that this was recognized and the collaboration with the Division of Human Services continues to be a major component in the National Skills Development Center . The results chain had weaknesses . While certified training and accreditation are relevant for signaling purpose to employers, evaluation of several training programs show that additional measures are needed to improve the employability of youth including job search assistance and attention to job creation . Also, the main outcome indicator related to employability was not measured and dropped during the second restructuring . 4. Achievement of Objectives (Efficacy): Objective : Increase the employability of youth through private -sector driven training Original and Revised : Substantial The original indicator targets were revised in December 2010 and then a second time in April 2013, five months before project closure.  Output National Qualifications register was operational in March 2012. Action plan to improve the relevance and sustainability of training was achieved in March 2012. The TVET unit at the Ministry of Education was accredited in March 2103 (Caribbean Vocational Qualifications - CVQ status) by CARICOM, which is an independent endorsement of training standards . However, the Bank's labor needs assessment (2013) shows that only 9% of firms interviewed were aware of the CVQ assessment and only 2% said they have used it to some extent . 1,170 unemployed youth were enrolled in the training, not meeting the original target of 1,850, but surpassing both the first revised target of 870 (revised in 2011) and the second revised target of 945 youth (revised in 2013). 95.6% of enrolled youth certified one year after the start of the training surpassing both the original target of 71% and revised target of 65% (revised in 2011). 50% of enrolled youth in training in other sectors than hospitality surpassing both the original target of 30% and revised target of 38% (revised in 2013). 122 pre-qualified or accredited training providers in competency -based training, surpassing both the original target of 100 and both revised targets of 65 (2011) and 117 (2013). Drop-out rates (from training) was reduced from 26% in 2012 to 8% in 2013 due to increased stipend amount paid to trainees, expedited payments, expanded training options and reduced training time span . No data on drop-out rates prior to 2012. 277 employers provided traineeship (no target). Indicator added during the 2011 restructuring: 101 standards were validated by 30 industry advisers surpassing the target of 87 standards. Outcome 1,119 youth were certified at Level 1 or above not meeting the original target of 1,265, but exceeding both the first revised target of 726 and the second revised target of 407 youth. 234 Level-2 international certifications were awarded in the areas of hospitality; 694 in Level-1, and 218 in Level-2 (with 27 youth receiving 2 certificates). This assessment is competency based and integrates the perspective of employers on trainees’ skills and behaviors that are critical to gain employment and being successful in a given field. Trainees only pass the internship programs if they are rated Satisfactory by the employers . The ICR reports that 43% of certified trainees (or 481 youth) were employed within less than 6 months after the start of the training. Percentage of enrolled youth employed fifteen months after the start of the training was not measured by the project and dropped in April 2013. The original target was 75% and the revised target of 65% was set in 2011, before the indicator was dropped . However, the St. Lucia National Skills Development Center reports that at least 66% of certified trainees were employed 15 months since training start meeting the revised target but not meeting the original target, 74% after a period of 22 months and 80% after 34 months. 5. Efficiency: Modest The PAD indicated that the project through its support to youth training would affect the probability of employment and wages for employed youth . Various scenarios of employment and wage impact were used to compute the Net Present Value and Rate of Return of Project investments . An increase in average wage income of 20% would have been needed to ensure a positive rate of return at a discount rate of 5%, 7.5% and 10%. The corresponding Rate of Return was anticipated to range from 27% (discount rate of 5%) to 2.43% (at 10% discount rate). The ICR calculated Internal Economic Rate of Return (IERR) of 8.5% for component 1, representing 57% of actual project cost. It assumed that training participants will have a 20% higher employment rate and 18.2% higher wages. However, no information is provided to confirm the actual values of these assumptions at project closure. The ICR does not explain how training has increased productivity based on which employers are willing to pay higher wages. The ICR notes several additional benefits that could possibly materialize in the future as a result of competency-based training. The government subsequently reported that average salaries for certified trainees have increased nominally from XCD 830 to XCD 1,110, reflecting a 33% nominal increase. The unit costs per person who has graduated under this project is US$ 4,754 (actual project costs divided by  1,119 youth). Unit costs are considerably higher and amount to $ 11,060 per certified trainee employed within 6 months. These total costs include start up, training and administrative expenses financed under the different project components. The ICR reports that training costs were more expensive than anticipated as the government focus changed to costlier Level -2 training and international certification . The reduction in dropout rates suggest that the delivery of training programs funded under this project, had become more efficient towards project closure, and delays in stipend payments were addressed (p. 34). However, employers complained about the excessive administrative work related to internships, which was complex and time consuming for employers and training providers, and created additional costs . Also, the project needed 18 additional months to implement the activities originally planned . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal Yes 27% 100% ICR estimate Yes 8.5% 57% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Project under the original and revised PDO targets : Moderately Satisfactory The sub-ratings under the original, first and second restructured PDO targets are identical . Relevance of the project objective is High while the relevance of the original and revised design is Modest . Achievement of the objective is Substantial under the original and revised PDO indicator targets . Efficiency is rated Modest. The Outcome Rating is therefore Moderately Satisfactory . According to OPCS/IEG guidelines for restructured projects, the final outcome rating is determined according to the percentage of the credit that disbursed before and after project restructuring . This IDA credit had disbursed 54% of actual loan amount at the time of the first restructuring in December 2010 and 81% at the second restructuring in April 2013. But weighting is not needed in this case because the ratings are identical before and after the restructuring. The combined outcome rating is Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR states that the strong support by the government for the newly established vocational framework and youth development as well as the accreditation review planned for August 2014 by the regional accreditation authority, will help sustain progress made in vocational training . Commitment to the program is also shown by mainstreaming the utilization of competency -based standards in publicly funded training . The risk to improved employability is considered significant if demand -side factors related to job creation do not improve . Similarly, the risk that internships with stipends paid by the project will not continue is reduced as the government is now co-paying a stipend to trainees . There are other financial challenges that pose a risk to project outcomes and sustainability including the continuous training and financing of trainers (p. 40). The government finances the existing services by the NSDC and is implementing an action plan to sustain the reforms introduced under this project. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The Bank team developed the project in collaboration with stakeholders including employers from the tourism sector to identify shortages in skills . Project preparation fostered collaboration with other countries, particularly Grenada where a similar operation was to be implemented . With the change in government in 2006, the Bank was flexible and agreed to expand training to other sectors . However, the team developed a design that was over-ambitious and did not identify the relevant risks . The Bank team did not recognize that the demand for training and employability will be affected by structural changes in the economy and job creation. Also, the ICR (p. 29-30) mentions several activities that should have been implemented (such as the CVQ qualification framework) during the project preparation phase, and which would have helped to receive CVQ status earlier and increase the number of CVQs issued . The Bank missed the opportunity to accurately adapt the design (which was a replication from other Latin American countries ) to the local context, and this resulted in delays (p. 30). The results framework had weaknesses as it did not include a plan for how to measure employability . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Supervision was conducted regularly . The Bank also organized a South -South exchange to Columbia to visit training centers. However, initially the Bank was inflexible (p. 39). The team became proactive and restructured the project twice shortly before the original and revised closing dates . The first restructuring at the end of December 2011 was late to address the reduced demand for employer -paid internships which started to decline in 2009 when the global economic crisis affected the tourism sector . When employers did not contribute to training cost, the team was proactive and ensured financing by the Government and under the project. The Bank supported the government in its collaboration with the private sector collaboration . The team did not use the opportunity to revise the M&E framework to measure project outcome . Instead target values were reduced for input and output indicators . There was insufficient attention to the development impact of the project . The main outcome indicator was not measured and then dropped shortly before project closure. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR mentions that while the institutional arrangements were in place, leadership and ownership of the project was a major concern (p. 30). Still, the government (under both administrations in office ) showed support for the project. Borrower contribution was considerably higher than estimated at appraisal despite the tight fiscal circumstances. The steering committee did not meet regularly, but still, coordination was good between the Ministry of Education and other Ministries . The Ministry of Agriculture has stated its commitment to adopt competency based training for all its extension services training . The 2013 graduation ceremony was attended by the country ’s highest authorities, including the Governor General, the Permanent Secretary (PS) in the Office of the Prime Minister, the Minister of Education, the PS in the Minister of Finance PS, members of the diplomatic core, and foreign ambassadors . Government Performance Rating Satisfactory b. Implementing Agency Performance:  The Ministry of Education and NSDC were responsible for project management and training, whereas the Project Unit was in charge of fiduciary activities . Coordination with key stakeholders and the private sector was effective. The Project Unit provided effective support to the TVET unit and the National Skills Development Center. Monitoring of indicators was done regularly . Project implementation was delayed by the absence of an operational system, staffing to support the system of certification and an insufficient number of training providers (ICR p. 31). These were all part of the qualification framework that was being set up under the Project and had to be established as a system . Financial Management were deemed satisfactory. Overall, procurement was satisfactory . Procurement weaknesses (see Section 11b) were addressed in 2012 (see ICR, page 7). There was effective collaboration with Grenada on the M&E framework for both projects. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The results framework included several process indicators, but it lacked an explicit indicator to assess "employability". The design at appraisal included one employability outcome indicator which was not measured and dropped (percentage of youth enrolled in the training scheme who are employed fifteen months after the start of their training) during the restructuring in April 2013. To measure employability the design could have included surveys to elicit employers' view on trainees and quality of training, as well as tracer surveys to collect data on employment and wages among graduates . Also the M&E design did not capture the effect of the vocational qualification framework which was a major focus of the project . b. M&E Implementation: The M&E design built upon the NSDC's information system . A management information system (MIS) has been developed with Grenada to support the M&E needs of the newly established vocational qualification framework . The ICR mentions that it would have been more beneficial for the project manager to be housed in the Ministry of Education for effective M&E of the project (p. 29). The PAD (p. 17) notes that the M&E of outcomes/results would include a tracer study that will track the impact of training in terms of employment and salary of the trainees. However, the ICR does not report any results from the tracer study . c. M&E Utilization: Project progress was monitored based on input and output indicators . The MIS was launched and provides a web based application system to allow Grenada and St Lucia to monitor the training process . The first registry of labor skills certificates being issued was created . The information management system could provide the basis for a regional labor-skills certification registry. Also the Ministry of Education has conducted an impact assessment to fully learn from project implementation . Although the project did not collect the employment data, the task team subsequently reported employment information collected by the St Lucia National Skills Development Center. M&E Quality Rating : Modest 11. Other Issues a. Safeguards:  The Environmental Screening Category was C . No safeguard policies were triggered . b. Fiduciary Compliance: Financial Management was in accordance wit h guidelines and audits were submitted on time . The ICR does not note whether any audits were qualified . Procurement was slow. In 2012, procurement was downgraded because the Project Coordination Unit did not follow Bank guidelines which resulted in delays . Corrective actions were taken and procurement was subsequently upgraded (see ICR, page 7). c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Satisfactory Efficiency is rated Modest. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Moderately Satisfactory There were moderate weaknesses in project implementation including slow procurement. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR presents some interesting lessons, that have been adapted by the IEG as follows : It is possible to measure employability of graduates without conducting a randomized control trial . While not done in this project, the experience from other countries shows that training institutions can conduct tracer surveys of their graduates to assess employment and wage data . The PAD (p. 12) envisaged a tracer study to identify the impact of the investment . In addition, employer surveys provide information on satisfaction with training and how it affects employability . Several life factors affect dropout rates that are beyond control of trainers . When training institutes receive  an output based payment per graduate, factors they cannot influence should be considered in the payment formula. Also, if vulnerable youth are the target groups for training, adding a social worker or counselor may be helpful for youth and trainers, to keep them in training and enter employment In this case, there were only 5 counselors, which was not enough for more than 1,000 trainees (ICR p. 33). Administrative workload for internships can pose obstacles . In this case, employers complained about the excessive administrative work which was complex and time consuming for employers and training providers . Administrative complexity creates extra cost that can be prevented . The IEG youth employment evaluation has highlighted the need for cross -sector collaboration . In this case, collaboration between different teams that work on education and job creation in different sectors would have been useful to bring together more effectively the demand and supply side . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is satisfactory in its presentation of evidence . The analysis could have been more critical . The borrower ICR provides helpful additional analysis, and very practical and interesting lessons and discussion of project implementation. Lessons are based on evidence and analysis . The ICR is internally consistent and consistent with guidelines. The ICR shows actual costs to be US$ 5.52 million equivalent, but it shows actual financing to total US$5.32 million. Total project cost should equal total project financing . a.Quality of ICR Rating : Satisfactory