EXECUTIVE BOARD 135th session Provisional agenda item 6.1
EB135/5 7 May 2014
Evaluation: annual report 1. The Executive Board at its 131st session approved the WHO evaluation policy.1 The policy, inter alia, requires the Office of Internal Oversight Services to report annually to the Executive Board through the Programme, Budget and Administration Committee on progress in the implementation of evaluation activities. The present document (a) provides information on the progress made by the Secretariat in implementing the WHO evaluation policy; (b) presents a summary of the lessons learnt from evaluations conducted at WHO, and (c) proposes the WHO-wide evaluation work plan for 2014–2015 for consideration.
PROGRESS MADE BY THE SECRETARIAT IN IMPLEMENTING THE WHO EVALUATION POLICY Strengthening of the Secretariat’s capacity to implement the corporate evaluation function 2. The Office of Internal Oversight Services is in the process of filling four vacancies for staff to work on areas related to evaluation in 2014.
Global Network on Evaluation in 2014 3. The participants in the annual meeting of the Global Network on Evaluation 2 held in December 2013 discussed the progress made in implementing the Network’s plans and the experiences and challenges they had faced. They agreed on a new plan for 2014 that addresses needs related to the institutionalization of evaluation in WHO at two levels. 4. Firstly, at senior management level, there is a need to foster strategic support for evaluation through appropriate advocacy and communication approaches. This would include re-orienting and re-energizing the Network to address the functional gaps relating to the composition of its members or its task forces. In addition, it would include finding the support needed for the systems and formally establishing the procedures by which the Network operates and communicates with its regional and country levels.
1 2
See decision EB131(1).
The Network has a total of 26 members drawn from the three levels of the Organization, as well as from related global functions.
EB135/5
5. Second, the Network will concentrate on several outputs. Some of these continue actions initiated in 2013, such as the conversion of the WHO evaluation practice handbook 1 into an electronic tool for learning, the completion and updating of the evaluation registry, 2 and the analysis of evaluation products. Others are new activities, such as the launch of a quality assurance system for evaluation across WHO, and specific activities to strengthen capacity and knowledge transfer using modern network technology.
LESSONS LEARNT FROM EVALUATIONS 6. The following summary sets out the key lessons drawn from the 28 reports across the Organization that qualify as evaluations. Using the definition of “evaluation” in the WHO evaluation policy and assessing coverage in terms of the categories established in the Twelfth General Programme of Work, 39% (n=11) of the evaluations fall under Category 1, 29% (n=8) under Category 4, and the remaining 32% under Category 2, 3 and 6 (Figure 1). Evaluations in the African Region constitute 36%, and those in headquarters and the South-East Asia Region each represent 21%, with the remaining 22% coming from the Region of the Americas, the European Region and the Western Pacific Region combined (Figure 2). Figure 1. Evaluation reports by category under the Twelfth General Programme of Work, 2014–2019 12 10 8 6 4 3 2 0 2 0 Category 1 Category 2 Category 3 Category 4 Category 5 Category 6 4 8
11
Source: Evaluation registry (https://extranet.who.int/evaluationregistry/Report.aspx)
1 2
WHO evaluation practice handbook. Geneva: World Health Organization; 2013.
The WHO evaluation registry includes 226 reports (2008 to 2014) of which 54 (24%) are available electronically. Working through the Global Network on Evaluation, the Office of Internal Oversight Services used a two-step process to (a) check the compliance of the reports with the WHO evaluation policy and (b) assess the quality of those reports classified as “evaluation reports”. The Office of Internal Oversight Services developed, field tested and revised the set of quality control tools and submitted them to the Global Network on Evaluation for validation and endorsement.
2
EB135/5
Figure 2. Evaluations commissioned, by WHO major office 12 10 8 6 6 4 2 0 0 AFRO AMRO EMRO EURO SEARO WPRO HQ/Global 3 2 1 6 10
Source: Evaluation registry (https://extranet.who.int/evaluationregistry/Report.aspx)
7. Of the 28 evaluations, 46% were programmatic, 29% were thematic and 25% were office-specific. WHO had a major role in managing the programme, project or function at global, regional or country level in 43% of the evaluations, whereas 57% of the evaluations concerned national programmes led by country stakeholders.
8. The guidelines contained in the WHO evaluation policy require that the quality assessment of evaluation reports includes criteria in the areas of planning, implementation and reporting. The assessment (Table 1) found that 75% of the reports were of “good” quality. The structure of the reports, the description of the purpose, objectives and scope of the evaluations, and their findings and recommendations were either “good” or “acceptable” in most cases. However, only 57% followed “good” or “acceptable” methodology, while 79% of the reports lacked references as to how the evaluation addressed issues of gender, human rights and equality. Other main gaps identified related to (a) insufficient information about the criteria on which the evaluation was made; (b) incompleteness of evaluation reports; and (c) lack of clarity with respect to a strategy for the public dissemination of evaluation reports. Table 1. Completeness and quality of evaluation reports (n=28) Rating Criterion 1. Planning of the evaluation Specifications of the terms of reference Planning proposal 2. 3. Quality assurance mechanisms during the evaluation Governance Quality of evaluation reports Structure of the report Description of the subject of the evaluation Evaluation purpose, objectives and scope Methodology Findings 23 24 20 10 22 82 86 71 36 78 0 2 5 6 5 0 7 18 21 18 5 2 3 12 1 18 7 11 43 3 10 36 17 61 1 3 15 2 53 7 10 9 36 32 3 17 11 61 Good Number % Acceptable Number % Poor Number %
3
EB135/5
Rating Criterion Conclusions Recommendations Gender, human rights and equality 21 18 1 Good Number % 75 64 3 Acceptable Number 3 8 5 % 11 29 18 4 2 22 Poor Number % 14 7 79
Lessons learnt from the evaluation process 9. Standardized information needs to be provided in reports about the methodology used in evaluations, to allow for their systematic review and to enhance the evidence that they provide. The approach to the evaluation and its quality control mechanisms were well documented in 11 evaluation reports. These (programmatic) evaluations followed a WHO guideline for national programme reviews and thus used identical methodology and tools. 1 However, in most of the remaining evaluation reports, this information was judged to be insufficient. For instance, those reports mentioned the fact that the evaluation included a combination of desk reviews and interviews with key informants, including WHO staff and other stakeholders, but they often lacked information about the criteria adopted or the process followed to ensure appropriate quality and/or independence of the evaluation process. 10. Evaluation reports also need to address the expected and unexpected results of what is being evaluated, in order to allow for proper institutional learning. Only eight reports (29%) addressed the challenges faced when assessing the results of a programme or function. Some of the challenges reported included a lack of clarity about the indicators used to assess results; insufficient information to validate claimed achievements; and inefficient systems for documenting activities and results. Reports also mentioned inadequate monitoring and analysis of data and poor sharing of information among partners as additional challenges during the evaluation process.
Lessons learnt from implementation 11. A high level of political commitment and good programme governance are the main enablers of the success of national or global health programmes (expressed in terms of the attainment of goals and objectives). The evaluation reports suggest that the vast majority of programmes, projects or functions “met most of their objectives/goals”, while a few of them “exceeded most of the objectives”. The most important factor that helped programmes, projects or functions to meet their objectives was their ability to secure a high level of political commitment through sustained advocacy. Other key factors that helped them to meet or exceed their objectives included establishing proper governance and coordination mechanisms, involving key partners at the respective levels (global, regional, country); developing strategic plans and monitoring their implementation; and making the programme, project or function part of the national development agenda.
Malaria programme reviews: a manual for reviewing the performance of malaria control and elimination programmes. Trial edition, March 2010. Geneva: World Health Organization; 2010.
1
4
EB135/5
12. A few evaluation reports suggest that the programme, project or function being evaluated was “doubtful of meeting the objectives”. The factors most frequently cited as hindering the attainment of objectives included increased dependency on donors, a low budget allocation, a lack of sustained leadership, weak implementation capacity owing to high staff turnover, and/or insufficient intersectoral collaboration. 13. There is a need to improve the methodology for evaluating the impact and sustainability of a programme, project or function. The focus of most evaluations was limited to the attainment of objectives. Some evaluations reported on impact, while only a few considered issues of sustainability. The nature and type of impact varied, according to the nature of the objectives or goals of the programme, project or function, and the degree of its development. For example, some evaluations focused on the coverage of a programme’s health services in a country, while anoth er looked at the decline of a given public health problem as a result of the national prevention and control programme. 14. Some reports noted the challenges faced by evaluation teams in assessing impact or attributing it to the programme, project or function. The three main reasons cited included that: (a) the evaluation happened too early in the results chain; (b) it was difficult to attribute impact to stakeholders in collaborative arrangements; and (c) it was difficult to attribute changes in the population’s health status to the programme, project or function. Those evaluations that addressed the issue of sustainability signalled the engagement of stakeholders, including target communities, as one of the key enablers for sustaining the impact of the programme. The difficulty faced by a programme, project or function in securing funding was cited as the most common issue hindering its sustainability. 15. An appropriate level of resources is vital for delivery of a programme, project or function. The challenges in programme implementation most frequently cited in the evaluation reports are linked to financial constraints, the management of human resources and/or the setting in which the programme is implemented and relations with stakeholders. In particular, the evaluation reports on WHO programmes and functions mentioned low budget allocations, reductions in staff owing to financial constraints, unrealistically optimistic financial projections, increased staff costs resulting in reduced activity costs, and reductions in contributions from major donors. The area of human resources stands out as the key challenge for programme delivery. Specific issues include staff cuts, delays in recruitment, frequent staff turnover and related lack of continuity of leadership, and the lack of capacity of staff to deliver. Other important challenges to the effective implementation of a programme or function relate to the lack of strategic planning, sustained follow-up to the good work done initially, clarity about the functions of the different stakeholders and goals of the programme, and intersectoral collaboration.
16. The WHO Secretariat needs to become more efficient, in order to improve its effectiveness at beneficiary level. WHO managed or co-managed 43% of the programmes or functions evaluated, and those evaluation reports make reference to the roles played by WHO in implementation and attainment of the goals. The key factors that affected effective implementation of WHO-managed programmes or functions are related to administrative delays in signing contractual arrangements with partners, such as memoranda of understanding, releasing funds and recruiting staff. 17. WHO needs to act on the recommendations of the evaluations and to monitor and follow up their implementation. On an average, there are approximately 22 recommendations per evaluation (n=608; range 1 to 79). Over 60% of the reports contain less than 20 recommendations, while about 25% of them contain over 30 recommendations (Table 2).
5
EB135/5
Table 2. Evaluation reports, by number of recommendations Number of recommendations 1–10 11–20 21–30 31–40 >40 Total Number of evaluation reports 10 7 4 3 4 28 % 36 25 14 11 14 100
18. The recommendations address aspects related to results in 49% of cases, to processes in 34% and to settings 1 in 17%. Among the specific recommendations, the three most frequent relate to the efficiency of a programme’s management and implementation, its effectiveness in terms of coverage and surveillance, and its strategic presence and partnerships (including resource mobilization). Table 3 provides further information on the recommendations. Few of the evaluations suggest that the evaluators shared the draft report among the key stakeholders for feedback before finalizing the report. Furthermore, none of the reports provides any clear indication on the follow-up or use made of the recommendations by those who commissioned the evaluation. Table 3. Recommended areas for improvement in a total of 28 evaluations (n=608) Broad Component No. Category Setting 1 Strategic presence and partnerships 2 Accountability 3 Relevance of programme Process 1 Efficiency Subcategory Governance Total, setting Implementation capacity Programme management Strategic information Human resources Financial management Procurement Value for money Total, process Coverage Surveillance Infrastructure Subtotal Subtotal 3 Advocacy Communication Leadership Subtotal 4 Impact at beneficiary level Quality of services/interventions Programme impact Subtotal Number 69 22 14 105 71 47 38 23 18 11 1 209 41 35 4 80 53 53 38 19 57 37 9 46 % 11 4 2 17
34
Results
1
Effectiveness
13 9
2
Programme monitoring and evaluation
9
8
1
The “setting” refers to the strategic presence and partnerships, accountability and relevance of a programme or
project.
6
EB135/5
Broad Component
No. Category 5 Sustainability
Subcategory Financial Programme Subtotal
6 7
Programme research Subtotal Normative functions Legislation Subtotal Total, results GRAND TOTAL
Number 28 11 39 15 15 4 4 294 608
%
7 2 1 49 100
Cross-cutting issues 19. Evaluations need to address and report on how the programme or function deals with gender perspectives (gender equality and women’s empowerment). The Global Network on Evaluation assessed how the evaluation reports had addressed gender, human rights and equality, as recommended in the WHO evaluation policy and the WHO evaluation practice handbook. It found that 79% of the reports lacked specific reference as to how the evaluations had addressed those issues. Additionally, the Gender Equity and Human Rights unit also assessed the aspects of gender equality and women’s empowerment in the evaluation reports, using the scorecard tool developed in the context of the UN System-wide Action Plan. 1 WHO results show a low average score (0.30) and hence do not meet the requirements for gender equality and women’s empo werment. The average score was brought down by a lack of: (a) evaluation questions that specifically addressed how gender equality and women’s empowerment had been integrated into the design, planning, implementation of the intervention and the results achieved; (b) a gender-responsive evaluation approach and methodology; (c) methods and tools to collect data related to gender equality and women’s empowerment; (d) data analysis techniques that included gender analysis; and (e) a process for validating the findings, conclusions and recommendations of the evaluations with specific reference to the participation of women, men, boys and girls from stakeholder groups. Moreover, this assessment noted the absence in the reports of any gender-responsive conclusions and recommendations. As WHO must report annually on the indicators agreed as part of the UN System-wide Action Plan, all managers will need to include the scorecard tool in the evaluation of their programmes, projects or functions.
Concluding remarks 20. This assessment is the first step in the systematic analysis of evaluations and the follow-up and monitoring of their recommendations in WHO. It has been made at a time when the evaluation registry is still incomplete. The insights gained and lessons learnt with regard to the evaluation process itself and the implementation of public health programmes may therefore not be wholly representative. 2 In addition, the fact that only a few programmes have standard approaches to evaluation and reporting makes it difficult to carry out a systematic assessment of these evaluation reports. 3 As the implementation of the WHO evaluation policy progresses, the supporting systems will improve and The tool has a grading system across 15 performance indicators, and a score of between 0 and 0.50 points is ranked as “missing”; 0.51–1.25 points is graded as “approaches requirements” and 1.26–1.75 means that the entity “meet requirements” related to gender equality and women’s empowerment aspects. 2 3 1
There is an over-representation of evaluation reports on national malaria control programmes (11/28). Reports use different formats and present marked differences in the nature of their recommendations.
7
EB135/5
the data available will be more complete. This will result in a more comprehensive analysis of and report on the quality of evaluations and the implementation of their recommendations. 21. Despite the above-mentioned limitations, this assessment has identified areas and aspects that the Secretariat needs to address in the short term, in order to make evaluation useful for organizational change. These include strengthening of capacities of those commissioning evaluations and the staff involved in carrying them out, to ensure that all evaluations: (a) are in accordance with the evaluation policy and the WHO evaluation practice guidelines; (b) integrate the UN System-wide Action Plan scorecard tool to assess gender equality and women’s empowerment; and (c) address the impact and sustainability of programmes, projects or functions. 22. This assessment has also identified factors associated with the success or failure of programmes, projects or functions. As such, it provides guidance on what WHO needs to ensure when working in countries. These factors include ensuring political commitment through the establishment of good governance mechanisms and engagement of stakeholders, better planning and allocation of resources, and programme management. On the other hand, the assessment also indicates that the Secretariat should improve in two main areas that are an intrinsic component of the WHO reform agenda: financial planning and human resources management, and monitoring and following up the implementation of recommendations from evaluations.
ORGANIZATION-WIDE EVALUATION WORK PLAN FOR 2014–2015 23. Following the presentation of the proposed Organization-wide evaluation work plan for 2014–2015 to the Executive Board at its 134th session in January 2014 (document EB134/38), the Office of Internal Oversight Services has held further consultations with senior management in the WHO Secretariat. At present, the main challenge lies in funding the plan. This is partly due to the timing of the overall planning process for the current biennium, which has not allowed for thorough bottom-up consultation with WHO country offices and regions. Regional teams have reported a lack of guidance from global level about how evaluation should be integrated into the planning process. As a result, only three regions have conducted regional consultations with the network of planning officers, while the rest of the major offices have deferred the process until global guidelines become available. 24. Some units commissioning evaluations have informed the Office of Internal Oversight Services that although they initially plan for these evaluations to be carried out in the current biennium, they could be deferred to 2016 depending on the availability of funding. The Annex contains the Organization-wide evaluation work plan for 2014–2015.
ACTION BY THE EXECUTIVE BOARD 25. The Board is invited to note the report and approve the Organization-wide evaluation work plan for 2014–2015.
8
EB135/5
ANNEX ORGANIZATION-WIDE EVALUATION WORK PLAN FOR 2014–2015 Category in Twelfth General Programme of Work
Area of evaluation
Year
Justification
Comments
Comprehensive implementation plan on maternal, infant and young child nutrition FAO/WHO Project and Fund for Enhanced Participation in Codex (Codex Trust Fund) Normative function of WHO
2014
Requirement
As recommended by the World Health Assembly in resolution WHA65.6. Required under collaborative arrangements. End of project. Requested by the Consultative Group for the Trust Fund. Relates to several resolutions of the 1 World Health Assembly; consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management. Envisaged in WHO evaluation policy. As considered by the Executive Board at its 129th session. Regional commitment. Several countries will evaluate their national programmes.
2
2014
Requirement Significance Utility Utility
5
2014 WHO presence in countries Utilization of national professional officers at country level Implementation of WHO evaluation policy Impact of publications in WHO WHO normative work on civil registration and vital statistics systems in the South-East Asia Region WHO normative work in relation to the International Health Regulations (2005) 2015
6
2014
Utility
6
2014
Utility
6
2015 2015
Requirement Requirement
6 4
2015
Requirement Significance
4
Significance
High rank by significance. Linked to global commitments and to regional/ international initiatives, such as the Asia Pacific Strategy for Emerging Diseases, Integrated Disease Surveillance and Response at regional level, and the Global Outbreak Alert and Response Network. Requested by the World Health Assembly in resolution WHA62.15. High rank by significance. Several countries are evaluating part or all of their national programmes. Subject to consultation with senior management.
5
Multidrug-resistant and extensively drug-resistant tuberculosis
2015
Requirement Significance
1
Independent Monitoring Board of the Global Polio Eradication Initiative and its work
2015
Significance
6
See, for instance, resolutions WHA62.14 on reducing health inequities through action on the social determinants of health and WHA65.3 on strengthening noncommunicable disease policies to promote active ageing.
1
9
EB135/5
Area of evaluation
Year
Justification
Comments
Category in Twelfth General Programme of Work
The Transformative Agenda of the Inter-Agency Standing Committee for health clusters in each country experiencing a protracted emergency WHO normative work on malaria Leadership and management in WHO Partnerships and collaborative arrangements eHealth standardization and interoperability Mobility and rotation in the Western Pacific Region National capacity strengthening by WHO (training) Recruitment in WHO
Subject to consultation with senior management. 2015 Significance 6
2015
Significance Significance Utility Utility Requirement Utility Utility Utility
High rank by significance. Several countries are evaluating their national programmes. Evaluation of WHO reform, 3rd stage. Subject to consultation with senior management. Requested by the World Health Assembly in resolution WHA66.24. Subject to consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management.
1
2015 2015 2015 or 2016 2015 or 2016 2015 or 2016 2015 or 2016
6 4 6 6 6
=
=
=
10