EXECUTIVE BOARD 134th session Provisional agenda item 11.1
EB134/38 3 January 2014
Evaluation Update report and proposed workplan for 2014–2015 1. The Executive Board at its 131st session approved the WHO evaluation policy. 1 The policy requires the Office of Internal Oversight to report annually to the Executive Board through the Programme, Budget and Administration Committee on progress in the implementation of evaluation activities. The present document provides information on the progress made by the Secretariat in implementing the approved evaluation policy and contains a preliminary draft of the proposed workplan for 2014–2015.
PROGRESS AND ENABLERS 2. To foster a culture of evaluation and its active use across the Organization, WHO established the Global Network on Evaluation in April 2013. The Network is composed of representatives from the three levels of the Organization as well as from related global functions. 2 In accordance with its terms of reference, the Network functions through task forces to address identified needs. 3. In accordance with the agreed plan of work for 2013, the Network has: (a) drawn up the criteria for including items in the biennial Organization-wide evaluation workplan; (b) established a quality assurance system for evaluation products; (c) developed a comprehensive evaluation reporting framework; (d) disseminated and made use of evaluation findings and lessons learned; (e) developed an integrated evaluation registry; (f) strengthened evaluation capacity across WHO; (g) identified a communication strategy for evaluation; and (h) carried out the framework analysis related to evaluation in WHO. In addition, the Network has finalized a WHO evaluation practice handbook, which clarifies the roles and responsibilities for evaluation and documents the main procedures, tools and methods, in order to help staff conduct evaluations in accordance with the policy. 4. Finally, the Network is working to develop guidance on specific issues related to evaluation (such as tracking the cost of evaluations, resourcing implementation of the evaluation policy, defining evaluation of impact in the WHO context) and to compile a database of evaluation experts. Details of the progress made with regard to the Network’s “deliverables” and plans for 2014 are contained in Annex 1. 1 2
Decision EB131(1).
The Network has a total of 26 members: one country office representative per region (six); one regional office representative per region (six); one representative per cluster (six) and Director General’s Office (one) at headquarters; one representative from each of the departments addressing cross-cutting issues of relevance to the evaluation policy (seven): Country Collaboration; Gender, Equity and Human Rights; Internal Oversight Services; Knowledge Management and Sharing; Planning, Resource Coordination, and Performance Management; Communications; Information Technology and Telecommunications.
EB134/38
5. The Network is planning to hold a coordination meeting on 9–10 December 2013 to: (a) assess the effectiveness of current arrangements to instil an evaluation culture in WHO; (b) take stock of the achievements in 2013; and (c) with guidance from the various ongoing external evaluations and stakeholders, identify a plan of work for 2014 and the resources needed to further develop the function, practice and utilization of evaluation across the Organization.
WHO BIENNIAL EVALUATION WORKPLAN FOR 2014–2015 6. The evaluation policy calls for a systematic approach to drawing up a biennial Organization-wide evaluation workplan as part of the Organization’s planning and budgeting cycle. In this regard, the policy defines the role and responsibilities of the Executive Board as, among other things: providing oversight of the evaluation function within the Organization and, specifically in relation to the workplan, input on the items of specific interest to Member States; and approving the biennial WHO evaluation workplan. This evaluation workplan is to be submitted to the Board, through the Programme Budget and Administration Committee. 7. The approach adopted to draw up the evaluation workplan included: (a) consideration of selected criteria (see Annex 2) in relation to the planned outcomes as defined in the Programme Budget 2014–2015; (b) mapping of those evaluations to which WHO is formally committed (by its governing bodies, partnerships or collaborative arrangements); and (c) the results of a consultation process within the Network and with senior management across WHO on evaluation priorities. 8. The consultation process for drawing up the evaluation workplan consisted of three phases: (a) Follow-up of the evaluations planned at regional and country levels with senior management at these levels. This resulted in a list of programmatic evaluations that WHO will support (technically and financially) or commission. At global level, a similar systematic process has been followed to identify the evaluations planned for 2014 –2015. Most of these evaluations relate to existing commitments within collaborative arrangements or summative evaluations of programmes. (b) Discussions with the managers responsible for outcomes at the level of categories and subcategories on the desirability and feasibility of undertaking evaluations in 2014 –2015. (c) Consultation with senior management at WHO headquarters and in the regions to prioritize and consolidate proposed evaluations and ensure that cross-cutting issues are sufficiently covered. 9. The result of the prioritization and consultation processes is the proposed evaluation workplan for 2014–2015 (Annex 3). It comprises the priority evaluations that WHO will support at the corporate level, either by commissioning, managing or conducting the evaluation work, or else by ensuring quality assurance and technical support to the units responsible for these evaluations. 10. The underlying approach, criteria and tools for drawing up the evaluation workplan are the responsibility of a task force within the Global Network on Evaluation, which has endorsed the relevance of the model for guiding the prioritization process. However, it will take some time to collate and validate the data sets underpinning compilation of the workplan – mainly because a considerable proportion of data at the level of disaggregation required by the model will only become available progressively as operational planning advances. The model will therefore become
2
EB134/38
increasingly useful and will continue to be refined and adapted as needed. The final workplan will be presented to the Executive Board in May 2014.
Survey on perception of evaluation capacity and identification of needs 11. In order to set a baseline for its work, the Network undertook a global staff survey in July 2013 to assess the Organization’s effectiveness in institutionalizing the evaluation policy and the need for development of evaluation capacity across WHO. Despite its inherent limitations (based on self-assessment and a participation rate of 8.2% – 577 responses received), the survey has identified three main areas where better communication about evaluation across WHO is required. These areas will inform the future work of the Network: (a) effective transition of the evaluation policy into a corporate culture of evaluation, with commitment by senior management to champion evaluation and promote change. This area includes the need to ensure that programme managers clearly understand the added value of the different types of evaluation in WHO and systematically engage in evaluations; (b) efficient capture and integration of the evaluative work under way at the programme and country levels, which is not registered or appropriately valued at corporate level; and (c) establishment of an appropriate learning programme to address the strong interest of staff (92% of respondents) in enhancing their technical capacities in an area that is considered to be strategic in the context of the WHO reform.
Development of the registry platform and quality control of evaluation products 12. Assessment of the existing evaluation registry in July 2013 revealed major gaps in terms of its completeness and deficiencies when assessed against the revised quality criteria. For example, of the 226 products reported as evaluations by technical units, a formal report had been completed in only 31% of cases, while only 24% were available for consultation. Of the 54 available e-reports, 28 (52%) met the working definition of evaluation, while the remainder were classified as programme reviews (28%), technical consultations/ad hoc reports (9%), performance assessments (5%), operational research/evaluation research (4%) and global surveys (2%). Using the new standard quality checklist for evaluation reports, 75% of the reports were ranked as “good”. However, 88% of the reports lacked references as to how the evaluation addressed issues of equality and of gender, equity and human rights. 13. The results of this assessment confirmed the validity of the checklists used for quality control of evaluation products and served to finalize the WHO evaluation practice handbook and to pilot test the approach to quality assurance related to the new global evaluation registry platform. 14. The new platform supports integrating quality control of evaluations in the stages of programme planning, implementation and reporting. It also links evaluations to the biennial evaluation workplan and the Twelfth General Programme of Work 2014–2019, and it provides information on all WHO evaluations. This development should yield the improvements required to prepare evaluation reports for publication on the WHO website.
3
EB134/38
CHALLENGES AND ISSUES 15. Instilling an evaluation culture throughout WHO requires various challenges to be taken up. From a demand perspective, evaluation is concentrated in programmes that have a long tradition of evaluation, which are often externally funded, and where evaluation is a formal requirement of the collaboration agreement. One of the often cited challenges and a barrier to the Network is that the resources devoted to evaluation at the corporate level remain inadequately low. A similar situation with regard to the allocation of financial and human resources for evaluation in most regions is also putting at risk the momentum generated by adoption of the evaluation policy. 16. Many of the evaluations carried out at country level focus on national programmes and do not necessarily address the performance of WHO vis-à-vis these programmes. Nevertheless, these evaluations provide input to the role that WHO plays in building national capacities for evaluation. 17. As the evaluation policy is operationalized, there is a need to clarify further the organization, roles and responsibilities for evaluation at the country, regional and global levels, as well as the funding arrangements across WHO. Until 2013, these aspects have been dealt with on an ad hoc basis. Guidelines should be in place for “bottom-up” operational planning for the biennium 2016–2017, to ensure that evaluation is part of the workplan implementation cycle at those three levels. In addition, operating systems such as the Global Management System need to be adapted so that information relevant to evaluation (on the resources planned and utilized, for instance) can be collected systematically rather than through parallel mechanisms. 18. Throughout the Organization, there is a need to continue to foster the evaluation culture among staff and management. Recognizing the important role played by a critical mass of staff championing evaluation, the Network has been working on a communication strategy for evaluation. With the current initiatives and plans for 2014, and if resources are made available both for the Network and its task forces, as well as for funding the WHO-wide biennial evaluation workplan, it can be expected that the situation will continue to improve during 2014.
ACTION BY THE EXECUTIVE BOARD 19. The Board is invited to note the report and to comment on the proposed WHO biennial evaluation workplan for 2014–2015.
4
ANNEX 1 Global Network on Evaluation – Deliverables, progress and plans for 2014 Deliverable WHO-wide biennial evaluation workplan Progress so far Framework identified and prioritization model under development System established and analysis made of available reports on evaluation products Some components (quality checklists, plans to introduce peer review function, registry update system) in place Ad hoc reporting Plans for 2014 Further refine prioritization model and consultation process for revision of WHOwide annual evaluation workplan for 2015 Continue analysis as evaluation products become available Comments Timing of operational planning and information on resources available for evaluation not conducive to timely compilation of workplan; process to be fully integrated for 2016–2017 Train staff in areas needing improvement (reference to gender and human rights and to evaluation management committees, completeness of reporting and utilization of evaluations) Planned for implementation as part of WHO reform process
Quality control of evaluation products
Quality assurance of evaluation in WHO
Finalize quality assurance system for use in 2015
Comprehensive reporting framework Evaluation registry Learning programme
Develop framework for reporting on results of Planned for implementation in 2014, when other processes evaluation and on knowledge management are stabilized and utilization Further utilization of registry and enhancement as needed Increasing prospective use of registry system for planning of evaluations and to support quality assurance Preparation under way dependent on additional resources Corporate programme to be compulsory for selected categories of staff Distribution strategy identified Community of practice, webinars started Institutionalization in progress Links to other deliverables to boost evaluation culture across WHO Integration of policy, functions and practice of evaluation in WHO Ongoing discussions within framework of United Nations Evaluation Group
System developed and integrated with other corporate systems
Needs identified, e-learning tool in Pilot implementation and assessment prior to development, programme in full implementation Identification of learning opportunities in preparation. regions Completed Elements of strategy identified Start in December 2013 Drawn up Web page on evaluation in preparation Develop into e-learning tool Integrate and implement full strategy Full implementation and roll-out
WHO evaluation practice handbook Knowledge management Evaluation communication strategy
Framework analysis of Elements of framework identified evaluation function in WHO Database of evaluation experts 5 Not yet formalized
Assess progress against relevant milestones Include peer review of evaluation function System and platform to be established
EB134/38
EB134/38
ANNEX 2 LIST OF CRITERIA USED TO PRIORITIZE EVALUATIONS FOR INCLUSION IN THE WORKPLAN
Organizational requirement • Formal external commitment to evaluation owing to collaborative arrangements (with partners, partnerships, etc.)
Organizational significance • Leadership priorities as defined in the General Programme of Work 2014 –2019 • Global, regional or international programme commitments as reflected in resolutions of the World Health Assembly and regional committees (on the United Nations Millennium Development Goals, disease eradication or elimination, the International Health Regulations (2005), etc.) • Level of investment as reflected in the programme budgets 2014–1015 (planned) and 2012–2013 (expended) • Level of perceived risk (inherent relative complexity, reputational risk, other oversight and evaluation findings) • Historical performance: programmatic progress identified in routine monitoring and performance assessment (such as mid-term reviews and end-of-biennium assessments)
Organizational utility • Potential area for institutional learning (innovation, catalytic initiative) • Degree of comparative advantage (relation to WHO’s core functions, production of global public goods) • Policy-related issues (lessons learned, reorientation/revision of policies) • Other judgment factors (recommendations by the Global Network on Evaluation or management)
6
EB134/38
ANNEX 3 PROPOSED ORGANIZATION-WIDE EVALUATION WORKPLAN 2014–2015 Area of evaluation FAO/WHO Project and Fund for Participation in Codex (Codex Trust Fund) Comprehensive implementation plan on maternal, infant and young child nutrition Global strategy and plan of action on public health, innovation and intellectual property Monitoring of achievement of health-related Millennium Development Goals eHealth standardization and interoperability Impact of publications in WHO United Nations SystemWide Action Plan on gender equality and empowerment of women WHO normative work on civil registration and vital statistics systems in SouthEast Asia Region WHO normative work in relation to International Health Regulations (2005) Year 2014 Justification Requirement Comments End of project. Requested by Consultative Group for Trust Fund Relates to Health Assembly resolution WHA65.6. Required under collaborative arrangement. As considered by Executive Board at its 133rd session in May 2013. Progress report in 2015, full independent evaluation in 2017. Requested by Health Assembly in resolution WHA63.15.
2014
Requirement
2015
Requirement
2015–2016
Requirement
2015–2016 2014–2015 2015
Requirement Requirement Requirement
Requested by Health Assembly in resolution WHA66.24. As considered by Executive Board at its 129th session. Called for by United Nations General Assembly in resolution A/RES/67/226. Regional commitment. Several countries will evaluate their national programmes. Global commitment, regional/international initiatives such as Asia Pacific Strategy for Emerging Diseases, Integrated Disease Surveillance and Response at regional level, and Global Outbreak Alert and Response Network. Envisaged in WHO evaluation policy. Requested by Health Assembly in resolution WHA62.15. High significance. Several countries evaluating part or all of national programmes.
2015
Requirement Significance
2015
Requirement Significance
Implementation of WHO evaluation policy Multidrug-resistant and extensively drug-resistant tuberculosis
2015 2015
Requirement Requirement Significance
7
EB134/38
Annex 3
Area of evaluation WHO normative work on malaria Environmental health
Year 2015
Justification Significance
Comments High rank by significance. Several countries evaluating national programmes. High rank by significance. Several countries evaluating national programmes. Evaluation of WHO reform, 3rd stage. Relates to several resolutions of the World Health Assembly; 1 consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management. Subject to consultation with senior management.
2015
Significance
Leadership and management in WHO Normative function of WHO
2015 2014
Significance Utility
Recruitment in WHO WHO presence in countries Mobility and rotation in Western Pacific Region Utilization of national professional officers at country level Partnerships and collaborative arrangements National capacity strengthening by WHO (training)
2014 2014 2014 2015
Utility Utility Utility Utility
2015 2015
Utility Utility
=
=
=
For instance, resolution WHA62.14 on Reducing health inequities through action on the social determinants of health and resolution WHA65.3 on Strengthening noncommunicable disease policies to promote active ageing.
1
8