EXECUTIVE BOARD 130th session Provisional agenda item 5
EB130/5 Add.8 22 December 2011
WHO reform WHO evaluation policy
BACKGROUND 1. In 2002, WHO developed a framework on Programme Management, focusing on monitoring, evaluation and reporting. This guidance outlined evaluation in the context of the principles of resultsbased management and identified four types of evaluations in WHO’s agenda: (a) Programme budget performance evaluation (end-of-biennium performance assessment report); (b) (c) (d) Programmatic evaluation; Thematic evaluation; Country evaluation.
2. Evaluation guidelines were established in 2006 to describe the application of, and the responsibilities for, evaluation at WHO and, in particular, to guide the work of the Office of Internal Oversight Services. 3. In the last eight years, the Secretariat has increased the number of Organization-wide programmatic and thematic evaluations, and country evaluations. However, from a broader institutional perspective, it has been less successful in fostering an evaluation culture, developing evaluation capacity at all levels of the Organization, and in promulgating participatory approaches to evaluations. The causes for this include the institutional arrangements for the evaluation function (including a lack of a direct mechanism for oversight by the governing bodies) and the absence of an effective budgetary allocation mechanism to resource the evaluation function. Most of the evaluations within WHO are still commissioned by individual technical programmes. 4. The evaluation policy has been informed by best practices, WHO’s own experience, and the United Nations Evaluation Group.1
The Group’s work on defining norms and standards for the evaluation function was undertaken in response to United Nations General Assembly resolution 59/250 (2004).
1
EB130/5 Add.8
PURPOSE 5. The purpose of this policy is to foster the culture and use of evaluation across the Organization, to provide a consolidated institutional framework for evaluation at the three levels of WHO, and to facilitate conformity of evaluation at WHO1 with best practices and with the norms and standards for evaluation of the United Nations Evaluation Group. 6. The accountability framework of WHO includes several types of assessments. WHO considers that all are crucial to programme development and institutional learning. The current policy addresses only the assessments qualifying as “Evaluation” and excludes other forms of assessments conducted in WHO, such as monitoring, performance assessment, surveys, programme reviews and audit.
POLICY STATEMENT 7. Evaluation is an essential function at WHO, carried out at all levels of the Organization. It ensures accountability and oversight for performance and results, and reinforces organizational learning in order to inform policy for decision-makers and support individual learning.
EVALUATION DEFINITION 8. “An evaluation is an assessment, as systematic and impartial as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area institutional performance (…).”2 (a) It focuses on expected and achieved accomplishments, examining the results chain, processes, contextual factors and causality, in order to understand achievements or the lack thereof; (b) It aims at determining the relevance, effectiveness, efficiency and sustainability of the interventions and contributions of the Organization; (c) It provides evidence-based information that is credible, reliable and useful, enabling the timely incorporation of findings, recommendations and lessons learnt into the decision-making processes of the Organization; (d) It is an integral part of each stage of the programming cycle and not only an end-ofprogramme activity.
1
This policy replaces the previous guidance and the WHO Evaluation Guidelines (WHO/IOS/06.2, July 2006).
2 As defined in the United Nations Evaluation Group document entitled “Norms for evaluation in the UN System” (document UNEG/FN/Norms(2005), http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=21 (accessed 14 December 2011).
2
EB130/5 Add.8
PRINCIPLES AND NORMS1 9. This policy provides a framework to ensure the systematic application of the key principles to the evaluation function in WHO. The key principles set out below are interrelated and underpin the approach to evaluation in WHO.
Impartiality 10. Impartiality is the absence of bias in due process; it requires methodological rigour, and the objective consideration and presentation of achievements and challenges. Impartiality contributes to the credibility of evaluation and reduces bias in the data gathering, analysis, formulation of findings, conclusions and recommendations. 11. All evaluations shall be conducted in an impartial manner at all stages of the evaluation process. An evaluation management group will be established for each evaluation to ensure oversight of the evaluation process.
Independence 12. Independence is the freedom from the control, or undue influence, of others. Independence provides legitimacy to evaluation and reduces the potential for conflicts of interest that could arise if policy-makers and managers were solely responsible for the evaluation of their own activities. 13. Independence must be ensured at organizational, functional, and behavioural levels. At the organizational level, the evaluation function must be separated from those responsible for the design and implementation of the programmes and operations being evaluated. At the functional level, there must be mechanisms that ensure independence in the planning, funding, and reporting of evaluations. At the behavioural level, there must be a code of conduct that is ethics-based. This code of conduct will seek to prevent or appropriately manage conflicts of interest. 14. Evaluators shall not be directly responsible for the policy, design, or overall management of the subject under review. WHO staff performing evaluations shall abide by the ethical principles and conduct of staff outlined in the Compilation of WHO’s policies and practices. External contractors shall abide by the WHO requirements for external contractual agreements. Evaluators must maintain the highest standards of professional and personal integrity during the entire evaluation process. They are expected to ensure that evaluations address gender and equity; and be sensitive to contextual factors, such as the beliefs, manners and customs of the social and cultural environments evaluated. 15. The whistleblower policy and other relevant policies will protect staff participating in evaluations from retaliation or repercussions.
See United Nations Evaluation Group document entitled “Norms for evaluation in the UN System” (document .UNEG/FN/Norms(2005), http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=21 (accessed 14 December 2011), and OECD/Development Assistance Committee. Principles for evaluation of development assistance. Paris, Organization for Economic Co-operation and Development, 1991, http://www.oecd.org/dataoecd/31/12/2755284.pdf (accessed 14 December 2011).
1
3
EB130/5 Add.8
Utility 16. Utility relates to the impact of the evaluation on decision-making and requires that evaluation findings be relevant and useful, presented in a clear and concise way, and monitored for implementation. The utility of an evaluation depends on its timeliness, relevance to the needs of the programme and stakeholders, the credibility of the process and products, and the accessibility of reports. 17. Utility will be ensured through the systematic prioritizing of the evaluation agenda based on established criteria and consultation with relevant stakeholders; the systematic follow-up of recommendations; public access to the evaluation products; and alignment with the results-based management framework.
Quality 18. Quality relates to the appropriate and accurate use of evaluation criteria, impartial presentation and analysis of evidence, and coherence between findings, conclusions and recommendations. 19. Quality will be ensured through (a) the continuous adherence to WHO evaluation methodology, applicable guidelines and the norms and standards for evaluation of the United Nations Evaluation Group; (b) oversight by the evaluation management group; and (c) peer-review of the evaluation report when justified. Other mechanisms such as periodic meta-evaluations will also be considered.
Transparency 20. To achieve transparency, stakeholders should be aware of the reason for the evaluation, the selection criteria, and the purposes for which the findings will be used. Transparency of process is also important, as is the accessibility of evaluation materials and products. 21. Transparency will be ensured through the following approaches. The commissioner of the evaluation will ensure a continuous consultation process with relevant stakeholders at all stages of the evaluation process. The evaluation report shall contain details of evaluation methodologies, approaches, sources of information and costs incurred. In accordance with the WHO disclosure policy, evaluation plans, reports, management responses and follow-up reports will be made public on the WHO’s evaluation web site.
TYPES OF EVALUATIONS 22. The WHO Secretariat commissions the following main types of evaluations. (a) Thematic evaluations focus on selected topics, such as a new way of working, a crosscutting theme or core function; or they address an emerging issue of corporate institutional interest. Thematic evaluations provide insight into relevance, effectiveness, sustainability and broader applicability. They require an in-depth analysis of a topic and cut across organizational structures. The scope of these evaluations may range from the entire Organization to a single WHO office.
4
EB130/5 Add.8
(b) Programmatic evaluations focus on a specific programme. This type of evaluation provides an in-depth understanding of how and why results and outcomes have been achieved over several years and examines their relevance, effectiveness, sustainability, and efficiency. Programmatic evaluations address achievements in relation to WHO’s results chain, and require a systematic analysis of the programme under review. The scope of programmatic evaluations may range from a country to interregional or global levels. (c) Office-specific evaluations are systematic assessments of the work of the Organization in and with a country, region or at headquarters in respect of WHO’s objectives and commitments. 23. The Executive Board may, at its discretion, also commission an evaluation of any aspects of WHO, either internally or externally.
EXTERNAL EVALUATIONS 24. Evaluations may be commissioned by the governing bodies to be conducted by external evaluators independent from the Secretariat. Other stakeholders, such as Member States, donors or partners, may also commission external evaluations of the work of WHO for the purpose of assessing performance and accountability or prior to placing reliance on the work of the Organization. 25. WHO encourages participation in joint evaluations and the Secretariat will fully cooperate in such external evaluations through a process of disclosure of appropriate information and facilitation. The results of such external evaluations, when made available, will be disclosed on the WHO evaluation web site.
PLANNING AND PRIORITIZATION OF EVALUATIONS 26. WHO will develop an annual, Organization-wide evaluation workplan as part of the Organization’s planning and budgeting cycle. 27. The workplan shall be established in consultation with senior management and the global working group on evaluation based on established criteria. The workplan shall be submitted to the Executive Board for approval.
EVALUATION METHODOLOGY 28. The following are the main components of an evaluation process.1
A WHO evaluation practice handbook will be developed to provide further guidance on the evaluation methodology.
1
5
EB130/5 Add.8
Design 29. Terms of reference for an evaluation shall include detailed information on the following elements: (a) (b) (c) Context of the evaluation; Purpose and objectives of the evaluation; Scope (outlining what is, and what is not, covered by the evaluation);
(d) Evaluation criteria (inter alia, relevance, efficiency, effectiveness and sustainability) and key evaluation questions; (e) Users (owner and audience) of the evaluation results;
(f) Methodology (approach for data collection and analysis, and involvement of stakeholders); (g) (h) (i) (j) Evaluation team (size, knowledge, skills and qualifications); A detailed workplan including a timetable, organization and budget; Deliverables (including report, distribution strategy and follow-up); Evaluation management group (including technical staff requirements).
Evaluation management group 30. When warranted by the size and complexity of the evaluation, a reference group shall be assembled to assist in the conduct of the evaluation. The functions of this group include reviewing, and commenting on, the terms of reference and the draft report. The group shall be kept informed of progress by the evaluation team leader and should be available to respond to queries from the evaluation team. As the evaluation process progresses, the group may refer additional ideas and suggestions to the evaluation team for consideration.
Team selection 31. The following should be considered in the selection of the evaluation team members: (a) technical and sectoral expertise;
(b) in-depth understanding and experience of quantitative and qualitative evaluation methodology; (c) (d) previous experience of conducting reviews and evaluations; demonstrated analytical and writing skills.
6
EB130/5 Add.8
32. The evaluation team leader shall be responsible for interactions among the evaluation team members and have overall responsibility for the evaluation outputs. 33. The team selection process must ensure that no member of the evaluation team has a conflict of interest. 34. A written report is an essential requirement of the evaluation process. The final evaluation report shall be logically structured and contain evidence-based findings, conclusions, lessons learnt and recommendations. 35. The report must: (a) (b) (c) (d) (e) include only information relevant to the overall purpose and objectives of the evaluation; describe the purpose of the evaluation and attach the terms of reference; answer the key questions detailed in the terms of reference; describe the methodology used to collect and analyse the information; indicate any limitations of the evaluation; and
(f) include the evidence on which the conclusions, lessons learnt, and recommendations are based.
FINANCING OF EVALUATION1 36. The Director-General shall ensure that there are adequate resources to implement the annual evaluation workplan. 37. Regional Directors, Assistant Directors-General, Directors and Heads of WHO Country Offices must ensure that resources are adequate to implement their respective components of the Organizationwide evaluation plan. An appropriate evaluation budget must be an integral part of the operational workplan of a programme, and shall be discussed as necessary with stakeholders during the planning phase of each project/programme/initiative. 38. In estimating the amount required to finance evaluation in WHO, estimations provided by other organizations have been considered. According to these, the overall programme budget might contain, as an integral part, a figure for evaluation that is equivalent to between 3% and 5% of that budget.
1
Further work will deal with the sources of funding of the evaluations.
7
EB130/5 Add.8
ACCOUNTABILITY AND OVERSIGHT 39. The accountability framework defines from whom, and to whom, authority flows and for what purpose. It further defines the accountability of those with authority and their responsibility in exercising that authority. This section defines the roles and responsibilities for the main actors in the evaluation process as well as the monitoring mechanism used to implement the evaluation policy.
Roles and responsibilities 40. The Executive Board of WHO1 shall: (a) (b) (c) determine the evaluation policy and subsequent amendments, as needed; provide oversight of the evaluation function within the Organization; encourage the performance of evaluations as an input to planning and decision-making;
(d) provide input to the annual evaluation workplan on the items of specific interest to Member States; (e) approve the annual Organization-wide evaluation workplan;
(f) consider and take note of the annual report of the implementation of the annual Organization-wide evaluation workplan; (g) periodically revise the evaluation policy, as necessary.
41. The global working group on evaluation is a working group of staff acting to promote the culture of evaluation within their area, facilitate the sharing of information and knowledge management in respect of evaluation at WHO by: (a) (b) (c) (d) participating in the preparation of the annual Organization-wide evaluation workplan; submitting relevant evaluation reports to the evaluation inventory; tracking information on management responses to evaluation recommendations; advising programmes across WHO on evaluation issues, as needed.
42. The Office of Internal Oversight Services is the custodian of the evaluation function and reports directly to the Director-General, and annually in a report for consideration by the Executive Board, on matters relating to evaluation at WHO. The Office of Internal Oversight Services is responsible for the following functions related to evaluation: (a) leading the development of an annual Organization-wide evaluation workplan through internal or external evaluations;
1
WHO Executive Board and its subsidiary organ the Programme, Budget and Administration Committee.
8
EB130/5 Add.8
(b) informing senior management on evaluation-related issues of Organization-wide importance; (c) facilitating the input of evaluation findings and lessons learnt for programme design across WHO; (d) (e) (f) (g) convening the global working group on evaluation; maintaining a system to track management responses to evaluations; maintaining an on-line inventory of evaluations performed across WHO; maintaining a roster of experts with evaluation experience;
(h) providing good-quality guidance material for the preparation, conduct and follow-up of decentralized evaluations; (i) ensuring that evaluation reports conform to the requirements of the policy;
(j) strengthening capacities in evaluation among WHO staff (for example, making available standardized methodologies or training on evaluation); (k) submitting an annual report on evaluation activities to the Executive Board through the Director-General; (l) supporting the periodic review and updates to the policy as needed.
USE OF EVALUATION FINDINGS Utilization and follow-up of recommendations 43. Recommendations contained in the evaluation report reflect the value added by the evaluation process. Each evaluation shall have an identified owner, such as the responsible officer of a cluster, programme, office or project. It is the responsibility of the owner to utilize the findings of the evaluation and develop an action plan for implementing the recommendations. 44. The evaluation owner shall ensure that an appropriate management response is issued in a timely manner to the appropriate Assistant Director-General at headquarters, or to the Regional Director in the regions and countries. 45. The Office of Internal Oversight Services shall monitor the follow-up of the implementation of evaluation recommendations in a systematic manner, coordinating efforts with the global working group on evaluation. The Office of Internal Oversight Services shall issue periodic status reports on progress in the implementation of the recommendations to senior management and report annually to the Executive Board.
9
EB130/5 Add.8
Disclosure and dissemination of evaluation reports 46. WHO shall make evaluation reports available in accordance with the Organization’s disclosure policy. 47. Lessons learnt from evaluations shall be distilled, reported and disseminated as appropriate.
ACTION BY THE EXECUTIVE BOARD 48. The Board is invited to note this draft policy and consider it for approval.
=
=
=
10