Practical guide to evaluation for programme managers and evaluation staff

0 Practical guide to evaluation for programme managers and evaluation staff WHO/DGO/EVL/2023.3 © World Health Organization 2023 All rights reserved. This health information product is intended for a restricted audience only. It may not be reviewed, abstracted, quoted, reproduced, transmitted, distributed, translated or adapted, in part or in whole, in any form or by any means. The designations employed and the presentation of the material in this health information product do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. The World Health Organization does not warrant that the information contained in this health information product is complete and correct and shall not be liable for any damages incurred as a result of its use. This guide is an independent publication by the WHO Evaluation Office. This publication does not necessarily reflect the views of the World Health Organization. The text has not been edited to official publication standards, and WHO accepts no responsibility for any errors. iii CONTENTS 1 • Introduction 2 • Planning 3 • Preparation 4 • Designing 5 • Data collection & analysis 6 • Validation & finalization 7 • Dissemination & follow-up 8 • Annexes Planning Data collectio and anal is Validation and finalizati Dissemination and follo p Ann xes Introduction Preparation Designing iv Note on terms used in this publication To enhance readability, the following terms are used in this guide to represent broader groups: The programme is the subject of evaluation. It could actually be a project, a component of a programme, or a set of activities to achieve defined goals such as a campaign, a humanitarian action, a workstream, or a combination of projects and activities. The evaluation commissioner is normally the manager of the programme to be evaluated, who has the responsibility to ensure the evaluation will take place and has budget authority over the programme. It could be a person delegated with this responsibility by the commissioner. The evaluation manager is the person designated by the evaluation commissioner or the programme manager to oversee and manage the entire evaluation process. The term also covers evaluation management groups formed for joint evaluations. 1 The Practical Guide to Evaluation for Programme Managers and Evaluation Staff aims to support managers and staff who are not professional evaluators to plan and commission evaluations. It also aims to share a common approach to evaluation within the organization at all levels that reflects standards applied broadly by the professional evaluation community. The Guide provides practical guidance to plan and commission evaluations under the framework of the WHO Evaluation Policy (2018), updating the previous guidance in the WHO Evaluation Practice Handbook (2013). Efforts were taken to make the Guide as practical, concise and user-friendly as possible, while not diverging from the previous guidance in substance. Therefore, people who have been following the Evaluation Practice Handbook could continue to do so. At the same time, the Guide adds some new perspectives that are now considered good practice. It incorporates some practical ways to further enhance stakeholder engagement and their buy-in, and it covers recent thinking on how to deal with complex or “fuzzy” subjects, such as the influence of WHO normative work or contributions to high-level policy goals. For readability and user-friendliness, the Guide intentionally avoids elaboration of typologies, theories and other elements that are not essential in planning and commissioning evaluations. For readers interested in more discussion and information on these subjects, useful references are provided in Annex 4. Section 2 explains how to plan evaluations for the entire programme cycle (such as a biennium), normally done at the outset of the cycle by programme managers and evaluation staff. Thereafter, from Section 3 and onwards, step-by-step guidance is provided on how to commission each individual evaluation. 1 Introduction 2 2.1 WHY DO WE EVALUATE? In planning for evaluations, what is most important is to think through why each proposed evaluation should take place. For example, it could be an input to a major strategic review with partners; to fulfil the needs of funding partners; for programme managers to know the results achieved in the field in order to inform the design of future plans; or for accountability to the supported population. This consideration affects the choice of evaluations, their timing and eventually their design. Evaluations aim to examine results – changes led or instigated by the programme’s activities and products. Evaluations ask “What has happened as a result of your work?” This distinguishes evaluations from audits or management reviews (see Box 1). For credibility, evaluations should be conducted by an independent and professional team of evaluators. There are certain protocols to follow to ensure the rigour, credibility and utility of evaluations. Therefore, evaluations are generally more costly and time-consuming than ad hoc management or technical reviews. Before deciding to plan for an evaluation, it is important to think about whether an evaluation is the best means to address the issue in question. Evaluations should operate on the guiding principles of intersectionality, inclusion, participation and ethics to document how multiple and intersecting factors of vulnerability (gender, age, ethnicity, disability and other factors of discrimination) interplay, and ensure vulnerable groups can have a say in the evaluation design, data collection and interpretation of results.1 An evaluation collects information on the results achieved and provides an evidence-based analysis on how the programme has been performing, what is working and what is not working. Therefore, it could be a useful exercise when the programme manager wishes to know how the programme has been affecting the behaviour of counterparts and beneficiaries. On the other hand, a study for which the information is available internally could be conducted by hiring a management consultant. An evaluation 1 Guidance note on integrating health equity, gender equality, disability inclusion and human rights in WHO evaluations (2023) Geneva: World Health Organization; 2023. 2 Planning This section outlines the following main tasks of the planning phase: • Assessing why the evaluation should take place. • Preparing an evaluation plan. • Estimating the budget. • Planning for joint evaluations. 3 could also serve as an accountability tool for external stakeholders such as a funding partner or the programme counterparts. If the objective is purely to learn lessons, an internal exercise may be sufficient. Box 1. Evaluation, audit and management reviews Evaluation and audit are both functions for accountability and improvement, conducted independently of the management for their credibility. An evaluation typically looks at whether the organization is doing the right things in the right way to achieve the results intended. An audit looks at whether the organization is doing things as intended in the right way, minimizing the risks of mistakes and wrongdoing. Other public service entities have tried to explain the differences between evaluation and audit, as the examples below demonstrate: Internal auditing … is a professional, independent and objective appraisal function that uses a disciplined, evidence-based approach to assess and improve the effectiveness of risk management, control and governance processes. Evaluation is the systematic and neutral collection and analysis of evidence to judge merit, worth or value... Evaluations typically focus on programs, policies and priorities and examine questions related to relevance, effectiveness and efficiency... Evaluations generally employ social science research methods. – Government of Canada, 2017 Auditing grew out of the accounting discipline, and evaluation grew out of the social sciences. An auditor looks for particular instances of things going wrong. [They] place great weight on the need for accuracy of administrative records, and [are] likely to be suspicious of statistical inferences based on aggregate data if [they] cannot find actual cases confirming the general conclusion. The social science evaluator mistrusts administrative records and is predisposed to seek [their] own direct observations and a much more generalized view of what happened. [They] prefer large volumes of data and mistrust small numbers of specific events because [they have] little confidence that what was found in one case is typical. – United States Government Accountability Office, 1980 A management review is a more generic term referring to any study conducted or commissioned by the management to address questions that it wishes to be answered. It therefore does not require independent investigation and the strict protocols that characterize internal audit and evaluation. 4 2.1 EVALUATION PLAN At the outset of a programme cycle, such as a biennium, programme managers should prepare an evaluation plan for their area of responsibility. An evaluation plan lists what subjects will be evaluated. For each evaluation, the plan should indicate: • when it will be conducted; • who will be responsible for it; • how much budget should be allocated for the evaluation; • what funding source is envisaged. It is good practice to note the purpose of the evaluation in the plan – that is, why this evaluation is planned. The evaluation plans should be consolidated at regional or departmental levels. Smaller programmes that do not envisage many evaluations could skip preparing the plans separately and directly integrate them into the regional or departmental plan. These regional or departmental plans will be further consolidated into the biennial Organization-wide evaluation workplan, prepared by the Evaluation Office and submitted for the approval of the Executive Board at the beginning of each biennium. It is possible to revise the evaluation plans as the situation or needs evolve. The relevant stakeholders should be consulted if the change of plan would affect them. In deciding on the timing of evaluations, one should take into account that the evaluation process generally takes six to eight months. 2.2 ESTIMATING THE BUDGET The cost of evaluations varies but generally in the range US$ 60 000–600 000. The factors influencing the budget include: • the size of the programme to be evaluated; • the desirable size and composition of the evaluation team, considering the needs for different expertise; • the cost of international travel if the evaluators are expected to be recruited from outside the country, and the cost of domestic travel, which could be higher in a challenging environment; • the need to conduct field research; • the availability of secondary data, such as monitoring data and project reports; 5 • the extent to which the evaluation management support, such as arranging interviews and field trips by the evaluation team, is outsourced as a package to a contracted consultancy firm. The elements to be estimated to arrive at the final budget figure include: • consultant fees; • travel (international and domestic), and per diem if applicable; • operational costs, such as for communication and rental of premises and equipment for stakeholder workshops and meetings; • evaluation management costs, whether in staff time or financial terms, which generally cover two to four working months spread over the evaluation period. For the funding source, the principle is to fund the evaluation from the source fund of the programme, and for the evaluation to be budgeted as a programme activity. For example, if it is a project funded by a voluntary contribution, it should be included in the project budget as an activity. 2.3 JOINT EVALUATIONS A joint evaluation is an evaluation owned and managed jointly by multiple entities, including external partners. When the evaluated programme is a joint programme, it is recommended to conduct evaluations jointly as well. Joint evaluations are also an option for thematic evaluations that address high-level policy objectives shared among multiple partners. They could also be considered for programmes where beneficiaries cannot distinguish actors that support them (e.g. humanitarian actions, where beneficiaries only know the local distributors who work for multiple agencies playing different roles). A joint evaluation is normally managed by a joint evaluation management group (EMG). The role of the EMG is basically the same as that of an evaluation manager. When many entities participate in a joint evaluation for practical reasons, the number of EMG members could be limited to representatives of a few of these entities (e.g. three or four people), while representatives of the remaining entities could participate in the reference group that reviews key products. Although there are advantages of conducting evaluations jointly, joint evaluations can be very time- consuming because of the need to coordinate on various aspects and have the products reviewed and accepted by all partners. In some cases, therefore, the partners decide among themselves to designate one of them to manage the evaluation instead of forming an EMG, or to take over all the administrative aspects (e.g. recruitment, travel management, publishing). 6 3.1 APPOINTING THE EVALUATION MANAGER In many offices, an evaluation manager is already designated to implement the evaluation plan. If not, the programme manager or the evaluation commissioner should appoint an evaluation manager to start the process. 3.2 LAUNCHING THE EVALUATION AND SETTING UP THE MANAGEMENT STRUCTURE The evaluation manager launches the evaluation, informing relevant stakeholders about the evaluation, and sets up its management structure. If it is a joint evaluation, the evaluation manager should consult other partners to form a joint evaluation management group or have an alternative arrangement (e.g. one of the partners will act as the sole evaluation manager on behalf of all partners). The evaluation manger should consider establishing a reference group. The reference group comprises key stakeholders who are familiar with or have an interest in the programme. The reference group could include those responsible for implementing the programme if it is not expected to compromise the independence of the evaluation. Its role is to advise the evaluation manager at key junctures of the evaluation, reviewing and discussing the draft terms of reference, the inception report and the final draft report. The evaluation manager could also consult the reference group on other important issues, such as the overall approach to the evaluation, the selection of the evaluation team, the selection of case studies, or the organization of the stakeholder workshop. At least one person in the reference group should have knowledge or experience of health equity, gender equality, disability and human rights. The evaluation manager could also set up an advisory group if considered useful. The advisory group comprises evaluation or subject matter experts who do not have a direct stake in the programme. The advisory group could provide technical advice to the evaluation manager or the evaluation team when needed and requested. Preparation This section outlines the following main tasks of the preparatory phase: • Appointing the evaluation manager. • Launching the evaluation and setting up its management structure. • Preparing the terms of reference. • Recruiting the evaluation team. • Preparing the information package for the evaluation team. • Assessing the evaluability. 7 The evaluation manager may request the Evaluation Office to assign a quality assurance advisor to the evaluation, who assists the evaluation manager on technical aspects of the evaluation and helps ensure and validate the quality of evaluation products. The quality assurance advisor is an experienced evaluation manager (normally an external consultant) who conducts reviews of key products such as the terms of reference, the inception report, and the first and final draft reports. The evaluation manager may also seek ad hoc advice from the quality assurance advisor throughout the evaluation process. The figure below presents a typical management structure. Figure 1. Management structure 3.3 PREPARING THE TERMS OF REFERENCE The terms of reference are the basis of the evaluation exercise. As terms of reference are attached to the contract, they should be simple and provide clear instructions on what the evaluation team is obliged to do and produce. Additional information that is useful for the evaluation team, such as the detailed explanation of the context or the programme, could be provided in a separate document or in an annex to the terms of reference. Box 2. shows a sample outline of the terms of reference. The evaluation terms of reference normally contain the following elements: 8 • Background and purpose of the evaluation – this explains why and for what purpose the evaluation needs to be conducted, and the main questions expected to be answered by the evaluation. • Scope of the evaluation – this defines precisely what the evaluation will and will not cover, including temporal and geographical considerations, with some necessary information on the context and the programme to explain the scope. • Objectives/evaluation questions – this is a core part of the terms of references that explains what the Evaluation Team aims to do through its research. The evaluation questions should be composed in such a way as to allow the team to draw broad conclusions that would answer the main questions posed at the outset. The questions could be categorized by the standard evaluation criteria (relevance, impact, effectiveness, efficiency, sustainability, coherence), but it is not obligatory to cover all the standard evaluation criteria. For evaluations with a broader scope, questions can be divided further into sub-questions. Questions examining the aspects of health equity, gender equality, human rights and disability should be included.2 See Annex 2 for more explanation on formulating evaluation questions. • Approach/methodology – this provides an indication of the approach or methodology to be adopted. For example, it may suggest the use of quantitative analyses or outcome harvesting to answer certain questions. In large-scale evaluations, it may suggest a case study approach with an indication of how the cases should be selected, or suggest dividing the evaluation into components such as assessments of normative versus operational work or global versus country work. • Evaluation team – this defines the desired profile of the evaluation team. It must include a team leader with a substantial evaluation experience and ideally a good understanding of the subject matter. The team leader should be supported by team members with profiles that are complementary to each other, such as subject matter experts. For large-scale evaluations, the team may benefit from junior team members who can support information-gathering, data analysis and organization of activities. At this stage, it is acceptable to indicate only the areas of expertise required but not how many members the team should have, or how these areas of expertise are shared among team members, to retain some flexibility in composing the team because some members may cover multiple areas of expertise. The evaluation team should have at least one member with knowledge or experience of health equity, gender equality, disability and human rights. When this is not feasible, the evaluation team should have access to such expertise and be able to receive appropriate advice when needed. The evaluation manager should facilitate such access. • Management arrangement – this provides the management structure of the evaluation and identifies the evaluation commissioner, the evaluation manager, the quality assurance 2 Guidance note on integrating health equity, gender equality, disability inclusion and human rights in WHO evaluations (2023) Geneva: World Health Organization; 2023. 9 advisor (if designated), the reference group, the advisory group (if desirable) and the external evaluation team, which reports to the evaluation manager while remaining independent on the contents of evaluation. • Deliverables – the deliverables to be produced by the evaluation team should include the inception report, which describes the methodology and how the evaluation will be conducted, and the final evaluation report. The deliverables may also include intermediary draft reports (e.g. the first draft for internal review, the second draft for circulation to stakeholders for comments, the final draft for presentation at a stakeholder workshop), presentation materials (e.g. PowerPoint presentation, one-page summary for wider dissemination, articles for online platforms) and in-person presentations at stakeholder workshops and briefings. • Timeframe and budget/payment schedule – this indicates the timing of the launch of the evaluation, the expected delivery dates of the inception report, the draft and final reports, and the budget and payment schedule (e.g. 20% at submission of the inception report, 20% on first draft, 40% on final draft, and 20% on completion of all tasks). • It is recommended to provide in the terms or reference or as an annex instructions on the structure of the inception and evaluation reports (see Boxes 3 and 4 for samples) and the writing style (see Annex 3). According to the evaluation policy and other recent guidance from the Organization, all evaluations must give due consideration to the normative aspects, including gender equality, respect for human rights, disability inclusion, environmental sustainability, and overall the Leave No One Behind principle. The general approach would be to include evaluation questions relating to these normative aspects, with findings, conclusions and recommendations that respond to the questions. If the programme has elements that need to include such aspects, there could be more specific and tailored evaluation questions on these aspects. The terms of reference should be reviewed by the evaluation commissioner, the reference group and the quality assurance advisor, if available. It is good practice for key stakeholders who are not in the reference group but who will be engaged by the evaluation team to review the terms of reference too. 10 Box 2. Sample terms of reference outline [Evaluation title] Terms of reference 1. Background 1.1. Background (e.g. mandate, context). 1.2. Purpose: - Indicate what needs to be answered to achieve the purpose. 1.3. Programme: - Some explanation of the programme to give an idea of what is to be evaluated. - References for further details. 2. Scope - Programme and period to be evaluated. - Indicate if there is any limitation in the scope (e.g. geographical locations, programme components). 3. Objectives and evaluation questions - Research objectives and evaluation questions. - See annex on evaluation questions. 4. Approach and methodology - General approach on how to conduct the evaluation – for example, mixed methods (both qualitative and quantitative data and analyses), case studies, household surveys, field studies. - Any particular methodology linked to evaluation questions. - Indicate whether any part of the research could be further outsourced (e.g. analysis of website usage). 5. Evaluation team 5.1. Team leader. 5.2. Team - Size of the team. - Expertise collectively required or desirable in the team. - Could be composed of a core team and separate national teams for country case studies. 6. Management arrangement 6.1. Evaluation commissioner/owner, who funds and clears the evaluation. 6.2. Evaluation manager, who supervises and supports the evaluation team. 6.3. Reference group, which reviews the contents of key products (e.g. terms of reference, inception report, draft reports). 6.4. Advisory group (optional), composed of experts who do not have a direct interest in the evaluation – could be evaluation or subject matter experts. 7. Deliverables 7.1. Inception report. 7.2. Draft reports. 7.3. Information materials. - Presentations for stakeholder workshop, briefings. - Briefing notes, summaries. 8. Timeframe - Give sufficient time to develop a good design (desk study, inception interviews). - Ensure sufficient time is allocated for the review process and revisions (inception report, first and second drafts), especially with external stakeholder reviews (e.g. three weeks). 9. Budget and payment schedule - Overall budget for evaluation team. - Payment schedule for intermediary products. 11 3.4 RECRUITING THE EVALUATION TEAM Once the terms of reference are prepared, the evaluation team must be recruited. Most evaluations are conducted by a team composed of two to five members. With a large evaluation for which country case studies are planned, the team could be composed of a core global or regional team and national teams, the latter supporting case studies in countries. There are also cases where a part of the evaluation activities is outsourced to a specialized firm, for instance to conduct an analysis of data usage. Generally, there are two ways to get the evaluation team on board – to recruit consultants individually, or to procure the services of a specialized firm. The former is more time-consuming for the evaluation manager, as the evaluation team members need to be recruited one by one, and the team usually requires more administrative, logistical and editorial support throughout the evaluation process. When the service of a consultant firm is procured, the firm may take over a large part of such support work. This means the costs are normally higher when using a firm than recruiting a team individually. If the evaluation team is composed of consultants recruited individually, one should first aim to recruit the team leader. The evaluation manager may then enlist the help of the team leader in composing the rest of the team. This allows the team leader to compose a team that suits their intended evaluation design and with whom they are comfortable working. To facilitate the identification of qualified evaluators by the evaluation managers, the Evaluation Office maintains a roster of prequalified evaluation experts that is updated regularly with a view to improving coverage of regions and sectors. The roster can be made available to evaluation managers wishing to recruit individual evaluators or to form a small team of evaluators for a particular evaluation. The Evaluation Office has identified qualified firms or institutions to be contracted to undertake evaluations. Some of them will enter into long-term agreements with the Organization to undertake evaluations.3 Evaluation managers are also invited to put forward names of individual consultants, firms and institutions that have done good-quality evaluation work for them and could be recommended for the roster for future assignments. When the evaluation team is formed, each team member must sign a declaration of Interest for WHO experts, a confidentiality undertaking, and a pledge of ethical conduct in evaluation. All three forms, and the entirety of the Ethical Guidelines for Evaluation (UNEG), are available from the Evaluation Office. 3.5 PREPARING THE INFORMATION PACKAGE Before the evaluation team starts its work, the evaluation manager should prepare a package of information and documents that would be necessary or useful for the team in the design and conduct of the evaluation. This includes information and documents pertaining to: 3 As of February 2023, the process is still under way to establish such long-term agreements. 12 • the programme, the project or the activities to be evaluated, such as the programme or project document, activity records, products such as publications and training materials, monitoring data and performance reports; • the background and the context, which provides the background against and the context in which the programme or project was conceived, designed and possibly modified; • the stakeholder mapping, which lists the programme or project stakeholders, their role in designing or implementing the programme or project activities, and their contact information to be used in arranging interviews and workshops; • past evaluations and reviews if available. 3.6 ASSESSING THE EVALUABILITY The evaluability assessment is a results-based management tool applied at the programme design stage and at the early implementation stage to ensure there is a clear programme logic, robust monitoring framework, strategy against changes that influence programme implementation, robust resource planning, and so on. It could also be conducted quickly at the preparation stage of an evaluation, especially when such an assessment has not been conducted earlier. When evaluability is assessed at the outset of an evaluation, the points to check include: • whether the programme has a clear programme logic or theory of change – if not, the evaluation team may need to clarify it with the programme team in designing the evaluation; • whether the programme has had credible and relevant monitoring data, including on gender, disability and any other disaggregated data relevant to the context – if not, the evaluation team may need to collect more data to assess the performance of the programme; • whether the achievements of the programme have been well documented, such as in the progress reports – if not, the evaluation team may need to collect more data to assess the effectiveness and impact of the programme; • whether there are political or logistical obstacles (e.g. security threats) in conducting or using the evaluation – if there are, the timing or the resources allocated for the evaluation may need to be revisited. 13 4.1 THE INCEPTION PHASE Once the core members of the evaluation team are on board, the inception phase of the evaluation starts. The main objective of this phase is for the evaluation team to design the evaluation and plan the activities. The design and the plan are presented in the inception report. The evaluation manager should conduct the initial briefing of the evaluation team, provide them with all the relevant information and documents, and arrange for the team to meet key stakeholders. In some cases, this involves an inception mission to the field for the team to have a good understanding of the issues and operational environment. 4.2 THE INCEPTION REPORT The evaluation team prepares the draft inception report, which elaborates how the team will conduct the evaluation. The inception report should: • follow the terms of reference; • explain the general approach to the evaluation, such as the methodology to be applied, whether case studies or surveys will be conducted, and how the performance indicators will be used to triangulate qualitative data. The methodology selected for data collection and analysis applies a health equity, disability and gender lens, and analyses human rights implications; • if necessary, develop the theory of change; • further elaborate evaluation questions, for example to develop evaluation sub-questions to answer evaluation questions in a practical manner; • identify data collection methods for each evaluation question and sub-question – an evaluation matrix is one way of doing this; • conduct a stakeholder analysis to adequately cover relevant data sources; • provide a concrete plan of data collection activities, such as interviews, a desk study or a field trip, and an indication of the timeframe for these activities; 4 Designing This section outlines the following main tasks of the design phase: • Beginning the inception phase. • Preparing a draft inception report. • Finalizing the inception report. 14 • propose a list of interviewees or group discussions – in many cases, the identity of individuals may not be known at this stage and so they could be indicated by their affiliation or type; the list should include stakeholders of different genders, people with disabilities, people from identified vulnerable groups and/or people from their representative organizations. The draft inception report submitted by the evaluation team should be reviewed for its quality by the evaluation manager, the reference group and the quality assurance advisor if available. It is good practice to also obtain feedback from key stakeholders on the draft report to ensure the accuracy of the information contained therein, ensure stakeholder buy-in, and avoid disagreements on evaluation questions or activities at a later stage. The stakeholders could be invited to provide the identity of individuals and specific fieldtrip locations that are unknown to the evaluation team. The evaluation team incorporates the stakeholder feedback to finalize the inception report. 15 Box 3. Sample inception report outline [Evaluation title] Inception report [Evaluation team] 10. Introduction 10.1. Purpose: - The mandate of the evaluation team and the purpose of the report (how the evaluation will be conducted – i.e. overall approach, methodologies, evaluation workplan). 10.2. Programme: - Explain the programme, as seen by the evaluation team and as needed. For example, if the programme will be evaluated in parts (e.g. by type of activity, by country or by sector), it should be explained here that the programme is composed of these parts. - If appropriate, refer to a theory of change. 10.3. Scope: - Explain the scope, especially if there is a limitation (e.g. it is not possible to cover a certain period or region). 11. Objectives and evaluation questions - Reaffirm, revise or further elaborate the objectives as needed. - Further elaborate evaluation questions as needed for the evaluation team. Often, this involves further developing sub-questions, keeping in mind data collection methods. 12. Approach and methodology - Reaffirm, revise or further elaborate the overall approach as needed (e.g. case studies, field research). - Explain how each evaluation question will be answered, and the proposed data sources. - Present a stakeholder analysis to ensure relevant data sources are covered adequately. 13. Evaluation workplan - Present a concrete workplan with the timeframe and location proposed for each activity. - Separately, a list of stakeholders to be interviewed could be prepared, organized by type and affiliation and possibly also by evaluation questions. This allows identification of individuals to be interviewed with the help of key stakeholders and informants. 14. Timeframe and deliverables - Propose the schedule of delivery of products, especially if the evaluation workplan suggests some changes from what is assumed in the terms of reference. 15. Resource requirements - Present resource requirements for different sets of activities, especially if the evaluation workplan suggests some changes from what is assumed in the terms of reference. 16 5.1 DATA COLLECTION Once the inception report is agreed, the evaluation team starts collecting data. The evaluation manager should facilitate this process, if necessary arranging meetings and fieldtrips or making additional documentation available. During the data collection process, the evaluation manager and the team should keep the lines of communication open. The evaluation manager should be aware of what the team is doing and give advice when necessary. It is good practice to have the evaluation team debrief the evaluation manager periodically, without breaching the confidentiality of the statements made by the stakeholders, in order to keep the programme manager and evaluation commissioner updated on the progress. On some occasions, such as an interview of a high-level government official or a group meeting of beneficiaries in the field, the programme staff need to accompany the team to the premises and introduce the team members. The staff should not remain on the premises once the interview or the meeting has started, however, to ensure confidentiality and give the stakeholders a space to express their views freely. 5.2 DATA ANALYSIS As the data are collected, the evaluation team should start the data analysis. The evaluation manager should ensure sufficient time is allocated for the team to engage in this analysis. During this time, the team compares the data collected by different members; conducts qualitative and quantitative analyses as needed; discusses the preliminary findings to address each evaluation question and sub-question; triangulates the data and validates the findings; and discusses preliminary conclusions and recommendations. In some cases, the team may need to collect further data if the original data are not sufficient to allow triangulation. It is a good idea to keep some flexibility in the schedule during this time to allow for such additional activities. Participatory and inclusive data collection methods should be used to ensure findings capture the views of stakeholders from different vulnerable groups. In large-scale evaluations – for example, of a multi-country programme in which county case studies are conducted – the data collection and analysis are undertaken in layers. The activity plan and the timeframe should be designed accordingly. 5 Data collection and analysis This section outlines the following main tasks of the data collection phase: • Arrangements for data collection. • Analysis of data. 17 6.1 FIRST DRAFT AND QUALITY CONTROL Once the data are collected and analysed, and the preliminary findings, conclusions and recommendations are formulated, the evaluation team prepares the first draft of the evaluation report and submits it to the evaluation manager. The evaluation manager and the quality assurance advisor, if available, should conduct the quality control, ensuring: • the draft report is written in a clear, logical and readable manner, and nothing is a surprise or incomprehensible; • the draft report responds to the terms of reference, addresses all the evaluation questions and sub-questions appropriately, contains all the components needed for the first draft, and follows the structure as requested; • there are no emotive, offensive or politically inappropriate terms or expressions in the draft report – emotive language implies expressions that convey subjective views or feelings of the author, such as “the progress made was admirable” instead of the more objective “the progress made was beyond the initial expectation”; • there is a logical coherence and rigour, each finding is supported by the data and analysis, the conclusions are drawn from the findings, and the recommendations are based on the findings and the conclusions; • the evaluation findings, conclusions and recommendation reflect a health equity, disability and gender analysis, and provide evidence on human rights implications. Although the first draft may not contain all the components of the final report, such as the front pages, executive summary and annexes, it should contain the main parts of the report, from the introduction to the conclusions and the recommendations. See Box 4. for the structure of the evaluation report. 6 Validation and finalization This section outlines the following main tasks of the validation and finalization phase: • Preparing the first draft of the evaluation report and quality control. • Preparing the second draft and stakeholder validation. • Preparing the draft management response. • Organizing a stakeholder workshop. 18 Box 4. Sample evaluation report outline Cover page [Title, evaluation team, date of issue, issuing entity] Front pages - Acknowledgements - Publication details (if needed) - Table of contents, including annexes (an annex contains material that is part of the report but is not placed within the main text for readability) - Lists of charts and tables - List of annexes (an annex has contents that are self-standing, such as terms of reference, component studies, survey results) - List of abbreviations Executive summary 1. Introduction 1.1. Mandate and purpose: - Why it was conducted. 1.2. Programme and scope: - What was evaluated. 1.3. Methodology: - How it was conducted. - Overall approach. - Evaluation questions and methods used. 1.4. Limitations: - What could not be done. - How the limitations have been overcome. 1.5. Structure of report. 2. Assessment - Generally, follow the evaluation question structure. If the programme is a multicomponent one, it could be structured by components first, as appropriate. - For each evaluation question, present the finding(s), followed by the supporting data and analyses. - Enumerate findings for the ease of reference, for example in a validation process. 3. Conclusions - Present conclusions that are drawn from the findings, respond to the overall evaluation questions, and serve the stated purpose of the evaluation. - Enumerate conclusions. 4. Recommendations - Present recommendations that follow findings and conclusions and serve the stated purpose of the evaluation. - Enumerate recommendations. 5. Appendices 6. Management response (optional) - This could be a part of the evaluation report or a separate document. 19 6.2 SECOND DRAFT AND STAKEHOLDER VALIDATION Once the first draft is quality controlled and revised accordingly by the evaluation team, the evaluation manager should circulate the second draft to the relevant stakeholders, including stakeholders from different vulnerable groups or their representative organizations for their review, requesting comments in writing – known as an audit trail. The standard format for the audit trail is provided in Figure 2. This is the critical stage for the credibility of the report. The evaluation manager should ensure the draft is reviewed by all internal and external stakeholders who engaged in or have substantial knowledge of programme development, implementation or performance. The stakeholder comments could contain general comments and comments on specific paragraphs or statements made in the report. If a stakeholder objects to a statement in the draft, the evaluation manager should invite the stakeholder to provide additional evidence to counter the statement. The evaluation manager should plan at least a few weeks for external stakeholder review, as sometimes the draft is circulated within counterpart organizations and comments are consolidated. As the stakeholder comments are received, the evaluation manager should pass them on to the evaluation team for consideration. The team should record in the audit trail how each comment was addressed in the final draft. When the team does not agree with a comment and does not reflect it in the final draft, the reason should be provided. The stakeholder comments and the team’s response should be kept as the audit trail, which could be referenced in case of disputes or disagreements, and ultimately for the credibility of the evaluation. Having the audit trail in a standardized format facilitates the revision process by the evaluation team and final quality control by the evaluation manager and quality assurance advisor. The evaluation manager should conduct a final quality control by checking whether the evaluation team has adequately and appropriately addressed all the comments. If not, the evaluation manager should return to the evaluation team and discuss how best to address the comments. Figure 2. Sample audit trail format 6.3 DRAFT MANAGEMENT RESPONSE As the final draft is being shaped, the draft and the recommendations should be provided to the programme manager who is primarily responsible for the implementation of the recommendations so that the programme manager can start preparing the draft management response. If the Text/paragraph commented Comments Evaluation team responses (Added after comments are received) Indicate which text or paragraph was commented on. The stakeholder conducting the review provides the comments. Evaluation team indicates how the comments were addressed. 20 recommendations are addressed to multiple stakeholders, more time should be planned for the programme manager to consult and consolidate the draft management response. The draft management response should generally contain overall comments on the report and the response to individual recommendations. A sample format to provide responses to individual recommendations is provided in Figure 3. The response to each recommendation should comprise: • whether it is accepted, partially accepted (or rejected) or fully rejected; • actions planned if the recommendation is fully or partially accepted, and the reasons why the recommendation is fully or partially rejected; • if the recommendation is accepted and the actions are planned, the responsibility and the timeframe to implement them. Figure 3. Sample management response format for specific comments 6.4 STAKEHOLDER WORKSHOP Whenever possible, the evaluation manager should organize a stakeholder workshop before finalization of the report. The stakeholder workshop discusses the final draft report and the draft management response to develop a consensus among all the key stakeholders on the assessment of the programme’s performance – what was working and what was not working – and the way forward, as contained in the draft management response. A sample agenda of the stakeholder workshop is provided in Figure 4. The purpose of the stakeholder workshop is to ensure all key stakeholders accept the report and the management response, creating their buy-in and securing collaboration in implementing the planned actions and, if necessary, reflect their viewpoints in the report and the management response. Participants to invite may include: • the programme manager and the programme staff; • representatives from sectoral departments or the regional office involved in the programme; • government counterpart officials and key policy decision-makers; Recommendation Response Actions planned/reasons for rejection Responsible/timeframe Quote the recommendation concerned Accepted, partially accepted, or rejected Elaborate actions planned for accepted or partially accepted recommendations Explain the reasons if it is partially accepted or rejected Indicate which entity will be responsible for each action planned, and the indicative timeframe to complete it 21 • actual and potential funding partners; • implementing partners; • collaborating partners; • beneficiary representatives or partners representing beneficiary interests; • stakeholders from identified vulnerable groups or from their representative organizations. Once the stakeholder workshop is finished, the evaluation team and the programme manager could adjust the report and the management response as appropriate and proceed to finalizing them. If it is logistically impossible or not preferable to conduct one workshop for all the stakeholders to participate, this can be done more than once – for example, a workshop can be held with internal stakeholders first, the drafts and presentations adjusted accordingly, and then another workshop organized for a broader audience. Figure 4. Sample stakeholder workshop agenda Agenda 1. Welcome from organizers – 5 min. 2. Introduction of purpose and agenda of the meeting – 10 min. 3. Presentation of evaluation report (final draft) – 20 min. 4. Questions and clarifications with evaluation team – 15 min. [Break – 10 min.] 5. Presentation of way forward/management response (draft) – 20 min. 6. Comments from key stakeholders – 20 min. 7. Discussion with floor – 1 hour. 8. Concluding remarks by programme manager, reflecting on comments and discussions – 15 min. 22 7.1. DISSEMINATING THE REPORT When it is finalized, the evaluation report should be submitted to the people who requested the evaluation and the people who will be involved in key decision-making for the programme (e.g. senior management, programme manager, funding partner(s), programme steering committee, government counterparts). The report should also be shared with the people who will be involved in the implementation of the programme and the management response (e.g. programme staff, implementing partners). For these core stakeholders, separate briefing sessions could be organized, especially for those who did not participate in the stakeholder workshops. Beyond these core stakeholders, a communication material (e.g. a briefing note) that summarizes key messages from the report and the management response could be prepared and distributed widely. For evaluations that are strategically important or need to address public accountability, key messages could be disseminated more broadly, such as through information sessions with partners and beneficiaries, articles in the programme’s newsletter, or video clips on the programme website or other online platforms. The report, management response and associated materials (e.g. the briefing note) should be sent to the WHO Evaluation Office to be uploaded to the Organization-wide evaluation repository. This allows cross-use of evaluation reports within and outside the Organization. The Evaluation Office will include information on the evaluation in the annual evaluation report submitted to the Executive Board. 7.2 FOLLOWING UP ON IMPLEMENTATION It is good practice for the programme manager to check on the implementation of the actions planned in the management response. This could be done, for example, a few years later with the steering group of the programme or with government counterparts in reviewing the programme implementation. Under the guidance of the Executive Board, the Organization has prepared a database containing governing body decisions and recommendations from the oversight bodies, and their implementation status. Recommendations from all evaluations that are accepted in the management response should be a part of this database. The programme managers, or officers responsible for the relevant segment of the database, must update the information periodically according to the set procedure. The 7 Dissemination and follow-up This section outlines the following main tasks of the dissemination and follow-up phase: • Disseminating the report. • Following up on implementation. 23 implementation of these decisions and recommendations will be monitored by the Organization and the Executive Board. 24 8.1 PROGRAMME THEORIES AND EVALUATION The logical framework, typically represented in a 4×4 matrix, was the main programming instrument until the 1990s and is still used to provide a broad framework for programmes, mainly for accountability purposes. The original Organisation for Economic Co-operation and Development Assistant Committee evaluation criteria (relevance, impact, effectiveness, efficiency, sustainability) were developed partly to be aligned to the structure of the logical framework. Sample logical framework Description Indicators of achievement Means of verification Assumptions Goal Relevance Impact Purpose Effectiveness Outputs Efficiency Activities/Inputs The logical framework was criticized, however, as representing a too simplistic linear model, in which one thing is supposed to lead to another, given the right conditions. It was also criticized as treating social, economic and political contexts as external factors outside the consideration, except as conditions to meet the assumptions, and this was not helpful in development programmes where these factors are of great importance. Over the past few decades, therefore, programmes and evaluations have increasingly used the theory of change as an instrument to describe the programme logic. With the theory of change, the programme designer can internalize contextual factors and interrelationships freely into the programme logic and represent the reality more closely than with the regimented logical framework. 8 Annexes 25 The theory of change is useful for expressing the programme logic and discussing the programme design. Evaluators usually use it to gain a common understanding of the programme logic with the programme staff, and to define which causal relationships to focus on, elaborate evaluation questions around them and identify appropriate evaluation methods. Because it comes in multiple forms and shapes, however, organizations have found it unsuitable for use as an accountability framework. The theory of change still has some deterministic features, in that it gives the impression that one thing (or several things) should lead to another. It is often viewed as a more complicated version of the logical framework. Some programme designers and evaluators therefore found it not suitable to apply it to more “fuzzy” and complex cases such as to assess policy influence or awareness-raising. Analogies used were that the “complicated” case was like building a rocket, in which multiple factors interact in a complicated but still predictable way, while the “complex” case was like raising a child, in that a child could react in different ways at different times even to the same parental guidance. One programming instrument developed in 2001 that could be applied to these “fuzzy” cases is outcome mapping. This looks somewhat similar to the logical framework in structure, but it redefines the causal relationship as influencing rather than causing. It focuses on outcomes that are defined as changes in behaviour of immediate partners that were supposed to be influenced (“boundary partners”), rather than tangible and measurable outcomes. It does not attribute the change (outcome) to the action (cause) but considers the plausibility that the action has influenced the change. It recognizes that such changes in behaviour could ultimately lead to impact, which is described as the change in state, a more tangible change. Source: https://about.makerble.com/blog/2018/7/24/a-practical-way-to-think-about-your-theory-of-change Variations of the Theory-of-Change 26 Although the evaluation methodology of outcome mapping is not yet firmly established, one methodology proposed and practiced is outcome harvesting. This proposes to identify outcomes first – that is, the changes that were plausibly influenced by the actions and plausibly contribute to the change in state. It then triangulates the outcomes and the hypothesized plausibility. Since WHO is a specialized agency providing considerable normative and technical guidance and advice, this programming and evaluation methodology is worth considering. 8.2. EVALUATION QUESTIONS Defining good evaluation questions is critical for a well-designed evaluation. The evaluation should have a purpose – why do we need to conduct this evaluation? This should lead to the identification of questions that the evaluation should answer to make it useful and serve the purpose of the evaluation. These questions are sometimes called the overall or main evaluation questions, to be raised in the first section of the terms of reference, and should be answered in the evaluation’s conclusions in one way or another. Such main evaluation questions, however, are often not directly answerable by research and need to be converted into a set of answerable questions, corresponding to the research methodology. These questions, often called sub-questions or methodological questions, should be described in the section on “Objectives and evaluation questions” in the terms of reference. It is usually necessary to dissect the subject matter and build these methodological evaluation questions by evaluation criteria, type of activity, sector, target beneficiary group, boundary partner, and so on. For instance, one needs to develop different data collection methods to assess activities that are aimed at different target beneficiary and vulnerable groups, or the different aspects of the programme. Therefore, in most cases, these evaluation questions are structured in a nested manner with sub- questions and sub-sub-questions. The guidance note on integrating health equity, gender equality and human rights in WHO evaluations provides examples of evaluation questions based on the type of evaluation conducted. Outputs/ actions Outcomes/ changes in behaviour Impact/changes of state Sphere of control Sphere of influence Sphere of interest Outcome Mapping 27 The evaluation manager should develop such evaluation questions and sub-questions in the terms of reference to a reasonable extent – that is, to the extent to be clear on what the evaluation manager wants from the evaluators. The evaluation manager could, however, leave the elaboration and the finalization of the evaluation questions to the evaluators, to be proposed and agreed in the inception report. In the evaluation report, the answers to the evaluation questions and sub-questions should generally be presented as the findings. Having a clear correspondence among the evaluation questions, the findings, and the supporting evidence and analysis enhances the quality and readability of the evaluation. Some funding partners require the evaluation to use the OECD-DAC criteria (relevance, impact, effectiveness, efficiency, sustainability, coherence), but it is not obligatory otherwise to apply all these criteria. The important thing is to conduct evaluation for its stated purpose, and this would not always require applying all the OECD-DAC criteria. 28 Box 5 is a demonstration of how to structure evaluation questions, using a hypothetical evaluation. Box 5. Evaluation of an awareness-raising programme on nutrition requirements at the community level Programme activities: • Provide community-based clinics with information materials to raise nutrition awareness of families. • Introduce a short course on nutrition requirements in teachers’ training so the teachers can in turn educate students and raise awareness of their families. Evaluation purpose: • To decide whether to continue the programme as it is. • To see whether resources should be reallocated between the two activities. Evaluation questions and sub-questions: Q1. How effective was the awareness-raising activity through the clinics? Data collection methods/sources: survey of clinic workers who received the information material, interviews of sample of clinic workers, household survey of families in sample communities, analysis of project records. Q1.1 What was the extent of the coverage of the clinics and the families? Q1.2 To what extent has this activity led to behavioural changes in the families? Q1.3 What were the major obstacles in instigating the behavioural changes? Q2. How effective was the awareness-raising activity through the teachers’ programme? Data collection methods/sources: survey of teachers who took the course, interviews of sample of teachers, household survey of families in sample communities, analysis of project records. Q2.1 To what extent were the teachers engaged in the nutrition education of students, and what would be the resulting coverage in terms of the families? Q2.2 To what extent has this activity led to behavioural changes in the families? Q2.3 What were the major obstacles in instigating the behavioural changes through this activity? Q3. Should resources be reallocated between the activities for enhanced effectiveness? Data collection methods/sources: comparative cost/benefit analysis of two activities, group discussions at selected communities with health workers and families. Q3.1 How does the cost/benefit compare between the two activities? Q3.2 To what extent would a resource increase in each activity raise effectiveness? Q4. Have the programme design and implementation considered gender equality objectives properly? Data collection methods/sources: group discussions at selected communities with health workers and families, programme documents and reports. Q3.1 How has the programme design taken into consideration gender equality, health equity and human rights objectives? Q3.2 Has the programme applied principles and measures to promote gender equality, health equity and human rights in its implementation? Q3.3 Has the programme contributed to the reduction or elimination of barriers to access to health for people from vulnerable populations (e.g. women, adolescents and youth, people with disabilities, people from indigenous communities, people from sexual diversities), particularly those that are furthest behind? 29 8.3 WRITING STYLE OF THE EVALUATION REPORT The evaluation report should be written in such a way that readers can see the findings are supported by evidence. Each finding should be followed immediately by the data and analysis that support it. The findings should not be put together in one big chunk, such as at the outset of a section. The findings should be written distinctly from the main text (e.g. in bold or set in a box) and enumerated. A similar principle applies to the conclusions and recommendations, which should be written distinctly and enumerated. Sample writing style of findings Finding 5. The technical guidance on ... has been accepted by most countries engaged, and about half of the countries were in the process of implementation, revising the regulations or ... The country case studies and the country survey revealed that, out of 80 countries that were ... Evaluation reports are an instrument for public accountability and will be read by external partners. Abbreviations should be avoided as much as possible, especially those that are internal to the Organization (e.g. names of departments). If abbreviations are used, they should be defined in a list in the front pages. The executive summary should not be an abridged version of the evaluation report. It should clearly convey the key messages from the evaluation report and focus on elements appropriate for high-level decision-makers and key external partners – “the executives”. In general, the Organization’s editorial style manual4 and other standard style guides for academic publications should be followed. 4 https://apps.who.int/iris/handle/10665/36842 30 8.4 REFERENCES The two key documents that provide the foundation for WHO evaluations are: • Evaluation policy (2018). Geneva: World Health Organization; 2018 (Evaluation: evaluation policy (2018) (who.int) accessed 4 April 2023) • United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation; 2016 (Norms and Standards for Evaluation (iom.int), accessed 20 April 2023) Other important reference documents from UNEG include: • UNEG Ethical Guidelines for Evaluation; 2020 (2020 Ethical Guidelines for Evaluation.pdf (iom.int), accessed 20 April 2023) • UNEG Evaluation Competency Framework; 2016 (www.unevaluation.org/2016-Evaluation- Competency-Framework, accessed 20 April 2023) • UNEG principles for stakeholder engagement. New York: United Nations Evaluation Group; 2017 (UNEG Principles for Stakeholder Engagement Final Nov2017.pdf., accessed 20 April 2023) • Integrating Human Rights and Gender Equality in Evaluations; 2014 (http://www.unevaluation.org/document/download/2107, accessed 20 April 2023) • Guidance on Integrating Disability Inclusion in Evaluations and Reporting on the UNDIS Entity; 2022 (Guidance on Integrating Disability Inclusion in Evaluations and Reporting on the UNDIS Entity Accountability Framework Evaluation Indicator, accessed 20 April 2023) • Guidelines for the evaluation of the United Nations Sustainable Development Cooperation Framework. New York: United Nations Evaluation Group; 2021 (http://www.unevaluation.org/document/download/3737, accessed 20 April 2023) • The Humanitarian-Development Nexus – What do evaluations have to say?; 2018 (http://evaluation.uneval.org/document/download/2872, accessed 20 April 2023) Some online portals that are used widely by evaluation professionals include: • Better Evaluation (https://www.betterevaluation.org/) • Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) (https://www.alnap.org/) Any enquiries about this evaluation should be addressed to: Evaluation Office, World Health Organization Email: evaluation@who.int Website: https://www.who.int/about/what-we-do/evaluation