The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects

Department of Reproductive Health and Research World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland Fax: +41 22 791 4171 E-mail: reproductivehealth@who.int Website: www.who.int/reproductivehealth The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978 92 4 151656 3 19072_Sample cover_3.5mm spine.indd 1 08/10/19 08:28

The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health and rights projects ISBN 978-92-4-151656-3 © World Health Organization 2019 Some rights reserved. This work is available under the Creative Commons Attribution-Non Commercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. Suggested citation. The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health and rights projects. Geneva: World Health Organization; 2019. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at https://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see https://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see https://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Design and layout by L’IV Com Sàrl Printed in Switzerland Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. How was this guidance developed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Who should use this guidance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. What is a post-project evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 5. Why is there a need for post-project evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 6. What are the common reasons for and benefits of conducting post-project evaluations of ASRH projects? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 7 What can you expect when planning and conducting a post-project evaluation? . . 10 8. Steps in planning a post-project evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 9. Common challenges and strategies to address them . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 9a. Defining the purpose and scope of the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 9b. Establishing the post-project evaluation team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 9c. Addressing the lack of data availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 9d. Developing feasible sampling strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 9e. Bringing external context into evaluation consideration . . . . . . . . . . . . . . . . . . . . . . . 35 9f. Assessing sustainability and emerging and unanticipated outcomes . . . . . . . . . . . . 38 10. Assessing cross-cutting project principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 10a. Positive youth development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 10b. Gender equality and human rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 10c. Poverty alleviation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 11. How can evaluators, programmes and funders support post-project evaluation? . . . . 46 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Appendix A: Analysis report – review of post-project evaluations of ASRH projects . . . . . . . 50 Appendix B: Post-project evaluation evaluability checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Appendix C: Evaluation websites and clearinghouses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S i i i Acknowledgements The World Health Organization (WHO) is grateful for the contributions that many individuals and organizations have made to the development of this guidance. This guidance was coordinated by Venkatraman Chandra-Mouli of the WHO Department of Reproductive Health and Research, with support from Marina Plesons and oversight from Lale Say. WHO extends sincere thanks to Susan Igras, who carried out the literature review and developed the first draft of the guidance. WHO is also grateful to participants of a WHO technical consultation for their review and feedback to the first and subsequent versions of the guidance: Susanne Carai, Sarah Castle, Bregje de Kok, Fariyal Fikree, Kylie Hutchinson, Emma Jackson, Rebecka Lundgren, Bella Mansanya, Kristien Michielsen, Ghazaleh Samandari, Shantanu Sharma, Manahil Siddiqi, and Judith Westeneng. Lastly, WHO thanks Vivien Stone for editing the document, L’IV Com for designing the document, and Graham Ogilvie for preparing the illustrations. This work was funded by the John D. and Catherine T. MacArthur Foundation. The views of the funding bodies have not influenced the content of this guidance. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!iv Abbreviations ALNAP Active Learning Network for Accountability and Performance ASRH adolescent sexual and reproductive health AYSRH adolescent and youth sexual reproductive health CA contribution analysis CiDA Canadian International Development Agency DAC Development Assistance Committee (OECD) DFiD Department for International Development iPPF International Planned Parenthood Federation iPv intimate partner violence JiCA Japan International Cooperation Agency LMiCs low- and middle-income countries MSC most significant change OECD Organisation for Economic Co-operation and Development PYD positive youth development QUiP Qualitative Impact Protocol REM ripple effect mapping SDGs Sustainable Development Goals UNiCEF United Nations Children’s Fund USAiD DEC United States Agency for International Development Development Experience Clearinghouse WHO World Health Organization P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S v

Overview 1 In many countries, initiatives that aim to improve adolescent1 sexual and reproductive health (ASRH) implement projects without well thought out plans for evaluation. A range of reasons help explain this phenomenon, including limited training and understanding of the value of evaluation and dedicated evaluation resources within implementing and technical entities. However, the end result is that baselines and endlines may not exist, there may be no comparison of the project area to a non-intervention area, and implementation monitoring data may not provide much information on quality and other less easily counted impacts or results of implementation. The resultant lack of evaluation limits the ability to demonstrate what was achieved, what approaches worked, and what approaches did not work, thus losing out on critical learnings. Nevertheless, managers or funders with an interest in designing future projects or expanding existing projects may later commission a post-project evaluation to assess impact and learning in the months or years after a project has ended. Many ASRH projects encounter and address policy and programmatic challenges, and in doing so, learn valuable lessons. Since careful documentation and rigorous evaluation are the exception rather than the norm, though, the lessons from these efforts are not extracted and placed in the public arena. Post-project evaluations thus have a role to complement prospective studies for new or follow-on projects. However, there is a lack of guidance and literature on post-project evaluations, and technical guidance from funding agencies for conducting post-project evaluation is rare. To begin to bridge this gap more formally, the World Health Organization (WHO) developed this practical guidance for conducting post-project evaluations of ASRH projects. 1 WHO defines adolescents as people between the ages of 10–19 years. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 1 2 A literature review was conducted to understand common post-project evaluation rationales, challenges and solutions. The review included:  peer-reviewed and grey literature articles and post-project evaluation reports of ASRH projects, identified through a search of seven evaluation clearinghouses from low- and middle-income countries (LMICs) between 2010 and 2017 (Appendix A); and  a review of post-project evaluation guidelines published by international organizations, governments, and development agencies. WHO then convened a technical consultation of experts to review and provide inputs to the first draft of the guidance document, which led to revisions and the eventual final document. How was this guidance developed? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!2 3 Aimed primarily at evaluators, this guidance may also be of interest to project managers and funders. Its purpose is to help evaluators to think through the key decisions to be made to yield the most rigorous evaluation possible given available financial and human resources and project documentation. This guidance includes challenges, tips and lessons learned from the literature on post-project evaluations to inspire ideas for future evaluations. Case studies highlight selected issues as well as the responses taken by evaluators to address them. Although focused on post-project evaluation, this guidance is also relevant to evaluators of final project evaluations in situations where evaluation was not planned from the start or where projects have shifted focus midway. Who should use this guidance? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 3 4 There is no common definition of post-project evaluation, and many terms are commonly used (see Box 1). We use a definition informed by the Japan International Cooperation Agency (JICA) because it describes what this type of evaluation is, what its objectives are, and how it is carried out: Post-project evaluations are performed within a certain period after a project is completed and most are not planned while projects are still operating. As staff may no longer be available and project activities are not observable, post-project evaluation is based on existing reports, monitoring records and, other written information, and often includes additional data collection. While such evaluations may assess the extent that projects implemented planned activities and how well outcomes were achieved, given they occur after projects end such evaluations may also focus on longer term impact and sustainability. Post-project evaluations consider retrospectively circumstances surrounding a project and other elements that may have influenced implementation, impact and sustainability of results. The definition focuses on the timing and purpose of post-project evaluation to distinguish it from final or endline evaluations which are conducted as projects are ending – when staff, project counterparts and project beneficiaries are present and many project activities are still observable to assess quality and fidelity of implementation. Figure 1 shows the relationship of evaluation to project planning and implementation, as well as where and how post-project evaluation can support project assessment and learning. While post-project evaluation can answer many typical evaluation questions, they are particularly well placed to determine if project outcomes (such as improvements in adolescent-friendly sexual and reproductive health services or shifts in gender attitudes) remain viable over time when support ends. What is post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!4 box 1Related terms for post-project evaluation • after-action review • ex-post evaluation • forensic evaluation • post-completion evaluation • post-hoc evaluation • post-implementation evaluation • post-implementation review • post-project evaluation • retrospective impact evaluation • sustainability impact evaluation • sustained and emerging impacts evaluation Problem/issue Project design and plan Activities/outputs Outcomes Effects/impact Post-project outcomes/effects Resources/inputs Relevance and appropriateness Does it make sense? • Did the project objectives address the need? • Did the project intervention address the right issues? • Is there (still) a need? Effectiveness Did it work? • Did the project achieve the desired objectives/outcomes? • Was the intervention based on knowledge and research to improve the likelihood of success? Effects/impacts • Was the project worthwhile? • Does it have merit? Efficiency Was it efficient? • What was the relationship between the project’s inputs and outputs? • Could resources have been better used? Process surrounding the interventions Was it well-managed? • Did planning and decision-making processes ensure the project’s success? • Did management processes ensure success? • Did processes for developing activities ensure their success?    Contextual and external factors outside project boundaries that may have influenced the project trajectory    Sustainability and unanticipated and emerging effects • Are the project’s effects lasting? • Was the project as worthwhile and as meritable as thought? Figure 1. The logic of a project and its relationship to key evaluation themes Source: Adapted from Ryan et al., 2016. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 5 5 Evaluation is still variable and evolving as a state responsibility within many governments (World Bank, 2013) and the evaluation of social development projects is still an emergent field of inquiry. Evaluators often operate in contexts where data systems are weak and multiple interventions complicate the assessment of project contributions to outcomes. In an era of heightened demands for accountability by governments and organizations, evaluation increasingly is seen as a key tool to assess the impact of interventions and investments (Morra- Imas & Rist, 2009).  In this context, project evaluations vary widely. A relatively small number of applied research projects employ elegant evaluation designs focused on impact. Many projects do not have resources, evaluation training or a proclivity in a politicized environment to conduct evaluation (World Bank, 2013). Post-project evaluations are even rarer, representing less than 1% of international development projects that are evaluated (Zivetz et al., 2017).  Projects operating in conflict-affected and emergency contexts often remain unevaluated given the special challenges of implementation and short project timeframes (Puri et al., 2015).  There are fewer evaluations of ASRH projects available in evaluation clearinghouses compared with evaluations of other sectors. Of 900 evaluations available in a range of evaluation clearinghouses (see Appendix C), only 41 were focused on adolescent and youth sexual and reproductive health (AYSRH) projects. Of these 41 evaluations, only 6 were post- project AYSRH evaluations. Compared with ASRH projects in high-income countries, there is a paucity of project evaluation from LMICs linked to a theory of change and outcomes, and the body of evidence on good practices is small (WHO, 2004; Chandra-Mouli et al., 2015). Consequently, many ASRH projects do not include evaluation in their project design, or consider it as an afterthought. If a project is exploratory or seems to be performing very well and there is a desire to expand it, only then might evaluation be considered to provide important evidence. However, by this point, there is often no funding for evaluation within project budgets. Why is there a need for post- project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!6 Effective evaluation seeks to assess the merit or worth of projects according to five evaluation criteria – the projects’ relevance, effectiveness, efficiency, impact and sustainability (see Box 2) (JICA, 2004; OECD, 2000). Post-project evaluation is most suitable for assessing impact and sustainability and, possibly, relevance (JICA, 2004). It can also provide insights into effectiveness and efficiency. Rigorous evaluation planned from the initiation of a project should be the norm. Where this is not possible, or even if endline evaluation has occurred, post-project evaluation offers some unique possibilities for learning to:  Explore planned or additional project impacts or effects that may have occurred after the end-of-project evaluation period.  Assess the sustainability of a project’s impacts or effects. Findings from an end-of-project evaluation are not always adequate predictors of sustainability. Post-project evaluation can add to learning about sustainability by:  Understanding what elements of a project were sustained (or not) after the project ended.  Gaining deeper insight into how project-introduced approaches and activities have spread, diffused or replicated through anticipated and unanticipated pathways.  Gaining a better understanding of causal or associated factors related to the above (the why and how).  Identifying unexpected and emerging outcomes that came about after the project’s end. 6 What are the common reasons for and benefits of conducting post-project evaluations of ASRH projects? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 7 box 2  Potentially collecting information on a project’s relevance. Staff, stakeholders and end- users/participants, if still available, may have had more time to reflect on the project and may have additional insights to add regarding its relevance. For ASRH projects post-project evaluation can:  Reach older adolescents and young adults who may offer new and/or different insights into project impact, sustainability of its impact, and its relevance.  Contribute to a better understanding of whether the theory of change mechanisms were valid or not. This can inform the development of promising project approaches and activities and contribute to greater clarity about change mechanisms that lead to improved adolescent health and well-being. Table 1 shows the range of reasons for conducting post-project ASRH evaluations. Criteria for evaluating development assistance Relevance: This seeks to assess the extent to which the activity is suited to the priorities and policies of the target group, recipient and funding agency. Effectiveness: This seeks to assess the extent to which an activity attains its objectives. Efficiency: This seeks to assess the outputs – qualitative and quantitative – in relation to the inputs. It is an economic term that signifies that the activity uses the least costly resources possible in order to achieve the desired results. Impact: This seeks to assess the positive and negative changes produced by a project, intended or unintended. This involves the main impacts and effects – intended and unintended – resulting from the activity on local social, economic, environmental and other development indicators. Sustainability: This seeks to assess the extent to which the benefits of an activity are likely to continue after funding has been withdrawn. Source: Adapted from the Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD), 2000. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!8 Accountability  Did the project meet the expectations of adolescents and other stakeholders and funders?  What difference did the project make in the wider environment in which it operates? Evidence of effect  Did the project achieve its intended outcomes? Justification for future funding  Is the project strategy worthwhile enough to warrant future investment in similar projects?  If so, under what contexts and with what adjustments? Organizational and project learning  What evidence exists for an organization to implement a similar project going forward, or to scale up a successful project?  If so, under which contexts and with what adjustments? Sustainability  Are the project’s effects still visible?  If new structures, services or projects were put into place, are they still operating? Unanticipated or emerging impacts  What unanticipated effects have been observed since the project ended?  Have any unforeseen impacts – positive or negative – emerged since the project ended? Table 1. Typical purposes of post-project ASRH evaluations P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 9 7 Each post-project evaluation situation will be different, with different resources and constraints. As a result, the evaluator will have to determine the most appropriate evaluation design amidst some necessary methodological compromises. Some typical challenges that may be encountered (which are also common in other types of evaluations) include:  Project design documentation is of uneven quality. For example, there is no articulated theory of change or logic framework.  Qualitative descriptions of implementation are not available. For example, project reports are focused on explaining outputs and outcomes rather than processes.  Quantitative project data are patchy. For example, monitoring data were never collected or are not available for review; some critical activities are not included in monitoring systems; or the project has not conducted evaluation or has only done a baseline evaluation. The evaluator will likely have to play a variety of roles in addition to traditional evaluator, such as technical advisor to recreate or refine former project theories of change that will guide evaluation questions, or investigative reporter to uncover project documents or former project staff and partners. The evaluator will likely also play a liaison role between the funding agency and the organization that implemented the project. See sections 8 and 9, which share practical guidance for those planning and implementing a post-project evaluation based on experience and lessons learned on post-project evaluation of ASRH projects. What can you expect when planning and conducting a post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!10 box 3Typical challenges of post-project evaluations: the TUSEME Project in the United Republic of Tanzania The TUSEME Project (“Let’s Speak Out” in Kiswahili) was implemented in the United Republic of Tanzania, and employed a rights-oriented, gender-responsive approach to empower girls to understand and overcome problems that constrain their development, including academic achievement. Initially implemented in seven secondary schools in five regions, TUSEME was expanded to 27 schools and was eventually mainstreamed into the national Secondary Education Development Plan and replicated in 237 secondary schools. A post-project evaluation of the TUSEME clubs was carried out to assess the sustainability of student participation and the impact of participation on girls’ empowerment and acquisition of life skills. However, evaluators faced several challenges:  Framing the evaluation: Due to financial and time constraints, the evaluation methodology could not be designed to test effectiveness with a comparison group (e.g. schools with TUSEME clubs and schools without clubs). As a result, the initial terms of reference were renegotiated to include only schools with TUSEME clubs.  Data availability and sampling: The project had no baseline data. Similarly, other documentation was largely lacking, including a project log frame, monitoring and evaluation reports and financial documents.  Primary data collection: Interviewing adolescents was challenged by the need to find former project participants, and once found, obtain consent from students’ parents and informed assent from students below 18 years. Additionally, many parents were not available to talk with evaluators at the time of the school visit when interviews were conducted. Most who were available did not have children in the clubs. Source: Mhando, 2015. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 11 8 The following steps are similar to the steps taken for an evaluation conducted when projects are still functioning or have just ended. However, one key exception is the critical importance of doing an evaluability assessment at the start to determine the extent to which a post-project evaluation can be conducted reliably and credibly. STEP 1: Define the purpose, scope and initial design of the post-project evaluation. Determine the purpose and scope of the evaluation, including the evaluation questions, with key stakeholders, such as implementation organizations, relevant government ministries/ departments and funding agencies.  What are stakeholder expectations for the evaluation?  What do they want to learn and why?  What is the intended use of the evaluation findings and how will they be disseminated? Consider engaging stakeholders beyond the implementing organization and immediate partners, such as government stakeholders and future users of evaluation findings. Their early inputs can contribute to the credibility of the evaluation and use of findings to improve current and/or future projects. Steps in planning a post- project evaluation T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!12 STEP 2: Conduct a post-project evaluation evaluability assessment. An evaluability assessment can provide critical information from multiple perspectives on the extent to which a post-project evaluation can be conducted reliably and can generate credible results. The evaluability assessment should address:  Organizational considerations to ensure stakeholder consensus on the need for, and utility of, the evaluation to inform policy and programmatic decision-making.  information availability to support evidence-based evaluation.  Methodological considerations to be able to answer evaluation questions.  Resource considerations to ensure sufficient financial, human and material resources are available for evaluation.  Timing considerations to collect and analyse data in a logical and timely manner.  Acceptability considerations to gain stakeholder consensus on the type and level of evidence that is acceptable. An evaluability assessment checklist (see Appendix B) can be a useful tool for this critical activity. STEP 3: Identify information gaps and design a feasible methodology with the maximum rigour possible. Once the evaluation purpose(s) has been specified with project stakeholders and the results of the evaluability assessment are available, it will be possible to make choices and understand what compromises might be necessary regarding the evaluation design. Figure 2 provides a decision tree of possible evaluation design choices based on key learning objectives and information/data possibilities. Who will be involved in the evaluation process? How will adolescents be engaged (e.g. will they have roles besides being respondents, such as members of the evaluation team or as advisors to the evaluation team)? Are theories of change and logic frameworks available and will they need to be adjusted for use in the evaluation? Are data and process documentation available? What are the possibilities for sampling and additional data collection? How will you approach analysis and the interpretation of findings? These common challenges are addressed in detail in section 8. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 13 Figure 2. Decision tree on possible evaluation design choices Impact/ effectiveness Process/ implementation Post-test only with no comparison group Quasi-experimental pre/post-test design with non-random group assignment Post-test only with non-random group assignment Time series design (if multiple data points are available) Pre/post-test design with no comparison group (if only two data points are available) Use qualitative methods to deepen understanding of impact and gauge unintended effects Independent analysis of process data Reconstruct process through: Mixed methods including interviews, focus group discussions, case studies, etc. with key project staff, partners, beneficiaries, and the community. Efficiency could be examined if cost data can be linked to activities Create and analyze process data using administrative/ performance data Where and when feasible, engage young people in the evaluation design, data collection, and interpretation of findings. Sustainability Independent analysis of sustainability data Use mixed methods to collect data on: • Political support • Environmental/ contextual factors • Funding stability • Partnerships • Organizational capacity • Project adaptation • Project weaknesses • Project strengths • Communication • Strategic plan • Sustainability of impact What do you want to learn about the intervention? Supplement with additional data, where possible Did the project collect sustainability data? Did the project collect process data? Can you access administrative/ performance data? Is there adequate data on a group that did not receive the intervention to create a comparison group retrospectively? Can you collect data from both the comparison group and the intervention group? Is there adequate data on the intervention group to create data points retrospectively? Supplement with additional data, where possible Yes No T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!14 STEP 4: Confirm the evaluation methodology with key stakeholders to obtain buy-in for later utilization. It is time to share the post-project evaluation plan – the evaluation purpose, evaluation questions and data sources, methods to analyse existing data and collect and analyse additional data, and dissemination approach – for feedback and to ensure the consensus of project stakeholders. In particular, the plan should be shared with those most likely to use the evaluation findings, whether that is the implementing organization, government or funding agency(ies). A caveat: at this point, it may be necessary to go back to the organization or funding agency commissioning the evaluation to explain that some of the initial evaluation questions are unanswerable with the information and/or resources available. STEP 5: Secure ethical approval for the evaluation. It is critical to ensure the protection of adolescents in any evaluation of an ASRH project (WHO, 2018). The evaluator should secure approval from an institutional review board or a waiver of formal ethical review if the evaluation is considered to be a routine project assessment, and revise the proposal as needed. STEP 6: The evaluation is ready to begin! With a clear plan, consensus from project stakeholders, and ethical approval, data collection is ready to begin. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 15 9 This section shares commonly cited challenges during the evaluation process that are unique or critical to post-project evaluation, drawn from a variety of sources referenced in this document, and suggests how they could be mitigated or overcome. Case studies, drawn from actual evaluations, serve to highlight some of the challenges. This section will not cover challenges common to all evaluations, such as those related to financial, time and/or human resource constraints and those related to navigating political environments. While stakeholders may be interested in learning about cost–effectiveness, it is also not addressed in this guidance. Collecting retrospective, reliable cost data linked to project activities is a significant challenge that requires former project staff to recall the time and effort spent on specific activities and that requires former project finance staff to link finance data with activity data. Common challenges and strategies to address them T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!16 9a. Defining the purpose and scope of the evaluation Defining the evaluation’s purpose and scope, as well as its key boundaries, are essential to designing a good evaluation. Measurable performance indicators, theories of change and a clear definition of beneficiary groups provide the intentions and pathways through which project changes are expected to occur. When the post-project evaluation aims to assess impact, an evaluator’s key concerns revolve around assessing possible association or, where possible, causality; whether a feasible comparison group can be found; and whether baseline and endline evaluations exist. When the evaluation aims to assess sustainability or the project’s implementation process, key concerns revolve, respectively, around retrospectively defining the expected sustainability outcomes and attempting to recreate processes if these are not well described in existing documentation. Challenge: No quantitatively measurable indicators or performance targets were established before or during the project against which to assess change. Strategy: Develop performance indicators using documentation on project objectives and aims, in addition to secondary data from national surveys or similar projects if available. Present the new performance indicators to stakeholders and gain consensus for their use in the evaluation. If there are baseline indicators but no established performance targets, discuss with stakeholders whether changes observed in these indicators from baseline represent reasonable performance for the project. If indicators are not meaningful (i.e. are either too ambitious to establish causality or are not logical within the project context), discuss with stakeholders and develop consensus for redefined performance targets. Challenge: No written theory of change exists to guide understanding of association or causality. if such documentation does exist, it may be outdated given changes that occurred during project implementation. Strategy: Work with available project staff and other stakeholders to develop a theory of change and/or a logic framework. The former explains pathways leading to change, and the latter provides an orderly structure to monitor project implementation. If theories of change and logic frameworks do exist, review and revise them as needed (e.g. if there were shifts in implementation strategies or assumptions underlying implementation). Note, though, that creating a theory of change retrospectively may present an additional limitation in the interpretation of findings. Resource: For suggestions on how to create a theory of change or logic framework retrospectively with project stakeholders see: https://actionevaluation.org/theory-of- actiontheory-of-change-tools-resources/. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 17 box 4Reconstructing the project logic framework: the Tarunya Project in India The Tarunya Project, which was implemented by Engender Health and partners over a 5-year period (2008–2013) in Jharkhand State, India, was designed to:  strengthen health systems and health service provision;  enhance community engagement and links between the services and community activities; and  strengthen ASRH programme leadership and management by the state and district health management teams. WHO was invited to conduct a post-project evaluation and the evaluation team quickly learned that no explicit programme theory of change or logic framework existed, making it impossible to develop suitable evaluation questions. The team collected all of the relevant project documents and used information from multiple sources to construct a logic framework to explain what the project had set out to do and what it had actually achieved. In doing so, the team put effort into ensuring that the logic framework was entirely faithful to the information in the project plans and reports. The logic framework then guided development of a set of evaluation questions related to Tarunya’s design, implementation, outputs, outcomes and impact. Upon presenting the logic model to project staff, they asked, “Where did this come from?” Evaluation team members explained that the project’s own plans and reports were the source of the logic framework. By pointing to the sources of specific information in the logic framework, the team showed project staff that the model was not invented but rather articulated a structure that was never made explicit on paper. Source: Barua & Chandra-Mouli, 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!18 Challenge: Concepts underpinning theories of change, such as “reaching vulnerable adolescents”, were not well defined operationally, making later evaluation decisions on sampling and measurement of project coverage difficult. Strategy: Work with project staff to articulate precise definitions for these concepts, based on their experience and understanding. If the original project staff are not available, identify potential definitions from the literature and interview stakeholders to develop consensus on an operational definition for use in the evaluation. Note, though, that creating a new definition retrospectively may present an additional limitation in the interpretation of findings. Challenge: Stakeholders are interested in assessing impact quantitatively but design options are limited. Under-used but adequate evaluation designs do exist for such post- project evaluations, but they are not always well understood by some evaluators. Strategy: Be prepared, with references, to explain design options to fellow evaluators, as well as the rationale for proposing the selected design. Because comparative designs that include only post-test assessments with actual or statistically constructed comparison groups are not always well understood, the evaluator will need to develop this understanding among stakeholders to secure buy-in for their use. Note, though, that many of these design options require the team to have access to statistical support. Resources:  For a practical overview of different evaluation design options, including post-project evaluation designs, see ALNAP’s Evaluation of humanitarian action guide (Chapter 11 Evaluation designs for answering evaluation questions) (https://www.alnap.org/ system/files/content/resource/files/main/alnap-evaluation-humanitarian-action-2016. pdf).  For a discussion (with helpful examples) of studies having adequate versus plausible versus probable impact, see the 1999 International Journal of Epidemiology seminal article by Habicht et al., “Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact” (https://www.measureevaluation. org/resources/training/capacity-building-resources/m-e-of-hiv-aids-programs-in-india- english/session-5-gis-and-research-evaluation/research-evaluation/Int.%20J.%20 Epidemiol.-1999-Habicht-10-8.pdf).  For a helpful perspective to understand the necessity and utility of design compromises in certain evaluation contexts, see the 2012 AIDS journal article by Laga et al., “Evaluating HIV prevention effectiveness: the perfect as the enemy of the good” (https://www.ncbi.nlm.nih.gov/pubmed/22313952). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 19 box 5Operationally defining “migration” and “trafficking”: a post- project evaluation in Mali In Mali, international agencies and the government requested a post-project evaluation to explain why a cross-border strategy aiming to reduce trafficking of vulnerable adolescents between Mali and the cocoa plantations of Côte d’Ivoire was unsuccessful. The project was guided by the legal definition of human trafficking in Article 3 of the United Nations Protocol to Prevent, Suppress and Punish Trafficking in Persons as: “the recruitment, transportation, transfer, harbouring or receipt of persons, by means of threat or use of force or other forms of coercion, of abduction, of fraud, of deception...to achieve the consent of a person having control over another person, for the pursue of exploitation. Exploitation shall include…forced labour or services, slavery, or practices similar to slavery, servitude…” The challenge was how to operationalize “migration” and “trafficking” in contexts where family members and the young people themselves are routinely placed by intermediaries in work situations as agricultural labourers. While the anti- trafficking programme was designed to return adolescents home, most did not want to be “freed” given the enormous peer pressure to go on migration as a rite of passage. Upon return to their poor, rural villages, most adolescents returned to Côte d’Ivoire to seek work. The evaluation team used appreciative inquiry techniques – focusing on young people’s strengths in complex realities rather than their weaknesses – to operationalize “migration” and “trafficking”, informed by the legal definition. They then used these operational terms to develop culturally appropriate interventions for safe migration by young people. Source: Busza J, et al., 2004. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!20 9b. Establishing the post-project evaluation team Evaluation teams are ideally constructed to ensure diversity of skills needed to undertake the evaluation with sufficient content and contextual knowledge of the project being evaluated. During a typical final project evaluation, the evaluation team can regularly consult with project staff and partners about the project’s history, its context, and any implementation issues. However, in post-project evaluations such people are often not readily available and project content and contextual knowledge need to be defined retroactively. Challenge: An external evaluator, who is not familiar with the project’s history, implementation, and larger operating context, is hired as team lead, and has limited or no formal access to former staff. Strategy: Try to complement the team lead’s external objectivity with team members who can provide necessary project content and contextual knowledge and complementary technical skills. Ideally, these team members would be former project staff or people familiar with the organization and project. If these people are not available, invite evaluators of similar projects or working in a similar geographical area who may have a good understanding of the context. Another strategy would be to invite former staff to serve as a consultative group for the evaluation team. Challenge: While they can provide historical context and detail, team members who “own” or have “owned” the project may also be biased in their long-term recollection and interpretation of events and results. if former project staff or partners become team members, be aware of potential biases they may hold. Strategy: An important starting point is for the team lead to acknowledge that insider bias is normal and establish a sense of mutual accountability to ensure that the evaluation is as evidence-based as possible. The team lead and team members all have an important role to play in “fact checking” and insisting that available evidence (not opinion) is used to support the analyses and findings. When undertaking data collection, it is important to organize the team to avoid situations of friends interviewing friends. Likewise, the team should discuss informed consent and strategies for maintaining anonymity and confidentiality when interviewing former colleagues and friends. Similarly, the team should be cognizant of situations where team members who are former project staff may be vulnerable to risks and potential harms when interviewing former colleagues, such as when interviewing a former supervisor with whom there were tensions. In these situations, it is especially important to build multiple lines of inquiry into data collection to allow for triangulation of more subjective evidence. Identifying more objective sources of information can be helpful in this regard. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 21 box 6Establishing a common understanding of the project: the CERCA Project in the Plurinational State of Bolivia, Ecuador and Nicaragua The Community-Embedded Reproductive Health Care for Adolescents (CERCA) Project was implemented in the Plurinational State of Bolivia, Ecuador and Nicaragua (2011–2014) to test the effectiveness of contextually adapted interventions to prevent teenage pregnancies. Across the three countries, CERCA worked at multiple levels – engaging adolescents, their peers, parents and health care providers. A post-project evaluation, which assessed both the process and the impact of the project, was undertaken to examine if and how CERCA’s design, implementation and regular use of monitoring data to adapt interventions affected the results. With limited project documentation available for review and a diversity of stakeholders involved in CERCA’s design, it was important that CERCA partners participated in the evaluation not only as respondents but also as evaluators who collected and analysed data. When working with project staff to develop retrospectively a project theory of change, it became apparent that different staff understood the project’s implementation process and outcomes/impact quite differently. To address potential bias, the team needed to establish a common understanding of the project through the participatory development of a theory of change. Additionally, the team prompted and facilitated subsequent discussions on how to reduce the potential for bias during data collection and analysis. Source: Ivanova et al., 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!22 Challenge: Adolescents who were associated with the project are not considered for evaluation roles except as respondents. Although it is a recognized as good practice to engage young people in an evaluation of their own project, to ensure the inclusion of adolescents’ perspectives and to contribute to positive youth development, it is rarely done. Strategy: Thinking of adolescents as only evaluation respondents is short-sighted. If they are available, engage former project participants to work with external evaluators in as many phases of the post-project evaluation as possible. At a minimum, if financial resources and time are limited, a well-designed and facilitated meeting with former project participants to seek input on plans for the evaluation can contribute to important adjustments in later evaluation phases. Organizing a subsequent analysis meeting with these adolescents to validate the results of the evaluation would also enrich its findings. Ideally, if resources allow, work with adolescents in the design, implementation and dissemination of the evaluation. They are well placed to develop data collection tools, and their involvement in data collection can yield more truthful data as young informants may be more open to talking about issues related to sexual and reproductive health with other young people with whom they can identify, as compared with adult researchers. Note, though, that adolescents who may be engaged are not necessarily representative of the target population. Careful facilitation is needed to mitigate potential resistance by adults to young people evaluating them and their work. An evaluator may need to advocate to the funding agency to include meaningful adolescent engagement to provide additional support for their inclusion. Resources:  For guidance and facilitator manuals to create conditions for successful youth participation and to train and support young people to conduct qualitative data collection for research and monitoring and evaluation, see Explore: a toolkit for involving young people as researchers in sexual and reproductive health programmes, developed by Rutgers and International Planned Parenthood Federation (IPPF) (https://www.rutgers.international/our-products/tools/explore).  For a practical approach to training and engaging young people in evaluation see Youth participatory evaluation: strategies for engaging young people (Kim Sabo Flores, 2007), which draws on theories of children’s play, evaluation and youth development (https://www.wiley.com/en-us/Youth+Participatory+Evaluation %3A+Strategies+for+Engaging+Young+People-p-9780787983925). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 23 box 7Putting into practice adolescent-adult partnerships: an evaluation in Ethiopia, Kenya, Indonesia and Pakistan In 2015, an evaluation was carried out in Ethiopia, Kenya, Indonesia and Pakistan to explore how local organizations involved young people within their ASRH programmes. Each country’s evaluation team included one adult evaluator and three to eight adolescent and youth co-evaluators. The co-evaluators were trained on the key programme concepts, research ethics, and techniques of conducting interviews and focus group discussions, based on the Rutgers and IPPF Explore toolkit. First, the young evaluators developed their own definition of meaningful youth participation, which formed the basis of their study. Additionally, they reviewed, translated, tested and adapted tools during practice sessions. They gathered information from heads of organizations, project staff, service providers, young people engaged in the programmes, and external stakeholders using interviews and focus group discussions. The young evaluators also observed intervention activities, such as provision of services at youth centres and health clinics, provision of peer education, and community mobilization. The young people were quick to learn. Every evening, the adult researcher organized de-briefing meetings with the research team to discuss the day’s findings and analyse the data. This helped the young evaluators to see any gaps in their data that they could cover the next day. Additionally, the evaluation team tried to de-brief each organization on the key findings of the evaluation to validate conclusions and formulate recommendations together. Source: Van Reeuwijk M & Singh A, 2018. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!24 Challenge: Where appropriate, the evaluation team lead needs to ensure parental/ caregiver approval as well as adolescent safeguarding during participation in evaluation activities in a context where former project structures and protection mechanisms no longer exist. Strategy: To ensure ethical and acceptable approaches are used to safeguard participants, obtain the advice of research committees in the area. To obtain parental/ caregiver consent, provide a written summary of the evaluation, describe in what contexts and for how long the adolescent will be engaged, and seek oral or written consent from their parent/caregiver. If it is very challenging to find parents/caregivers, it may be appropriate to engage school principals or other leaders to reach parents/ caregivers. In addition, the team should agree to child safety standards, such as ensuring minors safely move between the evaluation site and home, or having adult members accompany minors if they collect data or share findings in contexts that may place them in harm’s way. If the project was implemented by a child-focused organization, use their child protection guidelines to guide the evaluation safeguarding activities. Resources:  For recommendations on how to protect and support young evaluators engaged in ASRH project evaluation from harm, see Rutgers’s Explore toolkit (Part II: Creating conditions for successful youth participation in research) (https://www.rutgers. international/sites/rutgersorg/files/pdf/Explore-instructions_web.pdf).  For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/hand le/10665/273792/9789241508414-eng.pdf?ua=1).  For ethical considerations of participatory evaluation that place evaluators in the context of community power dynamics, beyond human subject protections, see Banks et al. 2013 article in Contemporary Social Science, “Everyday ethics in community- based participatory research” (https://www.tandfonline.com/doi/pdf/10.1080/21582 041.2013.769618). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 25 9c. Addressing the lack of data availability Data availability, or the lack thereof, is one of the most common issues that evaluators confront in post-project evaluations, since reliance on existing documentation is critical to the evaluation task. Challenges, such as those below, focus on issues of finding key informants from a past project, accessing usable quantitative outcome data and process documentation, managing recall issues, and interviewing adolescents whose age and cognition levels due to maturation no longer reflect the project population. Suggestions for sampling and viable qualitative impact evaluation methods are discussed in later sub-sections. Challenge: Key project personnel are not available due to migration, transfers to new projects, or other reasons after the project has ended. Strategy: Attempt to track down telephone numbers via colleagues and other community contacts and conduct phone interviews with former staff. Ask current employers if former project staff could attend a half-day or full-day meeting during which post-project data could be collected. If this is not possible, try to interview former staff in less formal settings such as coffee shops. If this is not possible, interview staff and people within the host organization and its implementation partners who have some knowledge of the former project. Challenge: Key adolescent beneficiaries and beneficiary lists are not available. Strategy: Young people are often an especially mobile population and the evaluator may need to get creative to find young people who participated in the project. Identify existing structures such as schools or youth clubs where past beneficiaries might be located. Ask school principals or other managers to contact beneficiaries on your behalf. If structures are still active, sample from still-functioning groups formerly supported by the project – comparing those groups with groups or individuals who never participated in the project. If it is not possible to contact former beneficiaries, consider collecting information from people who are familiar with the former beneficiaries. Challenge: There are no quantitative baseline, midline or endline data and using recall information is the best option to assess change. Strategy: Use participant recall data to assess impact in areas such as personal knowledge, attitudes and behaviour changes. For example, use Likert scales with statements that ask participants to assess the extent of change between two time points. Incorporate similar-but-different statements (e.g. two different statements about the same attitude) to assess variability in responses to check recall bias. To help distinguish context from factors more directly affected by the project, refer to timelines of internal and external events during key informant interviews to remind team members of the larger context. Likewise, interviewing senior management may be useful, as people in senior management positions often operate with the larger context in mind. Triangulate recall data with other available data. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!26 box 8Finding adolescents 14 years after the project’s end: the Pro.Star Project in Jamaica Some 14 years after the Pro.Star Project in Jamaica ended, evaluators used a case control study design to retrospectively assess the effects of the project, a school-based and parent education intervention on adolescents’ transition to adulthood. Intent on finding former students to learn how they had progressed in the 14 years post-project, evaluators had to use multiple means to find former participants, including internet, social media (Facebook and WhatsApp), and word of mouth to locate students. Teachers, guidance counsellors and current students helped evaluators to locate past programme participants – reaching out, for example, to their uncles, brothers, aunts and parents. On occasion, guidance counsellors accompanied interviewers to locate persons they had trouble finding. This proved effective because people were more trusting when credible community members accompanied interviewers. Once students were located, evaluators asked them to share names and contact information of other students. Determination and patience in tracking down respondents and flexibility in fitting data collection activities into their schedules were critical. Many of the students had moved out of the community – or, in fact, out of Jamaica – so interviewers conducted some interviews by phone. The results of this outreach were impressive – it was possible to locate and interview over 60% of sampled former students (155 participants) either in person or by telephone or email interviews. Source: Brown PR & Hardee K, 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 27 box 9Using a new evaluation design when the original design is not feasible: an innovations partnership project in Sierra Leone Recognizing that adolescent pregnancy in Sierra Leone is driven more by sociocultural than service access factors, UNICEF established a partnership with Child Fund, Save the Children, the Council of Churches of Sierra Leone, Restless Development, and the Bangladesh Rural Advancement Committee. Together, these organizations implemented five pilot projects in seven districts with the aim to address underlying drivers of adolescent pregnancy in the country. The post-project evaluation aimed to foster learning about the actors, contexts, and challenges of preventing/reducing adolescent pregnancy to inform future programming. Although each partner conducted a baseline, it was not possible to use findings from these baselines as a benchmark for assessing change or impact of the pilot projects due to inconsistencies in project approaches and survey methodologies across the five pilot projects. Instead, the evaluators selected an outcome mapping evaluation approach to assess the different dimensions of change in mindsets, contexts, attitudes and perceptions of individuals and groups that influence motivations, interactions, relationships and behaviour across the five pilot projects. It also explored the contributions of the pilot projects in supporting empowerment and increasing access to and use of sexual and reproductive health services. Source: Farzaneh, 2013. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!28 Challenge: The evaluator wants to align post-project data collection with data from earlier evaluations, but survey instruments and data are not comparable across survey points (baseline, midline, endline and post-project). Strategy: Build on existing survey instruments by making small adaptations that permit comparisons of most/all key outcomes over time. While questions may differ slightly (i.e. an early instrument may assess one specific area of parent-child communication, which is replaced in a later survey instrument to explore a different parent-child communication area) qualified interpretation of findings between time points is possible. If the evolved project aim means that adjusting questions from past survey instruments is not possible, map out key thematic domains currently addressed by the project and develop new questions to evaluate the domains. New data collection using survey methods opens the possibility of using statistical methods to answer many evaluation questions (i.e. factor analysis to determine associations of exposure to the intervention and outcomes, multiple logistics regression to determine the likelihood of change or influence due to project exposure). In the end, however, it may not be possible to undertake a quantitative study that will yield expected results. In such cases seek a different approach that contains a level of rigour to allow for the establishment of plausible causality. Resources: Going forward, recent funding agency mandates, once they are widely instituted, will make datasets open source and thus available to evaluators for additional analyses to answer questions posed in a post-project evaluation. In the meantime, for open access data on adolescents living in LMICs, see the GIRL Center Adolescent Data Hub (https://www.popcouncil.org/girlcenter/research/hub). Challenge: The project is heavily focused on adolescent outcomes and neglects to collect information on other outcome areas, such as improved parenting skills or improved teacher pedagogy. Strategy: It may be useful to review training and supervision reports. As these are produced at a lower level of the reporting system, they may include information on adults and their engagement and reactions to the project. Alternatively, such information gaps can be corrected by sampling adults in new data collection. Challenge: Post-project beneficiary movements, the ageing of adolescents, and other confounding factors associated with the passage of time make it difficult to gain meaningful information that can be generalized through a population-based study. Strategy: Consider using alternative sampling to suit the evaluation aim. For example, consider snowball sampling if the evaluation aims to access people of the same social group, diversity sampling if the evaluation aims to assess differences in sub-groups, or purposive sampling if the evaluation aims to intentionally study selected groups. Resource: For an overview of different sampling strategies, see Changing Minds’ overview on choosing a sampling method (http://changingminds.org/explanations/ research/sampling/choosing_sampling.htm). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 29 box 10Re-purposing existing project data for post-project evaluation: the Baylor Pediatric AIDS Initiative in the United Republic of Tanzania In August 2008 Baylor University began a seven-year project in the United Republic of Tanzania to strengthen services for prevention, early detection, treatment and care of pediatric HIV/AIDS in the Southern Highlands and the Lake zones. A post-project evaluation was carried out to assess the extent to which the project’s goals and objectives were achieved and to provide guidance and lessons learned for future projects, in particular related to effectiveness and sustainability. As an applied research project, the evaluation team had access to ample documentation and databases from the project period. Samples for quantitative data analysis, including a comparison group, were derived from databases of activities from the relatively larger and more complete population samples of the Baylor International Pediatric AIDS Initiative that evaluators were able to analyse in the way they wanted. Although this led to potential sampling bias, it negated the need for new data collection. Primary qualitative data were collected by the evaluation team through key informant interviews, focus group discussions and observations, and were used to supplement, complement and verify quantitative analyses. Source: USAID, 2015. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!30 Challenge: The maturation effect (i.e. ageing of adolescents who were enrolled in a project at an earlier age) introduces comparability biases in post-project evaluation instruments and data, such as in their interest in sexual experimentation and awareness of the larger social environment. Strategy: Ideally, create an unexposed group for comparison. If this is not possible, design post-project data collection to allow for the comparison of older adolescents with younger ones. For example, create two age cohorts: one mimicking the initial unexposed participant group as they entered the project, and the other representing older adolescents who participated in the project. Alternatively, include recall questions for older adolescent participants to inquire about changes they have observed between pre-project and post-project time points. Challenge: Data found in annual or other external reports of the project’s process or accomplishments do not appear realistic, are not sufficiently disaggregated, or do not align with outputs and outcomes identified in the logic framework. Strategy: If data seem unrealistic, try to verify information from project monitoring data. If not available, query the organization’s senior managers to understand what was reported and confirm its reliability. For example, critical contextual information on implementation, such as a large number of new staff recruitments, may not be included in the report but is necessary to understand the findings. If output and outcome data are not available in reports or are not sufficiently disaggregated for evaluation purposes, try to work backwards by reviewing monitoring data or other internal reports. If monitoring data sets are available and data are disaggregated by sex and age, consider doing additional analyses. If disaggregated data are unavailable, undertake new post-project data collection to elicit potentially different outcomes based on adolescent age or sex. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 31 9d. Developing feasible sampling strategies Sampling poses particular challenges for post-project evaluators, ranging from selecting the site to establishing a comparison group to selecting respondents. Without project outreach structures, it may be challenging to identify adolescent beneficiaries. Time and money, of course, also direct what sample is feasible, including whether or not to include a comparison group. When working with vulnerable adolescents, an assessment of the benefits and risks of contacting such individuals should also influence sampling decisions. When constraints do exist, it is important to acknowledge potential bias in the analysis. Challenge: Site selection is done in consultation with government officials or implementing partner staff, which may introduce bias. Alternatively, sites are selected based on real- life considerations, such as security issues or ease in reaching communities. Strategy: Be aware of site selection biases and advocate for transparency in decision- making and unbiased site selection as much as possible. Try to independently assess characteristics of the selected sample, including similarities between the intervention and comparison groups regarding the adolescent population, services, and other relevant infrastructure such as schools. Secondary data may be helpful in assessing comparability. Challenge: There is no comparison group or geographic area to control for the influence of confounding external factors. Strategy: If funding and time are available, create a comparison group or area as part of new data collection processes. Alternatively, use secondary data for comparison, if they are of reasonable quality, such as national survey results or service statistics. Assess for similarities between the intervention and comparison groups regarding the adolescent population, services, and relevant infrastructure such as schools. While preferable to include a comparison group, in certain circumstances it may not be desirable for ethical or logistical reasons. Challenge: The post-project evaluation needs to sample vulnerable adolescents and their parents/caregivers, or hard-to-access respondents, such as undocumented migrants or refugees. Strategy: Firstly, it is essential to conduct an upfront ethical risk/benefit analysis using a “do no harm” framework for evaluations of projects that reached vulnerable adolescents, such as adolescents living with HIV or sex workers. In addition to assuring child/adolescent protection, it is important to train and support interviewers in protection concepts and establish processes for data collection. Pay careful attention to issues such as site selection, sampling and interviewing vulnerable adolescents. To identify and find vulnerable adolescents, work with former project staff or staff from current outreach projects who serve the vulnerable group or an external advisory group, such as a research committee, to determine an appropriate and ethical way to identify and invite respondents to participate in the evaluation. It may be possible to create a sampling frame using network mapping or snowball sampling techniques. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!32 box 11Weighing the risks and benefits of establishing a comparison group: a theoretical post-project evaluation in South-East Asia Sometimes creating a comparison group is not desirable for ethical reasons. In a post-project evaluation of a programme for sex workers in South-East Asia, a conscious decision might be made not to create a comparison group because of the following considerations:  The research might be limited in time and financial resources.  It might be difficult to identify and gain access to a comparison group.  The analyses would need to correct for selection bias, given that participation in the intervention would have been voluntary. Sex workers who had not participated would likely differ in largely unobservable characteristics from those who did.  There would be no clear indication that the intervention would continue, thus participation in the intervention would not be able to be offered to the women in the comparison group. For these reasons, the evaluation team might decide that any benefit of a control group would be outweighed by the risk of exposing women to the community as sex workers. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 33 Ensure ethical considerations for privacy are met. Social media and use of WhatsApp can allow for discrete outreach, but may lead to bias in the sample since not all adolescents have access to these technologies. Ensure that evaluation objectives include budget line items for these types of inquiries to ensure they are adequately addressed. Resource: For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/han dle/10665/273792/9789241508414-eng.pdf?ua=1). Challenge: Former project structures that facilitated the logistics of finding vulnerable adolescent beneficiaries (and their parents/caregivers for provision of informed consent) are no longer available. Strategy: Contact the project’s implementing partners to share information about the post-project evaluation and to set up meetings with former beneficiaries. Ask leaders or key contacts from former implementation partners, such as school principals, to inform parents/caregivers of an upcoming evaluation visit and seek their consent for their child’s participation. (Note: This represents an opt-out approach; that is, parental consent is assumed unless the parents/caregivers indicate they do not want their child to participate. Although this is far from ideal and introduces important ethical considerations, in specific circumstances it may be the only viable approach to ensure parents/caregivers are both informed and that they provide consent.) Set up meetings to interview stakeholders – including adolescents – in discrete ways that do not put them in harm’s way of greater visibility or discrimination. Invite parents/caregivers when inviting adolescents to be interviewed in order to secure consent, as well as to increase the response rates of both adolescents and parents/caregivers. Resources:  For practical approaches to engage young people including vulnerable adolescents, as respondents, see working paper The ethics of social research with children and families in young lives: practical experiences (http://www.younglives.org.uk/sites/www. younglives.org.uk/files/YL-WP53-Morrow-EthicsOfResearchWithChildren.pdf).  For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/hand le/10665/273792/9789241508414-eng.pdf?ua=1). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!34 9e. Bringing external context into evaluation consideration External factors influence project trajectories. It is thus important to consider other projects, policies, and macro-environmental developments that may have influenced a project trajectory. As such, a post-project evaluation often needs to fill in missing contextual information to improve evidence of effectiveness and sustainability. Challenge: information on contextual factors to help explain the nature and trajectory of change is not available in the project documentation. There is little reference to situational analyses or anticipated risks. Project indicators are not linked to these contextual factors, either within the project cycle or after the project has ended. Strategy: Seek out a variety of information sources. Consult key informants for information on contextual factors, such as new policy developments or political forces that may have affected the project, but be cautious about potential biases. If possible, review pertinent information available from media sources, including news articles and other media records and Twitter feeds. Existing documentation (i.e. synthesis reviews, policy analyses) may provide contextual information. Try to recreate internal and external event timelines with available project or implementing partner staff and stakeholders to document factors that may have influenced implementation. Consider engaging with ethnographers to visit sites and employ ethnographic methods to explore and document contextual factors. It may also be useful to employ less-traditional evaluation frameworks; for example, if sufficient time and resources are available, a realist evaluation approach, based on developing explicit understanding of how context relates to outcomes, may help guide information gathering to address these contextual issues in non-biased ways. Resources:  For an explanation of how to create an events timeline, see Better Evaluation’s overview of timelines and time-ordered matrices (http://www.betterevaluation.org/ en/evaluation-options/timelines).  For information on using realist evaluation, see Better Evaluation’s overview of realist evaluation (https://www.betterevaluation.org/en/approach/realist_evaluation). Challenge: Other development projects have launched during project implementation or between the end of the project and the post-project evaluation, complicating the plausible attribution of findings and/or influencing post-project evaluation findings. Strategy: Use evaluation reports and assessment studies from the other projects, if available, to validate anecdotal information. If evaluation reports from the other projects are available, try to map overlapping activities. Try to determine, via an events timeline discussion, to what degree projects from different organizations supported or undermined each other’s effects between the end of the project and the post-project evaluation. Interview former staff and partners to inquire if prior projects and partnerships extending before the project years were important building blocks to the project under evaluation and include this as a component of the context documentation. Methods such as contribution analysis (CA) and Qualitative Impact Protocol (QUIP) provide a systematic approach to assess causal questions and infer plausible attribution in real-life project P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 35 box 12Gaining understanding of context to explain project outcomes: the strengthening collective response of the government to end child marriage through a district level convergence approach project in India The MAMTA-Health Institute for Mother and Child, New Delhi, in partnership with district administrations, undertook a 3-year project to support a cross-departmental convergence approach to synchronize government efforts to end child marriage in Sawai Madhopur, Rajasthan, and Jamui, Bihar, in India. As a convergence or systems change approach, it was important to understand external factors that could have helped or hindered project efforts to change institutional processes to work across departments during project implementation and in the time between project closing and the post- project evaluation. The creation of an events timeline, supplemented by secondary information, was critical to interpret the significant differences in programme outcomes in the two districts. The evaluation team worked with former project staff to create a timeline of internal and external project events, such as changes in child marriage policy at national and state levels and overlap with a state-sponsored programme to end child marriage in one state (Rajasthan) that began mid-project and worked across service departments (a similar approach to the project of interest). This information led to new questions about environmental forces, and the team subsequently supplemented and confirmed information about these events through a review of policy documents and stakeholder interviews. Source: Chandra-Mouli et al., 2018. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!36 evaluations in complex settings. Be willing to state that it is not possible to determine plausible attribution. Resources:  For information on CA – a qualitative technique that guides managers, researchers and policy-makers to a consensus on the contribution their project has made to particular outcomes through better understanding of why results were achieved (or not) and the roles played by the project and other factors – see Better Evaluation’s overview of contribution analysis (www.betterevaluation.org/en/plan/approach/ contribution_analysis).  For information on QUIP – a qualitative technique to evaluate a project implemented in a context of complexity that helps explain how changes can be attributed to different stakeholders or events – see the Assessing Rural Transformations’ QUIP guidelines for field use (http://www.bath.ac.uk/cds/projects-activities/assessing-rural- transformations/documents/complete-quip-guidelines.pdf). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 37 9f. Assessing sustainability and emerging and/or unanticipated outcomes Sustainability is almost always a desired impact and often a funding agency requirement, and post-project evaluations are ideally suited to assess this. Post-project evaluation also provides the possibility to explore whether any other outcomes occurred after the close of the project (Zivetz et al., 2017). These outcomes can be distinguished between unanticipated outcomes (arising from assumptions made in the theory of change) and emerging outcomes (due to efforts and resources of participants and partners after the project ended). Challenge: Sustainability outcomes are not well-defined in project documents but are a post-project evaluation aim. Strategy: Discuss with available staff whether exit or graduation strategies – planned project efforts to close out or transfer implementation support to another entity – existed in the project under evaluation, and what benchmarks or indicators signalled the end of, phasing out of, or reduction of project support. Work with available project staff and other stakeholders to create an operational definition of sustainability that can be used in planning and conducting the evaluation. One approach is to develop a hierarchy of sustainability (i.e. differentiations between sustained project inputs, outcomes or impacts) to define the original project’s sustainability aims. This will simplify the process of determining which methods can be employed to assess sustainability (e.g. observation to determine if activities and use of project materials continue in the post-project context). Resources:  For an illustrative example of a sustainability hierarchy that can guide evaluator-staff discussions on developing an operational definition of sustainability for a project under evaluation, see Valuing Voices Building the evidence base for post-project evaluation (also see Box 13) (http://valuingvoices.com/wp-content/uploads/2013/11/The-case- for-post-project-evaluation-Valuing-Voices-Final-2017.pdf).  For a practical overview of issues in evaluating the sustainability of health projects (including definitions and types of sustainability, specifications and measurements of dependent variables, definitions of independent variables or factors that influence sustainability, and suggestions for designs for research and data collection), see “An agenda for research on the sustainability of public health programs” (https://www. ncbi.nlm.nih.gov/pmc/articles/PMC3222409). Challenge: Anecdotal information indicates that unanticipated and/or emerging outcomes have occurred and need to be part of the evaluation frame to document lessons learned for future projects. Strategy: Several participatory evaluation techniques (listed below) have been developed to address such challenges regarding the wider impact of a project and plausible causality with a degree of rigour. Data collection and evaluation tools should be designed to search explicitly for unanticipated negative outcomes. While positive outcomes are easily shared by respondents, negative outcomes may not be captured T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!38 box 13Defining sustainability of an illustrative ASRH project using a hierarchy of results framework In this example, evaluators and stakeholder developed a hierarchy of sustainability to guide evaluation of an after-school club intervention for very young adolescents. Project stakeholders first defined sustainability of outputs, outcomes and impacts (y-axis). They also differentiated between types of impacts (x-axis) (i.e. whether project-expected achievements were sustained or whether broader sustainability impacts emerged that could be discerned only over a longer timeframe; that is, after a project ends). In addition, Zivetz et al. note there are two broader areas of sustainability to assess: emerging outcomes consider how participants used their own means to carry project initiatives forward, which could produce learning on incentivizing sustainability. This is in contrast to unanticipated outcomes, which are related to the project’s theory of change and may reveal the extent of and reasons why assumptions or objectives deviated from what was anticipated in the project design. Sustained impacts Emerging impacts N/A ASRH and gender equality improved within school structures Teachers are more comfortable teaching sexuality education. Adolescent financial literacy improved Sustainability of impacts Parent-teacher-student committees are maintained, with support from the national government After-school clubs are maintained with continued active engagement of adolescents and teachers Adolescents do small activities for cash to support continuation of the clubs Sustainability of outcomes Parent-teacher-student committees are in place and trained After-school clubs are in place and operational Adolescents and teachers are trained as facilitators Sustainability of outputs Source: Adapted from Zivetz et al., 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 39 box 14Intentionally exploring negative unintended project impacts: a family planning/intimate partner violence project in the Republic of Guinea In 2014, a facility-based youth family planning project in the Republic of Guinea decided to integrate intimate partner violence (IPV) screening as part of a comprehensive package of health services which included care for women who had experienced IPV. This involved training existing family planning staff on using an IPV screening checklist and procedures for referral to IPV services. While programme performance statistics and client interviews showed high levels of IPV screening acceptability and referral uptake, a post-project evaluation of the implementation process that explicitly sought to ascertain unintended consequences revealed that facility staff experienced a high degree of emotional distress due to the “heavy” nature of young women’s IPV stories and the staff’s inability to provide additional support on site. Furthermore, the evaluation also revealed that referrals were often incomplete, the IPV services at the referral sites were often inadequate, and that young clients were reluctant to engage with the referral organizations. Given these findings, the post-project evaluation recommended providing on-site psychosocial counseling for women who experienced IPV, as well as establishing regular group support sessions for staff to deal with the emotional burden of IPV screening and care. Source: Samandari G, et al., 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!40 unless they are intentionally and sensitively questioned and unless respondents are confident their responses will be confidential. Resources:  Outcome harvesting is a participatory process of collecting evidence of what has changed and then, working backwards, determining whether and how an intervention has contributed to these changes. Outcome harvesting has proven to be especially useful in complex situations. For more information, see Better Evaluation’s overview, Outcome harvesting (http://www.betterevaluation.org/en/plan/approach/outcome_ harvesting).  Most significant change (MSC) is a participatory process of engaging stakeholders in data collection and analysis to identify the most significant changes they experienced due to the project. Project stakeholders decide the sorts of changes (impact) that they observe, and analysis of multiple stories can confirm expected outcomes and reveal unanticipated and emerging outcomes. For more information, see Monitoring and Evaluation News’ overview on MSC (http://mande.co.uk/special-issues/most- significant-change-msc/).  Ripple effect mapping (REM) is a participatory approach that works somewhat like a group mind mapping, systematically engaging participants in a visual process to identify outcomes, impacts and unintended consequences, and determine next steps. For more information, see the Journal of Extension 2015 article “Using ripple effect mapping to evaluate program impact: choosing or combining the methods that work best for you” (https://joe.org/joe/2015april/tt1.php). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 41 10 The literature review conducted for this guidance indicated that several cross-cutting project principles relating to adolescents’ particular vulnerabilities and resilience, including principles of positive youth development (PYD), gender equality and human rights, and poverty alleviation, are often ignored or not explicitly discussed in project and evaluation reports (Denno et al., 2015). These cross-cutting factors influence project trajectories as they relate to adolescent health outcomes, and as they relate to conditions of sustainable development for young people and their communities. When these cross-cutting issues are applied to adolescent development projects, they alter the way in which the projects are designed, implemented, monitored and evaluated (UNICEF, 2012, p. 13). Post-project evaluation can play a role in highlighting the importance of such principles and concepts by analysing conceptual gaps and promoting the inclusion of these issues in new projects. For example, if a comprehensive sexuality education project is targeting only older adolescents or only girls, the principles review would create space to address the needs of younger adolescents and boys, as well. 10a. Positive youth development Tip: Assess the extent to which a project contributes to the promotion of PYD. Positive youth development offers a holistic lens upon which to assess projects. PYD engages young people along with their families, communities and governments, to support the empowerment of young people to reach their full potential (see Youth Power’s Positive youth development measurement toolkit below). Positive youth development approaches aim to build skills, assets and competencies; foster healthy relationships; strengthen a supportive environment; and transform systems to be more youth-friendly and to be accountable for young people. Central questions to consider related to PYD during a post-project evaluation include: Assessing cross-cutting project principles T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!42  Assets: Did the project design aim to provide adolescents with the necessary resources and support them to build skills and competencies to achieve ASRH project outcomes?  Agency: Did the project design aim to support adolescents to employ their assets to set their own goals and to make their own decisions about their lives to achieve desired outcomes?  Contribution: Were adolescents meaningfully engaged as a source of change for their own and their community’s positive development?  Enabling environment: Did the project create a more enabling environment for adolescents that develops and supports their assets, their agency, their contribution and their access to services and opportunities? Does it strengthen their ability to avoid risks and to stay safe and secure?1 Resource: For information on how to measure PYD, see Youth Power’s Positive youth development measurement toolkit (2017) (http://www.youthpower.org/resources/positive- youth-development-measurement-toolkit). 1 “Environment” includes social (relationships with peers and adults); normative (attitudes, norms and beliefs); structural (laws, project services and systems); and physical (supportive spaces) factors. Figure 3. Framework of positive youth development illustrating intersections leading to healthy adolescence Source: Adapted from Youth Power, 2017. Enabling environment • Bonding • Opportunities for pro-social involvement • Support • Pro-social norms • Value and recognition • Youth-responsive services • Gender-responsive services • Youth-friendly laws and policies • Gender-responsive laws and policies • Physical safety • Psychological safety • Positive identity • Self-efficacy • Goal-setting/ability to plan ahead • Perseverance and diligence • Positive beliefs about the future Agency • Youth engagement Contribution Healthy, productive and engaged youth Assets • Training • Formal education • Interpersonal skills (social and communication skills) • Higher-order thinking skills • Recognizing emotions • Self-control • Academic achievement P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 43 10b. Promotion of gender equality and human rights Tip: Assess the extent to which the project contributes to the promotion of gender equality and upholding of human rights. Gender norms, roles and relations influence people’s health, well-being and opportunities; as such, projects should understand and acknowledge the influence of gender and the impact of gender inequality on adolescents’ lives. Projects can be distinguished according to the extent to which they address gender. Gender-unequal projects perpetuate gender inequality by reinforcing unbalanced norms, roles and relations. Gender-blind projects ignore gender norms, roles and relations. Gender-sensitive projects consider gender norms, roles and relations, but do not address inequality generated by unequal norms, roles or relations. Gender-transformative projects consider gender norms, roles and relations and address the causes of gender-based health inequities (WHO, 2011). Likewise, projects should employ human rights-based approaches to health, which focus attention and provide strategies and solutions to redress inequalities, discriminatory practices (both real and perceived) and unjust power relations, which are often at the heart of inequitable health outcomes (WHO, 2019). Central gender equality and human rights questions to consider during a post-project evaluation are:  Design: Was the project conception gender-differentiated? Was a situational analysis conducted to understand differences due to gender inequalities? Were other key differences identified (i.e. class, disability, ethnicity, race, poverty and sexual orientation) and an assessment made of how the project would affect adolescents’ respective circumstances, status, opportunities and resources with regard to health and well-being?  Adolescent participation throughout the project: What were the respective roles of adolescents in designing the project? To what extent and with what level of equality were adolescents involved in implementing core project elements, such as having leadership roles in activity implementation? To what extent and with what level of equality were adolescents involved in reviewing progress towards project outcomes?  Monitoring: Do reporting systems disaggregate by age and sex of adolescents? Do reporting systems capture activities of groups beyond adolescents (i.e. parents, teachers, and others) to foster enabling environments? 10c. Poverty alleviation Tip: Assess the extent to which the project contributes to poverty alleviation, as outlined in the Global Strategy for Women’s, Children’s and Adolescents’ Health and the Sustainable Development Goals (SDGs). The vision of the Global Strategy for Women’s, Children’s and Adolescents’ Health is that by 2030, there will be a world in which every woman, child and adolescent in every setting realizes their rights to physical and mental health and well-being, has social and economic opportunities, and is able to participate fully in shaping sustainable and prosperous societies. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!44 Central questions to include in a post-project evaluation (drawn from the Global Strategy for Women’s, Children’s and Adolescents’ Health 2016–2013):  Was the project situated within a poverty-oriented structural environment, such as a national poverty reduction or SDG strategy?  Did the project embrace one or more guiding principles of the SDGs, reflecting understanding of the complexity of action needed to achieve the SDGs: be aligned with development effectiveness and humanitarian norms, and be human rights-based, gender-responsive, partnership-driven, people-informed and accountable?  Did the project contribute to national or subnational poverty-reducing effects? That is, did it contribute to ending preventable deaths, ensuring health and well-being, or expanding enabling environments for young people? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 45 11 To close, it is important to reiterate that post-project evaluation is not the ideal form of project evaluation, with the exception possibly being for the assessment of post-project sustainability and unanticipated and emerging impacts. When post-project evaluation is the only choice for project evaluation, then there is a need to ensure that it is done with as much rigour as possible and that the results and lessons learned are shared. To this end, there is a need to encourage more systematic reflection on the utility and application of post-project evaluation for:  learning about what works and why;  ensuring accountability to key stakeholders including adolescents;  exploring sustainability; and  documenting unanticipated and emerging impacts. 1. Build the monitoring and evaluation capacity of project staff in project and funding agencies to create a broader base of evaluation thinking and action With greater awareness of the importance of evaluation in project design, implementation and assessment of progress and impact and greater expertise in conducting evaluations, project and funding agency staff will develop evaluative thinking and begin to see new opportunities for the use of systematically collected data throughout a project – not only after it ends. How can evaluators, programmers and funders support post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!46 Additionally, evaluators must systematically determine appropriate and feasible roles of adolescents in evaluation and budgets for the resources and space to meaningfully engage young people. Evaluators and project directors can serve as advocates and actors for PYD, including through their engagement in evaluation. 2. Build ASRH project evidence To move the field forward, evaluation activities need to be included and budgeted in ASRH project proposals. When a post-project evaluation is determined to be the optimal option, it is important to understand and be able to speak to its limits as well as opportunities. To facilitate post-project evaluations, we encourage projects to retain data, in addition to reports, after they end. Additionally, post-project evaluations are not easily found in the public domain. Collaborative efforts are needed to disseminate and improve the sharing and use of findings from post- project evaluations. Budget lines should be included to allow time to write up findings of post- project evaluations and to publish them in peer-reviewed journals and popular newsletters. Organizations, including funding agencies and evaluation clearinghouses (see Appendix C), should systematically post evaluation reports, including post-project evaluation reports. Given the potential of post-project evaluation to improve learning and influence future project design and implementation, this guidance seeks to provide ideas and inspiration for future post-project evaluators and funders. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 47 References This list includes source documents that informed the main narrative. It does not include resources already listed below strategies in sections 8 in the main narrative or articles retained for the literature review, which are listed Appendix A. Books Bamburger M, Rugh J, Mabry M. Realworld evaluation: working under budget, time, data, and political constraints (second edition). Thousand Oaks (CA): Sage Publishers; 2012. Bilateral Funding agency documents/evaluation guidelines Austrian Development Agency. Guidelines for project and programme evaluations (final draft translated into English). Vienna: Austrian Development Cooperation/Evaluation Unit; 2009. JICA. Evaluation guidelines 2004. Tokyo: Japan International Cooperation Agency; 2004 (https://www.jica.go.jp/english/our_work/ evaluation/tech_and_grant/guides/pdf/guideline03-02.pdf, accessed 6 May 2019). Journal articles Barua A & Chandra-Mouli V. The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation. International Journal of Adolescent Medicine and Health. 2016;29(6) (doi: https://doi. org/10.1515/ijamh-2016-0024, accessed 6 May 2019). Busza J, Castle S, Diarra A. Trafficking and health. BMJ. 2004; 328(7452): 1369-1371. Chandra-Mouli V, Plesons M, Barua A, Sreenath P, Mehra S. How can collective action between government sectors to prevent child marriage be operationalized? Evidence from a post-hoc evaluation of an intervention in Jamui, Bihar and Sawai Madhopur, Rajasthan in India. Reproductive Health. 2018;15:118. Chandra-Mouli V, Svanemyr J, Amin A, Fogstad H, Say L, Girard F et al. Twenty years after International Conference on Population and Development: where are we with adolescent sexual and reproductive health and rights? Journal of Adolescent Health. 2015;56(1):S1– S6. Denno DM, Hoopes AJ, Chandra-Mouli V. Effective strategies to provide adolescent sexual and reproductive health services and to increase demand and community support. Journal of Adolescent Health. 2015;56(1):S22–S41. Ivanova O, Cordova-Pozo K, Segura ZE, Vega B, Chandra-Mouli V, Hindin MJ et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Evaluation and Program Planning. 2016;58:98–105. (https://doi.org/10.1016/j. evalprogplan.2016.06.007, accessed 6 May 2019). Ryan R, Wortley L, Ní Shé É. Evaluations of post-disaster recovery: a review of practice material. Evidence Base. 2016;1(4):1–33 (https:// www.anzsog.edu.au/resource-library/research/evaluations-of-post-disaster-recovery-a-review-of-practice-material, accessed 6 May 2019). Samandari G, Delamou A, Traore P, et al. Integrating intimate partner violence screening and counseling in a family planning clinic: Evaluation of a pilot project in Conakry, Guinea. African Journal of Reproductive Health. 2016; 20(2):86-93. Van Reeuwijk M & Singh A. Meaningful Youth Participation as a way to achieving success - Results from operational research on meaningful youth participation in a large-scale youth SRHR program in Africa and Asia. Canadian Journal of Children’s Rights. 2018; 5(1). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!48 Technical documents Bamburger M. Conducting quality impact evaluations under budget, time and data constraints. The World Bank Independent Consulting Group. Washington (DC): World Bank; 2006. Brown PR & Hardee K. Transitions to Adulthood in Jamaica: Retrospective Study of Youth.now Participation. Study Report. Kingston, Jamaica and Morristown, New Jersey: PEY & Associates and What Works Association; 2017. Farzaneh N. An evaluation of teenage pregnancy pilot projects in Sierra Leone. New York: UNICEF; 2013 (https://www.unicef.org/ evaldatabase/files/SierraLeone_2013-001_Evaluation_of_Teenage_Pregnancy_Pilot_Projects_in_Sierra_Leone. pdf, accessed 6 May 2019). Mhando NE, Shukia R, Mkumbo K. United Republic of Tanzania/UNICEF evaluation of TUSEME Programme in Tanzania. New York: UNICEF; 2015 (https://www.unicef.org/evaldatabase/files/TUSEME_REPORT_-_Final_(Tanzania_2015-008).pdf , accessed 6 May 2019). Morra-Imas LG & Rist RC. The road to results: designing and conducting effective development evaluations. Washington (DC): International Bank for Reconstruction and Development/World Bank; 2009 (https://openknowledge.worldbank.org/bitstream/ handle/10986/2699/52678.pdf, accessed 6 May 2019). OECD-DAC. Glossary of key terms in evaluation and results-based management. Paris: OECD-DAC; 2010. Puri J, Aladysheva A, Iversen V, Ghorpade Y, Brück T. What methods may be used in impact evaluations of humanitarian assistance? Bonn: The Institute for the Study of Labor (IZA); 2015. Social Cops Academy. The ultimate guide to effective data collection [Using data intelligence for better decision] [website]; 2017 (https://www.udemy.com/user/socialcops/, accessed 6 May 2019). UNICEF. Application of a human rights based approach and gender mainstreaming strategy for work with adolescents in Latin America and the Caribbean. Panama City, Panama: UNICEF Regional Office for Latin America and the Caribbean; 2012. USAID. Evaluation: Performance evaluation of the strengthening pediatric HIV and AIDS services in Tanzania program. Washington DC: USAID; 2015. WHO. Key Issues in the implementation of programmes for adolescent sexual and reproductive health. Geneva: World Health Organization; 2004 (https://apps.who.int/iris/bitstream/handle/10665/69695/WHO_FCH_CAH_04.3_eng.pdf;jsessionid =3C4E0CBAD79AED0C8573FC3CBB6D782D?sequence=1, accessed 6 May 2019). WHO. Gender mainstreaming for health managers: a practical approach. Facilitators’ guide. Participant’s notes. Geneva: World Health Organization; 2011 (https://www.who.int/gender-equity-rights/knowledge/health_managers_guide/en/, accessed 6 May 2019). WHO. Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents. Geneva: World Health Organization; 2018 (http://apps.who.int/iris/bitstream/handle/10665/273792/9789241508414-eng.pdf?ua=1, accessed 6 May 2019). Zivetz L, Cekan, J, Robbins K. Building the evidence base for post project evaluation: case study review and evaluability checklists [a report to the Faster Forward Fund] [website]. Valuing voices; 2017 (http://valuingvoices.com/building-the-evidence-base-for-post- project-evaluation-a-report-to-the-faster-forward-fund/, accessed 6 May 2019). Website articles/blog posts Mindtools. Post implementation reviews: making sure that what you delivered actually works. 2019 (https://www.mindtools.com/ pages/article/newPPM_74.htm, accessed 6 May 2019). OECD. DAC Criteria for evaluating development assistance. Fact sheet. 2000 (http://www.oecd.org/dac/evaluation/49756382.pdf, accessed 6 May 2019). WHO. Gender, equity and human rights. 2019 (https://www.who.int/gender-equity-rights/understanding/human-rights-definition/ en/, accessed 6 May 2019). World Bank. Monitoring and evaluation for better development results. Feature story, 2013 (http://www.worldbank.org/en/news/ feature/2013/02/14/monitoring-and-evaluation-for-better-development-results, accessed 6 May 2019). Youth Power. Positive Youth Development (PYD) Framework. 2017 (https://www.youthpower.org/positive-youth-development-pyd- framework, accessed 15 July 2019). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 49 Appendix A: Analysis report – review of post-project evaluations of ASRH projects To understand ASRH post-project evaluation rationales, challenges and solutions, a review of peer-reviewed and grey literature articles and post-project adolescent and AYSRH evaluation reports was conducted. 1. Methodology to identify evaluation reports retained for this review Using three levels of criteria to search seven evaluation databases, 99 evaluation reports were identified using first-level inclusion criteria (project evaluations, completed between 2010 and 2016, and operating in LMICs). After applying the second-level criteria (ASRH projects and programmes), 41 reports were retained. Each report was then examined using third-level criteria (post-project evaluations), yielding six reports. WHO provided seven additional reports and peer-reviewed articles, which brought the total number to 13 listed at the end of the report. Evaluation databases searched and results  USAID Development Experience Clearinghouse (DEC) (17 AYSRH evaluations identified of which 2 were post-project).  UNICEF evaluation database (24 identified of which 4 fit inclusion criteria).  Active Learning Network for Accountability and Performance in Humanitarian Action (none identified).  Canadian International Development Agency (none identified).  European Union Commission EuropeAid (none identified).  Independent Evaluation Group at the World Bank (none identified).  Poverty Action Lab (none identified). Inclusion criteria to search databases  First-level criteria: programme evaluations posted from 2011–2017 from LMICs (99 reports).  Second-level criteria: AYSRH projects and programmes (41 reports).  Third-level criteria: evaluations that were conducted post-project (6 reports). NB: Search criteria varied by database search functions, but included: Adolescent/Youth/ Adolescent sexual and reproductive health/Adolescent sexual health/Impact assessment/Impact evaluation/Implementation reviews). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!50 Notes on the process 1. Posting of evaluations on all databases is encouraged by funding agencies and clearinghouses but not required; thus, the evaluation report universe is a self-selected one. Only UNICEF applies an additional criterion for posting the evaluations; they must adhere to a specified level of rigour. 2. The search yielded few reports and highlighted a number of database search issues.  Different search terms included in the seven evaluation websites made the search imprecise. Only some of the websites had “youth and adolescent health” as a search term. None of the websites had “post-project evaluation” as a search term.  A total of 17 articles related to youth and adolescent health were identified on USAID’s DEC. However, only two of these 17 articles were post-project evaluations. Additionally, most of the excluded reports were focused on issues other than AYSRH.  A total of 24 articles were identified on UNICEF’s evaluation database from 2011–2017 by using “youth” and “adolescent” as search terms. However, only four of these 24 articles were relevant to sexual and reproductive health, and three of these four articles were post-project evaluations.  No articles that met the inclusion criteria were identified on the evaluation database of the Active Learning Network for Accountability and Performance in Humanitarian Action.  No articles that met the inclusion criteria were identified on the other databases and websites, including those of the Canadian International Development Agency, the European Union Commission, the Independent Evaluation Group of the World Bank and the Poverty Action Lab. 2. Key findings Why was the post-project evaluation done? The rationale for conducting the post-project evaluations varied across the reports. These included: to measure effectiveness, draw out lessons learned, understand how to improve the intervention for future projects, and to gauge the sustainability of the AYSRH project. Who commissioned and conducted the evaluation? The funding agency/technical agency – and sometimes the main implementing organizations – commissioned the evaluations. They then typically engaged external evaluators and/or research/evaluation firms to conduct the evaluations. How were project stakeholders, including young people, engaged? The engagement of stakeholders, including young people, varied across the reports. Eight evaluations reported engaging their stakeholders only as respondents. Two evaluations engaged adult stakeholders in the evaluation process as part of the evaluation team. Three evaluations engaged adolescents in the evaluation process, mostly through consultative meetings and – in one case – as co-evaluators throughout the evaluation process. What were the evaluation approaches? All the evaluations used qualitative approaches to collect their primary data, although some used a mixed methods approach. Only two evaluations employed pre-intervention/post-intervention measures. Several used atypical evaluation approaches including theory-driven evaluation and outcome mapping. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 51 Were there any reflections by evaluators on barriers and limitations of post-project evaluations? Evaluators across all 13 reports identified similar limitations. Recurring themes included: time constraints, small sampling size and lack of primary or secondary data to compare with a baseline or to verify results. In most of the reports, evaluators were not integrated into the project design process from the start, but were instead identified and engaged once the project had already ended. In many of the reports, limited availability of information and time constraints increased the pressure on evaluators, who had to balance the processes of conducting an evaluation and ensuring the quality of the data collection process. Were there any reflections by evaluators on the utility of post-project evaluations? Only a few evaluators explicitly discussed and explained the utility of post-project evaluations. None of the evaluators advocated for the need for this type of evaluation. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!52 Post-project evaluations included in this analysis Barua A & Chandra-Mouli V. The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation. International Journal of Adolescent Medicine and Health. 2016;29(6) (doi: https://doi.org/10.1515/ijamh-2016-0024, accessed 6 May 2019). Bernstein R, Harkins J, Mwanza M, Nandi S. Performance evaluation of the Strengthening Pediatric HIV and AIDS Services in Tanzania Program. USAID; 2015 (http://pdf.usaid.gov/pdf_docs/PA00KJFF.pdf, accessed 7 May 2019). Chandra-Mouli, V, Barua, A, Igras, S. Strengthening collective response of the government to end child marriage through a district level convergence approach in Jamui, Bihar, and Sawai Madhopur, Rajasthan, India. Geneva: WHO; 2016. Collins T, Yusuf Y, Tesfahun H, Dejene S. A rights-based approach to adolescent and youth development, Ethiopia. End of programme evaluation 2013. London: HLSP; 2013 (https://www.unicef.org/evaldatabase/files/Ethiopia_2013-023_ Evaluation_of_Adolescent_Programme-Final_Report_RNE.pdf, accessed 7 May 2019). Farzaneh N. An evaluation of teenage pregnancy pilot projects in Sierra Leone. UNICEF; 2013 (https://www.unicef.org/ evaldatabase/files/SierraLeone_2013-001_Evaluation_of_Teenage_Pregnancy_Pilot_Projects_in_Sierra_Leone. pdf, accessed 6 May 2019). Gotsadze T, Sharipova B, Ashurov S. Youth friendly health services program in Tajikistan 2006–2013: program evaluation report. UNICEF; 2014 (https://www.unicef.org/evaldatabase/files/Tajikistan_Youth_Friendly_Health_ Services_program_in_Tajikistan.pdf , accessed 7 May 2019). Ivanova O, Cordova-Pozo K, Segura ZE, Vega B, Chandra-Mouli V, Hindin MJ et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Evaluation and Program Planning. 2016;58:98–105. (https://doi.org/10.1016/j. evalprogplan.2016.06.007, accessed 6 May 2019). Mhando NE, Shukia R, Mkumbo K. United Republic of Tanzania/UNICEF evaluation of TUSEME Programme in Tanzania. UNICEF; 2015 (https://www.unicef.org/evaldatabase/files/TUSEME_REPORT_-_Final_(Tanzania_2015-008).pdf , accessed 6 May 2019). Rijsdijk L, Bos A, Ruiter R, Leerlooijer J, de Haas B, Schaalma H. The world starts with me: a multilevel evaluation of a comprehensive sex education programme targeting adolescents in Uganda BMC Public Health. 2011;(11):334. Swartz S, Deutsch C, Makoae M, Michel B, Harding J, Garzouzie G et al. Measuring change in vulnerable adolescents: findings from a peer education evaluation in South Africa. Journal of Social Aspects of HIV/AIDS. 2012;9(4):242–254. True Panacea LLC, Endres K, Rao P. ARC TWC evaluation: Guyana and Tanzania comparative report. USAID; 2010 (http:// pdf.usaid.gov/pdf_docs/PDACR615.pdf, accessed 6 May 2019). van Belle S, Marchal B, Dubourg D, Kegels G. How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC Public Health. 2010;(10):741. Van der Geugten J, van Meijel B, den Uhl MH de Vries N. Evaluation of a sexual and reproductive health education programme: students’ knowledge, attitude and behaviour in Bolgatanga Municipality, Northern Ghana. African Journal of Reproductive Health. 2015;19(3):126–36. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 53 Appendix B: Post-project evaluation evaluability checklist An evaluability assessment examines “the extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD DAC, 2010, p. 21). Evaluability assessments help determine if an evaluation is worthwhile to conduct in terms of anticipated benefits and costs and how it can be designed for maximum utility. The purpose of this checklist is to identify gaps in the data available and other shortcomings in the resources required to conduct a post-project evaluation. Knowledge of these limitations can highlight aspects of the evaluation that require mitigation if a post-project evaluation is to move forward. Note there is no minimum threshold of criteria required for proceeding with a post-project evaluation. Deciding whether or not to initiate a post-project evaluation will depend on the context of individual projects. Organizational considerations There is a clear purpose for conducting the post-project evaluation that is understood by all key stakeholders. Typical purposes include: Project and organizational learning: evidence of overall effectiveness evidence of unanticipated or emerging impacts1 evidence for improving project quality going forward or to scale up successful efforts evidence of the sustainability of outcomes/impacts. Promoting accountability for beneficiaries, partners, funders, and other stakeholders. Justifying future funding. The perceived utility of the results warrant the investment in a post-project evaluation. There is a clear opportunity for the results of the post-project evaluation to inform policy or programmatic decision- making or other worthwhile use. There is an explicit plan for the dissemination of findings, including to project participants and partners, and a pathway for organizational learning. There is strong leadership support for the post-project evaluation, as well as for evidence-informed decision-making overall. There is clarity and agreement on evaluation roles and responsibilities among the funder, implementing organization, project staff, other partners and the evaluator(s). 1 Zivetz, Cekan, Robbins (2017) distinguish between unanticipated outcomes (arising from assumptions made in the theory of change) and emerging outcomes (due to the efforts and resources of participants and partners after the project stopped). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!54 information availability There is basic documentation about the project available, which may include any of the following: Original project proposal Situation/context analysis Needs assessment/gap analysis Project description (including key contextual factors) Project design or plan Theory of change or logic model/log frame Monitoring plans Evaluation framework or results framework Work/implementation plans Progress reports or process reviews Endline or closeout reports Exit strategy documentation Formative/process/implementation evaluation reports Can the authors be contacted if more information or clarification is needed? There is a clear statement of the problem that the intervention was designed to address. There are original and evolved project goals and/or objectives. There is an explicit and coherent project theory or theory of change, or one can be reconstructed given the project documentation available. Inputs, activities, outputs, outcomes, impacts are specified and causal linkages are clearly indicated. Key stakeholders agree on the intended outcomes and impacts. It is possible to identify which aspects of the theory of change should be the focus of the post-project evaluation. It is possible to determine measurable outputs, outcomes and key indicators from the theory of change. The approximate start and end dates of the project are known. Individuals with some experience with, or knowledge about, the project are available for consultation. Individuals who can verify the fidelity of implementation to the original project model or any subsequent changes. Individuals who can comment on contextual factors present during and after the project that may have affected implementation and subsequent impacts. Information is available on how the intervention was aligned with other interventions and strategies at the time of operation. There are existing raw or analysed secondary project data available (baseline, midline or other) which may include: Monitoring data Activity records Attendance records Participant records Narrative reports Internal reviews Satisfaction surveys Outputs from project activities, e.g. visual outputs of participatory activities, other developed materials, tools, etc. Population-level data, national statistics Other (list here): There are data available for all sites. There are data available to demonstrate change on key outcomes. The sampling process used is clear. Project participation is clearly defined and distinguishable from nonparticipation. The data can be disaggregated by age, sex, gender, ethnicity, site, etc. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 55 The quality of the existing data appears acceptable for use regarding reliability and validity. Previously used data collection instruments are available for use. Information about other modifying factors that might have interacted with, influenced and contributed to the project’s outcomes post-completion is available (i.e. other development interventions in the area, social, political or economic trends, or climate, conflict, displacement, migration). Data collection Key stakeholders agree on the primary evaluation questions to be addressed in the post-project evaluation. Obtaining primary project data is still feasible even though the project has ended. There is a record of who was involved in what project activities, when and to what extent. Names, roles and contact lists of implementing actors, including project and organizational staff, funders and consultants, are available and potentially contactable. Names and contact lists of beneficiaries are available and potentially contactable. Knowledge of who was excluded (intentionally or unintentionally), and why. Key beneficiaries are willing and able to participate. An appropriate comparison group can be identified and contacted. Project sites are available for data collection where: There was a high level of programming during the project. The site(s) is reflective of the demographics and geography of some or all of the project intervention sites. In or out migration of the site is minimal, or is representative of the project sites overall. There has been minimal activity on the part of other development agencies in the same sectors in the years since the project ended (or something is known about these interventions). Any legacy effects of the implementing agency can be distinguished from the project under evaluation (in the case where a new project followed the one under evaluation). More than one data source (primary or secondary) is available to facilitate use of a mixed methods approach and the triangulation of multiple lines of evidence. The evaluation questions: Align with the areas of interest identified in the theory of change. Are realistic given the amount of primary and secondary data available and resources available to do the post- project evaluation. Explore alternative explanations for observed outcomes or impacts. Evaluation resources There are sufficient resources to employ the desired post-project evaluation methodology. Budget Staffing Equipment Other (list): The implementing organization has the capacity to assist the evaluator with data and other information requests related to the post-project evaluation. Staff and key stakeholders with the necessary skills are likely to be present. Staff and key stakeholders with the necessary history are likely to be present. There is an opportunity for staff and key stakeholders to participate in the design of the post-project evaluation and formulation of evaluation questions. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!56 Timing There is sufficient time available to conduct the post-project evaluation as designed. The results will be available in a timely way for decision-making. The evaluation can be timed to align with the original project end, particularly if seasonality may influence project outcomes and impact (i.e. agricultural and education cycles). Unique post-project evaluation considerations The evaluator can defend their choice of methods for the post-project evaluation. Primary intended users understand the limitations of the evaluation evidence and will view the results as credible. Stakeholders agree on the level of evidence required. Sources Bureau of Conflict and Stabilization Operations (CSO). CSO evaluability assessment checklist (working draft). Washington (DC): United States Department of State; n.d. Davies R. Planning evaluability assessments: a synthesis of the literature with recommendations. Report of a study commissioned by the Department for International Development. London, UK: DFID; 2013 (https://www.gov.uk/ government/uploads/system/uploads/attachment_data/file/248656/wp40-planning-eval-assessments.pdf, accessed 7 May 2019). JICA. Thematic evaluation: analysis for enhancing evaluability of JICA’s cooperation programs final report. Tokyo: Japan International Cooperation Agency; 2015 (https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/ program/thematic/c8h0vm000001rgwp-att/2015_01.pdf, accessed 7 May 2019). OECD-DAC. Glossary of key terms in evaluation and results-based management. Paris: OECD-DAC; 2010. Peersman G, Guijt I, Pasanen T. Evaluability assessment for impact evaluation: guidance, checklists and decision support. London, UK: Overseas Development Institute; 2015 (https://www.odi.org/publications/9442-evaluability- assessment-impact-evaluation-guidance-checklists-and-decision-support, accessed 7 May 2019). UNICEF. UNICEF Strategic plan 2014–2017: evaluability assessment. Final report annexes. New York (NY): UNICEF Evaluation Office; 2015 (https://www.unicef.org/evaldatabase/files/Evaluability_Assesment_of_UNICEFs_Strategic_ Plan_2014-2017_Annexes.pdf, accessed 7 May 2019). Zandniapour L & JBS International. Impact evaluability assessment tool. Washington (DC): Corporation for National Community Service; 2014. Zivetz L, Cekan J, Robbins K. Building the evidence base for post-project evaluation: case study review and evaluability checklists. Washington (DC): Valuing Voices; 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 57 Appendix C: Evaluation websites and clearinghouses SiDA — evaluations Swedish development cooperation Asian Development Bank Independent evaluations at the Asian Development Bank CiDA — evaluations Canadian Aid Agency DANiDA — evaluations Danish Aid Agency European Commission — aid evaluations Aid evaluations of the European Union independent Evaluation Group at the World Bank — Publications The Independent Evaluation Group evaluates the development effectiveness of the World Bank Group. Norad — evaluations Norwegian Aid Agency OECD DAC Evaluation Resource Centre (DEReC) The Development Assistance Committee (DAC) is the principal body through which the OECD deals with issues related to co-operation with developing countries. The Evaluation Resource Centre contains development evaluation reports and guidelines published by the network and its 30 bilateral and multilateral members. OECD DAC ongoing evaluations of development programmes The site for sharing of evaluation plans and work programmes of members of the DAC Network on Development Evaluation, for enhanced transparency and facilitation collaboration. It also provides an overview of current trends in development evaluation. Poverty Action Lab A collection of various impact evaluations, including some short summaries of the conclusions from impact studies on a variety of topics of relevance for development. Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) ALNAP is a unique network that incorporates many of the key humanitarian organizations and experts from across the humanitarian sector, including members from funding agencies, as well as, nongovernmental, Red Cross/Crescent, United Nations and independent/academic organizations. It includes evaluations and lessons learned on humanitarian actions. The Expert Group for Aid Studies (EBA) Independent evaluations of Swedish international development cooperation. UN Evaluation Group (UNEG) Country level evaluation database. USAiD — evaluations The United States Agency for International Development, with links to USAID publications including USAID evaluations. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!58

Department of Reproductive Health and Research World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland Fax: +41 22 791 4171 E-mail: reproductivehealth@who.int Website: www.who.int/reproductivehealth The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978 92 4 151656 3 19072_Sample cover_3.5mm spine.indd 1 08/10/19 08:28

Department of Reproductive Health and Research World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland Fax: +41 22 791 4171 E-mail: reproductivehealth@who.int Website: www.who.int/reproductivehealth The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978 92 4 151656 3 19072_Sample cover_3.5mm spine.indd 1 08/10/19 08:28

The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978-92-4-151656-3 © World Health Organization 2019 Some rights reserved. This work is available under the Creative Commons Attribution-Non Commercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. Suggested citation. The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects. Geneva: World Health Organization; 2019. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at https://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see https://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see https://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Design and layout by L’IV Com Sàrl Printed in Switzerland Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. How was this guidance developed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Who should use this guidance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. What is a post-project evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 5. Why is there a need for post-project evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 6. What are the common reasons for and benefits of conducting post-project evaluations of ASRH projects? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 7 What can you expect when planning and conducting a post-project evaluation? . . 10 8. Steps in planning a post-project evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 9. Common challenges and strategies to address them . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 9a. Defining the purpose and scope of the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 9b. Establishing the post-project evaluation team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 9c. Addressing the lack of data availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 9d. Developing feasible sampling strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 9e. Bringing external context into evaluation consideration . . . . . . . . . . . . . . . . . . . . . . . 35 9f. Assessing sustainability and emerging and unanticipated outcomes . . . . . . . . . . . . 38 10. Assessing cross-cutting project principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 10a. Positive youth development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 10b. Gender equality and human rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 10c. Poverty alleviation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 11. How can evaluators, programmes and funders support post-project evaluation? . . . . 46 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Appendix A: Analysis report – review of post-project evaluations of ASRH projects . . . . . . . 50 Appendix B: Post-project evaluation evaluability checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Appendix C: Evaluation websites and clearinghouses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S i i i Acknowledgements The World Health Organization (WHO) is grateful for the contributions that many individuals and organizations have made to the development of this guidance. This guidance was coordinated by Venkatraman Chandra-Mouli of the WHO Department of Reproductive Health and Research, with support from Marina Plesons and oversight from Lale Say. WHO extends sincere thanks to Susan Igras, who carried out the literature review and developed the first draft of the guidance. WHO is also grateful to participants of a WHO technical consultation for their review and feedback to the first and subsequent versions of the guidance: Susanne Carai, Sarah Castle, Bregje de Kok, Fariyal Fikree, Kylie Hutchinson, Emma Jackson, Rebecka Lundgren, Bella Mansanya, Kristien Michielsen, Ghazaleh Samandari, Shantanu Sharma, Manahil Siddiqi, and Judith Westeneng. Lastly, WHO thanks Vivien Stone for editing the document, L’IV Com for designing the document, and Graham Ogilvie for preparing the illustrations. This work was funded by the John D. and Catherine T. MacArthur Foundation. The views of the funding bodies have not influenced the content of this guidance. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!iv Abbreviations ALNAP Active Learning Network for Accountability and Performance ASRH adolescent sexual and reproductive health AYSRH adolescent and youth sexual reproductive health CA contribution analysis CiDA Canadian International Development Agency DAC Development Assistance Committee (OECD) DFiD Department for International Development iPPF International Planned Parenthood Federation iPv intimate partner violence JiCA Japan International Cooperation Agency LMiCs low- and middle-income countries MSC most significant change OECD Organisation for Economic Co-operation and Development PYD positive youth development QUiP Qualitative Impact Protocol REM ripple effect mapping SDGs Sustainable Development Goals UNiCEF United Nations Children’s Fund USAiD DEC United States Agency for International Development Development Experience Clearinghouse WHO World Health Organization P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S v

Overview 1 In many countries, initiatives that aim to improve adolescent1 sexual and reproductive health (ASRH) implement projects without well thought out plans for evaluation. A range of reasons help explain this phenomenon, including limited training and understanding of the value of evaluation and dedicated evaluation resources within implementing and technical entities. However, the end result is that baselines and endlines may not exist, there may be no comparison of the project area to a non-intervention area, and implementation monitoring data may not provide much information on quality and other less easily counted impacts or results of implementation. The resultant lack of evaluation limits the ability to demonstrate what was achieved, what approaches worked, and what approaches did not work, thus losing out on critical learnings. Nevertheless, managers or funders with an interest in designing future projects or expanding existing projects may later commission a post-project evaluation to assess impact and learning in the months or years after a project has ended. Many ASRH projects encounter and address policy and programmatic challenges, and in doing so, learn valuable lessons. Since careful documentation and rigorous evaluation are the exception rather than the norm, though, the lessons from these efforts are not extracted and placed in the public arena. Post-project evaluations thus have a role to complement prospective studies for new or follow-on projects. However, there is a lack of guidance and literature on post-project evaluations, and technical guidance from funding agencies for conducting post-project evaluation is rare. To begin to bridge this gap more formally, the World Health Organization (WHO) developed this practical guidance for conducting post-project evaluations of ASRH projects. 1 WHO defines adolescents as people between the ages of 10–19 years. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 1 2 A literature review was conducted to understand common post-project evaluation rationales, challenges and solutions. The review included:  peer-reviewed and grey literature articles and post-project evaluation reports of ASRH projects, identified through a search of seven evaluation clearinghouses from low- and middle-income countries (LMICs) between 2010 and 2017 (Appendix A); and  a review of post-project evaluation guidelines published by international organizations, governments, and development agencies. WHO then convened a technical consultation of experts to review and provide inputs to the first draft of the guidance document, which led to revisions and the eventual final document. How was this guidance developed? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!2 3 Aimed primarily at evaluators, this guidance may also be of interest to project managers and funders. Its purpose is to help evaluators to think through the key decisions to be made to yield the most rigorous evaluation possible given available financial and human resources and project documentation. This guidance includes challenges, tips and lessons learned from the literature on post-project evaluations to inspire ideas for future evaluations. Case studies highlight selected issues as well as the responses taken by evaluators to address them. Although focused on post-project evaluation, this guidance is also relevant to evaluators of final project evaluations in situations where evaluation was not planned from the start or where projects have shifted focus midway. Who should use this guidance? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 3 4 There is no common definition of post-project evaluation, and many terms are commonly used (see Box 1). We use a definition informed by the Japan International Cooperation Agency (JICA) because it describes what this type of evaluation is, what its objectives are, and how it is carried out: Post-project evaluations are performed within a certain period after a project is completed and most are not planned while projects are still operating. As staff may no longer be available and project activities are not observable, post-project evaluation is based on existing reports, monitoring records and, other written information, and often includes additional data collection. While such evaluations may assess the extent that projects implemented planned activities and how well outcomes were achieved, given they occur after projects end such evaluations may also focus on longer term impact and sustainability. Post-project evaluations consider retrospectively circumstances surrounding a project and other elements that may have influenced implementation, impact and sustainability of results. The definition focuses on the timing and purpose of post-project evaluation to distinguish it from final or endline evaluations which are conducted as projects are ending – when staff, project counterparts and project beneficiaries are present and many project activities are still observable to assess quality and fidelity of implementation. Figure 1 shows the relationship of evaluation to project planning and implementation, as well as where and how post-project evaluation can support project assessment and learning. While post-project evaluation can answer many typical evaluation questions, they are particularly well placed to determine if project outcomes (such as improvements in adolescent-friendly sexual and reproductive health services or shifts in gender attitudes) remain viable over time when support ends. What is post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!4 box 1Related terms for post-project evaluation • after-action review • ex-post evaluation • forensic evaluation • post-completion evaluation • post-hoc evaluation • post-implementation evaluation • post-implementation review • post-project evaluation • retrospective impact evaluation • sustainability impact evaluation • sustained and emerging impacts evaluation Problem/issue Project design and plan Activities/outputs Outcomes Effects/impact Post-project outcomes/effects Resources/inputs Relevance and appropriateness Does it make sense? • Did the project objectives address the need? • Did the project intervention address the right issues? • Is there (still) a need? Effectiveness Did it work? • Did the project achieve the desired objectives/outcomes? • Was the intervention based on knowledge and research to improve the likelihood of success? Effects/impacts • Was the project worthwhile? • Does it have merit? Efficiency Was it efficient? • What was the relationship between the project’s inputs and outputs? • Could resources have been better used? Process surrounding the interventions Was it well-managed? • Did planning and decision-making processes ensure the project’s success? • Did management processes ensure success? • Did processes for developing activities ensure their success?    Contextual and external factors outside project boundaries that may have influenced the project trajectory    Sustainability and unanticipated and emerging effects • Are the project’s effects lasting? • Was the project as worthwhile and as meritable as thought? Figure 1. The logic of a project and its relationship to key evaluation themes Source: Adapted from Ryan et al., 2016. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 5 5 Evaluation is still variable and evolving as a state responsibility within many governments (World Bank, 2013) and the evaluation of social development projects is still an emergent field of inquiry. Evaluators often operate in contexts where data systems are weak and multiple interventions complicate the assessment of project contributions to outcomes. In an era of heightened demands for accountability by governments and organizations, evaluation increasingly is seen as a key tool to assess the impact of interventions and investments (Morra- Imas & Rist, 2009).  In this context, project evaluations vary widely. A relatively small number of applied research projects employ elegant evaluation designs focused on impact. Many projects do not have resources, evaluation training or a proclivity in a politicized environment to conduct evaluation (World Bank, 2013). Post-project evaluations are even rarer, representing less than 1% of international development projects that are evaluated (Zivetz et al., 2017).  Projects operating in conflict-affected and emergency contexts often remain unevaluated given the special challenges of implementation and short project timeframes (Puri et al., 2015).  There are fewer evaluations of ASRH projects available in evaluation clearinghouses compared with evaluations of other sectors. Of 900 evaluations available in a range of evaluation clearinghouses (see Appendix C), only 41 were focused on adolescent and youth sexual and reproductive health (AYSRH) projects. Of these 41 evaluations, only 6 were post- project AYSRH evaluations. Compared with ASRH projects in high-income countries, there is a paucity of project evaluation from LMICs linked to a theory of change and outcomes, and the body of evidence on good practices is small (WHO, 2004; Chandra-Mouli et al., 2015). Consequently, many ASRH projects do not include evaluation in their project design, or consider it as an afterthought. If a project is exploratory or seems to be performing very well and there is a desire to expand it, only then might evaluation be considered to provide important evidence. However, by this point, there is often no funding for evaluation within project budgets. Why is there a need for post- project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!6 Effective evaluation seeks to assess the merit or worth of projects according to five evaluation criteria – the projects’ relevance, effectiveness, efficiency, impact and sustainability (see Box 2) (JICA, 2004; OECD, 2000). Post-project evaluation is most suitable for assessing impact and sustainability and, possibly, relevance (JICA, 2004). It can also provide insights into effectiveness and efficiency. Rigorous evaluation planned from the initiation of a project should be the norm. Where this is not possible, or even if endline evaluation has occurred, post-project evaluation offers some unique possibilities for learning to:  Explore planned or additional project impacts or effects that may have occurred after the end-of-project evaluation period.  Assess the sustainability of a project’s impacts or effects. Findings from an end-of-project evaluation are not always adequate predictors of sustainability. Post-project evaluation can add to learning about sustainability by:  Understanding what elements of a project were sustained (or not) after the project ended.  Gaining deeper insight into how project-introduced approaches and activities have spread, diffused or replicated through anticipated and unanticipated pathways.  Gaining a better understanding of causal or associated factors related to the above (the why and how).  Identifying unexpected and emerging outcomes that came about after the project’s end. 6 What are the common reasons for and benefits of conducting post-project evaluations of ASRH projects? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 7 box 2  Potentially collecting information on a project’s relevance. Staff, stakeholders and end- users/participants, if still available, may have had more time to reflect on the project and may have additional insights to add regarding its relevance. For ASRH projects post-project evaluation can:  Reach older adolescents and young adults who may offer new and/or different insights into project impact, sustainability of its impact, and its relevance.  Contribute to a better understanding of whether the theory of change mechanisms were valid or not. This can inform the development of promising project approaches and activities and contribute to greater clarity about change mechanisms that lead to improved adolescent health and well-being. Table 1 shows the range of reasons for conducting post-project ASRH evaluations. Criteria for evaluating development assistance Relevance: This seeks to assess the extent to which the activity is suited to the priorities and policies of the target group, recipient and funding agency. Effectiveness: This seeks to assess the extent to which an activity attains its objectives. Efficiency: This seeks to assess the outputs – qualitative and quantitative – in relation to the inputs. It is an economic term that signifies that the activity uses the least costly resources possible in order to achieve the desired results. Impact: This seeks to assess the positive and negative changes produced by a project, intended or unintended. This involves the main impacts and effects – intended and unintended – resulting from the activity on local social, economic, environmental and other development indicators. Sustainability: This seeks to assess the extent to which the benefits of an activity are likely to continue after funding has been withdrawn. Source: Adapted from the Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD), 2000. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!8 Accountability  Did the project meet the expectations of adolescents and other stakeholders and funders?  What difference did the project make in the wider environment in which it operates? Evidence of effect  Did the project achieve its intended outcomes? Justification for future funding  Is the project strategy worthwhile enough to warrant future investment in similar projects?  If so, under what contexts and with what adjustments? Organizational and project learning  What evidence exists for an organization to implement a similar project going forward, or to scale up a successful project?  If so, under which contexts and with what adjustments? Sustainability  Are the project’s effects still visible?  If new structures, services or projects were put into place, are they still operating? Unanticipated or emerging impacts  What unanticipated effects have been observed since the project ended?  Have any unforeseen impacts – positive or negative – emerged since the project ended? Table 1. Typical purposes of post-project ASRH evaluations P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 9 7 Each post-project evaluation situation will be different, with different resources and constraints. As a result, the evaluator will have to determine the most appropriate evaluation design amidst some necessary methodological compromises. Some typical challenges that may be encountered (which are also common in other types of evaluations) include:  Project design documentation is of uneven quality. For example, there is no articulated theory of change or logic framework.  Qualitative descriptions of implementation are not available. For example, project reports are focused on explaining outputs and outcomes rather than processes.  Quantitative project data are patchy. For example, monitoring data were never collected or are not available for review; some critical activities are not included in monitoring systems; or the project has not conducted evaluation or has only done a baseline evaluation. The evaluator will likely have to play a variety of roles in addition to traditional evaluator, such as technical advisor to recreate or refine former project theories of change that will guide evaluation questions, or investigative reporter to uncover project documents or former project staff and partners. The evaluator will likely also play a liaison role between the funding agency and the organization that implemented the project. See sections 8 and 9, which share practical guidance for those planning and implementing a post-project evaluation based on experience and lessons learned on post-project evaluation of ASRH projects. What can you expect when planning and conducting a post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!10 box 3Typical challenges of post-project evaluations: the TUSEME Project in the United Republic of Tanzania The TUSEME Project (“Let’s Speak Out” in Kiswahili) was implemented in the United Republic of Tanzania, and employed a rights-oriented, gender-responsive approach to empower girls to understand and overcome problems that constrain their development, including academic achievement. Initially implemented in seven secondary schools in five regions, TUSEME was expanded to 27 schools and was eventually mainstreamed into the national Secondary Education Development Plan and replicated in 237 secondary schools. A post-project evaluation of the TUSEME clubs was carried out to assess the sustainability of student participation and the impact of participation on girls’ empowerment and acquisition of life skills. However, evaluators faced several challenges:  Framing the evaluation: Due to financial and time constraints, the evaluation methodology could not be designed to test effectiveness with a comparison group (e.g. schools with TUSEME clubs and schools without clubs). As a result, the initial terms of reference were renegotiated to include only schools with TUSEME clubs.  Data availability and sampling: The project had no baseline data. Similarly, other documentation was largely lacking, including a project log frame, monitoring and evaluation reports and financial documents.  Primary data collection: Interviewing adolescents was challenged by the need to find former project participants, and once found, obtain consent from students’ parents and informed assent from students below 18 years. Additionally, many parents were not available to talk with evaluators at the time of the school visit when interviews were conducted. Most who were available did not have children in the clubs. Source: Mhando, 2015. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 11 8 The following steps are similar to the steps taken for an evaluation conducted when projects are still functioning or have just ended. However, one key exception is the critical importance of doing an evaluability assessment at the start to determine the extent to which a post-project evaluation can be conducted reliably and credibly. STEP 1: Define the purpose, scope and initial design of the post-project evaluation. Determine the purpose and scope of the evaluation, including the evaluation questions, with key stakeholders, such as implementation organizations, relevant government ministries/ departments and funding agencies.  What are stakeholder expectations for the evaluation?  What do they want to learn and why?  What is the intended use of the evaluation findings and how will they be disseminated? Consider engaging stakeholders beyond the implementing organization and immediate partners, such as government stakeholders and future users of evaluation findings. Their early inputs can contribute to the credibility of the evaluation and use of findings to improve current and/or future projects. Steps in planning a post- project evaluation T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!12 STEP 2: Conduct a post-project evaluation evaluability assessment. An evaluability assessment can provide critical information from multiple perspectives on the extent to which a post-project evaluation can be conducted reliably and can generate credible results. The evaluability assessment should address:  Organizational considerations to ensure stakeholder consensus on the need for, and utility of, the evaluation to inform policy and programmatic decision-making.  information availability to support evidence-based evaluation.  Methodological considerations to be able to answer evaluation questions.  Resource considerations to ensure sufficient financial, human and material resources are available for evaluation.  Timing considerations to collect and analyse data in a logical and timely manner.  Acceptability considerations to gain stakeholder consensus on the type and level of evidence that is acceptable. An evaluability assessment checklist (see Appendix B) can be a useful tool for this critical activity. STEP 3: Identify information gaps and design a feasible methodology with the maximum rigour possible. Once the evaluation purpose(s) has been specified with project stakeholders and the results of the evaluability assessment are available, it will be possible to make choices and understand what compromises might be necessary regarding the evaluation design. Figure 2 provides a decision tree of possible evaluation design choices based on key learning objectives and information/data possibilities. Who will be involved in the evaluation process? How will adolescents be engaged (e.g. will they have roles besides being respondents, such as members of the evaluation team or as advisors to the evaluation team)? Are theories of change and logic frameworks available and will they need to be adjusted for use in the evaluation? Are data and process documentation available? What are the possibilities for sampling and additional data collection? How will you approach analysis and the interpretation of findings? These common challenges are addressed in detail in section 8. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 13 Figure 2. Decision tree on possible evaluation design choices Impact/ effectiveness Process/ implementation Post-test only with no comparison group Quasi-experimental pre/post-test design with non-random group assignment Post-test only with non-random group assignment Time series design (if multiple data points are available) Pre/post-test design with no comparison group (if only two data points are available) Use qualitative methods to deepen understanding of impact and gauge unintended effects Independent analysis of process data Reconstruct process through: Mixed methods including interviews, focus group discussions, case studies, etc. with key project staff, partners, beneficiaries, and the community. Efficiency could be examined if cost data can be linked to activities Create and analyze process data using administrative/ performance data Where and when feasible, engage young people in the evaluation design, data collection, and interpretation of findings. Sustainability Independent analysis of sustainability data Use mixed methods to collect data on: • Political support • Environmental/ contextual factors • Funding stability • Partnerships • Organizational capacity • Project adaptation • Project weaknesses • Project strengths • Communication • Strategic plan • Sustainability of impact What do you want to learn about the intervention? Supplement with additional data, where possible Did the project collect sustainability data? Did the project collect process data? Can you access administrative/ performance data? Is there adequate data on a group that did not receive the intervention to create a comparison group retrospectively? Can you collect data from both the comparison group and the intervention group? Is there adequate data on the intervention group to create data points retrospectively? Supplement with additional data, where possible Yes No T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!14 STEP 4: Confirm the evaluation methodology with key stakeholders to obtain buy-in for later utilization. It is time to share the post-project evaluation plan – the evaluation purpose, evaluation questions and data sources, methods to analyse existing data and collect and analyse additional data, and dissemination approach – for feedback and to ensure the consensus of project stakeholders. In particular, the plan should be shared with those most likely to use the evaluation findings, whether that is the implementing organization, government or funding agency(ies). A caveat: at this point, it may be necessary to go back to the organization or funding agency commissioning the evaluation to explain that some of the initial evaluation questions are unanswerable with the information and/or resources available. STEP 5: Secure ethical approval for the evaluation. It is critical to ensure the protection of adolescents in any evaluation of an ASRH project (WHO, 2018). The evaluator should secure approval from an institutional review board or a waiver of formal ethical review if the evaluation is considered to be a routine project assessment, and revise the proposal as needed. STEP 6: The evaluation is ready to begin! With a clear plan, consensus from project stakeholders, and ethical approval, data collection is ready to begin. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 15 9 This section shares commonly cited challenges during the evaluation process that are unique or critical to post-project evaluation, drawn from a variety of sources referenced in this document, and suggests how they could be mitigated or overcome. Case studies, drawn from actual evaluations, serve to highlight some of the challenges. This section will not cover challenges common to all evaluations, such as those related to financial, time and/or human resource constraints and those related to navigating political environments. While stakeholders may be interested in learning about cost–effectiveness, it is also not addressed in this guidance. Collecting retrospective, reliable cost data linked to project activities is a significant challenge that requires former project staff to recall the time and effort spent on specific activities and that requires former project finance staff to link finance data with activity data. Common challenges and strategies to address them T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!16 9a. Defining the purpose and scope of the evaluation Defining the evaluation’s purpose and scope, as well as its key boundaries, are essential to designing a good evaluation. Measurable performance indicators, theories of change and a clear definition of beneficiary groups provide the intentions and pathways through which project changes are expected to occur. When the post-project evaluation aims to assess impact, an evaluator’s key concerns revolve around assessing possible association or, where possible, causality; whether a feasible comparison group can be found; and whether baseline and endline evaluations exist. When the evaluation aims to assess sustainability or the project’s implementation process, key concerns revolve, respectively, around retrospectively defining the expected sustainability outcomes and attempting to recreate processes if these are not well described in existing documentation. Challenge: No quantitatively measurable indicators or performance targets were established before or during the project against which to assess change. Strategy: Develop performance indicators using documentation on project objectives and aims, in addition to secondary data from national surveys or similar projects if available. Present the new performance indicators to stakeholders and gain consensus for their use in the evaluation. If there are baseline indicators but no established performance targets, discuss with stakeholders whether changes observed in these indicators from baseline represent reasonable performance for the project. If indicators are not meaningful (i.e. are either too ambitious to establish causality or are not logical within the project context), discuss with stakeholders and develop consensus for redefined performance targets. Challenge: No written theory of change exists to guide understanding of association or causality. if such documentation does exist, it may be outdated given changes that occurred during project implementation. Strategy: Work with available project staff and other stakeholders to develop a theory of change and/or a logic framework. The former explains pathways leading to change, and the latter provides an orderly structure to monitor project implementation. If theories of change and logic frameworks do exist, review and revise them as needed (e.g. if there were shifts in implementation strategies or assumptions underlying implementation). Note, though, that creating a theory of change retrospectively may present an additional limitation in the interpretation of findings. Resource: For suggestions on how to create a theory of change or logic framework retrospectively with project stakeholders see: https://actionevaluation.org/theory-of- actiontheory-of-change-tools-resources/. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 17 box 4Reconstructing the project logic framework: the Tarunya Project in India The Tarunya Project, which was implemented by Engender Health and partners over a 5-year period (2008–2013) in Jharkhand State, India, was designed to:  strengthen health systems and health service provision;  enhance community engagement and links between the services and community activities; and  strengthen ASRH programme leadership and management by the state and district health management teams. WHO was invited to conduct a post-project evaluation and the evaluation team quickly learned that no explicit programme theory of change or logic framework existed, making it impossible to develop suitable evaluation questions. The team collected all of the relevant project documents and used information from multiple sources to construct a logic framework to explain what the project had set out to do and what it had actually achieved. In doing so, the team put effort into ensuring that the logic framework was entirely faithful to the information in the project plans and reports. The logic framework then guided development of a set of evaluation questions related to Tarunya’s design, implementation, outputs, outcomes and impact. Upon presenting the logic model to project staff, they asked, “Where did this come from?” Evaluation team members explained that the project’s own plans and reports were the source of the logic framework. By pointing to the sources of specific information in the logic framework, the team showed project staff that the model was not invented but rather articulated a structure that was never made explicit on paper. Source: Barua & Chandra-Mouli, 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!18 Challenge: Concepts underpinning theories of change, such as “reaching vulnerable adolescents”, were not well defined operationally, making later evaluation decisions on sampling and measurement of project coverage difficult. Strategy: Work with project staff to articulate precise definitions for these concepts, based on their experience and understanding. If the original project staff are not available, identify potential definitions from the literature and interview stakeholders to develop consensus on an operational definition for use in the evaluation. Note, though, that creating a new definition retrospectively may present an additional limitation in the interpretation of findings. Challenge: Stakeholders are interested in assessing impact quantitatively but design options are limited. Under-used but adequate evaluation designs do exist for such post- project evaluations, but they are not always well understood by some evaluators. Strategy: Be prepared, with references, to explain design options to fellow evaluators, as well as the rationale for proposing the selected design. Because comparative designs that include only post-test assessments with actual or statistically constructed comparison groups are not always well understood, the evaluator will need to develop this understanding among stakeholders to secure buy-in for their use. Note, though, that many of these design options require the team to have access to statistical support. Resources:  For a practical overview of different evaluation design options, including post-project evaluation designs, see ALNAP’s Evaluation of humanitarian action guide (Chapter 11 Evaluation designs for answering evaluation questions) (https://www.alnap.org/ system/files/content/resource/files/main/alnap-evaluation-humanitarian-action-2016. pdf).  For a discussion (with helpful examples) of studies having adequate versus plausible versus probable impact, see the 1999 International Journal of Epidemiology seminal article by Habicht et al., “Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact” (https://www.measureevaluation. org/resources/training/capacity-building-resources/m-e-of-hiv-aids-programs-in-india- english/session-5-gis-and-research-evaluation/research-evaluation/Int.%20J.%20 Epidemiol.-1999-Habicht-10-8.pdf).  For a helpful perspective to understand the necessity and utility of design compromises in certain evaluation contexts, see the 2012 AIDS journal article by Laga et al., “Evaluating HIV prevention effectiveness: the perfect as the enemy of the good” (https://www.ncbi.nlm.nih.gov/pubmed/22313952). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 19 box 5Operationally defining “migration” and “trafficking”: a post- project evaluation in Mali In Mali, international agencies and the government requested a post-project evaluation to explain why a cross-border strategy aiming to reduce trafficking of vulnerable adolescents between Mali and the cocoa plantations of Côte d’Ivoire was unsuccessful. The project was guided by the legal definition of human trafficking in Article 3 of the United Nations Protocol to Prevent, Suppress and Punish Trafficking in Persons as: “the recruitment, transportation, transfer, harbouring or receipt of persons, by means of threat or use of force or other forms of coercion, of abduction, of fraud, of deception...to achieve the consent of a person having control over another person, for the pursue of exploitation. Exploitation shall include…forced labour or services, slavery, or practices similar to slavery, servitude…” The challenge was how to operationalize “migration” and “trafficking” in contexts where family members and the young people themselves are routinely placed by intermediaries in work situations as agricultural labourers. While the anti- trafficking programme was designed to return adolescents home, most did not want to be “freed” given the enormous peer pressure to go on migration as a rite of passage. Upon return to their poor, rural villages, most adolescents returned to Côte d’Ivoire to seek work. The evaluation team used appreciative inquiry techniques – focusing on young people’s strengths in complex realities rather than their weaknesses – to operationalize “migration” and “trafficking”, informed by the legal definition. They then used these operational terms to develop culturally appropriate interventions for safe migration by young people. Source: Busza J, et al., 2004. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!20 9b. Establishing the post-project evaluation team Evaluation teams are ideally constructed to ensure diversity of skills needed to undertake the evaluation with sufficient content and contextual knowledge of the project being evaluated. During a typical final project evaluation, the evaluation team can regularly consult with project staff and partners about the project’s history, its context, and any implementation issues. However, in post-project evaluations such people are often not readily available and project content and contextual knowledge need to be defined retroactively. Challenge: An external evaluator, who is not familiar with the project’s history, implementation, and larger operating context, is hired as team lead, and has limited or no formal access to former staff. Strategy: Try to complement the team lead’s external objectivity with team members who can provide necessary project content and contextual knowledge and complementary technical skills. Ideally, these team members would be former project staff or people familiar with the organization and project. If these people are not available, invite evaluators of similar projects or working in a similar geographical area who may have a good understanding of the context. Another strategy would be to invite former staff to serve as a consultative group for the evaluation team. Challenge: While they can provide historical context and detail, team members who “own” or have “owned” the project may also be biased in their long-term recollection and interpretation of events and results. if former project staff or partners become team members, be aware of potential biases they may hold. Strategy: An important starting point is for the team lead to acknowledge that insider bias is normal and establish a sense of mutual accountability to ensure that the evaluation is as evidence-based as possible. The team lead and team members all have an important role to play in “fact checking” and insisting that available evidence (not opinion) is used to support the analyses and findings. When undertaking data collection, it is important to organize the team to avoid situations of friends interviewing friends. Likewise, the team should discuss informed consent and strategies for maintaining anonymity and confidentiality when interviewing former colleagues and friends. Similarly, the team should be cognizant of situations where team members who are former project staff may be vulnerable to risks and potential harms when interviewing former colleagues, such as when interviewing a former supervisor with whom there were tensions. In these situations, it is especially important to build multiple lines of inquiry into data collection to allow for triangulation of more subjective evidence. Identifying more objective sources of information can be helpful in this regard. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 21 box 6Establishing a common understanding of the project: the CERCA Project in the Plurinational State of Bolivia, Ecuador and Nicaragua The Community-Embedded Reproductive Health Care for Adolescents (CERCA) Project was implemented in the Plurinational State of Bolivia, Ecuador and Nicaragua (2011–2014) to test the effectiveness of contextually adapted interventions to prevent teenage pregnancies. Across the three countries, CERCA worked at multiple levels – engaging adolescents, their peers, parents and health care providers. A post-project evaluation, which assessed both the process and the impact of the project, was undertaken to examine if and how CERCA’s design, implementation and regular use of monitoring data to adapt interventions affected the results. With limited project documentation available for review and a diversity of stakeholders involved in CERCA’s design, it was important that CERCA partners participated in the evaluation not only as respondents but also as evaluators who collected and analysed data. When working with project staff to develop retrospectively a project theory of change, it became apparent that different staff understood the project’s implementation process and outcomes/impact quite differently. To address potential bias, the team needed to establish a common understanding of the project through the participatory development of a theory of change. Additionally, the team prompted and facilitated subsequent discussions on how to reduce the potential for bias during data collection and analysis. Source: Ivanova et al., 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!22 Challenge: Adolescents who were associated with the project are not considered for evaluation roles except as respondents. Although it is a recognized as good practice to engage young people in an evaluation of their own project, to ensure the inclusion of adolescents’ perspectives and to contribute to positive youth development, it is rarely done. Strategy: Thinking of adolescents as only evaluation respondents is short-sighted. If they are available, engage former project participants to work with external evaluators in as many phases of the post-project evaluation as possible. At a minimum, if financial resources and time are limited, a well-designed and facilitated meeting with former project participants to seek input on plans for the evaluation can contribute to important adjustments in later evaluation phases. Organizing a subsequent analysis meeting with these adolescents to validate the results of the evaluation would also enrich its findings. Ideally, if resources allow, work with adolescents in the design, implementation and dissemination of the evaluation. They are well placed to develop data collection tools, and their involvement in data collection can yield more truthful data as young informants may be more open to talking about issues related to sexual and reproductive health with other young people with whom they can identify, as compared with adult researchers. Note, though, that adolescents who may be engaged are not necessarily representative of the target population. Careful facilitation is needed to mitigate potential resistance by adults to young people evaluating them and their work. An evaluator may need to advocate to the funding agency to include meaningful adolescent engagement to provide additional support for their inclusion. Resources:  For guidance and facilitator manuals to create conditions for successful youth participation and to train and support young people to conduct qualitative data collection for research and monitoring and evaluation, see Explore: a toolkit for involving young people as researchers in sexual and reproductive health programmes, developed by Rutgers and International Planned Parenthood Federation (IPPF) (https://www.rutgers.international/our-products/tools/explore).  For a practical approach to training and engaging young people in evaluation see Youth participatory evaluation: strategies for engaging young people (Kim Sabo Flores, 2007), which draws on theories of children’s play, evaluation and youth development (https://www.wiley.com/en-us/Youth+Participatory+Evaluation %3A+Strategies+for+Engaging+Young+People-p-9780787983925). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 23 box 7Putting into practice adolescent-adult partnerships: an evaluation in Ethiopia, Kenya, Indonesia and Pakistan In 2015, an evaluation was carried out in Ethiopia, Kenya, Indonesia and Pakistan to explore how local organizations involved young people within their ASRH programmes. Each country’s evaluation team included one adult evaluator and three to eight adolescent and youth co-evaluators. The co-evaluators were trained on the key programme concepts, research ethics, and techniques of conducting interviews and focus group discussions, based on the Rutgers and IPPF Explore toolkit. First, the young evaluators developed their own definition of meaningful youth participation, which formed the basis of their study. Additionally, they reviewed, translated, tested and adapted tools during practice sessions. They gathered information from heads of organizations, project staff, service providers, young people engaged in the programmes, and external stakeholders using interviews and focus group discussions. The young evaluators also observed intervention activities, such as provision of services at youth centres and health clinics, provision of peer education, and community mobilization. The young people were quick to learn. Every evening, the adult researcher organized de-briefing meetings with the research team to discuss the day’s findings and analyse the data. This helped the young evaluators to see any gaps in their data that they could cover the next day. Additionally, the evaluation team tried to de-brief each organization on the key findings of the evaluation to validate conclusions and formulate recommendations together. Source: Van Reeuwijk M & Singh A, 2018. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!24 Challenge: Where appropriate, the evaluation team lead needs to ensure parental/ caregiver approval as well as adolescent safeguarding during participation in evaluation activities in a context where former project structures and protection mechanisms no longer exist. Strategy: To ensure ethical and acceptable approaches are used to safeguard participants, obtain the advice of research committees in the area. To obtain parental/ caregiver consent, provide a written summary of the evaluation, describe in what contexts and for how long the adolescent will be engaged, and seek oral or written consent from their parent/caregiver. If it is very challenging to find parents/caregivers, it may be appropriate to engage school principals or other leaders to reach parents/ caregivers. In addition, the team should agree to child safety standards, such as ensuring minors safely move between the evaluation site and home, or having adult members accompany minors if they collect data or share findings in contexts that may place them in harm’s way. If the project was implemented by a child-focused organization, use their child protection guidelines to guide the evaluation safeguarding activities. Resources:  For recommendations on how to protect and support young evaluators engaged in ASRH project evaluation from harm, see Rutgers’s Explore toolkit (Part II: Creating conditions for successful youth participation in research) (https://www.rutgers. international/sites/rutgersorg/files/pdf/Explore-instructions_web.pdf).  For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/hand le/10665/273792/9789241508414-eng.pdf?ua=1).  For ethical considerations of participatory evaluation that place evaluators in the context of community power dynamics, beyond human subject protections, see Banks et al. 2013 article in Contemporary Social Science, “Everyday ethics in community- based participatory research” (https://www.tandfonline.com/doi/pdf/10.1080/21582 041.2013.769618). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 25 9c. Addressing the lack of data availability Data availability, or the lack thereof, is one of the most common issues that evaluators confront in post-project evaluations, since reliance on existing documentation is critical to the evaluation task. Challenges, such as those below, focus on issues of finding key informants from a past project, accessing usable quantitative outcome data and process documentation, managing recall issues, and interviewing adolescents whose age and cognition levels due to maturation no longer reflect the project population. Suggestions for sampling and viable qualitative impact evaluation methods are discussed in later sub-sections. Challenge: Key project personnel are not available due to migration, transfers to new projects, or other reasons after the project has ended. Strategy: Attempt to track down telephone numbers via colleagues and other community contacts and conduct phone interviews with former staff. Ask current employers if former project staff could attend a half-day or full-day meeting during which post-project data could be collected. If this is not possible, try to interview former staff in less formal settings such as coffee shops. If this is not possible, interview staff and people within the host organization and its implementation partners who have some knowledge of the former project. Challenge: Key adolescent beneficiaries and beneficiary lists are not available. Strategy: Young people are often an especially mobile population and the evaluator may need to get creative to find young people who participated in the project. Identify existing structures such as schools or youth clubs where past beneficiaries might be located. Ask school principals or other managers to contact beneficiaries on your behalf. If structures are still active, sample from still-functioning groups formerly supported by the project – comparing those groups with groups or individuals who never participated in the project. If it is not possible to contact former beneficiaries, consider collecting information from people who are familiar with the former beneficiaries. Challenge: There are no quantitative baseline, midline or endline data and using recall information is the best option to assess change. Strategy: Use participant recall data to assess impact in areas such as personal knowledge, attitudes and behaviour changes. For example, use Likert scales with statements that ask participants to assess the extent of change between two time points. Incorporate similar-but-different statements (e.g. two different statements about the same attitude) to assess variability in responses to check recall bias. To help distinguish context from factors more directly affected by the project, refer to timelines of internal and external events during key informant interviews to remind team members of the larger context. Likewise, interviewing senior management may be useful, as people in senior management positions often operate with the larger context in mind. Triangulate recall data with other available data. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!26 box 8Finding adolescents 14 years after the project’s end: the Pro.Star Project in Jamaica Some 14 years after the Pro.Star Project in Jamaica ended, evaluators used a case control study design to retrospectively assess the effects of the project, a school-based and parent education intervention on adolescents’ transition to adulthood. Intent on finding former students to learn how they had progressed in the 14 years post-project, evaluators had to use multiple means to find former participants, including internet, social media (Facebook and WhatsApp), and word of mouth to locate students. Teachers, guidance counsellors and current students helped evaluators to locate past programme participants – reaching out, for example, to their uncles, brothers, aunts and parents. On occasion, guidance counsellors accompanied interviewers to locate persons they had trouble finding. This proved effective because people were more trusting when credible community members accompanied interviewers. Once students were located, evaluators asked them to share names and contact information of other students. Determination and patience in tracking down respondents and flexibility in fitting data collection activities into their schedules were critical. Many of the students had moved out of the community – or, in fact, out of Jamaica – so interviewers conducted some interviews by phone. The results of this outreach were impressive – it was possible to locate and interview over 60% of sampled former students (155 participants) either in person or by telephone or email interviews. Source: Brown PR & Hardee K, 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 27 box 9Using a new evaluation design when the original design is not feasible: an innovations partnership project in Sierra Leone Recognizing that adolescent pregnancy in Sierra Leone is driven more by sociocultural than service access factors, UNICEF established a partnership with Child Fund, Save the Children, the Council of Churches of Sierra Leone, Restless Development, and the Bangladesh Rural Advancement Committee. Together, these organizations implemented five pilot projects in seven districts with the aim to address underlying drivers of adolescent pregnancy in the country. The post-project evaluation aimed to foster learning about the actors, contexts, and challenges of preventing/reducing adolescent pregnancy to inform future programming. Although each partner conducted a baseline, it was not possible to use findings from these baselines as a benchmark for assessing change or impact of the pilot projects due to inconsistencies in project approaches and survey methodologies across the five pilot projects. Instead, the evaluators selected an outcome mapping evaluation approach to assess the different dimensions of change in mindsets, contexts, attitudes and perceptions of individuals and groups that influence motivations, interactions, relationships and behaviour across the five pilot projects. It also explored the contributions of the pilot projects in supporting empowerment and increasing access to and use of sexual and reproductive health services. Source: Farzaneh, 2013. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!28 Challenge: The evaluator wants to align post-project data collection with data from earlier evaluations, but survey instruments and data are not comparable across survey points (baseline, midline, endline and post-project). Strategy: Build on existing survey instruments by making small adaptations that permit comparisons of most/all key outcomes over time. While questions may differ slightly (i.e. an early instrument may assess one specific area of parent-child communication, which is replaced in a later survey instrument to explore a different parent-child communication area) qualified interpretation of findings between time points is possible. If the evolved project aim means that adjusting questions from past survey instruments is not possible, map out key thematic domains currently addressed by the project and develop new questions to evaluate the domains. New data collection using survey methods opens the possibility of using statistical methods to answer many evaluation questions (i.e. factor analysis to determine associations of exposure to the intervention and outcomes, multiple logistics regression to determine the likelihood of change or influence due to project exposure). In the end, however, it may not be possible to undertake a quantitative study that will yield expected results. In such cases seek a different approach that contains a level of rigour to allow for the establishment of plausible causality. Resources: Going forward, recent funding agency mandates, once they are widely instituted, will make datasets open source and thus available to evaluators for additional analyses to answer questions posed in a post-project evaluation. In the meantime, for open access data on adolescents living in LMICs, see the GIRL Center Adolescent Data Hub (https://www.popcouncil.org/girlcenter/research/hub). Challenge: The project is heavily focused on adolescent outcomes and neglects to collect information on other outcome areas, such as improved parenting skills or improved teacher pedagogy. Strategy: It may be useful to review training and supervision reports. As these are produced at a lower level of the reporting system, they may include information on adults and their engagement and reactions to the project. Alternatively, such information gaps can be corrected by sampling adults in new data collection. Challenge: Post-project beneficiary movements, the ageing of adolescents, and other confounding factors associated with the passage of time make it difficult to gain meaningful information that can be generalized through a population-based study. Strategy: Consider using alternative sampling to suit the evaluation aim. For example, consider snowball sampling if the evaluation aims to access people of the same social group, diversity sampling if the evaluation aims to assess differences in sub-groups, or purposive sampling if the evaluation aims to intentionally study selected groups. Resource: For an overview of different sampling strategies, see Changing Minds’ overview on choosing a sampling method (http://changingminds.org/explanations/ research/sampling/choosing_sampling.htm). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 29 box 10Re-purposing existing project data for post-project evaluation: the Baylor Pediatric AIDS Initiative in the United Republic of Tanzania In August 2008 Baylor University began a seven-year project in the United Republic of Tanzania to strengthen services for prevention, early detection, treatment and care of pediatric HIV/AIDS in the Southern Highlands and the Lake zones. A post-project evaluation was carried out to assess the extent to which the project’s goals and objectives were achieved and to provide guidance and lessons learned for future projects, in particular related to effectiveness and sustainability. As an applied research project, the evaluation team had access to ample documentation and databases from the project period. Samples for quantitative data analysis, including a comparison group, were derived from databases of activities from the relatively larger and more complete population samples of the Baylor International Pediatric AIDS Initiative that evaluators were able to analyse in the way they wanted. Although this led to potential sampling bias, it negated the need for new data collection. Primary qualitative data were collected by the evaluation team through key informant interviews, focus group discussions and observations, and were used to supplement, complement and verify quantitative analyses. Source: USAID, 2015. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!30 Challenge: The maturation effect (i.e. ageing of adolescents who were enrolled in a project at an earlier age) introduces comparability biases in post-project evaluation instruments and data, such as in their interest in sexual experimentation and awareness of the larger social environment. Strategy: Ideally, create an unexposed group for comparison. If this is not possible, design post-project data collection to allow for the comparison of older adolescents with younger ones. For example, create two age cohorts: one mimicking the initial unexposed participant group as they entered the project, and the other representing older adolescents who participated in the project. Alternatively, include recall questions for older adolescent participants to inquire about changes they have observed between pre-project and post-project time points. Challenge: Data found in annual or other external reports of the project’s process or accomplishments do not appear realistic, are not sufficiently disaggregated, or do not align with outputs and outcomes identified in the logic framework. Strategy: If data seem unrealistic, try to verify information from project monitoring data. If not available, query the organization’s senior managers to understand what was reported and confirm its reliability. For example, critical contextual information on implementation, such as a large number of new staff recruitments, may not be included in the report but is necessary to understand the findings. If output and outcome data are not available in reports or are not sufficiently disaggregated for evaluation purposes, try to work backwards by reviewing monitoring data or other internal reports. If monitoring data sets are available and data are disaggregated by sex and age, consider doing additional analyses. If disaggregated data are unavailable, undertake new post-project data collection to elicit potentially different outcomes based on adolescent age or sex. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 31 9d. Developing feasible sampling strategies Sampling poses particular challenges for post-project evaluators, ranging from selecting the site to establishing a comparison group to selecting respondents. Without project outreach structures, it may be challenging to identify adolescent beneficiaries. Time and money, of course, also direct what sample is feasible, including whether or not to include a comparison group. When working with vulnerable adolescents, an assessment of the benefits and risks of contacting such individuals should also influence sampling decisions. When constraints do exist, it is important to acknowledge potential bias in the analysis. Challenge: Site selection is done in consultation with government officials or implementing partner staff, which may introduce bias. Alternatively, sites are selected based on real- life considerations, such as security issues or ease in reaching communities. Strategy: Be aware of site selection biases and advocate for transparency in decision- making and unbiased site selection as much as possible. Try to independently assess characteristics of the selected sample, including similarities between the intervention and comparison groups regarding the adolescent population, services, and other relevant infrastructure such as schools. Secondary data may be helpful in assessing comparability. Challenge: There is no comparison group or geographic area to control for the influence of confounding external factors. Strategy: If funding and time are available, create a comparison group or area as part of new data collection processes. Alternatively, use secondary data for comparison, if they are of reasonable quality, such as national survey results or service statistics. Assess for similarities between the intervention and comparison groups regarding the adolescent population, services, and relevant infrastructure such as schools. While preferable to include a comparison group, in certain circumstances it may not be desirable for ethical or logistical reasons. Challenge: The post-project evaluation needs to sample vulnerable adolescents and their parents/caregivers, or hard-to-access respondents, such as undocumented migrants or refugees. Strategy: Firstly, it is essential to conduct an upfront ethical risk/benefit analysis using a “do no harm” framework for evaluations of projects that reached vulnerable adolescents, such as adolescents living with HIV or sex workers. In addition to assuring child/adolescent protection, it is important to train and support interviewers in protection concepts and establish processes for data collection. Pay careful attention to issues such as site selection, sampling and interviewing vulnerable adolescents. To identify and find vulnerable adolescents, work with former project staff or staff from current outreach projects who serve the vulnerable group or an external advisory group, such as a research committee, to determine an appropriate and ethical way to identify and invite respondents to participate in the evaluation. It may be possible to create a sampling frame using network mapping or snowball sampling techniques. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!32 box 11Weighing the risks and benefits of establishing a comparison group: a theoretical post-project evaluation in South-East Asia Sometimes creating a comparison group is not desirable for ethical reasons. In a post-project evaluation of a programme for sex workers in South-East Asia, a conscious decision might be made not to create a comparison group because of the following considerations:  The research might be limited in time and financial resources.  It might be difficult to identify and gain access to a comparison group.  The analyses would need to correct for selection bias, given that participation in the intervention would have been voluntary. Sex workers who had not participated would likely differ in largely unobservable characteristics from those who did.  There would be no clear indication that the intervention would continue, thus participation in the intervention would not be able to be offered to the women in the comparison group. For these reasons, the evaluation team might decide that any benefit of a control group would be outweighed by the risk of exposing women to the community as sex workers. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 33 Ensure ethical considerations for privacy are met. Social media and use of WhatsApp can allow for discrete outreach, but may lead to bias in the sample since not all adolescents have access to these technologies. Ensure that evaluation objectives include budget line items for these types of inquiries to ensure they are adequately addressed. Resource: For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/han dle/10665/273792/9789241508414-eng.pdf?ua=1). Challenge: Former project structures that facilitated the logistics of finding vulnerable adolescent beneficiaries (and their parents/caregivers for provision of informed consent) are no longer available. Strategy: Contact the project’s implementing partners to share information about the post-project evaluation and to set up meetings with former beneficiaries. Ask leaders or key contacts from former implementation partners, such as school principals, to inform parents/caregivers of an upcoming evaluation visit and seek their consent for their child’s participation. (Note: This represents an opt-out approach; that is, parental consent is assumed unless the parents/caregivers indicate they do not want their child to participate. Although this is far from ideal and introduces important ethical considerations, in specific circumstances it may be the only viable approach to ensure parents/caregivers are both informed and that they provide consent.) Set up meetings to interview stakeholders – including adolescents – in discrete ways that do not put them in harm’s way of greater visibility or discrimination. Invite parents/caregivers when inviting adolescents to be interviewed in order to secure consent, as well as to increase the response rates of both adolescents and parents/caregivers. Resources:  For practical approaches to engage young people including vulnerable adolescents, as respondents, see working paper The ethics of social research with children and families in young lives: practical experiences (http://www.younglives.org.uk/sites/www. younglives.org.uk/files/YL-WP53-Morrow-EthicsOfResearchWithChildren.pdf).  For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/hand le/10665/273792/9789241508414-eng.pdf?ua=1). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!34 9e. Bringing external context into evaluation consideration External factors influence project trajectories. It is thus important to consider other projects, policies, and macro-environmental developments that may have influenced a project trajectory. As such, a post-project evaluation often needs to fill in missing contextual information to improve evidence of effectiveness and sustainability. Challenge: information on contextual factors to help explain the nature and trajectory of change is not available in the project documentation. There is little reference to situational analyses or anticipated risks. Project indicators are not linked to these contextual factors, either within the project cycle or after the project has ended. Strategy: Seek out a variety of information sources. Consult key informants for information on contextual factors, such as new policy developments or political forces that may have affected the project, but be cautious about potential biases. If possible, review pertinent information available from media sources, including news articles and other media records and Twitter feeds. Existing documentation (i.e. synthesis reviews, policy analyses) may provide contextual information. Try to recreate internal and external event timelines with available project or implementing partner staff and stakeholders to document factors that may have influenced implementation. Consider engaging with ethnographers to visit sites and employ ethnographic methods to explore and document contextual factors. It may also be useful to employ less-traditional evaluation frameworks; for example, if sufficient time and resources are available, a realist evaluation approach, based on developing explicit understanding of how context relates to outcomes, may help guide information gathering to address these contextual issues in non-biased ways. Resources:  For an explanation of how to create an events timeline, see Better Evaluation’s overview of timelines and time-ordered matrices (http://www.betterevaluation.org/ en/evaluation-options/timelines).  For information on using realist evaluation, see Better Evaluation’s overview of realist evaluation (https://www.betterevaluation.org/en/approach/realist_evaluation). Challenge: Other development projects have launched during project implementation or between the end of the project and the post-project evaluation, complicating the plausible attribution of findings and/or influencing post-project evaluation findings. Strategy: Use evaluation reports and assessment studies from the other projects, if available, to validate anecdotal information. If evaluation reports from the other projects are available, try to map overlapping activities. Try to determine, via an events timeline discussion, to what degree projects from different organizations supported or undermined each other’s effects between the end of the project and the post-project evaluation. Interview former staff and partners to inquire if prior projects and partnerships extending before the project years were important building blocks to the project under evaluation and include this as a component of the context documentation. Methods such as contribution analysis (CA) and Qualitative Impact Protocol (QUIP) provide a systematic approach to assess causal questions and infer plausible attribution in real-life project P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 35 box 12Gaining understanding of context to explain project outcomes: the strengthening collective response of the government to end child marriage through a district level convergence approach project in India The MAMTA-Health Institute for Mother and Child, New Delhi, in partnership with district administrations, undertook a 3-year project to support a cross-departmental convergence approach to synchronize government efforts to end child marriage in Sawai Madhopur, Rajasthan, and Jamui, Bihar, in India. As a convergence or systems change approach, it was important to understand external factors that could have helped or hindered project efforts to change institutional processes to work across departments during project implementation and in the time between project closing and the post- project evaluation. The creation of an events timeline, supplemented by secondary information, was critical to interpret the significant differences in programme outcomes in the two districts. The evaluation team worked with former project staff to create a timeline of internal and external project events, such as changes in child marriage policy at national and state levels and overlap with a state-sponsored programme to end child marriage in one state (Rajasthan) that began mid-project and worked across service departments (a similar approach to the project of interest). This information led to new questions about environmental forces, and the team subsequently supplemented and confirmed information about these events through a review of policy documents and stakeholder interviews. Source: Chandra-Mouli et al., 2018. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!36 evaluations in complex settings. Be willing to state that it is not possible to determine plausible attribution. Resources:  For information on CA – a qualitative technique that guides managers, researchers and policy-makers to a consensus on the contribution their project has made to particular outcomes through better understanding of why results were achieved (or not) and the roles played by the project and other factors – see Better Evaluation’s overview of contribution analysis (www.betterevaluation.org/en/plan/approach/ contribution_analysis).  For information on QUIP – a qualitative technique to evaluate a project implemented in a context of complexity that helps explain how changes can be attributed to different stakeholders or events – see the Assessing Rural Transformations’ QUIP guidelines for field use (http://www.bath.ac.uk/cds/projects-activities/assessing-rural- transformations/documents/complete-quip-guidelines.pdf). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 37 9f. Assessing sustainability and emerging and/or unanticipated outcomes Sustainability is almost always a desired impact and often a funding agency requirement, and post-project evaluations are ideally suited to assess this. Post-project evaluation also provides the possibility to explore whether any other outcomes occurred after the close of the project (Zivetz et al., 2017). These outcomes can be distinguished between unanticipated outcomes (arising from assumptions made in the theory of change) and emerging outcomes (due to efforts and resources of participants and partners after the project ended). Challenge: Sustainability outcomes are not well-defined in project documents but are a post-project evaluation aim. Strategy: Discuss with available staff whether exit or graduation strategies – planned project efforts to close out or transfer implementation support to another entity – existed in the project under evaluation, and what benchmarks or indicators signalled the end of, phasing out of, or reduction of project support. Work with available project staff and other stakeholders to create an operational definition of sustainability that can be used in planning and conducting the evaluation. One approach is to develop a hierarchy of sustainability (i.e. differentiations between sustained project inputs, outcomes or impacts) to define the original project’s sustainability aims. This will simplify the process of determining which methods can be employed to assess sustainability (e.g. observation to determine if activities and use of project materials continue in the post-project context). Resources:  For an illustrative example of a sustainability hierarchy that can guide evaluator-staff discussions on developing an operational definition of sustainability for a project under evaluation, see Valuing Voices Building the evidence base for post-project evaluation (also see Box 13) (http://valuingvoices.com/wp-content/uploads/2013/11/The-case- for-post-project-evaluation-Valuing-Voices-Final-2017.pdf).  For a practical overview of issues in evaluating the sustainability of health projects (including definitions and types of sustainability, specifications and measurements of dependent variables, definitions of independent variables or factors that influence sustainability, and suggestions for designs for research and data collection), see “An agenda for research on the sustainability of public health programs” (https://www. ncbi.nlm.nih.gov/pmc/articles/PMC3222409). Challenge: Anecdotal information indicates that unanticipated and/or emerging outcomes have occurred and need to be part of the evaluation frame to document lessons learned for future projects. Strategy: Several participatory evaluation techniques (listed below) have been developed to address such challenges regarding the wider impact of a project and plausible causality with a degree of rigour. Data collection and evaluation tools should be designed to search explicitly for unanticipated negative outcomes. While positive outcomes are easily shared by respondents, negative outcomes may not be captured T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!38 box 13Defining sustainability of an illustrative ASRH project using a hierarchy of results framework In this example, evaluators and stakeholder developed a hierarchy of sustainability to guide evaluation of an after-school club intervention for very young adolescents. Project stakeholders first defined sustainability of outputs, outcomes and impacts (y-axis). They also differentiated between types of impacts (x-axis) (i.e. whether project-expected achievements were sustained or whether broader sustainability impacts emerged that could be discerned only over a longer timeframe; that is, after a project ends). In addition, Zivetz et al. note there are two broader areas of sustainability to assess: emerging outcomes consider how participants used their own means to carry project initiatives forward, which could produce learning on incentivizing sustainability. This is in contrast to unanticipated outcomes, which are related to the project’s theory of change and may reveal the extent of and reasons why assumptions or objectives deviated from what was anticipated in the project design. Sustained impacts Emerging impacts N/A ASRH and gender equality improved within school structures Teachers are more comfortable teaching sexuality education. Adolescent financial literacy improved Sustainability of impacts Parent-teacher-student committees are maintained, with support from the national government After-school clubs are maintained with continued active engagement of adolescents and teachers Adolescents do small activities for cash to support continuation of the clubs Sustainability of outcomes Parent-teacher-student committees are in place and trained After-school clubs are in place and operational Adolescents and teachers are trained as facilitators Sustainability of outputs Source: Adapted from Zivetz et al., 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 39 box 14Intentionally exploring negative unintended project impacts: a family planning/intimate partner violence project in the Republic of Guinea In 2014, a facility-based youth family planning project in the Republic of Guinea decided to integrate intimate partner violence (IPV) screening as part of a comprehensive package of health services which included care for women who had experienced IPV. This involved training existing family planning staff on using an IPV screening checklist and procedures for referral to IPV services. While programme performance statistics and client interviews showed high levels of IPV screening acceptability and referral uptake, a post-project evaluation of the implementation process that explicitly sought to ascertain unintended consequences revealed that facility staff experienced a high degree of emotional distress due to the “heavy” nature of young women’s IPV stories and the staff’s inability to provide additional support on site. Furthermore, the evaluation also revealed that referrals were often incomplete, the IPV services at the referral sites were often inadequate, and that young clients were reluctant to engage with the referral organizations. Given these findings, the post-project evaluation recommended providing on-site psychosocial counseling for women who experienced IPV, as well as establishing regular group support sessions for staff to deal with the emotional burden of IPV screening and care. Source: Samandari G, et al., 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!40 unless they are intentionally and sensitively questioned and unless respondents are confident their responses will be confidential. Resources:  Outcome harvesting is a participatory process of collecting evidence of what has changed and then, working backwards, determining whether and how an intervention has contributed to these changes. Outcome harvesting has proven to be especially useful in complex situations. For more information, see Better Evaluation’s overview, Outcome harvesting (http://www.betterevaluation.org/en/plan/approach/outcome_ harvesting).  Most significant change (MSC) is a participatory process of engaging stakeholders in data collection and analysis to identify the most significant changes they experienced due to the project. Project stakeholders decide the sorts of changes (impact) that they observe, and analysis of multiple stories can confirm expected outcomes and reveal unanticipated and emerging outcomes. For more information, see Monitoring and Evaluation News’ overview on MSC (http://mande.co.uk/special-issues/most- significant-change-msc/).  Ripple effect mapping (REM) is a participatory approach that works somewhat like a group mind mapping, systematically engaging participants in a visual process to identify outcomes, impacts and unintended consequences, and determine next steps. For more information, see the Journal of Extension 2015 article “Using ripple effect mapping to evaluate program impact: choosing or combining the methods that work best for you” (https://joe.org/joe/2015april/tt1.php). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 41 10 The literature review conducted for this guidance indicated that several cross-cutting project principles relating to adolescents’ particular vulnerabilities and resilience, including principles of positive youth development (PYD), gender equality and human rights, and poverty alleviation, are often ignored or not explicitly discussed in project and evaluation reports (Denno et al., 2015). These cross-cutting factors influence project trajectories as they relate to adolescent health outcomes, and as they relate to conditions of sustainable development for young people and their communities. When these cross-cutting issues are applied to adolescent development projects, they alter the way in which the projects are designed, implemented, monitored and evaluated (UNICEF, 2012, p. 13). Post-project evaluation can play a role in highlighting the importance of such principles and concepts by analysing conceptual gaps and promoting the inclusion of these issues in new projects. For example, if a comprehensive sexuality education project is targeting only older adolescents or only girls, the principles review would create space to address the needs of younger adolescents and boys, as well. 10a. Positive youth development Tip: Assess the extent to which a project contributes to the promotion of PYD. Positive youth development offers a holistic lens upon which to assess projects. PYD engages young people along with their families, communities and governments, to support the empowerment of young people to reach their full potential (see Youth Power’s Positive youth development measurement toolkit below). Positive youth development approaches aim to build skills, assets and competencies; foster healthy relationships; strengthen a supportive environment; and transform systems to be more youth-friendly and to be accountable for young people. Central questions to consider related to PYD during a post-project evaluation include: Assessing cross-cutting project principles T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!42  Assets: Did the project design aim to provide adolescents with the necessary resources and support them to build skills and competencies to achieve ASRH project outcomes?  Agency: Did the project design aim to support adolescents to employ their assets to set their own goals and to make their own decisions about their lives to achieve desired outcomes?  Contribution: Were adolescents meaningfully engaged as a source of change for their own and their community’s positive development?  Enabling environment: Did the project create a more enabling environment for adolescents that develops and supports their assets, their agency, their contribution and their access to services and opportunities? Does it strengthen their ability to avoid risks and to stay safe and secure?1 Resource: For information on how to measure PYD, see Youth Power’s Positive youth development measurement toolkit (2017) (http://www.youthpower.org/resources/positive- youth-development-measurement-toolkit). 1 “Environment” includes social (relationships with peers and adults); normative (attitudes, norms and beliefs); structural (laws, project services and systems); and physical (supportive spaces) factors. Figure 3. Framework of positive youth development illustrating intersections leading to healthy adolescence Source: Adapted from Youth Power, 2017. Enabling environment • Bonding • Opportunities for pro-social involvement • Support • Pro-social norms • Value and recognition • Youth-responsive services • Gender-responsive services • Youth-friendly laws and policies • Gender-responsive laws and policies • Physical safety • Psychological safety • Positive identity • Self-efficacy • Goal-setting/ability to plan ahead • Perseverance and diligence • Positive beliefs about the future Agency • Youth engagement Contribution Healthy, productive and engaged youth Assets • Training • Formal education • Interpersonal skills (social and communication skills) • Higher-order thinking skills • Recognizing emotions • Self-control • Academic achievement P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 43 10b. Promotion of gender equality and human rights Tip: Assess the extent to which the project contributes to the promotion of gender equality and upholding of human rights. Gender norms, roles and relations influence people’s health, well-being and opportunities; as such, projects should understand and acknowledge the influence of gender and the impact of gender inequality on adolescents’ lives. Projects can be distinguished according to the extent to which they address gender. Gender-unequal projects perpetuate gender inequality by reinforcing unbalanced norms, roles and relations. Gender-blind projects ignore gender norms, roles and relations. Gender-sensitive projects consider gender norms, roles and relations, but do not address inequality generated by unequal norms, roles or relations. Gender-transformative projects consider gender norms, roles and relations and address the causes of gender-based health inequities (WHO, 2011). Likewise, projects should employ human rights-based approaches to health, which focus attention and provide strategies and solutions to redress inequalities, discriminatory practices (both real and perceived) and unjust power relations, which are often at the heart of inequitable health outcomes (WHO, 2019). Central gender equality and human rights questions to consider during a post-project evaluation are:  Design: Was the project conception gender-differentiated? Was a situational analysis conducted to understand differences due to gender inequalities? Were other key differences identified (i.e. class, disability, ethnicity, race, poverty and sexual orientation) and an assessment made of how the project would affect adolescents’ respective circumstances, status, opportunities and resources with regard to health and well-being?  Adolescent participation throughout the project: What were the respective roles of adolescents in designing the project? To what extent and with what level of equality were adolescents involved in implementing core project elements, such as having leadership roles in activity implementation? To what extent and with what level of equality were adolescents involved in reviewing progress towards project outcomes?  Monitoring: Do reporting systems disaggregate by age and sex of adolescents? Do reporting systems capture activities of groups beyond adolescents (i.e. parents, teachers, and others) to foster enabling environments? 10c. Poverty alleviation Tip: Assess the extent to which the project contributes to poverty alleviation, as outlined in the Global Strategy for Women’s, Children’s and Adolescents’ Health and the Sustainable Development Goals (SDGs). The vision of the Global Strategy for Women’s, Children’s and Adolescents’ Health is that by 2030, there will be a world in which every woman, child and adolescent in every setting realizes their rights to physical and mental health and well-being, has social and economic opportunities, and is able to participate fully in shaping sustainable and prosperous societies. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!44 Central questions to include in a post-project evaluation (drawn from the Global Strategy for Women’s, Children’s and Adolescents’ Health 2016–2013):  Was the project situated within a poverty-oriented structural environment, such as a national poverty reduction or SDG strategy?  Did the project embrace one or more guiding principles of the SDGs, reflecting understanding of the complexity of action needed to achieve the SDGs: be aligned with development effectiveness and humanitarian norms, and be human rights-based, gender-responsive, partnership-driven, people-informed and accountable?  Did the project contribute to national or subnational poverty-reducing effects? That is, did it contribute to ending preventable deaths, ensuring health and well-being, or expanding enabling environments for young people? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 45 11 To close, it is important to reiterate that post-project evaluation is not the ideal form of project evaluation, with the exception possibly being for the assessment of post-project sustainability and unanticipated and emerging impacts. When post-project evaluation is the only choice for project evaluation, then there is a need to ensure that it is done with as much rigour as possible and that the results and lessons learned are shared. To this end, there is a need to encourage more systematic reflection on the utility and application of post-project evaluation for:  learning about what works and why;  ensuring accountability to key stakeholders including adolescents;  exploring sustainability; and  documenting unanticipated and emerging impacts. 1. Build the monitoring and evaluation capacity of project staff in project and funding agencies to create a broader base of evaluation thinking and action With greater awareness of the importance of evaluation in project design, implementation and assessment of progress and impact and greater expertise in conducting evaluations, project and funding agency staff will develop evaluative thinking and begin to see new opportunities for the use of systematically collected data throughout a project – not only after it ends. How can evaluators, programmers and funders support post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!46 Additionally, evaluators must systematically determine appropriate and feasible roles of adolescents in evaluation and budgets for the resources and space to meaningfully engage young people. Evaluators and project directors can serve as advocates and actors for PYD, including through their engagement in evaluation. 2. Build ASRH project evidence To move the field forward, evaluation activities need to be included and budgeted in ASRH project proposals. When a post-project evaluation is determined to be the optimal option, it is important to understand and be able to speak to its limits as well as opportunities. To facilitate post-project evaluations, we encourage projects to retain data, in addition to reports, after they end. Additionally, post-project evaluations are not easily found in the public domain. Collaborative efforts are needed to disseminate and improve the sharing and use of findings from post- project evaluations. Budget lines should be included to allow time to write up findings of post- project evaluations and to publish them in peer-reviewed journals and popular newsletters. Organizations, including funding agencies and evaluation clearinghouses (see Appendix C), should systematically post evaluation reports, including post-project evaluation reports. Given the potential of post-project evaluation to improve learning and influence future project design and implementation, this guidance seeks to provide ideas and inspiration for future post-project evaluators and funders. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 47 References This list includes source documents that informed the main narrative. It does not include resources already listed below strategies in sections 8 in the main narrative or articles retained for the literature review, which are listed Appendix A. Books Bamburger M, Rugh J, Mabry M. Realworld evaluation: working under budget, time, data, and political constraints (second edition). Thousand Oaks (CA): Sage Publishers; 2012. Bilateral Funding agency documents/evaluation guidelines Austrian Development Agency. Guidelines for project and programme evaluations (final draft translated into English). Vienna: Austrian Development Cooperation/Evaluation Unit; 2009. JICA. Evaluation guidelines 2004. Tokyo: Japan International Cooperation Agency; 2004 (https://www.jica.go.jp/english/our_work/ evaluation/tech_and_grant/guides/pdf/guideline03-02.pdf, accessed 6 May 2019). Journal articles Barua A & Chandra-Mouli V. The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation. International Journal of Adolescent Medicine and Health. 2016;29(6) (doi: https://doi. org/10.1515/ijamh-2016-0024, accessed 6 May 2019). Busza J, Castle S, Diarra A. Trafficking and health. BMJ. 2004; 328(7452): 1369-1371. Chandra-Mouli V, Plesons M, Barua A, Sreenath P, Mehra S. How can collective action between government sectors to prevent child marriage be operationalized? Evidence from a post-hoc evaluation of an intervention in Jamui, Bihar and Sawai Madhopur, Rajasthan in India. Reproductive Health. 2018;15:118. Chandra-Mouli V, Svanemyr J, Amin A, Fogstad H, Say L, Girard F et al. Twenty years after International Conference on Population and Development: where are we with adolescent sexual and reproductive health and rights? Journal of Adolescent Health. 2015;56(1):S1– S6. Denno DM, Hoopes AJ, Chandra-Mouli V. Effective strategies to provide adolescent sexual and reproductive health services and to increase demand and community support. Journal of Adolescent Health. 2015;56(1):S22–S41. Ivanova O, Cordova-Pozo K, Segura ZE, Vega B, Chandra-Mouli V, Hindin MJ et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Evaluation and Program Planning. 2016;58:98–105. (https://doi.org/10.1016/j. evalprogplan.2016.06.007, accessed 6 May 2019). Ryan R, Wortley L, Ní Shé É. Evaluations of post-disaster recovery: a review of practice material. Evidence Base. 2016;1(4):1–33 (https:// www.anzsog.edu.au/resource-library/research/evaluations-of-post-disaster-recovery-a-review-of-practice-material, accessed 6 May 2019). Samandari G, Delamou A, Traore P, et al. Integrating intimate partner violence screening and counseling in a family planning clinic: Evaluation of a pilot project in Conakry, Guinea. African Journal of Reproductive Health. 2016; 20(2):86-93. Van Reeuwijk M & Singh A. Meaningful Youth Participation as a way to achieving success - Results from operational research on meaningful youth participation in a large-scale youth SRHR program in Africa and Asia. Canadian Journal of Children’s Rights. 2018; 5(1). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!48 Technical documents Bamburger M. Conducting quality impact evaluations under budget, time and data constraints. The World Bank Independent Consulting Group. Washington (DC): World Bank; 2006. Brown PR & Hardee K. Transitions to Adulthood in Jamaica: Retrospective Study of Youth.now Participation. Study Report. Kingston, Jamaica and Morristown, New Jersey: PEY & Associates and What Works Association; 2017. Farzaneh N. An evaluation of teenage pregnancy pilot projects in Sierra Leone. New York: UNICEF; 2013 (https://www.unicef.org/ evaldatabase/files/SierraLeone_2013-001_Evaluation_of_Teenage_Pregnancy_Pilot_Projects_in_Sierra_Leone. pdf, accessed 6 May 2019). Mhando NE, Shukia R, Mkumbo K. United Republic of Tanzania/UNICEF evaluation of TUSEME Programme in Tanzania. New York: UNICEF; 2015 (https://www.unicef.org/evaldatabase/files/TUSEME_REPORT_-_Final_(Tanzania_2015-008).pdf , accessed 6 May 2019). Morra-Imas LG & Rist RC. The road to results: designing and conducting effective development evaluations. Washington (DC): International Bank for Reconstruction and Development/World Bank; 2009 (https://openknowledge.worldbank.org/bitstream/ handle/10986/2699/52678.pdf, accessed 6 May 2019). OECD-DAC. Glossary of key terms in evaluation and results-based management. Paris: OECD-DAC; 2010. Puri J, Aladysheva A, Iversen V, Ghorpade Y, Brück T. What methods may be used in impact evaluations of humanitarian assistance? Bonn: The Institute for the Study of Labor (IZA); 2015. Social Cops Academy. The ultimate guide to effective data collection [Using data intelligence for better decision] [website]; 2017 (https://www.udemy.com/user/socialcops/, accessed 6 May 2019). UNICEF. Application of a human rights based approach and gender mainstreaming strategy for work with adolescents in Latin America and the Caribbean. Panama City, Panama: UNICEF Regional Office for Latin America and the Caribbean; 2012. USAID. Evaluation: Performance evaluation of the strengthening pediatric HIV and AIDS services in Tanzania program. Washington DC: USAID; 2015. WHO. Key Issues in the implementation of programmes for adolescent sexual and reproductive health. Geneva: World Health Organization; 2004 (https://apps.who.int/iris/bitstream/handle/10665/69695/WHO_FCH_CAH_04.3_eng.pdf;jsessionid =3C4E0CBAD79AED0C8573FC3CBB6D782D?sequence=1, accessed 6 May 2019). WHO. Gender mainstreaming for health managers: a practical approach. Facilitators’ guide. Participant’s notes. Geneva: World Health Organization; 2011 (https://www.who.int/gender-equity-rights/knowledge/health_managers_guide/en/, accessed 6 May 2019). WHO. Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents. Geneva: World Health Organization; 2018 (http://apps.who.int/iris/bitstream/handle/10665/273792/9789241508414-eng.pdf?ua=1, accessed 6 May 2019). Zivetz L, Cekan, J, Robbins K. Building the evidence base for post project evaluation: case study review and evaluability checklists [a report to the Faster Forward Fund] [website]. Valuing voices; 2017 (http://valuingvoices.com/building-the-evidence-base-for-post- project-evaluation-a-report-to-the-faster-forward-fund/, accessed 6 May 2019). Website articles/blog posts Mindtools. Post implementation reviews: making sure that what you delivered actually works. 2019 (https://www.mindtools.com/ pages/article/newPPM_74.htm, accessed 6 May 2019). OECD. DAC Criteria for evaluating development assistance. Fact sheet. 2000 (http://www.oecd.org/dac/evaluation/49756382.pdf, accessed 6 May 2019). WHO. Gender, equity and human rights. 2019 (https://www.who.int/gender-equity-rights/understanding/human-rights-definition/ en/, accessed 6 May 2019). World Bank. Monitoring and evaluation for better development results. Feature story, 2013 (http://www.worldbank.org/en/news/ feature/2013/02/14/monitoring-and-evaluation-for-better-development-results, accessed 6 May 2019). Youth Power. Positive Youth Development (PYD) Framework. 2017 (https://www.youthpower.org/positive-youth-development-pyd- framework, accessed 15 July 2019). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 49 Appendix A: Analysis report – review of post-project evaluations of ASRH projects To understand ASRH post-project evaluation rationales, challenges and solutions, a review of peer-reviewed and grey literature articles and post-project adolescent and AYSRH evaluation reports was conducted. 1. Methodology to identify evaluation reports retained for this review Using three levels of criteria to search seven evaluation databases, 99 evaluation reports were identified using first-level inclusion criteria (project evaluations, completed between 2010 and 2016, and operating in LMICs). After applying the second-level criteria (ASRH projects and programmes), 41 reports were retained. Each report was then examined using third-level criteria (post-project evaluations), yielding six reports. WHO provided seven additional reports and peer-reviewed articles, which brought the total number to 13 listed at the end of the report. Evaluation databases searched and results  USAID Development Experience Clearinghouse (DEC) (17 AYSRH evaluations identified of which 2 were post-project).  UNICEF evaluation database (24 identified of which 4 fit inclusion criteria).  Active Learning Network for Accountability and Performance in Humanitarian Action (none identified).  Canadian International Development Agency (none identified).  European Union Commission EuropeAid (none identified).  Independent Evaluation Group at the World Bank (none identified).  Poverty Action Lab (none identified). Inclusion criteria to search databases  First-level criteria: programme evaluations posted from 2011–2017 from LMICs (99 reports).  Second-level criteria: AYSRH projects and programmes (41 reports).  Third-level criteria: evaluations that were conducted post-project (6 reports). NB: Search criteria varied by database search functions, but included: Adolescent/Youth/ Adolescent sexual and reproductive health/Adolescent sexual health/Impact assessment/Impact evaluation/Implementation reviews). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!50 Notes on the process 1. Posting of evaluations on all databases is encouraged by funding agencies and clearinghouses but not required; thus, the evaluation report universe is a self-selected one. Only UNICEF applies an additional criterion for posting the evaluations; they must adhere to a specified level of rigour. 2. The search yielded few reports and highlighted a number of database search issues.  Different search terms included in the seven evaluation websites made the search imprecise. Only some of the websites had “youth and adolescent health” as a search term. None of the websites had “post-project evaluation” as a search term.  A total of 17 articles related to youth and adolescent health were identified on USAID’s DEC. However, only two of these 17 articles were post-project evaluations. Additionally, most of the excluded reports were focused on issues other than AYSRH.  A total of 24 articles were identified on UNICEF’s evaluation database from 2011–2017 by using “youth” and “adolescent” as search terms. However, only four of these 24 articles were relevant to sexual and reproductive health, and three of these four articles were post-project evaluations.  No articles that met the inclusion criteria were identified on the evaluation database of the Active Learning Network for Accountability and Performance in Humanitarian Action.  No articles that met the inclusion criteria were identified on the other databases and websites, including those of the Canadian International Development Agency, the European Union Commission, the Independent Evaluation Group of the World Bank and the Poverty Action Lab. 2. Key findings Why was the post-project evaluation done? The rationale for conducting the post-project evaluations varied across the reports. These included: to measure effectiveness, draw out lessons learned, understand how to improve the intervention for future projects, and to gauge the sustainability of the AYSRH project. Who commissioned and conducted the evaluation? The funding agency/technical agency – and sometimes the main implementing organizations – commissioned the evaluations. They then typically engaged external evaluators and/or research/evaluation firms to conduct the evaluations. How were project stakeholders, including young people, engaged? The engagement of stakeholders, including young people, varied across the reports. Eight evaluations reported engaging their stakeholders only as respondents. Two evaluations engaged adult stakeholders in the evaluation process as part of the evaluation team. Three evaluations engaged adolescents in the evaluation process, mostly through consultative meetings and – in one case – as co-evaluators throughout the evaluation process. What were the evaluation approaches? All the evaluations used qualitative approaches to collect their primary data, although some used a mixed methods approach. Only two evaluations employed pre-intervention/post-intervention measures. Several used atypical evaluation approaches including theory-driven evaluation and outcome mapping. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 51 Were there any reflections by evaluators on barriers and limitations of post-project evaluations? Evaluators across all 13 reports identified similar limitations. Recurring themes included: time constraints, small sampling size and lack of primary or secondary data to compare with a baseline or to verify results. In most of the reports, evaluators were not integrated into the project design process from the start, but were instead identified and engaged once the project had already ended. In many of the reports, limited availability of information and time constraints increased the pressure on evaluators, who had to balance the processes of conducting an evaluation and ensuring the quality of the data collection process. Were there any reflections by evaluators on the utility of post-project evaluations? Only a few evaluators explicitly discussed and explained the utility of post-project evaluations. None of the evaluators advocated for the need for this type of evaluation. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!52 Post-project evaluations included in this analysis Barua A & Chandra-Mouli V. The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation. International Journal of Adolescent Medicine and Health. 2016;29(6) (doi: https://doi.org/10.1515/ijamh-2016-0024, accessed 6 May 2019). Bernstein R, Harkins J, Mwanza M, Nandi S. Performance evaluation of the Strengthening Pediatric HIV and AIDS Services in Tanzania Program. USAID; 2015 (http://pdf.usaid.gov/pdf_docs/PA00KJFF.pdf, accessed 7 May 2019). Chandra-Mouli, V, Barua, A, Igras, S. Strengthening collective response of the government to end child marriage through a district level convergence approach in Jamui, Bihar, and Sawai Madhopur, Rajasthan, India. Geneva: WHO; 2016. Collins T, Yusuf Y, Tesfahun H, Dejene S. A rights-based approach to adolescent and youth development, Ethiopia. End of programme evaluation 2013. London: HLSP; 2013 (https://www.unicef.org/evaldatabase/files/Ethiopia_2013-023_ Evaluation_of_Adolescent_Programme-Final_Report_RNE.pdf, accessed 7 May 2019). Farzaneh N. An evaluation of teenage pregnancy pilot projects in Sierra Leone. UNICEF; 2013 (https://www.unicef.org/ evaldatabase/files/SierraLeone_2013-001_Evaluation_of_Teenage_Pregnancy_Pilot_Projects_in_Sierra_Leone. pdf, accessed 6 May 2019). Gotsadze T, Sharipova B, Ashurov S. Youth friendly health services program in Tajikistan 2006–2013: program evaluation report. UNICEF; 2014 (https://www.unicef.org/evaldatabase/files/Tajikistan_Youth_Friendly_Health_ Services_program_in_Tajikistan.pdf , accessed 7 May 2019). Ivanova O, Cordova-Pozo K, Segura ZE, Vega B, Chandra-Mouli V, Hindin MJ et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Evaluation and Program Planning. 2016;58:98–105. (https://doi.org/10.1016/j. evalprogplan.2016.06.007, accessed 6 May 2019). Mhando NE, Shukia R, Mkumbo K. United Republic of Tanzania/UNICEF evaluation of TUSEME Programme in Tanzania. UNICEF; 2015 (https://www.unicef.org/evaldatabase/files/TUSEME_REPORT_-_Final_(Tanzania_2015-008).pdf , accessed 6 May 2019). Rijsdijk L, Bos A, Ruiter R, Leerlooijer J, de Haas B, Schaalma H. The world starts with me: a multilevel evaluation of a comprehensive sex education programme targeting adolescents in Uganda BMC Public Health. 2011;(11):334. Swartz S, Deutsch C, Makoae M, Michel B, Harding J, Garzouzie G et al. Measuring change in vulnerable adolescents: findings from a peer education evaluation in South Africa. Journal of Social Aspects of HIV/AIDS. 2012;9(4):242–254. True Panacea LLC, Endres K, Rao P. ARC TWC evaluation: Guyana and Tanzania comparative report. USAID; 2010 (http:// pdf.usaid.gov/pdf_docs/PDACR615.pdf, accessed 6 May 2019). van Belle S, Marchal B, Dubourg D, Kegels G. How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC Public Health. 2010;(10):741. Van der Geugten J, van Meijel B, den Uhl MH de Vries N. Evaluation of a sexual and reproductive health education programme: students’ knowledge, attitude and behaviour in Bolgatanga Municipality, Northern Ghana. African Journal of Reproductive Health. 2015;19(3):126–36. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 53 Appendix B: Post-project evaluation evaluability checklist An evaluability assessment examines “the extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD DAC, 2010, p. 21). Evaluability assessments help determine if an evaluation is worthwhile to conduct in terms of anticipated benefits and costs and how it can be designed for maximum utility. The purpose of this checklist is to identify gaps in the data available and other shortcomings in the resources required to conduct a post-project evaluation. Knowledge of these limitations can highlight aspects of the evaluation that require mitigation if a post-project evaluation is to move forward. Note there is no minimum threshold of criteria required for proceeding with a post-project evaluation. Deciding whether or not to initiate a post-project evaluation will depend on the context of individual projects. Organizational considerations There is a clear purpose for conducting the post-project evaluation that is understood by all key stakeholders. Typical purposes include: Project and organizational learning: evidence of overall effectiveness evidence of unanticipated or emerging impacts1 evidence for improving project quality going forward or to scale up successful efforts evidence of the sustainability of outcomes/impacts. Promoting accountability for beneficiaries, partners, funders, and other stakeholders. Justifying future funding. The perceived utility of the results warrant the investment in a post-project evaluation. There is a clear opportunity for the results of the post-project evaluation to inform policy or programmatic decision- making or other worthwhile use. There is an explicit plan for the dissemination of findings, including to project participants and partners, and a pathway for organizational learning. There is strong leadership support for the post-project evaluation, as well as for evidence-informed decision-making overall. There is clarity and agreement on evaluation roles and responsibilities among the funder, implementing organization, project staff, other partners and the evaluator(s). 1 Zivetz, Cekan, Robbins (2017) distinguish between unanticipated outcomes (arising from assumptions made in the theory of change) and emerging outcomes (due to the efforts and resources of participants and partners after the project stopped). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!54 information availability There is basic documentation about the project available, which may include any of the following: Original project proposal Situation/context analysis Needs assessment/gap analysis Project description (including key contextual factors) Project design or plan Theory of change or logic model/log frame Monitoring plans Evaluation framework or results framework Work/implementation plans Progress reports or process reviews Endline or closeout reports Exit strategy documentation Formative/process/implementation evaluation reports Can the authors be contacted if more information or clarification is needed? There is a clear statement of the problem that the intervention was designed to address. There are original and evolved project goals and/or objectives. There is an explicit and coherent project theory or theory of change, or one can be reconstructed given the project documentation available. Inputs, activities, outputs, outcomes, impacts are specified and causal linkages are clearly indicated. Key stakeholders agree on the intended outcomes and impacts. It is possible to identify which aspects of the theory of change should be the focus of the post-project evaluation. It is possible to determine measurable outputs, outcomes and key indicators from the theory of change. The approximate start and end dates of the project are known. Individuals with some experience with, or knowledge about, the project are available for consultation. Individuals who can verify the fidelity of implementation to the original project model or any subsequent changes. Individuals who can comment on contextual factors present during and after the project that may have affected implementation and subsequent impacts. Information is available on how the intervention was aligned with other interventions and strategies at the time of operation. There are existing raw or analysed secondary project data available (baseline, midline or other) which may include: Monitoring data Activity records Attendance records Participant records Narrative reports Internal reviews Satisfaction surveys Outputs from project activities, e.g. visual outputs of participatory activities, other developed materials, tools, etc. Population-level data, national statistics Other (list here): There are data available for all sites. There are data available to demonstrate change on key outcomes. The sampling process used is clear. Project participation is clearly defined and distinguishable from nonparticipation. The data can be disaggregated by age, sex, gender, ethnicity, site, etc. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 55 The quality of the existing data appears acceptable for use regarding reliability and validity. Previously used data collection instruments are available for use. Information about other modifying factors that might have interacted with, influenced and contributed to the project’s outcomes post-completion is available (i.e. other development interventions in the area, social, political or economic trends, or climate, conflict, displacement, migration). Data collection Key stakeholders agree on the primary evaluation questions to be addressed in the post-project evaluation. Obtaining primary project data is still feasible even though the project has ended. There is a record of who was involved in what project activities, when and to what extent. Names, roles and contact lists of implementing actors, including project and organizational staff, funders and consultants, are available and potentially contactable. Names and contact lists of beneficiaries are available and potentially contactable. Knowledge of who was excluded (intentionally or unintentionally), and why. Key beneficiaries are willing and able to participate. An appropriate comparison group can be identified and contacted. Project sites are available for data collection where: There was a high level of programming during the project. The site(s) is reflective of the demographics and geography of some or all of the project intervention sites. In or out migration of the site is minimal, or is representative of the project sites overall. There has been minimal activity on the part of other development agencies in the same sectors in the years since the project ended (or something is known about these interventions). Any legacy effects of the implementing agency can be distinguished from the project under evaluation (in the case where a new project followed the one under evaluation). More than one data source (primary or secondary) is available to facilitate use of a mixed methods approach and the triangulation of multiple lines of evidence. The evaluation questions: Align with the areas of interest identified in the theory of change. Are realistic given the amount of primary and secondary data available and resources available to do the post- project evaluation. Explore alternative explanations for observed outcomes or impacts. Evaluation resources There are sufficient resources to employ the desired post-project evaluation methodology. Budget Staffing Equipment Other (list): The implementing organization has the capacity to assist the evaluator with data and other information requests related to the post-project evaluation. Staff and key stakeholders with the necessary skills are likely to be present. Staff and key stakeholders with the necessary history are likely to be present. There is an opportunity for staff and key stakeholders to participate in the design of the post-project evaluation and formulation of evaluation questions. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!56 Timing There is sufficient time available to conduct the post-project evaluation as designed. The results will be available in a timely way for decision-making. The evaluation can be timed to align with the original project end, particularly if seasonality may influence project outcomes and impact (i.e. agricultural and education cycles). Unique post-project evaluation considerations The evaluator can defend their choice of methods for the post-project evaluation. Primary intended users understand the limitations of the evaluation evidence and will view the results as credible. Stakeholders agree on the level of evidence required. Sources Bureau of Conflict and Stabilization Operations (CSO). CSO evaluability assessment checklist (working draft). Washington (DC): United States Department of State; n.d. Davies R. Planning evaluability assessments: a synthesis of the literature with recommendations. Report of a study commissioned by the Department for International Development. London, UK: DFID; 2013 (https://www.gov.uk/ government/uploads/system/uploads/attachment_data/file/248656/wp40-planning-eval-assessments.pdf, accessed 7 May 2019). JICA. Thematic evaluation: analysis for enhancing evaluability of JICA’s cooperation programs final report. Tokyo: Japan International Cooperation Agency; 2015 (https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/ program/thematic/c8h0vm000001rgwp-att/2015_01.pdf, accessed 7 May 2019). OECD-DAC. Glossary of key terms in evaluation and results-based management. Paris: OECD-DAC; 2010. Peersman G, Guijt I, Pasanen T. Evaluability assessment for impact evaluation: guidance, checklists and decision support. London, UK: Overseas Development Institute; 2015 (https://www.odi.org/publications/9442-evaluability- assessment-impact-evaluation-guidance-checklists-and-decision-support, accessed 7 May 2019). UNICEF. UNICEF Strategic plan 2014–2017: evaluability assessment. Final report annexes. New York (NY): UNICEF Evaluation Office; 2015 (https://www.unicef.org/evaldatabase/files/Evaluability_Assesment_of_UNICEFs_Strategic_ Plan_2014-2017_Annexes.pdf, accessed 7 May 2019). Zandniapour L & JBS International. Impact evaluability assessment tool. Washington (DC): Corporation for National Community Service; 2014. Zivetz L, Cekan J, Robbins K. Building the evidence base for post-project evaluation: case study review and evaluability checklists. Washington (DC): Valuing Voices; 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 57 Appendix C: Evaluation websites and clearinghouses SiDA — evaluations Swedish development cooperation Asian Development Bank Independent evaluations at the Asian Development Bank CiDA — evaluations Canadian Aid Agency DANiDA — evaluations Danish Aid Agency European Commission — aid evaluations Aid evaluations of the European Union independent Evaluation Group at the World Bank — Publications The Independent Evaluation Group evaluates the development effectiveness of the World Bank Group. Norad — evaluations Norwegian Aid Agency OECD DAC Evaluation Resource Centre (DEReC) The Development Assistance Committee (DAC) is the principal body through which the OECD deals with issues related to co-operation with developing countries. The Evaluation Resource Centre contains development evaluation reports and guidelines published by the network and its 30 bilateral and multilateral members. OECD DAC ongoing evaluations of development programmes The site for sharing of evaluation plans and work programmes of members of the DAC Network on Development Evaluation, for enhanced transparency and facilitation collaboration. It also provides an overview of current trends in development evaluation. Poverty Action Lab A collection of various impact evaluations, including some short summaries of the conclusions from impact studies on a variety of topics of relevance for development. Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) ALNAP is a unique network that incorporates many of the key humanitarian organizations and experts from across the humanitarian sector, including members from funding agencies, as well as, nongovernmental, Red Cross/Crescent, United Nations and independent/academic organizations. It includes evaluations and lessons learned on humanitarian actions. The Expert Group for Aid Studies (EBA) Independent evaluations of Swedish international development cooperation. UN Evaluation Group (UNEG) Country level evaluation database. USAiD — evaluations The United States Agency for International Development, with links to USAID publications including USAID evaluations. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!58

Department of Reproductive Health and Research World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland Fax: +41 22 791 4171 E-mail: reproductivehealth@who.int Website: www.who.int/reproductivehealth The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978 92 4 151656 3 19072_Sample cover_3.5mm spine.indd 1 08/10/19 08:28

Department of Reproductive Health and Research World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland Fax: +41 22 791 4171 E-mail: reproductivehealth@who.int Website: www.who.int/reproductivehealth The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978 92 4 151656 3 19072_Sample cover_3.5mm spine.indd 1 08/10/19 08:28

The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978-92-4-151656-3 © World Health Organization 2019 Some rights reserved. This work is available under the Creative Commons Attribution-Non Commercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. Suggested citation. The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects. Geneva: World Health Organization; 2019. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at https://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see https://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see https://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Design and layout by L’IV Com Sàrl Printed in Switzerland Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. How was this guidance developed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Who should use this guidance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. What is a post-project evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 5. Why is there a need for post-project evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 6. What are the common reasons for and benefits of conducting post-project evaluations of ASRH projects? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 7 What can you expect when planning and conducting a post-project evaluation? . . 10 8. Steps in planning a post-project evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 9. Common challenges and strategies to address them . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 9a. Defining the purpose and scope of the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 9b. Establishing the post-project evaluation team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 9c. Addressing the lack of data availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 9d. Developing feasible sampling strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 9e. Bringing external context into evaluation consideration . . . . . . . . . . . . . . . . . . . . . . . 35 9f. Assessing sustainability and emerging and unanticipated outcomes . . . . . . . . . . . . 38 10. Assessing cross-cutting project principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 10a. Positive youth development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 10b. Gender equality and human rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 10c. Poverty alleviation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 11. How can evaluators, programmes and funders support post-project evaluation? . . . . 46 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Appendix A: Analysis report – review of post-project evaluations of ASRH projects . . . . . . . 50 Appendix B: Post-project evaluation evaluability checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Appendix C: Evaluation websites and clearinghouses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S i i i Acknowledgements The World Health Organization (WHO) is grateful for the contributions that many individuals and organizations have made to the development of this guidance. This guidance was coordinated by Venkatraman Chandra-Mouli of the WHO Department of Reproductive Health and Research, with support from Marina Plesons and oversight from Lale Say. WHO extends sincere thanks to Susan Igras, who carried out the literature review and developed the first draft of the guidance. WHO is also grateful to participants of a WHO technical consultation for their review and feedback to the first and subsequent versions of the guidance: Susanne Carai, Sarah Castle, Bregje de Kok, Fariyal Fikree, Kylie Hutchinson, Emma Jackson, Rebecka Lundgren, Bella Mansanya, Kristien Michielsen, Ghazaleh Samandari, Shantanu Sharma, Manahil Siddiqi, and Judith Westeneng. Lastly, WHO thanks Vivien Stone for editing the document, L’IV Com for designing the document, and Graham Ogilvie for preparing the illustrations. This work was funded by the John D. and Catherine T. MacArthur Foundation. The views of the funding bodies have not influenced the content of this guidance. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!iv Abbreviations ALNAP Active Learning Network for Accountability and Performance ASRH adolescent sexual and reproductive health AYSRH adolescent and youth sexual reproductive health CA contribution analysis CiDA Canadian International Development Agency DAC Development Assistance Committee (OECD) DFiD Department for International Development iPPF International Planned Parenthood Federation iPv intimate partner violence JiCA Japan International Cooperation Agency LMiCs low- and middle-income countries MSC most significant change OECD Organisation for Economic Co-operation and Development PYD positive youth development QUiP Qualitative Impact Protocol REM ripple effect mapping SDGs Sustainable Development Goals UNiCEF United Nations Children’s Fund USAiD DEC United States Agency for International Development Development Experience Clearinghouse WHO World Health Organization P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S v

Overview 1 In many countries, initiatives that aim to improve adolescent1 sexual and reproductive health (ASRH) implement projects without well thought out plans for evaluation. A range of reasons help explain this phenomenon, including limited training and understanding of the value of evaluation and dedicated evaluation resources within implementing and technical entities. However, the end result is that baselines and endlines may not exist, there may be no comparison of the project area to a non-intervention area, and implementation monitoring data may not provide much information on quality and other less easily counted impacts or results of implementation. The resultant lack of evaluation limits the ability to demonstrate what was achieved, what approaches worked, and what approaches did not work, thus losing out on critical learnings. Nevertheless, managers or funders with an interest in designing future projects or expanding existing projects may later commission a post-project evaluation to assess impact and learning in the months or years after a project has ended. Many ASRH projects encounter and address policy and programmatic challenges, and in doing so, learn valuable lessons. Since careful documentation and rigorous evaluation are the exception rather than the norm, though, the lessons from these efforts are not extracted and placed in the public arena. Post-project evaluations thus have a role to complement prospective studies for new or follow-on projects. However, there is a lack of guidance and literature on post-project evaluations, and technical guidance from funding agencies for conducting post-project evaluation is rare. To begin to bridge this gap more formally, the World Health Organization (WHO) developed this practical guidance for conducting post-project evaluations of ASRH projects. 1 WHO defines adolescents as people between the ages of 10–19 years. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 1 2 A literature review was conducted to understand common post-project evaluation rationales, challenges and solutions. The review included:  peer-reviewed and grey literature articles and post-project evaluation reports of ASRH projects, identified through a search of seven evaluation clearinghouses from low- and middle-income countries (LMICs) between 2010 and 2017 (Appendix A); and  a review of post-project evaluation guidelines published by international organizations, governments, and development agencies. WHO then convened a technical consultation of experts to review and provide inputs to the first draft of the guidance document, which led to revisions and the eventual final document. How was this guidance developed? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!2 3 Aimed primarily at evaluators, this guidance may also be of interest to project managers and funders. Its purpose is to help evaluators to think through the key decisions to be made to yield the most rigorous evaluation possible given available financial and human resources and project documentation. This guidance includes challenges, tips and lessons learned from the literature on post-project evaluations to inspire ideas for future evaluations. Case studies highlight selected issues as well as the responses taken by evaluators to address them. Although focused on post-project evaluation, this guidance is also relevant to evaluators of final project evaluations in situations where evaluation was not planned from the start or where projects have shifted focus midway. Who should use this guidance? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 3 4 There is no common definition of post-project evaluation, and many terms are commonly used (see Box 1). We use a definition informed by the Japan International Cooperation Agency (JICA) because it describes what this type of evaluation is, what its objectives are, and how it is carried out: Post-project evaluations are performed within a certain period after a project is completed and most are not planned while projects are still operating. As staff may no longer be available and project activities are not observable, post-project evaluation is based on existing reports, monitoring records and, other written information, and often includes additional data collection. While such evaluations may assess the extent that projects implemented planned activities and how well outcomes were achieved, given they occur after projects end such evaluations may also focus on longer term impact and sustainability. Post-project evaluations consider retrospectively circumstances surrounding a project and other elements that may have influenced implementation, impact and sustainability of results. The definition focuses on the timing and purpose of post-project evaluation to distinguish it from final or endline evaluations which are conducted as projects are ending – when staff, project counterparts and project beneficiaries are present and many project activities are still observable to assess quality and fidelity of implementation. Figure 1 shows the relationship of evaluation to project planning and implementation, as well as where and how post-project evaluation can support project assessment and learning. While post-project evaluation can answer many typical evaluation questions, they are particularly well placed to determine if project outcomes (such as improvements in adolescent-friendly sexual and reproductive health services or shifts in gender attitudes) remain viable over time when support ends. What is post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!4 box 1Related terms for post-project evaluation • after-action review • ex-post evaluation • forensic evaluation • post-completion evaluation • post-hoc evaluation • post-implementation evaluation • post-implementation review • post-project evaluation • retrospective impact evaluation • sustainability impact evaluation • sustained and emerging impacts evaluation Problem/issue Project design and plan Activities/outputs Outcomes Effects/impact Post-project outcomes/effects Resources/inputs Relevance and appropriateness Does it make sense? • Did the project objectives address the need? • Did the project intervention address the right issues? • Is there (still) a need? Effectiveness Did it work? • Did the project achieve the desired objectives/outcomes? • Was the intervention based on knowledge and research to improve the likelihood of success? Effects/impacts • Was the project worthwhile? • Does it have merit? Efficiency Was it efficient? • What was the relationship between the project’s inputs and outputs? • Could resources have been better used? Process surrounding the interventions Was it well-managed? • Did planning and decision-making processes ensure the project’s success? • Did management processes ensure success? • Did processes for developing activities ensure their success?    Contextual and external factors outside project boundaries that may have influenced the project trajectory    Sustainability and unanticipated and emerging effects • Are the project’s effects lasting? • Was the project as worthwhile and as meritable as thought? Figure 1. The logic of a project and its relationship to key evaluation themes Source: Adapted from Ryan et al., 2016. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 5 5 Evaluation is still variable and evolving as a state responsibility within many governments (World Bank, 2013) and the evaluation of social development projects is still an emergent field of inquiry. Evaluators often operate in contexts where data systems are weak and multiple interventions complicate the assessment of project contributions to outcomes. In an era of heightened demands for accountability by governments and organizations, evaluation increasingly is seen as a key tool to assess the impact of interventions and investments (Morra- Imas & Rist, 2009).  In this context, project evaluations vary widely. A relatively small number of applied research projects employ elegant evaluation designs focused on impact. Many projects do not have resources, evaluation training or a proclivity in a politicized environment to conduct evaluation (World Bank, 2013). Post-project evaluations are even rarer, representing less than 1% of international development projects that are evaluated (Zivetz et al., 2017).  Projects operating in conflict-affected and emergency contexts often remain unevaluated given the special challenges of implementation and short project timeframes (Puri et al., 2015).  There are fewer evaluations of ASRH projects available in evaluation clearinghouses compared with evaluations of other sectors. Of 900 evaluations available in a range of evaluation clearinghouses (see Appendix C), only 41 were focused on adolescent and youth sexual and reproductive health (AYSRH) projects. Of these 41 evaluations, only 6 were post- project AYSRH evaluations. Compared with ASRH projects in high-income countries, there is a paucity of project evaluation from LMICs linked to a theory of change and outcomes, and the body of evidence on good practices is small (WHO, 2004; Chandra-Mouli et al., 2015). Consequently, many ASRH projects do not include evaluation in their project design, or consider it as an afterthought. If a project is exploratory or seems to be performing very well and there is a desire to expand it, only then might evaluation be considered to provide important evidence. However, by this point, there is often no funding for evaluation within project budgets. Why is there a need for post- project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!6 Effective evaluation seeks to assess the merit or worth of projects according to five evaluation criteria – the projects’ relevance, effectiveness, efficiency, impact and sustainability (see Box 2) (JICA, 2004; OECD, 2000). Post-project evaluation is most suitable for assessing impact and sustainability and, possibly, relevance (JICA, 2004). It can also provide insights into effectiveness and efficiency. Rigorous evaluation planned from the initiation of a project should be the norm. Where this is not possible, or even if endline evaluation has occurred, post-project evaluation offers some unique possibilities for learning to:  Explore planned or additional project impacts or effects that may have occurred after the end-of-project evaluation period.  Assess the sustainability of a project’s impacts or effects. Findings from an end-of-project evaluation are not always adequate predictors of sustainability. Post-project evaluation can add to learning about sustainability by:  Understanding what elements of a project were sustained (or not) after the project ended.  Gaining deeper insight into how project-introduced approaches and activities have spread, diffused or replicated through anticipated and unanticipated pathways.  Gaining a better understanding of causal or associated factors related to the above (the why and how).  Identifying unexpected and emerging outcomes that came about after the project’s end. 6 What are the common reasons for and benefits of conducting post-project evaluations of ASRH projects? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 7 box 2  Potentially collecting information on a project’s relevance. Staff, stakeholders and end- users/participants, if still available, may have had more time to reflect on the project and may have additional insights to add regarding its relevance. For ASRH projects post-project evaluation can:  Reach older adolescents and young adults who may offer new and/or different insights into project impact, sustainability of its impact, and its relevance.  Contribute to a better understanding of whether the theory of change mechanisms were valid or not. This can inform the development of promising project approaches and activities and contribute to greater clarity about change mechanisms that lead to improved adolescent health and well-being. Table 1 shows the range of reasons for conducting post-project ASRH evaluations. Criteria for evaluating development assistance Relevance: This seeks to assess the extent to which the activity is suited to the priorities and policies of the target group, recipient and funding agency. Effectiveness: This seeks to assess the extent to which an activity attains its objectives. Efficiency: This seeks to assess the outputs – qualitative and quantitative – in relation to the inputs. It is an economic term that signifies that the activity uses the least costly resources possible in order to achieve the desired results. Impact: This seeks to assess the positive and negative changes produced by a project, intended or unintended. This involves the main impacts and effects – intended and unintended – resulting from the activity on local social, economic, environmental and other development indicators. Sustainability: This seeks to assess the extent to which the benefits of an activity are likely to continue after funding has been withdrawn. Source: Adapted from the Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD), 2000. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!8 Accountability  Did the project meet the expectations of adolescents and other stakeholders and funders?  What difference did the project make in the wider environment in which it operates? Evidence of effect  Did the project achieve its intended outcomes? Justification for future funding  Is the project strategy worthwhile enough to warrant future investment in similar projects?  If so, under what contexts and with what adjustments? Organizational and project learning  What evidence exists for an organization to implement a similar project going forward, or to scale up a successful project?  If so, under which contexts and with what adjustments? Sustainability  Are the project’s effects still visible?  If new structures, services or projects were put into place, are they still operating? Unanticipated or emerging impacts  What unanticipated effects have been observed since the project ended?  Have any unforeseen impacts – positive or negative – emerged since the project ended? Table 1. Typical purposes of post-project ASRH evaluations P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 9 7 Each post-project evaluation situation will be different, with different resources and constraints. As a result, the evaluator will have to determine the most appropriate evaluation design amidst some necessary methodological compromises. Some typical challenges that may be encountered (which are also common in other types of evaluations) include:  Project design documentation is of uneven quality. For example, there is no articulated theory of change or logic framework.  Qualitative descriptions of implementation are not available. For example, project reports are focused on explaining outputs and outcomes rather than processes.  Quantitative project data are patchy. For example, monitoring data were never collected or are not available for review; some critical activities are not included in monitoring systems; or the project has not conducted evaluation or has only done a baseline evaluation. The evaluator will likely have to play a variety of roles in addition to traditional evaluator, such as technical advisor to recreate or refine former project theories of change that will guide evaluation questions, or investigative reporter to uncover project documents or former project staff and partners. The evaluator will likely also play a liaison role between the funding agency and the organization that implemented the project. See sections 8 and 9, which share practical guidance for those planning and implementing a post-project evaluation based on experience and lessons learned on post-project evaluation of ASRH projects. What can you expect when planning and conducting a post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!10 box 3Typical challenges of post-project evaluations: the TUSEME Project in the United Republic of Tanzania The TUSEME Project (“Let’s Speak Out” in Kiswahili) was implemented in the United Republic of Tanzania, and employed a rights-oriented, gender-responsive approach to empower girls to understand and overcome problems that constrain their development, including academic achievement. Initially implemented in seven secondary schools in five regions, TUSEME was expanded to 27 schools and was eventually mainstreamed into the national Secondary Education Development Plan and replicated in 237 secondary schools. A post-project evaluation of the TUSEME clubs was carried out to assess the sustainability of student participation and the impact of participation on girls’ empowerment and acquisition of life skills. However, evaluators faced several challenges:  Framing the evaluation: Due to financial and time constraints, the evaluation methodology could not be designed to test effectiveness with a comparison group (e.g. schools with TUSEME clubs and schools without clubs). As a result, the initial terms of reference were renegotiated to include only schools with TUSEME clubs.  Data availability and sampling: The project had no baseline data. Similarly, other documentation was largely lacking, including a project log frame, monitoring and evaluation reports and financial documents.  Primary data collection: Interviewing adolescents was challenged by the need to find former project participants, and once found, obtain consent from students’ parents and informed assent from students below 18 years. Additionally, many parents were not available to talk with evaluators at the time of the school visit when interviews were conducted. Most who were available did not have children in the clubs. Source: Mhando, 2015. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 11 8 The following steps are similar to the steps taken for an evaluation conducted when projects are still functioning or have just ended. However, one key exception is the critical importance of doing an evaluability assessment at the start to determine the extent to which a post-project evaluation can be conducted reliably and credibly. STEP 1: Define the purpose, scope and initial design of the post-project evaluation. Determine the purpose and scope of the evaluation, including the evaluation questions, with key stakeholders, such as implementation organizations, relevant government ministries/ departments and funding agencies.  What are stakeholder expectations for the evaluation?  What do they want to learn and why?  What is the intended use of the evaluation findings and how will they be disseminated? Consider engaging stakeholders beyond the implementing organization and immediate partners, such as government stakeholders and future users of evaluation findings. Their early inputs can contribute to the credibility of the evaluation and use of findings to improve current and/or future projects. Steps in planning a post- project evaluation T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!12 STEP 2: Conduct a post-project evaluation evaluability assessment. An evaluability assessment can provide critical information from multiple perspectives on the extent to which a post-project evaluation can be conducted reliably and can generate credible results. The evaluability assessment should address:  Organizational considerations to ensure stakeholder consensus on the need for, and utility of, the evaluation to inform policy and programmatic decision-making.  information availability to support evidence-based evaluation.  Methodological considerations to be able to answer evaluation questions.  Resource considerations to ensure sufficient financial, human and material resources are available for evaluation.  Timing considerations to collect and analyse data in a logical and timely manner.  Acceptability considerations to gain stakeholder consensus on the type and level of evidence that is acceptable. An evaluability assessment checklist (see Appendix B) can be a useful tool for this critical activity. STEP 3: Identify information gaps and design a feasible methodology with the maximum rigour possible. Once the evaluation purpose(s) has been specified with project stakeholders and the results of the evaluability assessment are available, it will be possible to make choices and understand what compromises might be necessary regarding the evaluation design. Figure 2 provides a decision tree of possible evaluation design choices based on key learning objectives and information/data possibilities. Who will be involved in the evaluation process? How will adolescents be engaged (e.g. will they have roles besides being respondents, such as members of the evaluation team or as advisors to the evaluation team)? Are theories of change and logic frameworks available and will they need to be adjusted for use in the evaluation? Are data and process documentation available? What are the possibilities for sampling and additional data collection? How will you approach analysis and the interpretation of findings? These common challenges are addressed in detail in section 8. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 13 Figure 2. Decision tree on possible evaluation design choices Impact/ effectiveness Process/ implementation Post-test only with no comparison group Quasi-experimental pre/post-test design with non-random group assignment Post-test only with non-random group assignment Time series design (if multiple data points are available) Pre/post-test design with no comparison group (if only two data points are available) Use qualitative methods to deepen understanding of impact and gauge unintended effects Independent analysis of process data Reconstruct process through: Mixed methods including interviews, focus group discussions, case studies, etc. with key project staff, partners, beneficiaries, and the community. Efficiency could be examined if cost data can be linked to activities Create and analyze process data using administrative/ performance data Where and when feasible, engage young people in the evaluation design, data collection, and interpretation of findings. Sustainability Independent analysis of sustainability data Use mixed methods to collect data on: • Political support • Environmental/ contextual factors • Funding stability • Partnerships • Organizational capacity • Project adaptation • Project weaknesses • Project strengths • Communication • Strategic plan • Sustainability of impact What do you want to learn about the intervention? Supplement with additional data, where possible Did the project collect sustainability data? Did the project collect process data? Can you access administrative/ performance data? Is there adequate data on a group that did not receive the intervention to create a comparison group retrospectively? Can you collect data from both the comparison group and the intervention group? Is there adequate data on the intervention group to create data points retrospectively? Supplement with additional data, where possible Yes No T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!14 STEP 4: Confirm the evaluation methodology with key stakeholders to obtain buy-in for later utilization. It is time to share the post-project evaluation plan – the evaluation purpose, evaluation questions and data sources, methods to analyse existing data and collect and analyse additional data, and dissemination approach – for feedback and to ensure the consensus of project stakeholders. In particular, the plan should be shared with those most likely to use the evaluation findings, whether that is the implementing organization, government or funding agency(ies). A caveat: at this point, it may be necessary to go back to the organization or funding agency commissioning the evaluation to explain that some of the initial evaluation questions are unanswerable with the information and/or resources available. STEP 5: Secure ethical approval for the evaluation. It is critical to ensure the protection of adolescents in any evaluation of an ASRH project (WHO, 2018). The evaluator should secure approval from an institutional review board or a waiver of formal ethical review if the evaluation is considered to be a routine project assessment, and revise the proposal as needed. STEP 6: The evaluation is ready to begin! With a clear plan, consensus from project stakeholders, and ethical approval, data collection is ready to begin. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 15 9 This section shares commonly cited challenges during the evaluation process that are unique or critical to post-project evaluation, drawn from a variety of sources referenced in this document, and suggests how they could be mitigated or overcome. Case studies, drawn from actual evaluations, serve to highlight some of the challenges. This section will not cover challenges common to all evaluations, such as those related to financial, time and/or human resource constraints and those related to navigating political environments. While stakeholders may be interested in learning about cost–effectiveness, it is also not addressed in this guidance. Collecting retrospective, reliable cost data linked to project activities is a significant challenge that requires former project staff to recall the time and effort spent on specific activities and that requires former project finance staff to link finance data with activity data. Common challenges and strategies to address them T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!16 9a. Defining the purpose and scope of the evaluation Defining the evaluation’s purpose and scope, as well as its key boundaries, are essential to designing a good evaluation. Measurable performance indicators, theories of change and a clear definition of beneficiary groups provide the intentions and pathways through which project changes are expected to occur. When the post-project evaluation aims to assess impact, an evaluator’s key concerns revolve around assessing possible association or, where possible, causality; whether a feasible comparison group can be found; and whether baseline and endline evaluations exist. When the evaluation aims to assess sustainability or the project’s implementation process, key concerns revolve, respectively, around retrospectively defining the expected sustainability outcomes and attempting to recreate processes if these are not well described in existing documentation. Challenge: No quantitatively measurable indicators or performance targets were established before or during the project against which to assess change. Strategy: Develop performance indicators using documentation on project objectives and aims, in addition to secondary data from national surveys or similar projects if available. Present the new performance indicators to stakeholders and gain consensus for their use in the evaluation. If there are baseline indicators but no established performance targets, discuss with stakeholders whether changes observed in these indicators from baseline represent reasonable performance for the project. If indicators are not meaningful (i.e. are either too ambitious to establish causality or are not logical within the project context), discuss with stakeholders and develop consensus for redefined performance targets. Challenge: No written theory of change exists to guide understanding of association or causality. if such documentation does exist, it may be outdated given changes that occurred during project implementation. Strategy: Work with available project staff and other stakeholders to develop a theory of change and/or a logic framework. The former explains pathways leading to change, and the latter provides an orderly structure to monitor project implementation. If theories of change and logic frameworks do exist, review and revise them as needed (e.g. if there were shifts in implementation strategies or assumptions underlying implementation). Note, though, that creating a theory of change retrospectively may present an additional limitation in the interpretation of findings. Resource: For suggestions on how to create a theory of change or logic framework retrospectively with project stakeholders see: https://actionevaluation.org/theory-of- actiontheory-of-change-tools-resources/. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 17 box 4Reconstructing the project logic framework: the Tarunya Project in India The Tarunya Project, which was implemented by Engender Health and partners over a 5-year period (2008–2013) in Jharkhand State, India, was designed to:  strengthen health systems and health service provision;  enhance community engagement and links between the services and community activities; and  strengthen ASRH programme leadership and management by the state and district health management teams. WHO was invited to conduct a post-project evaluation and the evaluation team quickly learned that no explicit programme theory of change or logic framework existed, making it impossible to develop suitable evaluation questions. The team collected all of the relevant project documents and used information from multiple sources to construct a logic framework to explain what the project had set out to do and what it had actually achieved. In doing so, the team put effort into ensuring that the logic framework was entirely faithful to the information in the project plans and reports. The logic framework then guided development of a set of evaluation questions related to Tarunya’s design, implementation, outputs, outcomes and impact. Upon presenting the logic model to project staff, they asked, “Where did this come from?” Evaluation team members explained that the project’s own plans and reports were the source of the logic framework. By pointing to the sources of specific information in the logic framework, the team showed project staff that the model was not invented but rather articulated a structure that was never made explicit on paper. Source: Barua & Chandra-Mouli, 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!18 Challenge: Concepts underpinning theories of change, such as “reaching vulnerable adolescents”, were not well defined operationally, making later evaluation decisions on sampling and measurement of project coverage difficult. Strategy: Work with project staff to articulate precise definitions for these concepts, based on their experience and understanding. If the original project staff are not available, identify potential definitions from the literature and interview stakeholders to develop consensus on an operational definition for use in the evaluation. Note, though, that creating a new definition retrospectively may present an additional limitation in the interpretation of findings. Challenge: Stakeholders are interested in assessing impact quantitatively but design options are limited. Under-used but adequate evaluation designs do exist for such post- project evaluations, but they are not always well understood by some evaluators. Strategy: Be prepared, with references, to explain design options to fellow evaluators, as well as the rationale for proposing the selected design. Because comparative designs that include only post-test assessments with actual or statistically constructed comparison groups are not always well understood, the evaluator will need to develop this understanding among stakeholders to secure buy-in for their use. Note, though, that many of these design options require the team to have access to statistical support. Resources:  For a practical overview of different evaluation design options, including post-project evaluation designs, see ALNAP’s Evaluation of humanitarian action guide (Chapter 11 Evaluation designs for answering evaluation questions) (https://www.alnap.org/ system/files/content/resource/files/main/alnap-evaluation-humanitarian-action-2016. pdf).  For a discussion (with helpful examples) of studies having adequate versus plausible versus probable impact, see the 1999 International Journal of Epidemiology seminal article by Habicht et al., “Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact” (https://www.measureevaluation. org/resources/training/capacity-building-resources/m-e-of-hiv-aids-programs-in-india- english/session-5-gis-and-research-evaluation/research-evaluation/Int.%20J.%20 Epidemiol.-1999-Habicht-10-8.pdf).  For a helpful perspective to understand the necessity and utility of design compromises in certain evaluation contexts, see the 2012 AIDS journal article by Laga et al., “Evaluating HIV prevention effectiveness: the perfect as the enemy of the good” (https://www.ncbi.nlm.nih.gov/pubmed/22313952). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 19 box 5Operationally defining “migration” and “trafficking”: a post- project evaluation in Mali In Mali, international agencies and the government requested a post-project evaluation to explain why a cross-border strategy aiming to reduce trafficking of vulnerable adolescents between Mali and the cocoa plantations of Côte d’Ivoire was unsuccessful. The project was guided by the legal definition of human trafficking in Article 3 of the United Nations Protocol to Prevent, Suppress and Punish Trafficking in Persons as: “the recruitment, transportation, transfer, harbouring or receipt of persons, by means of threat or use of force or other forms of coercion, of abduction, of fraud, of deception...to achieve the consent of a person having control over another person, for the pursue of exploitation. Exploitation shall include…forced labour or services, slavery, or practices similar to slavery, servitude…” The challenge was how to operationalize “migration” and “trafficking” in contexts where family members and the young people themselves are routinely placed by intermediaries in work situations as agricultural labourers. While the anti- trafficking programme was designed to return adolescents home, most did not want to be “freed” given the enormous peer pressure to go on migration as a rite of passage. Upon return to their poor, rural villages, most adolescents returned to Côte d’Ivoire to seek work. The evaluation team used appreciative inquiry techniques – focusing on young people’s strengths in complex realities rather than their weaknesses – to operationalize “migration” and “trafficking”, informed by the legal definition. They then used these operational terms to develop culturally appropriate interventions for safe migration by young people. Source: Busza J, et al., 2004. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!20 9b. Establishing the post-project evaluation team Evaluation teams are ideally constructed to ensure diversity of skills needed to undertake the evaluation with sufficient content and contextual knowledge of the project being evaluated. During a typical final project evaluation, the evaluation team can regularly consult with project staff and partners about the project’s history, its context, and any implementation issues. However, in post-project evaluations such people are often not readily available and project content and contextual knowledge need to be defined retroactively. Challenge: An external evaluator, who is not familiar with the project’s history, implementation, and larger operating context, is hired as team lead, and has limited or no formal access to former staff. Strategy: Try to complement the team lead’s external objectivity with team members who can provide necessary project content and contextual knowledge and complementary technical skills. Ideally, these team members would be former project staff or people familiar with the organization and project. If these people are not available, invite evaluators of similar projects or working in a similar geographical area who may have a good understanding of the context. Another strategy would be to invite former staff to serve as a consultative group for the evaluation team. Challenge: While they can provide historical context and detail, team members who “own” or have “owned” the project may also be biased in their long-term recollection and interpretation of events and results. if former project staff or partners become team members, be aware of potential biases they may hold. Strategy: An important starting point is for the team lead to acknowledge that insider bias is normal and establish a sense of mutual accountability to ensure that the evaluation is as evidence-based as possible. The team lead and team members all have an important role to play in “fact checking” and insisting that available evidence (not opinion) is used to support the analyses and findings. When undertaking data collection, it is important to organize the team to avoid situations of friends interviewing friends. Likewise, the team should discuss informed consent and strategies for maintaining anonymity and confidentiality when interviewing former colleagues and friends. Similarly, the team should be cognizant of situations where team members who are former project staff may be vulnerable to risks and potential harms when interviewing former colleagues, such as when interviewing a former supervisor with whom there were tensions. In these situations, it is especially important to build multiple lines of inquiry into data collection to allow for triangulation of more subjective evidence. Identifying more objective sources of information can be helpful in this regard. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 21 box 6Establishing a common understanding of the project: the CERCA Project in the Plurinational State of Bolivia, Ecuador and Nicaragua The Community-Embedded Reproductive Health Care for Adolescents (CERCA) Project was implemented in the Plurinational State of Bolivia, Ecuador and Nicaragua (2011–2014) to test the effectiveness of contextually adapted interventions to prevent teenage pregnancies. Across the three countries, CERCA worked at multiple levels – engaging adolescents, their peers, parents and health care providers. A post-project evaluation, which assessed both the process and the impact of the project, was undertaken to examine if and how CERCA’s design, implementation and regular use of monitoring data to adapt interventions affected the results. With limited project documentation available for review and a diversity of stakeholders involved in CERCA’s design, it was important that CERCA partners participated in the evaluation not only as respondents but also as evaluators who collected and analysed data. When working with project staff to develop retrospectively a project theory of change, it became apparent that different staff understood the project’s implementation process and outcomes/impact quite differently. To address potential bias, the team needed to establish a common understanding of the project through the participatory development of a theory of change. Additionally, the team prompted and facilitated subsequent discussions on how to reduce the potential for bias during data collection and analysis. Source: Ivanova et al., 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!22 Challenge: Adolescents who were associated with the project are not considered for evaluation roles except as respondents. Although it is a recognized as good practice to engage young people in an evaluation of their own project, to ensure the inclusion of adolescents’ perspectives and to contribute to positive youth development, it is rarely done. Strategy: Thinking of adolescents as only evaluation respondents is short-sighted. If they are available, engage former project participants to work with external evaluators in as many phases of the post-project evaluation as possible. At a minimum, if financial resources and time are limited, a well-designed and facilitated meeting with former project participants to seek input on plans for the evaluation can contribute to important adjustments in later evaluation phases. Organizing a subsequent analysis meeting with these adolescents to validate the results of the evaluation would also enrich its findings. Ideally, if resources allow, work with adolescents in the design, implementation and dissemination of the evaluation. They are well placed to develop data collection tools, and their involvement in data collection can yield more truthful data as young informants may be more open to talking about issues related to sexual and reproductive health with other young people with whom they can identify, as compared with adult researchers. Note, though, that adolescents who may be engaged are not necessarily representative of the target population. Careful facilitation is needed to mitigate potential resistance by adults to young people evaluating them and their work. An evaluator may need to advocate to the funding agency to include meaningful adolescent engagement to provide additional support for their inclusion. Resources:  For guidance and facilitator manuals to create conditions for successful youth participation and to train and support young people to conduct qualitative data collection for research and monitoring and evaluation, see Explore: a toolkit for involving young people as researchers in sexual and reproductive health programmes, developed by Rutgers and International Planned Parenthood Federation (IPPF) (https://www.rutgers.international/our-products/tools/explore).  For a practical approach to training and engaging young people in evaluation see Youth participatory evaluation: strategies for engaging young people (Kim Sabo Flores, 2007), which draws on theories of children’s play, evaluation and youth development (https://www.wiley.com/en-us/Youth+Participatory+Evaluation %3A+Strategies+for+Engaging+Young+People-p-9780787983925). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 23 box 7Putting into practice adolescent-adult partnerships: an evaluation in Ethiopia, Kenya, Indonesia and Pakistan In 2015, an evaluation was carried out in Ethiopia, Kenya, Indonesia and Pakistan to explore how local organizations involved young people within their ASRH programmes. Each country’s evaluation team included one adult evaluator and three to eight adolescent and youth co-evaluators. The co-evaluators were trained on the key programme concepts, research ethics, and techniques of conducting interviews and focus group discussions, based on the Rutgers and IPPF Explore toolkit. First, the young evaluators developed their own definition of meaningful youth participation, which formed the basis of their study. Additionally, they reviewed, translated, tested and adapted tools during practice sessions. They gathered information from heads of organizations, project staff, service providers, young people engaged in the programmes, and external stakeholders using interviews and focus group discussions. The young evaluators also observed intervention activities, such as provision of services at youth centres and health clinics, provision of peer education, and community mobilization. The young people were quick to learn. Every evening, the adult researcher organized de-briefing meetings with the research team to discuss the day’s findings and analyse the data. This helped the young evaluators to see any gaps in their data that they could cover the next day. Additionally, the evaluation team tried to de-brief each organization on the key findings of the evaluation to validate conclusions and formulate recommendations together. Source: Van Reeuwijk M & Singh A, 2018. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!24 Challenge: Where appropriate, the evaluation team lead needs to ensure parental/ caregiver approval as well as adolescent safeguarding during participation in evaluation activities in a context where former project structures and protection mechanisms no longer exist. Strategy: To ensure ethical and acceptable approaches are used to safeguard participants, obtain the advice of research committees in the area. To obtain parental/ caregiver consent, provide a written summary of the evaluation, describe in what contexts and for how long the adolescent will be engaged, and seek oral or written consent from their parent/caregiver. If it is very challenging to find parents/caregivers, it may be appropriate to engage school principals or other leaders to reach parents/ caregivers. In addition, the team should agree to child safety standards, such as ensuring minors safely move between the evaluation site and home, or having adult members accompany minors if they collect data or share findings in contexts that may place them in harm’s way. If the project was implemented by a child-focused organization, use their child protection guidelines to guide the evaluation safeguarding activities. Resources:  For recommendations on how to protect and support young evaluators engaged in ASRH project evaluation from harm, see Rutgers’s Explore toolkit (Part II: Creating conditions for successful youth participation in research) (https://www.rutgers. international/sites/rutgersorg/files/pdf/Explore-instructions_web.pdf).  For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/hand le/10665/273792/9789241508414-eng.pdf?ua=1).  For ethical considerations of participatory evaluation that place evaluators in the context of community power dynamics, beyond human subject protections, see Banks et al. 2013 article in Contemporary Social Science, “Everyday ethics in community- based participatory research” (https://www.tandfonline.com/doi/pdf/10.1080/21582 041.2013.769618). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 25 9c. Addressing the lack of data availability Data availability, or the lack thereof, is one of the most common issues that evaluators confront in post-project evaluations, since reliance on existing documentation is critical to the evaluation task. Challenges, such as those below, focus on issues of finding key informants from a past project, accessing usable quantitative outcome data and process documentation, managing recall issues, and interviewing adolescents whose age and cognition levels due to maturation no longer reflect the project population. Suggestions for sampling and viable qualitative impact evaluation methods are discussed in later sub-sections. Challenge: Key project personnel are not available due to migration, transfers to new projects, or other reasons after the project has ended. Strategy: Attempt to track down telephone numbers via colleagues and other community contacts and conduct phone interviews with former staff. Ask current employers if former project staff could attend a half-day or full-day meeting during which post-project data could be collected. If this is not possible, try to interview former staff in less formal settings such as coffee shops. If this is not possible, interview staff and people within the host organization and its implementation partners who have some knowledge of the former project. Challenge: Key adolescent beneficiaries and beneficiary lists are not available. Strategy: Young people are often an especially mobile population and the evaluator may need to get creative to find young people who participated in the project. Identify existing structures such as schools or youth clubs where past beneficiaries might be located. Ask school principals or other managers to contact beneficiaries on your behalf. If structures are still active, sample from still-functioning groups formerly supported by the project – comparing those groups with groups or individuals who never participated in the project. If it is not possible to contact former beneficiaries, consider collecting information from people who are familiar with the former beneficiaries. Challenge: There are no quantitative baseline, midline or endline data and using recall information is the best option to assess change. Strategy: Use participant recall data to assess impact in areas such as personal knowledge, attitudes and behaviour changes. For example, use Likert scales with statements that ask participants to assess the extent of change between two time points. Incorporate similar-but-different statements (e.g. two different statements about the same attitude) to assess variability in responses to check recall bias. To help distinguish context from factors more directly affected by the project, refer to timelines of internal and external events during key informant interviews to remind team members of the larger context. Likewise, interviewing senior management may be useful, as people in senior management positions often operate with the larger context in mind. Triangulate recall data with other available data. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!26 box 8Finding adolescents 14 years after the project’s end: the Pro.Star Project in Jamaica Some 14 years after the Pro.Star Project in Jamaica ended, evaluators used a case control study design to retrospectively assess the effects of the project, a school-based and parent education intervention on adolescents’ transition to adulthood. Intent on finding former students to learn how they had progressed in the 14 years post-project, evaluators had to use multiple means to find former participants, including internet, social media (Facebook and WhatsApp), and word of mouth to locate students. Teachers, guidance counsellors and current students helped evaluators to locate past programme participants – reaching out, for example, to their uncles, brothers, aunts and parents. On occasion, guidance counsellors accompanied interviewers to locate persons they had trouble finding. This proved effective because people were more trusting when credible community members accompanied interviewers. Once students were located, evaluators asked them to share names and contact information of other students. Determination and patience in tracking down respondents and flexibility in fitting data collection activities into their schedules were critical. Many of the students had moved out of the community – or, in fact, out of Jamaica – so interviewers conducted some interviews by phone. The results of this outreach were impressive – it was possible to locate and interview over 60% of sampled former students (155 participants) either in person or by telephone or email interviews. Source: Brown PR & Hardee K, 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 27 box 9Using a new evaluation design when the original design is not feasible: an innovations partnership project in Sierra Leone Recognizing that adolescent pregnancy in Sierra Leone is driven more by sociocultural than service access factors, UNICEF established a partnership with Child Fund, Save the Children, the Council of Churches of Sierra Leone, Restless Development, and the Bangladesh Rural Advancement Committee. Together, these organizations implemented five pilot projects in seven districts with the aim to address underlying drivers of adolescent pregnancy in the country. The post-project evaluation aimed to foster learning about the actors, contexts, and challenges of preventing/reducing adolescent pregnancy to inform future programming. Although each partner conducted a baseline, it was not possible to use findings from these baselines as a benchmark for assessing change or impact of the pilot projects due to inconsistencies in project approaches and survey methodologies across the five pilot projects. Instead, the evaluators selected an outcome mapping evaluation approach to assess the different dimensions of change in mindsets, contexts, attitudes and perceptions of individuals and groups that influence motivations, interactions, relationships and behaviour across the five pilot projects. It also explored the contributions of the pilot projects in supporting empowerment and increasing access to and use of sexual and reproductive health services. Source: Farzaneh, 2013. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!28 Challenge: The evaluator wants to align post-project data collection with data from earlier evaluations, but survey instruments and data are not comparable across survey points (baseline, midline, endline and post-project). Strategy: Build on existing survey instruments by making small adaptations that permit comparisons of most/all key outcomes over time. While questions may differ slightly (i.e. an early instrument may assess one specific area of parent-child communication, which is replaced in a later survey instrument to explore a different parent-child communication area) qualified interpretation of findings between time points is possible. If the evolved project aim means that adjusting questions from past survey instruments is not possible, map out key thematic domains currently addressed by the project and develop new questions to evaluate the domains. New data collection using survey methods opens the possibility of using statistical methods to answer many evaluation questions (i.e. factor analysis to determine associations of exposure to the intervention and outcomes, multiple logistics regression to determine the likelihood of change or influence due to project exposure). In the end, however, it may not be possible to undertake a quantitative study that will yield expected results. In such cases seek a different approach that contains a level of rigour to allow for the establishment of plausible causality. Resources: Going forward, recent funding agency mandates, once they are widely instituted, will make datasets open source and thus available to evaluators for additional analyses to answer questions posed in a post-project evaluation. In the meantime, for open access data on adolescents living in LMICs, see the GIRL Center Adolescent Data Hub (https://www.popcouncil.org/girlcenter/research/hub). Challenge: The project is heavily focused on adolescent outcomes and neglects to collect information on other outcome areas, such as improved parenting skills or improved teacher pedagogy. Strategy: It may be useful to review training and supervision reports. As these are produced at a lower level of the reporting system, they may include information on adults and their engagement and reactions to the project. Alternatively, such information gaps can be corrected by sampling adults in new data collection. Challenge: Post-project beneficiary movements, the ageing of adolescents, and other confounding factors associated with the passage of time make it difficult to gain meaningful information that can be generalized through a population-based study. Strategy: Consider using alternative sampling to suit the evaluation aim. For example, consider snowball sampling if the evaluation aims to access people of the same social group, diversity sampling if the evaluation aims to assess differences in sub-groups, or purposive sampling if the evaluation aims to intentionally study selected groups. Resource: For an overview of different sampling strategies, see Changing Minds’ overview on choosing a sampling method (http://changingminds.org/explanations/ research/sampling/choosing_sampling.htm). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 29 box 10Re-purposing existing project data for post-project evaluation: the Baylor Pediatric AIDS Initiative in the United Republic of Tanzania In August 2008 Baylor University began a seven-year project in the United Republic of Tanzania to strengthen services for prevention, early detection, treatment and care of pediatric HIV/AIDS in the Southern Highlands and the Lake zones. A post-project evaluation was carried out to assess the extent to which the project’s goals and objectives were achieved and to provide guidance and lessons learned for future projects, in particular related to effectiveness and sustainability. As an applied research project, the evaluation team had access to ample documentation and databases from the project period. Samples for quantitative data analysis, including a comparison group, were derived from databases of activities from the relatively larger and more complete population samples of the Baylor International Pediatric AIDS Initiative that evaluators were able to analyse in the way they wanted. Although this led to potential sampling bias, it negated the need for new data collection. Primary qualitative data were collected by the evaluation team through key informant interviews, focus group discussions and observations, and were used to supplement, complement and verify quantitative analyses. Source: USAID, 2015. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!30 Challenge: The maturation effect (i.e. ageing of adolescents who were enrolled in a project at an earlier age) introduces comparability biases in post-project evaluation instruments and data, such as in their interest in sexual experimentation and awareness of the larger social environment. Strategy: Ideally, create an unexposed group for comparison. If this is not possible, design post-project data collection to allow for the comparison of older adolescents with younger ones. For example, create two age cohorts: one mimicking the initial unexposed participant group as they entered the project, and the other representing older adolescents who participated in the project. Alternatively, include recall questions for older adolescent participants to inquire about changes they have observed between pre-project and post-project time points. Challenge: Data found in annual or other external reports of the project’s process or accomplishments do not appear realistic, are not sufficiently disaggregated, or do not align with outputs and outcomes identified in the logic framework. Strategy: If data seem unrealistic, try to verify information from project monitoring data. If not available, query the organization’s senior managers to understand what was reported and confirm its reliability. For example, critical contextual information on implementation, such as a large number of new staff recruitments, may not be included in the report but is necessary to understand the findings. If output and outcome data are not available in reports or are not sufficiently disaggregated for evaluation purposes, try to work backwards by reviewing monitoring data or other internal reports. If monitoring data sets are available and data are disaggregated by sex and age, consider doing additional analyses. If disaggregated data are unavailable, undertake new post-project data collection to elicit potentially different outcomes based on adolescent age or sex. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 31 9d. Developing feasible sampling strategies Sampling poses particular challenges for post-project evaluators, ranging from selecting the site to establishing a comparison group to selecting respondents. Without project outreach structures, it may be challenging to identify adolescent beneficiaries. Time and money, of course, also direct what sample is feasible, including whether or not to include a comparison group. When working with vulnerable adolescents, an assessment of the benefits and risks of contacting such individuals should also influence sampling decisions. When constraints do exist, it is important to acknowledge potential bias in the analysis. Challenge: Site selection is done in consultation with government officials or implementing partner staff, which may introduce bias. Alternatively, sites are selected based on real- life considerations, such as security issues or ease in reaching communities. Strategy: Be aware of site selection biases and advocate for transparency in decision- making and unbiased site selection as much as possible. Try to independently assess characteristics of the selected sample, including similarities between the intervention and comparison groups regarding the adolescent population, services, and other relevant infrastructure such as schools. Secondary data may be helpful in assessing comparability. Challenge: There is no comparison group or geographic area to control for the influence of confounding external factors. Strategy: If funding and time are available, create a comparison group or area as part of new data collection processes. Alternatively, use secondary data for comparison, if they are of reasonable quality, such as national survey results or service statistics. Assess for similarities between the intervention and comparison groups regarding the adolescent population, services, and relevant infrastructure such as schools. While preferable to include a comparison group, in certain circumstances it may not be desirable for ethical or logistical reasons. Challenge: The post-project evaluation needs to sample vulnerable adolescents and their parents/caregivers, or hard-to-access respondents, such as undocumented migrants or refugees. Strategy: Firstly, it is essential to conduct an upfront ethical risk/benefit analysis using a “do no harm” framework for evaluations of projects that reached vulnerable adolescents, such as adolescents living with HIV or sex workers. In addition to assuring child/adolescent protection, it is important to train and support interviewers in protection concepts and establish processes for data collection. Pay careful attention to issues such as site selection, sampling and interviewing vulnerable adolescents. To identify and find vulnerable adolescents, work with former project staff or staff from current outreach projects who serve the vulnerable group or an external advisory group, such as a research committee, to determine an appropriate and ethical way to identify and invite respondents to participate in the evaluation. It may be possible to create a sampling frame using network mapping or snowball sampling techniques. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!32 box 11Weighing the risks and benefits of establishing a comparison group: a theoretical post-project evaluation in South-East Asia Sometimes creating a comparison group is not desirable for ethical reasons. In a post-project evaluation of a programme for sex workers in South-East Asia, a conscious decision might be made not to create a comparison group because of the following considerations:  The research might be limited in time and financial resources.  It might be difficult to identify and gain access to a comparison group.  The analyses would need to correct for selection bias, given that participation in the intervention would have been voluntary. Sex workers who had not participated would likely differ in largely unobservable characteristics from those who did.  There would be no clear indication that the intervention would continue, thus participation in the intervention would not be able to be offered to the women in the comparison group. For these reasons, the evaluation team might decide that any benefit of a control group would be outweighed by the risk of exposing women to the community as sex workers. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 33 Ensure ethical considerations for privacy are met. Social media and use of WhatsApp can allow for discrete outreach, but may lead to bias in the sample since not all adolescents have access to these technologies. Ensure that evaluation objectives include budget line items for these types of inquiries to ensure they are adequately addressed. Resource: For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/han dle/10665/273792/9789241508414-eng.pdf?ua=1). Challenge: Former project structures that facilitated the logistics of finding vulnerable adolescent beneficiaries (and their parents/caregivers for provision of informed consent) are no longer available. Strategy: Contact the project’s implementing partners to share information about the post-project evaluation and to set up meetings with former beneficiaries. Ask leaders or key contacts from former implementation partners, such as school principals, to inform parents/caregivers of an upcoming evaluation visit and seek their consent for their child’s participation. (Note: This represents an opt-out approach; that is, parental consent is assumed unless the parents/caregivers indicate they do not want their child to participate. Although this is far from ideal and introduces important ethical considerations, in specific circumstances it may be the only viable approach to ensure parents/caregivers are both informed and that they provide consent.) Set up meetings to interview stakeholders – including adolescents – in discrete ways that do not put them in harm’s way of greater visibility or discrimination. Invite parents/caregivers when inviting adolescents to be interviewed in order to secure consent, as well as to increase the response rates of both adolescents and parents/caregivers. Resources:  For practical approaches to engage young people including vulnerable adolescents, as respondents, see working paper The ethics of social research with children and families in young lives: practical experiences (http://www.younglives.org.uk/sites/www. younglives.org.uk/files/YL-WP53-Morrow-EthicsOfResearchWithChildren.pdf).  For guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health with adolescents, see the WHO Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents (http://apps.who.int/iris/bitstream/hand le/10665/273792/9789241508414-eng.pdf?ua=1). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!34 9e. Bringing external context into evaluation consideration External factors influence project trajectories. It is thus important to consider other projects, policies, and macro-environmental developments that may have influenced a project trajectory. As such, a post-project evaluation often needs to fill in missing contextual information to improve evidence of effectiveness and sustainability. Challenge: information on contextual factors to help explain the nature and trajectory of change is not available in the project documentation. There is little reference to situational analyses or anticipated risks. Project indicators are not linked to these contextual factors, either within the project cycle or after the project has ended. Strategy: Seek out a variety of information sources. Consult key informants for information on contextual factors, such as new policy developments or political forces that may have affected the project, but be cautious about potential biases. If possible, review pertinent information available from media sources, including news articles and other media records and Twitter feeds. Existing documentation (i.e. synthesis reviews, policy analyses) may provide contextual information. Try to recreate internal and external event timelines with available project or implementing partner staff and stakeholders to document factors that may have influenced implementation. Consider engaging with ethnographers to visit sites and employ ethnographic methods to explore and document contextual factors. It may also be useful to employ less-traditional evaluation frameworks; for example, if sufficient time and resources are available, a realist evaluation approach, based on developing explicit understanding of how context relates to outcomes, may help guide information gathering to address these contextual issues in non-biased ways. Resources:  For an explanation of how to create an events timeline, see Better Evaluation’s overview of timelines and time-ordered matrices (http://www.betterevaluation.org/ en/evaluation-options/timelines).  For information on using realist evaluation, see Better Evaluation’s overview of realist evaluation (https://www.betterevaluation.org/en/approach/realist_evaluation). Challenge: Other development projects have launched during project implementation or between the end of the project and the post-project evaluation, complicating the plausible attribution of findings and/or influencing post-project evaluation findings. Strategy: Use evaluation reports and assessment studies from the other projects, if available, to validate anecdotal information. If evaluation reports from the other projects are available, try to map overlapping activities. Try to determine, via an events timeline discussion, to what degree projects from different organizations supported or undermined each other’s effects between the end of the project and the post-project evaluation. Interview former staff and partners to inquire if prior projects and partnerships extending before the project years were important building blocks to the project under evaluation and include this as a component of the context documentation. Methods such as contribution analysis (CA) and Qualitative Impact Protocol (QUIP) provide a systematic approach to assess causal questions and infer plausible attribution in real-life project P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 35 box 12Gaining understanding of context to explain project outcomes: the strengthening collective response of the government to end child marriage through a district level convergence approach project in India The MAMTA-Health Institute for Mother and Child, New Delhi, in partnership with district administrations, undertook a 3-year project to support a cross-departmental convergence approach to synchronize government efforts to end child marriage in Sawai Madhopur, Rajasthan, and Jamui, Bihar, in India. As a convergence or systems change approach, it was important to understand external factors that could have helped or hindered project efforts to change institutional processes to work across departments during project implementation and in the time between project closing and the post- project evaluation. The creation of an events timeline, supplemented by secondary information, was critical to interpret the significant differences in programme outcomes in the two districts. The evaluation team worked with former project staff to create a timeline of internal and external project events, such as changes in child marriage policy at national and state levels and overlap with a state-sponsored programme to end child marriage in one state (Rajasthan) that began mid-project and worked across service departments (a similar approach to the project of interest). This information led to new questions about environmental forces, and the team subsequently supplemented and confirmed information about these events through a review of policy documents and stakeholder interviews. Source: Chandra-Mouli et al., 2018. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!36 evaluations in complex settings. Be willing to state that it is not possible to determine plausible attribution. Resources:  For information on CA – a qualitative technique that guides managers, researchers and policy-makers to a consensus on the contribution their project has made to particular outcomes through better understanding of why results were achieved (or not) and the roles played by the project and other factors – see Better Evaluation’s overview of contribution analysis (www.betterevaluation.org/en/plan/approach/ contribution_analysis).  For information on QUIP – a qualitative technique to evaluate a project implemented in a context of complexity that helps explain how changes can be attributed to different stakeholders or events – see the Assessing Rural Transformations’ QUIP guidelines for field use (http://www.bath.ac.uk/cds/projects-activities/assessing-rural- transformations/documents/complete-quip-guidelines.pdf). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 37 9f. Assessing sustainability and emerging and/or unanticipated outcomes Sustainability is almost always a desired impact and often a funding agency requirement, and post-project evaluations are ideally suited to assess this. Post-project evaluation also provides the possibility to explore whether any other outcomes occurred after the close of the project (Zivetz et al., 2017). These outcomes can be distinguished between unanticipated outcomes (arising from assumptions made in the theory of change) and emerging outcomes (due to efforts and resources of participants and partners after the project ended). Challenge: Sustainability outcomes are not well-defined in project documents but are a post-project evaluation aim. Strategy: Discuss with available staff whether exit or graduation strategies – planned project efforts to close out or transfer implementation support to another entity – existed in the project under evaluation, and what benchmarks or indicators signalled the end of, phasing out of, or reduction of project support. Work with available project staff and other stakeholders to create an operational definition of sustainability that can be used in planning and conducting the evaluation. One approach is to develop a hierarchy of sustainability (i.e. differentiations between sustained project inputs, outcomes or impacts) to define the original project’s sustainability aims. This will simplify the process of determining which methods can be employed to assess sustainability (e.g. observation to determine if activities and use of project materials continue in the post-project context). Resources:  For an illustrative example of a sustainability hierarchy that can guide evaluator-staff discussions on developing an operational definition of sustainability for a project under evaluation, see Valuing Voices Building the evidence base for post-project evaluation (also see Box 13) (http://valuingvoices.com/wp-content/uploads/2013/11/The-case- for-post-project-evaluation-Valuing-Voices-Final-2017.pdf).  For a practical overview of issues in evaluating the sustainability of health projects (including definitions and types of sustainability, specifications and measurements of dependent variables, definitions of independent variables or factors that influence sustainability, and suggestions for designs for research and data collection), see “An agenda for research on the sustainability of public health programs” (https://www. ncbi.nlm.nih.gov/pmc/articles/PMC3222409). Challenge: Anecdotal information indicates that unanticipated and/or emerging outcomes have occurred and need to be part of the evaluation frame to document lessons learned for future projects. Strategy: Several participatory evaluation techniques (listed below) have been developed to address such challenges regarding the wider impact of a project and plausible causality with a degree of rigour. Data collection and evaluation tools should be designed to search explicitly for unanticipated negative outcomes. While positive outcomes are easily shared by respondents, negative outcomes may not be captured T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!38 box 13Defining sustainability of an illustrative ASRH project using a hierarchy of results framework In this example, evaluators and stakeholder developed a hierarchy of sustainability to guide evaluation of an after-school club intervention for very young adolescents. Project stakeholders first defined sustainability of outputs, outcomes and impacts (y-axis). They also differentiated between types of impacts (x-axis) (i.e. whether project-expected achievements were sustained or whether broader sustainability impacts emerged that could be discerned only over a longer timeframe; that is, after a project ends). In addition, Zivetz et al. note there are two broader areas of sustainability to assess: emerging outcomes consider how participants used their own means to carry project initiatives forward, which could produce learning on incentivizing sustainability. This is in contrast to unanticipated outcomes, which are related to the project’s theory of change and may reveal the extent of and reasons why assumptions or objectives deviated from what was anticipated in the project design. Sustained impacts Emerging impacts N/A ASRH and gender equality improved within school structures Teachers are more comfortable teaching sexuality education. Adolescent financial literacy improved Sustainability of impacts Parent-teacher-student committees are maintained, with support from the national government After-school clubs are maintained with continued active engagement of adolescents and teachers Adolescents do small activities for cash to support continuation of the clubs Sustainability of outcomes Parent-teacher-student committees are in place and trained After-school clubs are in place and operational Adolescents and teachers are trained as facilitators Sustainability of outputs Source: Adapted from Zivetz et al., 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 39 box 14Intentionally exploring negative unintended project impacts: a family planning/intimate partner violence project in the Republic of Guinea In 2014, a facility-based youth family planning project in the Republic of Guinea decided to integrate intimate partner violence (IPV) screening as part of a comprehensive package of health services which included care for women who had experienced IPV. This involved training existing family planning staff on using an IPV screening checklist and procedures for referral to IPV services. While programme performance statistics and client interviews showed high levels of IPV screening acceptability and referral uptake, a post-project evaluation of the implementation process that explicitly sought to ascertain unintended consequences revealed that facility staff experienced a high degree of emotional distress due to the “heavy” nature of young women’s IPV stories and the staff’s inability to provide additional support on site. Furthermore, the evaluation also revealed that referrals were often incomplete, the IPV services at the referral sites were often inadequate, and that young clients were reluctant to engage with the referral organizations. Given these findings, the post-project evaluation recommended providing on-site psychosocial counseling for women who experienced IPV, as well as establishing regular group support sessions for staff to deal with the emotional burden of IPV screening and care. Source: Samandari G, et al., 2016. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!40 unless they are intentionally and sensitively questioned and unless respondents are confident their responses will be confidential. Resources:  Outcome harvesting is a participatory process of collecting evidence of what has changed and then, working backwards, determining whether and how an intervention has contributed to these changes. Outcome harvesting has proven to be especially useful in complex situations. For more information, see Better Evaluation’s overview, Outcome harvesting (http://www.betterevaluation.org/en/plan/approach/outcome_ harvesting).  Most significant change (MSC) is a participatory process of engaging stakeholders in data collection and analysis to identify the most significant changes they experienced due to the project. Project stakeholders decide the sorts of changes (impact) that they observe, and analysis of multiple stories can confirm expected outcomes and reveal unanticipated and emerging outcomes. For more information, see Monitoring and Evaluation News’ overview on MSC (http://mande.co.uk/special-issues/most- significant-change-msc/).  Ripple effect mapping (REM) is a participatory approach that works somewhat like a group mind mapping, systematically engaging participants in a visual process to identify outcomes, impacts and unintended consequences, and determine next steps. For more information, see the Journal of Extension 2015 article “Using ripple effect mapping to evaluate program impact: choosing or combining the methods that work best for you” (https://joe.org/joe/2015april/tt1.php). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 41 10 The literature review conducted for this guidance indicated that several cross-cutting project principles relating to adolescents’ particular vulnerabilities and resilience, including principles of positive youth development (PYD), gender equality and human rights, and poverty alleviation, are often ignored or not explicitly discussed in project and evaluation reports (Denno et al., 2015). These cross-cutting factors influence project trajectories as they relate to adolescent health outcomes, and as they relate to conditions of sustainable development for young people and their communities. When these cross-cutting issues are applied to adolescent development projects, they alter the way in which the projects are designed, implemented, monitored and evaluated (UNICEF, 2012, p. 13). Post-project evaluation can play a role in highlighting the importance of such principles and concepts by analysing conceptual gaps and promoting the inclusion of these issues in new projects. For example, if a comprehensive sexuality education project is targeting only older adolescents or only girls, the principles review would create space to address the needs of younger adolescents and boys, as well. 10a. Positive youth development Tip: Assess the extent to which a project contributes to the promotion of PYD. Positive youth development offers a holistic lens upon which to assess projects. PYD engages young people along with their families, communities and governments, to support the empowerment of young people to reach their full potential (see Youth Power’s Positive youth development measurement toolkit below). Positive youth development approaches aim to build skills, assets and competencies; foster healthy relationships; strengthen a supportive environment; and transform systems to be more youth-friendly and to be accountable for young people. Central questions to consider related to PYD during a post-project evaluation include: Assessing cross-cutting project principles T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!42  Assets: Did the project design aim to provide adolescents with the necessary resources and support them to build skills and competencies to achieve ASRH project outcomes?  Agency: Did the project design aim to support adolescents to employ their assets to set their own goals and to make their own decisions about their lives to achieve desired outcomes?  Contribution: Were adolescents meaningfully engaged as a source of change for their own and their community’s positive development?  Enabling environment: Did the project create a more enabling environment for adolescents that develops and supports their assets, their agency, their contribution and their access to services and opportunities? Does it strengthen their ability to avoid risks and to stay safe and secure?1 Resource: For information on how to measure PYD, see Youth Power’s Positive youth development measurement toolkit (2017) (http://www.youthpower.org/resources/positive- youth-development-measurement-toolkit). 1 “Environment” includes social (relationships with peers and adults); normative (attitudes, norms and beliefs); structural (laws, project services and systems); and physical (supportive spaces) factors. Figure 3. Framework of positive youth development illustrating intersections leading to healthy adolescence Source: Adapted from Youth Power, 2017. Enabling environment • Bonding • Opportunities for pro-social involvement • Support • Pro-social norms • Value and recognition • Youth-responsive services • Gender-responsive services • Youth-friendly laws and policies • Gender-responsive laws and policies • Physical safety • Psychological safety • Positive identity • Self-efficacy • Goal-setting/ability to plan ahead • Perseverance and diligence • Positive beliefs about the future Agency • Youth engagement Contribution Healthy, productive and engaged youth Assets • Training • Formal education • Interpersonal skills (social and communication skills) • Higher-order thinking skills • Recognizing emotions • Self-control • Academic achievement P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 43 10b. Promotion of gender equality and human rights Tip: Assess the extent to which the project contributes to the promotion of gender equality and upholding of human rights. Gender norms, roles and relations influence people’s health, well-being and opportunities; as such, projects should understand and acknowledge the influence of gender and the impact of gender inequality on adolescents’ lives. Projects can be distinguished according to the extent to which they address gender. Gender-unequal projects perpetuate gender inequality by reinforcing unbalanced norms, roles and relations. Gender-blind projects ignore gender norms, roles and relations. Gender-sensitive projects consider gender norms, roles and relations, but do not address inequality generated by unequal norms, roles or relations. Gender-transformative projects consider gender norms, roles and relations and address the causes of gender-based health inequities (WHO, 2011). Likewise, projects should employ human rights-based approaches to health, which focus attention and provide strategies and solutions to redress inequalities, discriminatory practices (both real and perceived) and unjust power relations, which are often at the heart of inequitable health outcomes (WHO, 2019). Central gender equality and human rights questions to consider during a post-project evaluation are:  Design: Was the project conception gender-differentiated? Was a situational analysis conducted to understand differences due to gender inequalities? Were other key differences identified (i.e. class, disability, ethnicity, race, poverty and sexual orientation) and an assessment made of how the project would affect adolescents’ respective circumstances, status, opportunities and resources with regard to health and well-being?  Adolescent participation throughout the project: What were the respective roles of adolescents in designing the project? To what extent and with what level of equality were adolescents involved in implementing core project elements, such as having leadership roles in activity implementation? To what extent and with what level of equality were adolescents involved in reviewing progress towards project outcomes?  Monitoring: Do reporting systems disaggregate by age and sex of adolescents? Do reporting systems capture activities of groups beyond adolescents (i.e. parents, teachers, and others) to foster enabling environments? 10c. Poverty alleviation Tip: Assess the extent to which the project contributes to poverty alleviation, as outlined in the Global Strategy for Women’s, Children’s and Adolescents’ Health and the Sustainable Development Goals (SDGs). The vision of the Global Strategy for Women’s, Children’s and Adolescents’ Health is that by 2030, there will be a world in which every woman, child and adolescent in every setting realizes their rights to physical and mental health and well-being, has social and economic opportunities, and is able to participate fully in shaping sustainable and prosperous societies. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!44 Central questions to include in a post-project evaluation (drawn from the Global Strategy for Women’s, Children’s and Adolescents’ Health 2016–2013):  Was the project situated within a poverty-oriented structural environment, such as a national poverty reduction or SDG strategy?  Did the project embrace one or more guiding principles of the SDGs, reflecting understanding of the complexity of action needed to achieve the SDGs: be aligned with development effectiveness and humanitarian norms, and be human rights-based, gender-responsive, partnership-driven, people-informed and accountable?  Did the project contribute to national or subnational poverty-reducing effects? That is, did it contribute to ending preventable deaths, ensuring health and well-being, or expanding enabling environments for young people? P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 45 11 To close, it is important to reiterate that post-project evaluation is not the ideal form of project evaluation, with the exception possibly being for the assessment of post-project sustainability and unanticipated and emerging impacts. When post-project evaluation is the only choice for project evaluation, then there is a need to ensure that it is done with as much rigour as possible and that the results and lessons learned are shared. To this end, there is a need to encourage more systematic reflection on the utility and application of post-project evaluation for:  learning about what works and why;  ensuring accountability to key stakeholders including adolescents;  exploring sustainability; and  documenting unanticipated and emerging impacts. 1. Build the monitoring and evaluation capacity of project staff in project and funding agencies to create a broader base of evaluation thinking and action With greater awareness of the importance of evaluation in project design, implementation and assessment of progress and impact and greater expertise in conducting evaluations, project and funding agency staff will develop evaluative thinking and begin to see new opportunities for the use of systematically collected data throughout a project – not only after it ends. How can evaluators, programmers and funders support post-project evaluation? T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!46 Additionally, evaluators must systematically determine appropriate and feasible roles of adolescents in evaluation and budgets for the resources and space to meaningfully engage young people. Evaluators and project directors can serve as advocates and actors for PYD, including through their engagement in evaluation. 2. Build ASRH project evidence To move the field forward, evaluation activities need to be included and budgeted in ASRH project proposals. When a post-project evaluation is determined to be the optimal option, it is important to understand and be able to speak to its limits as well as opportunities. To facilitate post-project evaluations, we encourage projects to retain data, in addition to reports, after they end. Additionally, post-project evaluations are not easily found in the public domain. Collaborative efforts are needed to disseminate and improve the sharing and use of findings from post- project evaluations. Budget lines should be included to allow time to write up findings of post- project evaluations and to publish them in peer-reviewed journals and popular newsletters. Organizations, including funding agencies and evaluation clearinghouses (see Appendix C), should systematically post evaluation reports, including post-project evaluation reports. Given the potential of post-project evaluation to improve learning and influence future project design and implementation, this guidance seeks to provide ideas and inspiration for future post-project evaluators and funders. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 47 References This list includes source documents that informed the main narrative. It does not include resources already listed below strategies in sections 8 in the main narrative or articles retained for the literature review, which are listed Appendix A. Books Bamburger M, Rugh J, Mabry M. Realworld evaluation: working under budget, time, data, and political constraints (second edition). Thousand Oaks (CA): Sage Publishers; 2012. Bilateral Funding agency documents/evaluation guidelines Austrian Development Agency. Guidelines for project and programme evaluations (final draft translated into English). Vienna: Austrian Development Cooperation/Evaluation Unit; 2009. JICA. Evaluation guidelines 2004. Tokyo: Japan International Cooperation Agency; 2004 (https://www.jica.go.jp/english/our_work/ evaluation/tech_and_grant/guides/pdf/guideline03-02.pdf, accessed 6 May 2019). Journal articles Barua A & Chandra-Mouli V. The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation. International Journal of Adolescent Medicine and Health. 2016;29(6) (doi: https://doi. org/10.1515/ijamh-2016-0024, accessed 6 May 2019). Busza J, Castle S, Diarra A. Trafficking and health. BMJ. 2004; 328(7452): 1369-1371. Chandra-Mouli V, Plesons M, Barua A, Sreenath P, Mehra S. How can collective action between government sectors to prevent child marriage be operationalized? Evidence from a post-hoc evaluation of an intervention in Jamui, Bihar and Sawai Madhopur, Rajasthan in India. Reproductive Health. 2018;15:118. Chandra-Mouli V, Svanemyr J, Amin A, Fogstad H, Say L, Girard F et al. Twenty years after International Conference on Population and Development: where are we with adolescent sexual and reproductive health and rights? Journal of Adolescent Health. 2015;56(1):S1– S6. Denno DM, Hoopes AJ, Chandra-Mouli V. Effective strategies to provide adolescent sexual and reproductive health services and to increase demand and community support. Journal of Adolescent Health. 2015;56(1):S22–S41. Ivanova O, Cordova-Pozo K, Segura ZE, Vega B, Chandra-Mouli V, Hindin MJ et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Evaluation and Program Planning. 2016;58:98–105. (https://doi.org/10.1016/j. evalprogplan.2016.06.007, accessed 6 May 2019). Ryan R, Wortley L, Ní Shé É. Evaluations of post-disaster recovery: a review of practice material. Evidence Base. 2016;1(4):1–33 (https:// www.anzsog.edu.au/resource-library/research/evaluations-of-post-disaster-recovery-a-review-of-practice-material, accessed 6 May 2019). Samandari G, Delamou A, Traore P, et al. Integrating intimate partner violence screening and counseling in a family planning clinic: Evaluation of a pilot project in Conakry, Guinea. African Journal of Reproductive Health. 2016; 20(2):86-93. Van Reeuwijk M & Singh A. Meaningful Youth Participation as a way to achieving success - Results from operational research on meaningful youth participation in a large-scale youth SRHR program in Africa and Asia. Canadian Journal of Children’s Rights. 2018; 5(1). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!48 Technical documents Bamburger M. Conducting quality impact evaluations under budget, time and data constraints. The World Bank Independent Consulting Group. Washington (DC): World Bank; 2006. Brown PR & Hardee K. Transitions to Adulthood in Jamaica: Retrospective Study of Youth.now Participation. Study Report. Kingston, Jamaica and Morristown, New Jersey: PEY & Associates and What Works Association; 2017. Farzaneh N. An evaluation of teenage pregnancy pilot projects in Sierra Leone. New York: UNICEF; 2013 (https://www.unicef.org/ evaldatabase/files/SierraLeone_2013-001_Evaluation_of_Teenage_Pregnancy_Pilot_Projects_in_Sierra_Leone. pdf, accessed 6 May 2019). Mhando NE, Shukia R, Mkumbo K. United Republic of Tanzania/UNICEF evaluation of TUSEME Programme in Tanzania. New York: UNICEF; 2015 (https://www.unicef.org/evaldatabase/files/TUSEME_REPORT_-_Final_(Tanzania_2015-008).pdf , accessed 6 May 2019). Morra-Imas LG & Rist RC. The road to results: designing and conducting effective development evaluations. Washington (DC): International Bank for Reconstruction and Development/World Bank; 2009 (https://openknowledge.worldbank.org/bitstream/ handle/10986/2699/52678.pdf, accessed 6 May 2019). OECD-DAC. Glossary of key terms in evaluation and results-based management. Paris: OECD-DAC; 2010. Puri J, Aladysheva A, Iversen V, Ghorpade Y, Brück T. What methods may be used in impact evaluations of humanitarian assistance? Bonn: The Institute for the Study of Labor (IZA); 2015. Social Cops Academy. The ultimate guide to effective data collection [Using data intelligence for better decision] [website]; 2017 (https://www.udemy.com/user/socialcops/, accessed 6 May 2019). UNICEF. Application of a human rights based approach and gender mainstreaming strategy for work with adolescents in Latin America and the Caribbean. Panama City, Panama: UNICEF Regional Office for Latin America and the Caribbean; 2012. USAID. Evaluation: Performance evaluation of the strengthening pediatric HIV and AIDS services in Tanzania program. Washington DC: USAID; 2015. WHO. Key Issues in the implementation of programmes for adolescent sexual and reproductive health. Geneva: World Health Organization; 2004 (https://apps.who.int/iris/bitstream/handle/10665/69695/WHO_FCH_CAH_04.3_eng.pdf;jsessionid =3C4E0CBAD79AED0C8573FC3CBB6D782D?sequence=1, accessed 6 May 2019). WHO. Gender mainstreaming for health managers: a practical approach. Facilitators’ guide. Participant’s notes. Geneva: World Health Organization; 2011 (https://www.who.int/gender-equity-rights/knowledge/health_managers_guide/en/, accessed 6 May 2019). WHO. Guidance on ethical considerations in planning and reviewing research studies on sexual and reproductive health in adolescents. Geneva: World Health Organization; 2018 (http://apps.who.int/iris/bitstream/handle/10665/273792/9789241508414-eng.pdf?ua=1, accessed 6 May 2019). Zivetz L, Cekan, J, Robbins K. Building the evidence base for post project evaluation: case study review and evaluability checklists [a report to the Faster Forward Fund] [website]. Valuing voices; 2017 (http://valuingvoices.com/building-the-evidence-base-for-post- project-evaluation-a-report-to-the-faster-forward-fund/, accessed 6 May 2019). Website articles/blog posts Mindtools. Post implementation reviews: making sure that what you delivered actually works. 2019 (https://www.mindtools.com/ pages/article/newPPM_74.htm, accessed 6 May 2019). OECD. DAC Criteria for evaluating development assistance. Fact sheet. 2000 (http://www.oecd.org/dac/evaluation/49756382.pdf, accessed 6 May 2019). WHO. Gender, equity and human rights. 2019 (https://www.who.int/gender-equity-rights/understanding/human-rights-definition/ en/, accessed 6 May 2019). World Bank. Monitoring and evaluation for better development results. Feature story, 2013 (http://www.worldbank.org/en/news/ feature/2013/02/14/monitoring-and-evaluation-for-better-development-results, accessed 6 May 2019). Youth Power. Positive Youth Development (PYD) Framework. 2017 (https://www.youthpower.org/positive-youth-development-pyd- framework, accessed 15 July 2019). P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 49 Appendix A: Analysis report – review of post-project evaluations of ASRH projects To understand ASRH post-project evaluation rationales, challenges and solutions, a review of peer-reviewed and grey literature articles and post-project adolescent and AYSRH evaluation reports was conducted. 1. Methodology to identify evaluation reports retained for this review Using three levels of criteria to search seven evaluation databases, 99 evaluation reports were identified using first-level inclusion criteria (project evaluations, completed between 2010 and 2016, and operating in LMICs). After applying the second-level criteria (ASRH projects and programmes), 41 reports were retained. Each report was then examined using third-level criteria (post-project evaluations), yielding six reports. WHO provided seven additional reports and peer-reviewed articles, which brought the total number to 13 listed at the end of the report. Evaluation databases searched and results  USAID Development Experience Clearinghouse (DEC) (17 AYSRH evaluations identified of which 2 were post-project).  UNICEF evaluation database (24 identified of which 4 fit inclusion criteria).  Active Learning Network for Accountability and Performance in Humanitarian Action (none identified).  Canadian International Development Agency (none identified).  European Union Commission EuropeAid (none identified).  Independent Evaluation Group at the World Bank (none identified).  Poverty Action Lab (none identified). Inclusion criteria to search databases  First-level criteria: programme evaluations posted from 2011–2017 from LMICs (99 reports).  Second-level criteria: AYSRH projects and programmes (41 reports).  Third-level criteria: evaluations that were conducted post-project (6 reports). NB: Search criteria varied by database search functions, but included: Adolescent/Youth/ Adolescent sexual and reproductive health/Adolescent sexual health/Impact assessment/Impact evaluation/Implementation reviews). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!50 Notes on the process 1. Posting of evaluations on all databases is encouraged by funding agencies and clearinghouses but not required; thus, the evaluation report universe is a self-selected one. Only UNICEF applies an additional criterion for posting the evaluations; they must adhere to a specified level of rigour. 2. The search yielded few reports and highlighted a number of database search issues.  Different search terms included in the seven evaluation websites made the search imprecise. Only some of the websites had “youth and adolescent health” as a search term. None of the websites had “post-project evaluation” as a search term.  A total of 17 articles related to youth and adolescent health were identified on USAID’s DEC. However, only two of these 17 articles were post-project evaluations. Additionally, most of the excluded reports were focused on issues other than AYSRH.  A total of 24 articles were identified on UNICEF’s evaluation database from 2011–2017 by using “youth” and “adolescent” as search terms. However, only four of these 24 articles were relevant to sexual and reproductive health, and three of these four articles were post-project evaluations.  No articles that met the inclusion criteria were identified on the evaluation database of the Active Learning Network for Accountability and Performance in Humanitarian Action.  No articles that met the inclusion criteria were identified on the other databases and websites, including those of the Canadian International Development Agency, the European Union Commission, the Independent Evaluation Group of the World Bank and the Poverty Action Lab. 2. Key findings Why was the post-project evaluation done? The rationale for conducting the post-project evaluations varied across the reports. These included: to measure effectiveness, draw out lessons learned, understand how to improve the intervention for future projects, and to gauge the sustainability of the AYSRH project. Who commissioned and conducted the evaluation? The funding agency/technical agency – and sometimes the main implementing organizations – commissioned the evaluations. They then typically engaged external evaluators and/or research/evaluation firms to conduct the evaluations. How were project stakeholders, including young people, engaged? The engagement of stakeholders, including young people, varied across the reports. Eight evaluations reported engaging their stakeholders only as respondents. Two evaluations engaged adult stakeholders in the evaluation process as part of the evaluation team. Three evaluations engaged adolescents in the evaluation process, mostly through consultative meetings and – in one case – as co-evaluators throughout the evaluation process. What were the evaluation approaches? All the evaluations used qualitative approaches to collect their primary data, although some used a mixed methods approach. Only two evaluations employed pre-intervention/post-intervention measures. Several used atypical evaluation approaches including theory-driven evaluation and outcome mapping. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 51 Were there any reflections by evaluators on barriers and limitations of post-project evaluations? Evaluators across all 13 reports identified similar limitations. Recurring themes included: time constraints, small sampling size and lack of primary or secondary data to compare with a baseline or to verify results. In most of the reports, evaluators were not integrated into the project design process from the start, but were instead identified and engaged once the project had already ended. In many of the reports, limited availability of information and time constraints increased the pressure on evaluators, who had to balance the processes of conducting an evaluation and ensuring the quality of the data collection process. Were there any reflections by evaluators on the utility of post-project evaluations? Only a few evaluators explicitly discussed and explained the utility of post-project evaluations. None of the evaluators advocated for the need for this type of evaluation. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!52 Post-project evaluations included in this analysis Barua A & Chandra-Mouli V. The Tarunya Project’s efforts to improve the quality of adolescent reproductive and sexual health services in Jharkhand state, India: a post-hoc evaluation. International Journal of Adolescent Medicine and Health. 2016;29(6) (doi: https://doi.org/10.1515/ijamh-2016-0024, accessed 6 May 2019). Bernstein R, Harkins J, Mwanza M, Nandi S. Performance evaluation of the Strengthening Pediatric HIV and AIDS Services in Tanzania Program. USAID; 2015 (http://pdf.usaid.gov/pdf_docs/PA00KJFF.pdf, accessed 7 May 2019). Chandra-Mouli, V, Barua, A, Igras, S. Strengthening collective response of the government to end child marriage through a district level convergence approach in Jamui, Bihar, and Sawai Madhopur, Rajasthan, India. Geneva: WHO; 2016. Collins T, Yusuf Y, Tesfahun H, Dejene S. A rights-based approach to adolescent and youth development, Ethiopia. End of programme evaluation 2013. London: HLSP; 2013 (https://www.unicef.org/evaldatabase/files/Ethiopia_2013-023_ Evaluation_of_Adolescent_Programme-Final_Report_RNE.pdf, accessed 7 May 2019). Farzaneh N. An evaluation of teenage pregnancy pilot projects in Sierra Leone. UNICEF; 2013 (https://www.unicef.org/ evaldatabase/files/SierraLeone_2013-001_Evaluation_of_Teenage_Pregnancy_Pilot_Projects_in_Sierra_Leone. pdf, accessed 6 May 2019). Gotsadze T, Sharipova B, Ashurov S. Youth friendly health services program in Tajikistan 2006–2013: program evaluation report. UNICEF; 2014 (https://www.unicef.org/evaldatabase/files/Tajikistan_Youth_Friendly_Health_ Services_program_in_Tajikistan.pdf , accessed 7 May 2019). Ivanova O, Cordova-Pozo K, Segura ZE, Vega B, Chandra-Mouli V, Hindin MJ et al. Lessons learnt from the CERCA project, a multicomponent intervention to promote adolescent sexual and reproductive health in three Latin America countries: a qualitative post-hoc evaluation. Evaluation and Program Planning. 2016;58:98–105. (https://doi.org/10.1016/j. evalprogplan.2016.06.007, accessed 6 May 2019). Mhando NE, Shukia R, Mkumbo K. United Republic of Tanzania/UNICEF evaluation of TUSEME Programme in Tanzania. UNICEF; 2015 (https://www.unicef.org/evaldatabase/files/TUSEME_REPORT_-_Final_(Tanzania_2015-008).pdf , accessed 6 May 2019). Rijsdijk L, Bos A, Ruiter R, Leerlooijer J, de Haas B, Schaalma H. The world starts with me: a multilevel evaluation of a comprehensive sex education programme targeting adolescents in Uganda BMC Public Health. 2011;(11):334. Swartz S, Deutsch C, Makoae M, Michel B, Harding J, Garzouzie G et al. Measuring change in vulnerable adolescents: findings from a peer education evaluation in South Africa. Journal of Social Aspects of HIV/AIDS. 2012;9(4):242–254. True Panacea LLC, Endres K, Rao P. ARC TWC evaluation: Guyana and Tanzania comparative report. USAID; 2010 (http:// pdf.usaid.gov/pdf_docs/PDACR615.pdf, accessed 6 May 2019). van Belle S, Marchal B, Dubourg D, Kegels G. How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC Public Health. 2010;(10):741. Van der Geugten J, van Meijel B, den Uhl MH de Vries N. Evaluation of a sexual and reproductive health education programme: students’ knowledge, attitude and behaviour in Bolgatanga Municipality, Northern Ghana. African Journal of Reproductive Health. 2015;19(3):126–36. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 53 Appendix B: Post-project evaluation evaluability checklist An evaluability assessment examines “the extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD DAC, 2010, p. 21). Evaluability assessments help determine if an evaluation is worthwhile to conduct in terms of anticipated benefits and costs and how it can be designed for maximum utility. The purpose of this checklist is to identify gaps in the data available and other shortcomings in the resources required to conduct a post-project evaluation. Knowledge of these limitations can highlight aspects of the evaluation that require mitigation if a post-project evaluation is to move forward. Note there is no minimum threshold of criteria required for proceeding with a post-project evaluation. Deciding whether or not to initiate a post-project evaluation will depend on the context of individual projects. Organizational considerations There is a clear purpose for conducting the post-project evaluation that is understood by all key stakeholders. Typical purposes include: Project and organizational learning: evidence of overall effectiveness evidence of unanticipated or emerging impacts1 evidence for improving project quality going forward or to scale up successful efforts evidence of the sustainability of outcomes/impacts. Promoting accountability for beneficiaries, partners, funders, and other stakeholders. Justifying future funding. The perceived utility of the results warrant the investment in a post-project evaluation. There is a clear opportunity for the results of the post-project evaluation to inform policy or programmatic decision- making or other worthwhile use. There is an explicit plan for the dissemination of findings, including to project participants and partners, and a pathway for organizational learning. There is strong leadership support for the post-project evaluation, as well as for evidence-informed decision-making overall. There is clarity and agreement on evaluation roles and responsibilities among the funder, implementing organization, project staff, other partners and the evaluator(s). 1 Zivetz, Cekan, Robbins (2017) distinguish between unanticipated outcomes (arising from assumptions made in the theory of change) and emerging outcomes (due to the efforts and resources of participants and partners after the project stopped). T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!54 information availability There is basic documentation about the project available, which may include any of the following: Original project proposal Situation/context analysis Needs assessment/gap analysis Project description (including key contextual factors) Project design or plan Theory of change or logic model/log frame Monitoring plans Evaluation framework or results framework Work/implementation plans Progress reports or process reviews Endline or closeout reports Exit strategy documentation Formative/process/implementation evaluation reports Can the authors be contacted if more information or clarification is needed? There is a clear statement of the problem that the intervention was designed to address. There are original and evolved project goals and/or objectives. There is an explicit and coherent project theory or theory of change, or one can be reconstructed given the project documentation available. Inputs, activities, outputs, outcomes, impacts are specified and causal linkages are clearly indicated. Key stakeholders agree on the intended outcomes and impacts. It is possible to identify which aspects of the theory of change should be the focus of the post-project evaluation. It is possible to determine measurable outputs, outcomes and key indicators from the theory of change. The approximate start and end dates of the project are known. Individuals with some experience with, or knowledge about, the project are available for consultation. Individuals who can verify the fidelity of implementation to the original project model or any subsequent changes. Individuals who can comment on contextual factors present during and after the project that may have affected implementation and subsequent impacts. Information is available on how the intervention was aligned with other interventions and strategies at the time of operation. There are existing raw or analysed secondary project data available (baseline, midline or other) which may include: Monitoring data Activity records Attendance records Participant records Narrative reports Internal reviews Satisfaction surveys Outputs from project activities, e.g. visual outputs of participatory activities, other developed materials, tools, etc. Population-level data, national statistics Other (list here): There are data available for all sites. There are data available to demonstrate change on key outcomes. The sampling process used is clear. Project participation is clearly defined and distinguishable from nonparticipation. The data can be disaggregated by age, sex, gender, ethnicity, site, etc. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 55 The quality of the existing data appears acceptable for use regarding reliability and validity. Previously used data collection instruments are available for use. Information about other modifying factors that might have interacted with, influenced and contributed to the project’s outcomes post-completion is available (i.e. other development interventions in the area, social, political or economic trends, or climate, conflict, displacement, migration). Data collection Key stakeholders agree on the primary evaluation questions to be addressed in the post-project evaluation. Obtaining primary project data is still feasible even though the project has ended. There is a record of who was involved in what project activities, when and to what extent. Names, roles and contact lists of implementing actors, including project and organizational staff, funders and consultants, are available and potentially contactable. Names and contact lists of beneficiaries are available and potentially contactable. Knowledge of who was excluded (intentionally or unintentionally), and why. Key beneficiaries are willing and able to participate. An appropriate comparison group can be identified and contacted. Project sites are available for data collection where: There was a high level of programming during the project. The site(s) is reflective of the demographics and geography of some or all of the project intervention sites. In or out migration of the site is minimal, or is representative of the project sites overall. There has been minimal activity on the part of other development agencies in the same sectors in the years since the project ended (or something is known about these interventions). Any legacy effects of the implementing agency can be distinguished from the project under evaluation (in the case where a new project followed the one under evaluation). More than one data source (primary or secondary) is available to facilitate use of a mixed methods approach and the triangulation of multiple lines of evidence. The evaluation questions: Align with the areas of interest identified in the theory of change. Are realistic given the amount of primary and secondary data available and resources available to do the post- project evaluation. Explore alternative explanations for observed outcomes or impacts. Evaluation resources There are sufficient resources to employ the desired post-project evaluation methodology. Budget Staffing Equipment Other (list): The implementing organization has the capacity to assist the evaluator with data and other information requests related to the post-project evaluation. Staff and key stakeholders with the necessary skills are likely to be present. Staff and key stakeholders with the necessary history are likely to be present. There is an opportunity for staff and key stakeholders to participate in the design of the post-project evaluation and formulation of evaluation questions. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!56 Timing There is sufficient time available to conduct the post-project evaluation as designed. The results will be available in a timely way for decision-making. The evaluation can be timed to align with the original project end, particularly if seasonality may influence project outcomes and impact (i.e. agricultural and education cycles). Unique post-project evaluation considerations The evaluator can defend their choice of methods for the post-project evaluation. Primary intended users understand the limitations of the evaluation evidence and will view the results as credible. Stakeholders agree on the level of evidence required. Sources Bureau of Conflict and Stabilization Operations (CSO). CSO evaluability assessment checklist (working draft). Washington (DC): United States Department of State; n.d. Davies R. Planning evaluability assessments: a synthesis of the literature with recommendations. Report of a study commissioned by the Department for International Development. London, UK: DFID; 2013 (https://www.gov.uk/ government/uploads/system/uploads/attachment_data/file/248656/wp40-planning-eval-assessments.pdf, accessed 7 May 2019). JICA. Thematic evaluation: analysis for enhancing evaluability of JICA’s cooperation programs final report. Tokyo: Japan International Cooperation Agency; 2015 (https://www.jica.go.jp/english/our_work/evaluation/tech_and_grant/ program/thematic/c8h0vm000001rgwp-att/2015_01.pdf, accessed 7 May 2019). OECD-DAC. Glossary of key terms in evaluation and results-based management. Paris: OECD-DAC; 2010. Peersman G, Guijt I, Pasanen T. Evaluability assessment for impact evaluation: guidance, checklists and decision support. London, UK: Overseas Development Institute; 2015 (https://www.odi.org/publications/9442-evaluability- assessment-impact-evaluation-guidance-checklists-and-decision-support, accessed 7 May 2019). UNICEF. UNICEF Strategic plan 2014–2017: evaluability assessment. Final report annexes. New York (NY): UNICEF Evaluation Office; 2015 (https://www.unicef.org/evaldatabase/files/Evaluability_Assesment_of_UNICEFs_Strategic_ Plan_2014-2017_Annexes.pdf, accessed 7 May 2019). Zandniapour L & JBS International. Impact evaluability assessment tool. Washington (DC): Corporation for National Community Service; 2014. Zivetz L, Cekan J, Robbins K. Building the evidence base for post-project evaluation: case study review and evaluability checklists. Washington (DC): Valuing Voices; 2017. P R A C T I CA L G U I DA N C E F O R CO N D U C T I N G P O S T- P R OJ E C T E VA LUAT I O N S O F A D O L E S C E N T S E X UA L A N D R E P R O D U C T I V E H E A LT H P R OJ E C T S 57 Appendix C: Evaluation websites and clearinghouses SiDA — evaluations Swedish development cooperation Asian Development Bank Independent evaluations at the Asian Development Bank CiDA — evaluations Canadian Aid Agency DANiDA — evaluations Danish Aid Agency European Commission — aid evaluations Aid evaluations of the European Union independent Evaluation Group at the World Bank — Publications The Independent Evaluation Group evaluates the development effectiveness of the World Bank Group. Norad — evaluations Norwegian Aid Agency OECD DAC Evaluation Resource Centre (DEReC) The Development Assistance Committee (DAC) is the principal body through which the OECD deals with issues related to co-operation with developing countries. The Evaluation Resource Centre contains development evaluation reports and guidelines published by the network and its 30 bilateral and multilateral members. OECD DAC ongoing evaluations of development programmes The site for sharing of evaluation plans and work programmes of members of the DAC Network on Development Evaluation, for enhanced transparency and facilitation collaboration. It also provides an overview of current trends in development evaluation. Poverty Action Lab A collection of various impact evaluations, including some short summaries of the conclusions from impact studies on a variety of topics of relevance for development. Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) ALNAP is a unique network that incorporates many of the key humanitarian organizations and experts from across the humanitarian sector, including members from funding agencies, as well as, nongovernmental, Red Cross/Crescent, United Nations and independent/academic organizations. It includes evaluations and lessons learned on humanitarian actions. The Expert Group for Aid Studies (EBA) Independent evaluations of Swedish international development cooperation. UN Evaluation Group (UNEG) Country level evaluation database. USAiD — evaluations The United States Agency for International Development, with links to USAID publications including USAID evaluations. T H E P R OJ E C T H A S E N D E D, B U T W E CA N S T I L L L E A R N F R O M I T!58

Department of Reproductive Health and Research World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland Fax: +41 22 791 4171 E-mail: reproductivehealth@who.int Website: www.who.int/reproductivehealth The project has ended but we can still learn from it! Practical guidance for conducting post-project evaluations of adolescent sexual and reproductive health projects ISBN 978 92 4 151656 3 19072_Sample cover_3.5mm spine.indd 1 08/10/19 08:28