Всемирная организация здравоохранения (ВОЗ / WHO) · Technical Documents

Facilitator’s guide: using research in the EVIPNet framework

31 декабря 2018 г. Всемирная организация здравоохранения

Полный текст размещён на сайте публикующей организации. lawenc.com индексирует метаданные и ведёт на официальный источник.

Полный текст

The Evidence-informed Policy Network (EVIPNet) Europe Using research in the EVIPNet framework FACILITATOR’S GUIDE EVIDENCE-INFORMED POLICY-MAKING November 2017 The Evidence-informed Policy Network (EVIPNet) Europe Using research in the EVIPNet framework FACILITATOR’S GUIDE EVIDENCE-INFORMED POLICY-MAKING ABSTRACT The Evidence-informed Policy Network (EVIPNet) Europe is an initiative of the WHO Regional Office for Europe operating under the WHO European Health Information Initiative and in alignment with the United Nations Sustainable Development Goals. EVIPNet Europe aims to increase country capacity in developing health policies informed by the best available research evidence, supporting implementation of the European policy framework Health 2020 and the Action Plan to Strengthen the Use of Evidence, Information and Research for Policy-making in the WHO European Region. Facilitators are trained to run workshops on evidence-informed policy-making in their own settings, thus increasing national knowledge translation capacity. This guide provides complete instructions and materials needed to conduct a workshop in which participants are guided towards (i) acquiring, assessing, adapting and applying evidence that is relevant to their policy needs and (ii) preparing evidence briefs for policy and policy dialogues. The materials can be used as a three-day workshop or tailored to the specific needs of the organizations and participants involved. The resources and tools need to be adjusted to local, national or regional circumstances. KEYWORDS Capacity Building, Teaching, Research, Health Information Systems, Evidence-Based Practice, Policy Making, Health Policy SUGGESTED CITATION EVIPNet Europe. Evidence-informed policy-making: using research in the EVIPNet framework. Copenhagen: WHO Regional Office for Europe; 2018. Licence: CC BY-NC-SA 3.0 IGO. Document number: WHO/EURO:2018-3018-42776-59687 © World Health Organization 2018 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercialShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Address requests about publications of the WHO Regional Office for Europe to: Publications WHO Regional Office for Europe UN City, Marmorvej 51 DK-2100 Copenhagen, Denmark Alternatively, complete an online request form for documentation, health information, or for permission to quote or translate, on the Regional Office website (http://www.euro.who.int/pubrequest). CONTENTS ACKNOWLEDGEMENTS ....................................................................................................... IV ABBREVIATION ........................................................................................................................V INTRODUCTION .........................................................................................................................1 Train-the-trainer strategy ......................................................................................................................1 Workshop objectives and design ............................................................................................................1 Facilitator’s Guide ................................................................................................................................ 2 SECTION I. PLANNING THE WORKSHOP .......................................................................... 5 SECTION II. GUIDELINES FOR FACILITATION .................................................................11 SECTION III. FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS .. 17 Day 1 ..................................................................................................................................................18 Day 2 ................................................................................................................................................40 Day 3 ................................................................................................................................................ 64 SECTION IV. TOOLS FOR FACILITATORS .........................................................................79 Sample agenda ..................................................................................................................................80 Materials checklist ..............................................................................................................................81 Pre-Workshop survey ......................................................................................................................... 82 Clarifying a problem: case studies .......................................................................................................84 Finding and using research evidence ................................................................................................... 85 AMSTAR 2 checklist ...........................................................................................................................88 SURE checklist ................................................................................................................................... 92 Evidence brief on sugar-sweetened beverages ..................................................................................... 98 Policy dialogue demonstration: patient safety ...................................................................................... 99 Daily feedback form ..........................................................................................................................105 Final evaluation form ........................................................................................................................106 Facilitator feedback ..............................................................................................................................................................108 Energizers ........................................................................................................................................ 113 Quick exercises to assess participant learning ..................................................................................... 115 Dealing with disrupters: prevention and intervention .......................................................................... 116 Instructional methods ....................................................................................................................... 118 REFERENCES AND RESOURCES .....................................................................................126 ACKNOWLEDGEMENTS This publication was produced by the Division of Information, Evidence, Research and Innovation of the WHO Regional Office for Europe, under the leadership of Dr Claudia Stein, Director, and the supervision of Tanja Kuchenmüller, Unit Leader i.a. The principal authors of this publication were: • Susan Kingston, Global Health Consulting, Seattle, United States • Tanja Kuchenmüller, Unit Leader i.a., Knowledge Management, Evidence and Research for Policy-Making, WHO Regional Office for Europe, Copenhagen, Denmark • Rhona Mijumbi, Regional East African Community Health Policy Initiative, Makerere University, Uganda • Vicki Carpenter Erickson, Professional Facilitator, Copenhagen, Denmark. We would like to acknowledge the contribution through critical review of Kaelan Moat (Managing Director McMaster Health Forum) and Tomás Pantoja (Associate Professor, Department of Family Medicine, Pontificia Universidad Católica de Chile). We would also like to thank the members of the (EVIPNet Europe) WHO Secretariat, in particular Clara Wollburg (Intern, WHO Regional Office for Europe, Copenhagen, Denmark). We are grateful to the Structured Operational Research and Training Initiative (SORT IT) for co-founding this publication. iv FACILITATOR'S GUIDE ABBREVIATIONS EBP evidence brief for policy EIP evidence-informed policy-making EVIPNet Evidence-informed Policy Network KT knowledge translation PD policy dialogue WHO World Health Organization ABBREVIATIONS v 1INTRODUCTION International attention has been focused on bridging the gap between health research and policy-making. In 2005, WHO launched the Evidence-informed Policy Network (EVIPNet) with the aim of empowering and strengthening country policy-makers, researchers and civil society to use evidence in policy-making. Influenced by the successful implementation in other regions, in October 2012 the WHO Regional Office for Europe established its regional network: EVIPNet Europe. The vision of this network is in alignment with the Sustainable Development Goals (1) and the key areas of the WHO European Health Information Initiative (2). The network’s mandate to strengthen evidence-informed policy-making (EIP) and build national knowledge translation capacity further contributes to the implementation of the European health policy framework Health 2020 (3) and the Action Plan to Strengthen the Use of Evidence, Information and Research for Policy-making in the WHO European Region (4). EVIPNet Europe focuses on national empowerment to address policy challenges in the participating countries. With the understanding that each country needs to develop its own capacity in EIP, EVIPNet Europe aims to support national stakeholders in their efforts to apply EIP methodologies to their particular context. Promoting an environment favourable to the systematic and transparent use of evidence in policy-making requires continuity and a sufficient scale of human resources able to find and use research evidence to inform work related to prioritizing problems and developing and implementing policies and programmes. Train-the-trainer strategy As one step towards strengthening the human resources and capacities required at national level, EVIPNet Europe initiated a train-the-trainer strategy, with the aim of developing national champions throughout the Region so that they could facilitate future workshops on EIP in their own settings. The strategy is being rolled out gradually with two key steps. First, a training-the-trainers workshop is being offered. This workshop comprises of brief presentations on EVIPNet Europe and its support for EIP, followed by facilitated interactive sessions. Emphasis is placed on helping participants to acquire the practical pedagogical skills needed to become effective facilitators in EIP. Second, to ensure that the designated trainers have the knowledge and skills required to organize and conduct workshops on their own, they will start off co-facilitating EVIPNet Europe’s capacity-building workshops (both in their own country and in the Region) in tandem with more experienced EVIPNet facilitators to gain hands-on experience. This will enable them to take a leading role in the future. Workshop objectives and design After completing this training phase, facilitators will be responsible for running the EIP Workshop, which is intended for a wide range of stakeholders such as public health FACILITATOR'S GUIDE 2 planners, health system managers and health researchers. The primary objectives of the Workshop are to strengthen the participants’ capacity to: • acquire, assess, adapt and apply research evidence; and • at a fundamental level, prepare evidence briefs for policy and organize policy dialogues. By the end of the Workshop, participants will have greater knowledge and skills in EIP and more specifically in EVIPNet’s methodologies and their application. The EIP Workshop is based on the principle that in order to sustain the learning, participants need to be active and have time to reflect: it is centred on the learner. Therefore, the role of the facilitator is not only to expertly present the material and guide participants through the various sessions but also to draw on and share the collective knowledge of the participants. This approach will make the Workshop’s content more relevant to the participants and thus increase the likelihood of it having a lasting impact. Although the Workshop is designed to run for three consecutive days, it can be delivered in several ways depending on time constraints or particular circumstances. The material should be tailored and adjusted to local, national or regional circumstances and the specific needs of the organizations and participants involved. Facilitator’s Guide This Guide provides facilitators with complete instructions and materials needed to run the entire EIP Workshop. The Guide further describes different approaches depending on participants’ needs and demands to make the Workshop as relevant as possible to the target audience. Accordingly, individual sessions could be presented as stand-alone topics to meet ad hoc training needs, or selected sessions could be integrated with other related presentations and activities to address specific audience requirements or to guide participants through a specific policy project. In addition, the Facilitator’s Guide includes prompts for questions and comments during the presentation of the slides, as well as brief facilitated discussions at the end of each session on the topics presented. Furthermore, time is allocated for hands-on activities to enable participants to apply the concepts that have been covered. While the curriculum already utilizes a variety of instructional methods (e.g. case studies, demonstrations, discussions, brainstorming), facilitators should be encouraged to adapt or use additional methods to tailor the Workshop most creatively to the needs of the audience. Hence, the choice of the best training method depends on the learning objectives and the method’s unique advantages and disadvantages, among other factors. To illustrate how evidence has been used to inform policy on different global health issues, the Workshop presentations also include several real-life examples. Facilitators can build upon these and present their own real-life examples or share stories from their experiences to further customize the Workshop. The Facilitator’s Guide is organized into four main sections: I. planning the Workshop II. guidelines for facilitation III. facilitation instructions, and IV. tools for facilitators. Section I on planning the Workshop is targeted at the facilitator as well as others on the organizing team. It covers all practical aspects that need to be considered to ensure INTRODUCTION: OVERVIEW OF WORKSHOP 3 a successful training session: from selecting participants to organizing venue, travel and meeting room logistics to sending pre-Workshop surveys and assignments. It also provides a comprehensive pre-Workshop checklist to avoid anything being overlooked. Section II offers tips on how to make facilitation as effective as possible. Since the Workshop is learner centred, the facilitator needs to be mindful of adult-learning principles and how best to support small group work. The role of the facilitator is to engage the participants while guiding them through the curriculum. Section III lays out the entire three-day curriculum. It provides the specific agenda for each day, the objectives for the various sessions and step-by-step instructions to facilitate the sessions, including the activities, suggested talking points and slides. Over the three days, the Workshop builds steadily. Day 1 explains how evidence is defined and how to use it to clarify problems and identify options to address these high-priority issues. Day 2 covers finding and assessing systematic reviews and implementing policy options. Day 3 moves to more advanced topics: preparing evidence briefs and organizing policy dialogues. The Workshop ends with a policy dialogue demonstration. The Guide also includes instructions on how to open and wrap up each day to enable the facilitator to manage the participants’ expectations and make adjustments if required. The slides are also available as PowerPoint files and can be requested from the WHO EVIPNet Europe Secretariat. The fourth and final section is intended to support the facilitator practically by providing a comprehensive set of tools: sample agendas; materials checklist; complete session handouts; facilitation tools, such as energizers and exercises for assessing learning; and evaluation questionnaires. The evaluations provided by the participants and the feedback shared among the facilitators will later be a valuable source of information to assess whether the Workshop has fulfilled its objectives. References 1. Transforming our world: the 2030 agenda for sustainable development. New York: United Nations; 2015 (A/RES/70/1; (https://sustainabledevelopment.un.org/content/documents/21252030%20Agenda%20 for%20Sustainable%20Development%20web.pdf, accessed 1 November 2017). 2. European Health Information Initiative. Copenhagen: WHO Regional Office for Europe; 2017 (http:// www.euro.who.int/en/data-and-evidence/european-health-information-initiative-ehii/european- health-information-initiative, accessed 1 November 2017). 3. Health 2020: a European policy framework and strategy for the 21st century. Copenhagen: WHO Regional Office for Europe; 2013 (http://www.euro.who.int/__data/assets/pdf_file/0011/199532/ Health2020-Long.pdf?ua=1, accessed 1 November 2017). 4. Action plan to strengthen the use of evidence, information and research for policy-making in the WHO European Region. Copenhagen: WHO Regional Office for Europe; 2016 (EUR/RC66/12; (http://www.euro. who.int/__data/assets/pdf_file/0006/314727/66wd12e_EIPActionPlan_160528.pdf?ua=1, accessed 1 November 2017).

SECTION I. PLANNING THE WORKSHOP 6FACILITATOR'S GUIDE PLANNING THE WORKSHOP Workshop organizers will need to complete the following steps to plan a successful Workshop. 1. Select participants 2. Assemble the Workshop team 3. Review and adapt the curriculum (as needed) 4. Manage the venue/travel logistics 5. Conduct a meeting with the Workshop team 6. Prepare the meeting room. Select participants Selecting and inviting engaged and committed participants is important for a successful Workshop. The impact of the Workshop will last longer if participants have the opportunity and support to put their learning into action following the Workshop. Participants should ideally include those who: • have demonstrated interest in research and policy-making; • have support from their supervisor; • have an opportunity to incorporate their learning into current or future activities; and • have a strong understanding of their local political landscape. A group of 10 to 20 participants is an ideal size for one to two facilitators to effectively manage discussions and small group work. After selecting the final participants, facilitators should learn more about the participants’ professional backgrounds, health interests and how much experience they have with policy work. This information can be obtained through a pre-Workshop survey, which will help the facilitator to: • determine whether the curriculum or agenda needs to be adapted in any way; • manage participant expectations; and • draw on the knowledge and experience of the participants to make the Workshop more interactive. A sample of a pre-Workshop participant survey is included in Section IV. This should be sent to participants at least four weeks prior to the Workshop. A reminder should be sent to the non-responders (ideally one and two weeks before the Workshop). Assemble the Workshop team The Workshop team will work closely together during all stages of the Workshop to facilitate sessions and manage logistics. The Workshop team should ideally include: • a lead facilitator • potential co-facilitators; and • a logistics coordinator to handle communication with the venue, catering, etc. Facilitators should have experience in group facilitation, health research, policy-making and in the regions represented at the Workshop. SECTION I: PLANNING THE WORKSHOP 7 A note about translation Determine the main language that will be used during the Workshop. If translators are necessary, it is best to appoint a translator who has a technical vocabulary for research and health policy. Facilitators should not serve as the primary translator(s). Review and adapt the curriculum This curriculum is flexible enough to be conducted with a wide variety of audiences. Each day builds upon material and examples from the previous day, so sessions should be conducted in the order presented here. However, facilitators may wish to adapt the presentations or activities, based on participants’ learning needs, time available or size of the group, while adhering to the original session sequence and instructions. This includes adding, deleting or updating the PowerPoint slides to better meet the needs of participants. While the curriculum already utilizes a variety of instructional methods (e.g. case studies, demonstrations, discussions, brainstorming), facilitators are encouraged to adapt or use additional methods to meet the needs of the audience most creatively. Choosing the best training method depends on the learning objectives and the method’s unique advantages and disadvantages, among other factors. Instructional methods in Section IV give a complete description of the most common training methods. The presentations also include several real-life examples to illustrate how evidence has been used to inform policy on different global health issues. Facilitators can further customize the Workshop with their own real-life examples or share stories from their experiences. Be sure to select examples that reflect participants’ interests and needs. If less time is available for the Workshop, consider using fewer activities or examples to illustrate concepts and/or reducing the amount of time for group work or the number of teams that report out after small group exercises. If more time is available, offer an additional day to provide individual or peer-to-peer technical assistance to refine or further develop the specific policy projects. Once the curriculum has been reviewed, ensure that there is an online site (e.g. Dropbox) where participants can access any materials to review before the Workshop. Manage the venue/travel logistics Make sure that the venue is spacious enough to accommodate all participants. Meeting rooms should ideally be set up with round tables already organized for group work so that plenary sessions and the small group work can be held in the same room. Other logistics to consider include: • catering for breaks and meals • participant per diems (if provided) • hotel reservations • travel and local transportation • audiovisual needs • planning for an evening social event, if desired. 8FACILITATOR'S GUIDE Conduct a facilitators’ meeting Facilitators should meet before the Workshop to review logistics and adapt the agenda as necessary. The following pre-Workshop checklist may be useful to make sure you are prepared. • Assemble participant packets, attendance sheet, name tags. • Plan the Day 1 registration process. • Coordinate for meals and coffee/tea breaks. • Review final participant list and results from any pre-Workshop surveys/ assignments. • Review participant per diem/other logistics. • Decide how best to divide participants into small groups (if needed). • Assign roles and sessions to facilitator(s). • Make adjustments to Workshop schedule, curriculum or materials. • If needed, appoint a rapporteur and brief him/her on the Workshop content and expectations. • Review language management plan (if needed): • translator process (who will translate and when); • how to best present when using translators; • slides, flip charting in another language. • Organize a social event (optional). • Select material for participants to review beforehand and send a weblink to those materials. Prepare the meeting room The day before the Workshop, conduct a “walk through” of the meeting room to be sure it is properly arranged and equipped, including: • adequate tables and chairs • optimal table arrangement • lighting and temperature control • microphones (if needed) • audiovisuals, electrical cords, screens • flip charts, markers (best are black, blue or green), tape. There are generally two useful ways to arrange the tables. Each has advantages and disadvantages for various Workshop components. Based on the size of your group and room, select an option that will function most effectively. • U-shape, with facilitators and easels at the open end, ensures that everyone can easily see, hear and interact with each other but it is not conducive for small group work. This shape is best for a smaller workshop group. • Cabaret-style, with several round tables spaced closely around the room with facilitators at the front of room, is effective for small group work but reduces interaction between participants. Before the session, post the necessary flip charts on the wall in a clearly visible location in the room. Also, place participant packets at each seat. Packets should include: • agenda • copies of presentation slides • participant roster • notepaper • worksheets • pen/pencil • name tag and name placard. SECTION I: PLANNING THE WORKSHOP 9 Planning schedule In order to complete each of these parts in a timely manner, planning should begin several months before the Workshop. The schedule in Table 1 is suggested as a guideline. Table 1. Suggested schedule for arranging a Workshop TIMELINE TASK 2 months before • Assemble the Workshop team • Secure a venue • Invite/notify participants and send pre-Workshop survey 1 month before • Confirm list of participants • Arrange travel logistics (earlier if visas are required) for participants and facilitator(s) • Collect pre-Workshop participant survey • Administer pre-Workshop assignment, if any • Review and adjust Workshop curriculum and agenda • Assign roles and presentations among facilitators 2–3 weeks before • Finalize and produce materials, worksheets, etc. • Confirm venue logistics 1 week before • Assemble participants’ packets and materials • Create participant roster with contact information 1–2 days before • Facilitator(s) travels to Workshop location, if applicable • Conduct facilitators’ meeting • Prepare the Workshop room

SECTION II. GUIDELINES FOR FACILITATION 12 FACILITATOR'S GUIDE GUIDELINES FOR FACILITATION A facilitator is a person who helps others to move through a process of learning or completing tasks (or often both). A facilitator must give explanations, do demonstrations, answer questions, conduct exercises and lead group discussions—all while keeping to a time schedule! Although each facilitator has his/her own personal style, great facilitators create an energetic, collaborative learning environment that encourages participants and facilitators to interact, discuss and practise skills together. Strategies to keep participants energized and attentive This Workshop is meant to be active. The focus is on engaged, collaborative learning that encourages participants and facilitators to interact, discuss and practise skills together. Some of the common techniques are outlined next. Qualities and techniques of effective facilitators Some fundamental practices used by all effective facilitators are listed here. Be prepared • Always arrive early to organize the room and materials; check that all audio and computer equipment is working properly. • Know your daily objectives and what needs to be accomplished for each day/ session. Create a welcoming atmosphere • Make sure to introduce yourself and greet participants as they arrive. • Learn the names of each participant and use their names frequently; this creates a collegial atmosphere and helps participants to feel valued and included. Stay enthusiastic • The energy level of participants will reflect your own, so stay upbeat and positive. Stay flexible with the schedule • Manage and adjust time so all participants adequately understand concepts. • If you spend more time on one area, plan to make up time elsewhere. • Be sure to cover all relevant material over the course of the Workshop. Use the “wisdom in the room” • Acknowledge the expertise of participants. • Encourage them to contribute their knowledge and share relevant experiences. Be mindful of your physical presence in the room • Do not turn your back to the room but equally do not stay frozen in one spot. • Move around so everyone can hear, see and stay fully engaged. • Maintain eye contact with participants so that everyone feels included, avoid always looking at the same person. Looking at a participant for a few seconds will often prompt a reply, even from a shy participant. SECTION II: GUIDELINES FOR FACILITATION 13 Be eager for participants to learn • Frequently ask questions to make sure participants understand the material, even if they do not ask for help. • Avoid simple yes or no questions and ask questions that begin with what, why or how. Observe and respond • Watch the energy, mood and reaction of the participants. If the room energy feels flat, do not hesitate to pause and perhaps insert a quick energizer (see Section IV) or invite participants to volunteer to lead an energizer of their own. • Adjust content based on the tempo and productivity of the group to meet their needs. • Check to see whether participants are having any problems, even if they do not ask for help. If participants know that someone is genuinely interested in their learning, they are more likely to ask for help when they need it. Remain neutral • Do not take sides in disagreements that may arise. • Actively solicit alternative opinions. • Do not criticize what participants say. Ask open-ended questions • Solicit more information and opinions from participants beyond simple yes or no responses. • Encourage participants to elaborate on their comments (e.g. “What more can you say about that?”). Encourage critical thinking • Encourage participants to pose questions to each other, not just to you. • Do not answer every question right away. • Reflect questions back to participants or invite others to respond first. Listen actively • Allow participants to think and take time to answer a question rather than rushing to fill silence. • Encourage participants to answer each other’s questions, then summarize or connect ideas. Get everyone engaged • Make it safe for everyone to speak and ask the group to agree on guidelines for working together (ground rules such as demonstrate mutual respect, agree to disagree, all questions are welcome, give honest feedback). • Enthusiastically thank individuals when they ask questions or volunteer for activities. • Help quieter people to participate by giving them chances to succeed. • Mix up groups and connect with all participants so everyone has a chance to contribute. Trust your own advocacy expertise and facilitation skills • Use the Facilitator’s Guide but do not rely on it as a script. • Give real-life examples or change order of content as necessary. Request and use feedback • Collect participant feedback each day and adjust the next day’s content to re-emphasize topics or adapt teaching approaches. • Negative feedback is actually quite helpful if it inspires creative improvements. 14 FACILITATOR'S GUIDE Deal quickly with “disrupters” • If a participant disrupts by dominating discussions, having side conversations or challenging your authority with their own, take steps to curb these behaviours right away. There are several suggestions in Section IV under Dealing with disrupters. Tips for teaching adults Adult learning is an instructional approach that recognizes that adults learn in a different way to children or young people. Adults have their own unique motivations, needs and learning styles. Adults are autonomous and self-directed. They appreciate having a choice to participate or not. Therefore, it is best to facilitate rather than dictate the training process. Adults are goal-oriented. They have a clear reason for attending and appreciate an organized training session with specific objectives. Learning must be relevant and practical. Adults must see an obvious reason for learning and that the content will be useful to them. Know your audience and what they need. Always try to connect concepts to real work or life experiences. Adults are sensitive to wasted time. Pay attention to start and end times but be willing to adjust to meet the needs of participants. Repeat material for a clear reason. Adults may feel anxious about activities where they might appear less competent. They appreciate safety, fairness and an equal learning environment. Encourage participants not to fear mistakes and praise all attempts to participate, risk an incorrect answer or try a new skill. Remember that participants in this Workshop will be experienced professionals. The knowledge and skills that they bring to the Workshop are important to the learning process. Facilitators should frequently acknowledge this expertise and attentively encourage participants to contribute their knowledge, share relevant work experiences and provide different perspectives. The learning experience will be enriched for all involved when facilitators and participants learn collaboratively. Facilitating small group work During small group work, the facilitator should float among teams as a mentor or “table coach”. The primary role of a table coach is to make sure participants understand the task, follow instructions and work productively. You are closely positioned to look for areas where participants may be confused and to answer questions as they arise. Effective table coaches: • allow teams to work independently before jumping in and position themselves nearby while groups work independently to observe; • pay attention to the conversation, and if the team is getting off track or members clearly do not understand the task, redirect or clarify ideas without any hesitation; • encourage the group’s critical thinking, by responding with another question if the team asks a question as this will get them thinking; • watch the clock as teams may get into vigorous discussions, easily losing track of time and thus failing to complete an activity; time reminders can be given at mid- point and five to ten minutes before the activity ends; and • encourage the group to assign roles as teams are more likely to function efficiently if they assign a timekeeper, recorder and spokesperson at the beginning of an activity. SECTION II: GUIDELINES FOR FACILITATION 15 Co-facilitation Although a single facilitator is sufficient for this Workshop, an assistant or co-facilitator can be useful. A facilitating partner can record brainstorm ideas on flip charts, serve as a note- taker and help with logistics so the lead facilitator can focus on the process and discussion. When two facilitators choose to work together, there are some suggestions for effective co-facilitation: Divide the Workshop by parts and switch off throughout the day. Assign sessions that are a good match with each other’s talents, knowledge or background. While one facilitates, the other assumes the assistant role (e.g. recording on flip charts, time keeping, etc.). If you are assisting, allow your partner to manage the flow of the discussion. Resist the urge to interrupt or take control, unless support is clearly requested. If you are the lead facilitator, engage your partner to answer participant questions that align with his/her unique expertise. Establish cues to use during presentations to subtly communicate feedback. Examples are “Time is running out”, “Participants don’t understand”, “We need a quick time out” or “Please help me explain this concept”. Be prepared to facilitate each other’s sessions in case something unexpected happens (e.g. co-facilitator falls ill).

SECTION III. FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS This section contains complete agendas, learning objectives, instructions and key talking points for each session. The slide show icon indicates when to use the PowerPoint slides: Call for facilitator feedback Since this Workshop will be organized and run by different people in different contexts, and for different target groups, it would be useful to exchange experiences, tips and comments among Workshop organizers and facilitators. Therefore, as you progress through each session, note how each session went and what changes you might make for future Workshops. If possible, please record and submit your impressions and suggestions on the Facilitator feedback form in Section IV. This feedback will be shared among organizers and facilitators to learn from each other to continually improve the Workshop. 18 FACILITATOR'S GUIDE Table 2. Sample agenda for Day 1 SAMPLE AGENDA ESTIMATED TIME MATERIALS Opening session 45 minutes PowerPoint slides Laptop LCD projector Attendance sheet Name tags Flip charts Coloured markers Participant folders Copies of handouts: • Clarifying a problem: case studies • Daily feedback form What is evidence? 75 minutes Break Clarifying a problem 60 minutes Activity time 45 minutes Lunch Identifying options 60 minutes Break Activity time 90 minutes Wrap up 15 minutes DAY 1 The Opening session of the Workshop orients participants to the objectives, content and methods of the Workshop, begins to establish rapport among participants and sets the collegial tone of the Workshop. The morning session then continues with a brief overview of the various types and roles of evidence in policy-making (What is evidence?) and then introduces the different ways to clarify and frame health problems to address with a programme or a policy (Clarifying a problem). This is the first phase of the policy process. Before moving to the next session, there is time to include a hands-on activity appropriate for the goals of the audience. The focus of the afternoon session is on the second phase of the policy process, where participants learn how to identify and evaluate potential policy options to address the problem (Identifying options). There is a second opportunity for a hands-on activity during which participants can apply their learning so far or continue working on a specific group project. In the discussions throughout the day, encourage participants to compare the “ideal” EIP process with the policy process they currently experience in their work. Table 2 gives a sample agenda, with the materials needed for the whole day. 1 SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 19 OPENING SESSION As participants arrive, direct them to sign the attendance sheet and distribute name tags and participant folders. FACILITATION STEPS 1. WELCOME everyone to the Workshop. 2. INTRODUCE yourself and other facilitators. 3. ASK participants to introduce themselves and describe their professional roles and their experience with using research evidence in policy-making (apart from getting to know the participants, this will also allow a quick assessment of the level of knowledge, skills and experience of participants in terms of knowledge translation). Options are to: • rotate around the table, one participant at a time; • put participants into pairs and have partners introduce each other to the group; or • in the introductions, ask participants to also include something personal about themselves to create more rapport (e.g. ask them to tell the group the best/most difficult part of their job, or tell the group something about themself that we might not guess). 4. EXPLAIN the Workshop objectives. • This Workshop was developed to build the capacity of countries to design and implement health policies informed by research evidence. In this Workshop you will: • gain skills to acquire, assess, adapt and apply research evidence; and • learn the fundamentals of how to prepare evidence briefs for policy and organize policy dialogues. • By the end of this Workshop, you will be able to find and use research evidence to inform work related to prioritizing problems and developing and implementing policies and programmes. 5. ASK participants to share one or two topics they most want to learn about in this Workshop. Record responses on a flip chart. 6. REVIEW Workshop logistics (e.g. restrooms, emergency exits, meals, per diems, etc.). 7. SHOW the participant folders. Point out the contents including copies of the slide presentations, which also contain areas for taking notes. 8. REVIEW the Workshop agenda. Point out the variety of teaching methods: presentation, demonstration and hands-on activities. 9. HIGHLIGHT the agenda for today. 10. ASK for final questions about the Workshop. 45 minutes 20 FACILITATOR'S GUIDE Slide 1 75 minutes WHAT IS EVIDENCE? The purpose of this session is to build a common understanding of evidence-informed policy-making and the role of evidence in the policy-making process. Use the discussion after the slide presentation to help participants to assess how evidence is used – or not used – in current policy-making in their work. Objectives By the end of this session, participants will be able to: • define the term evidence • identify the role of evidence in making policy • define evidence-informed policy-making • describe common challenges in evidence-informed policy-making. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers. FACILITATION STEPS 1. EXPLAIN the objectives for this session (5 minutes). 2. ACTIVITY: pairs brainstorm (10 minutes). • Participants form pairs and are asked them to brainstorm reasons why evidence- informed policy-making is important. • After three minutes, pairs share their responses with the full group while you record their responses on a flip chart. Examples could include: • results in better decisions and more effective policies • can assess different policy options and their impact • accountability and transparent democratic processes. • Debrief by asking participants to identify similarities and differences in the responses. 3. PRESENT the slide set: What is evidence? (45 minutes) What is Evidence? EVIPNet Europe What is Evidence? Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World Health Org nization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 21 What is Evidence? Objectives of the session • Define the term evidence. • Identify the role of evidence in making policy. • Discuss the concepts of evidence-informed policy making and knowledge translation. • Describe the different forms of evidence- informed policy making. What is Evidence? Key questions 1. What is evidence? 2. What is its role in the policy-making process? 3. What is evidence-informed policy making? What is Evidence? 1. What is evidence? “Evidence concerns facts (actual or asserted) intended for use in support of a conclusion”* • A fact is something known by experience or observation. • Evidence is used to support a conclusion. It is not the same as the conclusion. Source: Lomas J, Culyer T, McCutcheon C, McAuley L, Law S (2005). Conceptualizing and combining evidence for health system guidance. Hamilton: Canadian Health Services Research Foundation. What is Evidence? Types of evidence Context-free scientific evidence Context-sensitive scientific evidence Tacit knowledge medical-effectiveness or biomedical research evidence put into a particular operational setting expertise, view and realities of stakeholders Slide 2 READ the objectives. Slide 3 We will answer these three questions in this session. Slide 4 For our purposes, we will use this definition. æ ASK a volunteer to read the definition. æ ENCOURAGE responses with these prompts if needed. • Is evidence just hard facts? • Are the facts only valid if they are supported by scientific research? • What about things we can’t touch or measure? Slide 5 This slide is animated Think about how different forms of evidence may apply. For example, whether the evidence is context-free or context-sensitive. Is it tacit knowledge or hard facts? Always consider the type of evidence. CLICK Context-free evidence is what works in general, or knowledge about the overall “potential” of something. In other words, this evidence will generally appear the same in any setting or environment. This is typically used within medical effectiveness or biomedical research. For example, the fact that aspirin can relieve a headache is not context specific. It will work against headache in any environment. CLICK Context-sensitive evidence is evidence put into a particular operational setting. For example, what happens when we want to make aspirin available in rural villages? How will we communicate about the use and benefits of aspirin? How will we distribute aspirin? This evidence will be unique to this particular context of rural villages. But the fact that aspirin works is not context specific. Think of context-sensitive evidence as where quantitative meets qualitative or where the theory meets the reality. Both types of evidence are captured in systematic reviews, in other syntheses (e.g. a policy brief), in single studies and in pilot or case studies. CLICK 22 FACILITATOR'S GUIDE Slide 5 (contd) This slide is animated A third and often less acknowledged category is tacit knowledge. Tacit knowledge comes from experience, lessons learned, organizational tradition and best practice. It is a mixture of intuition, common sense, know-how and pragmatism. In some cases, tacit knowledge may help to address areas where other evidence is inconclusive or non-existent. æ FACILITATE a brief discussion with prompts such as: • What is the implication of having different types of evidence? • What does it mean for you as a researcher that we have these different types of evidence? Slide 6 Tacit knowledge comes from experience, lessons learned, organizational tradition and best practice. By comparison, explicit knowledge is scientific, structured, verifiable and replicable evidence. Evidence synthesis products such as systematic reviews are highest on the hierarchy of robust evidence because they are more comprehensive and reliable than a single study. The importance of combining explicit and tacit knowledge is to create robust evidence that will enhance design of context-specific solutions that are operational and applicable to local context. Slide 7 Some evidence may be more convincing than other evidence. Scientific research is generally more valued and trusted than an “informed opinion”. Yet how do we know what is good science or not? Was the research conducted using solid methods or was it perhaps influenced by donor money? Therefore, we must always consider the quality and reliability of evidence. Slide 8 Not all evidence has equal quality. So how do I determine how much confidence to have in given evidence? Many times the evidence may look good at first glance. But factors such as study design, methodology, biases, sample size, sampling methods and so on may decrease our level of confidence in the evidence. Slide 9 Making judgements in a systematic and transparent way could protect against errors and facilitate knowledge transfer. Likewise this may help to resolve arguments by focusing the deliberations on the available evidence and the interpretation given to specific findings rather than in ideology or other non-declared interests (e.g. financial conflicts of interest). What is Evidence? Types of knowledge Source: Campbell S (2012). The knowledge translation curriculum. Ottawa: Canadian Coalition for Global; Health Research. What is Evidence? Implications of this definition • Not all evidence is equally convincing. • Evidence based on scientific research is usually more convincing than that based on random observation. • Must judge (explicitly or implicitly) our confidence in the quality of different types of evidence. What is Evidence? Judgement about confidence Judge evidence (for effects) on criteria such as: NOT on who says it or how they say it. • What sort of observations? study design • How well they were done? risk of bias • How consistent they are? consistency • How directly relevant they are? directness • How many are there? precision • How strong is an association? large effects What is Evidence? Source: GRADE Working Group (http://www.gradeworkinggroup.org/). Judgement about confidence When judgements are made systematically and explicitly, it helps to: • protect against errors • resolve disagreements • facilitate critical appraisal • communicate information. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 23 Slide 10 This slide is animated æ ASK participants to raise their hands if they think expert opinion is evidence. Then ask for a show of hands of those who think it is NOT evidence. Ask each side to explain their opinions. CLICK Expert opinion is much more than evidence. It is facts laced with personal attitudes, experiences, bias and human sentiment. It is evidence plus opinion and conclusions. An example. A lot of expert opinion says that migration is commonly seen as the result of poverty and violence in origin countries. So migrants mostly move to seek a better life abroad. While this may be partly true and is based on some evidence, it is embedded in the experience that these experts have with the system. But in fact, research evidence shows that growing prosperity in poorer countries increases migration and that the level of migration is largely determined by labour demand in destination countries (The Determinants of International Migration (DEMIG) project, based at the University of Oxford). When using expert opinion, try to get to the actual facts behind that opinion and then appraise the credibility of the opinion. The strength of an expert’s opinion should be based on its connection with facts, not on how articulate or persuasive the expert is. æ ASK: What do you think are the pros and cons of expert opinion? For example, it can help non-experts to interpret data but the bias of the expert might lead you to an incorrect or narrow conclusion. Slide 11 In policy-making, evidence is just one factor or ingredient going into the process. Many inputs come into play. For example, new politicians come into office or pressure groups might veto what you say. These other factors may influence how we introduce evidence into the policy-making process. For example, we may have to present the evidence in a way that is sensitive to local culture and tradition or wait to introduce the evidence until the political context is more favourable. æ ASK participants to share any examples from their own work of how factors like these influenced a policy. Be prepared to share your own example if needed. Slide 12 Developing public policy is not a single event. And although the slide almost implies that it is a linear process with a clear starting point and clear-cut sections, it is not. However, for our learning purposes, we identify three phases in the process. These include the setting of the agenda, where policy-makers identify problems, prioritize these problems for the policy agenda and identify desired goals (solutions). What is Evidence? Expert opinion • Evidence + opinions and conclusions. • There is evidence behind expert opinions. • To use expert opinion appropriately: • identify the facts (experience or observations) behind the opinions; and • appraise how well the facts support the conclusions (not how persuasive the expert is). POLICY- MAKING Evidence Experience Expertise Political context Values judgementsResources Policy narratives Pressure groups Culture Tradition 2. What is the role of evidence in the policy-making process? What is Evidence? What is the role of evidence? Agenda • Evidence Analysis • Evidence Action • Evidence 24 FACILITATOR'S GUIDE Slide 12 (contd) In the analysis phase, the issues are carefully considered and various possible courses of action are proposed. Policy-makers then discuss a list of recommendations and make a decision that ideally leads to action. Evidence can inform each stage of the policy-making process. For example, a recent disease outbreak might provide new data to show a critical shortage of health care workers. This issue now becomes an agenda priority over competing issues. æ FACILITATE a brief discussion on this issue with these prompts: • How can we use evidence to set a policy agenda? • How can we use evidence to analyse policy options? • How can we use evidence to determine actions? Evidence will then be needed to analyse causes of the health care worker shortage and propose solutions. Perhaps there are data to show low output from training schools, low retention due to low pay or poor distribution of workers between urban and rural areas. The policy-makers also need evidence to help them to understand the benefits, costs or potential consequences of implementing different options. These options could include shifting tasks to lower cadres, training more health workers, or giving incentives to those already in service. We will talk more in this training about finding and using evidence for this process. Slide 13 One should endeavour to combine research evidence from both local assessment and global evidence, usually from systematic reviews, as appropriate as each has different benefits. A systematic review compiles experiences from several settings to address a particular research issue. Assessing local evidence reveals answers to a research issue as assessed contextually, in this case in a given local setting. Such local contextual factors include underlying causes of a given problem and local needs, values and resources. Let’s take the problem of health worker migration to higher-income countries. First, we would look at local data and research findings to uncover the causes in our own setting, gauge the scale of our problem (Is it widespread? More in some provinces? Why is it happening?) and examine the effectiveness of previous policies to address this problem. A look at systematic reviews may then reveal what policy options have been beneficial or harmful in other countries struggling with health worker migration. This helps us to judge which policy options might work in our own setting and how we may need to adapt those options to address the specific factors revealed by local evidence. Later we will talk much more about systematic reviews. Systematic reviews and local evidence La vi s JN , P er m an an d G , O xm an A D , L ew in S , F re th ei m A. S U PP O R T to ol s fo r e vi de nc e- in fo rm ed h ea lth P ol ic y- m ak in g (S TP ) 1 : W ha t i s ev id en ce -in fo rm ed p ol ic ym ak in g. H ea lth R es Po lic y Sy st /B io M ed C en tra l [ w eb si te ]. 20 09 ;7 (1 ):S 1. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 25 Slide 14 This slide is animated æ ASK a volunteer to read the first box. CLICK to reveal the second box and READ. To ensure that evidence is relevant and reliable, we should appraise it systematically by asking: • Is this the best available evidence? • Have we looked everywhere we possibly could? • How do we know we have looked everywhere? Our systematic process must also be transparent so that others can examine what research evidence was used and how it was judged. Slide 15 Shifting from policy-making informed by beliefs and opinions towards arguments supported by research is not always easy. There can be many barriers to evidence- informed policy-making. For example, there may be no office with the mandate or funding to do knowledge translation activities or policy-makers may not even want the evidence. æ REVIEW briefly some of the challenges in the middle column of the chart. æ ASK participants to share challenges they have faced in incorporating evidence into policy-making. Remember that, in daily practice, evidence itself does not make the decision. It can only help to influence the decision. Slide 16 æ REVIEW the key points (or ask a volunteer to read them). æ ASK for any comments or questions. Slide 17 Resources (see slide deck) FACILITATION STEPS (contd) 4. FACILITATE a brief discussion with these prompts (10 minutes) • In what ways does your current policy-making process resemble (or not) the evidence-informed policy-making process? • What evidence is typically used to make policies in your work context? 5. ASK for final questions. EXPLAIN that in the next session we will talk about how to clarify a policy problem that requires research evidence, including how to describe or frame the policy question or problem and how to describe its size in relation to evidence-informed policy discussions. What is Evidence? 3. What is evidence-informed policy making? “Evidence-informed health policymaking is an approach to policy decisions that is intended to ensure that decision making is well-informed by the best available research evidence.” Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. (2009) .SUPPORT tools for evidence-informed health policymaking (STP) 1: what is evidence-informed policymaking. Health Res Policy Syst . 7(Suppl 1):S1. “…access to and appraisal of evidence as an input into the policymaking process is both systematic and transparent” What is Evidence? Challenges to evidence-informed policy making La vi s JN , P er m an an d G , O xm an A D , L ew in S , F re th ei m A (2 00 9) . S U P P O R T to ol s fo r ev id en ce -in fo rm ed h ea lth p ol ic y- m ak in g (S TP ) 1 : w ha t i s ev id en ce -in fo rm ed p ol ic ym ak in g. H ea lth R es P ol ic y S ys t. 7( S up pl 1) :S 1. . What is Evidence? Key points • “Evidence” is facts that are used to support a conclusion. Evidence itself is not the conclusion. • Evidence can be used in all stages of the policy making process, but it is not the only factor in policy decisions. • Evidence should come from systematic reviews and local assessment of the benefits and potential harms of a policy option. 26 FACILITATOR'S GUIDE CLARIFYING A PROBLEM This session introduces the concept of problem clarification. Four key questions are presented that can be used to help to clarify a policy problem. We will emphasize how the different answers to these questions can have important implications for the types of information and research evidence that will be relevant. Objectives By the end of this session, participants will be able to: • outline a process and key questions to help to clarify a policy problem; • compare various frameworks to describe or frame a health problem; and • discuss ways to describe the size and cause of a problem. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers. FACILITATION STEPS BEFORE you begin this session, consider leading a brief learning assessment activity to determine how well participants have learned key concepts from the previous session. It also helps to focus the group energy and to bring participants together after the break. Ideas for brief learning assessments are given in Section IV. 1. EXPLAIN the objectives for this session (5 minutes). 2. PRESENT the slide set: Clarifying a problem (45 minutes). 60 minutes Slide 1 We have just discussed different types of evidence. The first step in using that evidence to inform policy is to identify a very specific health problem to address with policy. This is a very important and often difficult step in the process of evidence- informed health policy-making. Clarifying the problem helps policy-makers to choose solutions that will actually address the problem. Slide 2 » ASK a volunteer to read the objectives. Clarifying the Problem EVIPNet Europe Clarifying a Problem Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World Health Orga ization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Clarifying the Problem Objectives of the session • Outline a process and key questions to help to clarify a health problem. • Compare various frameworks to describe or “frame” a health problem. • Discuss ways to describe the size and cause of a problem. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 27 Slide 3 There are three main phases in this framework or model of evidence-informed policy-making: • clarifying a problem • framing options (possible solutions) for the problem • implementng an option. Each phase requires evidence. Slide 4 When clarifying a policy problem these are the four questions to explore: æ READ the questions (or ask a volunteer to read them). These questions give us an organized process to follow to clarify any health policy question presented. We will explore them in detail one by one. Slide 5 æ First, what is the problem? Stakeholders with different backgrounds and lenses may see the same problem differently. This step helps to get everyone on the same page. For example, a problem may be initially presented as lack of access to medical services. Different stakeholders may attribute this problem to inadequate operating hours, distance from users, terrain of the area, absent medical workers and equipment, weather conditions, and so on. It is important that stakeholders agree on which factor to focus on, even though all of the factors may contribute to overall lack of access to medical services. This may require several meetings to be sure what the specific issue is. Slide 6 Sometimes the problem is not very clear, especially at the health system level where problems can be complex and multisectoral. Often we jump to solutions before uncovering the real problem. It is advisable to involve policy-makers and implementers in the process of defining the problem. Problem definition may not necessarily happen in just one or two meetings. Often, the more time we spend analysing an issue, the clearer the primary causes and drivers of the problem become. Then we can judge best what the problem is. Do not be tempted to rush through the process of defining the problem. Slide 7 How did this issue come to your attention? This factor may be prompting the policy brief. It may also influence the speed in which the issue must be addressed or the way different people are compelled to act. æ FACILITATE a brief discussion on this issue with these prompts: • In your work, what “problems” have come up or have received more attention recently? Clarifying the Problem Source: Lavis JN, Wilson M, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4. Clarifying the Problem Clarifying the problem 1. What is the problem and how did it come to attention? 2. How has the problem been framed (described) and what are the consequences of that? 3. How big is the problem? 4. What is the cause of the problem? Clarifying the Problem 1. What is the problem? • How was the problem initially described? • What was the motivation for preparing this particular policy brief? • Later iterations and analysis may lead to a better framing. Clarifying the Problem What is the true problem? • Health system problems can be unclear at first: – solutions or diagnoses often appear before the real problem – must clarify the problem before you identify options. • Discuss further with policy-makers and other stakeholders to uncover the true problem. Clarifying the Problem 1. How did it come to attention? • Why are we talking about this? For how long have we been aware of this? • This may influence speed of response or the reasons different stakeholders feel compelled to act. 28 FACILITATOR'S GUIDE Slide 7 (contd) • How did this problem get the attention? • Who brought it to your attention? æ OFFER your own example to help participants’ thinking. Slide 8 Knowing how the problem came to attention can help to clarify the problem and the extent to which it warrants attention. There are many ways that a problem can surface. æ READ a few examples from the list. HIGHLIGHT where they overlap with the participants’ examples. æ FACILITATE a brief discussion with these questions: • Can you think of other ways? • Who can share a story from your work of how a problem came to your attention? • How did the way it came to attention influence what happened next? Slide 9 æ Second, we should ask how the problem has been framed or described. Slide 10 How different stakeholders perceive the problem may be influenced by their background, their professional training, and so on. You will need to involve different people in the discussion, so you must be able to speak their “language”. This process often needs repeating as the issue becomes clearer. Slide 11 A problem can be framed or described in different ways by or for different audiences. æ REVIEW the key points. Describing a problem in terms of disease or risk factors may resonate with the Ministry of Health but maybe not with the parliament or with labour unions. These stakeholders might be more interested in the financial or health worker aspect of the problem presented. Therefore presenting it in that way may make more sense. æ List on a flip chart the following ways to frame a health worker shortage problem (these appear on the next slide): • inadequate supply of health workers • insufficient use of health workers Clarifying the Problem Ways that problems surface • Specific event (often political) • Change in an indicator • Lack of progress towards established goals • Advocacy • Public dissatisfaction (e.g. through polls or mass media) • Political consensus • Pressure from donors or international agencies • Priority-setting process Clarifying the Problem Clarifying the problem 1. What is the problem and how did it come to attention? 2. How has the problem been framed (described) and what are the consequences of that? 3. How big is the problem? 4. What is the cause of the problem? Clarifying the Problem 2. How has the problem been framed? • This can determine: – types of option available to address the problem – extent to which stakeholders feel the problem warrants attention. • Helps to present the problem in a way that: – resonates with stakeholders – helps in identifying appropriate options. • Likely to be an iterative process. Clarifying the Problem How has the problem been framed? • Can describe problems in terms of: – a risk factor or disease – coverage, quality of care, cost of care or equitable access to care (related to a service or programme) – delivery, financial or governance arrangements – implementation of agreed policies or programmes. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 29 Slide 11 (contd) æ List on a flip chart (contd) • distribution of health workers • inadequate provision of care æ ASK for ideas on other ways to frame a health worker shortage problem. Slide 12 This table is difficult to see, so you will need to either prepare a handout in an enhanced version or summarize it for the audience. Tables can be useful ways to compare the advantages and disadvantages of different framing options. This table shows different ways to frame the problem of shifting tasks from doctors to other professionals within the health system. As could be seen the same problem of “task shifting” could be framed in different ways that probably makes more sense to different audiences (e.g. health professionals, unions, policy-makers) æ READ some examples from the table. æ ASK for any final comments or questions about how to frame a problem. Slide 13 Part of clarifying a problem also involves estimating its size. Slide 14 æ The third question is how big is the problem? You may be able to measure the problem quantitatively, for example as proportions or in absolute numbers. The literature will often give you some common indicators such as: • how many people are at risk of developing a given disease; • how many are covered by a given programme (versus those that it should cover); or • how many policies have been implemented with what outcome. Slide 15 Or you may be able to show the size by making implicit or explicit comparisons. æ REVIEW the slide with examples. æ FACILITATE a brief discussion with these prompts: • What other ways can you use to show the scale of a problem? • What have you used in the past to describe problems? æ ASK for any final comments or questions about showing the size of a problem. Clarifying the Problem Constructing a table may help clarify how best to frame the problem SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3. Clarifying the problem, version 2.1. SURE Collaboration Clarifying the Problem Clarifying the problem 1. What is the problem and how did it come to attention? 2. How has the problem been framed (described) and what are the consequences of that? 3. How big is the problem? 4. What is the cause of the problem? Clarifying the Problem 3. How big is the problem? • Different indicators may be relevant to estimating the size of a problem, depending on whether the problem is described in terms of: – a risk factor or disease – coverage, quality of care, cost of care or equitable access to care – delivery, financial or governance arrangements – implementation of agreed policies or programmes. Clarifying the Problem Using comparisons to show size • Compare with goals – MDGs for maternal or child mortality. • Compare over time – increase in treatment failures or growing prevalence of a disease or risk factor. • Compare across areas within a country – variations in access to or utilization of services. • Compare with other countries – differences in mortality or prevalence rates (or changes over time) in comparable countries. 30 FACILITATOR'S GUIDE Slide 16 The first two questions are often the most difficult ones to answer. Slide 17 æ Finally, we need to ask what the cause of the problem is. It is important to identify the causes as these will later directly link to the options or possible solutions. But causes are complex and not easy to disentangle. There are many layers of influencers. The term causes is used here to refer to the “factors underlying the problem” and may not always make reference to a definitive cause– effect mechanism (in the epidemiological sense of the terms). Slide 18 These are some methods to uncover and clarify possible causes. æ ASK participants. • Can you think of any other ways? • What methods have you used in your work? Slide 19 This table is difficult to see, so you will need to either prepare a handout in an enhanced version or summarize it for the audience. Again, visual frameworks can help to summarize ideas and organize your thinking. This is a general framework that has been useful. æ WRITE the following words on a flip chart: • delivery • finance • governance. æ REVIEW the table briefly. To give an example, if we have a problem of inadequate health care services, and you would like to look exhaustively where the causes may lie, you could use a framework as shown in the slide. Is it delivery arrangements or is it a governance problem, as in how decisions and policies have been made about where services can and cannot be availed? Is it financial arrangements, about who pays for what? Could it be health systems information and research where the needs of particular areas are not communicated back to the central ministry well enough for them to adjust the decisions of delivery? The slide shows an example of such an analysis using three of the building blocks of the WHO health system framework. Clarifying the Problem Clarifying the problem 1. What is the problem and how did it come to attention? 2. How has the problem been framed (described) and what are the consequences of that? 3. How big is the problem? 4. What is the cause of the problem? Clarifying the Problem 4. What is the cause of the problem? • Understanding the cause of a problem can help: – to identify and select appropriate options – to frame the problem. • Causes of health system problems are complex and uncertain: – clarifying the cause is not always simple or linear – be transparent about what you know and do not know. Clarifying the Problem • Strategies include: – using broad or more specific frameworks – reviewing relevant research evidence – brainstorming – interviews with key informants. What is the cause of the problem? Clarifying the Problem A broad framework for health system problems Source: SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3. Clarifying the problem, version 2.1. SURE Collaboration (http://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/sure guides.html). SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 31 Clarifying the Problem For example • A problem was initially framed in terms of delivery arrangements: – a shortage of health workers in rural areas. • The cause of this shortage may be related to financial or governance arrangements: – how health workers are paid. – licensing of different types of health worker to perform specific tasks. Slide 20 Here is another example: æ REVIEW the slide. Slide 21 æ REVIEW the sources. Please keep in mind though that frameworks are only meant as a guide allowing you to identify the problem and its causes systematically. Frameworks are usually general and can be applied widely for different subjects. We will show you a few additional frameworks that you may find useful. Slide 22 A useful framework is the WHO health systems framework (and we do acknowledge that there are other health systems frameworks that have different components or categories), which identifies six blocks under which elements of a health system can be categorized: • service delivery • health workforce • health information systems • access to essential medicines • financing • leadership/governance These elements all interact at different levels. As we have been saying, health systems issues are complex. Hence any problem may have causes related to any combination of these blocks. One needs to be sure where the problem lies before identifying solutions. Slide 23 This table is difficult to see, so you will need to summarize it for the audience. æ REVIEW the chart briefly. Slide 24 This table is difficult to see, so you will need to summarize it for the audience. æ REVIEW the graph briefly. Clarifying the Problem Finding frameworks • Google Scholar or PubMed: – combine the word framework with key words describing the problem. • Talk with experts in the specific area (often the most efficient way). • Other sources: – other policy briefs on closely related issues – policy analyses – systematic reviews (or their overviews). Clarifying the Problem WHO health systems framework So ur ce : ht tp :// w w w .w ho .in t/ he al th in fo /s ys te m s/ W HO _M BH SS _2 01 0_ fu ll_ w eb .p df Clarifying the Problem A human resources framework Source: Chopra M, Munro S, Lavis JN, Vist G, Bennett S (2008). Effects of policy. options for human resources for health: an analysis of systematic reviews. Clarifying the Problem A health care financing framework Ku tz in J ( 20 01 ). A de sc rip tiv e fra m ew or k f or co un tr y- le ve l a na ly sis o f he al th ca re fi na nc in g ar ra ng em en ts . H ea lth P ol icy . 5 6: 17 1– 20 4. 32 FACILITATOR'S GUIDE Slide 25 Brainstorming is another way to help in the process of clarifying the problem and its causes, particularly with people who have direct knowledge of the context. Consult with your experts several times to generate ideas and check your hypotheses. æ INVITE participants to share examples of how they have used brainstorming to clarify a problem. Slide 26 Often we are tempted to jump right to the options. But do not rush when you are facilitating a process like this. Take time to clarify the problem so you move in a unified, transparent way towards options. Also remember to frame the problem from the beginning in terms that policy-makers can understand. They are your final target. Slide 27 This is just to recap on the four questions used for deliberations about clarifying a problem. Slide 28 Resources (see slide deck) FACILITATION STEPS (contd) 3. ASK for any final comments or questions about clarifying a problem (10 minutes). Clarifying the Problem Key points • Clarifying the problem is a key step in evidence- informed policy-making. Don’t rush. • Interactions with stakeholders (policy-makers) are extremely important. • Match the framing to the particular audience. • Use a general framework with which you are familiar. Clarifying the Problem Clarifying a problem 1. What is the problem and how did it come to attention? 2. How has the problem been framed (described) and what are the consequences of that? 3. How big is the problem? 4. What is the cause of the problem? Clarifying the Problem Brainstorming or creative thinking • Can be structured (with a framework) or unstructured. • Involve people with: – different perspectives – a broad knowledge of the health system. • May be an iterative process: – hypothesize potential causes – find information to support or refute hypotheses – discuss the causes again. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 33 ACTIVITY TIME By now, participants will be ready for something active after sitting through two slide presentations. Use this time slot to lead an activity that will encourage participants to apply what they have learned in an interactive exercise. There are three options to consider below. IDEAS FOR ACTIVITIES Small group work This allows participants to work on a real project within their local context. If the project involves clarifying a health problem for policy work, the following questions may be useful in discussions.1 • What questions could you ask to clarify the problem? • How did the problem come to attention? How likely is the prospect the problem will be addressed? • What indicators can be used or what comparisons could be made to establish the magnitude of the problem? • How can a problem be framed (or described) in a way that will motivate different groups? Who are these groups that need to be reached? Case study This activity helps participants to think critically about how they would clarify a problem in three different scenarios. Follow these steps to facilitate. 1. Divide participants into three teams and distribute the handout Clarifying a problem: case studies. Assign each team a scenario. 2. Give teams about 15 minutes to read their scenario and to answer the discussion questions. 3. As a full group, ask each team to discuss its scenario and its responses to the discussion questions (about 15 minutes per team). Brief presentation This can also be an open time for a brief presentation to be made on a side topic, to invite a guest speaker or to allow a participant to make a special presentation. 45 minutes 1 Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). 34 FACILITATOR'S GUIDE IDENTIFYING OPTIONS In the previous session, we discussed how to clarify the problem. In this session, we discuss how to identify various options to address that problem. Again, the slide presentation follows a series of guiding questions that can be used to identify and assess various policy options. Objectives By the end of this session, participants will be able to: • describe ways to generate options to address a problem; and • review tools and strategies to evaluate and select policy options. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers. FACILITATION STEPS 1. EXPLAIN the objectives for this session (5 minutes). 2. PRESENT the slide set: Identifying options (45 minutes). 60 minutes Slide 1 In the previous session, we talked about how to clarify the problem. In this session, we will talk about how to identify the various options – the policy solutions – to address that problem. Slide 2 æ REVIEW the objectives. Slide 3 With regard to the evidence-informed policy-making framework, we are now at the second stage after clarifying the problem. We will now look into framing options (possible solutions) to address the problem. Identifying Options EVIPNet Europe Identifying Options Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World Health Organization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Identifying Options Objectives of the session •Describe ways to generate options to address a problem. •Review tools and strategies to evaluate and select policy options. Identifying Options Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 35 Slide 4 There are five questions that can help to organize our discussions and help us to think through the process of option framing. This structure is not rigid. Use these questions flexibly. æ ASK volunteers to read each question one at a time. There are two main themes here: • What are the possible options? • What evidence do we need in order to evaluate these options? Slide 5 Options should stem directly from the causes (underlying factors) of the problem. If we are clear about the causes we can identify options to impact those causes. For example, an option could be a new policy to provide a programme or service, a policy change in the health system, or a new way of implementing an existing policy. æ ASK everyone to think of an option related to a problem identified in the earlier session. æ INVITE each participant to share his/her option. ENCOURAGE them to keep this option in mind throughout this session. Slide 6 A good place to start is the causes of the problem and where those causes lie within the health system (e.g. delivery, financial, governance). For example, Is the problem caused by a lack of human resources or finances? To facilitate the process of identifying the options, you can look at the results of a systematic review related to your problem. Perhaps the results of a systematic review on the effectiveness of lay health workers might help to outline options to solve your human resource problem among health care workers. Look at how other regional neighbours have addressed the same problem. You do not need to copy their interventions but you can learn from their experience. You can also consult key informants who have specific knowledge in this area. Finally, your team can brainstorm its own ideas. æ FACILITATE a brief discussion with these prompts: • Which one of these strategies might be useful with your chosen problem? • Have you tried any of these so far? If so, what was the outcome? æ ASK: What questions do you have at this point? Slide 7 The causes of and solutions to health system problems are complex. Sometimes, the best option will be a specific intervention. Other times it will be a broad-based intervention or even a number of interventions linked together to address the problem. æ READ the example. This is a complex intervention involving also community health workers and will likely need multiple policy components. Identifying Options Questions to consider 1. What are the possible options to address the problem? 2. What benefits/harms are likely with each option? Which ones are important to those who will be affected? 3. What are the local costs of each option? Is there local evidence about their cost–effectiveness? 4. How can each option be adapted? How might that alter its benefits, harms and costs? 5. Whose opinions might influence the acceptability of an option and its benefits, harms and costs? Identifying Options 1. What are the possible options? • Should flow logically from the description of the problem and its causes. • Types of option: – new policy to provide cost-effective programme, service or drug – policy change in the health system framework (governance, financial, delivery) – new strategy to better implement an existing policy. Identifying Options • Consider where the cause lies within the health system (a delivery, financial and governance problem). • Look at interventions described in systematic reviews or overviews of reviews. • Consider ways in which other jurisdictions have addressed the problem. • Consult key informants. • Use brainstorming. Strategies to identify potential options Identifying Options Health system problems are complex • So solutions may be complex, e.g. – organizing a system to recruit, train, support and retain community health workers to deliver maternal child care. • May need to combine several options: – including different delivery, financial and governance arrangements. 36 FACILITATOR'S GUIDE Slide 8 Consider effectiveness studies that have been conducted globally to see the effect of the intervention. Remember to consider particularly the effects on those groups easily marginalized by policies and practices. Ask yourself questions such as: • What will the effect be on special groups (like children under 5 years, migrants, pregnant women)? • What might we observe if we intervene and if we do not intervene (keep the status quo)? • How will process indicators (such as coverage rates) of health benefits change under the given intervention? Slide 9 There may be some unintended consequences of an option. Again, look at the local and global literature for negative consequences of policies, especially among vulnerable groups. Studies designed to examine harmful effects or effectiveness can be helpful if they have data comparing two or more groups, settings or situations. We can then apply those results to our own setting. Slide 10 Before recommending a policy, you may need to review economic evaluations to consider what it will cost to implement and other economic implications. Economic evaluations of policies and programmes compare the costs and consequences of alternative interventions to determine the best use of scarce resources. This comparison may be done from the viewpoint of a payer (such as government and insurance companies), from the viewpoint of a provider (such as a health manager or hospital board) or from that of health care recipients or taxpayers. Stakeholders may view costs differently. What a provider considers an expense may be seen as a benefit by society. One option may appeal to the government but be rejected by citizens because it increases taxes. æ INVITE participants to share any experiences they have using economic evaluations, if any. æ ASK participants which cost issues or debates existed in their particular health policy. DISCUSS briefly. Slide 11 The evidence supporting various options often comes from elsewhere. How do we know whether it will work in our setting? Many of the studies we have just mentioned are quantitative and talk about the positive and negative effects of a policy or programme. Qualitative studies and qualitative evidence synthesis consider settings and context and note variables that can be modified. Identifying Options 2. What are the potential benefits? • Look at effectiveness studies among different populations, comparing different variables or assessing particular outcomes: ̶ effects of policy options on groups (e.g. children under-5 years, pregnant women, immigrants, teenagers) ̶ comparing with status quo. ̶ range of process indicators (e.g. coverage rates achieved) and outcomes (e.g. mortality). Identifying Options 2. What are the potential harms or unintended consequences? • Look at studies that examine effectiveness or harms among different populations, comparing different variables or assessing particular outcomes: ̶ same participants and variables ̶ assess a diverse set of potential harms. Identifying Options 3. What are the costs and cost– effectiveness of each option? • Look at economic evaluations conducted in the same setting and using a similar viewpoint (e.g. payer, provider or society at large). • Stakeholders may view costs differently. Identifying Options 4. How can options be adapted? • Look at qualitative studies carried out with a study of effects to determine how and why an intervention works. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 37 Slide 12 Reports about stakeholder experiences with the proposed option may also be informative. They may reveal, for example, how policy-makers view the option, how local opinion leaders speak about it or how the media portrays it. For example, contraception is a good policy option but what does the Catholic church say about it (and how many people listen to the church)? Consider the views of stakeholders of all kinds. All of these steps help you to extensively explore a given option before you present it to a policy-maker. æ ASK: Who are the influential opinion leaders for your particular health problem? DISCUSS briefly. Slide 13 A table like this can help you to compare the options using these criteria. It can also highlight areas of missing information and keeps the process systematic and transparent. Do not judge an option's feasibility in advance because you anticipate strong reactions. Do not eliminate an option just because you think it will not be popular or that local culture or politics will not accept it. This is falling back into the practice of opinion-based policy-making rather than evidence-informed policy-making. Leave the final choice for the decision-makers. Remember that evidence-informed health policy-making is not decision-making. It simply brings in arguments. Decision- making is a political process. æ ASK for any comments or questions. Slide 14 æ REVIEW the key points (or ask volunteers to read them). Slides 15 and 16 Resources (see slide deck) FACILITATION STEPS (contd) 3. ASK for any final comments or questions about identifying options (10 minutes). Identifying Options 5. Whose opinions about these options might be influential? • Look at qualitative studies of stakeholders’ views and experiences. Identifying Options Comparing options Option 1 Option 2 Benefits Harms Cost–effectiveness Stakeholder views Identifying Options • Policy options should stem naturally from the description of the problem and its cause(s). • Revisit the problem often to get an optimal list of options. • Use the five questions to explore the problem and possible options. • Consider the options being discussed currently by decision-makers. Key points 38 FACILITATOR'S GUIDE ACTIVITY TIME Use this time slot to lead an activity that will encourage participants to apply what they have learned in an interactive exercise. There are three options to consider below. IDEAS FOR ACTIVITIES Small group work Time for participants to work on a real project within their local context. Groups can continue their work from the previous activity and identify options to address the problem using the approach described in this session. Group brainstorming exercise One participant describes a real-life health system problem that is a current focus of policy work in their area. Facilitate a discussion with the full group to help to identify various policy options for this problem, using questions from the presentation: • What are the possible options to address the problem? • What benefits/harms are likely with each option? Which ones are important to those who will be affected? • What are the local costs of each option? Is there local evidence about their cost–effectiveness? • How can each option be adapted? How might that alter its benefits, harms and costs? • Whose opinions might influence the acceptability of an option and its benefits, harms and costs? Brief presentation Use this time to make another brief presentation, invite a guest speaker, show a video or allow a participant to make a special presentation, etc. 90 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 39 WRAP UP This session summarizes the key points from the day and encourages participants to reflect on their learning and give input on the Workshop delivery. Objectives By the end of this session, participants will be able to: • summarize the day’s topics and achievements; and • provide feedback about the Workshop so far. Materials • Handout: Daily feedback. FACILITATION STEPS 1. THANK participants for their enthusiasm and hard work. 2. REVIEW the main topics from the day quickly. 3. ASK participants for any final questions or comments about the day. 4. DISTRIBUTE the Daily feedback handout. Invite participants to write what they enjoyed the most, where they learned the most, where they still feel confused and what should be changed tomorrow. 5. ATTEND to any remaining logistics and CLOSE the day. AFTER THE DAY • REVIEW participants' feedback. • PREPARE a SUMMARY for the following morning. • Review today's progress and tomorrow's agenda. Adjust the content and timing as needed based on participant feedback. • Remove flip charts that are no longer needed and prepare materials for the next day. • Record any impressions and/or suggestions for future Workshops on the Facilitator feedback form. 15 minutes 40 FACILITATOR'S GUIDE DAY 2 The EIP framework that we are working with has three phases. On Day 1, participants explored the first two phases of the EIP framework: clarifying a problem and identifying options to address the problem. Today's sessions focus on the third phase of the framework: implementation considerations of policy options. Before addressing the third phase, the morning sessions, concentrate on how to use systematic reviews as a source of evidence in policy-making. They cover what systematic reviews are and where to find them (Finding evidence: systematic reviews) and how to evaluate systematic reviews for methodological quality (Assessing a systematic review). Both sessions give participants hands-on experience in finding reviews online and using checklists to appraise them, which emphasizes practical skill building. The activity time session can be used for more of this hands-on practise or to apply these skills to a specific project. The afternoon begins with a presentation on identifying implementation considerations and identifying strategies to address these barriers (Implementing policy options). There is a second opportunity for a hands-on activity to which participants can apply their learning so far or for continuing working on a specific group project. The sample agenda is given in Table 3, with the materials needed for the whole day. By the end of today, participants should have a better understanding of the three phases of the EIP framework. 2 Table 3. Sample agenda for Day 2 SAMPLE AGENDA ESTIMATED TIME MATERIALS Outlook on Day 2 15 minutes PowerPoint slides Laptop LCD projector Attendance sheet Name tags Flip charts Coloured markers Participant folders Copies of handouts: • Finding and using research evidence • AMSTAR 2 checklist • SURE checklist • Daily feedback form Finding evidence: systematic reviews 75 minutes Break Assessing a systematic review 90 minutes Activity time 45 minutes Lunch Implementing policy options 60 minutes Break Activity time 90 minutes Wrap up 15 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 41 OUTLOOK ON DAY 2 As you review the key concepts from yesterday, highlight again the three phases of the policy process. This will help participants to see how they are moving from the two phases discussed yesterday to the third phase in today’s work. 1. WELCOME everyone back to the Workshop. 2. SUMMARIZE briefly the participant feedback from yesterday and describe any adjustments planned for today. 3. ASK participants to name the main accomplishments or outcomes from yesterday. Responses may include: • group rapport • better understanding of the policy process • know more ways to describe or frame a health problem. 4. REVIEW the main topics and activities planned for the day. Explain that yesterday we covered the first two phases of the policy process: clarifying the problem and framing options. Today we move to the third phase of the policy process: implementing options. 5. ASK for questions. 6. LEAD a brief learning assessment exercise to review key concepts from yesterday. For ideas, see Quick exercises to assess participant learning (Section IV). 15 minutes 42 FACILITATOR'S GUIDE FINDING EVIDENCE: SYSTEMATIC REVIEWS This session will introduce participants to systematic reviews as a source of evidence to assess policy options. Some participants may be less familiar with systematic reviews and may need more time to navigate the websites in the hands-on portion of the presentation. Consider using more experienced participants as peer mentors during this activity. Objectives By the end of this session, participants will be able to: • identify sources of research evidence about a problem, options and implementation strategies; • describe the main features of systematic reviews and their advantages over single studies; • clarify myths and misunderstandings of systematic reviews; and • review useful sources of systematic reviews and other types of evidence. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers • Handout: Finding and using research evidence. FACILITATION STEPS 1. EXPLAIN the objectives for this session (5 minutes). 2. PRESENT the slide set: Finding evidence: systematic reviews (55 minutes). 90 minutes Slide 1 We have clarified the problem and considered different policy options. Next, we will need evidence to assess these options. Systematic reviews are a good source of that evidence. Slide 2 æ ASK a volunteer to read the objectives. Finding Evidence: Systematic Reviews EVIPNet Europe Finding Evidence: Systematic Reviews Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World Health Organization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Finding Evidence: Systematic Reviews Objectives of the session • Identify sources of research evidence about a problem, options and implementation strategies. • Describe the main features of systematic reviews and their advantages over single studies. • Clarify myths and misunderstandings of systematic reviews. • Review useful sources of systematic reviews and other types of evidence. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 43 Slide 3 æ REFER to the handout Finding and using research evidence. This handout from McMaster University lists some useful sources of evidence to answer particular questions. The information in this first section relates to clarifying a problem. æ POINT to this section on the handout and highlight any key information. Slide 4 The middle section relates to finding options. æ POINT to this section on the handout and highlight any key information. Slide 5 æ ASK participants who have ever used a systematic review to raise their hands æ INVITE one of them to define a systematic review. æ READ the definition of a systematic review. A systematic review is broader than a classic meta-analysis. Systematic reviews sometimes include meta-analyses in their process. Slide 6 A meta-analysis is a statistical approach to combine the data derived from systematic reviews. Slide 7 This slide is animated This is the process of a systematic review. CLICK to display each step Remember that reviewers do not make recommendations. They simply interpret the evidence and summarize it. Slide 8 Systematic reviews can be conducted for several types of research. æ REVIEW the types. Finding Evidence: Systematic Reviews Finding evidence: the problem Source: Lavis JN (2012). Finding and using research evidence. Hamilton: Canada: McMaster Health Forum. Finding Evidence: Systematic Reviews Finding evidence: options Source: Lavis JN (2012). Finding and using research evidence. Hamilton: Canada: McMaster Health Forum. Finding Evidence: Systematic Reviews Systematic review • A systematic review summarizes the results of available health care studies (controlled trails) to offer a high level of evidence on the effectiveness of health care interventions. • Reviewers methodically follow a protocol that describes: - the way existing studies are found - how the studies are judged in terms of their usefulness in answering the review questions - how the results of the separate studies are brought together to give overall measure of effectiveness. Finding Evidence: Systematic Reviews Meta-analysis “…a statistical analysis of the results from independent studies, which generally aims to produce a single estimate of a treatment effect.” Source: Egger M, Smith GD, Altman DG (2008). Systematic reviews in health care, second edition. London: BMJ Books. Finding Evidence: Systematic Reviews Process of a systematic review • Formulate the question • Define criteria to include and exclude studies Ask a focused question • Search for and select the studies • Assess study quality (risk of bias) Identify and select studies • Collect relevant data • Analyse and summarize (meta-analysis) Extract data • Interpret the resultsForm conclusions Finding Evidence: Systematic Reviews Systematic reviews may also involve: • Administrative database studies and community surveys that help to place problems in comparative perspective. • Observational studies that help to characterize an option’s likely harms. • Qualitative studies that help to understand the meanings that individuals or groups attach to a problem, how and why options work, and stakeholders’ views about and experiences with particular options. 44 FACILITATOR'S GUIDE Slide 9 æ ASK: Can anyone explain the difference between a systematic review, an overview of systematic reviews and an evidence brief for policy? ENCOURAGE responses. æ REVIEW the definitions from the slide. æ ASK if everyone is clear about the differences between meta analysis, systematic review, overviews, and an evidence brief. ENCOURAGE questions. Slide 10 This slide is animated æ ASK participants to brainstorm some advantages that systematic reviews have over single studies. ENCOURAGE responses. CLICK With so many studies being published, it is challenging to keep up with the literature. Single studies may have results that apply only to a select population or may even contradict results from another similar study. When these studies are looked at together within a systematic review, a clearer (and more consistent) picture will emerge. Using reviews that summarize the outcomes of multiple intervention studies saves time and shows stakeholders that the evidence has been compiled systematically. Slide 11 When using results from multiple studies summarized in a systematic review, policy- makers are more likely to make balanced decisions and have more confidence on what they can expect from those policy decisions. Slide 12 The classic systematic review looks at clinical evidence produced by randomized controlled trials. But there are a number of other types of systematic review of qualitative research evidence (or systematic reviews of both qualitative and quantitative research evidence in the same review). They go by many names. æ READ the list. These types of qualitative review can be just as useful for policy-making. æ ASK: Do you know about these or have you actually done these? Most policy-makers are not familiar with systematic reviews, how to do them or how to use the results. You may need to explain systematic reviews and their value in the policy-making process more fully. Also, you may have to teach policy-makers where to find systematic reviews and how to appraise their quality and utility. This will help to build local capacity for evidence-informed policy-making. Finding Evidence: Systematic Reviews Some jargon • Systematic reviews. • Overview of systematic reviews: ̶ mapping and synthesis of all reviews in a given domain (e.g. human resources for health, primary health care). • Evidence briefs for policy: ̶ synthesis of reviews used specifically to inform problems, frame policy options and plan implementation. Finding Evidence: Systematic Reviews Why systematic reviews? Advantages over single studies in defining problems or framing options. • Saves time for policy-makers. Evidence has already been collected and summarized, so policy-makers can focus on applying the results locally. • Stakeholders can constructively contest research evidence when presented in a more systematic and transparent way. Finding Evidence: Systematic Reviews Why systematic reviews? Advantages over single studies in describing the effectiveness (benefits) of a policy option. • Reduce the likelihood that policy-makers will be misled by single study outcomes. • Increase policy-maker confidence in what they can expect from an intervention (by increasing the number of units for study). Finding Evidence: Systematic Reviews Other names for qualitative and mixed-methods reviews Narrative summary/thematic analysis/ grounded theory/meta-ethnography/ meta-study/realist synthesis/ cross-case techniques/content analysis/ case survey/qualitative comparative analysis/ Bayesian meta-analysis. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 45 Slide 13 A number of constraints have limited the use of systematic reviews in policy-making. Apart from the common misconceptions about them shown in the slide, sometimes systematic reviews can be difficult to locate within different databases. They can also be hard to understand because they are long and are often written in highly technical language. Slide 14 æ REVIEW the stages. In this session, we will discuss how to find and select systematic reviews. We will cover the final two stages in the next session. Slide 15 These are some useful sources of systematic reviews. æ REVIEW the list and ask if anyone has used any of these sources before. æ INVITE participants to open their laptops to look at the sites during the discussion. Slide 16 Health Systems Evidence compiles syntheses of research evidence specifically on health systems covering topics such as governance, financial and delivery arrangements within health systems, and about implementation strategies that can support change in health systems. It is updated regularly. It can help you frame your questions and to find evidence about options. Slide 17 æ REVIEW the different elements of the search refinement. Slide 18 Notice how these boxes reflect the same framework categories/health system arrangements we talked about in the previous sessions (governance, finance and delivery). Finding Evidence: Systematic Reviews Constraints in using systematic reviews in policy-making • Common misconceptions: • only for randomized controlled trials • only for effects • only for those working in biomedical paradigms • only if statistical synthesis is possible. • Hard to retrieve – need “one-stop shopping”. • Hard to understand – need more user-friendly summaries. Finding Evidence: Systematic Reviews Using systematic reviews: main steps • Find available systematic reviews. • Select relevant systematic reviews. • Judge how well a systematic review applies. • Judge how much confidence to place in a systematic review. Finding Evidence: Systematic Reviews Finding systematic reviews • Health Systems Evidence www.healthsystemsevidence.org • The Cochrane Library www.thecochranelibrary.com • Health Evidence http://www.healthevidence.org/ • PDQ Evidence http://www.pdq-evidence.org/ • PubMed www.pubmed.gov • SUPPORT summaries http://www.support- collaboration.org/summaries.htm Health Systems Evidence Source:www.healthsystemsevidence.org Health Systems Evidence Source:www.healthsystemsevidence.org Health Systems Evidence Source:www.healthsystemsevidence.org 46 FACILITATOR'S GUIDE Slide 19 We will take 10 minutes to go through this interface. You must register before you can navigate the site. æ ASSIST participants to register and guide them through this interface. Slide 20 This is the interface that appears when you click on one of the findings. It summarizes the topics, content and quality of a systematic review. Slide 21 The Cochrane Library is another source for high-quality, accessible synthesized research evidence for topics in health care. You may have to pay for access to the Cochrane Library. Sometimes you can access this in your local library. You can enter search criteria into the top boxes and then select which domain you would like to search. In this example, we are searching for reviews about community health workers within the databases of Consumers and Communication and Effective Practise/Health Systems. æ POINT to red arrows. Slide 22 Here are the results of our search. If we click on the first record, the full article will appear. æ ASSIST participants, as needed, to navigate the screen. Slide 23 This is the article. Slide 24 The Cochrane guidelines require that information like that shown above is presented explicitly. This is helpful in appraising the quality of a given systematic review. Health Systems Evidence Source:www.healthsystemsevidence.org Health Systems Evidence Source:www.healthsystemsevidence.org Finding Evidence: Systematic Reviews The Cochrane Library So ur ce : w w w .th ec oc hr an el ib ra ry .c om Finding Evidence: Systematic Reviews The Cochrane Library So ur ce : w w w .th ec oc hr an el ib ra ry .c om Finding Evidence: Systematic Reviews The Cochrane Library So ur ce : w w w .th ec oc hr an el ib ra ry .c om Finding Evidence: Systematic Reviews The Cochrane Library So ur ce : w w w .th ec oc hr an el ib ra ry .c om SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 47 Slide 25 On the Cochrane site you can search using a key search term (or MeSH term) in addition to using the advanced search feature with more details (e.g. year of publication). æ POINT out these key features. Slide 26 Health Evidence is another source of systematic reviews focused on the effects of public health interventions/programmes. It currently includes close to 5000 systematic reviews critically appraised regarding their "quality" (reliability) and relevance. The reviews are identified using electronic and hand searches. A short summary with the main findings and practice and policy implications is also included. Slide 27 PDQ (pretty darn quick) Evidence is another useful resource where you can find systematic reviews and overviews related to health systems interventions. Slide 28 The search is relatively simple and you can use free text words in seven different languages. Slide 29 We have used an example related to a policy on tobacco plain package. In the following slides you can see what PDQ is able to display from each record. Slide 30 On the right side, there is a list of the records identified with additional information for each of them. Finding Evidence: Systematic Reviews The Cochrane Library So ur ce : w w w .th ec oc hr an el ib ra ry .c om Finding Evidence: Systematic Reviews Health Evidence So ur ce : h ttp :// w w w .h ea lth ev id en ce .o rg / Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / 48 FACILITATOR'S GUIDE Slide 31 Then, for each individual record… Slide 32 … you can find the number of studies included in the review and whether the review has been included in any overview of systematic reviews (identifying related reviews in the field). Slide 33 You can also find general summarized information about the review. Slide 34 æ DIRECT participants to open the PubMed website. PubMed lists more than systematic reviews. So be sure to focus your search. Slide 35 æ POINT to the red arrow. Here we want to search systematic reviews about task shifting. Slide 36 The top box shows the terms you searched for. æ POINT to the upper arrow. You will also see links to citations related to the publication listed. æ POINT to the lower arrow. Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / Finding Evidence: Systematic Reviews PDQ-Evidence So ur ce : h tt p: // w w w. pd q- ev id en ce .o rg / PubMed Source: www.pubmed.gov PubMed Source: www.pubmed.gov PubMed Source: www.pubmed.gov SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 49 Slide 37 You can manage the filters in a given search using the links shown by the arrow. You can filter the papers by language, publication date, and so on. æ POINT to the filter area near the red arrow. Slide 38 æ DIRECT participants to open the SUPPORT website. SUPPORT presents results for health system questions in a more user-friendly way. It is most relevant to low-income countries. They are summaries of systematic reviews prepared by health system researchers independent from the reviews' authors. Slide 39 Here is a SUPPORT summary on the impact of user fees on access to health services. Notice the Key Messages section and the summary of what is included and not included. æ POINT to these sections. Slide 40 The next page offers some background and tells us how the systematic review was actually conducted. Slide 41 The summary of findings is very easy to read and understand. Slide 42 This chart explains how relevant the findings might be to low- and middle-income countries. Finding Evidence: Systematic Reviews PubMed Source: www.pubmed.gov SUPPORT summaries Source: http://www.support-collaboration.org/summaries.htm SUPPORT summaries Source: http://www.support-collaboration.org/summaries.htm SUPPORT summaries Source: http://www.support-collaboration.org/summaries.htm SUPPORT summaries Source: http://www.support-collaboration.org/summaries.htm SUPPORT summaries Source: http://www.support-collaboration.org/summaries.htm 50 FACILITATOR'S GUIDE Slide 43 And finally, this page shows where to go for additional information. Slide 44 There are many systematic reviews available, so have clear criteria to select those that meet your current need. You may choose to include or exclude studies based on methodology, target populations, types of intervention, and so on. Slide 45 If you want to define a specific problem in a specific setting, you may want to restrict your reviews to that or similar settings. On the other hand, you may need to look extensively at what other jurisdictions have done to find the right solution to the problem. So you may not want to restrict the setting. Always consider the context in which the study was conducted compared with yours even if it is in the same country, age group, time, etc. Slide 46 æ ASK for any questions or comments about finding or selecting systematic reviews. Slide 47 æ REVIEW the key points (or ask a volunteer to read them). Slide 48 Resources (see slide deck) FACILITATION STEPS (contd) 3. In the time remaining (approximately 30 minutes), ENCOURAGE participants to practise navigating the different websites by searching for reviews on their own personal topic of interest. 4. Offer assistance and answer questions as you walk around the room. SUPPORT summaries Source: http://www.support-collaboration.org/summaries.htm Finding Evidence: Systematic Reviews Selecting systematic reviews • Explicit selection criteria help to guide judgements and add transparency. • Criteria might specify: – minimum methodological requirements – population, patient or problem types that are addressed – Types of intervention, option, health systems arrangement or implementation strategy that are addressed. Finding Evidence: Systematic Reviews Selecting systematic reviews Should we establish criteria for “setting” (e.g. primary care, global region)? • Restricting reviews to specific countries or low- and middle-income countries may exclude important evidence (sometimes the best available evidence). • Can be more informative to include reviews with studies from diverse settings and then assess the applicability of the results to your setting. Finding Evidence: Systematic Reviews Questions or comments? Finding Evidence: Systematic Reviews Key points • Systematic reviews are just a good starting point. They should not be your only evidence. • Get familiar with a few places to find systematic reviews (e.g. Health Systems Evidence, The Cochrane Library). • Look for a local person involved in Cochrane or another “SR organization” to build partnerships. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 51 ASSESSING A SYSTEMATIC REVIEW This is another hands-on skill-building session in which participants practise using two tools to assess elements of systematic reviews. The following activity time can be used to extend this practise or for participants to assess a systematic review for a real-life policy project. Objectives By the end of this session, participants will be able to: • discuss the importance of appraising systematic reviews in evidence-informed policy making; • describe techniques to appraise systematic reviews; and • explore ways to appraise other types of study. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers • Handouts: AMSTAR 2 checklist, SURE checklist. FACILITATION STEPS 1. EXPLAIN the objectives for this session (5 minutes). 2. PRESENT the slide set: Assessing a systematic review (75 minutes). 90 minutes Slide 1 In the last session we became familiar with locating systematic reviews. Now we will discuss how to evaluate those reviews. Slide 2 After we select our evidence, the next step is to assess how much confidence we can place in the given reviews. We will look at the advantages of different methods. We will also discuss how to appraise other sources of evidence if we do not have a systematic review. Assessing a Systematic Review EVIPNet Europe Assessing a Systematic Review Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World Health Organization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Assessing a Systematic Review Objectives of the session • Discuss the importance of appraising systematic reviews in evidence-informed policy-making. • Describe techniques to appraise systematic reviews. • Explore ways to appraise other types of study. 52 FACILITATOR'S GUIDE Slide 3 This slide is animated æ ASK: What do we mean by confidence in a study or a review? ENCOURAGE responses. CLICK to display the text. æ ASK a volunteer to read the definition. Slide 4 Note that confidence in a systematic review is not the same as confidence in an estimate of effect derived from the review. In order to make this distinction and be able to answer any additional query from the participants, the facilitator should have a good understanding of the GRADE methodology. Slide 5 This slide is animated æ ASK: What do we mean by bias? ENCOURAGE responses. CLICK to display the text. æ READ the definition. Slide 6 æ REVIEW the different types of bias briefly. æ ASK participants to give examples of any of the types of bias. ENCOURAGE responses. Slide 7 æ DEFINE the terms. æ ASK for any questions. Assessing a Systematic Review Confidence in a systematic review • The extent to which we can be sure that the review provides a complete and accurate summary of the best available evidence. • Based on methods used to: – identify, include, and critically appraise studies – analyse the findings. Assessing a Systematic Review Confidence in an estimate of effect • The extent to which we can be sure that an estimate of effect is correct (or adequate to support a particular decision) is based on judgements about: – risk of bias – imprecision – consistency – directness. Assessing a Systematic Review Bias • Deviation from the truth in results or inferences. • Systematic error introduced into sampling or testing by selecting or encouraging one outcome or answer over others. Source: Viswanathan M et al. (2012). Assessing the risk of bias of individual studies in systematic reviews of health care interventions. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville (MD): Agency for Healthcare Research and Quality; 2012and Comparative Effectiveness Reviews. Rockville (MD): Agency for Healthcare Research and Quality (US);. Assessing a Systematic Review Bias • Systematic differences in: – comparison groups (selection bias) – intervention that is provided or exposure to factors apart from the intervention (performance bias) – withdrawals or exclusions of people entered into a study (attrition bias) – how outcomes are assessed (detection bias) – reporting of outcomes (reporting bias). Source: Higgins JPT, Green S, editors (2011). Cochrane handbook for systematic reviews of interventions, version 5.1.0. London: The Cochrane Collaboration. Assessing a Systematic Review Risk of bias • Extent to which bias may be responsible for a study’s findings. • Assessing risk of bias is similar to assessing the validity or quality of a study. • Validity: extent to which a result (of a measurement or study) is likely to be true. • Quality: vague notion of the strength or validity of a study. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 53 Slide 8 æ REFER participants to their handouts of the SURE and AMSTAR checklists. There are many ways to assess systematic reviews. These are just two. AMSTAR stands for A Measurement Tool to Assess Systematic Review. SURE stands for Supporting the Use of Research Evidence. SURE guides are an important instrument to EVIPNet Europe to strengthen evidence-informed policy- making at country level. Results of the SURE team include evidence briefs for policy, policy dialogues, rapid responses, SURE guides and international workshops among others. The primary reason to use a checklist is to help you to think critically about the review. It helps you to assess if an intervention works and if it is cost-effective. It also helps you to determine if their conclusions are grounded in systematic rules/standards. You can apply these same principles to systematic reviews and to single studies. æ ASK: • Has anyone experience using either of these tools? • How did you use it? Slide 9 æ REFER participants to the AMSTAR 2 checklist. AMSTAR has significantly advanced the practice of assessing the methodological quality of systematic reviews. You could also use this checklist as you prepare a systematic review to ensure a high-quality review. Slide 10 AMSTAR has many advantages. Most importantly, it is very practical and easy to use. It only takes a few times for the checklist process to feel natural. æ REVIEW briefly the questions on the AMSTAR 2 guide. Slide 11 æ REVIEW the steps, referring often to the handout. Sometimes you will not have clear answers and will have to use value judgements. But this checklist gives you a systematic and transparent way to make that judgement. Assessing a Systematic Review Appraisal checklists • AMSTAR • SURE checklist Assessing a Systematic Review • Valid, reliable and useable instrument to help users to differentiate between systematic reviews, focusing on their methodological quality and expert consensus. • To facilitate the development of high-quality reviews. AMSTAR Assessing a Systematic Review AMSTAR’s advantages • Strong interrater agreement, test – retest reliability, face and construct validity. • Endorsed by the Canadian Agency for Drugs and Technologies in Health (CADTH) and several authors (cited over 200 times in 2014 –2017). • High reliability of the total AMSTAR score. • Practical: short completion time, easy to arrive at a final decision because of the comprehensible guidelines. Assessing a Systematic Review Using AMSTAR • Read each question and answer yes or no. • Total the number of “yes” answers. The more you have, the more confident you can be in the systematic review. • Does each “yes” actually weigh the same? 54 FACILITATOR'S GUIDE Slide 12 æ REFER participants to the SURE checklist handout. The primary benefit of SURE is that it draws on other checklists (including AMSTAR) to update and get a reliable summary. It is more focused on health systems questions than AMSTAR. Slide 13 æ REVIEW briefly the questions in the SURE checklist. Slide 14 After the assessment, place the review into one of three categories: • reliable • with limitations • with fatal flaws. If there are only minor limitations, the review can be used. It would be prudent, however, to search for more reviews. Slide 15 A fatal flaw is a limitation so significant that the review should not be used. You may, however, draw some key messages or useful concepts from the review. Slide 16 The SURE collaboration has a guide to assessing single studies. Check on the SURE website. These guides ease the process but it is still your value judgement. æ ASK for any comments or questions. Assessing a Systematic Review SURE checklist • Based on other similar checklists. • Developed by applying a widely used checklist to systematic reviews of health system arrangements and implementation strategies. • Helps you to judge how likely a review provides a reliable summary of the best available evidence of the impacts of these complex interventions. Assessing a Systematic Review SURE checklist • Two sections describe the methods used to: – identify, select and critically appraise studies – analyse the results of included studies. • Summary assessments are based on the questions: – minor, moderate or major limitations. – can guide the use of reviews in policy briefs. Assessing a Systematic Review Summary assessments • Reliable: good to use. • With limitations: – can use the systematic review but limitations are important enough to searching for another systematic review – interpret results cautiously if no better review available – can supplement information from this review with additional searches or include studies in the policy brief. Assessing a Systematic Review Summary assessments • A summary assessment may reveal fatal flaws: ̶ limitations are sufficiently important to render the results of the review unreliable ̶ do not use the results in the policy brief ̶ may draw some key messages or useful information from the review (e.g. framework to identify potential options). Assessing a Systematic Review If you don’t have a systematic review… • Search for individual studies to supplement or stand in place of a systematic review. • Use the same processes for a systematic review: – use systematic and transparent methods to find, select and critically appraise studies and to synthesize the results. – describe your methods in an appendix to the policy brief. Source: Evidence-Informed Policy Network (2017). SURE guides for preparing and using evidence-based policy briefs. Geneva: World Health Organization . SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 55 Slide 17 æ ASK participants to form pairs or small groups. æ GIVE instructions from the slide and allow 30 minutes to complete the activity. æ ROTATE among groups to answer questions. æ FACILITATE a brief discussion with the full group using these prompts: • What did you learn from this activity? • Was any part confusing or difficult to use? æ DEBRIEF with these key points. • The abstract may not reveal all of the information you need. • You will soon know where to go automatically (to which sections) to answer particular questions on the checklist. • Find systematic reviews from different sources (e.g. Cochrane, Health System) and then score them using the different tools. This will help you to see the pros and cons, values and limits of each instrument. Slide 18 æ REVIEW the key points (or ask a volunteer to read them). æ ASK for final questions or comments. Slides 19 and 20 Resources (see slide deck) FACILITATION STEPS (contd) 3. FACILITATE a brief discussion with these prompts (10 minutes). • How likely do you think you will be to use these tools in your work? • Can you describe a situation in your current work in which you might use a systematic review? • What might be some of the challenges you could face as you try to incorporate systematic reviews into your work or into local policy-making? How might you address these challenges? 4. ASK for final questions. 5. EXPLAIN that in the next session we will see how systematic reviews and other evidence are involved in the third phase of policy-making: implementing policy options. Assessing a Systematic Review Exercise • Choose one systematic review you found in the previous session. • Use the AMSTAR framework to appraise its quality. Assessing a Systematic Review Key points • There are many tools and instruments available. Keep it simple. • Checklists help you to think, but you must make the final decision. • Even if you find flaws, a systematic review or single study can still be useful. 56 FACILITATOR'S GUIDE ACTIVITY TIME Use this time slot to lead an activity that will encourage participants to apply what they have learned in an interactive exercise. There are three options to consider below. IDEAS FOR ACTIVITIES Small group work Time for participants to work on a real project within their local context. Finding and assessing a review Participants can work individually or in groups to find a systematic review that is pertinent to a real-life policy situation. Participants can also use one of the checklists to evaluate that review. Brief presentation If you are integrating the evidence-informed policy-making sessions into other topic areas, use this time to make another brief presentation, invite a guest speaker, allow a participant to make a special presentation, etc. 45 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 57 IMPLEMENTING POLICY OPTIONS After sessions on clarifying problems and identifying options, we now move into the third phase to identify next steps once the options are considered. The emphasis is on anticipating factors that can promote (facilitators) or interfere (barriers) with implementation. Objectives By the end of this session, participants will be able to: • describe facilitators and barriers to implementing policy options; and • use different methods to identify implementation facilitators and barriers and strategies to address them. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers. FACILITATION STEPS Before you begin this session, consider leading a brief learning assessment activity to determine how well participants have learned key concepts from the previous session. It also helps to focus the group energy and to bring participants together after the break. Ideas for brief learning assessments are given in Section IV. 1. EXPLAIN the objectives for this session (5 minutes). 2. PRESENT the slide set: Implementing policy options (45 minutes). 60 minutes Slide 1 So far we have talked about clarifying problems and identifying options. Now we move into the third phase in which we consider how we will put these options into action. Slide 2 Putting options into practice and leveraging change are not always easy or straightforward. These changes will be more successful if you anticipate the barriers – those factors that may inhibit or make these changes difficult. You also need to develop strategies to address these barriers. æ REVIEW the objectives. Implementation considerations EVIPNet Europe Implementing policy options Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World H al h Organization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Implementation considerations Objectives of the session • Describe barriers and enablers to implementing policy options. • Consider the importance of identifying strategies to address implementation barriers. • Review ways to develop and select implementation strategies. 58 FACILITATOR'S GUIDE Slide 3 As we will see, evidence can also be used to inform how we might implement options. Slide 4 Putting options into practice and leveraging change are not always easy or straightforward. Little is known about the effectiveness of different methods for identifying barriers and tailoring interventions to address those barriers. Slide 5 Barriers and enablers are often, but not always, mirror images of each other. These are other words you might see in the literature. æ REVIEW the slide. Slide 6 In policy analysis, we tend to worry about the barriers and forget to look for factors that can work for us. Do not let the barriers overshadow what enablers might exist. There are many examples of enablers in Evidence Briefs for Policy on the WHO EVIPNet website. Slide 7 It is important to identify barriers and enablers in a structured way, just as we did in the previous steps. We do not want to miss any important factors, waste time on unimportant factors or fail to seek evidence of potential impacts. One way to add structure to this process is to use a checklist or a table that lists all levels where barriers or enablers might occur. A checklist can help us to answer questions. • Who will implement this policy option? • Who will embrace it? • Who will benefit or be challenged by it? Implementation considerations So ur ci e: La vi s JN , W ils on M G , O xm an AD , L ew in S, F re th ei m A (2 00 9) . SU PP O R T to ol s fo r e vi de nc e- in fo rm ed h ea lth p ol ic ym ak in g (S TP ) 4 : u si ng re se ar ch e vi de nc e to c la rif y a pr ob le m . H ea lth R es P ol ic y Sy st . 7 (S up pl 1) :S 4. Implementation considerations Implementing options • May require changes at various levels, e.g. – behavioural changes of health care recipients and providers; organizational changes. • Strategies to facilitate change will be more successful if they: – address important barriers to change – maximize enablers. Implementation considerations Barriers and enablers Barriers to change and enablers of change: •moderators and mediators •problems and incentives •barriers and facilitators. Implementation considerations Where barriers/enablers may occur • Recipients of care • Providers of care • Other stakeholders (e.g. media, civil society organizations) • Health system • Social and political systems Implementation considerations Framework to identify barriers/enablers • Structured approach helps to ensure we do not: – overlook important factors – pay too much attention to unimportant ones – fail to assess the evidence of their potential impact. • Using a checklist can be helpful. SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 59 Slide 8 Barriers and enablers can occur at individual, organizational and systems/ environmental levels. You can find many examples of barriers and enablers in real-life in Evidence Briefs for Policy on the EVIPNet website. æ ASK participants to quickly brainstorm one or two examples of barriers at each level. æ ASK participants to give examples of any enablers at these different levels. Slide 9 Here are some factors to examine at each stakeholder level. • Do they know what we are proposing? • Will they have the skills to implement this policy? • Are they willing to do it and have the tools to do it? • What might motivate them to take desired actions? The answers to these questions may be very different for each stakeholder group. Yet look for ways in which they are similar. A barrier or enabler that appears for multiple stakeholders is obviously a crucial one. Slide 10 Then consider system level issues – what you might need and what capacity exists to implement the policy option. æ READ a few examples from the table. Slide 11 This slide is animated æ ASK: What barriers might you encounter in the broader social and political environment? PROMPT with an example from the table to encourage responses. CLICK to display the table and READ the examples. Slide 12 These are some ways to identify and clarify the importance of potential barriers. Remember to consult any systematic reviews. æ REVIEW the list. æ ASK for any comments or questions. Implementation considerations Where barriers/enablers may occur • Recipients of care • Providers of care • Other stakeholders (e.g. media, civil society organizations) • Health system • Social and political systems Implementation considerations Recipients and providers of care Knowledge Competency Attitudes Access to care/supplies Motivation to change Implementation considerations Health system constraints Inadequate financial resources Inadequate processes Inadequate human resources Inadequate incentives Inadequate training Inadequate management Inadequate supervision Inadequate leadership Inadequate internal communication Inadequate procurement and distribution systems Inadequate external communication Inadequate allocation of authority Inadequate information systems Inadequate accountability Inadequate facilities Bureaucracy Implementation considerations Social/political constraints Ideology Inadequate contracts Short-term thinking Inappropriate donor practices Conflicting interests Opinion leaders Inappropriate norms Corruption Competing priorities Political instability Implementation considerations Methods to identify and clarify barriers • Brainstorming with a team of experts. • Contact key informants. • Search for published or unpublished studies: – country reports – case studies – interview studies – focus group studies – direct observations – Surveys. 60 FACILITATOR'S GUIDE Slide 13 There is no magic strategy that will address all barriers. You will need creative thinking to develop the best strategy to implement your policy option. Brainstorming brings together people with different types of expertise and perspectives to generate ideas. Slide 14 Brainstorming takes time but yields many benefits. æ REVIEW the slide. Slide 15 There is a lack of sufficient research evidence to support the use of specific interventions for specific barriers. But we can still add theory-based approaches to brainstorming to generate ideas about implementation. Sound theories add structure and can help to frame our thinking. Yet we still need to use our own best judgement on how to apply these theories to our policy situation. Slide 16 This is an example of how you might generate a list of strategies to address barriers at the level of health care recipients. For example, the strategy to address the barrier of poor knowledge may be to disseminate information. Again the strategies are very context-specific. Strategies to address the same barrier may differ across settings. æ REVIEW the table. Slide 17 This is an example of how you might generate a list of strategies to implement a policy at the level of health care providers. æ REVIEW the table. æ ASK participants to share examples from their own work of when a new policy encountered barriers from health providers or recipients. What was the barrier and what strategy was implemented to address that barrier? Slide 18 This is an example of how you might generate a list of strategies to address health system constraints. æ REVIEW the table. Implementation considerations Strategies to address barriers • Matching implementation strategies with key barriers and enablers relies often on brainstorming. • Structured or unstructured approach. • Typically face-to-face (or virtually). – participants respond to each other’s suggestions – generate a complete list of potential strategies and then evaluate ideas. Implementation considerations Advantages of brainstorming • Generating as many potential solutions as possible may increase the chance of finding a good one among them. • Discussing the merits of each solution idea can focus attention on the most promising ones. • Promotes ownership among real people who will help you in implementation. Implementation considerations Using theories to select interventions • Theory-based approaches can be used to add structure to brainstorming. • Provides a framework and approach to identify interventions. • Using theories still relies on logic and judgements. Implementation considerations Possible interventions for barriers: health care recipients Barriers Examples of implementation strategies Knowledge Disseminate information that is reliable and accessible, e.g. via mass media or community health workers Competency Provide training and support Attitudes Disseminate information on the size of the problem, including relevant comparisons Access to care Reduce financial or physical barriers to care Motivation to change Disseminate information to motivate people to seek care, use financial or material incentives, etc. Implementation considerations Possible interventions for barriers: health care providers Barriers Examples of implementation strategies Knowledge Disseminate educational materials Competency Educational meetings or outreach visits Attitudes Disseminate information on the size of the problem, including relevant comparisons; use opinion leaders Access to supplies Improve the distribution of necessary supplies Motivation to change Disseminate information to motivate health workers to change their practice; financial or other incentives; reduce the burden of changing practices Implementation considerations Possible interventions for health system constraints Barriers Examples of implementation strategies Inadequate financial resources Strategies to generate the necessary resources or reduce the cost of implementing the option Inadequate human resources Task shifting; training and support; adjust incentives to recruit and retain health workers where they are needed Inadequate training Change training programmes or develop new training programmes Inadequate supervision Train people to provide training/support; set appropriate incentives for supervisors; formal agreements and monitoring arrangements Inadequate internal communication Structured referral sheets, involve consultants in primary care educational activities SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 61 Slide 19 This is an example of how you might generate a list of strategies to address social and political constraints. æ REVIEW the table. Slide 20 æ ASK for any final questions or comments. Slide 21 æ REVIEW the key points (or ask volunteers to read them). æ ASK for any final questions or comments. Slide 22 Resources (see slide deck) FACILITATION STEPS (contd) 3. FACILITATE a brief discussion with these prompts (10 minutes). • Who can share an example from their own work of when a new policy encountered barriers from health providers or recipients? • What was the barrier and what strategy was implemented to address that barrier? 4. ASK for final questions. Implementation considerations Possible interventions for social and political constraints Barriers Examples of implementation strategies Ideology Systematic and transparent approaches to use evidence to inform decisions; more transparent decision-making processes Short-term thinking More systematic and transparent approaches to judge the pros and cons of options and make decisions Conflicting interests Exposure of conflicts of interest; declare and manage conflicts of interest in policy development processes Inappropriate norms Engage opinion leaders; consensus processes; set professional standards Competing priorities More systematic and transparent priority-setting processes; provide evidence of the size of the problem and relevant comparisons Implementation considerations Questions or comments? Implementation considerations Key points • Don’t just drop an option if you find a barrier. Look at strategies. • Include “implementers” in the discussion (people who will be implementing on the ground). • Someone who knows implementation literature could be a key resource. • Use graphic tools to show the links between barriers and potential strategies to address them. • Strategies to facilitate change will be more successful if they minimize barriers and maximize enablers. 62 FACILITATOR'S GUIDE ACTIVITY TIME Use this time slot to lead an activity that will encourage participants to apply what they have learned in an interactive exercise. There are three options to consider below. IDEAS FOR ACTIVITIES Small group work Time for participants to work on a real project within their local context. Group brainstorming exercise Ask a participant to describe a policy option currently being considered in their work, along with a short list of barriers (or use examples emerging from previous exercises). Then divide participants into three teams, each with a facilitator, to brainstorm strategies to address these barriers and possible sources of evidence to inform these strategies. Emphasize that the strategies need to be informed by evidence. In plenary, compare and contrast the team results. Brief presentation If you are integrating the evidence-informed policy-making sessions into other topic areas, use this time to make another brief presentation, invite a guest speaker, allow a participant to make a special presentation, etc. 90 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 63 WRAP UP This session summarizes the key points from the day and encourages participants to reflect on their learning and give input on the Workshop delivery. Objectives By the end of this session, participants will be able to: • summarize the day’s topics and achievements; and • provide feedback about the Workshop so far. Materials • Handout: Daily feedback. FACILITATION STEPS 1. THANK participants for their enthusiasm and hard work. 2. REVIEW the main topics from the day quickly. 3. ASK participants for any final questions or comments about the day. 4. DISTRIBUTE the Daily feedback handout. Invite participants to write what they enjoyed the most, where they learned the most, where they still feel confused and what should be changed tomorrow. 5. ATTEND to any remaining logistics and CLOSE the day. Note: The afternoon session tomorrow involves a policy dialogue demonstration in which several participants will read roles from a script. Consider asking for those volunteers now so they can get familiar with the script before the session. AFTER THE DAY • REVIEW participants' feedback • PREPARE a SUMMARY for the following morning. • Review today's progress and tomorrow's agenda. Adjust the content and timing as needed based on participant feedback. • Remove flip charts that are no longer needed and prepare materials for the next day. • Record any impressions and/or suggestions for future Workshops on the Facilitator feedback form. 15 minutes 64 FACILITATOR'S GUIDE DAY 3 The final day of the Workshop spotlights two methods commonly used to communicate about policy evidence. The first session (Preparing evidence briefs for policy) introduces the evidence brief for policy, a tool to summarize evidence for a particular policy issue. After the presentation, participants practise developing outlines for evidence briefs. The next session (Organizing policy dialogues) describes a useful method to bring stakeholders together to discuss that evidence: the policy dialogue. In the afternoon, participants have the opportunity to experience a brief simulation of a policy dialogue (Policy dialogue demonstration). The facilitation team can assume the roles in the demonstration or you can invite participants to assume those roles. Be sure to assign those roles before breaking for lunch and that all volunteers have a copy of the demonstration script (provided in Section IV). The Workshop then closes as participants reflect on their learning and evaluate the Workshop. Table 4 has a sample agenda for Day 3. 3 Table 4. Sample agenda for Day 3 SAMPLE AGENDA ESTIMATED TIME MATERIALS Outlook on Day 3 15 minutes PowerPoint slides Laptop LCD projector Attendance sheet Flip charts Coloured markers Participant folders Copies of handouts: • Policy dialogue demonstration • Evidence brief on sugar- sweetened beverages • Final evaluation form Preparing evidence briefs for policy 90 minutes Break Activity: preparing evidence briefs 60 minutes Organizing policy dialogues 90 minutes Lunch Policy dialogue demonstration 60 minutes Break Closing session 45 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 65 OUTLOOK ON DAY 3 As you review the key concepts from yesterday and the three phases of the policy process, emphasize that the tools discussed today can be used to help to put evidence into practical use throughout the policy process. FACILITATION STEPS 1. WELCOME everyone back to the Workshop. 2. SUMMARIZE briefly the participant feedback from yesterday and describe any adjustments planned for today. 3. ASK participants to name the main accomplishments or outcomes from yesterday. Responses may include: • more skilled at finding evidence • better understanding of systematic reviews and how to evaluate them • know more ways to develop and select implementation strategies. 4. REVIEW the main topics and activities planned for the day. Explain that we will move into the third phase of the policy process, which includes special focus on evidence briefs for policy and policy dialogues. 5. ASK for questions. 6. LEAD a brief learning assessment exercise to review key concepts from yesterday. For ideas, see Quick exercises to assess participant learning (Section IV). 15 minutes 66 FACILITATOR'S GUIDE PREPARING EVIDENCE BRIEFS FOR POLICY This session brings previous concepts together into a document called an evidence brief for policy. After the presentation, you will guide participants through a real-life evidence brief. You can use the example provided or one of your own. Objectives By the end of this session, participants will be able to: • define an evidence brief for policy and describe its key components; and • review the main steps to prepare an evidence brief. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers • Handout: Evidence brief on sugar-sweetened beverages. FACILITATION STEPS 1. EXPLAIN the objectives for this session (5 minutes). 2. PRESENT the slide set: Preparing evidence briefs for policy (45 minutes). 90 minutes Slide 1 This session brings everything from our previous sessions together as we write an evidence brief for policy (EBP). Slide 2 æ REVIEW the objectives. æ ASK how many participants have seen or used a policy brief (by show of hands). Slide 3 æ ASK a volunteer to read the definition. An evidence brief for policy brings together global research evidence (from systematic reviews) and local evidence to inform deliberations about health policies. It is used to inform policy-makers and serves as a foundation for future policy dialogues. The intentional emphasis on evidence is what distinguishes these briefs from a more generic policy brief. EVIPNet Europe Preparing Evidence Briefs for Policy Based on: • SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) • SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) • EVIPNet (Evidence-Informed Policy Network), World Health Organization. Partially based on: • a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Preparing Evidence Briefs for Policy Objectives of the session • Define an evidence brief for policy and describe its key components. • Review the main steps to prepare an evidence brief. Preparing Evidence Briefs for Policy What is an evidence brief for policy? • “…research syntheses in a user-friendly format, offering evidence-informed policy options. The EBP is to convince the target audience of the urgency of the current problem and the need to adopt the preferred alternatives or strategies of intervention.” World Health Organization (2017). Evidence briefs for policy [website]. Geneva: World Health Organization (http://www.who.int/evidence/resources/policy_briefs/en/). SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 67 Slide 4 While policy briefs are often not based on systematic reviews and the method is not transparent, an evidence brief for policy brings together global research evidence (usually from systematic reviews) and local evidence to inform deliberations about health policies. It is used to inform policy-makers and may serve as a foundation for future policy dialogues. Slide 5 An EBP first summarizes the best available evidence to clarify the problem. Then it describes the key options to address the problem. Finally, it considers potential enablers and barriers to implementing the options and strategies to address them. Evidence is referenced in each of these components. Slide 6 An EBP should contain a summary of key messages plus an executive summary that describes the problem, policy options and strategies for implementation. The key messages and executive summary may be the only pages your policy-makers read. The 1:3:25 format means 1 page of key messages, 3 pages of executive summary and 25 pages for the full report. See SUPPORT collaboration resources for evidence briefs for a full description of this format and the reasons for it. æ ASK: • Why do you think this is the rule? • Why is it tiered this way? ENCOURAGE ideas. Slide 7 This slide is animated We've already emphasized this first step in other sessions. CLICK to display the next bullet. The next step is to decide on a timeline and a work plan. Typically, an EBP is prepared in weeks or months. This is different to a rapid response brief, which may need to be prepared in a few days. This will also depend on: • size of your team; • competing job responsibilities among the team; and • level of internal expertise or need to outsource key tasks. æ FACILITATE a brief discussion with these prompts: • How would these steps look in your own setting? • Do you have a team already designated for preparing evidence briefs for policy? What expertise is there? Preparing Evidence Briefs for Policy Policy brief vs. evidence brief for policy • Policy briefs are a wide range of documents that summarize the background of policy issues, analyse options or make recommendations. • Evidence briefs are a new approach to packaging research evidence for policy-makers and stakeholders. • Evidence briefs compile global research evidence (from systematic reviews) and local evidence to inform deliberations about health policies. Preparing Evidence Briefs for Policy • Summarize best available evidence to clarify size and nature of policy problem. • Assess likely impacts of key options to address the problem. • Identify potential barriers to implement the options and strategies to address those barriers. Key components Preparing Evidence Briefs for Policy Primary content of an EBP • Key messages, 1 page: – bullet point summary of key messages. • Executive summary, 3 pages: – synopsis of the problem, policy options and implementation considerations. • Full report, 25 pages: – the problem – policy options – implementation considerations. Remember the 1:3:25 rule for content Preparing Evidence Briefs for Policy Steps to prepare an EBP • First step: agree on the problem: – use explicit criteria and systematic processes to decide which issues to prioritize. • Next step: decide on a timeline and a work plan: – often determined by external factors – EBPs usually prepared in weeks or months. 68 FACILITATOR'S GUIDE Slide 8 æ REVIEW the key tasks of the work plan. Include the timeline and the people responsible for different tasks. Slide 9 æ REVIEW the tips. Teams will vary for different briefs. The expert team for the core process is usually constant, but content experts will vary based on the particular policy topic. Define what process experts are and what content experts are. æ INVITE any participants who have experience with evidence briefs for policy to offer their own tips. Slide 10 Health Research Policy and Systems publishes a series of SUPPORT tools for evidence-informed health policy-making. This tool describes how to prepare and use evidence briefs for policy. Slide 11 æ ASK: Who has actual experience developing a policy brief? æ FACILITATE a brief discussion with questions, such as: • What was your process? • What was the uptake? • How can we improve the effectiveness of these policy briefs? æ ASK for any comments or questions. Slide 12 The SURE guides build upon the SUPPORT tools and offer more guidance on writing evidence briefs. Slide 13 Here are some topics on the SURE website. æ REVIEW the sample topics. Work plan for a evidence briefs for policy Source: SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3 (http://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html). Preparing Evidence Briefs for Policy • Set a team with “real time” to write the brief. • Have a mix of junior and senior people. • Rely on partnerships to search for and appraise evidence. • Rely on your regional network to review the final brief (merit review). Tips for writing an evidence brief Preparing Evidence Briefs for Policy SUPPORT tools www.health-policy- systems.com/content/7/ S1/S13 Preparing Evidence Briefs for Policy Questions or comments? Preparing Evidence Briefs for Policy SURE guides http://www.who.int/evidence/sure/guides/en/ Preparing Evidence Briefs for Policy SURE guides for preparing and using policy briefs Background • Getting started • Prioritizing topics for policy briefs Preparing policy briefs • Clarifying the problem • Deciding on and describing policy options to address the problem • Identifying and addressing barriers to implementing policy options • Clarifying uncertainties and needs for monitoring and evaluation Using policy briefs • Organizing and running policy dialogues • Informing and engaging stakeholders SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 69 Slide 14 æ REVIEW the key points (or ask volunteers to read them). Facilitators could emphasize that the starting point for an EBP is the policy issue and not the research evidence or research questions. The research evidence is collected focusing on the problem and the options being considered in the evidence brief. Slide 15 Resources (see slide deck) FACILITATION STEPS (contd) 3. DIVIDE participants into three small groups and DISTRIBUTE copies of the handout: Evidence brief on sugar-sweetened beverages and negative health impact in Estonia. 4. WRITE these components of an EBP on a flip chart: • policy problem • policy options • implementation considerations. 5. ASSIGN one component to each group. 6. EXPLAIN that this is the Executive summary of an evidence brief for policy developed by Estonia. It asserts that the consumption of sugar-sweetened beverages is more associated with increased energy intake, weight gain, overweight and obesity than any other food or beverage, as well as with the development of several noncommunicable diseases and poor oral health. 7. INSTRUCT each group to locate its component in the executive summary and note the key points made in that section. 8. RECONVENE the groups after 20 minutes and INVITE each group to report on its findings (in this order: problem, options and then implementation). 9. SUMMARIZE with these following key points: • this is only the Executive summary and the complete evidence brief would also describe sources of evidence that inform the policy brief; and • a policy-maker may have time to read only an executive summary and so it is essential that it provides a concise yet complete summary of the problem, policy options and implementation issues. 10. ASK for final questions. 11. EXPLAIN that in the next presentation we will see how evidence briefs are used to engage stakeholders in policy discussions. Preparing Evidence Briefs for Policy Key points • An evidence brief for policy synthesizes the evidence about a problem, options and implementation strategies for policy-makers. • Make writing an evidence brief as collaborative as possible. • A work plan and timeline will help you to organize tasks and complete the brief on time. 70 FACILITATOR'S GUIDE ACTIVITY: PREPARING EVIDENCE BRIEFS Use this time slot to lead an activity that will encourage participants to apply what they have learned in an interactive exercise. There are two options to consider below. IDEAS FOR ACTIVITIES Small group work Time for participants to outline an evidence brief for policy for a real project within their local context. Case study 1. Select an evidence brief for policy (examples can be found at http://www.who.int/ evidence/resources/policy_briefs/en/index3.html). 2. Divide participants into four teams. 3. Write the following questions on a flip chart. • How does the brief describe the nature and scope of the problem? • How does the brief describe various policy options? • What are key implementation considerations? • What methods were used to identify, select and assess research evidence? 4. Assign a question to each team and give them 30 minutes to review the brief and discuss their responses. 5. In plenary, ask each team to share its responses. 6. Debrief with discussion questions such as: • If you were a policy-maker, how likely would this evidence brief influence your opinions? • How would you improve this evidence brief? • What are the most important qualities you think an evidence brief should have? 60 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 71 ORGANIZING POLICY DIALOGUES This session shows how evidence briefs are used to engage stakeholders in a process called a policy dialogue (PD). The session starts with a short video. Internet access is needed to view the video. Objectives By the end of this session, participants will be able to: • define a policy dialogue; • review six key questions to ask when organizing policy dialogues; and • describe how to organize and run policy dialogues. Materials • PowerPoint slides • Laptop and projector • Flip chart or dry erase board • Markers. FACILITATION STEPS 1. EXPLAIN the objectives for this session (5 minutes). 2. SHOW the video: Reflections on organizing and hosting policy dialogues (10 minutes). www.youtube.com/watch?v=tj_PdGYSDOA# 3. DEBRIEF the video by asking the following questions: • How would you define a policy dialogue in your own words? • What is the purpose of a policy dialogue? What do we want to achieve with a policy dialogue? • What are some of the benefits of a policy dialogue? Some of the challenges? 4. PRESENT the slide set: Organizing policy dialogues (55 minutes). 90 minutes Slide 1 Evidence-informed policy dialogues are becoming more common as an element of a broader knowledge transfer strategy. This session will describe a standardized yet flexible framework for organizing policy dialogues. Slide 2 æ REVIEW the objectives. EVIPNet Europe Organizing Policy Dialogues Based on: •SUPPORT Collaboration (Supporting Policy relevant Reviews and Trials) •SURE Collaboration (Supporting Use of Research Evidence for Policy in African Health Systems) •EVIPNet (Evidence-Informed Policy Network), World Health Organization. Partially based on: •a course designed by Dr John Lavis, McMaster Health Forum, WHO Collaborating Centre for Evidence Informed Policymaking Organising Policy Dialogues Objectives of the session • Define a policy dialogue. • Review six key questions to ask when organizing policy dialogues. • Describe how to organize and run policy dialogues. 72 FACILITATOR'S GUIDE Slide 3 This slide is animated æ READ the definition (NB this is the WHO definition). æ ASK participants to identify keywords or ideas in the definition. CLICK to display the remaining bullets. A policy dialogue begins by presenting the research evidence in an evidence brief for policy. It complements the explicit knowledge of the evidence brief with tacit knowledge (the views and knowledge of stakeholders) during a deliberative, highly interactive process. A policy dialogue is not just a "consultation" where stakeholders are asked what they think as a gesture of inclusion. The essential feature of a dialogue is that stakeholder input is considered an invaluable component of the policy solution, side by side with evidence. You must bring in evidence! Policy dialogues are complementary to the policy brief and the ideas collected from the dialogue are usually incorporated into the final brief. Remember the dialogue is not trying to reach a consensus but to elicit as much information as possible from the expertise present. Slide 4 Many times, we engage in debate and not in a dialogue. æ REVIEW some of the differences between dialogue and debate. æ ASK: • Who has experienced a policy dialogue and what was it like? • Was it more like a dialogue or a debate? ENCOURAGE participants to share their experiences. Slide 5 Policy-makers need more than just data to make decisions. Policy dialogues offer a structured way to bring in critical perspectives and ideas of stakeholders who can provide "real-life" context and understanding of that evidence. It is critical to understand that the scope of a policy dialogue is not necessarily to end up with consensus on a policy decision. It is to inform the decision-makers, while at the same time collecting their and other stakeholders' tacit knowledge, complementing the published and grey literature found and included in the evidence brief for policy. Manage this expectation or people may feel disappointed, frustrated or turned off from the process. This could hamper your efforts. Know how policy is made and how a dialogue supplements and can blend into this process at different phases. You can use policy dialogues early if you need help scoping the problem. But they can also come later as you work out the options for policy or how to implement them. Organising Policy Dialogues Policy dialogue: definition • Structured discussion carefully designed to present research evidence and exchange information between stakeholders to inform policy problems and solutions. • Deliberative process based on a pre-circulated evidence brief for policy. • Brings research evidence together with views, experiences and tacit knowledge of stakeholders. • Enables interaction between policy-makers, researchers and other stakeholders. Organising Policy Dialogues Dialogue vs debate Dialogue Debate Collaborative Oppositional Common ground Winning Enlarges perspectives Affirms perspectives Searches for agreement Searches for differences Causes introspection Causes critique Looks for strengths Looks for weaknesses Re-evaluates assumptions Defends assumptions Listening for meaning Listens for countering Remains open-ended Implies a conclusion So ur ce : A da pt ed fr om Jo ne s C M , M itt el m ar k M B (2 00 7) . T he IU HP E bl ue pr in t f or d ire ct a nd su st ai ne d di al og ue in p ar tn er sh ip in iti at iv es . I n: M cQ ue en D , Jo ne s C M , e di to rs . G lo ba l p er sp ec tiv es o n he al th pr om ot io n ef fe ct iv en es s. Ne w Yo rk : S pr in ge r:3 3– 40 . Organising Policy Dialogues • Research evidence is only one input into the policy- making process. Dialogues bring in critical opinions and perspectives. • Provides locally contextualized “decision support” for policy-makers and stakeholders. • Stakeholders can add significant value to this process and take action to address high-priority issues, alongside policy-makers. Rationale for policy dialogues SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 73 Slide 6 These six questions provide an easy-to-use framework to organize a policy dialogue. We will look at each question in detail. æ READ each question (or ask volunteers to read). Slide 7 æ READ each question (or ask volunteers to read). Slide 8 If the issue is not a priority for policy-makers or stakeholders, there will be little motivation to participate in a dialogue or much momentum behind action afterwards. Whatever motivated the evidence brief for policy should motivate the policy dialogue. Slide 9 Although most dialogues are called at the end to address each of the topics shown, dialogues may also be called to focus on a particular aspect of the framework or brief, such as defining the problem or identifying options. This may influence your choice of who to invite to ensure the most appropriate voices are at the table. Slide 10 Package the evidence into a policy brief and distribute it before the dialogue so all participants are prepared for the discussion. Everyone should accept the evidence brief for policy as final and accurate. The dialogue afterwards is directly linked to the policy brief. Start the dialogue with a summary of key points from the policy brief. Some participants may prefer PowerPoint slides, others a printed page of bullet points and still others an executive summary. Use different mechanisms to deliver the information. Tailor your knowledge transfer to your audience. Be sure the discussions cover important contextual factors that can impact a policy option, such as: • What do interest groups prefer and how much power do they have? • Maybe the policy option is good but the institution to implement it has another policy against it. • Perhaps there is strong evidence for the option but the belief systems of the community will not support it. • How interested are donors in this particular topic? Organising Policy Dialogues Organizing policy dialogues: key questions 1. Does the dialogue address a high-priority issue? 2. Are there ample opportunities to discuss the problem, options to address the problem and key implementation issues? 3. Is the dialogue informed by a precirculated policy brief and discussion on all factors that can influence the policy-making process? Organising Policy Dialogues 4. Is there fair representation of those who will be involved in, or affected by, decisions on this issue? 5. Does the dialogue engage a facilitator, follow rules about attributing comments and clearly state that its goal is not consensus? 6. Will outputs be produced and follow-up steps taken to support action? Organizing policy dialogues: key questions Organising Policy Dialogues Q1. Priority issue Does the dialogue address a high-priority issue? • Issue must be on the governmental agenda and widely perceived by many, if not all, stakeholders as a priority. • Should be the same reasons why the evidence brief for policy was prepared to inform the policy dialogue. Organising Policy Dialogues Q2. Agenda Are there ample opportunities to discuss the problem, options to address the problem and key implementation issues? •Agenda should include separate deliberations about: ̶ problem ̶ each option, with pros and cons ̶ implementation considerations ̶ possible next steps for different constituencies •Aim to develop shared understanding. Organising Policy Dialogues Q3. Evidence and other considerations Is the dialogue informed by a precirculated policy brief and a discussion on all factors that can influence the policy-making process? •Circulate the evidence brief for policy at least 10 days before the policy dialogue. •The evidence brief should be assumed to be final and correct. •Discussions must address: ̶ institutional constraints ̶ interest group pressure ̶ values and beliefs ̶ external factors (e.g. economy, donors). 74 FACILITATOR'S GUIDE Slide 11 Make an effort to map the stakeholders. This is key to getting all relevant views and opinions to the table. We do not always have choice or influence on the invitation list, but try to ensure that each invitee will be willing to integrate, share in the process and take the time to think critically. People who are stuck in a firm position will not be helpful. Remember that it is not only high-power people who can champion changes. People on the ground can be powerful catalysts. Do not underestimate them. Slide 12 The facilitator is a key player. Consider several facilitators and prepare them well on the topic, participants and objectives of the dialogue in addition to any relevant background or history. Create a pool of people upon whom you can rely for facilitation help. The role of the facilitator is to: • clarify the goal, objectives and expectations of a dialogue; • draw out different opinions, values and beliefs while maximizing the participants' contributions to the group in a fair, inclusive and respectful manner; and • intervene and steward as necessary in order to keep the participants to the agenda and objectives. The facilitator should be credible (i.e. skilled and knowledgeable) as well as perceived as neutral. He/she establishes house rules that will guide the discussion; for example, a rule that notes and reports will not reveal who made particular comments. This is the Chatham House rule commonly used for debates and discussions, particularly on controversial issues. This encourages open, honest discussion. Do we have to achieve consensus or not? In general, consensus is not a goal of the dialogue because we want to hear as many diverse ideas and opinions as possible. Slide 13 The dialogue itself is not the end of the process. Always produce a report and distribute widely. Consider other follow-up activities to keep up momentum. æ READ the examples. There are some additional examples on the McMaster website. Will the team for the evidence brief and dialogue be available for the next year to provide additional evidence if needed? Sometimes there may be decisions that you did not foresee and policy-makers will need support. Organising Policy Dialogues Q4. Fair representation Is there fair representation of those who will be involved in, or affected by, decisions on this issue? •Stakeholder mapping to list policy-makers, local government, managers, professional leaders, civil society leaders, researchers. •Choose dialogue participants based on their ability to: ̶ articulate views and experiences of a particular constituency ̶ constructively engage with and learn from other participants ̶ champion actions to address the issue within their constituencies. Organising Policy Dialogues Q5. Facilitator, attributions, goal Does the dialogue engage a facilitator, follow a rule on attributing comments and clearly state that its goal is not consensus? •Facilitator must be seen as skilled, knowledgeable and neutral. •Set a rule on attributing comments (e.g. Chatham House rule). •Clarify that consensus is not the main goal for the dialogue. But embrace consensus if it emerges naturally. Organising Policy Dialogues Q6. Next steps Will outputs be produced and follow-up steps taken to support action? Dialogue summary (without attributing comments) should be produced and disseminated widely. Optional: ̶ personalized briefings to key policy-makers? ̶ media interviews with dialogue participants? ̶ video interviews with dialogue participants (for posting on YouTube)? ̶ year-long evidence service? SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 75 Slide 14 Here are some additional points to remember. • Choose a title for the dialogue that will grab attention and stir curiosity. • Be sure your invitation letter names the organizers, members of the steering committee and funders. • Think about evaluation. æ ASK participants with experience in policy dialogues to share any additional lessons learned. Slide 15 This slide is animated There are a number of tools to help you to organize a policy dialogue. CLICK This is an article on SUPPORT tools for policy dialogues. CLICK The SURE guides also have some information. CLICK This section is on organizing and running policy dialogues. Slide 16 æ ASK for any questions or comments. Slide 17 æ REVIEW the key points (or ask volunteers to read them). Slide 18 Resources (see slide deck) FACILITATION STEPS (contd) 5. ASK for any final questions. 6. EXPLAIN that after lunch we will do a brief simulation of a policy dialogue. Organising Policy Dialogues Other considerations • Chose an engaging title. • Send invitation letter: ̶ organizers and their affiliations ̶ steering committee members and their affiliations (if applicable) ̶ Funders. • Formative evaluation ̶ adapt survey questions to match the design features of the dialogue ̶ core team can assist with this. Organising Policy Dialogues Tools Source: SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3. Clarifying the problem, version 2.1. Organising Policy Dialogues Questions or comments? Organising Policy Dialogues • Define in advance and very clearly the objective of the dialogue. Be clear in what you want to achieve. • Use the six questions to guide your organizing efforts. • Spend time mapping the stakeholders. Having the right people at the table is critical. Key points 76 FACILITATOR'S GUIDE POLICY DIALOGUE DEMONSTRATION The demonstration in this session brings the concepts just discussed about policy dialogues to life as participants observe an abbreviated policy dialogue simulation. It also adds energy (and even some fun) to the afternoon. Objectives By the end of this session, participants will be able to: • apply theoretical concepts in a more realistic policy dialogue experience; and • describe a typical agenda and flow of a policy dialogue. Materials • Flip chart • Markers • Script for policy dialogue demonstration: patient safety. PREPARATION 1. Arrange five chairs in a semi-circle facing the audience. The chair for the facilitator should be set slightly apart yet facing the four dialogue participants. 2. Write the following on a flip chart: • high priority issue • agenda (problem, options, implementation) • evidence brief • fair representation of stakeholders • facilitator, ground rules, goal • next steps. FACILITATION STEPS 1. INVITE the volunteers to take their seats at the front of the room. 2. EXPLAIN that this demonstration is a very abbreviated version of a policy dialogue. Real-life dialogues usually take at least one day to complete. 3. REFER to the flip chart and INSTRUCT participants to watch for these key questions just discussed in the previous session. (You may want to review these key points briefly.) 4. CONDUCT the demonstration (30 minutes). 5. FACILITATE a brief discussion on each of the key questions from the flip chart. Use the following prompts. • Did you see this element included in the dialogue and how? • What did the facilitator do to include or emphasize this element? Do you remember any participant comments or interactions related to this element? • What were your impressions overall of this dialogue? • How would this same dialogue look in your own work/community setting? What might be different? 6. ASK participants for any final questions or comments about policy dialogues. 60 minutes SECTION III: FACILITATION INSTRUCTIONS FOR THE WORKSHOP SESSIONS 77 CLOSING SESSION This session is used to help participants to plan how they will carry their learning forward into their own work. Objectives By the end of this session, participants will be able to: • reflect on their accomplishments; and • evaluate the Workshop. Materials • Flip chart or dry erase board • Markers • Handout: Final evaluation. FACILITATION STEPS 1. INVITE everyone to reflect on the work completed and what they have learned during the Workshop. 2. ASK each person to share something about their reflection. (You may wish to record responses on a flip chart.) Options include: • one word to describe your future role in evidence-informed policy-making • one thing you learned this week that you will definitely apply in your work • one specific action you will take to integrate evidence more into policy-making. 3. INVITE participants to share any thoughts or comments. 4. DISTRIBUTE copies of the Final evaluation sheet. Allow time for participants to complete. 5. INVITE closing remarks from participants, facilitators or others and conclude the Workshop. 6. THANK the participants for their attendance and active participation, and wish them well in their efforts to incorporate evidence into policy-making. END OF WORKSHOP 30 minutes

SECTION IV. TOOLS FOR FACILITATORS This section contains useful documentation and handouts for a Workshop. PREPARATORY MATERIAL Sample agenda, Materials checklist and Pre-Workshop survey SESSION HANDOUTS Clarifying a problem: case studies, Finding and using research evidence, AMSTAR 2 checklist, SURE checklist, Evidence brief on sugar-sweetened beverages, Policy dialogue demonstration: patient safety EVALUATION TOOLS Daily feedback form, Final evaluation form, Facilitator feedback form FACILITATION TOOLS Energizers, Quick exercises to assess participant learning, Dealing with disrupters, Instructional methods 80 FACILITATOR'S GUIDE Sample agenda Below is a sample agenda for presenting the sessions as a three-day Workshop. DAY 1 08:30–09:00 Registration 09:00–09:45 Opening session 09:45–11:00 What is evidence? 11:00–11:15 Break 11:15–12:15 Clarifying a problem 12:15–13:00 Activity time 13:00–14:00 Lunch 14:00–15:00 Identifying options 15:00–15:15 Break 15:15–16:45 Activity time 16:45–17:00 Wrap up of Day 1 DAY 2 08:30–08:45 Outlook on Day 2 08:45–10:15 Finding evidence: systematic reviews 10:15–10:30 Break 10:30–12:00 Assessing a systematic review 12:00—13:00 Activity time 13:00–14:00 Lunch 14:00–15:00 Implementing policy options 15:00–15:15 Break 15:15–16:45 Activity 16:45–17:00 Wrap up of Day 2 DAY 3 08:30–08:45 Outlook on Day 3 08:45–10:15 Preparing evidence briefs for policy 10:15–10:30 Break 10:30–11:30 Group exercise on preparing an annotated outline of an evidence brief for policy 11:30–13:00 Organizing policy dialogues 13:00–14:00 Lunch 14:00–15:00 Policy dialogue demonstration 15:00–15:15 Break 15:15–16:00 Closing session SECTION IV: TOOLS FOR FACILITATORS 81 Materials checklist These are the materials required to facilitate the Workshop. • Attendance sheets • name tags • participant folders • PowerPoint slides • copies of PowerPoint slides printed as handouts (with space for note taking) • laptop • LCD projector and screen • flip charts or dry erase board • markers, extra pens • copies of handouts: • Clarifying a problem: case studies • Finding and using research evidence • AMSTAR 2 checklist • SURE checklist • Evidence brief on sugar-sweetened beverages and their negative health impact in Estonia • Policy dialogue demonstration script • Daily feedback • Final evaluation 82 FACILITATOR'S GUIDE Pre-Workshop survey Evidence-informed Policy-making Date: Venue: We would like to know more about your knowledge and experience with evidence-informed policy-making (EIP) so we can better target the Workshop to your needs and expectations. The information you provide will only be shared among the facilitator and Workshop organizers. Please complete the survey and return it to [name of facilitator, email address] ................................................................................... ......................................................................................................................................................................................................................................... by [date]................................................................................... Name: Job title/position: What do you hope to gain by participating in this Workshop? What would you most like to learn during the Workshop? SECTION IV: TOOLS FOR FACILITATORS 83 Are you already familiar with: Definition of evidence and the different types of evidence No Yes How to use evidence in EIP No Yes How to search for evidence No Yes If yes, which databases have you used? How to appraise evidence No Yes If yes, which tools have you used? Ways to frame a health problem No Yes Ways to generate policy options to address a health problem and tools to evaluate and select policy options No Yes Systematic reviews No Yes If yes, what are the key advantages of systematic reviews compared to single studies? Preparing policy briefs No Yes If yes, give examples: Running policy dialogues No Yes If yes, give examples: Thank you for completing this questionnaire. 84 FACILITATOR'S GUIDE Clarifying a problem: case studies The following scenarios were borrowed from the SUPPORT tools (1). Scenario 1 You are a senior civil servant and have been asked to submit a briefing note to the Minister about a health system problem in which she has a personal interest, namely that many of her constituents and family members say that they can't find a primary health care physician. You are concerned about whether the current draft of the briefing note prepared by a junior policy analyst does justice to the problem. Scenario 2 You work in the Ministry of Health and are preparing a briefing note about a health system problem. You have been told that the problem is about many citizens not having access to primary health care providers and services. Scenario 3 You work in an independent unit that supports the Ministry of Health in its use of research evidence in policy-making and are preparing a policy brief for the Ministry of Health on barriers to accessing primary health care. You want guidance on how to clarify the problem in a systematic and comprehensive way. Discussion points 1. What is the problem? 2. How did the problem come to attention and has this process influenced the prospect of it being addressed? 3. What indicators can be used or collected to establish the magnitude of the problem and to measure progress in addressing it? 4. What comparisons can be made to establish the magnitude of the problem and to measure progress in addressing it? 5. How can a problem be framed (or described) in a way that will motivate different groups? Reference 1. Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 2009;7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). SECTION IV: TOOLS FOR FACILITATORS 85 Finding and using research evidence The Health Systems Learning tools from McMaster University (1) include a summary sheet for finding and using research evidence. About clinical and public health issues About health system issues CLARIFYING A PROBLEM 1. What is the problem (and its causes)? • A risk factor, disease or condition • A programme, service or drug being used • Current health system (governance, financial and delivery) arrangements within which programmes, services and drugs are provided • Current degree of implementation of an agreed course of action 2. How did the problem come to attention and has this process influenced the prospect of it being addressed? National health and health care utilization databases (e.g. for Canada www.cihi.ca) LexisNexis for media coverage of health issues in all countries 3. What indicators can be used, or collected, to establish the magnitude of the problem and to measure progress in addressing it? As for 2 4. What comparisons can be made to establish the magnitude of the problem and to measure progress in addressing it? PubMed HSR queries: • process assessment • outcomes assessment Health Systems Evidence for health system arrangements 5. How can a problem be framed (or described) in a way that will motivate different groups? PubMed HSR queries: • qualitative research Health Systems Evidence for health system arrangements FRAMING OPTIONS 1. Has an appropriate set of options been identified to address the problem (within one or more of the areas where problems were identified)? • Introducing, changing or discontinuing a programme, service or drug • Introducing, changing or discontinuing a health system arrangement that contributes to whether the right mix of programmes, services and drugs gets to those who need them • Implementing an agreed course of action 2. What benefits are important to those who will be affected and which benefits are likely to be achieved with each option? Cochrane Library for clinical programmes, services and drugs Health Systems Evidence for public health programmes and services Health Systems Evidence for health system arrangements 86 FACILITATOR'S GUIDE About clinical and public health issues About health system issues FRAMING OPTIONS (contd) 3. What harms are important to those who will be affected and which harms are likely to arise with each option? Cochrane Library for clinical programmes, services and drugs Health Systems Evidence for public health programmes and services As for 2 4. What are the local costs of each option and is there local evidence about their cost–effectiveness? Cochrane Library for economic evaluations of any option As for 2 5. What adaptations might be made to any given option and might they alter its benefits, harms and costs? PubMed HSR Queries for qualitative research As for 2 6. Which stakeholders’ views and experiences might influence the acceptability of an option and its benefits, harms and costs? PubMed HSR Queries for qualitative research As for 2 IDENTIFYING IMPLEMENTATION CONSIDERATIONS 1. What are the potential barriers to and facilitators of the successful implementation of the policy or programme (at each of the following levels)? • Patients/citizens (e.g. awareness of the availability of a free programme) • Health workers (e.g. adherence to guidelines) • Organizations (e.g. performance management) • Systems (e.g. enforcement of regulations) 2. What strategies should be considered in order to facilitate the necessary behavioural changes among patients/citizens? Health Systems Evidence for implementation strategies Rx for Change for descriptions of implementation strategies and summaries of their effectiveness Health Systems Evidence for implementation strategies Rx for Change for descriptions of implementation strategies and summaries of their effectiveness 3. What strategies should be considered in order to facilitate the necessary behavioural changes among health workers? As for 2 As for 2 4. What strategies should be considered in order to facilitate the necessary organizational changes? As for 2 As for 2 5. What strategies should be considered in order to facilitate the necessary system changes? As for 2 As for 2 SECTION IV: TOOLS FOR FACILITATORS 87 About clinical and public health issues About health system issues FOR SYSTEMATIC REVIEWS 1. What is the quality (AMSTAR) score? OR/NOT/AND • “user fees” ≠ user fees • (doctor AND nurse) OR pharmacist ≠ doctor AND (nurse OR pharmacist) nurs* = nurse OR nurses OR nursing 2. How locally applicable are the key messages? OR/NOT/AND • “user fees” ≠ user fees • (doctor AND nurse) OR pharmacist ≠ doctor AND (nurse OR pharmacist) nurs* = nurse OR nurses OR nursing Websites: Cochrane Library (www.cochranelibrary.com); Health Evidence (www.healthevidence.org); Health Systems Evidence (www.healthsystemsevidence.org); Lexis Nexis (www.lexisnexis.com/hottopics/lnacademic/); PubMed HSR queries (www.nlm.nih.gov/nichsr/hedges/search.html); Rx for Change (www.rxforchange.ca) Note: All reviews in Rx for Change are captured in Health Systems Evidence. Reference 1. McMaster Health Forum. Health systems learning. Hamilton (ON): McMaster University; 2013 (https://www.mcmasterforum.org/learn-how/ health-systems-learning, accessed 1 November 2017). 88 FACILITATOR'S GUIDE AMSTAR 2 checklist AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both (1) 1. Did the research questions and inclusion criteria for the review include the components of PICO? For Yes  Population  Intervention  Comparator group  Outcome Optional (recommended)  Timeframe for follow-up  Yes  No 2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? For Partial Yes: The authors state that they had a written protocol or guide that included ALL the following:  review question(s)  a search strategy  inclusion/exclusion criteria  a risk of bias assessment For Yes: As for partial yes, plus the protocol should be registered and should also have specified:  a meta-analysis/synthesis plan, if appropriate, and  a plan for investigating causes of heterogeneity  justification for any deviations from the protocol  Yes  Partial Yes  No 3. Did the review authors explain their selection of the study designs for inclusion in the review? For Yes, the review should satisfy ONE of the following:  Explanation for including only RCTs  OR Explanation for including only NRSI  OR Explanation for including both RCTs and NRSI  Yes  No 4. Did the review authors use a comprehensive literature search strategy? For Partial Yes (all the following):  searched at least 2 databases (relevant to research question)  provided key word and/or search strategy  justified publication restrictions (e.g. language) For Yes, should also have (all the following):  searched the reference lists / bibliographies of included studies  searched trial/study registries  included/consulted content experts in the field  where relevant, searched for grey literature  conducted search within 24 months of completion of the review  Yes  Partial Yes  No SECTION IV: TOOLS FOR FACILITATORS 89 5. Did the review authors perform study selection in duplicate? For Yes, either ONE of the following:  at least two reviewers independently agreed on selection of eligible studies and achieved consensus on which studies to include  OR two reviewers selected a sample of eligible studies and achieved good agreement (at least 80 percent), with the remainder selected by one reviewer.  Yes  No 6. Did the review authors perform data extraction in duplicate? For Yes, either ONE of the following:  at least two reviewers achieved consensus on which data to extract from included studies  OR two reviewers extracted data from a sample of eligible studies and achieved good agreement (at least 80 percent), with the remainder extracted by one reviewer.  Yes  No 7. Did the review authors provide a list of excluded studies and justify the exclusions? For Partial Yes:  provided a list of all potentially relevant studies that were read in full-text form but excluded from the review For Yes, must also have:  Justified the exclusion from the review of each potentially relevant study  Yes  Partial Yes  No 8. Did the review authors describe the included studies in adequate detail? For Partial Yes (ALL the following):  described populations  described interventions  described comparators  described outcomes  described research designs For Yes, should also have ALL the following:  described population in detail  described intervention in detail (including doses where relevant)  described comparator in detail (including doses where relevant)  described study’s setting  timeframe for follow-up  Yes  Partial Yes  No 9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? RCTs For Partial Yes, must have assessed RoB from:  unconcealed allocation, and  lack of blinding of patients and assessors when assessing outcomes (unnecessary for objective outcomes such as all-cause mortality) For Yes, must also have assessed RoB from:  allocation sequence that was not truly random, and  selection of the reported result from among multiple measurements or analyses of a specified outcome  Yes  Partial Yes  No  Includes only NRSI 90 FACILITATOR'S GUIDE 9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review? (contd) NRSI For Partial Yes, must have assessed RoB:  from confounding, and  from selection bias For Yes, must also have assessed RoB:  methods used to ascertain exposures and outcomes, and  selection of the reported result from among multiple measurements or analyses of a specified outcome  Yes  Partial Yes  No  Includes only RCTs 10. Did the review authors report on the sources of funding for the studies included in the review? For Yes:  Must have reported on the sources of funding for individual studies included in the review. Note: Reporting that the reviewers looked for this information but it was not reported by study authors also qualifies  Yes  No 11. If meta-analysis was performed did the review authors use appropriate methods for statistical combination of results? RCTs For Yes:  The authors justified combining the data in a meta-analysis  AND they used an appropriate weighted technique to combine study results and adjusted for heterogeneity if present.  AND investigated the causes of any heterogeneity For NRSI For Yes:  The authors justified combining the data in a meta-analysis  AND they used an appropriate weighted technique to combine study results, adjusting for heterogeneity if present  AND they statistically combined effect estimates from NRSI that were adjusted for confounding, rather than combining raw data, or justified combining raw data when adjusted effect estimates were not available  AND they reported separate summary estimates for RCTs and NRSI separately when both were included in the review  Yes  No  No meta- analysis conducted  Yes  No  No meta- analysis conducted 12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis? For Yes:  included only low risk of bias RCTs  OR, if the pooled estimate was based on RCTs and/or NRSI at variable RoB, the authors performed analyses to investigate possible impact of RoB on summary estimates of effect.  Yes  No  No meta- analysis conducted SECTION IV: TOOLS FOR FACILITATORS 91 13. Did the review authors account for RoB in individual studies when interpreting/discussing the results of the review? For Yes:  included only low risk of bias RCTs  OR, if RCTs with moderate or high RoB, or NRSI were included the review provided a discussion of the likely impact of RoB on the results  Yes  No 14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? For Yes:  There was no significant heterogeneity in the results  OR if heterogeneity was present the authors performed an investigation of sources of any heterogeneity in the results and discussed the impact of this on the results of the review  Yes  No 15. If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? For Yes:  performed graphical or statistical tests for publication bias and discussed the likelihood and magnitude of impact of publication bias  Yes  No  No meta- analysis conducted 16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? For Yes:  The authors reported no competing interests OR  The authors described their funding sources and how they managed potential conflicts of interest  Yes  No Reference 1. https://www.bmj.com/content/358/bmj.j4008.full Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017 Sep 21;358:j4008. 92 FACILITATOR'S GUIDE SURE checklist SURE has a checklist to judge how much confidence to place in a systematic review. The list here is adapted from Appendix 1. SUPPORT summaries checklist for making judgments about how much confidence to place in a systematic review (1) with revision to incorporate elements from the AMSTAR checklist (2). Review: Assessed by: Date: Section A. Methods used to identify, include and critically appraise studies A.1 Were the criteria used for deciding which studies to include in the review reported? Did the authors specify:  Types of study  Participants  Intervention(s)  Outcome(s) Coding guide – check the answers above: YES: All four should be yes  Yes  Can't tell / partially  No Comments (note important limitations or uncertainty) A.2 Was the search for evidence reasonably comprehensive? Were the following done:  Language bias avoided (no restriction of inclusion based on language)  No restriction of inclusion based on publication status  Relevant databases searched (MEDLINE + Cochrane Library)  Reference lists in included articles checked  Authors/experts contacted Coding guide – check the answers above: YES: All five should be yes PARTIALLY: relevant databases and reference lists are both ticked  Yes  Can't tell / partially  No Comments (note important limitations or uncertainty) SECTION IV: TOOLS FOR FACILITATORS 93 A.3 Is the review reasonably up to date? Were the searches done recently enough that more recent research is unlikely to be found or to change the results of the review?  Yes  Can't tell / not sure  No Comments (note important limitations or uncertainty) A.4 Was bias in the selection of articles avoided? Did the authors specify:  Explicit selection criteria  Independent screening of full text by at least two reviewers  List of included studies provided  List of excluded studies provided Coding guide – check the above YES: All four should be yes  Yes  Can't tell / partially  No Comments (note important limitations or uncertainty) A.5 Did the authors use appropriate criteria to assess the risk for bias in analysing the studies that are included?a  The criteria used for assessing the risk of bias were reported  A table or summary of the assessment of each included study for each criterion was reported  Sensible criteria were used that focus on the risk of bias (and not other qualities of the studies, such as precision or applicability)  List of excluded studies provided Coding guide – check the above YES: All four should be yes  Yes  Can't tell / partially  No Comments (note important limitations or uncertainty) 94 FACILITATOR'S GUIDE A.6 Overall: how would you rate the methods used to identify, include and critically appraise studies? Summary assessment score A relates to the five questions above If the option no or partial is used for any of the five questions above, the review is likely to have important limitations Examples of fatal flaws might include not reporting explicit selection criteria, not providing a list of included studies or not assessing the risk of bias in included studies  Fatal flaws (limitations that are important enough that the results of the review are not reliable and they should not be used in the policy brief)  Important limitations (limitations are important enough that it would be worthwhile to search for another systematic review and to interpret the results of this review cautiously, if a better review cannot be found)  Reliable (only minor limitations) Comments (note any fatal flaws or important limitations) Section B. Methods used to analyse the findings B.1 Were the characteristics and results of the included studies reliably reported, giving:  Independent data extraction by at least two reviewers?  A table or summary of the characteristics of the participants, interventions and outcomes for the included studies?  A table or summary of the results of the included studies? Coding guide – check the answers above: YES: All three should be yes  Yes  Partially  No  Not applicable (e.g. no included studies) Comments (note important limitations or uncertainty) B.2 Were the methods used by the review authors to analyse the findings of the included studies reported?  Yes  Partially  No  Not applicable (e.g. no studies or data) Comments (note important limitations or uncertainty) SECTION IV: TOOLS FOR FACILITATORS 95 B.3 Did the review describe the extent of heterogeneity?  Did the review ensure that included studies were similar enough that it made sense to combine them, sensibly divide the included studies into homogeneous groups, or sensibly conclude that it did not make sense to combine or group the included studies?  Did the review discuss the extent to which there were important differences in the results of the included studies?  If a meta-analysis was done, were tests for heterogeneity carried out or other appropriate statistic reported (I2, chi square test)?  Yes  Can't tell / Partially  No  Not applicable (e.g. no studies or data) Comments (note important limitations or uncertainty) B.4 Were the findings of the relevant studies combined (or not combined) appropriately relative to the primary question the review addresses and the available data? How was the data analysis done?  Descriptive only  Vote counting based on direction of effect  Vote counting based on statistical significance  Description of range of effect sizes  Meta-analysis  Meta-regression  Other: specify  Not applicable (e.g. no studies or no data) How were the studies weighted in the analysis?  Equal weights (this is what is done when vote counting is used)  By quality or study design (this is rarely done)  Inverse variance (this is what is typically done in a meta-analysis)  Number of participants  Other, specify  Not clear  Not applicable (e.g. no studies or no data) Did the review address unit of analysis errors?  Yes – took clustering into account in the analysis (e.g. used intra-cluster correlation coefficient)  No, but acknowledged problem of unit of analysis errors  No mention of issue  Not applicable – no clustered trials or studies included Coding guide – check the answers above If narrative OR vote counting (where quantitative analyses would have been possible) OR inappropriate table, graph or meta-analyses OR unit of analyses errors not addressed (and should have been) the answer is likely NO If appropriate table, graph or meta-analysis AND appropriate weights AND the extent of heterogeneity was taken into account, the answer is likely YES. If no studies/no data: NOT APPLICABLE If unsure: CAN'T TELL/PARTIALLY  Yes  Can't tell / Partially  No  Not applicable (e.g. no studies or data) 96 FACILITATOR'S GUIDE [B4 contd] Comments (note important limitations or uncertainty) B.5 Did the review examine the extent to which specific factors might explain differences in the results of the included studies?  Were factors that the review authors considered as likely explanatory factors clearly described?  Was a sensible method used to explore the extent to which key factors explained heterogeneity?  Descriptive/textual  Graphical  Meta-regression  Other  Yes  Can't tell / Partially  No  Not applicable (e.g. too few studies, no important differences in the results of the included studies, or the included studies were so dissimilar that it would not make sense to explore heterogeneity of the results) Comments (note important limitations or uncertainty) B.6 Overall, how would you rate the methods used to analyse the findings relative to the primary question addressed in the review? Summary assessment score B relates to the five questions in this section If the option no or partial is used for any of the five questions above, the review is likely to have important limitations Examples of fatal flaws might include not reporting critical characteristics of the included studies or not reporting the results of the included studies  Fatal flaws (limitations that are important enough that the results of the review are not reliable and they should not be used in the policy brief)  Important limitations (limitations that are important enough that it would be worthwhile to search for another systematic review and to interpret the results of this review cautiously if a better review cannot be found)  Reliable (only minor limitations) Use comments to specify if relevant, to flag uncertainty or need for discussion aRisk of bias is the extent to which bias may be responsible for the findings of a study. Bias is a systematic error or deviation from the truth in results or inferences (3). In studies of the effects of health care, the main types of bias arise from systematic differences in the groups that are compared (selection bias), the care that is provided or exposure to other factors apart from the intervention of interest (performance bias), withdrawals or exclusions of people entered into a study (attrition bias) or how outcomes are assessed (detection bias). Reviews of studies may also be particularly affected by reporting bias, where a biased subset of all the relevant data is available (3). Assessing the risk of bias is also referred to as assessing the validity or quality of a study. Validity is the extent to which a result (of a measurement or study) is likely to be true. Quality is a vague notion of the strength or validity of a study, often indicating the extent of control over bias (4). SECTION IV: TOOLS FOR FACILITATORS 97 References 1. Supporting the Use of Research Evidence (SURE) Collaboration. Appendix 1. SUPPORT summaries checklist for making judgments about how much confidence to place in a systematic review. Norwegian University of Science and Technology; 2017 (https://www.ntnu.edu/ documents/1269470669/1276564084/SR+checklist.pdf/9aab27bb-0b6e-42ec-872c-376f5f14f147, accessed 30 October 2017). 2. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. 3. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, version 5.1.0. London: The Cochrane Collaboration; 2011. 4. Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M et al. (2012). Assessing the risk of bias of individual studies in systematic reviews of health care interventions. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ methods for effective health care series; https://www.ncbi.nlm.nih.gov/books/NBK91433/, accessed 29 November 2017). 98 FACILITATOR'S GUIDE Evidence-informed Policy Network (EVIPNet) Europe Evidence brief for policy EVIPNet Europe Number 1 Reducing the consumption of sugar-sweetened beverages and their negative health impact in Estonia EVIPNet Europe. Evidence brief for policy 1: reducing the consumption of sugar-sweetened beverages and their negative health impact in Estonia. Copenhagen: WHO Regional Office for Europe; 2017 (http://www.euro.who.int/__data/assets/ pdf_file/0016/324205/EBP-1-Web.pdf?ua=1, accessed 1 November 2017). SECTION IV: TOOLS FOR FACILITATORS 99 Policy dialogue demonstration: patient safety This demonstration requires five participants: one facilitator and four participants acting as dialogue stakeholders. Opening FACILITATOR: Thank you for attending this policy dialogue to examine the problem of patient safety in our district hospitals. Last year the Ministry of Health initiated efforts to strengthen patient safety using research to inform its policy revision. You as key stakeholders have been invited to provide your unique experiences and perspectives on this problem and to consider potential policy solutions. Thank you to: • Dr Mona, researcher from National University • Mr Bena, Quality Assurance Manager with the Ministry of Health • Ms Francis, Nurse Manager at District X hospital • Ms King, consumer advocate from the Patient Rights Consortium. Excellent policies lying on shelves will never get a chance to benefit people. That is why each contribution is precious to help to clarify the problem of patient safety, to discuss options to address the problem and to find the best ways to make sure that the policy options are successfully implemented. Before we start, we need to agree on some basic guidelines for this dialogue. • As your facilitator, my role is to make sure all have an equitable time to speak and contribute. • We must establish a rule about whether or not comments can be attributed (e.g. the Chatham House Rule states that participants are free to use the information received during the meeting, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be revealed.) • We will not neccessarily aim for consensus but will welcome it if it emerges. • There will be a final report on the dialogue and it will be circulated to all participants. The problem FACILITATOR: We have just heard Dr Mona from National University present the evidence brief for policy on hospital patient safety, which you have received in advance. She described the national scope of the problem, primarily errors in medication prescribing and spread of hospital-acquired infections. I'd like to begin our dialogue by hearing your thoughts. What factors do you feel contribute to this problem? And as we talk, I'd like to remind you about the Ground Rules you generated earlier. Who would like to begin? NURSE: I believe the scope of the problem is much larger than Dr Mona described. There is not really a safety culture at hospitals. Health workers are often punished if they put pressure on managers about safety conditions or report any mistakes made by doctors. So I'm sure medical incidents are underreported. ADVOCATE: Usually frontline workers are blamed, but management decisions also impact how services are delivered by health workers. To what extent should we hold management accountable to ensure that health facilities function properly? Who is going to be penalized for not establishing this culture of safety – the nurse or a hospital manager who has failed to supervise? 100 FACILITATOR'S GUIDE NURSE: As we look at the safety of patients, we should not forget the safety of the health care providers. Today you are a health worker. Tomorrow you could be the patient. Lack of adequate personal protective equipment for health providers should also be noted in the policy brief. ADVOCATE: To me, the policy brief seemed to suggest that the problem of patient safety is a result of the faulty health system. Patient safety is only a subset of overall system quality: efficiency, reducing wait times, effectiveness of services, and patient- centeredness. All these impact quality of care. FACILITATOR: I think most of us would agree that these are all important issues and that they do have some relevance to patient safety and should be recorded. Patient- centeredness is definitively at the core of efforts to promote better patient safety. It has an impact on other considerations that we must examine today, such as the option of strengthening human resources that Dr Mona mentioned. However, given our limited time today, I suggest we keep our focus primarily on patient safety itself. Besides, I would like to leave some additional time for final comments and you can come back in a moment with your helpful contribution about patient-centred health care. ADVOCATE: But the Ministry of Health should more effectively promote cleanliness, quality and improving patient safety. There are also surgical errors and medical records errors. We should be broader than this to incorporate all the other aspects of errors. How do we make sure health care personnel are well trained? How do we handle issues of confidentiality regarding safety of patients? FACILITATOR: Thank you, Ms King. We appreciate your willingness to look at this issue in a broad context. Mr Bena, we have not heard as much from you. What thoughts would you like to add? QA MANAGER: People have the attitude that it is the government's job to look after their health. It is your personal responsibility to look after your health. Hospital costs are currently staggering. We have taken steps as a Ministry including changing to new devices, improving on health waste management and also restricting some practices that are harmful to the patients. I am sure this dialogue will further these improvements. The Ministry is committed to improving patient safety in its planning processes and implementation strategies. But patients must be educated about their role. ADVOCATE: But it is your responsibility to strengthen facility inspection and supervision. Does the Ministry of Health ensure that registered facilities have trained personnel? It is the responsibility of the Ministry of Health to provide oversight to health care, but I see that this is sorely lacking. QA MANAGER: But the problem should also focus on patients. The problem statement should spell out how the low level of awareness among patients about their rights is a factor that affects patient safety. Patients have unrealistic expectations of the health system. This needs to be addressed. It would be good to have sensitization at the community level. Posters need to be put around health facilities educating patients on how to protect themselves. ADVOCATE: I do agree that we need to create a demand for safety. Patients feel the providers are more knowledgeable than they are, so whatever health providers do is right. Patients do not know they can question the health care they receive. We need to transform this culture of paternalism in health care. Poor staffing levels hinder effective communication among health workers, leading to provider fatigue and creating an environment for medical incidents. FACILITATOR: Is it accurate to say that you both agree that patient empowerment is important and that the Ministry of Health has a role to play in patient education? Manager and Advocate agree. SECTION IV: TOOLS FOR FACILITATORS 101 FACILITATOR: Great. So far, we have the issues of inadequate staffing, facility oversight, lack of patient awareness of their safety rights. What other factors contribute to the problem of patient safety? NURSE: Many of the problems regarding nurses started when health training schools were transferred from the Ministry of Health to the Ministry of Education. This undid the ethical framework that was emphasized when medical schools were still under the Ministry of Health and produced nurses with inadequate skills. This policy dialogue should recommend that medical training institutions go back under the Ministry of Health. FACILITATOR: It's great that you are beginning to think about solutions, but let's stay focused on defining the problem a bit more. Please remember your idea when we talk about policy options. NURSE: Decentralizing health services has created more challenges. Some districts have hospitals that are dysfunctional while some new districts do not have hospitals. That is why my hospital receives many patients from other districts but we don't get money from other districts to serve these patients. With its limited budget, my hospital does not have the capacity to provide quality care to all these patients. FACILITATOR: And how would you say that impacts patient safety? NURSE: In our new decentralized system, health care is only motivated by profit. The more medicine prescribed the more a patient feels that he/she is dealing with a genuine doctor. Doctors today prescribe to a patient many types of medicine to treat the same disease. That raises questions on ethical behaviour of health workers. Patient demand is driven by what they get from radio, the Internet and television. So people visit health workers with the belief that they need particular medicines. ADVOCATE: Instead of using the term patient, I like to use the term people. People have lost confidence in the health system, so everybody has become his own doctor. They rely on people who have no medical training or they self-treat at the pharmacy before they seek care from the health system. This leads to complications and wastage of medicines. FACILITATOR: Thank you. To summarize the discussion so far, you have mentioned many health system challenges that impact patient safety such as low demand for safety among consumers; shortage of health workers, especially nurses; insufficient nurse training; poor financing; decentralization; inefficient safety assessment; and enforcement practices. Mr Bena, I'd like to give you equal chance to comment before we move to our next discussion. ADVOCATE interrupts: I just want to quickly mention… FACILITATOR: Please, Ms King, let us first give Mr Bena a chance to add his comment. Remember we will have time for you to add final comments at the wrap up. QA MANAGER: Provider behaviour, such as carelessly prescribing antibiotics, is also critical. Clinical misdiagnosis leads to mismanagement, with exposure to unnecessary procedures, drugs, while at the same time not dealing with the actual problems patients are suffering from. FACILITATOR: Thank you for this rich discussion. I appreciate your willingness to share opinions that don't always agree. That's what makes our analysis more complete and is in fact why we invited such a broad variety of stakeholders. I also appreciate your willingness to hear opposing points of view with respect. Do we feel ready to focus now on policy options? Everyone agrees. 102 FACILITATOR'S GUIDE Policy option: nurse staffing models FACILITATOR: Let's now move onto the first policy option that Dr Mona outlined, which is revising nurse staffing models. As we heard, nurse staffing model interventions include changes to nurse staffing levels (nurse per patient ratio) or in the nursing skill mix (proportion of different nursing grades and levels of qualification, expertise). It could also involve the educational preparation of nurses, work shift patterns, and the use of overtime and agency staff. Who would like to begin? Long pause of silence… FACILITATOR: Everyone was talking so briskly before and now there is silence. What is the reason for this silence? Do you feel confused about our process? Do you need more information? Do you need some time to think about the information that has been presented? More silence. Eventually the QA Manager begins. QA MANAGER: I myself just needed some time to think. I have many thoughts about this and didn't know where to start. FACILITATOR: Feel free to begin anywhere you wish. Your experience is of great value to this dialogue. QA MANAGER: OK, thank you. The issue of staffing is a very big problem and cuts across many professions. (NURSE starts and maintains a side conversation with ADVOCATE). Many doctors are task shifting their core roles to nurses so that the doctor is free to run his private clinic and the nurse also pushes tasks to the nursing aides. Effective supervision of the human resources is required at all the levels of the health care system. Task shifting to lower level health cadres might lead to errors because they are not trained to perform these new tasks. We need to come up with a realistic staffing model. FACILITATOR: Excuse me, Ms King and Ms Francis. Please remember our rule about side conversations. We are here to talk together, so please be willing to share your comments with the full group. Is there something either of you wish to contribute? ADVOCATE: I agree with Mr Bena. We need an appropriate mix of staff skills. Even if you have enough nurses, if the other clinicians are not there the nurses will end up doing their job and services will not improve. The health training institutions need to be aware of where there is greatest demand for health cadres and train accordingly to fill the human resource gaps. NURSE: Well, I disagree with the contribution that there is need to consider the entire health workforce instead of nurses. A nurse is involved when a patient is being worked on. And a nurse spends more time with the patient compared with other health professionals. We need to open up and recruit more nurses instead of having one nurse running around a hospital floor the whole day. QA MANAGER: The evidence presented today is from high-income countries and not specific to our context here. How does this apply to our context here? Evidence used in a policy brief like this should be local because evidence from other countries may not be applicable here. And what about the costs? FACILITATOR: Dr Mona, can you address this concern about data relevance? RESEARCHER: We do have national data on health staffing levels. A human resources audit showed unfilled vacancies at public sector health facilities at 39%, and 64% of nurses and midwives serve the central urban region, which includes only 27% of the population. Specialist nursing posts in public health, psychiatry and nutrition at the national hospitals record 17% vacancies, while at the 11 regional referral hospitals this comes to 24%. SECTION IV: TOOLS FOR FACILITATORS 103 NURSE: Nurses spend a lot of time with patients, more than other health cadres, and emphasizing the role of nurses is important here. Fewer nurses means that a nurse does not get to adequately discharge her duties and it is not possible for them to provide adequate supervision. RESEARCHER: I know of one high-quality systematic review assessing hospital nurse staffing models. The reviewers assert that some nurse staffing models probably reduce death in hospitalized patients, reduce length of stay in hospital but could slightly increase readmission rates. Since current evidence is limited, we would need more rigorous evaluation to determine an optimal staffing model. ADVOCATE: The problem is bigger than nurses. So if we are to go with the option targeting professionals, then it should be all inclusive. Managing a patient is teamwork. There is no one health cadre that is solely responsible for patient safety. We should do it as a team. This is what I have observed in my experience. So this policy option should target all the health professionals. Implementation issues FACILITATOR: I'm sure we could continue this discussion for much longer, but I want to keep us moving so we can address all of the topics on our agenda today. Whenever we discuss policy options we must also consider some of the factors that impede policy implementation so we can address these when policy is being made. QA MANAGER: We know anecdotally that inadequate staffing and retention are influenced by insufficient training capacity, low pay and unattractive living situations, particularly in rural districts. But scaling up of nurse staffing costs money. I would need to see some evidence to show that investment in these areas will actually result in improved staff retention and productivity. FACILITATOR: Dr Mona, what evidence do we have on this issue? RESEARCHER: A systematic review by Willis–Shattuck examined factors affecting retention of health workers in lower income settings. Motivational factors such as adequate financial incentives, career development, health worker recognition can improve morale significantly. Another systematic review by Penaloza and colleagues affirms that, in addition to financial rewards, other factors can also help such as improving hospital infrastructure and hospital management. NURSE: Another barrier is inadequate supply and distribution of specialist nurses. There was a policy shift introducing comprehensive nursing about 15 years ago. This two-and- a-half-year training produced nurses who cannot do everything as expected. So in this policy option, what are we saying? Are we going to go back to the original way of training nurses that produced specialized nurses like midwives or continuing with comprehensive nurses, which has become a problem? QA MANAGER: There are other things that need to be included to enable our health care delivery system to deliver better. For example, can we name and shame our bad health workers? We should name and even shame health facilities that do not stick to treatment guidelines or do unethical things. NURSE: That kind of approach will do nothing to motivate health workers to change or adopt new behaviours. We don't need punishment. We need support! Maybe things like continuing education workshops, educational outreach visits by a trained person to health workers in their own settings, audits and supportive feedback could help. QA MANAGER: I want to emphasize strengthening of quality assurance. Funds allocated for quality assurance are little. This country is big and so is this problem. So all of the actors in this issue must engage government to provide more resources for quality assurance. And patients must be informed about the various offices where they can 104 FACILITATOR'S GUIDE register their complaints so that these can be addressed. This will help empower patients as we discussed earlier. FACILITATOR: (to ADVOCATE) Ms King, you haven't spoken yet. Is there something you wish to add? ADVOCATE stays quiet, looks down. FACILITATOR allows silence. ADVOCATE: Truthfully, this is the stage in the discussion where I just feel frustrated and start to lose hope that anything will change. Everyone is quick to assign blame and solutions. But those solutions always seem so impossible to implement. This is not the first time we have talked about this issue of patient safety and nothing has changed so far. FACILITATOR: So you feel discouraged and sceptical that this policy dialogue will produce any tangible results. ADVOCATE: Yes. FACILITATOR: How many others feel like this to some degree? Everyone raises his hands. Next steps and closing FACILITATOR: It is very clear that patient safety is important to everyone here. Although this problem was initially analysed for us in the evidence brief, our interaction here has certainly deepened our understanding of this problem. And yet deeper understanding can also sometimes makes us feel solutions may be too difficult to actually achieve. This can be a common experience in policy dialogues. One way to help improve optimism and lay a foundation for success is to develop some very specific action steps. As our dialogue today comes to a close, how would you suggest we move forward? ADVOCATE: I myself would like to see a type of matrix of short-term, medium-term and long-term action steps, with responsibilities for action assigned to specific stakeholders and with a specific time line. QA MANAGER: Coordination with various actors is very important and I hope in these documents the various actors are identified and coordination mechanisms documented. FACILITATOR: What final comments would you like to add? ADVOCATE: As I mentioned before, I prefer to use the term people, instead of patient. When health care professionals do their jobs, they must put themselves in the place of a person arriving at the hospital. That is a people-centred, humane approach that can help to prevent many mistakes. FACILITATOR: We certainly must always remember that our efforts are directed at improving and protecting people's health. With these words we will close the dialogue and commit to circulate the report of the meeting to all of you. SECTION IV: TOOLS FOR FACILITATORS 105 Date: What did you enjoy most today? Where did you learn the most? Where do you still feel confused? What should we change tomorrow? Daily feedback form Date: What did you enjoy most today? Where did you learn the most? Where do you still feel confused? What should we change tomorrow? Date: What did you enjoy most today? Where did you learn the most? Where do you still feel confused? What should we change tomorrow? 106 FACILITATOR'S GUIDE Final evaluation form Workshop location: Date: Please give us feedback on your experience with this training. Your comments and suggestions will help us refine and improve the design, content and delivery of the Workshop. Strongly disagree Somewhat disagree Neutral Somewhat agree Strongly agree 1. The workshop aims and learning objectives were met 2. The sessions were relevant to my work and my organization 3. The information in the technical sessions was adequate 4. The sessions were delivered in an interesting or engaging manner 5. The length of the Workshop was appropriate 6. The facilitators were helpful and knowledgeable 7. There was enough time for the sessions 8. The handouts were helpful 9. The Workshop venue was convenient and comfortable 10. Travel arrangements and information before the Workshop were adequate 11. Administrative support during the Workshop (e.g. timekeeping, catering) was adequate SECTION IV: TOOLS FOR FACILITATORS 107 12. Name one task given before or during the Workshop that was particularly useful to you and why. 13. Name one activity during the Workshop that should be changed or removed and why. 14. Name any topics that you feel were missing and should be included in this Workshop. 15. How we can make this Workshop better? Thank you. Your feedback is highly appreciated. 108 FACILITATOR'S GUIDE Facilitator feedback Since the Workshop will be organized and run by different people in different contexts for different target groups, it would be useful to be able to exchange experiences, tips and comments among workshop organizers and facilitators. This feedback will be shared by the organizer/facilitator community so that we can learn from each other and make adjustments, if necessary. Send completed forms to [name, email address]. Workshop venue: Dates: Workshop organizers: Facilitator(s): General comments/recommendations 1. Did the participants have the necessary background and experience to understand the material and actively participate in the workshop? What would you do differently to select participants? 2. How did you adapt the curriculum and why? SECTION IV: TOOLS FOR FACILITATORS 109 3. Was the meeting room large enough and set up to maximize interaction? What would you recommend to future facilitators about the meeting space? 4. Was there sufficient time to cover the material, answer participant questions, respond to comments and facilitate discussions? If not, how would you adjust the agenda? 5. Was there sufficient time to run the separate activity sessions? If not, what would you suggest be done differently next time? 6. Were there activities you incorporated into the workshop that were not part of the guide? If so, please describe: 110 FACILITATOR'S GUIDE Feedback on the specific sessions DAY 1 Questions Comments Opening session What is evidence Did the activity on brainstorming work? Did the participants have any problems understanding the slides? Would you present the concepts differently in future? Clarifying a problem Did the participants have any problems understanding the slides? How would you present the concepts differently? Activity time What type of activity did you choose to do? What worked well? What could have worked better? Identifying options Did the participants have any problems understanding the slides? How would you present the concepts differently? Activity time What type of activity did you choose to do? What worked well? What could have worked better? Wrap up SECTION IV: TOOLS FOR FACILITATORS 111 DAY 2 Questions Comments Outlook on Day 2 Did participants clearly understand the key points from Day 1? Finding evidence: systematic reviews Did the participants have any problems understanding the slides? How would you present the concepts differently? Did the individual exploration of websites work well? Assessing a systematic review Did the participants have any problems understanding the slides? How would you present the concepts differently? Activity time What type of activity did you choose to do? What worked well? What could have worked better? Implementing policy options Did the participants have any problems understanding the slides? How would you present the concepts differently? Activity time What type of activity did you choose to do? What worked well? What could have worked better? Wrap up 112 FACILITATOR'S GUIDE DAY 3 Questions Comments Outlook on Day 3 Did participants clearly understand the key points from Day 2? Preparing evidence briefs for policy Did the participants have any problems understanding the slides? How would you present the concepts differently? Assessing a systematic review Did the participants have any problems understanding the slides? How would you present the concepts differently in future? Activity: Preparing evidence briefs What worked well? What could have worked better? Organizing policy dialogues Did the participants find the video useful? Did the participants have any problems understanding the slides? How would you present the concepts differently? Policy dialogue demonstration What worked well? What could have worked better? Closing session Additional comments or recommendations: Thank you for completing this feedback sheet. Feedback on the specific sessions (contd) SECTION IV: TOOLS FOR FACILITATORS 113 Energizers Different types of games can be used by facilitators for a variety of purposes, including helping people to get to know each other, increasing enthusiasm levels and encouraging team building (1). Tips for using energizers • Use energizers frequently during a workshop or meeting, whenever people look sleepy or tired or to create a natural break between activities. • Choose games that are appropriate for the local culture (e.g. when considering games that involve touch or gender interaction). • Select games in which everyone can participate (i.e. be sensitive to participants with disabilities, literacy levels). • Keep energizers short and move on to the next planned activity. Big fish, little fish Form a circle of participants and stand in the middle. As you face a participant, say "big fish" while holding up your hands in the opposite gesture (little fish). The participant facing you must respond with the opposite words ("little fish") and gesture with hands far apart ("big fish"). Move around the circle randomly changing frequently from big fish to little fish. As a participant gives the wrong verbal or gesture response, she/he must take one step back from the circle. As and Bs Ask everyone to choose silently someone in the room to be their "A" person and another person to be their "B" person. There are no set criteria for their choices – selections are entirely up to individuals. Once everyone has made their choices, tell them to get as close to their "A" person as possible, while getting as far away from their "B" person. People can move quickly but should not grab or hold anyone. After a few minutes, stop and reverse the process, participants getting close to their "B" person and avoiding their "A" person. Shakeout In a circle, everyone shakes out their left arm, right arm, left leg then right leg, starting with eight times for each limb, then repeats all four limbs with four, two and then one1. Count out loud for full effect – counting in other languages encouraged! Birthday line-up Ask the group to line themselves up in the order of their birthday (or height, for example) without speaking. What we have in common The facilitator calls out a characteristic of people in the group, such as "have children". All those who have children should move to one corner of the room. As the facilitator calls out more characteristics, such as "likes football", people with the characteristic move to the indicated space. 114 FACILITATOR'S GUIDE Coconut The facilitator shows the group how to spell out C-O-C-O-N-U-T by using full movements of the arms and the body. All participants then try this together. (If spelling in English is difficult for some, participants can spell their names). Group statues Ask the group to move around the room, loosely swinging their arms and gently relaxing their heads and necks. After a short while, shout out a word. The group must form themselves into statues that describe the word. For example, the facilitator shouts "peace". All the participants have to instantly adopt poses, without talking, that show what "peace" looks like to them. Repeat the exercise several times. Countdown Ask participants to form a circle. Explain that the group needs to count together from 1 to 50. However, they cannot say "seven" or any number that is a multiple of seven. Instead, they have to clap their hands. Once someone claps their hands, the group must count the numbers in reverse. If someone says seven or a multiple of seven, start the counting again. Rainstorm Everyone sits quietly in a circle, with their eyes closed, waiting for the facilitator's first movement. The facilitator rubs his/her palms together to create the sound of rain. The person to their right makes this sound, and then the next person until everyone in the group is making the same sound. Once everyone is rubbing palms, the facilitator makes the rain sound louder by snapping his/her fingers, and that sound in turn is passed around the circle. Then the facilitator claps both hands together, and that sound is passed around the circle to create a rainstorm. Then the facilitator slaps his/her thighs, and the group follows. When the facilitator and the group stomp their feet, the rain becomes a hurricane. To indicate the storm is stopping, the facilitator reverses the order, thigh slapping, then hand clapping, finger snapping, and palm rubbing, ending in silence. Mirror image Participants sort themselves into pairs. Each pair decides which one of them will be the "mirror." This person then copies (mirrors) the actions of their partner. After some time, ask the pair to swap roles so that the other person can be the "mirror." Shopping list The group forms a circle. One person starts by saying "I am going to the market to buy fish." The next person says, "I am going to the market to buy fish and potatoes." Each person repeats the list and then adds an item. The aim is to be able to remember all of the items that all of the people before you have listed. Reference 1. 100 ways to energize groups. Brighton: International HIV/AIDS Alliance; 2002 (https://www.aidsalliance.org/ resources/467-100-ways-to-energise-groups, accessed 1 November 2017). SECTION IV: TOOLS FOR FACILITATORS 115 Quick exercises to assess participant learning Use these activities throughout the day to assess if participants have learned key points and to identify areas of confusion. These exercises also serve as fun energizers! Raffle review Place five questions about the session's content into a "raffle" box, bowl or envelope. Invite a participant to select and answer a question. That participant then chooses the next participant to select and answer a question. Minute paper shuffle Ask participants to write a relevant question about the material, using no more than a minute, and collect them all. Shuffle and redistribute, asking each participant to answer the question. True or false? State a key point of fact from the material (or slightly alter it to make it incorrect) and ask, "True or False?" Instruct participants to stand up if they think the statement is true. Pass the marker Give a flip chart marker (or other item) to a participant who then states one key point from the session. They pass the marker to another participant at random who states another key point and so on. Ball toss This is another version of Pass the marker. Ask participants to stand in a circle. Hold a small ball (or crumpled piece of paper) and state one thing you learned from the session. Then toss the ball to another participant who shares a key point they remember from the session. They toss the ball to another participant and so on until all participants have shared a key point. Quote minus one Provide a quote relevant to your topic but leave out a crucial word and ask participants to guess what it might be. For example, "Never start a question with the word _____." Headlines Form groups of two to four participants and ask them to write a headline for a newspaper that summarizes the main learning point of the session. Invite teams to write their headlines on a flip chart 116 FACILITATOR'S GUIDE Dealing with disrupters: prevention and intervention Disrupters are people who try to dominate, stray from the agenda, have side conversations or who think they are right and ridicule and attack others' ideas (1). Try using these "preventions" to avoid disruption 1. Get agreement on the agenda, ground rules and outcomes. Agreeing on the process creates a sense of shared accountability and ownership of the meeting, joint responsibility for how the meeting is run, and group investment in whether the outcomes and goals are achieved. 2. Listen carefully. Avoid pretending to listen to what a participant is saying as people can tell. Listen closely to understand a point someone is making and check to see if you understood their idea correctly. 3. Show respect for experience. Encourage participants to share strategies, stories from the field and lessons they have learned. Value the experience and wisdom in the room. 4. Find out the group's expectations. Uncover at the start why participants think they are here. Then be clear about what will and will not be covered in this meeting. Make plans for how to cover issues that will not be dealt with: Write them down on flip charts ("parking lots") and agree to deal with them at the right time. 5. Do not be defensive. If you are attacked or criticized, take a "mental step" backwards before responding. Once you become defensive, you risk losing the group's respect and trust and might cause participants to feel they cannot be honest with you. 6. Ally with "power players". These participants can turn your meeting into a nightmare if they do not feel that their influence and role are acknowledged and respected. If possible, acknowledge them from the start. Try giving them roles to play during the meeting such as a "sounding board" for you at breaks, to check in with about how the meeting is going. Try using these "interventions" when disruption is happening If someone dominates the meeting, refuses to stick to the agenda, keeps bringing up the same point again and again or challenges how you are handling the meeting, you need to intervene to ensure a useful meeting for all participants. 1. Refer back to the agenda and the ground rules. Make a bargain to deal with their issue for a short period of time ("OK, let's deal with your issue for 5 minutes and then we ought to move on"). If that does not work, agree to defer the issue to the "parking lot" or end of the day. 2. Pull the group behind you. Get group consensus on how to proceed or correct an unhelpful or awkward dynamic within the group. 3. Use humour. Try a humorous comment or a joke, especially about yourself. Humour almost always lightens the mood and is particularly helpful if you are an outsider, there is a lot of tension in the room, if some people do not want to be there or others are scared/shy about participating. 4. Acknowledge the issue or unhelpful comments. Make it clear that you understand how important their issue is by saying, "It's a very important point and one I'm sure we all feel is critical". SECTION IV: TOOLS FOR FACILITATORS 117 5. Use body language. Move closer to conversers, or to the quiet ones. Make eye contact with them to get their attention and convey your intent. 6. Take a break. If case you have tried all of the above suggestions and nothing has worked, take a break. Invite the disruptive person outside the room and politely but firmly state your feelings about how disruptive their behaviour is to the group. Also try to find out what is going on, and see if there are other ways to address that person's concerns. But make it clear that the disruption needs to end. Reference 1. Community tool box. Lawrence (KS): Center for Community Health and Development, University of Kansas; 2017 (http://ctb.ku.edu, accessed 1 November 2017). 118 FACILITATOR'S GUIDE Instructional methods METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS INFORMATIONAL Lecture Lecturette (shorter lecture) Lecture forum (with question cards or question/ answer period) Conveys large sum of information; fast; efficient forum allows exploration of content in more detail Audience is largely passive Trainer should be an interesting speaker, able to self-limit and stick to time, and be able to facilitate questions effectively Panel Panel forum Expanding panel (vacant chair—individual can join panel when wishing to express opinion) Adds different points of view to content Audience is largely passive with exception of expanding panel; expanding panel not practical with groups larger than 20 Leader must express solid set of ground rules and have skills to enforce them Debate Provides different points of view; thought-provoking Audience is largely passive Same as for panel Presentation Presentation with listening teams (participants given listening assignment before presentation and question speaker afterward) Presentation with reaction panel (small group listens and forms panel following presentation) Keeps participants interested and involved; resources can be discovered and shared; learning can be observed; lots of information in a fast format with new points of view; a more organized question-and-answer format; reaction panel can speak Learning points can be confusing or lost; a few participants may dominate the discussion; time control is more difficult; audience is largely passive; a reaction panel may not represent all views of the group Trainer orally presents new information to the group; trainer should structure listening assignment with clear purpose; must select panelists from a cross- section of the group Film Prepared videotape Slides Educational television Reinforces content; adds entertainment; video allows lights to remain on; flexible start and stop for discussion Passive methods for an audience; possibility of equipment problems These methods are not appropriate early in a session; never use to start a session; always introduce and debrief a film, etc. with phrases such as "Look for___, ___, & __"; leader should set up and test equipment before session; have adapter plug and extra bulbs SECTION IV: TOOLS FOR FACILITATORS 119 METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS INFORMATIONAL Lecture Lecturette (shorter lecture) Lecture forum (with question cards or question/ answer period) Conveys large sum of information; fast; efficient forum allows exploration of content in more detail Audience is largely passive Trainer should be an interesting speaker, able to self-limit and stick to time, and be able to facilitate questions effectively Panel Panel forum Expanding panel (vacant chair—individual can join panel when wishing to express opinion) Adds different points of view to content Audience is largely passive with exception of expanding panel; expanding panel not practical with groups larger than 20 Leader must express solid set of ground rules and have skills to enforce them Debate Provides different points of view; thought-provoking Audience is largely passive Same as for panel Presentation Presentation with listening teams (participants given listening assignment before presentation and question speaker afterward) Presentation with reaction panel (small group listens and forms panel following presentation) Keeps participants interested and involved; resources can be discovered and shared; learning can be observed; lots of information in a fast format with new points of view; a more organized question-and-answer format; reaction panel can speak Learning points can be confusing or lost; a few participants may dominate the discussion; time control is more difficult; audience is largely passive; a reaction panel may not represent all views of the group Trainer orally presents new information to the group; trainer should structure listening assignment with clear purpose; must select panelists from a cross- section of the group Film Prepared videotape Slides Educational television Reinforces content; adds entertainment; video allows lights to remain on; flexible start and stop for discussion Passive methods for an audience; possibility of equipment problems These methods are not appropriate early in a session; never use to start a session; always introduce and debrief a film, etc. with phrases such as "Look for___, ___, & __"; leader should set up and test equipment before session; have adapter plug and extra bulbs 120 FACILITATOR'S GUIDE METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS INFORMATIONAL (CONTD) Group discussion (of given topic) Buzz groups (short, time-limited discussion on given subject) Keeps participants interested and involved; resources can be discovered and shared; learning can be observed; participants are active; get a chance to hear other points of view; quieter people can express viewpoints and ideas Learning points can be confusing or lost; a few participants may dominate the discussion; time control is more difficult; an inexperienced leader may be unable to use format for attitudinal purposes Trainer divides large group into small groups with groups of four to six most effective; a small group has a short time to discuss a topic or solve a problem; trainer should be able to give clear instructions and keep discussion on target; his or her main function is judging when to cut off discussion Brainstorming Can get all participants involved in collecting a lot of information and quickly generating ideas; good for problem-solving; quick change of pace; filler; allows all to participate; validates ideas of group The problem/issue must be clearly defined; time control is more difficult; needs clear trigger questions and evaluation/discussion afterwards; somewhat over-used method; requires careful facilitation For idea generation and creative group thinking; all participants present many ideas as rapidly as possible on a problem or issue; the group organizes list into categories for further discussion; do not evaluate, criticize, omit or discuss contributions until all are recorded in contributor's own words; use another person to record if possible Reading (alone or aloud) Reading with discussion or report Saves time (trainees can read faster than trainer can talk); material can be kept for later use; ensures consistency of information; engaging, active; provides a chance for in-depth insight and different perspectives Can be boring if used for too long without interruption; participants read at different paces and it is difficult to measure if people are learning; may require more reading/writing skills than participants have; leader may have to fill in after reports Written material is used to present new information to participants; requires skill to select relevant material; reading skill by participants ATTITUDINAL Role-play Mini-role-play Playing self "movie" See Behavioural methods See Behavioural methods See Behavioural methods Doodling Portraits (of self or others) Thought provoking; good for making thoughts and opinions more concrete, warm-up activity Requires participant self-direction Need to relate to relevant learning goals Simulation games See Behavioural methods See Behavioural methods See Behavioural methods SECTION IV: TOOLS FOR FACILITATORS 121 METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS INFORMATIONAL (CONTD) Group discussion (of given topic) Buzz groups (short, time-limited discussion on given subject) Keeps participants interested and involved; resources can be discovered and shared; learning can be observed; participants are active; get a chance to hear other points of view; quieter people can express viewpoints and ideas Learning points can be confusing or lost; a few participants may dominate the discussion; time control is more difficult; an inexperienced leader may be unable to use format for attitudinal purposes Trainer divides large group into small groups with groups of four to six most effective; a small group has a short time to discuss a topic or solve a problem; trainer should be able to give clear instructions and keep discussion on target; his or her main function is judging when to cut off discussion Brainstorming Can get all participants involved in collecting a lot of information and quickly generating ideas; good for problem-solving; quick change of pace; filler; allows all to participate; validates ideas of group The problem/issue must be clearly defined; time control is more difficult; needs clear trigger questions and evaluation/discussion afterwards; somewhat over-used method; requires careful facilitation For idea generation and creative group thinking; all participants present many ideas as rapidly as possible on a problem or issue; the group organizes list into categories for further discussion; do not evaluate, criticize, omit or discuss contributions until all are recorded in contributor's own words; use another person to record if possible Reading (alone or aloud) Reading with discussion or report Saves time (trainees can read faster than trainer can talk); material can be kept for later use; ensures consistency of information; engaging, active; provides a chance for in-depth insight and different perspectives Can be boring if used for too long without interruption; participants read at different paces and it is difficult to measure if people are learning; may require more reading/writing skills than participants have; leader may have to fill in after reports Written material is used to present new information to participants; requires skill to select relevant material; reading skill by participants ATTITUDINAL Role-play Mini-role-play Playing self "movie" See Behavioural methods See Behavioural methods See Behavioural methods Doodling Portraits (of self or others) Thought provoking; good for making thoughts and opinions more concrete, warm-up activity Requires participant self-direction Need to relate to relevant learning goals Simulation games See Behavioural methods See Behavioural methods See Behavioural methods 122 FACILITATOR'S GUIDE METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS ATTITUDINAL (CONTD) Task groups Sustained interaction allows quieter people to express themselves; validates participants Time-consuming; requires great degree of self-direction and group maturity Keep groups small and diverse with sustained interaction and clear purpose Pantomime skits Engaging, active; good for warm-up Willingness and trust of group necessary Need to provide direction and purpose; relate method to relevant goals BEHAVIOURAL Role play Mini-role play Helps retention; allows participants to practise new skills in a controlled environment; participants are actively involved; observers can impact attitude and behaviour Requires preparation time; may be difficult to tailor to all situations; needs sufficient class time for exercise completion and feedback; requires maturity and willingness of groups; requires trainer have excellent facilitation skills Participants act out problem-solving situations similar to those they will encounter in their workplace; trainer needs skill and understanding as he or she must get people into roles, give directions, establish a climate of trust; trainer needs insight into how an activity may pose a threat to some individuals; ability to help group process and debrief; use in well-formed group; can be structured into dyad, triad and fishbowl Movie (role play assisted by feedback, "more__, or less __") Useful in rehearsing new skills, behaviours Same as for role play, intensive and time-consuming Simulation games Intense involvement; practise skills in problem solving and decision-making; competitive Competitive; requires a game and possibly a consultant to help facilitate; time-consuming A package game requires preparation time for the leader to learn the rules and directions Tape recording with playback Videotape recording with playback Very concrete learning tool; participant involved in judging own performance Criteria must be clear; feedback and assessment based on specific behaviours; requires equipment Trainer should establish purpose and performance criteria clearly Case study Mini-case study (problem situations for small groups to analyse) Critical incident (small section of case stating most critical or dramatic moment) Requires active participant involvement; can simulate performance required after training; learning can be observed; opportunity to apply new knowledge; requires judgement; good assessment tool; participants active; chance to practise skills Information must be precise and kept up-to-date; needs sufficient class time for participants to complete the case; participants can become too interested in the case content; case study must be relevant to learner's needs and daily concerns Participants are given information about a situation and directed to come to a decision or solve a problem concerning the situation; trainer needs to have knowledge and skills to "solve" the problem; may need to design own studies; compare approaches of several groups and reinforce best solutions SECTION IV: TOOLS FOR FACILITATORS 123 METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS ATTITUDINAL (CONTD) Task groups Sustained interaction allows quieter people to express themselves; validates participants Time-consuming; requires great degree of self-direction and group maturity Keep groups small and diverse with sustained interaction and clear purpose Pantomime skits Engaging, active; good for warm-up Willingness and trust of group necessary Need to provide direction and purpose; relate method to relevant goals BEHAVIOURAL Role play Mini-role play Helps retention; allows participants to practise new skills in a controlled environment; participants are actively involved; observers can impact attitude and behaviour Requires preparation time; may be difficult to tailor to all situations; needs sufficient class time for exercise completion and feedback; requires maturity and willingness of groups; requires trainer have excellent facilitation skills Participants act out problem-solving situations similar to those they will encounter in their workplace; trainer needs skill and understanding as he or she must get people into roles, give directions, establish a climate of trust; trainer needs insight into how an activity may pose a threat to some individuals; ability to help group process and debrief; use in well-formed group; can be structured into dyad, triad and fishbowl Movie (role play assisted by feedback, "more__, or less __") Useful in rehearsing new skills, behaviours Same as for role play, intensive and time-consuming Simulation games Intense involvement; practise skills in problem solving and decision-making; competitive Competitive; requires a game and possibly a consultant to help facilitate; time-consuming A package game requires preparation time for the leader to learn the rules and directions Tape recording with playback Videotape recording with playback Very concrete learning tool; participant involved in judging own performance Criteria must be clear; feedback and assessment based on specific behaviours; requires equipment Trainer should establish purpose and performance criteria clearly Case study Mini-case study (problem situations for small groups to analyse) Critical incident (small section of case stating most critical or dramatic moment) Requires active participant involvement; can simulate performance required after training; learning can be observed; opportunity to apply new knowledge; requires judgement; good assessment tool; participants active; chance to practise skills Information must be precise and kept up-to-date; needs sufficient class time for participants to complete the case; participants can become too interested in the case content; case study must be relevant to learner's needs and daily concerns Participants are given information about a situation and directed to come to a decision or solve a problem concerning the situation; trainer needs to have knowledge and skills to "solve" the problem; may need to design own studies; compare approaches of several groups and reinforce best solutions 124 FACILITATOR'S GUIDE METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS BEHAVIOURAL (CONTD) "In-basket" (form of case study in which letters, memos, etc. are given to participants for response) Helps participants to clarify and crystallize thoughts, opinions, values; opportunity to apply knowledge to "real" situation Requires writing skills; must be relevant to participants Leader needs knowledge of participants' daily concerns/needs; ability to critique responses Demonstration Demonstration with practise (by participants) Aids comprehension and retention; stimulates participants' interest; can give participants model to follow; allows for optional modelling of desired behaviour/skill; can be active; good for learning simple skills Must be accurate and relevant to participants; written examples can require lengthy preparation time; trainer demonstrations may be difficult for all participants to see well; method more effective if participants are active; feedback must follow immediately after practise Participants are shown the correct steps for completing a task or are shown an example of a correctly completed task; requires skill to model desired behaviour; break procedure down into simple steps; ability to provide feedback Skills practise laboratory (small participant groups practise together) Different points of view and feedback; participants active; good for translating information into skills Group should have enough knowledge or insight to coach one another Act as a resource to groups PLANNING Group discussion with decision-making regarding a new action Individual or group planning session with report Validates maturity and needs of group members; members have best insight into their problems and needs for their job; group leaves session with practical, constructive and mutual goals; groups get ideas from one another Requires mature group that can self-direct and stay on task; time-consuming Leader serves as resource once directions are given SECTION IV: TOOLS FOR FACILITATORS 125 METHOD ADVANTAGES POSSIBLE DISADVANTAGES COMMENTS BEHAVIOURAL (CONTD) "In-basket" (form of case study in which letters, memos, etc. are given to participants for response) Helps participants to clarify and crystallize thoughts, opinions, values; opportunity to apply knowledge to "real" situation Requires writing skills; must be relevant to participants Leader needs knowledge of participants' daily concerns/needs; ability to critique responses Demonstration Demonstration with practise (by participants) Aids comprehension and retention; stimulates participants' interest; can give participants model to follow; allows for optional modelling of desired behaviour/skill; can be active; good for learning simple skills Must be accurate and relevant to participants; written examples can require lengthy preparation time; trainer demonstrations may be difficult for all participants to see well; method more effective if participants are active; feedback must follow immediately after practise Participants are shown the correct steps for completing a task or are shown an example of a correctly completed task; requires skill to model desired behaviour; break procedure down into simple steps; ability to provide feedback Skills practise laboratory (small participant groups practise together) Different points of view and feedback; participants active; good for translating information into skills Group should have enough knowledge or insight to coach one another Act as a resource to groups PLANNING Group discussion with decision-making regarding a new action Individual or group planning session with report Validates maturity and needs of group members; members have best insight into their problems and needs for their job; group leaves session with practical, constructive and mutual goals; groups get ideas from one another Requires mature group that can self-direct and stay on task; time-consuming Leader serves as resource once directions are given 126 FACILITATOR'S GUIDE REFERENCES AND RESOURCES Slide set 1: What is evidence? Campbell S (2012). The knowledge translation curriculum. Ottawa: Canadian Coalition for Global Health Research (http://www.ccghr.ca/resources/knowledge-translation, accessed 11 December 2017). GRADE Working Group (http://www.gradeworkinggroup.org/, accessed 11 December 2017). International Migration Institute (2017). Determinants of International Migration (DEMIG) project. Oxford: International Migration Institute, University of Oxford (https://www.imi.ox.ac.uk/completed-projects/ demig, accessed 1 November 2017). Kouyate B, Nana V (2010). Policy brief on improving access to artemisinin-based combination therapies for malaria in Burkina Faso. Int J Technol Assess Health Care. 26(2):233–36. Lavis JN, Panisset U (2010). EVIPNet Africa’s first series of policy briefs to support evidence-informed policy-making. Int J Technol Assess Health Care. 26:229–32. Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policy-making (STP) 1: what is evidence-informed policymaking. Health Res Policy Syst. 7(Suppl 1):S1 (https://health-policy-systems.biomedcentral.com/articles/10.1186/1478-4505-7-S1-I1, accessed 1 November 2017). Lomas J, editor (2005). Defining evidence and its role in guiding policy and practice. In: WHO Meeting on knowledge translation in global health. Geneva: World Health Organization (http://www.who.int/kms/ events/DefiningEvidence_JLomas.pdf?ua=1, accessed 30 October 2017). Lomas J, Culyer T, McCutcheon C, McAuley L, Law S (2005). Conceptualizing and combining evidence for health system guidance. Hamilton: Canadian Health Services Research Foundation (http://www.cfhi-fcass. ca/migrated/pdf/insightAction/evidence_e.pdf, accessed 11 December 2017). Tikki P (2007). Evidence to action in the developing world: what evidence is needed? Bull World Health Organ. 85(4):247. Walshe K, Rundall TG (2001). Evidence-based management: from theory to practice in health care. Milbank Q. 79(3):440–1. Slide set 2: Clarifying a problem Chopra M, Munro S, Lavis JN, Vist G, Bennett S (2008). Effects of policy. options for human resources for health: an analysis of systematic reviews. Lancet. 371:668–74. Kutzin J (2001). A descriptive framework for country-level analysis of health care financing arrangements. Health Policy. 56:171–204. Lavis JN, Wilson M, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3. Clarifying the problem, version 2.1. SURE Collaboration (http://epoc.cochrane.org/sites/epoc.cochrane.org/files/ public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html, accessed 11 December 2017). World Health Organization (2010). Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies. Geneva: World Health Organization (http://www.who.int/ healthinfo/systems/WHO_MBHSS_2010_full_web.pdf, accessed 11 December 2017). Day 1 activities Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). SECTION IV: TOOLS FOR FACILITATORS 127 Slide set 3: Identifying options Lavis JN, Greco L (2010). Health Systems Evidence: evidence to support decision-making – an online repository of synthesized research evidence for health system policymakers, managers and stakeholders. Hamilton: McMaster Health Forum. Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). Lavis JN, Wilson MG, Oxman AD, Grimshaw J, Lewin S. Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 5: using research evidence to frame options to address a problem. Health Res Policy Syst. 7(Suppl 1):S5 (https://health-policy-systems.biomedcentral.com/ articles/10.1186/1478-4505-7-S1-S5, accessed 11 December 2017). Slide set 4: Finding evidence: systematic reviews Egger M, Smith GD, Altman DG (2008). Systematic reviews in health care, second edition. London: BMJ Books (http://onlinelibrary.wiley.com/doi/10.1002/9780470693926.fmatter/pdf, accessed 11 December 2017). Gough D, Oliver S, Thomas J, editors (2017). An introduction to systematic reviews, 2nd edition. London: Sage. Lavis JN (2012). Finding and using research evidence. Hamilton: Canada: McMaster Health Forum. Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3. Clarifying the problem, version 2.1. SURE Collaboration (http://epoc.cochrane.org/sites/epoc.cochrane.org/files/ public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html, accessed 11 December 2017). Websites Health Systems Evidence https://www.healthsystemsevidence.org/ Cochrane Library www.thecochranelibrary.com Health Evidence https://www.healthevidence.org/ PDQ Evidence http://www.pdq-evidence.org/ PubMed www.pubmed.gov SUPPORT summaries http://www.support-collaboration.org/summaries.htm Slide set 5: Assessing a systematic review Evidence-Informed Policy Network (2017). SURE guides for preparing and using evidence-based policy briefs. Geneva: World Health Organization (http://www.who.int/evidence/sure/guides/en/, accessed 30 October 2017). Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P et al. for the GRADE Working Group (2008). Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 336:924–6. Higgins JPT, Green S, editors (2011). Cochrane handbook for systematic reviews of interventions, version 5.1.0. London: The Cochrane Collaboration. Lavis JN (2009). How can we support the use of systematic reviews in policy-making? PLOS Med. 6(11): e1000141. Lavis JN (2010). Criteria for assessing systematic reviews. Hamilton: McMaster University Program in Policy Decision-Making [an example of applying AMSTAR to assess a systematic review]. Lavis JN, Oxman AD, Grimshaw J, Johansen M, Boyko JA, Lewin S et al. (2009). SUPPORT tools for evidence-informed health policymaking (STP) 7: finding systematic reviews. Health Res Policy Syst. 7(Suppl 1):S7 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271834/, accessed 11 December 2017). 128 FACILITATOR'S GUIDE Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A (2009). SUPPORT tools for evidence- informed health policymaking (STP) 9: assessing the applicability of the findings of a systematic review. Health Res Policy Syst. 7(Suppl 1):S9 (https://health-policy-systems.biomedcentral.com/ articles/10.1186/1478-4505-7-S1-S9, accessed 11 December 2017). Lewin S, Oxman A, Lavis JN, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 8: deciding how much confidence to place in a systematic review. Health Res Policy Syst. 7(Suppl 1):S8 (https://health-policy-systems.biomedcentral.com/articles/10.1186/1478-4505-7-S1-S8, accessed 11 December 2017). Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M et al. (2012). Assessing the risk of bias of individual studies in systematic reviews of health care interventions. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ methods for effective health care series; https://www.ncbi.nlm.nih.gov/books/NBK91433/, accessed 29 November 2017). Slide set 6: Implementing policy options Evidence-Informed Policy Network (2017). Evidence briefs for policy. Geneva: World Health Organization (http://www.who.int/evidence/resources/policy_briefs/en/, accessed 30 October 2017). Fretheim A, Munabi-Babigumira S, Oxman AD, Lavis JN, Lewin S. (2009). SUPPORT tools for evidence- informed policymaking in health 6: using research evidence to address how an option will be implemented. Health Res Policy Syst. 7(Suppl 1):S6 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271833, accessed 11 December 2017). Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). Slide set 7: Preparing evidence briefs for policy Evidence-Informed Policy Network (2017). SURE guides for preparing and using evidence-based policy briefs. Geneva: World Health Organization (http://www.who.int/evidence/sure/guides/en/, accessed 30 October 2017). Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 13: preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 7(Suppl 1):S13 (https://health-policy-systems.biomedcentral.com/ articles/10.1186/1478-4505-7-S1-S13, accessed 11 December 2017). Mijumbi R (2011). How applicable are the 2010 WHO guidelines for infant feeding in the context of HIV in low income countries like Uganda? SURE Collaboration (SURE Rapid Response; (http://epoc.cochrane. org/sites/epoc.cochrane.org/files/public/uploads/SURE-Guides-v2.1/Collectedfiles/source/additional_ resources/guide_01_getting_started/01%20rapid%20response%20example%20-%20infant%20 feeding%20guidelines%20in%20hiv%202011%2011.pdf, accessed 11 December 2017). SURE Collaboration (2011). SURE guides for preparing and using evidence-based policy briefs. 3. Clarifying the problem, version 2.1. SURE Collaboration (http://epoc.cochrane.org/sites/epoc.cochrane.org/files/ public/uploads/SURE-Guides-v2.1/Collectedfiles/sure_guides.html, accessed 11 December 2017). World Health Organization (2017). Evidence briefs for policy [website]. Geneva: World Health Organization (http://www.who.int/evidence/resources/policy_briefs/en/, accessed 11 December 2017). Slide set 8: Organizing policy dialogues Jones CM, Mittelmark MB (2007). The IUHPE blueprint for direct and sustained dialogue in partnership initiatives. In: McQueen D, Jones CM, editors. Global perspectives on health promotion effectiveness. New York: Springer:33–40. Lavis JN, Boyko A, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 14: organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 7(Suppl 1):S14 (https://health-policy-systems.biomedcentral.com/ articles/10.1186/1478-4505-7-S1-S14, accessed 11 December 2017). SECTION IV: TOOLS FOR FACILITATORS 129 Section IV Center for Community Health and Development (2017). Community tool box. Lawrence (KS): Center for Community Health and Development, University of Kansas (http://ctb.ku.edu, accessed 1 November 2017). EVIPNet Europe. Evidence brief for policy 1: reducing the consumption of sugar-sweetened beverages and their negative health impact in Estonia. Copenhagen: WHO Regional Office for Europe; 2017 (http://www. euro.who.int/__data/assets/pdf_file/0016/324205/EBP-1-Web.pdf?ua=1, accessed 1 November 2017). International HIV/AIDS Alliance (2002). 100 ways to energize groups. Brighton: International HIV/AIDS Alliance (https://www.aidsalliance.org/resources/467-100-ways-to-energise-groups, accessed 1 November 2017). Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A (2009). SUPPORT tools for evidence-informed health policymaking (STP) 4: using research evidence to clarify a problem. Health Res Policy Syst. 7(Suppl 1):S4 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3271831/, accessed 30 October 2017). Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non- randomised studies of healthcare interventions, or both. BMJ. 2017 Sep 21;358:j4008. (https://www.bmj. com/content/358/bmj.j4008.full, accessed 22 May 2018). Supporting the Use of Research Evidence (SURE) Collaboration (2017). Appendix 1. SUPPORT summaries checklist for making judgments about how much confidence to place in a systematic review (https:// www.ntnu.edu/documents/1269470669/1276564084/SR+checklist.pdf/9aab27bb-0b6e-42ec-872c- 376f5f14f147, accessed 30 October 2017). The WHO Regional Office for Europe The World Health Organization (WHO) is a specialized agency of the United Nations created in 1948 with the primary responsibility for international health matters and public health. The WHO Regional Office for Europe is one of six regional offices throughout the world, each with its own programme geared to the particular health conditions of the countries it serves. Member States Albania Andorra Armenia Austria Azerbaijan Belarus Belgium Bosnia and Herzegovina Bulgaria Croatia Cyprus Czechia Denmark Estonia Finland France Georgia Germany Greece Hungary Iceland Ireland Israel Italy Kazakhstan Kyrgyzstan Latvia Lithuania Luxembourg Malta Monaco Montenegro Netherlands Norway Poland Portugal Republic of Moldova Romania Russian Federation San Marino Serbia Slovakia Slovenia Spain Sweden Switzerland Tajikistan The former Yugoslav Republic of Macedonia Turkey Turkmenistan Ukraine United Kingdom Uzbekistan Document number: WHO/EURO:2018-3018-42776-59687 World Health Organization Regional Office for Europe UN City, Marmorvej 51, DK-2100 Copenhagen Ø, Denmark Tel.: +45 45 33 70 00 Fax: +45 45 33 70 01 Email: eurocontact@who.int

Основные сведения

Организация Всемирная организация здравоохранения (ВОЗ / WHO)

Тип документа Technical Documents

Дата принятия 31 декабря 2018 г.

Источник Всемирная организация здравоохранения