How to build an infodemic insights report in six steps

aPANDE MIC INFODEMIC INSIGHTS REPORT HOW TO BUILD AN IN SIX STEPS OUTBREAK EPIDEMIC b How to build an infodemic insights report in six steps How to build an infodemic insights report in six steps ISBN (WHO) 978-92-4-007565-8 (electronic version) ISBN (WHO) 978-92-4-007566-5 (print version) ISBN (UNICEF) 978-92-806-5468-4 © World Health Organization and the United Nations Children’s Fund (UNICEF), 2023 This joint report reflects the activities of the World Health Organization (WHO) and the United Nations Children’s Fund (UNICEF). Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO or UNICEF endorses any specific organization, products or services. The unauthorized use of the WHO or UNICEF names or logos is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO) or the United Nations Children’s Fund (UNICEF). Neither WHO nor UNICEF are responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization (http://www.wipo.int/amc/en/mediation/rules). Suggested citation. How to build an infodemic insights report in six steps. Geneva: World Health Organization and the United Nations Children’s Fund (UNICEF), 2023. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. UNICEF and WHO Photographs. UNICEF and WHO photographs are copyrighted and are not to be reproduced in any medium without obtaining prior written permission. Permissions may be granted for one-time use in a context that accurately represents the real situation and identity of all human beings depicted. UNICEF and WHO photographs are not to be used in any commercial context; content may not be digitally altered to change meaning or context; assets may not be archived by any non-WHO or non-UNICEF entity. Requests for permission to reproduce UNICEF photographs should be addressed to UNICEF, Division of Communication, 3 United Nations Plaza, New York 10017, USA (email: nyhqdoc. permit@unicef.org). Requests for permission to reproduce WHO photographs should be addressed to: http://www.who. int/about/licensing/copyright_form/en/ General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO or UNICEF concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO or UNICEF in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO and UNICEF to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO or UNICEF be liable for damages arising from its use. Design and layout by QUO Bangkok. Acknowledgements iii Glossary iv OVERVIEW 1 What an infodemic insights report is 1 Manual objectives 2 Who this manual is for 3 How to use this manual 3 Before you get started 4 2.1 Identifying data sources 7 2.2 Assess and select data sources 10 2.3 Develop an analysis plan for each data source 11 3.1 Overview of steps in an integrated analysis 13 3.2 Defining risk assessment criteria 14 3.3 Analyse each data source to derive intelligence 18 3.4 Integrate intelligence from data sources into insights 20 CONTENTS CHOOSE THE QUESTION THAT INFODEMIC MANAGEMENT INSIGHTS COULD HELP ANSWER 5 STEP 1 IDENTIFY AND SELECT DATA SOURCES AND DEVELOP AN ANALYSIS PLAN FOR EACH DATA SOURCE 7 STEP 2 CONDUCT AN INTEGRATED ANALYSIS ACROSS DATA SOURCES 13 STEP 3 ANNEXES 31 A1 Best practices for an infodemic manager in emergency response 32 A2 Competencies for infodemic management team members 33 A3 Resources related to the ethics of social listening and infodemic management 38 A4 How to build a workflow for the infodemic insights report 39 A5 Troubleshooting common challenges 41 A6 Key messages for advocating the use of infodemic insights reports 43 A7 Scaling up or scaling down infodemic insights processes and reports, 44 depending on context A8 Data source evaluation checklist 47 A9 How to run a cross-disciplinary infodemic insights meeting 48 A10 How to develop a taxonomy 49 A11 Sample taxonomy for generating infodemic insights 51 A12 Resources on developing a style guide and aiming for inclusive language 55 A13 Taking an infodemic insights report to implementation 56 APPENDICES 57 Appendix 1 How to develop infodemic insights on zero-dose children 58 Appendix 2 How to develop infodemic insights on polio outbreaks and campaigns 61 Appendix 3 How to develop infodemic insights on seasonal influenza vaccination 64 Appendix 4 How to develop infodemic insights on vaccine safety events 67 Appendix 5 How to develop infodemic insights to inform planning for mass gatherings 70 to prevent outbreaks DEVELOP STRATEGIES AND RECOMMENDATIONS 21 STEP 4 DEVELOP AN INFODEMIC INSIGHTS REPORT 25 STEP 5 DISSEMINATE THE INFODEMIC INSIGHTS REPORT AND TRACK ACTIONS TAKEN 30 STEP 6 iii Acknowledgements The six steps to developing an infodemic insights report was written by Tina Purnat (WHO), and Elisabeth Wilhelm (UNICEF), under the direction of Tim Nguyen (WHO) and Surangani Abeyesekera (UNICEF). Their work was directed by Sylvie Briand, Director of the Department of Epidemic and Pandemic Preparedness and Prevention in the WHO Health Emergencies Programme and Ephrem T. Lemango, Associate Director – Health, Chief of Immunization Programme Group, UNICEF New York Headquarters. The manual has been developed in an iterative and consultative manner with a diverse group of experts working in emergency preparedness and response, social and behaviour change, immunization, infodemic management, digital health and health information analysis. The manual was conceptualized based on discussions, experience sharing and feedback from the WHO infodemic manager community during global conferences, training sessions and through collaborations on scientific publications between 2020 and 2023. The process the manual describes reflects the learning and experience of WHO, UNICEF and other partner organizations, such as members of the Africa Infodemic Response Alliance, the Robert Koch Institute and the US Centers for Disease Control and Prevention, of conducting social listening and preparing reports on infodemic insights during the COVID-19 pandemic. The manual was revised several times based on an iterative consultation process between November 2022 and March 2023, during which WHO and UNICEF collected feedback and discussed the manual in webinars, workshops, group and individual discussions. Feedback was also collected digitally from all levels of their organizations, from the wider social and behaviour change communication community at their international summit in Morocco in December 2022, the WHO infodemic manager network, via social media channels related to infodemic management, and partners in the Digital Information Environment Workstream of the Vaccination Demand Hub (including Gavi, the Vaccine Alliance, Johns Hopkins University and the Sabin Vaccine Institute among others). Special thanks for substantial comments, review and discussion to: Amaia Glaria Artazcoz, WHO | Susana Barragan, European Centre for Disease Prevention and Control | Catherine Bertrand-Ferrandis, OLYLO, France | Christopher Chadwick, WHO | Shoshanna Goldin, WHO | Karen Greiner, UNICEF West and Central Africa Regional Office | Elodie Ho, WHO Regional Office for Africa and Africa Infodemic Response Alliance | Harim Humayun, UNICEF Regional Office for South Asia | Grev Lester Hunt, UNICEF West and Central Africa Regional Office | Alhassan Ibrahim, WHO | Atsuyoshi Ishizumi, WHO | Hamid Kashan, UNICEF | Rokayya Konsowa, WHO Regional Office for Eastern Mediterranean | Fabrice Laurentien, UNICEF Mediterranean and North Africa | Sandra Machiri, WHO | Renata Mares, AdaptableFolks, Canada | Lisa Menning, WHO | Rhys O’Neill, WHO Regional Office for Africa and Africa Infodemic Response Alliance | Jun Ryan Orbina, WHO | Adnan Shahzad, UNICEF | Sylvia Sommariva, UNICEF | Aleksandar Stevanovic, University of Belgrade, Serbia | Anisha Verma, US Centers for Disease Control and Prevention, USA | Lieke Visser, WHO Regional Office for Western Pacific | Christopher Voegeli, US Centers for Disease Control and Prevention, USA | Ninglan Wang, WHO | Claire Wardle, Brown University USA | Silvan Wehrli, Robert Koch Institute, Germany | Becky White, WHO | Brian Yau, WHO. iv Glossary Codebook (in qualitative analysis) A list of terms that are used to tag or group qualitative data to aid in thematic analysis. The codebook also includes definitions of each code word and examples. Directionality Identification the direction of change of narrative over time, specifically whether it is emerging, persistent, re-emerging or declining in the analysed conversations. Hazard Something that is potentially harmful to someone’s health or well-being. In infodemic management context, hazards can be present in different circulating narratives, especially those that may advocate against adhering to health guidance, promote non-evidence-based treatments, promote stigma or harm against groups of people and erode trust in health authorities. Infodemic An overabundance of information, accurate or not, in the digital and physical space, accompanying an acute health event such as an outbreak or epidemic. Infodemics (pl) When there are multiple co-occurring acute health events (for example, a measles outbreak while a seasonal influenza outbreak is occurring), multiple infodemics can co-occur, shifting circulating narratives and impacting people’s perceptions, attitudes and behaviors of people. If outbreaks co-occur, communities may experience mixed messaging by health authorities and confusion in general public, especially vulnerable groups, and interrupt access to routine health information and service delivery. Infodemic insights Infodemic insights are learnings about circulating narratives that are generated through an integrated analysis of diverse data sources. Infodemic insights are often presented in infodemic insights reports which describe narratives, their level or risk, and identify accompanying recommended actions to address the learning from the infodemic insights so that public health can be protected. Infodemic intelligence Infodemic intelligence are pieces of key information that result from analysis of an individual data source. When intelligence from different data sources is analysed in an integrated manner, this yields infodemic insights which in turn are used to form recommendations in the infodemic insights report. Narrative A description of a series of events or a story that connects and provides meaning to disparate pieces of information. In infodemic management context, narratives are units of analysis because they embed meaning, values and emotions that can explain how people understand and share pieces of information. These embedded components of a narrative can be hijacked or can blend with mis- and disinformation narratives which can divert people’s attention into inaccurate understanding and interpretation of information. In the context of health, narratives help explain popular understanding of outbreaks and emergencies, health guidance, level of risk related to certain behaviour, modes of transmission, infection routes and prevention measures, etc. These narratives sometimes do not reflect scientific fact, but instead reflect a collective understanding of a health issue which can influence people’s perceptions and behaviour. vRisk The likelihood of a hazard causing harm to an individual or group. In infodemic management context, the risk of a narrative depends on its spread across channels or its disproportionate impact among a vulnerable population. Risk assessment/Risk levels A tool, combined with expert judgement, to determine the degree of likelihood of a hazard causing harm. In infodemic management context, a risk assessment is conducted on narratives to determine a level of potential harm of this narrative, therefore informing prioritization of recommended actions by a health authority. Spread The extent to which a narrative appears across different (online/offline) communities, internet platforms, data sources. Sticky/stickiness (of a narrative) The quality of a narrative to stay in a person’s mind after exposure. This can sometimes be described as challenging a person’s imagination, maintaining interest in a narrative and readily recalling or remembering a narrative. An infodemic manager works to make health information sticky to better compete against mis- and disinformation narratives that tend to have sticky qualities. Taxonomy A systematic classification of terms. In infodemic management context, this describes the terms used to organize coding of narratives, questions, concerns, information voids and circulating mis- and disinformation about a public health issue. Velocity Velocity of the spread of a narrative indicates (1) the speed with which the narrative spreads through different (online/offline) communities, internet platforms, and data sources. and (2) the directionality of the spread of the narrative over time.

1An infodemic is an overabundance of information, accurate or not, in the digital and physical space, accompanying an acute health event such as an outbreak or epidemic. Infodemics contain questions, concerns, information voids (where people seek credible, accurate information but cannot find it) and circulating mis- and disinformation. An infodemic is accelerated and amplified through digital media and offline, causing information overload and confusion. An infodemic can promote stigma, erode trust in health authorities, affect mental health and negatively influence health decisions and behaviours, thereby making it more difficult for health authorities to respond effectively and protect the population’s health. In order to mitigate an infodemic that is affecting a population and the public health response, we must understand it by developing an infodemic insights report. What an infodemic insights report is Within a noisy information environment that affects people’s perceptions, attitudes and health decisions, there is often a gap between health guidance and the population’s behaviour. In these cases, infodemic insights can help to inform an emergency response or the response of a health programme. In the early stage of an outbreak, infodemic insights process can be stood up rapidly to inform the adaptation of outbreak response as the situation rapidly evolves. The infodemic insights process can be adapted depending on the phase of the outbreak and available resourcing. An insights report is a rapid synthesis of: • what has been learned from an integrated analysis of data from diverse sources on how people, communities, health systems and societies are being affected by an infodemic, how it is shaping the information environment, and how it may impact attitudes, awareness, perceptions and health behaviours; • the implications this has for a health or emergency programme; and • recommendations for actions or strategies that can be taken to address these issues. OVERVIEW Rapid infodemic insight reports are not a replacement for the more comprehensive evidence generation that occurs in routine health programming. Infodemic insights are specifically aimed at generating rapid and reproducible insights to inform a public health response when time is short. They include recommendations for action that shape the information environment, whereas other types of evidence gathering focus on assessing the public health impact, burden, or prevalence of a particular pathogen or health issue. These two are complementary and do not replace each other. PRO TIP How do infodemic insights reports relate to other types of health reports? 2 How to build an infodemic insights report in six steps This manual provides a quick overview of the steps required to develop an infodemic insights report that can be used during an emergency response or for routine health programming (where so-called low-level infodemics may be more common). The steps are: 1. Choose the question that infodemic management insights could help to answer 2. Identify and select the data sources and develop an analysis plan for each data source Manual objectives 3. Conduct an integrated analysis across those data sources 4. Develop strategies and recommendations 5. Develop an infodemic insights report 6. Disseminate the infodemic insights report and track the actions taken. Each of the following chapters of the manual is dedicated to explaining one step of this process Choose the question that infodemic management insights could help to answer S T E P Conduct an integrated analysis across those data sources Identify and select the data sources and develop an analysis plan for each data source S T E P Develop strategies and recommendationsS T E P S T E P Develop an infodemic insights report S T E P INFODEMIC INSIGHTS S T E P Disseminate the infodemic insights report and track the actions taken INFODEMIC INSIGHTS 3Who this manual is for Depending on the emergency response or health programme structure, people in various roles may be involved in generating infodemic insights and developing insight reports. Different stakeholders may have infodemic management functions – from national-level public health institutes to local health departments to factchecking organizations to community-based organizations focused on health topics – all can develop and benefit from infodemic insight reports. This manual is written for people with exposure to digital analytics, digital health, communications, behavioural science, health information analysis, health promotion, emergency response and related areas in the context of public health. No one has all the requisite skills to be the perfect infodemic manager because the phenomenon is complex and the field is still developing – but this manual is designed to provide enough guidance to effectively build a report with only some exposure to the fields listed above. In fact, people called “infodemic managers” who do this work full time are rare, and it is more common that people in different positions in an emergency response or health programme may be assigned infodemic management duties. This manual is designed to be used by someone just starting out on the task of setting up an insights unit or developing an insights report, who will not have unlimited financial, human or technical resources, but needs to establish a process and deliver an actionable report quickly. Basic digital or public health concepts will not be explained because a baseline knowledge is assumed. Moreover, details on how different types of data sources are set up and managed will not be covered in this manual. How to use this manual The manual provides simple steps and templates to help you map out the process of developing insights and creating a report. These templates are explained in the steps can be downloaded for use in various formats. Additional guidance in the steps includes examples to help clarify explanations and Pro tips that call attention to important considerations and advice from lessons learned on specific topics. At the end of the manual, extensive annexes provide support for common tasks, reference materials, and relevant resources and tools. After the annexes, there are topic-specific appendices that serve as short topic guides, which can be used separately as handouts for analysts working in specific contexts. Training materials are also available as a companion to this manual to help teach this process to others. Take the manual and its components and adapt it to your context and needs. Remember that the aim is to design an evidence-based, reproducible process for developing insights that are going to be used to improve public health outcomes. The processes you establish now will determine the usefulness of your infodemic insights, but be open to refining your process as you develop reports and learn how to optimize the various elements. This manual is accompanied by courses on the OpenWHO “Infodemic management” channel. For a list of resources, tools, research papers, events, and other information relevant to infodemic managers, subscribe to the WHO infodemic management news flash. Additional courses are available on the UNICEF Agora platform. Infodemic monitoring, analysis and reporting processes that are described in this manual can be applied to a large range of health and development related topics. Specific topic-based handouts are currently available in the Appendix for the following topics: • zero-dose children • polio • seasonal influenza • vaccine safety events • mass gathering events If you have used this manual to develop insights reports on other topics and would like to submit a similar handout for inclusion in an online repository, please email infodemicmanagement@ who.int. Additional topic-based handouts will be made available in the Google Drive folder for this manual. Overview 4 How to build an infodemic insights report in six steps Infodemic management is a team sport. Because the volume of data can be massive and many different skills are needed to analyse it, it often helps to share the load and work with a team. If you are the only person managing infodemic-related tasks, you can still build an informal network of advisers and partners that can help verify each step of the process. However, if you have the ability to share duties with other team members or partners, you can build an infodemic insights team. Consider what types of skills you need to analyse and interpret the data. It’s really important to include people with subject-matter expertise who can make informed recommendations (e.g. if your question Before you get started is related to the zoonotic transmission of a novel pathogen, involve a One Health expert who has worked on the human–animal interface and the public health teams that are involved in managing it). Consider including expertise from the relevant health or emergency field, communications, digital health, data analytics, informatics, epidemiology, sociobehavioural science and health promotion to provide a cross-disciplinary perspective in the analysis phase. Team members can be brought together on an ad hoc basis or regularly tasked with analysing and interpreting datasets that they are familiar with. For general guidance and principles to help before you get started, see annexes A1 through A7: A1 Best practice for an infodemic manager in emergency response A2 Competencies for infodemic management team members A3 Resources related to the ethics of social listening and infodemic management A4 How to build a workflow for the infodemic insights report A5 Troubleshooting common challenges A6 Key messages for advocating the use of infodemic insights reports A7 Scaling up or scaling down infodemic insights processes and reports, depending on context 5Talk to health programme or emergency response colleagues about what behaviours or trends are worrying them or an issue they want to understand better. This may require some coaching, because there is a human tendency to describe problems instead of developing actionable research questions. Here are some examples of types of questions that would be good: 1. Despite widespread availability of vaccines and strong recommendations for people in Community X to get vaccinated, why is Community X still undervaccinated? 2. What questions or concerns do pregnant women have about Zika infection and prevention measures? 3. How did the misinformation narratives about microchips in COVID-19 vaccines circulate within communities and affect perceptions? 4. How well do caregivers understand the changing guidance about catch-up vaccination for their children? 5. What concerns do religious leaders and religious communities have about the upcoming measles–rubella campaign? 6. How are people hearing, sharing and reacting to the concern that the new Ebola outbreak can’t be prevented by currently available vaccines? 7. What alternative cures, treatments or preventives are being discussed, marketed or sold to a target audience that are not recommended? How is this affecting people’s perceptions and health behaviour related to the pathogen/health topic? CHOOSE THE QUESTION THAT INFODEMIC MANAGEMENT INSIGHTS COULD HELP ANSWER It is important to choose a question that is linked to understanding conversations or behaviour that has an impact on achieving a public health objective. For example, doing a general search on everything people are talking about regarding breast cancer or the flu season is too broad to yield programmatically useful insights and will take a very long time or a lot of person-hours to analyse very quickly. Ask a question that can actually be answered with your available data sources and capacity for analysis and that can be addressed within the scope and activities of the emergency response or health programme team you’re working with. PRO TIP A useful analysis starts with choosing the right question STEP 1 6 How to build an infodemic insights report in six steps • Do make sure you define your population in the question of concern. • Do choose a topic that has experienced change to allow for directionality in analysis (for example, investigating a spike in demand for antibiotics despite recommendations against antibiotic use for a current outbreak pathogen). • Do identify the gap that you’re trying to address in the question of concern (e.g. questions are being asked that suggest confusion about recommended public health guidance). • Do ask a question that can be addressed by public health interventions. • Despite widespread availability of [service] and strong recommendations for people in [population/ community] to receive [service], why is [population/community] still not taking up the service? • What questions or concerns do [population] have about [disease] infection and prevention measures? • How did the misinformation narratives about [misinformation] circulate within communities and affect perceptions of [health intervention]? • How well do [population] understand the changing guidance about [health topic]? • What concerns do religious leaders and religious communities have about [health topic]? • How are people hearing, sharing and reacting to [news development about health topic]? • What alternative cures, treatments, or preventives are being discussed, marketed or sold to a target audience? How is this affecting people’s perceptions and health behaviour related to [pathogen/health topic]? • What are the values or reasons that people in [community] are raising in their discussion of [health topic]? • What emotions and reactions are associated with the latest major event related to [health topic] online? • What offline actions in response to [health topic] are being advocated for by [community]? • Which communities are adjacent to the community where the harmful narrative on [topic] is spreading? How are they responding and reacting to it? • How is [narrative/meme/piece of misinformation] changing and evolving over time, and how is it appearing in different communities and what responses does it generate? • Don’t choose very broad questions that are difficult to answer and too general to inform public health action (for example, how many pieces of misinformation are there about bananas causing COVID-19?). • Don’t try to investigate issues related to the reputation of your organization, because this is not the objective of infodemic management (though it is the job of your external communication team). • Don’t attempt to investigate individual behaviours, because infodemic insights look at the population level. Do Don’t “Starter” questions to tailor to your question of concern. Use Template 1 to keep a record of the agreed question of concern that infodemic insights report will help to answer. Findthetemplatehere Template 1 The infodemic insights report will aim to respond to this question of concern [Your question here] _________________________________________________________________________________________? 7You want to identify data sources that will help you respond to your question of concern, and that will help you better understand the information environment that your population of interest lives in, how this population seeks health information and what their health behaviours are. It is also important to assess the quality and provenance of data sources before including them. Aim for more than five regularly recurring data sources in your insights report and at least three data sources that examine a specific population. Finally, choose an appropriate analytic method for each source bearing in mind the objective of the analysis. Remember that data are often collected and held by others. You don’t have to do all the data collections yourself – actually it’s preferred that you use routinely collected data sources that already exist. This is why building a network or team and developing relationships with people who manage or have access to diverse data sources is important. AnnexA8 provides data source evaluation checklist. 2.1 Identifying data sources IDENTIFY AND SELECT DATA SOURCES AND DEVELOP AN ANALYSIS PLAN FOR EACH DATA SOURCE Infodemic insights should not be a quick fix for a lack of field or desk research, but instead complement longer-term and routine data collection, analysis and use. Data sources that you develop through insight development can become part of routine health information systems. STEP 2 • the possibility of human error (e.g. incorrect data entry or code cleaning; bias) • system design flaws in automated data collection or coding • tool and analytical algorithmic limitations (e.g. only able to track users who interact with content but not lurkers; not flagging bot-generated content) • data sharing policy limitations (e.g. anonymized data presented in aggregate with no known denominator; no geolocation data available) • a reliance on automated tools for translation of unstructured data, with few quality checks for multilingual content • access and structural challenges (e.g. not everyone is online; online censorship shaping online behaviour) • a lack of timeliness (e.g. data may be weeks old by the time it becomes available) • the cognitive heuristics of the users that the dataset describes (e.g. digital literacy; social desirability bias) • small samples (sometimes data from very small samples can be valuable if coming from under- represented communities or community leaders) • small absolute changes in trends over time with a large denominator (which results in large changes in relative metrics, needing care to interpret). However, when considering which data to include, keep some of the following limitations in mind: PRO TIP There is no perfect data source and it does not need to be representative of a population to be useful to generate infodemic insights PRO TIP Infodemic insights complement other types of health and data analysis 8 How to build an infodemic insights report in six steps First, ask yourself these key questions when identifying and gathering potential data sources: 1. Who is monitoring or tracking people’s questions, concerns, information voids, narratives and circulating mis- and disinformation? 2. Who is tracking health behaviours and downstream health system effects related to this topic? 3. Where is this data collected from? 4. Where is it managed? 5. How often is it collected? 6. What format is the data collected and managed in? 7. What language is the data in? 8. What are the data and process quality control methods applied to the data source and its data? 9. What data gathering or analytics methods are being used on this dataset? Are they published? 10. How is this data source currently being used? 11. Who is the best point of contact for this dataset? Are they willing or able to share the dataset on a regular basis? 12. What ethics, privacy or data governance considerations are there with this data source? 13. What are the strengths of this data source? 14. What are the limitations of this data source? Consider data generation methods, data quality, data freshness, data reliability, language, representativeness of the population. 15. How relevant or useful will this data source be in answering your infodemic question? 16. What communities are over- or underrepresented in these data? For question 1, consider organizations and data sources including: a. Emergency response (ad hoc set-up for a specific emergency): i. Situational reports ii. Requests for technical assistance iii. Rumour tracking iv. Field reports v. Emergency hotlines vi. Ad hoc networks of responders vii. Outbreak investigation-related datasets viii. Feedback from formal mechanisms (e.g. accountability to affected populations in a humanitarian emergency) b. Health system communications: i. Hotline call log ii. Email/chat queries iii. Press inquiries iv. Patient feedback surveys v. Other formal feedback mechanisms c. Digital environment and user behaviour: i. Website analytics ii. Search trends iii. Social media monitoring iv. Digital infodemic insights analysis (external reports) v. Discussion forums vi. Customer feedback/review sites vii. Other digital data sources d. Society and community: i. Media monitoring ii. Feedback from community events iii. Opinion polls. For question 2, consider data sources including: a. Health system data: i. Use of medicines and diagnostics ii. Service utilization iii. Poison control hotline and surveillance systems on the use of medicines and medical devices iv. System for adverse events reporting (if relevant) v. Cluster surveys, seroprevalence surveys and other epidemiological datasets vi. Health campaign data (e.g. microplans, independent monitoring reports) vii. Health data collected by apps for outbreak investigation or management b. Regulatory data on medicines and medical devices: i. Adverse event monitoring (if relevant) ii. Data supporting marketing authorization applications c. Sociobehavioural studies, including ethnographic studies, community assessments, knowledge, attitude, practice and belief studies, etc. d. Behavioural risk factor surveillance datasets e. Mobility and location data f. Population-based surveys such as multiple indicator cluster surveys, demographic health surveys, health interview surveys, etc. g. Peer-reviewed research h. Grey literature. 9Template 2.1 Mapping available data sources for integrated analysis for infodemic insights Data source Focalpoint Target audienceor population Whatcan thisdatatell us? Format Frequency Strengths Weaknesses Relevance toinfodemic insightsquestion: _____________? Example: Social media dashboard External communications, Ministry of Health Users who interact online with the Ministry of Health’s social channels How people are reacting and sharing Ministry of Health- authored information Digital dashboard that can be exported into Excel Weekly or on-demand • Can be directly accessed by the infodemic insights team • Includes Twitter, Facebook and TikTok analytic data • Limited data available on Instagram or LinkedIn • Cannot customize dashboard with public health terms • Only limited content has been posted on the channels related to the topic of concern and is therefore harder to analyse than posts on all the other topics that appear on the channel High relevance Next, fill out Template 2.1 to the best of your ability to create a landscape of all possible datasets. See following with guidance to the type of material to input, and findthetemplatehere E X A M P L E Step 2: Identify and select data sources and develop an analysis plan for each data source 10 How to build an infodemic insights report in six steps Then, sit with your insights team to discuss the following questions: 1. What data sources are the most rigorous and reliable and should be included? (e.g. nationally representative telephone surveys versus a convenience-based online survey, or national hotline call logs versus online Q&A posts about the health topic on a discussion website). 2. Which least reliable data sources should be excluded? 3. Which data sources may be repetitive of other data sources and could therefore be excluded without losing information from the analysis? 4. What are one-off data sources that could be considered? (e.g. field reports, annual surveys, an in-depth social listening report from a social media platform). 5. Which data sources may not be timely enough to yield useful insights? 6. Which data sources track behaviour versus just describing programme performance? (e.g. analysing the click-through rate, bounce- rate and navigation paths on the Ministry of Health website, versus how fast the webpage loads and which websites have linked to it). 7. Which data sources are the hardest to obtain? Which are the easiest? (for operational, technical or political reasons). 8. Do you have enough data sources of sufficient quality that directly or indirectly measure the community of interest? Aim for at least three data sources to use. If you have fewer than three, either look for additional data sources or collect more data. Decideasateamwhichdatasourcestoinclude andwhichtoexclude. 2.2 Assess and select data sources Choosing sources carefully will reduce the analytic burden. However, it is important to include enough data sources that enable common trends and themes to be identified, which will reinforce the validity of the results, especially if they are different data sources collecting information focused on the same population. For example, you may include in your analysis a knowledge, attitude and practices (KAP) survey, a nationally representative poll from a reputable polling agency, and an analysis of Instagram reactions and comments from parents about routine immunization. All of these data sources indicate that among caregivers there is lower risk perceptions for measles infection than for other vaccine-preventable diseases, such as polio and pertussis. Therefore, you would have a higher degree of confidence in reaching a conclusion that was supported by multiple reputable data sources, rather than just one. 1. Who is included in the process of making meaning in this dataset, especially when the data is related to a group not represented on the team? 2. What tradeoffs about equity are you making when determining data sources to include, and how can you invite feedback about them? 3. How are you centering the voices and perspectives of individuals and communities included in the data sources? 4. What types of equity are you prioritizing, making explicit, and who decides? 5. What assumptions and values are you using to determine the importance of the data sources? 6. What context about systems and power dynamics are you considering when identifying data sources? Consider questions such as: PRO TIP One of the biggest challenges an infodemic manager might face is access to too much information PRO TIP When reviewing, selecting and analysing data sources, consider how equity considerations are reflected in your decisions 11 Once you have decided on the data sources, fill out Template2.2 to help you develop analysis plans. See following page with guidance to the type of material to input, and findthetemplatehere Aim to include more than five regularly recurring data sources in your insights report. As the insights unit gains more human resources and analytic capacity, more sources and more complex analytic approaches can be added. You should routinely revisit this template to update your sources, and if resources permit, consider targeted investment in developing a routine data source that would cover a blind spot in your coverage. For each data source, an analytic method will need to be identified. Analytic methods are diverse and must be selected to fit the type of data source and the objective of the analysis. They can include: • quantitative summary statistics (e.g. a frequency table or word cloud of common words used in hotline calls) • correlations or simple statistical models (most often applicable to health information, health programme, facility and epidemiological data sources) • a qualitative social network analysis • a qualitative coding and thematic analysis (looking for repetition, indigenous categories and missingness) • a quantitative analysis using machine learning/ artificial intelligence (AI) (e.g. sentiment analysis, network analysis, automated classification by topic or linguistic patterns) • a taxonomy-based summary of computationally identified narratives (e.g. using a social listening taxonomy for linguistic analysis of social media data). Each analytic method might require in-depth knowledge of a specific topic or field. Therefore, it is important to match the skills you have on your team with the analytic methods that you choose. 2.3 Develop an analysis plan for each data source Your institution may have guidance on the ethical use of social media monitoring data and other identifiable information obtained. However, you may need to create your own if it doesn’t already exist. You may have access to personally identifiable information from data sources, and it is your responsibility to ensure that no such individual-level information appears in a public report or is reverse-identifiable. This is because in most cases you will be using a post or piece of content as an illustrative example of a larger narrative trend, rather than suggesting that the individual is responsible for this negative narrative. If you choose to include screenshots or content from individuals, then you can blur out usernames and images, or paraphrase a common sentiment so that the exact wording can’t be traced back to the owner. PRO TIP Consider how to anonymize data sources in reporting Step 2: Identify and select data sources and develop an analysis plan for each data source 12 How to build an infodemic insights report in six steps Template 2.2 Developing analysis plans for infodemic data sources Datasource Analyticmethod(withinclusion/exclusioncriteria) Variables(if applicable) Indicators(ifapplicable) Software/ skillsrequired toanalyse data Assigned analyst Actionstoreduce bias Expectedtime commitment tocomplete analysis Example: Health hotline call logs, exported in Excel on a monthly basis, anonymized • Descriptive statistics by the categories used at the call centre and of content/ usage data by hotline operators • If call logs contain more than 1000 queries, prioritize the analysis of existing indicators over a custom analysis of other variables • Exclude queries that come from health workers • Topics that there are no prepared responses for • Calls that have been escalated to a second-line subject-matter expert • Number of times a query is received on a particular topic • Number of new topics generated in the last month • Number of times that a query was logged as being related to a piece of misinformation • Descriptive statistics • Thematic analysis • Can be done in Excel Tandava • Analytical plan reviewed by a second person • Native speakers analyse specific language content 1 day 13 CONDUCT AN INTEGRATED ANALYSIS ACROSS DATA SOURCES STEP 3 Once you have chosen your question of concern (recorded in Template 1), selected your data sources and drawn up your analysis plans (recorded in Template 2.2), it is time to plan how these data analyses will link to public health action. Infodemic analyses, like any analysis of health information, are designed to identify problems or health risks to help formulate solutions and improve health service delivery or health outcomes. The analysis plan for each data source will help to yield intelligence, which when analysed 3.1 Overview of steps in an integrated analysis alongside intelligence from the other data sources, will yield integrated insights, forming the basis of the recommendations to be used in the insights report (see Figure 1). When compiling intelligence across multiple data sources to derive insights, a risk assessment should be used to determine how large a risk the subject of a particular insight is to promoting a specific health behaviour or outcome. In this section of Step3, you are developing analysis plans (blue boxes in Figure1) that will be used in the next section of this step to lead to intelligence. Figure 1 Moving from data source, to intelligence, insights and recommendations to form an infodemic insights report Datasource1 Datasource2 Datasource3 Datasource4 Integratedanalysis Analysisplan1 Analysisplan2 Analysisplan3 Analysisplan4 RecommendationsInsights Intelligence Intelligence Intelligence Intelligence Riskassessment topublichealth basedonmatrix 14 How to build an infodemic insights report in six steps At the end of Step 3, you should be able to make recommendations for strategies to rapidly respond to identified issues that have the highest potential to cause harm to public health. Table 1 gives an example of how several pieces of intelligence from various data sources, coupled with a risk assessment, can lead to insight and recommendations. Table 1 Example Intelligence combining with a risk assessment (including spread and velocity) can determine an insight, followed by recommendations Intelligence Riskassessment Insight Recommendation A rumour log shows widespread belief that COVID-19 can be cured with antibiotics. Moderate risk, because it appears in two other data sources (broad spread) and the number of mentions has increased dramatically since the last report (velocity is increasing). Some consumers believe that COVID-19 can be cured with antibiotics and have easily obtained prescriptions from pharmacists. • Address misconception with updated FAQs for health workers. • Create debunk content and share it with factchecking networks. • Partner with the national pharmacists association to improve counselling for customers on antibiotic use for COVID-19 treatment. You will be actively assessing risk of identified intelligence and insights to public health and well- being. Therefore, before you start any analysis, you will need to define your risk assessment criteria. Before conducting an integrated analysis in the context of public health, assessing and mitigating risk is a critical activity to protect population health. Public health mitigation or prevention measures are prioritized based on risk assessments that take into account the following: • The route of transmission or spread of the hazard • Potential adverse effects • The safety level for each exposure • The exposure points and levels • Potential adverse outcomes. Epidemiologically (and infodemiologically), when assessing risk you also consider aspects of: • Timing • Place (platform) • Population (community). “Hazards” in an infodemic context usually appear as narratives. Note that information or misinformation itself is not hazardous but it can contribute to harm when it becomes part of a larger narrative that leverages values or emotions, is shared widely or responded to by a specific community. It then has a high potential for being “sticky”. Stickiness, or the ability of a narrative or piece of content to be recalled by someone exposed to it, makes it more likely that a person will share the narrative, content algorithms will amplify it online, and that it will affect the person’s health perceptions and potential behaviour. Highly sticky content that reinforces an existing narrative is likely to generate reactions and variations (e.g. memes, content shared on new platforms) and be reflected in the news cycles. The level of risk depends on: • The narrative’s timing: Is it old? Is it new? It is gaining traction and spreading more quickly? • Which platforms it is appearing on: What social or traditional media channels? Where is it being repeated or amplified or changing? Is it jumping channels? • The community it is affecting: Do you have any data on how the community is responding? How are the most vulnerable groups being discussed or reacting to the narrative? 3.2 Defining risk assessment criteria 15 Furtherconsiderationsonnarrative formation,spreadandimplications forrisk Narratives come in many shapes and sizes, and depending on your data sources, you can more clearly define where they emerge, how they grow and change and which platforms and communities they affect. It’s important to remember that a narrative that is narrow and only circulates in one community, especially a vulnerable one, can be as harmful as a widely circulating narrative. Therefore, avoid using absolute measures and metrics such as volume to dictate which narratives need to be prioritized for action. For example, if a rumour is spread that refugees are not eligible for vaccination against a disease during an outbreak, even though public health policies encourage everyone to get vaccinated regardless of refugee status, this can impact this refugee group’s willingness to be vaccinated. Related issues of stigma, fear about legal jeopardy and a lack of understanding of the rights of refugees in the host community, and a lack of information available in the refugees’ languages can all reinforce the potential harms of this narrative. The level of risk is also linked to the epidemiological characteristics of the outbreak or emergency or health issue in question. For example, very few health topics affect “the general public” but rather affect very specific groups in specific locations. Diseases often carry social characteristics and may come with historical baggage of how this disease or emergency may have been addressed previously. Depending on which community you’re focusing on for your research question, you will define a level of risk based on these factors. For example, stigma is associated more closely with some diseases than others, such as for sexually transmitted infections or for pathogens like Ebola, COVID-19, Marburg and mpox that are deadly and/or highly transmissible. Other diseases may affect already vulnerable populations such as Zika affecting pregnant women, or cholera occurring during an earthquake response, and can also be stigmatizing. Therefore, you would assess narratives that discuss or express stigma in relation to a particular population as higher risk, even if the overall volume of conversation is not high. Infodemic management should be implemented through public health structures and therefore, needs to serve emergency response and programme objectives which will look different from place to place and from health topic to health topic. For example, if you’re developing a risk matrix for a local health department, your risk assessment will look different from a colleague’s national-level perspective at the Ministry of Health. Different countries will assign a different level of risk to specific narratives or concerns, recognizing that for some issues there may be a large gap between the true public health threat and what is generating lots of conversations, concerns and press coverage that indicate a high degree of risk perception that may be unwarranted. For example, in the 2014–2016 West Africa Ebola outbreak, there was a lot of social media discussion in Europe and North America about the potential spread of Ebola within national borders, even though the epidemiological risk was extremely low. If you compared the volume of social media conversation there with that in the countries experiencing the outbreak, which had lower Internet penetration and fewer social media conversations, someone could reach the incorrect conclusion that Ebola was a high risk for international spread and that the level of anxiety detected on social media North America and Europe was proportional to the threat of infection. Just because a lot of people are talking about a public health risk, doesn’t mean it’s a true reflection of public health risk. In such a situation, it’s important to correct the high degree of risk perception that is unwarranted, because if that is left unaddressed, this can encourage the formation of misinformation and a perception of inaction by health authorities. Additionally, stigmatization and victim blaming may become part of the narrative, as it did in the above example for travellers from the African continent and aid workers who returned home with possible Ebola infection. Step 3: Conduct an integrated analysis across data sources 16 How to build an infodemic insights report in six steps Customizingariskassessmentmatrixfor yourcontext Every country and health authority will have a different set of programme objectives and priorities and different tolerances for risk. This is informed by health system and emergency response capacity, previous experience with similar emergencies, and defined gaps where you suspect that a narrative could easily form (e.g. the government responded poorly in a previous emergency and this resulted in a major loss of trust in the authorities, and the Ministry of Health is sensitive to discussions related to trust and linking previous emergency responses to this one). Below are the four broad categories that make up a risk assessment matrix and that can be tailored to your needs: The narrative does not apply to your population, does not have widespread circulation, does not provoke strong or emotional reactions, does not lead to a spread of questions, concerns or confusion on this topic, and there is no evidence that it is negatively impacting health behaviour. • Example: People are discussing the health guidance from another country for a rare mosquito-borne virus that doesn’t exist in your country. The narrative has some application to your population, has some circulation in your country including modest amplification or evidence of message adaptation to local worries or concerns, there is some evidence that it is provoking strong or emotional reactions, triggering some questions, concerns or confusion on this topic, and there is anecdotal evidence that it is negatively impacting health behaviour. • Example: A candidate running in a provincial election claims that masks reduce oxygen to the brain and mandatory masking in schools in the past year had led to lower test scores in children in schools. Her comments are amplified on social media and searches for a relationship between mask wearing and test scores increase. However, today, 92 % of schools do not have mandatory masking requirements in place, but health authorities are concerned about resistance to masking in the approaching winter season where new variants of COVID-19 may infect school children. Although this tends to be rarer than concerning or negative narratives, positive narratives can be identified and utilized as a protective factor, where pro-healthy behaviours are encouraged, or trust in health systems or the government is reinforced, or adherence to health guidance is encouraged. • Example: The government’s mental health support campaign to connect young people with older citizens in nursing homes to prevent pandemic isolation is chronicled in an award- winning documentary. This is widely discussed online and inspires different municipalities to adapt similar programmes. Theseriskassessmentmatricesshouldbe definedinrelationtoyourinfodemicinsights questionofconcernandareaofhealth,aswellas theappetiteforriskthatyourorganizationhas, becauseeachrisklevelwillleadtoadifferentset ofrecommendationsorstrategies. The narrative is very relevant to your population, has widespread circulation in your country including amplification or evidence of message adaptation to local worries or concerns in multiple communities, there is evidence that it is triggering strong or emotional reactions, many questions, concerns or a lot of confusion on this topic, and there is systematic evidence that it is negatively impacting health behaviour. • Example: Fertility concerns are expressed around pandemic flu vaccination after a new scientific publication confirms that flu vaccines can affect women’s menstrual cycles. The article gains traction in social media. One family is suing the government for a miscarriage suffered after vaccination. This is widely covered in the press, and a prominent obstetrician advises delaying vaccination until after birth. Vaccination coverage among pregnant women is extremely low.Lowrisk Moderaterisk Positivesentiment Highrisk 17 Highly sticky narratives tend to jump audiences, languages, communities and platforms. And sometimes “zombie” narratives may surface reusing old visuals and previously circulating misinformation and adapting it to a new topic. The reason why common misinformation tropes repeat and “go viral” is because they exploit people’s closely held values and exacerbate their concerns. For example, a pernicious narrative that appears with every vaccination effort is that vaccines cause infertility or affect reproductive health. This is a narrative that has appeared for polio, Ebola, COVID-19, measles, HPV, influenza and other routine vaccines. The reason it keeps appearing is because linking vaccination to something so personal and integral to someone’s identity and sense of family and community will always generate a reaction. When such a narrative is detected, it is helpful to provide context for how and where this narrative appeared, and to note that it has appeared before and been proven multiple times to be false. Refer to experienced response or programme colleagues who have encountered this narrative before and addressed it in specific communities when you are analysing insights and developing recommendations. PRO TIP Why high-risk narratives repeat themselves across outbreaks and emergencies and what to do about it Template 3.1 Infodemic risk matrix with definition of risk criteria and levels for your question of concern and context Risklevel Definition Example Typeofrecommendation/ strategy/actionthisrisk levelwouldprompt Low risk Moderate risk High risk Positive sentiment This risk assessment matrix can be updated over time as you identify new risks or actions that can or should be taken. In the assessment of a topic, consider the spread and velocity of specific narratives, and whether the topic is emerging, persistent, re-emerging or declining. Depending on the velocity and spread, narratives may disappear or resurface, or change, which will be captured after several insights reports have been developed over time. Fill out Template 3.1 before moving to the next step. Findthetemplatehere As mentioned, before starting an analysis, define the risk assessment criteria for low, moderate and high risk and positive sentiment for your context so that you can generate reproducible insights and prioritize actions. Remember that high-risk narratives often require more swift and far-reaching action and low-risk narratives require little or no response. Step 3: Conduct an integrated analysis across data sources 18 How to build an infodemic insights report in six steps Recall Figure 1, the visual depiction of how data sources, coupled with analysis plans, lead to individual pieces of intelligence, which, analysed together, form insights that lead to recommendations. By now, you will have identified data sources and their associated analysis plans, and defined the risk matrix for your question of concern. In this section, you will perform the integrated analysis (yellow box in Figure 1) for the intelligence that will lead to insights. It is important to remember that to get the most useful insights from the infodemic analysis, you need to use qualitativemethods for integrated analysis that are driven by human analysts. Even though datasets such as social media monitoring reports or surveys report quantitative data, they are analysed in the context of a specific infodemic-related question alongside several other data sources. This is why most infodemic management integrated analyses will not attempt to establish associations, confidence intervals or use statistical testing. This type of additional qualitative analysis takes time, which you may not have in an emergency, but mature infodemic management systems that have access to robust data sources and sufficient people and time can carry out more sophisticated analyses. However, your analysis should indicate the directionof change: whether it was emerging, persistent, re-emerging or declining. These directionality trends can only be detected if you track the same issue over time – in your first insights report you will not be able to indicate the direction of change because you have no baseline to compare it with. Avoid reporting absolute numbers or defining how often a term is used, because this cannot be translated into an actionable recommendation for public health programming. Instead, focus onestablishingandanalysingtheformation, evolutionandvelocityofspecificthemes ornarrativesorinformationvoids.Couple themwithariskassessmentandidentify towhatdegreethisposesathreattopublic health. For example, counting every single piece of misinformation that uses the hashtag #diedsuddenly in English isn’t going to be very helpful for answering a public health question. A more useful alternative could be to understand which social media influencers in your country have used the hashtag, how they discussed it, which communities in your country have shared it within their own networks, and whether there was any national press coverage related to #diedsuddenly. 3.3 Analyse each data source to derive intelligence These are often provided through commercial platforms used for social media analyses and were originally intended to understand positive or negative emotions, keywords and reactions associated with a brand or company. These types of binary classifications of expressions are not actionable or accurate, because sentiment analysis algorithms have difficulty accurately labeling sarcasm, humour or other complex emotions, or picking up expressions from emojis and visual content. When sentiment analyses are used in reports, the reader often pays attention to the percentage of negative sentiment, suggesting that an infodemic manager’s job is to improve the positivity score on a particular health topic. This propagates the incorrect assumption that an infodemic manager’s job is to make the internet a sunny, happy place where no one expresses any discontent. In fact, an infodemic manager’s job is to accurately relay the entire range of emotions and reactions a community is expressing in relation to a health topic, because emotions affect behaviour. Automated sentiment analysis is notoriously unreliable, and can give a false sense of security, thinking that we know how a group of people feel based on a small social media dataset derived from a platform that is not representative of the population. Language and expression of emotion are far more complex. If you choose to investigate emotion and how it surfaces in narratives, consider using Plutchick’s wheel of emotions and participatory methods for sentiment analysis. PRO TIP Use automated sentiment analyses with caution 19 Template 3.2 Compiling data source intelligence guided by analysis plan for each data source Datasource Whatquestions dopeople have? What concerns dopeople have? What information voidsare there? (people can’tfind information theyare lookingfor) What narrativesare circulating onthistopic? (explanations for motivations andreasons forcurrent state) Whatmis- and disinformation iscirculating onthistopic? Whatother beliefsor behaviours dopeople haveonthis topic? Other relevant observations Summarizethetheme/ gistinonesentence (orseveralsentences iftherearemultiple themes) Directionality –isthisan emerging, persistent, re-emerging ordeclining topic?(add designation pertheme) Risk level Example: Videos related to home remedies for treating mpox symptoms, and comments shared on a prominent subReddit for the LGBTQ+ community • Where on the body do rashes associated with mpox show up? • What home remedies are recommended by medical professionals? • What home remedies are recommended by community members? • Lack of access to vaccine of choice • Stigma for seeking treatment • Worry about revealing status to close contacts • Definition of what a close contact is • Who to contact to learn if they should get vaccinated • Available mpox vaccines are not safe • Vaccines are more painful than the rash • Health system not meeting needs of LGBTQ+ community to get care. • Mpox is a side-effect of COVID-19 vaccines • Putting chamomile tea bags on rashes can cure them. • People report challenges accessing doctors who can diagnose them and provide care. • People in the HIV+ community are sharing their home remedies on mpox- related threads. • There is a perceived lack of access to medical care and vaccines for the LGBTQ+ community which is driving conversations about home remedies for treatment. • HIV-positive community members are sharing coping and home- remedy strategies for people with suspected mpox infection. Emerging Emerging Moderate Once you have completed the analysis of each data source from Template 2.2, consolidate the analyses into Template 3.2, leaving fields blank if not applicable. This is how you begin to integrate your analyses. Findthetemplatehere Step 3: Conduct an integrated analysis across data sources 20 How to build an infodemic insights report in six steps Meet as a team and discuss the compiled Template 3.2. The last three columns of the template relate to summarizing the theme or gist, determining the velocity of the narrative and assigning a level of risk will probably be the most difficult and may require discussion with your group and consultation with a subject-matter expert. When filling out the themes column, consider which themes are most important (e.g. that appear the most frequently, cause the most concern, or were previously not detected) and which may be less important (e.g. probably have a low impact on health behaviour, have only been detected in a less reliable source or smaller sample size, provide out-dated information, the insights are not actionable by the health authority). Each of these themes for each data source is your intelligence. Determining the velocity of the narrative and assigning a level of risk for the last column will probably be the most difficult and may require discussion within your team and consultation with a subject-matter expert. From the information in Template 3.2, discuss with your team and choose the top intelligence that best represents the overall picture of people’s perceptions, attitudes, discussions, behaviour and reactions and the narrative related to the specific health topic and your question of concern. This is your insight. For additional information on team meetings see A9 How to run a cross-disciplinary infodemic insights meeting. Next, assign a risk level (low, moderate, high or positive sentiment) to each insight. This can be compiled using Template 3.3. Findthetemplatehere 3.4 Integrate intelligence from data sources into insights Template 3.3 Identifying major themes across data sources and intelligence and prioritizing by risk Majorthemes Themeemergedfromwhatdatasources? Overall,whatlevel ofriskshouldthis themebeassigned? Example: LGBTQ+ community is attempting to cope with a perceived lack of access to health services or effective preventive interventions by using home-based remedies, some of which are not recommended and may be harmful (emerging) Reddit thread analysis Media monitoring report YouTube video analysis of the 25 most popular videos mentioning mpox prevention and cures National LGBTQ+ health hotline call logs Community feedback report from consortium of organizations serving the LGBTQ+ community, based on six listening sessions High It is important to begin with an inductive approach, developing themes from evidence emerging from the data and creating or adjusting a framework that exists to organize them, rather than starting with a framework to define what you should look for and then deductively “fitting in” what themes you found to a pre-existing model that was not built for infodemic insights. Otherwise you may miss themes or discount topics that don’t “fit”, even if they are of concern to a large audience and should be addressed. Remember that your data sources and risk assessments will constantly be updated, and conversations you’re tracking will always be shifting. This means that your codebook will expand, so you may need to update the taxonomies you’ve developed. For more information on taxonomies see A10 How to develop a taxonomy and A11 Sample taxonomy for generating infodemic insights PRO TIP Develop themes based on evidence you have identified 21 If you repeat an insights report with a set of routinely available data sources and similar infodemic questions of concern, you can develop a codebook of re-emerging themes and a framework for organizing themes. Depending on the health topic, this framework may use behavioural, epidemiological or other models used by the health programme to define the parameters. For example: • To understand the drivers of vaccination you may include thematic buckets for “practical factors” such as accessibility and vaccine supply; “thinking and feeling” such as risk perception of disease, confidence and safety; and “socio-behavioural factors” such as social support and health care provider recommendations, using the “Behavioural and social drivers of vaccination” model from WHO. • If you’re investigating an infodemic accompanying an outbreak, you may group together themes related to “community spread” or “hospitalization” or “treatments”, based on the COVID-19, mpox and respiratory pathogen social listening taxonomies from WHO. PRO TIP More insights can be derived from tracking them over time DEVELOP STRATEGIES AND RECOMMENDATIONS STEP 4 When moving from insights to recommendations, the next step is to consider what strategies and recommendations can be paired with high- priority insights. The types of recommendations you can make will be heavily shaped by your institutional capacity and objectives, and the strengths of your partnerships. An insights report should be useful to multiple stakeholders, who can see their role in the report and recommendations. For example, very often in an emergency response you would work closely with the communications team, the data team, the health operations team, the health information team and external stakeholders such as civil society, the media, fact checkers, professional organizations, online self-organized communities and technology platforms. Consider how recommendations can be taken up and acted on by stakeholders, even if you and your team can’t implement the recommendation yourselves. What type of recommendations, strategies or opportunities for action may be suggested for each of the major themes from Template 3.3? • Address health service supply issues • Address health service access issues • Address health service delivery issues • Improve coordination and governance • Clarify health guidance or policies • Improve communication • Improve community engagement • Reduce harmful actions • Address a structural issue with the information environment • Conduct more research in the area. 22 How to build an infodemic insights report in six steps Ideally, you will want to prioritize and recommend strategies that are immediately implementable by stakeholders. Rapidly addressing infodemic issues you have identified can prevent the growth of narratives or misinformation that can feed population anxiety, dampen risk perception, lower compliance with health guidance and lead to behaviours that don’t protect health. Avoid trying to make recommendations that go beyond your organizational mandate or are not informed by an expert that you have consulted. For example, the health authority does not regulate the Internet and recommendations to force take-downs of misinformation by technology companies will probably be ineffective without a government mechanism or policy to enforce this or could lead to unintended harm and misuse. When developing recommendations, consider that it is unlikely that you can influence large political, policy or social challenges that pre-date the emergency. For example, the COVID-19 pandemic highlighted major gaps in health equity in different populations, but actions based on an infodemic insights report is unlikely to fix health equity challenges that persisted before the pandemic. However, awareness of them can inform your analysis and prioritization and prevent future unintended harm from infodemic management actions on this population. To understand how recommendations are applied to insights, Table 2 gives examples of high- priority infodemic insights from analysing the COVID-19 epidemic from the point of view of a national Ministry of Health. The table includes sample recommendations and their type. Table 2 Examples of high-priority infodemic insights from analysing the COVID-19 pandemic and example recommendations Infodemicinsight Recommendations Typeofrecommendation Emerging: Community X is reporting a lack of availability of COVID-19 rapid tests and expressing frustration about their inability to follow health guidance that people should test before travelling. • Work with colleagues in the supply chain to address supply and distribution challenges. Address health service supply issues. • Communicate with Community X about what is being done to address the supply issue. Improve communication. • Address confusion or concern about the guidance by adding to the Q&A about what to do if rapid tests are not available and someone is planning on travelling. Clarify health guidance or policies. Re-emerging: Family members express frustration that although elderly people were prioritized for COVID-19 vaccination, they face access barriers and families are asking for more outreach or door-to- door services. • Work with the emergency response team focused on the elderly to review current efforts and promote mobile vaccination and other vaccination opportunities that reach the elderly. Address health service access issues. • Develop a new webpage providing resources for families who want to have older family members vaccinated, detailing how they can help and what community resources are available. Improve communication. • Host a webinar for community organizations and families who want to support elderly vaccination. Improve community engagement. 23 Table 2 Examples of high-priority infodemic insights from analysing the COVID-19 epidemic and example recommendations (continued) Infodemicinsight Recommendations Typeofrecommendation Emerging: Misinformation and concerns about the Ministry of Health partnering with the military to bring COVID-19 medical supplies, including vaccines, to local communities, spurring suspicion and loss of trust, especially among minority groups. • Clarify the role of the military in providing support for emergency response in public communication. Clarify health guidance or policies. • Add the clarification about military engagement to the FAQ webpage. Improve communication. • Identify what specific communities these concerns are being expressed in. Conduct more research in the area. Persistent: Conflicting guidance from the Ministry of Health, Ministry of Transportation and the Ministry of Tourism on travel restrictions, vaccination and testing requirements for tourists, business travellers and locals is causing confusion and frustration with the government response. • Suggest that all the relevant ministries convene to discuss the guidance and implementation of more harmonized information for the public. Improve coordination and governance. • Update the webpages and social media content for tourists, business travellers and locals to reduce confusion. Improve communication. • Work with pharmacies and local health care associations to provide testing and vaccination services at travel hubs. Address health service delivery issues. Emerging: High profile cases of health workers being harassed, threatened or attacked delivering COVID-19 health care are being discussed and applauded online as demanding medical freedom by certain groups. • Engage medical and health associations to understand the full impact of these types of threats against health workers and what local strategies have been implemented. Improve community engagement. • Work with technology platforms to heavily moderate content advocating violence against health workers. Improve coordination and governance. • Coordinate with law enforcement colleagues to communicate to the public what steps are being taken to protect health workers and the consequences of threats and acts of violence against them. Reduce harmful actions. Emerging: An antivaccine documentary questioning the safety of COVID-19 vaccines has received over a million views and is being discussed on multiple platforms, including being translated into other languages by users, alongside encouragement not to get the COVID-19 vaccine (#notaguineapig). • Work with factcheckers to address vaccine misinformation claims and provide an updated myth buster page on the Ministry of Health website for factchecking organizations and media to link to. Address a structural issue with the information environment. • Further analyse the communities adjacent to those where this documentary is circulating, particularly vulnerable communities that speak the languages the documentary has been translated into. Conduct further research. • Update the metadata and content of vaccine safety webpages to better align with Google searches to make credible content easier to find. Address a structural issue with the information environment. Step 4: Develop strategies and recommendations 24 How to build an infodemic insights report in six steps Resist the urge to default to communications-related recommendations, which do not address the multitude of reasons that people may not be following health guidance. For example, recommending more messages to be shared doesn’t address the root of the problem of a community’s feeling of exclusion and anger when they report supply and access issues and poor treatment by health workers. Another example: if rampant misinformation is circulating in a minority community that distrusts the government, more messages from the government or debunking will probably not address that lack of trust and weak community ties. This is why it is also important to provide evidence and recommendations that would improve health service delivery, seek out nontraditional communities to engage with, or improve policy and guidance, even if this is not the mandate of your team to consider or implement. You want to make recommendations based on your evidence and also recommend actions that are within the mandate of the programme you’re supporting. You want to make recommendations that can be accomplished in the short and medium term. They need to be formulated with strong knowledge of how the public health programme works. It is not advisable to develop recommendations without involving a subject matter expert on that topic. Nor is it advisable to write an insights report that only delivers a description of the issue (“admiring the problem”) and doesn’t provide any concrete recommendations and strategies to address infodemic issues. If traditional emergency response or public communications actions were sufficient to mitigate the infodemic, there would be no need for an infodemic insights report. PRO TIP High-quality communication is necessary but not sufficient for managing infodemics PRO TIP When developing recommendations, it’s important to have the right people involved 25 Insights reports are the single most important product infodemic managers generate. They are a synthesis of what has been learned from evaluating data sources, including intelligence, the implications this has for a health or emergency programme, and recommendations for actions or strategies. Insights reports should be generated with no more than two-week-old data (this can be longer in “peace time”), because narratives shift, as does the epidemiology of an outbreak in a public health response, and decisions must be based on relevant data. Users of insights reports will be pressed for time, doing too much with not enough resources, so you will need to prioritize for them. This means focusing on the most concerning narratives, especially those flagged as high risk, and recommending strategies that can be rapidly implemented – think low-hanging fruit instead of long-term massive systemic change. This is especially true in emergencies where time is of the essence to protect lives and welfare. The report also needs to be presented in a format that will be quickly actionable – long reports are not desirable because they are time consuming to produce, clear and design, and even more time consuming to read and use. Shorter and more concise is better. An insights report can take many forms: PowerPoint slides, an infographic, a document or webpage. However, regardless of the format, it is important to tailor it to your audience – the people meant to take action with the infodemic insights reported, which will often be people within your own organization. Understanding how and when target audiences would prefer to receive infodemic insights and in what format will aid in its uptake and dissemination. Remember that you may need to educate target audiences, especially if some strategies in the report are novel (e.g. social inoculation) or require close collaboration (e.g. debunking with factchecking organizations). Avoid technical details and aim for short sentences. The key components of an insights report are (see also Template 4.1): • Executivesummary(bottomlineupfront) o A three-sentence summary of the major themes, their risk assessment and which actions to address them should be prioritized. • Explanationofpurposeoftheinsightsreport o A one-sentence descriptor of the report’s purpose and approach and provide a point of contact. • Majorthemeswithrecommendations foraction o Several sentences explaining the gist of the important themes or intelligence. Include directionality and the risk assessment for each theme. Explain: • which community or stakeholder is involved; • the relevant platforms/data sources; • an overview of the relevant questions, concerns, information voids, narratives and circulating mis- and disinformation; • points of confusion or gaps between official guidance and stakeholder or community actions and other reactions. o An illustrative, anonymized example, as appropriate. o Recommendations for action: What communications, programmatic, socio- behavioural, policy, or other structural strategies would you recommend to address DEVELOP AN INFODEMIC INSIGHTS REPORT STEP 5 26 How to build an infodemic insights report in six steps this issue in the short and medium term that are feasible and evidence-informed? Which stakeholders should be involved? Be specific! o When possible the recommendations should include the following: • Summary of the evidence supporting the recommendation (1-2 sentences). • Suggestions of different ways to implement the recommendation. • Indication of who is supposed to take action on the recommendation. • Estimated costs of implementing the recommendation • Specification of the benefits of implementing the recommendations • Barriers to implementing the recommendation and how to mitigate them. • Generally, provide enough detail so that an individual can read it and know exactly what they need to do ** next. They should be able to pick up the phone and take action. o Additional considerations: • Consider a formal action planning process for supporting changes. • Consider using a participatory data analysis process/activity with intended users and decision makers. • Persistentthemes o Provide updates on any themes mentioned in previous reports that have evolved or require additional attention or actions. o Provide updated recommendations. • Methods o Several sentences briefly describing the methods. o A narrative or table with an overview of the data sources, any relevant links, frequency, and other important information about them (e.g. language if working in multiple languages). o An explanation of the risk matrix. o A statement of the limitations. o If the methods or report are published online, date and post a permalink. Other components that might appear in an insights report include supporting materials, points of contact or links for addressing specific infodemic issues. When developing the narrative for the report, begin with an overview of the themes or intelligence emerging from the infodemic data sources (e.g. hotline call logs, social listening report, knowledge, attitude, practice and belief survey). Additional context may be provided by highlighting events, newsworthy developments or milestones that may have affected people’s perceptions and discussions. Then, develop recommendations or strategies for action. For editorial support, see A12 Resources on developing a style guide and aiming for inclusive language. Often, insights reports are primarily designed for operational internal use by a health system or programme. However, if it is to be released to the public, it may need to go through multiple rounds of clearance, such as scientific clearance, communications clearance, and clearance from the programme/emergency response leadership, depending on the circulation of the report. Clearance can dramatically slow down publication and dissemination of a report (see Step 6), especially if there are additional steps required to design, lay out and publish the report or upload to a website or disseminate to different networks. Because narratives, questions and concerns change over time in populations, it is important that insights are immediately used for action. Work with your clearance chain to understand the need for speedy action through to publication and optimize it. Ensuring that the reports are effectively used means educating everyone involved in each step of the process ahead of time. These are not epidemiological or social media monitoring reports, and therefore may require more explanation, particularly of how digital environments function and how they can affect people’s perceptions, attitudes and behaviours related to specific health topics. Consider holding an “infodemic management 101” briefing for a wider audience before the first report is issued. PRO TIP Infodemic insights reports are novel publications and may not have been encountered before by people within health programmes or emergency response 27 Template 5.1 Template for an insights report Use this template to plan out the components of your insights report in narrative form and tailor it to your organizational and operational needs, as well as your formatting for distribution. Findthetemplatehere Sectionofreport Whattoinclude Headlineinformation Executivesummary • Provide the “bottom line up front” by including a three-sentence summary of the major themes, their risk assessment and which actions to address them should be prioritized. Explanationofthe purposeofthe insightsreport • A one-sentence descriptor of the report’s purpose and approach and provide a point of contact. Majorthemeswithrecommendationsforaction Theme#1with recommendations foraction • A one-sentence summary of the theme. • Several sentences explaining the gist of the important themes or intelligence. Include directionality and the risk assessment for each theme. Explain: o which community or stakeholder is involved; o the relevant platforms/ data source; o an overview of the relevant questions, concerns, information voids, narratives and circulating mis- and disinformation; o points of confusion or gaps between official guidance and stakeholder or community actions and other reactions. • An illustrative, anonymized example, as appropriate. • Recommendations for action: What communications, programmatic, sociobehavioural, policy, or other structural strategies would you recommend to address this issue in the short and medium term that are feasible and evidence-informed? What stakeholders should be involved? Theme#2with recommendations foraction • A one-sentence summary of the theme. • Several sentences explaining the gist of the important themes or intelligence. Include directionality and the risk assessment for each theme. Explain: o which community or stakeholder is involved; o the relevant platforms/ data source; o an overview of the relevant questions, concerns, information voids, narratives and circulating mis- and disinformation; o points of confusion or gaps between official guidance and stakeholder or community actions and other reactions. • An illustrative, anonymized example, as appropriate. • Recommendations for action: What communications, programmatic, sociobehavioural, policy, or other structural strategies would you recommend to address this issue in the short and medium term that are feasible and evidence-informed? What stakeholders should be involved? Step 5: Develop an infodemic insights report 28 How to build an infodemic insights report in six steps Template 5.1 Template for an insights report (continued) Sectionofreport Whattoinclude Theme#3with recommendations foraction • A one-sentence summary of the theme. • Several sentences explaining the gist of the important themes or intelligence. Include directionality and the risk assessment for each theme. Explain: o which community or stakeholder is involved; o the relevant platforms/ data source; o an overview of the relevant questions, concerns, information voids, narratives and circulating mis- and disinformation; o points of confusion or gaps between official guidance and stakeholder or community actions and other reactions. • An illustrative, anonymized example, as appropriate. • Recommendations for action: What communications, programmatic, sociobehavioural, policy, or other structural strategies would you recommend to address this issue in the short and medium term that are feasible and evidence-informed? What stakeholders should be involved? Persistentthemes Persistenttheme#1 • Provide updates on any themes mentioned in previous reports that have evolved or require additional attention or actions. • Provide updated recommendations. Persistenttheme#2 • Provide updates on any themes mentioned in previous reports that have evolved or require additional attention or actions. • Provide updated recommendations. Howthereportwasbuilt Methods • Several sentences briefly describing the methods. • A statement of limitations. Datasources • A narrative or table with an overview of the data sources, any relevant links, frequency, and other important information about them (e.g. language if working in multiple languages). Riskmatrix • An explanation of the risk matrix. Furtherinformation • If the methods or report are published online, date and post a permalink. • Links to related content and resources that will help address the themes. 29 When designing one-slide summaries in PowerPoint or for an infographic, adapt this template for your use: Template 5.2 [insert topic] Infodemic Insights: Key themes and recommendations based on integrated analysis ([date span]) Findthetemplatehere Change the icons for each of the three themes to adjust the colour coding to the risk and the type of narrative observed. [taxonomy category] – [gist headline of the narrative theme] [taxonomy category] – [gist headline of the narrative theme] [taxonomy category] – [gist headline of the narrative theme] [recommendations for action] [recommendations for action] [recommendations for action] [Narrative description of what is being discussed, by whom, in what community/language/etc. and “so what” – how is this linked to health programme or public health action.] [Narrative description of what is being discussed, by whom, in what community/language/etc. and “so what” – how is this linked to health programme or public health action.] [Narrative description of what is being discussed, by whom, in what community/language/etc. and “so what” – how is this linked to health programme or public health action.] * Level of risk to public health (impact of uptake of PHSM, vaccines, treatments, diagnostics) as determined by type of narrative, reach, dissemination, affected communities: high; moderate; low; positive sentiment Emerging Persistent Re-emerging Keythemes(bysociallisteningtaxonomy) Recommendationsforaction Step 5: Develop an infodemic insights report 30 “An insights report launched in a forest and there’s no one around, the report won’t make a sound.” An insights report is only as strong as its circulation and the willingness and ability of its stakeholders to act on the insights and recommendations. For more information, see A13 Taking an infodemic insights report to implementation. You can improve distribution of the insights report by: • offering insights in a variety of formats (e.g. a WhatsAppable message, 1 PowerPoint slide, an infographic); • promoting subscriptions to updates (e.g. email listserv, RSS feed, dedicated Telegram or Slack channel); Be sure to regularly speak with stakeholders to understand their opinions on the design and utility of the report and solicit feedback for improvement. They may even provide future questions for the insights team to investigate. If you have developed and shared several reports, they may have been forwarded, shared or used beyond your immediate mailing list. A brief survey accompanying the reports can provide further information on how far and where they have gone. • presenting insights at regularly occurring meetings where operational and programmatic decisions are made (e.g. incident management support team meetings, weekly leadership briefings, intra-campaign monitoring meetings); • embedding insights into existing communications products and channels (e.g. add an infodemic insights section to situation reports or a newsletter); • if appropriate, publishing insights reports online for the public and other stakeholders to access. In order to track what actions have been taken, follow up with key stakeholders. Record these actions and the ways the insights reports are being used, using Template 6.1. Findthetemplatehere DISSEMINATE THE INFODEMIC INSIGHTS REPORT AND TRACK ACTIONS TAKEN STEP 6 Template 6.1 Tracking infodemic actions taken following dissemination of the insights report Infodemic management organizational stakeholder Briefedon insights report101? Receives insights report? Associated recommendations Actions taken Measuresor outcomes Focalpoint 31 ANNEXES 32 Best practices for an infodemic manager in emergency responseAnnex A1 When an emergency is declared, a health authority or other government agency will set up an incident management support team. These structures vary from country to country but usually follow a format similar to WHO’s Emergency Response Framework. Being part of an emergency response can feel overwhelming, but the opportunity to inject infodemic insights into the response structure can help strengthen the overall response. Depending on the grade, size and scope of the emergency, there may be multiple teams set up to address different aspects of the information environment, including health communications, risk communication and community engagement, health information, behavioural surveillance and public health and safety measures. Infodemic management may be included in one of those functions or be an independent team. An infodemic manager operating in such an environment may face the following challenges: • A lot of noise • Difficulty sifting out important information • Competing priorities • Differing lines of communication • Unclear lines of authority • Everything being urgent and important • Not enough staff and time to accomplish objectives • High staff turnover • Data of uncertain provenance. The best way to quickly become effective as an infodemic manager is to understand the emergency response structure, understand the information needs of the incident management leadership team, and build partnerships across teams and with organizations outside of the response structure. These relationships will help you identify additional data sources, provide additional expertise you can leverage and also provide avenues of sharing and acting on infodemic insights reports. You may be able to bring on temporary staff or have staff deployed to your team to increase analytic capacity, but they may not be around for longer than a few days or weeks, so developing standard operating procedures (SOPs) and strong handover processes is important. You should also understand any existing clearance chains for scientific or communications clearance, and depending on the audience and distribution (internal, only with partners, the public?) of your infodemic insights report, you may need to seek one or both. Build this into your report development timeline. Remember, most emergency responses need a turnaround on data requests of hours and days, not weeks, so you may need to develop the first report extremely quickly with whatever information is to hand. Demonstrate value early by providing insights and other technical assistance quickly as you build your data pipeline and team. 33 Best practices for an infodemic manager in emergency response Competencies for infodemic management team membersAnnex A2 WHO has published a competency framework that can be used to identify the skills that are needed on an infodemic management team. Below are the competencies that may be most relevant for developing an infodemic insights report and taking action after the report, across the domains of: Listening, Delivery, Intervention and Promotion. You can use this as a starting point to develop terms of reference and outline workflows. Social listening and integrated analysis Planning (landscape analysis, goals/objectives, analysis planning, setting up systems) Develop and utilize standard operating procedures to collect, analyse and utilize information on a topic across various data sources Map preferred channels of communication with focus on vulnerable groups Perform audience analysis and segmentation Identify potential data sources and evaluate them for limitations and weakness inherent in each individual source or in the context of other sources to be used Conduct literature reviews and analysis related to topic Establish SMART goals and objectives Develop or adopt a taxonomy of classification for the topic, including the development of Boolean search strings and an associated code book Establish networks and connections with appropriate partners or staff members responsible for available data sources Connect infodemic management and social listening work goals and objectives to overarching programme goals and strategies Evaluate data collection and social listening tools and platforms to determine best use cases Conduct a desk review to identify available and appropriate resources, tools and templates Establish the criteria and approach for determining and applying a threat or impact matrix to analyse the topic and themes to adapt programme needs Establish transparent, bilateral data sharing mechanisms with collaborators and partners Determine how to weight various data sources against each other based on programmatic priorities Data collection Navigate social listening tools and platforms to collect appropriate data Access and collect data from routine data sources Analyse both quantitative and qualitative data Develop a system for data management and organization 34 How to build an infodemic insights report in six steps Social listening and integrated analysis Integrated analysis Analyse and evaluate individuals’ behaviours, focusing on personal, social and environmental determinants Detect key topics and narratives, information deficits and open questions, and mis/disinformation in offline and online populations, across multiple data sources and types Identify, analyse and evaluate the evidence-basis of the main narratives and claims over health issues circulating within the population Facilitate a collaborative process including staff with specialization in communications, behavioural science, informatics, data management and epidemiology to contribute to integrated analysis Determine how to apply a pre-determined threat or impact matrix to identified themes Discern appropriate data analysis techniques and approaches to different data sources and to address specific programme needs Delivery of high-quality health information and programming Strategic planning/theory and framework application Provide evidence-informed recommendations for priority follow-up actions on detected high-risk signals Apply analytical and evaluative frameworks to assess the quality and the risk of mis/disinformation to behaviours Report development Write and edit content for a variety of stakeholders about findings informed by theory and strategic planning including informing graphic design elements and visualizations Adapt or adjust threat or impact level determinations based on programmatic priorities and the appropriate behavioural theory/framework Be familiar with programme and policy to offer realistic short-term and long-term recommendations or opportunities for action that strength health programmes and systems Report dissemination/communicating findings Liaise closely with the HQ and regional counterparts ensuring coherence of strategy, materials and messages Action planning Identify targets for infodemic interventions from policy and the health system to community and individual levels Strategies, recommendations and intervention development Design Identify barriers to and facilitators of a desired health behaviour or outcome in the target population Define the knowledge or desired health behaviour and identify evidence-based or emerging best practices for intervention development Clearly define target audiences and map the available and applicable data or information Develop persona for target audiences and demographics for interventions Identify the most appropriate model of change to apply to intervention development and develop clear process and outcome indicators 35 Strategies, recommendations and intervention development Develop and implement interventions that address the individual, community, cultural and societal-level factors affecting trust and resilience Be familiar with human-centred design principles Have experience of pre-testing or piloting new or adapted interventions in the field with target populations Identify appropriate partners and stakeholders to enhance the development of interventions Be able to interpret the latest scientific research, literature on the evidence base and translate it into interventions Communications (content development) Craft content, key messages and talking points about identified themes Coordinate with communications, graphic design and social media partners or colleagues to develop and test a communication campaign or product Be familiar with the health authority’s channels and style of communication Support spokespersons and health officials through the development of talking points, press releases and other external communications Identify means for the internal communication of key messages to ensure message alignment and cohesion Conduct message testing with target audiences, be familiar with message testing design, theories and methods, including conducting A/B testing and focus groups Be familiar with different social media platform standards and best practices for message dissemination Have knowledge of health communication science best practices Be able to navigate scientific and organizational clearance processes Outreach and engagement/technical assistance Identify and establish partnerships with stakeholders and networks that can serve as amplifiers of content and interventions Explain and present the process, findings and interventions to different audiences Provide customer service support for internal and external partners for acute infodemic management needs Be ready to quickly conduct outreach to target communities and leverage digital tools to quickly collect feedback and adjust interventions and communications Research Determine how to expand or adapt current routine surveys or surveillance mechanisms to gather additional necessary data Identify partners and external stakeholders conducting research to better understand information voids, perceptions and opinions and their impact on the desired health behaviour Work with behavioural scientists on the development of research, testing interventions and understanding behaviours Discern the soundness of research methodologies and approaches Implementation Identify existing systems, structures and programmes to implement designed interventions Determine the resources, tools and personnel needed to implement and monitor interventions Coordinate with health programme staff to implement designed interventions and provide customer service Annex A2: Competencies for infodemic management team members 36 How to build an infodemic insights report in six steps Promotion of resilience to misinformation and other infodemic harms Risk communication and community engagement Apply risk communication principles, behavioural science theories and models, and people-centred approaches to the development of health communication campaigns for diverse audiences Identify and strengthen networks of trusted messengers and influencers Engage local media in providing tailored messages and accurate, credible information on health topics Design community engagement strategies and activities (i.e. social inoculation) Develop community and social listening feedback loops, including the creation of a tipline or email for concerns from the community about information gaps and misinformation Create and disseminate appropriate communication products (digital and print) Develop a mechanism for responding quickly to community questions and concerns Provide branded and unbranded content and materials that can be adapted by influencers and trusted messengers Create and maintain a frequently asked questions webpage and communication products, including the latest circulating rumours and mis/disinformation Build the capacity of health care workers to have empathetic conversations based on motivational interviewing best practices with patients, including listening to patients’ concerns, and address information gaps and misinformation Develop strategies that leverage social norms and social norm modelling to promote healthy or desired behaviours Partnership Establish and strengthen partnerships with medical and health care associations Establish and strengthen partnerships with news media and fact checking organizations Collaborate with community arts and cultural organizations Collaborate with faith-based organizations Collaborate with health and social services-focused organizations Collaborate with family and youth-focused organizations Establish and strengthen partnerships with technology companies Extend and strengthen partnerships across government agencies, especially across different levels of government from local to national Seek out and build partnerships with organizations that specifically address the needs of disproportionately affected or vulnerable populations Leverage existing partnerships to identify new relevant and reliable data and data sources Technical assistance Build the capacity of partners, influencers and trusted messenger to share accurate, credible information and address concerns, information gaps and misinformation effectively Assist partners and the local level of government to implement the interventions and campaigns Provide individualized consultations and feedback on infodemic-related issues among partners within government and appropriate nongovernmental organizations Support local government and organizations in conducting rapid community assessments to better understand the needs of target audiences and the necessary interventions 37 Strengthening of systems for preparedness, prevention and response Monitoring and evaluation Identify goals and objectives for an evaluation of the interventions and develop key indicators Be able to conduct rapid assessments about a specific topic or intervention Identify the evaluation data needs and collection methods Conduct research and an evaluation, and be able to integrate participatory methods Explain and present the findings of the evaluation Identify the audiences and plans for dissemination Develop a system for process improvement that is iterative and ongoing Policy Identify the key policy-makers and organizations within or supporting the health system Build partnerships to strengthen the internal and external policies related to infodemic management Identify the policy gaps or bottlenecks that limit the ability to assess infodemic issues and quickly react Write policy briefs and concept notes to translate findings for key stakeholders and the public Conduct a policy analysis to determine the existing gaps in a health system and determine the potential impacts of new policies Capacity-building Identify the types of stakeholders, partners and colleagues necessary to support both the diagnostics and interventions for infodemics Develop and present training and seminars for others within the health system about what infodemic management is Establish and maintain networks and relationships to expand the science of infodemic management and leverage expertise internally and externally Advocate for system and policy improvements to address the identified issues Quality and process improvement Identify the goals and actions needed to improve the processes and uptake and use of the insights reports Collect and apply feedback to improve processes and outputs of infodemic insights analysis Define and track metrics and measures for each identified action and regularly check how well they fit the set targets Promotion of resilience to misinformation and other infodemic harms Offer surge support during a crisis or emergency Build the capacity of other staff within the organization or government to conduct infodemic management activities and provide support to the infodemic management unit Annex A2: Competencies for infodemic management team members 38 Because the field of social listening and infodemic insights development and management is still growing, there isn’t much guidance currently available, although WHO is currently leading efforts to develop such guidance. However, it’s important to check whether your organization has any existing guidance. If not, consider these two sources: • The Association of Internet Researchers have published guidelines on conducting ethical Internet research • Ethical standards for research during public health emergencies: distilling existing guidance to support COVID-19 R&D Resources related to the ethics of social listening and infodemic managementAnnex A3 39 How to build a workflow for the infodemic insights reportAnnex A4 Each team and workflow will be different, but you can use the table below to begin organizing roles and assigning them to specific tasks or steps in the process. Responsible Details Time frame Before getting started: current make-up of the infodemic insights team Team member #1 Team member #2 Team member #3 STEP 1. Choose the question that infodemic management insights could help to answer Identify the question STEP 2. Identify and select the data sources and develop an analysis plan for each data source Data source #1 Data source #2 Data source #3 Data source #4 Data source #5 STEP 3. Conduct an integrated analysis across the data sources Integrated analysis meeting A Integrated analysis meeting B Integrated analysis meeting C STEP 4. Develop strategies and recommendations Expert #1 consulted Expert #2 consulted Expert #3 consulted STEP 5. Develop an infodemic insights report Writer #1 Writer #2 Designer Reviewer #1 Reviewer #2 (manager) Clearance 40 How to build an infodemic insights report in six steps Responsible Details Time frame STEP 6. Disseminate the infodemic insights report and track actions taken Publish/post report Disseminate on channel A Disseminate on channel B Disseminate on channel C Collect and analyse feedback/track actions 41 Troubleshooting common challengesAnnex A5 Common challenge Sample strategies It’s difficult to bring partners together to discuss or contribute data or provide input to the infodemic insights report • Reach out individually to partners, schedule a discussion where you explain infodemic insights reporting, how their organization can benefit, provide something of value (e.g. an insight from the latest report that is relevant), and invite them for a further discussion. • Ask a partner for advice on a specific issue, or ask them to join as a subject-matter expert whom you’d occasionally engage with to get their recommendation for what goes into a report. • Raise awareness of infodemic insights reports and share them in standing meetings. • Share in email or message-based groups and invite more partners to join future discussions. • Raise the issue with your manager and ask for help getting buy-in from leadership at partner organizations. It’s difficult to disseminate the report more widely to partners who might benefit • Provide the report in more formats (e.g. a single infographic, WhatsApp-formatted message). • Raise awareness of infodemic insights reports and share out in standing meetings. • Share on email or message-based groups and invite more partners to join future discussions. • Join partner meetings and ask to highlight the latest report findings. • Identify future opportunities where a report may be issued on emergency or programme activities where partners are expected to be included. There is little organizational capacity to understand what an infodemic insights unit does or what an infodemic insights report it, which means that the reports aren’t optimally used for public health action • Develop an “infodemic management 101” training and present it at an upcoming staff meeting. • Ask less engaged colleagues for advice on a specific issue, or ask them to join as a subject-matter expert whom you’d occasionally engage with to get their recommendation for what goes into a report. • Develop an insights report that investigates an issue of concern to the less engaged part of the organization and demonstrate the value of this approach. • Publicize how infodemic insights reports are being used to drive public health action and share the positive feedback your reports have received. • Recruit members to the insights team from other parts of the organization so that more teams become familiar with infodemic management work. • Develop a feedback loop with infodemic insights report recipients and ask how they are using the report, and how to improve it to make it more useful. High turnover of staff makes it difficult to maintain momentum • Develop standard operating procedures (SOPs) for all processes and include them in all onboarding presentations to staff. • Ask for team members to periodically review and update the SOPs with the latest learnings for their areas of work. • Ensure that a common filesharing and organizational structure exists where everyone is expected to upload and save their work products. • Consider alternative staffing approaches such as hiring consultants, working with fellows or interns, detailing people from other departments for the short term, or requesting additional staff to be deployed. • Ask partners for assistance, either by integrating volunteers into the team to work on activities that are a priority for their parent organization, or delegate specific tasks to specific partners. 42 How to build an infodemic insights report in six steps Common challenge Sample strategies Colleagues who work in other disciplines find it difficult to understand the work of the infodemic insights team or our reports • Develop a “cross walk” document that relates major concepts that are familiar to certain profiles (e.g. epidemiologists) to the terms and concepts used in infodemic management and share it. • Ensure that the style guide has a list of common terms and definitions that are linked or referenced in all insights reports. • Ask less engaged colleagues for advice on a specific issues, or ask them to join as a subject-matter expert whom you’d occasionally engage with to get their recommendation for what goes into a report. • Develop an “infodemic management 101” training and present at an upcoming staff meeting. • Publicize how infodemic insights reports are being used to drive public health action and share the positive feedback your reports have received. It’s difficult to monitor the impact of the infodemic insights report • Develop a simple feedback form that is linked and shared alongside all reports and request that recipients fill it out. • Develop a feedback loop with infodemic insights report recipients and ask how they are using the report, and how to improve it to make it more useful. • Create short links or QR codes on distributed reports to monitor the number of clicks or downloads; similarly, track downloads and links when posted online. • Set up a simple Google search alert for your infodemic insights report to learn where else it’s being referenced and reacted to online. 43 Key messages for advocating the use of infodemic insights reportsAnnex A6 Having key messages for advocating internally and externally for the use of infodemic insights reports can be helpful in a variety of situations, including raising awareness, increasing subscriber count and for securing further buy-in, resources and additional staffing and funding. Develop key messages based on your organizational structure and operational needs. Here are a few standard key messages you can tailor to your context: • In order to address [public health issue x] we need widespread trust in [name institution] and a willingness to follow public health guidance. However, we know that everyone is living in a noisy information environment that may influence their awareness, perceptions and behaviour. • There is a lot of misleading, outdated or inaccurate information out there that can erode that trust and make it less likely that people will follow health guidance. • For example, we saw that [high priority infodemic issue that was recognized by leadership or was recently in the news] affected people’s [perception/trust/willingness to follow guidance] which [describe the impact]. • Social listening and infodemic management approaches can help us understand that noisy information landscape and understand how it’s affecting different populations, and provide evidence-based recommendations for action. • How do we do it? Infodemic monitoring and insights generation encompasses a systematic analysis of different data sources and generation of intelligence. It turns that intelligence into infodemic insights and applies a structured judgement and risk matrix approach to generate recommendations for action. • We regularly publish and disseminate the [programme name] infodemic insights report to equip our colleagues with the information they need on how the public is reacting to the [emergency or health topic] so that they can adjust their response and improve its effectiveness. • For example, in the last report, we detected [issue x], which we found across [multiple data sources]. The conclusion we reached is that [describe major theme]. There are several infodemic management actions that can be taken to address this, including [recommendations for action]. • Since then, [describe how the insights report was used or how action was taken based on this insight]. • Right now, we are utilizing [number] data sources with a team that includes [describe team specializations] and have developed [number] infodemic insights reports since [first date of report], providing assistance to the [response/programme]. With more [staff/resources/funding/ organizational support] we could [include details of what you could accomplish scaling up]. 44 How to build an infodemic insights report in six steps Scaling up or scaling down infodemic insights processes and reports, depending on contextAnnex A7 Nearly every country is conducting social listening or infodemic insights generation, which recognizes the priority placed on infodemic management within health authorities. However, an expensive vision may not be possible to make a reality without a sufficient budget, enough trained human resources, enough time or enough organizational and political buy-in and demand for the insights by the policy-makers. Therefore, you may have to scale down your vision and your objectives based on the resources that you have available at the moment. In the table below, we compare different operational contexts to different types of vehicles, which have brief definitions in the column headers. Depending on the operating environment you are in, you may be the lone person working on infodemic insights (bicycle), have a few more people and resources (sturdy 4x4) or have a team and lots of resources (luxury vehicle). This section describes what can realistically be accomplished for someone in each type of environment, so you can accomplish the step on a small, medium or large scale. Bicycle Very constrained environment, likely a team of one or two people Sturdy 4x4 Some resources and supportive policies Luxury SUV Sufficient dedicated resources and policies to achieve multiple objectives Before you start • Find a way to consult subject-matter experts between steps 1 and 6 even if you are alone and even if there is no formally set process to do this. • Lean on your professional network to find advice, data sources and resources. • Include a subject-matter expert as a regular adviser to the team. • Determine how to structure your team, including roles and responsibilities and reporting lines up to the emergency response or health programme leadership. • Build in ways to request feedback and evaluate your efforts in steps 1–6, and implement improvements. • Include a subject-matter expert (or several) as part of the team. • Identify ways you can recruit and retain team members with the right skillsets from across different parts of your organization. • Consider creating a flying team or surge team that can be activated when an emergency strikes at national or subnational level. • Support the creation of subnational teams or the ability to develop subnational infodemic insights. • Seek your manager’s support to get institutional buy-in to insert and disseminate infodemic insights reports in formal communications and meetings. • Actively advocate for supportive policies for access to data and for uptake of the infodemic insights into programming. 45 Scaling up or scaling down infodemic insights processes and reports, depending on context Bicycle Very constrained environment, likely a team of one or two people Sturdy 4x4 Some resources and supportive policies Luxury SUV Sufficient dedicated resources and policies to achieve multiple objectives STEP 1: Choose the question that infodemic management insights could help answer • Identify a question of concern that narrowly identifies a smaller population to focus on. • Identify a question of concern that more broadly identifies a population to focus on. • Identify a question of concern that identifies several populations to focus on; or • Formulate several related questions on one population. STEP 2: Identify and select the data sources and develop an analysis plan for each data source • Choose data sources that are readily available to you. • Understand their limitations when interpreting them. • Identify additional data sources from the routine health programme and emergency response. • Consider integrating other social listening reports. • Build relationships to access data sources from outside the health system. • Focus on more languages and consider analysing more than only text conversations. • Leverage partnerships with civil society organizations, the media and academia to identify and include data sources. • Consider utilizing more advanced analysis methods. • If there are large gaps in the data coverage, develop and implement new data collection to generate a new source. • Routinely analyse images and video as part of the content analysis in multiple languages. STEP 3: Conduct an integrated analysis across all data sources • Focus on using only a sample of large datasets. • Use qualitative coding methods to identify themes. • Avoid relying on AI-generated analysis only. • Assign data collection and analysis tasks across team members. • Develop a structured team approach to discussing findings and identify themes. • Develop standard operating procedures, codebooks, taxonomies and templates. • If repeating insights reports on a topic, consider analysing long-term trends and comparing them with major developments in the response to the acute health event. • Leverage and customize AI-assisted analysis for specific data sources. • Connect the integrated analysis to existing models and frameworks that may be used in the programme or in emergency response for the research question of concern. STEP 4: Develop strategies and recommendations • Include strategies that can be implemented with the available staff and resources. • Make recommendations that target improving communications and community engagement. • Include strategies that can be implemented beyond your team and among external partners. • Develop recommendations that may address policy, structural, supply, service delivery or programmatic issues. • Include strategies that rely on intersectoral collaboration. • Include more explicit strategies that address more segmented vulnerable populations. • Include more human-centred design components within strategies and recommendations. • Identify which stakeholder should act on each recommendation. STEP 5: Develop an infodemic insights report • Focus on the highest-risk themes. • Use a very simple insights report template. • Include more themes and a deeper analysis of persistent and re-emerging themes. • Produce the report in multiple formats. • Offer the report at more frequent intervals. • Provide more ways for readers to engage with the report’s content, such as interactive visualizations. • Develop reports at subnational level or deep-dives into specific topics. Annex A7: Scaling up or scaling down infodemic insights processes and reports, depending on context 46 How to build an infodemic insights report in six steps How do you know if you’re graduating from a bicycle to a sturdy 4x4? • You receive requests from response leadership on specific insights or to troubleshoot specific narratives or misinformation. • Report recipients tell you that the report is helpful and that they shared it with colleagues. • You notice that the insights report is referenced in official documents and in talking points by leadership. • You are invited to regularly present your insights to the health programme or emergency response team. • Other people in partner organizations start generating their own insights reports because they find them useful. • People want to join your team because word has gotten out that your team is doing something useful and innovative. • Specific strategies from the insights report are implemented in the field. • Colleagues suggest new data sources or new partnerships to explore. • Colleagues or people in leadership express frustration with how long it takes to publish a report and want the insights more quickly. These types of feedback can be useful when advocating for additional resources and support to continue or expand infodemic insights reporting and infodemic management activities. Bicycle Very constrained environment, likely a team of one or two people Sturdy 4x4 Some resources and supportive policies Luxury SUV Sufficient dedicated resources and policies to achieve multiple objectives STEP 6: Disseminate the infodemic insights report and track actions taken • At the bare minimum, track how far the insights reports have been disseminated, who has been using them and how. • Disseminate through a self-managed mailing list or a messaging app group. • Disseminate in multiple formats, platforms and for multiple audiences. • Take part in regular technical or leadership meetings to share insights. • Assign a person to track the distribution and use of the report and to solicit and incorporate feedback. • Optimize the review and clearance processes to speed up the timeline between steps 5 and 6. • Collaborate with communications colleagues to develop better factcheck content and more effective messages based on the insights. 47 Data source evaluation checklistAnnex A8 Is this data source reliable? Does this data source have a high degree of accuracy? Is data “fresh” enough (less than two weeks old in emergencies) to still represent the current picture? Is this data source used by others in the emergency or health space? Does the source have any external endorsements of authority and credibility? Is this source a one-off or routine data source? (avoid one-offs) Is the data quality checked? Does the data source have any information related to bias, or is bias accounted for in the data collection? Are the underlying data collection and cleaning methods clear? Were the data collected in a consistent manner? Were the data collected in an ethical manner? Are the data available in a format that can be easily imported and analysed? Will the data help to address the infodemic question of concern? Is this data source affordable to include? If the data source requires additional analysis, does the team have the capacity and expertise to do so? 48 How to run a cross-disciplinary infodemic insights meetingAnnex A9 If you have more than one person working on the infodemic insights report, you may want to host regular meetings to discuss data sources, data analysis plans, and to combine datasets and intelligence for an integrated analysis, and finally to develop of the report. It can be helpful to follow a common meeting format, such as: • An update on related emergency or programmatic developments or events that may have implications for the report. • A review of the latest requests for support/feedback from response, programme or partners. • Give the current status of the infodemic insights report. • A round robin update on people’s status/progress on their assigned tasks. • Identify and discuss specific challenges. • If the meeting is for data synthesis or integrated insights development: • Project on screen and walk through a completed Template 3.2 Compiling data source intelligence. • Discuss each data source in turn and what was learned across all columns. • Then, discuss what commonalities there are. • Finally, identify and discuss the major themes and what risk level should be assigned for each and complete Template 3.3 Identifying major themes across data sources and intelligence and prioritizing by risk together. • Identify the next steps and action items. • Address any other business (AOB). 49 How to develop a taxonomy A taxonomy is a system of classification or structure that will allow you to align your search strategy with topics that are most relevant to your question of concern and cover the domains that your organization is responsible for. For example, if you’re asked to build infodemic insights reports to track a foodborne outbreak, you may format the taxonomy on domains related to an epidemiological investigation. A taxonomy can also help you organize intelligence from data and map the areas of a public health topic where the information, perception, discussion and behaviour are most concentrated, where conversation seems to be missing or silent, and where new topics or concerns may emerge. This map will probably change between infodemic insights reports, which is why taxonomies are living documents, and should be updated with new concerns, related keywords and instructions to the analyst. There are many ways of building a taxonomy, such as by audience, knowledge and behaviour, or domains of a public health issue, or by identified challenges to achieving a desired public health outcome. There is no right or wrong way to develop a taxonomy, but it is likely that it will dramatically change after one or two infodemic insights reports because you will update the domains and categories with what you have learned and according to how the discourse may have shifted. Here are some ways you can start the process of building a taxonomy. If you aren’t sure where to begin, start by asking yourself or the team the following questions: • What is the public health issue? • If a pathogen: What do we know about how it spreads and infects people? How can it be prevented? What strategies are there to intervene? How are people talking about it? • If an emergency: How is this emergency affecting different groups of people in the short, medium and long term? Who is responsible for addressing it? What strategies or policies are currently in place? • If another health topic: How is this health topic affecting different groups of people in the short, medium and long term? What are the current policies and guidance and what are the current perceptions and behaviours on this topic? • What is the gap between the guidance and the population’s behaviour? • What is the reaction to current guidance? • What are the positive behaviours? How are they being communicated and displayed? • What are the behaviours that don’t protect health? How are they being communicated and displayed? • How is the information environment affecting the discourse on this topic? • Who is talking about this topic and to whom? You can draw inspiration from the structure of the emergency response you’re a part of (e.g. if there is a team working on non-pharmaceutical interventions, or a team focused on vulnerable populations, you can bet these are priority topics; you need to understand the public discourse about them so should have a category for each). If you are part of a health programme, look at how the programme is arranged to determine potential areas of focus (e.g. focus on health workers if one of the main strategies for influenza preparedness is high vaccine confidence and uptake among health workers). Annex A10 50 How to build an infodemic insights report in six steps Another approach is to conduct a “fishbone” exercise (Figure A3.1), where the outcome may be the perfect public health outcome that is desired (e.g. reduced spread of viral disease among community members). You write this at the head of the “fish”. Then, work backwards to list the possible reasons why this has not yet been achieved, which are the “fish bones”. These should also be reasons that would surface in public discourse. Congratulations, you have just found potential domains of inquiry that you can turn into a taxonomy. Figure A3.1 Example of a fishbone exercise Desired Outcome Reason#5 Reason#6 Reason#7 Reason#8 Re as on # 4 Re as on # 3 Re as on # 2 is no tb ei ng ac hi ev ed Re as on # 1W hy o ut co m e 51 The taxonomy below was designed to track the spread of a zoonotic infectious disease (mpox). It can be adapted to other pathogens or types of emergencies. There is a column for the topic and related subtopics, as well as space to provide more specific guidance to the analyst. All of these should be adapted and constantly updated over the course of the emergency or for however long the infodemic insights reports are being generated. The important structure to maintain is focusing on categorizing by cause, illness, treatment, interventions and conversations about information. The taxonomy spans five categories of topics of conversation: 1. The cause – How did the virus emerge and how is it spreading? 2. The illness – What do we know about the disease, what are the symptoms and how is it transmitted? 3. The treatment – How can it be cured? 4. The interventions – What is being done by the authorities and institutions? 5. Conversations about information – Meta-conversations about guidance, reporting, misinformation and content. CATEGORY #1: How are people talking about the cause? Questions, concerns, confusion, information voids, narratives, mis/disinformation about how the virus emerged and how it is spreading Topic Subtopic Notes for analyst in keyword setup and interpretation The source of the virus For example: from animals/monkeys/small mammals /rodents, zoonoses, smallpox/variola, other Possible speculation about bioengineering Defrosting of tundra/environmental sources (smallpox concerns) Stigma For example: men who have sex with men, north–south, Africa, Nigeria Spread Endemic/nonendemic countries Travel Community contexts Use latest WHO guidance for endemic/nonendemic references Urban/rural Example: community contexts in nonendemic countries – household, workplace, school/nursery, sexual contacts, health care, houses of worship, transportation, sports, festivals, concerts, social gatherings, parties and any other recalled interactions. Spread via travel, including cross-border, air travel Sample taxonomy for generating infodemic insightsAnnex A11 52 How to build an infodemic insights report in six steps CATEGORY #2: How are people talking about the illness? Questions, concerns, confusion, information voids, narratives, mis/disinformation about the disease, what the symptoms are and how it is transmitted Topic Subtopic Notes for analyst in keyword setup and interpretation Symptoms Confirmed Other Confirmed: rash, blisters, headache, fever, swollen lymph nodes, muscle aches, back pain, weakness Example: confusion with other diseases that cause a rash, such as chickenpox Asymptomatic Asymptomatic transmission Not expected to occur often, but analysis should pick up worries and questions about this. Means of transmission Face-to-face exposure Direct physical contact, skin-to-skin, mouth-to-skin (including sexual contact) Contaminated materials Animal–human contact or consumption of meat Mother–fetus Other Example of face-to-face exposure: through body fluids, including respiratory droplets, also including health care workers without appropriate PPE) Example contaminated materials: clothing, bedding or utensils that have been used by a symptomatic person Including consumption of illegally traded meats, bushmeat Congenital mpox Sexual transmission Severity of disease By age groups Complications Children, adults <40, >40 and elderly people Secondary infections: bronchopneumonia, sepsis, encephalitis Protection from transmission/prevention Effectiveness of condoms (this will not prevent transmission but some people might think so) Risk Eating uncooked meat Age Sex Underlying conditions Health care workers Sex workers Caregivers Example eating meat: eating uncooked meat or other parts of animals (in particular bushmeat) Risk groups may be: children, pregnant women, parents caring for sick children, immunocompromised people, health care workers, people living with HIV/AIDS, sex workers Similarities to other orthopoxviruses (e.g. smallpox/variola virus) How smallpox and mpox are discussed together Poxvirus (e.g. smallpox/variola virus) Please note that chickenpox is not an orthopoxvirus Comparisons with COVID-19 Risk/severity Similarities to COVID-19 How mpox and COVID-19 are discussed together 53 CATEGORY #3: How are people talking about the treatment? Questions, concerns, confusion, information voids, narratives, mis/disinformation about how the disease can be prevented or cured Topic Subtopic Notes for analyst in keyword setup and interpretation Current treatment Note: antivirals (Tecovirimat)/clinical care There may be mention of brincidofovir (Tembexa) and brincidofovir (not licensed for smallpox) Vaccine Note: vaccine for smallpox MVA-BN (JYNNEOS/Imvamune/Imvanex); ACAM2000; LC16) Mpox vaccine; also MVA-BN Vaccine stockpiles/strategic reserves Research and development Vaccine nationalism Supply chain challenges Treatment myths CATEGORY #4: How are people talking about the interventions? Questions, concerns, confusion, information voids, narratives, mis/disinformation about what is being done by the authorities and institutions Topic Subtopic Notes for analyst in keyword setup and interpretation Personal protective equipment Example: gloves, gowns, medical masks and eye protection for health care workers, lab workers; masks, sheets/gowns for patients Quarantine Isolation and quarantine protocols Supportive care Health care Health care equipment Personal measures Example: washing hands, physical distancing, condom use Contact tracing Case isolation duration Close contact definition Inequalities in intervention access/use Example: in access to antiretrovirals, vaccine Other Measures in public settings Travel measures Note: not yet recommended, but people may already be discussing it Annex A11: Sample taxonomy for generating infodemic insights 54 How to build an infodemic insights report in six steps CATEGORY #5: How are people talking about the information? Questions, concerns, confusion, narratives in the meta-conversation about guidance, reporting, misinformation, content Topic Subtopic Notes for analyst in keyword setup and interpretation Data and evidence Health guidance Reactions, confusion, interpretation of guidance Mis- and disinformation Trust in health advice Trust in the government 55 Resources on developing a style guide and aiming for inclusive languageAnnex A12 A style guide is a set of standards or guidance on how to write a specific type of publication. You probably already adhere to a style guide, such as for referencing, that may be preferred by your organization. However, when developing infodemic insights and writing reports, you will develop a style for this new type of product. Capturing the way visual design and content come together for the report in a style guide will allow multiple report writers to write in the same way and for editors to ensure consistency of design and language. A style guide should not only encompass visual and design information (e.g. typography, logo use, colours and format) but also address common grammar, spelling, definitions/acronyms and best practice of writing style (technical to simple), tone (formal to informal) and finally, the language used. Usage is the most critically important for a report writer, because the language used to define groups of people and places should be clear, balanced and avoid stigmatizing or stereotypical language, even if the way the public talks about such issues is stigmatizing or stereotypical. If you include examples that use such language in the report, clearly indicate that you are citing specific use of a term; this almost always deserves more analysis and will point towards a public perception that needs to be addressed. For example, if you detect discussions of refugees as “dirty illegals” and increased use of the hashtag #throwthebumsout, this points to wide use of stigmatizing language that may be associated with a public health issue to be addressed, especially if a group of people are demonized or the spread of a pathogen is misattributed to a vulnerable group of people. Inthestyleguide,youshouldcarefullyoutlinehowyouwilldiscussspecificgroupsofpeopletoavoid usingstigmatizinglanguageinareportdesignedtopreventfurtherharmtothem.It’simportantto separateouthealthorlivingconditionfromtheidentityofgroupsofpeople. For example, instead of writing “the poor” say “people living in poverty” or “people with lower socioeconomic status”. Instead of writing “disabled people” write “people with disabilities”. Consult an expert or at least a person from that community if you’re not sure about what the appropriate terms are for different groups of people. You want to use inclusive language wherever possible. This also extends to any visuals you use. For example, avoid using pictures of people with a specific disease, especially if there is a stereotype or stigma related to the disease, unless it’s to underscore a specific point. For example, during early stages of the 2022 mpox outbreak, assumptions were made that mpox only affects Africans, which was reinforced by media imagery of Africans and graphic examples of mpox infections on darker skin tones. These images made it difficult for the public outside of Africa to imagine that they could be at risk, and made it difficult for clinicians to diagnose potential mpox infections on other skin tones. Examples of style guides: • Canadian National Collaborating Centre for Determinants of Health: Let᾽s Talk: Language of health equity • US CDC Global Public Health Equity Guiding Principles for Communication • UK National Institute of Care and Excellence style guide • American Medical Association: Advancing Health Equity: A Guide to Language, Narrative and Concepts. • US National Institutes of Health: Writing Respectfully: Person-First and Identity-First Language Although style guides that reference health equity topics and language may be more common in newsrooms or for research journals, updating your institutional style guide or developing a working style guide for your team on these topics can ensure that the infodemic insights reports that you generate don’t unintentionally perpetuate harmful stereotypes or narratives. Framinginfodemicissuesusingclear languageisimportantbecauseitalsocanaffecthowstrategiesandrecommendationsareunderstood, implementedandhowtheyaredescribedinthefuture. 56 Usually, infodemic managers and analysts will develop recommendations to be discussed in the response and taken forward by other teams, although it is common that recommendations and actions related to the digital information environment may be implemented by the infodemic management team. If you are contributing to the discussion of an implementation plan or have been asked to develop an implementation plan based on the infodemic insights report recommendations, consider including the following: • A summary of evidence supporting the recommendation • Suggestions of different ways to implement the recommendation • An indication of who is supposed to take action on the recommendation • The estimated costs of implementing the recommendation • The benefits of implementing the recommendation • Any barriers to implementing the recommendation and how to mitigate them. Provide enough detail for the decision-maker to assess and take action, but keep it to a maximum of one page. Taking an infodemic insights report to implementationAnnex A13 57 APPENDICES 58 How to build an infodemic insights report in six steps How to develop infodemic insights on zero-dose children Appendix 1 Step Considerations Step1:Choosethequestions thatinfodemicmanagement insightscouldhelpanswer Starterquestionsthatyoucanconsider • What concerns and questions about vaccines currently exist in community X? • In geographically-bound area of interest, what conversations and health topics are top of mind for parents and caregivers, and how does immunization feature in these conversations? • How are circulating concerns, questions and narratives affecting peoples’ intent to vaccinate their children? Can be further unpacked by using the framework on Behavioural and Social Drivers (BeSD) on immunization. Programmeorstakeholdersbestplacedtoidentifythemostrelevantquestionandwould benefitmostfrominfodemicinsightsreportsonthatquestion Immunization programme managers can best identify the most relevant questions and use the insights recommended by the infodemic insights report in driving toward interventions. The interventions will be context-specific and coordinated with immunization programme implementation.1 Zero-dose children Millions of children are missing out on life-saving doses of routine vaccines, and some have received no vaccines at all. Children who have completely missed all vaccinations fall into the zero-dose category. Zero-dose children are defined by GAVI as those who lack the first dose of diphtheria-tetanus- pertussis-containing vaccine (DTP1). There are also a large cohort of children who fall into the under- immunized category, where they have received some vaccines, but have not completed the entire vaccine schedule. Most of these children tend to live in missed communities where there are clusters of zero-dose and under-immunized children, who often face barriers to accessing social and health services. Understanding where these zero-dose or under-vaccinated children are and the perceptions and behaviors of their parents and caregivers can help immunization programmes plan more effective outreach and demand generation strategies. An infodemic insights report that focuses on zero-dose and under-immunized children can inform these strategies. It is important to note that missed communities often do not have the same level of internet penetration and mobile internet access compared to other communities that more commonly access social media and digital information sources that are often used as part of infodemic insights reports. Therefore, any infodemic insights reports being developed regarding zero-dose children should include a variety of data sources, including offline and community-level sources, that geographically pinpoint and describe these communities and the conversations and attitudes expressed among parents and caregivers. Note that because of this hyper-locality of analysis, special care must be taken to adhere to ethics and governance guidance on data collection, analysis and reporting of infodemic insights. Specialconsiderationsforzero-dosechildren 1 World Health Organization. (2022). Behavioural and social drivers of vaccination: tools and practical guidance for achieving high uptake. https://apps.who.int/iris/handle/10665/354459. 59 Step Considerations Step2:Identifyandselect datasourcesanddevelopan analysisplanforeachdata source Datasourcesspecificallyavailableonthistopic Multiple Indicator Cluster Surveys (MICS), Reaching Every District (RED) plans, field studies e.g., KAPBs on vaccination, Community Rapid Assessments on demand-related areas, vaccine-specific digital listening reports, rapid convenience monitoring (RCM) reports usually done during supplemental immunization activities that routinely gather information on reasons for non-vaccination, field reports from health workers and the teams involved in immunization activities, Community discussion reports, vaccination coverage reports at the local level and available demographic and social data, mobility reports and other GIS data. Specificcriteriatoassessqualityofdatasourcesonthistopic • Prioritize geographically-specific or vulnerable population-specific data sources. For example, include social media monitoring data if it is geotagged to geographic areas of missed communities. • Consider local radio stations/community radio stations and online channels for those radio stations where local discussions about vaccine-related topics might take place. • For urban contexts, it is more likely that missed children and communities face fewer physical access barriers to receiving vaccines. Consider hyper-local data sources, such as Facebook pages for neighbourhoods or local media and interest-based WhatsApp groups where caregivers and parents may discuss child health topics. • Leverage tiplines where health workers or community champions may report vaccine concerns or misinformation to understand what is on top of mind for parents and caregivers. Understand the limitations of available programme data e.g.: o RCMs are conveniently sampled and are highly specific to locality. o Local coverage data may have issues with population estimates. Step3:Conductan integratedanalysisacross datasources Specificvulnerableorat-riskgroupsthatneedtobeconsideredintheriskassessment • Populations where there is an ongoing outbreak or are at the highest risk for an impending outbreak. • Populations and settings where childhood vaccination coverage is particularly low, e.g., economically-disadvantaged populations, mobile populations, ethnic minority, populations in geographically-isolated areas, among others. • Populations that have previously experienced vaccine hesitancy or a vaccine-related event that affected vaccine confidence. Epidemiologicalsignalsthatarerelevantwhenassessingrisk • Vaccination coverage data • Incidence rates of vaccine-preventable diseases • Surveillance reports of vaccine preventable disease outbreaks Typesofexpertisetoconsultorincludeinyourinfodemicinsightsteamtointerpretthedata • Technical expertise from immunization programme managers/health workers for interpreting coverage and disease incidence data • Behavioural science expertise for interpreting available social and behavioural data • Communications and community engagement expertise for interpreting implications of vaccination narratives in online and offline conversations Appendix 1: How to develop infodemic insights on zero-dose children 60 How to build an infodemic insights report in six steps Step Considerations Step4:developstrategies andrecommendations Existingframeworksorguidancethatmaybereferencedwhendeveloping recommendationsorstrategies • GAVI Zero-Dose Analysis Card https://www.gavi.org/sites/default/files/support/Gavi_Zero- dose_AnalysisCard.pdf • World Health Organization. (2022). Behavioural and social drivers of vaccination: tools and practical guidance for achieving high uptake. https://apps.who.int/iris/ handle/10665/354459 Commonrecommendationsforactionthatcouldbeconsidered • Involve community members and leaders in the microplanning process to improve estimates and strategies to reach zero-dose children. • Activate a network of advocates and partners (e.g., medical societies, faith-based organizations, civil society organizations, community-based groups) and provide a centralized platform for the coordination of messages and actions. • Equip community leaders and vaccination advocates with tools and resources to engage in effective vaccine conversations. • Engage in conversations on social media rather than just posting information, include links to credible sources of vaccine information. • Identify and address any barriers to accessing health information or health services that could make getting vaccinated difficult or inconvenient. • Consider the role of community members from communities with zero-dose children and how they might be reached through digital channels to share vaccine information and promote vaccine uptake. Step5:Developaninfodemic insightsreport The infodemic insights report should be made suitable for identified target groups or geographical area, for tailored local interventions. Step6:Disseminatethe infodemicinsightsreportand trackactionstaken Waystheinfodemicinsightsreportsshouldbeformattedanddisseminatedtoreachtarget audiencesonthistopic • An action-oriented brief will be produced on a regular basis and disseminated to local or district level immunization programme and health promotion officers to assist in planning to promote vaccine demand. Existingmeetings,networksorotherpublicationopportunitieswherethisinsightsreport couldbeshared • Liaise with the immunization programme manager to identify the timing, frequency and proper channels for sharing the infodemic insights report in relation to strategic plans to reach zero-dose children. • It will also be important to connect with focal point/s for the risk communication and community engagement response. 61 How to develop infodemic insights on polio outbreaks and campaigns Appendix 2 Step Considerations Step1:Choosethequestions thatinfodemicmanagement insightscouldhelpanswer Starterquestionsthatyoucanconsider • How are people in this community talking about polio vaccines? Who seems to be influencing or leading the conversation? • How have polio-related questions, concerns and misinformation shifted over time in this country? • How are people discussing the recent polio outbreak in District X? • How are people characterizing their experiences of the ongoing polio campaign in Country X? How can we improve planning for the next campaign round to address specific concerns? • Are there any gender-specific qualities to the conversation and perceptions related to polio vaccine in Community X? • How have past vaccine related events contributed to narratives on polio? Programmeorstakeholderbestplacedtoidentifythemostrelevantquestionandwould benefitmostfrominfodemicinsightsreportsonthatquestion The polio programme and the immunization progamme at national and subnational levels can all identify relevant questions and would benefit from insights reports. Additional stakeholders who may share their data and benefit from the insights report recommendations might include UNICEF, WHO, other funded partners doing polio demand generation and communication work, medical associations, and faith leaders. Step2:Identifyandselect datasourcesanddevelop ananalysisplanforeach datasource Datasourcesspecificallyavailableonthistopic Field studies e.g., KAPBs on vaccination, vaccine-specific digital listening reports, rapid convenience monitoring (RCM) reports. RCMs, usually done during supplemental immunization activities, routinely gather information on reasons for non-vaccination, field reports from health workers and the teams involved in campaign activities, media monitoring reports, independent monitoring reports, microplans, community meetings, community- level feedback mechanisms, vaccination coverage reports at the local level and available demographic and socio-behavioural data. Specificcriteriatoassessqualityofdatasourcesonthistopic Discuss with the polio M&E colleagues about potential data sources to leverage and their degree of confidence in the quality and reliability of different data sources which will vary from country to country. Understand the limitations of available programme data e.g.: • RCMs are conveniently sampled and are highly specific to locality. • Local coverage data may have issues with population estimates. Polio We are closer than ever to ending polio, but outbreaks and challenges persist, meaning that thousands of children are still not receiving polio vaccines. Interruptions of vaccination campaigns during the COVID- 19 pandemic in children living in remote, fragile and conflict-affected areas have all contributed to fewer children receiving polio vaccines. In some countries, these immunity gaps are resulting in polio outbreaks. Maintaining confidence in polio vaccines and promoting high uptake for them during polio campaigns is critical for reaching global polio eradication goals and protecting children from paralysis. Unfortunately, concerns and misinformation about polio vaccines continue to affect parents’ trust and willingness to have their children vaccinated. Therefore, developing effective social listening systems and developing infodemic insights is critical for understanding the information environment that parents in communities are living in so that we might better address those questions and concerns and address misinformation more effectively. Specialconsiderationsforpoliooutbreaksandcampaigns: 62 How to build an infodemic insights report in six steps Step Considerations Step3:Conductan integratedanalysisacross datasources Specificvulnerableorat-riskgroupsthatneedtobeconsideredintheriskassessment • Zero-dose children • Populations where there is an ongoing polio outbreak or with large immunity gaps in a specific birth cohort that are more likely to suffer an outbreak. • Populations and settings where routine immunization coverage is particularly low, e.g., economically disadvantaged populations, mobile populations, ethnic minority, populations in geographically-isolated areas, among others. • Populations that have previously experienced vaccine hesitancy or a vaccine-related event that affected vaccine confidence. Epidemiologicalsignalsthatarerelevantwhenassessingrisk • Vaccination coverage data for routine vaccines • Polio campaign coverage data • Polio vaccination drop out data • Acute Flaccid Paralysis (AFP) surveillance data • Incidence rates and features of polio outbreak (e.g. wild polio virus, vaccine-derived polio virus) Typesofexpertisetoconsultorincludeinyourinfodemicinsightsteamtointerpretthedata • Technical expertise from polio and immunization programme managers/health workers for interpreting coverage and disease incidence data • Behavioural science expertise for interpreting available socio-behavioural data • Communications and community engagement expertise for interpreting implications of vaccination narratives in online and offline conversations • Expertise from the community of focus with deep familiarity with language, cultural and social customs Step4:Developstrategies andrecommendations Existingframeworksorguidancethatmaybereferencedwhendeveloping recommendationsorstrategies • Global Polio Eradication Initiative (GPEI) Outbreak Preparedness & Response guidance • Global Polio Eradication Initiative (GPEI) The polio toolkit • UNICEF vaccine misinformation management field guide Commonrecommendationsforactionthatcouldbeconsidered • Ensure high quality polio content is widely available on official health authority and immunization partner web presences and social media channels and available offline in multiple formats and languages. • Develop and FAQ page that can be used to debunk common misinformation posted on the health authority web site and updated on a regular basis. • Engage with young people to improve vaccine and digital literacy by teaching skills to recognize health misinformation and how to talk to family members about vaccines. • Work with faith leaders to develop content that addresses concerns or misinformation that have a religious framing. • Address other health and social priorities that a community is facing and communicate clearly about them alongside offering polio vaccinations in future campaigns. • Set up Q&A and tipline for community health workers who get questions or concerns or health misinformation that they may struggle addressing. The tipline can factcheck health claims and provide supportive messages and materials to health workers. • Train health workers and community health workers on the ability to have effective vaccine conversations with parents and caregivers, address vaccine hesitancy and concerns as well as misinformation. 63 Step Considerations Step5:Developaninfodemic insightsreport The infodemic insights report should be made suitable for identified target groups or geographical area, for tailored local interventions. Step6:Disseminatethe infodemicinsightsreport andtrackactionstaken Waystheinfodemicinsightsreportsshouldbeformattedanddisseminatedtoreachtarget audiencesonthistopic • The timing of the infodemic insights reports should be determined by the cadence of outbreak response and immunization campaign rounds. This means that infodemic monitoring and social listening activities should be activated before the outbreak, where possible, and surge with additional capacity to be able to develop a more rapid insights that leverage more complex data sources that become available during campaigns. • Different immunization partners will have different information needs. Any infodemic insights report should be formatted and delivered in a way that is most appropriate and useful to them. Existingmeetings,networksorotherpublicationopportunitieswherethisinsightsreport couldbeshared • Liaise with the polio or immunization programme manager to identify the timing, frequency and proper channels for sharing the infodemic insights report in relation to the cadence of polio programme activities. • If the EOC has been activated for a polio outbreak, include infodemic insights reports in standing meetings, situational reports, and other times when polio programme staff and partners are planning or discussing campaign performance. References: 1. Global Polio Eradication Initiative (GPEI) 2. Global Polio Eradication Initiative (GPEI) The polio toolkit Appendix 2: How to develop infodemic insights on polio outbreaks and campaigns 64 How to build an infodemic insights report in six steps WHO recommends that seasonal influenza vaccination be administered annually to health workers, older adults, individuals with comorbidities and underlying conditions, pregnant women, and other populations (e.g. children) as determined by national contexts. Infodemic monitoring and insights reports on seasonal influenza vaccination programmes can help countries to tailor their annual vaccination campaigns to understand and address circulating mis and dis-information. Through infodemic management, countries can identify barriers to uptake and implement solutions to improve vaccine acceptance and demand. Specialconsiderationsfor: Step Considerations Step1:Choosethequestions thatinfodemicmanagement insightscouldhelpanswer Starterquestionsthatyoucanconsider: • What are key concerns about seasonal influenza vaccination? • Are particular populations or geographic areas where misinformation and concerns about influenza vaccines are frequently circulating? • Have questions, concerns and rumours about seasonal influenza changed over time in a priority population? • If monitoring for a long period of time that allows for detection of trends: Is there a seasonality to conversations, concerns and misinformation about seasonal influenza? How does it compare to the actual influenza season? Programmeorstakeholderbestplacedtoidentifythemostrelevantquestionandwould benefitmostfrominfodemicinsightsreportsonthatquestion The team responsible for seasonal influenza vaccination at the national, regional, and local levels would be best places to identify their key question(s) and would benefit most from the insights reports. These teams might be part of immunization programmes, infectious disease units, disease surveillance units, etc. Step2:Identifyandselect datasourcesanddevelopan analysisplanforeachdata source Datasourcesspecificallyavailableonthistopic Data sources may include: social media, media outlets, internet forums, vaccinators, health facility leadership, community/religious leaders, community health workers, and hotlines. Additionally, socio-behavioural research that might include surveys, focus group discussions, or key informant interviews related to seasonal influenza may also be available. Specificcriteriatoassessqualityofdatasourcesonthistopic WHO monitors influenza vaccination policy and programme characteristics through the WHO/ UNICEF electronic Joint Reporting Form (eJRF). Country level influenza vaccination policy and vaccine availability data are collected annually. The eJRF can be used to understand the current state of influenza vaccination in a given country. How to develop infodemic insights on seasonal influenza vaccination Appendix 3 Seasonal influenza 65 Step Considerations Step3:Conductan integratedanalysisacross datasources Specificvulnerableorat-riskgroupsthatneedtobeconsideredintheriskassessment Target populations for seasonal influenza vaccination which should be outlined in the national influenza vaccination policy. For countries initiating for just expanding their vaccination programmes, WHO recommends: • Priority target groups: health workers, individuals with comorbidities/underlying conditions, older adults, and pregnant women • Additional groups to consider: children, individuals living in congregate-living settings (e.g., prisons, refugee camps), disadvantaged populations, indigenous populations Target group selection varies by country and should be driven by national disease burden, priorities, resources, and programmatic feasibility. Epidemiologicalsignalsthatarerelevantwhenassessingrisk For seasonal influenza vaccination, the epidemiology of the influenza season should be taken into consideration. Seasonal influenza vaccines should be given prior to the start of the primary period of increased influenza activity. For the Northern Hemisphere, primary influenza activity typically starts after October. For the Southern Hemisphere, primary influenza activity typically starts after April. Countries in the tropics and sub-tropics which experience multiple peaks of influenza activity year-round are recommended to analyze local surveillance data to determine the appropriate timing of influenza vaccination campaigns and the corresponding influenza vaccine formulation. This has implications for timing of risk communication and vaccine demand generation efforts which may fuel broader public attention and discussion on this topic. Typesofexpertisetoconsultorincludeinyourinfodemicinsightsteamtointerpretthedata Individuals with experience in one or more of the following areas: influenza epidemiology, seasonal influenza vaccination, vaccine safety, obstetrics and gynecology, quantitative and qualitative data analysis, risk communication and community engagement, customs/ requirements of the religious groups in the country, and expanded programme on immunization. Step4:Developstrategies andrecommendations Existingframeworksorguidancethatmaybereferencedwhendeveloping recommendationsorstrategies WHO has developed manuals to support seasonal influenza vaccination for two populations: • Health workers – How to implement seasonal influenza vaccination of health workers: an introduction manual for national immunization programme managers and policy makers: https://apps.who.int/iris/handle/10665/325906 • Pregnant women – How to implement influenza vaccination of pregnant women: An introduction manual for national immunization programme managers and policy makers: https://apps.who.int/iris/handle/10665/250084 Commonrecommendationsforactionthatcouldbeconsidered 1. Ensure the eligible populations can receive seasonal influenza vaccination for free 2. Provide seasonal influenza vaccination during day and night shifts of health facilities to enable easy access by health workers and target populations 3. Provide health facilities with campaign communications materials that explain who is eligible for vaccination, the benefits and risks, and where/when to access the vaccine 4. Create a network of champions: Encourage health facilities to ask vaccinated health workers to speak to other health workers and eligible populations about their decision to get vaccinated 5. Have a website and national hotline (phone and text) where people can ask questions and receive timely, accurate information about seasonal influenza vaccination and the campaign Appendix 3: How to develop infodemic insights on seasonal influenza vaccination 66 How to build an infodemic insights report in six steps Step Considerations Step5:Developaninfodemic insightsreport The infodemic insights report should include both challenges and potential solutions identified for the different target groups. If differences were identified by geographical area, this should be included in the report to support tailored local actions. Step6:Disseminatethe infodemicinsightsreportand trackactionstaken Waystheinfodemicinsightsreportsshouldbeformattedanddisseminatedtoreachtarget audiencesonthistopic Information about the seasonal influenza vaccination campaign is typically communicated to national, regional, and local stakeholders through an annual circular. Key findings from the infodemics insights report may be helpful to include in this circular. The findings should also be shared with the National Immunization Technical Advisory Group (or similar advisory body). Existingmeetings,networksorotherpublicationopportunitieswherethisinsightsreport couldbeshared It would be helpful to share these findings are national preparatory or end of campaign meetings focused on the seasonal influenza vaccination programmes as well as at regional meetings focused on vaccination or vaccine acceptance and demand. Where possible, insights report on seasonal influenza should accompany the surveillance reports on seasonal influenza. KeyResources: 1. Seasonal Influenza Vaccines: An Overview for Decision-Makers 2. Influenza prevention and control: OpenWHO course 3. Additional resources including campaign materials for seasonal influenza vaccination can be found in the Seasonal Influenza Vaccination Toolkit Additionalinformation: WHO is developing a quantitative and qualitative tool to improve influenza vaccination acceptance and demand data. The tool is anticipated to be field tested and validated in Q3 -Q4 2023. When available, the tool and key findings will published here 67 Public concerns about vaccine safety occur during routine immunization activities and campaigns and may increase during vaccine safety events such as adverse events following immunization (AEFIs). If not properly monitored, addressed and coordinated with programme implementation, misinformation on vaccination can further fuel concerns and negatively affect vaccine confidence and uptake, and trust in health authorities. Infodemic monitoring and insights reports can help inform immunization programme strategies to effectively address vaccine safety concerns. Specialconsiderationsforvaccinesafetyevents: Step Considerations Step1:Choosethequestions thatinfodemicmanagement insightscouldhelpanswer Starterquestionsthatyoucanconsider • What concerns and questions about vaccines currently exist in community X? • What misinformation narratives have emerged? Questions and concerns about vaccines can resurface and can fuel new narratives. • How are these concerns, questions and narratives affecting peoples’ intent to vaccinate? Can further be unpacked by using the framework on Behavioural and Social Drivers (BeSD) on immunization. Programmeorstakeholderbestplacedtoidentifythemostrelevantquestionandwould benefitmostfrominfodemicinsightsreportsonthatquestion Immunization programme managers can best identify the most relevant questions and use the insights recommended by the infodemic insights report in driving toward interventions. The interventions will be context-specific and coordinated with immunization programme implementation. Step2:Identifyandselect datasourcesanddevelopan analysisplanforeachdata source Datasourcesspecificallyavailableonthistopic Field studies e.g., KAPBs on vaccination, vaccine-specific digital listening reports, rapid convenience monitoring (RCM) reports. RCMs, usually done during supplemental immunization activities, routinely gather information on reasons for non-vaccination, risk assessment tools for specific conditions, field reports from health workers and the teams involved in reporting and investigating adverse events following immunization e.g., AEFI/AESI reporting forms, vaccination coverage reports at the local level and available demographic and socio-behavioural data. Specificcriteriatoassessqualityofdatasourcesonthistopic Check the sensitivity of digital social listening datasets by considering the following: • Boolean search strings that have been used, including variations on words related to vaccination, names of vaccines, and diseases they prevent • Timeliness of sampling, important in noting narratives • Consider internet penetration and representativeness of vaccine concerns captured through digital social listening Understand the limitations of available programme data e.g.: • RCMs are conveniently sampled and are highly specific to locality. • Local coverage data may have issues with population estimates. How to develop infodemic insights on vaccine safety events Appendix 4 Vaccine safety events 68 How to build an infodemic insights report in six steps Step Considerations Step3:Conductan integratedanalysisacross datasources Specificvulnerableorat-riskgroupsthatneedtobeconsideredintheriskassessment • Populations where there is an ongoing outbreak or are at the highest risk for an impending outbreak. • Populations and settings where coverage is particularly low, e.g., economically disadvantaged populations, mobile populations, ethnic minority, populations in geographically-isolated areas, among others. • Populations that have previously experienced vaccine hesitancy or a vaccine-related event that affected vaccine confidence. Epidemiologicalsignalsthatarerelevantwhenassessingrisk • Vaccination coverage data • Incidence rates of vaccine-preventable diseases • Past vaccine safety surveillance reports – past vaccine safety events influence current public perceptions Typesofexpertisetoconsultorincludeinyourinfodemicinsightsteamtointerpretthedata • Technical expertise from immunization programme managers/health workers for interpreting coverage and disease incidence data • Behavioural science expertise for interpreting available social and behavioural data • Communications and community engagement expertise for interpreting implications of vaccination narratives in online and offline conversations Step4:Developstrategies andrecommendations Existingframeworksorguidancethatmaybereferencedwhendeveloping recommendationsorstrategies • World Health Organization. (2022). Behavioural and social drivers of vaccination: tools and practical guidance for achieving high uptake. https://apps.who.int/iris/ handle/10665/354459. Commonrecommendationsforactionthatcouldbeconsidered Note: Responding to vaccine safety-related events should be closely aligned and coordinated with actions from the immunization programme and vaccine safety surveillance teams. • Activate a network of advocates and partners (e.g., medical societies, faith-based organizations, civil society organizations, community-based groups) and provide a centralized platform for the coordination of messages and actions • Equip community leaders and vaccination advocates with tools and resources to engage in conversations on the issue • Engage in conversations on social media rather than just posting information, include links to credible sources of vaccine information. • Update relevant web pages (of the Ministry of Health, professional medical societies) with information (e.g., Q & As on the issue, resources that can be amplified in other channels) and with features for search engine optimization. • Equip health workers with messages and skills in handling difficult conversations about vaccine safety. • Closely monitor the activity of groups of vocal vaccine deniers and related personalities. Note emerging narratives and expansion of networks. Use the insights to preposition messaging or plan strategies in case of further increase in vaccine denialism activity. • Identify and address any barriers to accessing health information or health services that could make getting vaccinated difficult or inconvenient. 69 Step Considerations Step5:Developaninfodemic insightsreport The infodemic insights report should be made suitable for identified target groups or geographical area, for tailored local interventions. Step6:Disseminatethe infodemicinsightsreportand trackactionstaken Waystheinfodemicinsightsreportsshouldbeformattedanddisseminatedtoreachtarget audiencesonthistopic • An action-oriented brief will be produced monthly or during initial roll out or supplementary immunization phases. Existingmeetings,networksorotherpublicationopportunitieswherethisinsightsreport couldbeshared • Liaise with the immunization programme manager to identify the timing, frequency and proper channels for sharing the infodemic insights report in relation to ongoing processes on the programme side (e.g., AEFI investigation, causality assessment, etc.). • It will also be important to connect with focal point/s for the risk communication and community engagement response. References: 1. World Health Organization. (2020). COVID-19 vaccines: safety surveillance manual. 2. World Health Organization. Regional Office for Europe. (2017). Four immediate steps when responding to an event that may erode trust: World Health Organization vaccine safety supporting document. World Health Organization. 3. World Health Organization. Western Pacific Regional Office. (2016). Vaccine safety communication: guide for immunization programme managers and national regulatory authorities. Appendix 4: How to develop infodemic insights on vaccine safety events 70 Strong community relationships and regular social listening are crucial for effective disease outbreaks and emergency response during mass gatherings. WHO advises using a risk-based preparedness approach to decide on modifying (holding postponing or cancelling) gatherings, tailored to the event›s characteristics and context, repeated regularly during planning and the conducting event phase, and adapting the response in a timely manner. The All-Hazards Mass Gatherings Risk Assessment tool identifies and estimates public health risks due to range of hazards, assesses event preparedness, and estimates the host country’s capacity to respond to potential health impacts; an infodemic insights report can help inform the risk assessment and planning. Specialconsiderationsfor: Step Considerations Step1:Choosethequestions thatinfodemicmanagement insightscouldhelpanswer Starterquestionsthatyoucanconsider: • How are people discussing the mass gathering and related health and safety issues? • Are there influencers or opinion leaders who shape the conversation and how in relation to this mass gathering and health? • If there is an existing outbreak elsewhere: How are narratives around the outbreak/ pathogen intersecting with discussions about the mass gathering? • How is official guidance to mitigate spread of infection in relation to the mass gathering being understood and reacted to by the public? Programmeorstakeholderbestplacedtoidentifythemostrelevantquestionandwould benefitmostfrominfodemicinsightsreportsonthatquestion • Mass gathering event organizers • Public health officials and Ministry of Health officials • Other stakeholders whose involvement in planning is critical (i.e. security forces • Community leaders • Community Health Teams • Risk Communication and Community Engagement Teams • Media covering the event Step2:Identifyandselect datasourcesanddevelopan analysisplanforeachdata source Datasourcesspecificallyavailableonthistopic • WHO Mass Gatherings Database • WHO social media monitoring for high visibility events • Events’ situational reports • Peer-reviewed research • Grey literature • Social media platforms • Official social media channels for the event • Local news media • Surveys and polls • Public health authorities • Event organizers • Event mobile applications • Travel and tourism operators • Mass transit and rideshare operators How to develop infodemic insights to inform planning for mass gatherings to prevent outbreaks Appendix 5 Mass gatherings 71 Step Considerations Specificcriteriatoassessqualityofdatasourcesonthistopic • Availability of data • Freshness of data • Ability to attribute specific questions or concerns to the specific event (e.g. references to the event specifically or use of a common hashtag) Step3:Conductan integratedanalysisacross datasources Riskassessmentcriteria development Specificvulnerableorat-riskgroupsthatneedtobeconsideredintheriskassessment Considering the below vulnerable or at-risk groups can help to ensure that the information provided during mass gatherings is accessible, relevant, and useful to all attendees, and that the unique needs and challenges of these groups are considered. Context and event specific: 1. Attendees of the event 2. Event organizers and staff 3. Host community of the event 4. Specific groups that are identified in the pre-event assessment planning, which may include: a. Children and youth b. Elderly c. Pregnant women d. Individuals with chronic illnesses or compromised immune systems: e. Migrants and refugees f. Low-income communities g. Rural populations Epidemiologicalsignalsthatarerelevantwhenassessingrisk • Event based surveillance • Timeline of the reported cases/public health events in relation to the gathering events (before, during or after the event) • Disease incidence and prevalence • Mode of disease transmission • Vaccination coverage • Health status of attendees • Disease severity and mortality rate • Geographic distribution Typesofexpertisetoconsultorincludeinyourinfodemicinsightsteamtointerpretthedata 1. Epidemiologists: They can provide insights into the disease outbreaks or health risks associated with mass gatherings. 2. Communication experts: They can help develop and deliver clear and effective messaging to the public and other stakeholders. 3. Social media analysts: They can help monitor and analyze social media platforms to identify emerging rumours and misinformation related to the event. 4. Public health experts: They can provide guidance on how to prevent or control the spread of infectious diseases during the event. 5. Community engagement specialists: They can help facilitate communication and engagement with the community and ensure that the public’s needs and concerns are heard and addressed. 6. Data analysts: They can help analyze and interpret data from various sources, including social media platforms, news outlets, and public health agencies, to identify trends and patterns that may be relevant to the event. 7. Risk assessment specialists: They can help develop a risk assessment framework that takes into account the specific characteristics and context of the mass gathering event. Appendix 5: How to develop infodemic insights to inform planning for mass gatherings to prevent outbreaks 72 How to build an infodemic insights report in six steps Step Considerations Step4:Developstrategies andrecommendations Existingframeworksorguidancethatmaybereferencedwhendeveloping recommendationsorstrategies • Public health for mass gatherings: key considerations • Public health advice for gatherings during the current monkeypox outbreak, 28 June 2022 • WHO mass gathering COVID-19 risk assessment tool: generic events, version 3 • WHO mass gathering COVID-19 risk assessment tool: generic events, version 3 Web Application • WHO generic all-hazards risk assessment and planning tool for mass gathering events • WHO mass gathering COVID-19 risk assessment tool – Generic events • WHO mass gathering COVID-19 risk assessment tool – Sports events • WHO mass gathering COVID-19 risk assessment tool – Religious events • WHO Policy Brief: Gatherings in the context of COVID-19, 19 January 2023 Commonrecommendationsforactionthatcouldbeconsidered • Improve availability and accessibility of health information related to the mass gathering event, including online, offline and multiple languages and formats • Provide frequently asked questions section on the mass gathering event’s web site where questions, concerns and misinformation might be addressed • Use geolocation-specific functions on social media pages to make it easier for people to locate different aspects of the venue related to health and safety (such as bathrooms, medical stations, security, etc.) • Ensure that influencers and other leaders that are officially connected to the mass gathering event are equipped with messages about the specific health topic that can be shared about multiple formats • Promote the use of an event-specific hashtag in digital conversations so that people can ask questions and get answers quickly • Event organizers should proactively share health information with registered attendees and encourage them to take action to prevent disease transmission (e.g. “get vaccinated before you pack your suit case”) • Be sure to indicate to attendees what official sources of information they should trust and share related to the mass gathering event, including the appropriate use of logos and consistent design Step5:Developaninfodemic insightsreport The infodemic insights report should include both challenges and potential solutions identified for the different target groups, including the event organizers and staff, attendees and the host community. If differences were identified by geographical area, this should be included in the report to support tailored local actions. 73 Step Considerations Step6:Disseminatethe infodemicinsightsreportand trackactionstaken Waystheinfodemicinsightsreportsshouldbeformattedanddisseminatedtoreachtarget audiencesonthistopic The format and dissemination of infodemic insights reports for mass gatherings may vary depending on the target audience and the context of the event. However, here are some general recommendations: • Use clear and concise language • Use visual aids • Tailor the report to the mass gathering target audience • Create two versions of the report: one with technical information for experts and another simplified version for the general public. • Provide actionable recommendations: The report should include clear, practical recommendations that can be easily implemented before and early in the event. This can include specific steps that event organizers or attendees can take to reduce the spread of misinformation and protect themselves and others from health risks. • Use multiple channels for dissemination including online and through the media, and all official event channels (e.g. web site, app, information booth) • Engage with stakeholders: Work with event organizers, local health authorities, and other stakeholders to ensure the report is relevant, timely, and useful. Solicit feedback on the report and consider making updates based on this feedback. • Include translations into different languages of the participants of a mass gatherings event to ensure the dissemination of the infodemic insights reports reaches the target audience, taking into account linguistic diversity and enabling individuals to access the information in their preferred language, thus maximizing the impact and understanding of the message Existingmeetings,networksorotherpublicationopportunitieswherethisinsightsreport couldbeshared • Regular event organizer meetings • Meetings between organizers and the health authority and other government stakeholders • Through internal communication by event organizers and event staff and partners • If appropriate, through event web site and official channels References: 1. Public health for mass gatherings: key considerations 2. WHO Policy Brief: Gatherings in the context of COVID-19, 19 January 2023 3. The generic all-hazards risk assessment and planning tool for mass gathering events Appendix 5: How to develop infodemic insights to inform planning for mass gatherings to prevent outbreaks