WHO guideline recommendations on digital interventions for health system strengthening Research Considerations WHO/RHR/19.9 © World Health Organization 2019 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. This publication forms part of the WHO guideline: recommendations on digital interventions for health system strengthening. It is being made publicly available as supplied by those responsible for its development for transparency purposes and information, as required by WHO (see the WHO handbook for guideline development, 2nd edition (2014)). Suggested citation. WHO guideline: recommendations on digital interventions for health system strengthening. Research considerations. Geneva: World Health Organization; 2019. (WHO/RHR/19.9) Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party- owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Editing: Green Ink, United Kingdom Design and layout: RRD Design LLC W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 1 Future research This chapter on future research highlights crosscutting evidence gaps observed across a range of interventions in relation to effectiveness, resource use and cost-effectiveness, and gender, equity and rights. In addition, specific research questions are provided for each of the interventions, based on the gaps identified through the evidence-to-decision framework and GDG. 1.1 Overarching research gaps The following sections describe the overarching research priorities identified through this guideline process. These reflect the general areas in which the available evidence was found to be of low or very low certainty or confidence, or where no direct evidence was identified. Where studies were available, in some cases the certainty or confidence of the evidence was affected by poor reporting of outcomes, studies with small numbers of participants, and limited representation across different settings. Annex 6 maps the state of evidence and its gaps based on the findings from the effectiveness reviews for the included digital health interventions. Effectiveness For many of the interventions, the available evidence on effectiveness was sparse. Future research should measure health system process improvements that may immediately result from the digital intervention, such as health workers’ adherence to recommended practice, as well as related distal health outcomes. Researchers should be realistic about the extent to which digital health interventions can impact on distal health outcomes, which are often affected by a variety of factors beyond the interaction with the digital intervention. Additionally, effectiveness studies need to include ways of concurrently monitoring technological performance (for example, do messages reach intended individuals?) and behavioural performance or user engagement (e.g. do individuals who get messages listen to or read them, and subsequently act on them?). W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 2 Resource use and cost-effectiveness The studies included in the systematic reviews of the effectiveness of the digital interventions considered by the guideline identified limited evidence on the resources used to implement these interventions. Costing studies should assess costs over a longer period, with appropriate accounting of amortization and maintenance of equipment and the continuous user support required. Future research should explore the cost-effectiveness, and potential for cost savings of the identified intervention and additional savings achieved through combining interventions. Gender, equity and rights Further research needs to encompass a wider range of contexts and populations, including populations with poor access to digital or conventional health services, in order to better understand and mitigate any potential negative impacts on gender, equity and rights. Key research questions include how digital health interventions can help to reduce disparities in linking to the wider health system and whether these interventions may create further inequities in some settings as a consequence of poor network coverage, limited control of mobile devices, or a lack of other resources. Research should also explore unintentional exacerbation of inequities based on who has access to digital devices, and who has access to network connectivity. Implementation research Due to the strong focus on integrated health systems and interoperability, future research should also examine the synergies across different combinations of digital health interventions to determine which packages of interventions are most effective and cost-effective. Addressing this question is important given the potential complexity of implementing packages of digital interventions and the costs of establishing and maintaining these systems. Specific questions include the following. Ⱥ What is the feasibility and effectiveness of different combinations of digital health interventions? Ⱥ What are the non-digital health and supporting interventions (for example, enhanced transportation, supervision) that should be packaged together with digital health interventions to ensure their effectiveness, acceptability and feasibility? Ⱥ What are the minimum requirements of a country’s enabling environment (infrastructure, governance, workforce, interoperability and standards) to support the different recommended digital health interventions? Ⱥ How can the fidelity (i.e. the roll out of all the critical components of the intervention as intended) of implementation at scale be facilitated? Frameworks such as RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) may be useful in structuring the implementation research (87). W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 3 1.2 Considerations for the design of future evaluations The GDG also identified several issues related to the design of future evaluations of digital health interventions, including the following: Ⱥ Health system focused digital interventions, such as stock management and birth and death notification, are often complex in the number of components, behaviours targeted, and organizational levels involved (89). These factors may make designs such as randomized controlled trials for evaluating the effectiveness of these interventions difficult to apply. Other designs may therefore need to be considered, such as controlled before-and-after studies, stepped-wedge randomized controlled trials and interrupted time series studies. Ⱥ While there is value in evaluating changes in client/patient health outcomes, intermediate outcomes are also critical for the evaluation of digital health interventions. For example, the effect of decision support on client/patient health outcomes are influenced not only by the information delivered through the digital system, but also by a host of other factors, including access to medicines, their cost, family support, and biomedical factors such as whether the individual responds appropriately to recommended treatments or has comorbidities. A logical framework of how the digital intervention functions may be helpful in understanding the pathways through which the intervention influences a targeted behaviour or health system challenge and in selecting appropriate outcomes along these pathways. Ⱥ Digital technologies provide new opportunities to capture research data for measuring the effectiveness of implementations in real time, thus facilitating the ability to conduct evaluations more rapidly. Incorporating the research data collection needs for primary and secondary outcomes of interest at the design stage can ensure that the data needed to measure these outcomes is captured alongside the implementation. Ⱥ Rapid changes in digital technologies and the iterative approaches often used for software development may force digital health interventions to evolve during evaluation periods, which may pose challenges for the evaluation process. Detailed process evaluations running alongside impact evaluations may be helpful in understanding the effects of incremental changes in the digital interventions over time. Ⱥ Future research efforts should establish common metrics and tools for assessing the effectiveness and cost-effectiveness of digital health interventions W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 4 Evidence maps and illustrative research questions The tables below illustrates the general trends in the evidence found in the effectiveness reviews, demonstrating low and very low certainty evidence across most interventions. For more details on the specific interventions and outcomes, please review the summary of findings in Web Supplement 1. In addition, specific research gaps and accompanying illustrative research questions are listed Table A5.4. These questions should be addressed using rigorous methods. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 5 Table A5.1 Effectiveness evidence for client interventions Digital intervention Unintended consequences Resource use Satisfaction and acceptability Utilization of health services Health behaviour, status and well-being TCC – adolescents TCC – adults TCC – pregnant + postpartum TCC – pregnant + postpartum with HIV TCC – children <5 Client-to- provider telemedicine TCC stands for targeted client communication. This intervention was reviewed across five population groups. This table does not reflect information on satisfaction and acceptability obtained from qualitative reviews. The comparison for all interventions reflected on these tables is standard care. Please see Web Supplement 1 for other comparison groups for TCC. Table A5.2 Effectiveness evidence for health worker (HW) interventions Digital intervention Unintended consequences Resource use Satisfaction/ acceptability HW performance HW skills/ attitudes HW knowledge Clients’ utilization of health services Clients’ health behaviour, health status/ well-being Provider-to- provider telemedicine Decision support Decision support + digital tracking Decision support + digital tracking + TCC mLearning Table A5.3 Effectiveness evidence for Health system interventions Digital intervention Unintended consequences Resource use Satisfaction/ acceptability Coverage of birth/death notification Timeliness of birth /death notification Coverage of newborn or child health services Timeliness of newborn or child health services Availability of commodities Quality and timeliness of stock management Birth notification Death notification Stock notification Key UNKNOWN LITTLE OR NO DIFFERENCE POSITIVE EFFECT NEGATIVE EFFECT Not applicable/Not measured May make little or no difference (low certainty evidence) May have benefits (low certainty evidence) May lead to harm (low certainty evidence) Uncertain effect because of very low certainty evidence Probably makes little or no difference (moderate certainty evidence) Probably has benefits (moderate certainty evidence) Probably leads to harm (moderate certainty evidence) no incidence No evidence identified Makes little or no difference (high certainty evidence) no incidence Has benefits (high certainty evidence) no incidence Leads to harm (high certainty evidence) no incidence Size of bubbles reflects the number of studies contributing to the outcome 1-37-10 4-6 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 6 Intervention-specific research gaps Table A5.4 outlines the specific research gaps, with illustrative research questions, identified for each of the interventions included in the guideline. These research questions should be addressed using rigorous methods. Table A5.4 Research gaps Intervention Evidence- to-decision domain Research gaps and illustrative research questions Birth and death notification Effectiveness Ⱥ What is the effect of birth and death notification on the quality and timeliness of birth and death reporting or on the accountability for responding to the data? Ⱥ Does notification by mobile devices lead to more timely and complete legal registration, in the case of births, increased coverage and timeliness of health and other social services (e.g. vaccination), or in the case of deaths, increased recording of the causes? Acceptability Ⱥ What is the acceptability of birth and death notification via mobile devices, rather than through standard practices of notification? Research should include how these interventions interact with the sociocultural norms and needs of different communities regarding births and deaths and their notification. Feasibility Ⱥ What are the legal, ethical, data security and policy requirements for allowing new groups of people or cadres of health worker to notify births and deaths? What types of modification to existing legal frameworks would be needed to implement birth and death notification by mobile devices at national scale? Ⱥ What are the most appropriate ways to train health workers and other people designated to use birth and death notification? Ⱥ In what ways do birth (and infant death) notification provide opportunities to link maternal health records with child health outcomes? Resource use Ⱥ See overarching research gaps in section 5.1 Gender, equity and rights Ⱥ How does this intervention increase or decrease health-related disparities? Are there population groups or settings that may not be able to benefit from this intervention, and how can this be addressed? W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 7 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Stock notification and commodity management Effectiveness Ⱥ What is the effect of stock notification and commodity management via mobile devices on improved availability/reduced stock-out of commodities at the point of care? Ⱥ What are the health system conditions that contribute to the effectiveness of this intervention (for example, supervision of health workers, effective transport of products, drug access/purchase policies)? Ⱥ Future research should also be conducted across a range of settings. Acceptability Ⱥ No research gaps identified Feasibility Ⱥ How can digital stock notification and commodity management systems be implemented so that they are aligned closely with both national ordering routines and local needs, and are also supported by well-functioning national and subnational commodity management? Ⱥ What can be learnt from practices in logistics management information systems used outside of the health sector that may be applicable to primary health care settings? Resource use Ⱥ What are the potential cost savings from introducing digital stock notification, for example through reducing the need for buffer stock and improving the accuracy of stock need forecasts? Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 8 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Client1-to- provider telemedicine Effectiveness Ⱥ What types of digital channel used in facilitating client-to-provider telemedicine are most effective (for example, transfer of images, voice, text, and other delivery channels)? Under which circumstances should these different channels be used? Ⱥ Future research should include the following outcomes: Ⱥ use of health services Ⱥ health behaviour, status and well-being Ⱥ health worker and client satisfaction Ⱥ unintended consequences, including the specific risks and safety concerns for implementing telemedicine different health domains or conditions. Acceptability Ⱥ How does this intervention influence health workers’ ability to communicate or explain information to clients, including issues of liability? Linked to this, in what ways does this intervention change interactions between clients/patients and health workers? Ⱥ Further research in low- and middle-income settings is especially needed. Feasibility Ⱥ What mechanisms can address identified implementation barriers, such as concerns about data privacy obtaining informed consent, and challenges in network connectivity that may compromise the quality of information exchanged (e.g. loss of quality of image files, interrupted connection)? Resource use Ⱥ What are the resources needed to implement client-to-provider telemedicine, and what is the cost-effectiveness of this intervention? This should include research on the cost-effectiveness of different delivery channels, such as voice- based consultations, image exchanges and other modalities to facilitate client- to-provider telemedicine for different health issues. Gender, equity and rights Ⱥ How does this intervention increase or decrease health-related disparities? Are there population groups or settings that may not be able to be able to benefit from this intervention, and how can this be addressed? 1 Although WHO’s Classification of digital health interventions v1.0 uses the term “client” (13), the terms “individual” and “patient” may be used interchangeably, where appropriate. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 9 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Provider- to-provider telemedicine Effectiveness Ⱥ What are the conditions that contribute to the effectiveness of provider-to- provider telemedicine? Ⱥ Future research should include the following outcomes: » health worker performance and adherence to recommended practice, quality of care provision » health behaviour, status and well-being » health worker and client satisfaction » unintended consequences, including the specific risks and safety concerns for implementing telemedicine different health domains or conditions. Acceptability Ⱥ How is provider-to-provider telemedicine perceived by health workers to influence inter-professional interactions and collaboration? Feasibility Ⱥ What are the potential barriers to implementing these interventions, and how can these be mitigated? Such barriers include, for example, challenges in connectivity and its resulting consequences on the quality of information exchange (e.g. loss of quality of image files, interrupted connections). Resource use Ⱥ What are the resources needed to implement provider-to-provider telemedicine, and what is the cost effectiveness of this intervention? This should include research on the cost-effectiveness of different delivery channels, such as voice-based consultations, image exchanges and other modalities, to facilitate provider-to-provider telemedicine for different health issues. Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 10 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Targeted client communication Effectiveness Ⱥ How does the frequency, dose, delivery channel and overall exposure to content of targeted client communication affect behaviour change and health outcomes? Ⱥ Future research on effectiveness should consider the following outcomes: » use of health services » health behaviour, status and well-being » satisfaction with services » knowledge and attitudes (for adolescent populations) » unintended consequences. Acceptability Ⱥ Most studies to date have asked people about their views were they to receive targeted communications via mobile devices, while some studies have evaluated people’s experiences within pilot projects or randomized trials. Future research should focus on the views of participants involved in national- scale targeted client communication programmes. Ⱥ What is the acceptability of different formats and delivery mechanisms across different sociocultural contexts and population groups, such as adolescents? Feasibility Ⱥ What strategies can be used to address privacy concerns and to mitigate any potential negative effects of transmitting sensitive health content, including ways to enforce consent and the ability to opt out of programmes? Ⱥ What ways can be used to maintain contact with clients who regularly change their phone numbers, or who have limited or shared access to mobile devices? Resource use Ⱥ What is the cost-effectiveness of different delivery channels, such as voice, text messages, USSD, and smartphone applications? Gender, equity and rights Ⱥ What strategies can be used to ensure equal access to and use of targeted client communication services for all groups, including people with poor access to mobile devices and/or poor network coverage, people who speak minority languages and people with low literacy or poor technological literacy and skills? Ⱥ Future research assessing the effectiveness of targeted client communication using mobile devices should make efforts to ensure that disadvantaged populations are included. Trials should avoid excluding, wherever possible, participants on the basis of mobile device ownership, literacy levels, language or participation in formal health care programmes. Other Ⱥ Where possible, research should take an integrated approach that includes outcomes across the continuum of care in pregnancy, childbirth and child health, as well as across sexual and reproductive health in general. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 11 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Health worker decision support Effectiveness Ⱥ What is the effectiveness of health worker decision support via mobile devices across different settings, health domains, levels of health care, and among health workers with different levels of training? Future research should focus on these outcomes: » health worker performance and adherence to recommended practice, quality of care provision » clients’/patients’ use of services » clients’/patients’ health behaviour, status and well-being » health worker and client satisfaction » unintended consequences. Acceptability Ⱥ How is decision support via mobile devices perceived by health workers and clients, and how does it influence their interactions in the provision of services? Feasibility Ⱥ What mechanisms can be used to validate the health content within decision support systems, to ensure that the recommended clinical practices are congruent with the best available evidence? Resource use Ⱥ See overarching research gaps in section 5.1 Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 Other Ⱥ What mechanisms can be used to ensure that decision support tools evolve with new clinical evidence and subsequent policy changes? The development of the clinical algorithms used within decision support systems is presently an inexact science. Further research is needed to identify best practice, to develop and refine these algorithms both in terms of their clinical effectiveness and their ease of use and acceptability for health workers and clients. The use of artificial intelligence for the development of decision support systems is an emerging area that may help to refine algorithms, but more research is needed on acceptability, feasibility and ethics. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 12 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Digital tracking with decision support and targeted client communication Effectiveness Ⱥ What is the effectiveness of digital tracking across different settings and health domains? Research should focus on these outcomes: » health worker performance and adherence to recommended practices; quality of care provision » clients’/patients’ use of health services, including follow-up services » quality of data on the services provided » clients’/patients’ health behaviour, status and well-being » health worker and client satisfaction » unintended consequences. Acceptability Ⱥ What approaches can be used to minimize the dual burden on health workers of operating paper and digital systems? Feasibility Ⱥ What are the policy requirements for transitioning from paper to digital systems for client health records, including the establishment and institutionalized use of unique identification mechanisms? Ⱥ What are the implementation approaches and requirements for maintaining a longitudinal client record across the continuum of care and for ensuring linkages of records across different facilities? Ⱥ How should service delivery be planned for those individuals and communities who opt out of tracking when digital tracking systems are implemented at scale? Resource use Ⱥ What are the resources needed to implement and maintain digital tracking combined with health worker decision support and/or targeted client communication? Ⱥ Future research should also identify the potential savings from removing or reducing the costs of printing and assess the cost-effectiveness of these interventions. Modelling approaches such as the Lives Saved Tool (88) may be helpful. Gender, equity and rights Ⱥ How can digital tracking be implemented among marginalized populations, such as migrants and displaced populations, which may not be included within a unique identification system? Other Ⱥ What are the key feasibility, acceptability, resource use and equity considerations linked to incorporating emerging technologies that use biometric identification data to uniquely identify each client, including infants? This includes technologies such as facial recognition and fingerprint and optical scanning. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 13 Intervention Evidence- to-decision domain Research gaps and illustrative research questions mLearning Effectiveness Ⱥ What are the health system conditions that contribute to the effectiveness of mLearning? Research should include these outcomes: » health worker skills and attitudes, including long-term effects on these outcomes » health worker performance and adherence to recommended practice; quality of care provision » client health behaviours » unintended consequences. Acceptability Ⱥ No research gaps identified Feasibility Ⱥ What are the potential barriers to implementing this intervention, including potential losses to the per diem remuneration received by health workers when shifting from face-to-face to mLearning modalities? Resource use Ⱥ What are the resources needed to implement mLearning, and what is the cost-effectiveness of these interventions? Research should consider the cost- effectiveness across different mLearning delivery channels. Ⱥ Resource use and cost-effectiveness was recognized as a cross-cutting research gap across all of the examined digital health interventions. Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 14
WHO guideline recommendations on digital interventions for health system strengthening Research Considerations WHO/RHR/19.9 © World Health Organization 2019 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. This publication forms part of the WHO guideline: recommendations on digital interventions for health system strengthening. It is being made publicly available as supplied by those responsible for its development for transparency purposes and information, as required by WHO (see the WHO handbook for guideline development, 2nd edition (2014)). Suggested citation. WHO guideline: recommendations on digital interventions for health system strengthening. Research considerations. Geneva: World Health Organization; 2019. (WHO/RHR/19.9) Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party- owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Editing: Green Ink, United Kingdom Design and layout: RRD Design LLC W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 1 Future research This chapter on future research highlights crosscutting evidence gaps observed across a range of interventions in relation to effectiveness, resource use and cost-effectiveness, and gender, equity and rights. In addition, specific research questions are provided for each of the interventions, based on the gaps identified through the evidence-to-decision framework and GDG. 1.1 Overarching research gaps The following sections describe the overarching research priorities identified through this guideline process. These reflect the general areas in which the available evidence was found to be of low or very low certainty or confidence, or where no direct evidence was identified. Where studies were available, in some cases the certainty or confidence of the evidence was affected by poor reporting of outcomes, studies with small numbers of participants, and limited representation across different settings. Annex 6 maps the state of evidence and its gaps based on the findings from the effectiveness reviews for the included digital health interventions. Effectiveness For many of the interventions, the available evidence on effectiveness was sparse. Future research should measure health system process improvements that may immediately result from the digital intervention, such as health workers’ adherence to recommended practice, as well as related distal health outcomes. Researchers should be realistic about the extent to which digital health interventions can impact on distal health outcomes, which are often affected by a variety of factors beyond the interaction with the digital intervention. Additionally, effectiveness studies need to include ways of concurrently monitoring technological performance (for example, do messages reach intended individuals?) and behavioural performance or user engagement (e.g. do individuals who get messages listen to or read them, and subsequently act on them?). W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 2 Resource use and cost-effectiveness The studies included in the systematic reviews of the effectiveness of the digital interventions considered by the guideline identified limited evidence on the resources used to implement these interventions. Costing studies should assess costs over a longer period, with appropriate accounting of amortization and maintenance of equipment and the continuous user support required. Future research should explore the cost-effectiveness, and potential for cost savings of the identified intervention and additional savings achieved through combining interventions. Gender, equity and rights Further research needs to encompass a wider range of contexts and populations, including populations with poor access to digital or conventional health services, in order to better understand and mitigate any potential negative impacts on gender, equity and rights. Key research questions include how digital health interventions can help to reduce disparities in linking to the wider health system and whether these interventions may create further inequities in some settings as a consequence of poor network coverage, limited control of mobile devices, or a lack of other resources. Research should also explore unintentional exacerbation of inequities based on who has access to digital devices, and who has access to network connectivity. Implementation research Due to the strong focus on integrated health systems and interoperability, future research should also examine the synergies across different combinations of digital health interventions to determine which packages of interventions are most effective and cost-effective. Addressing this question is important given the potential complexity of implementing packages of digital interventions and the costs of establishing and maintaining these systems. Specific questions include the following. Ⱥ What is the feasibility and effectiveness of different combinations of digital health interventions? Ⱥ What are the non-digital health and supporting interventions (for example, enhanced transportation, supervision) that should be packaged together with digital health interventions to ensure their effectiveness, acceptability and feasibility? Ⱥ What are the minimum requirements of a country’s enabling environment (infrastructure, governance, workforce, interoperability and standards) to support the different recommended digital health interventions? Ⱥ How can the fidelity (i.e. the roll out of all the critical components of the intervention as intended) of implementation at scale be facilitated? Frameworks such as RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) may be useful in structuring the implementation research (87). W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 3 1.2 Considerations for the design of future evaluations The GDG also identified several issues related to the design of future evaluations of digital health interventions, including the following: Ⱥ Health system focused digital interventions, such as stock management and birth and death notification, are often complex in the number of components, behaviours targeted, and organizational levels involved (89). These factors may make designs such as randomized controlled trials for evaluating the effectiveness of these interventions difficult to apply. Other designs may therefore need to be considered, such as controlled before-and-after studies, stepped-wedge randomized controlled trials and interrupted time series studies. Ⱥ While there is value in evaluating changes in client/patient health outcomes, intermediate outcomes are also critical for the evaluation of digital health interventions. For example, the effect of decision support on client/patient health outcomes are influenced not only by the information delivered through the digital system, but also by a host of other factors, including access to medicines, their cost, family support, and biomedical factors such as whether the individual responds appropriately to recommended treatments or has comorbidities. A logical framework of how the digital intervention functions may be helpful in understanding the pathways through which the intervention influences a targeted behaviour or health system challenge and in selecting appropriate outcomes along these pathways. Ⱥ Digital technologies provide new opportunities to capture research data for measuring the effectiveness of implementations in real time, thus facilitating the ability to conduct evaluations more rapidly. Incorporating the research data collection needs for primary and secondary outcomes of interest at the design stage can ensure that the data needed to measure these outcomes is captured alongside the implementation. Ⱥ Rapid changes in digital technologies and the iterative approaches often used for software development may force digital health interventions to evolve during evaluation periods, which may pose challenges for the evaluation process. Detailed process evaluations running alongside impact evaluations may be helpful in understanding the effects of incremental changes in the digital interventions over time. Ⱥ Future research efforts should establish common metrics and tools for assessing the effectiveness and cost-effectiveness of digital health interventions W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 4 Evidence maps and illustrative research questions The tables below illustrates the general trends in the evidence found in the effectiveness reviews, demonstrating low and very low certainty evidence across most interventions. For more details on the specific interventions and outcomes, please review the summary of findings in Web Supplement 1. In addition, specific research gaps and accompanying illustrative research questions are listed Table A5.4. These questions should be addressed using rigorous methods. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 5 Table A5.1 Effectiveness evidence for client interventions Digital intervention Unintended consequences Resource use Satisfaction and acceptability Utilization of health services Health behaviour, status and well-being TCC – adolescents TCC – adults TCC – pregnant + postpartum TCC – pregnant + postpartum with HIV TCC – children <5 Client-to- provider telemedicine TCC stands for targeted client communication. This intervention was reviewed across five population groups. This table does not reflect information on satisfaction and acceptability obtained from qualitative reviews. The comparison for all interventions reflected on these tables is standard care. Please see Web Supplement 1 for other comparison groups for TCC. Table A5.2 Effectiveness evidence for health worker (HW) interventions Digital intervention Unintended consequences Resource use Satisfaction/ acceptability HW performance HW skills/ attitudes HW knowledge Clients’ utilization of health services Clients’ health behaviour, health status/ well-being Provider-to- provider telemedicine Decision support Decision support + digital tracking Decision support + digital tracking + TCC mLearning Table A5.3 Effectiveness evidence for Health system interventions Digital intervention Unintended consequences Resource use Satisfaction/ acceptability Coverage of birth/death notification Timeliness of birth /death notification Coverage of newborn or child health services Timeliness of newborn or child health services Availability of commodities Quality and timeliness of stock management Birth notification Death notification Stock notification Key UNKNOWN LITTLE OR NO DIFFERENCE POSITIVE EFFECT NEGATIVE EFFECT Not applicable/Not measured May make little or no difference (low certainty evidence) May have benefits (low certainty evidence) May lead to harm (low certainty evidence) Uncertain effect because of very low certainty evidence Probably makes little or no difference (moderate certainty evidence) Probably has benefits (moderate certainty evidence) Probably leads to harm (moderate certainty evidence) no incidence No evidence identified Makes little or no difference (high certainty evidence) no incidence Has benefits (high certainty evidence) no incidence Leads to harm (high certainty evidence) no incidence Size of bubbles reflects the number of studies contributing to the outcome 1-37-10 4-6 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 6 Intervention-specific research gaps Table A5.4 outlines the specific research gaps, with illustrative research questions, identified for each of the interventions included in the guideline. These research questions should be addressed using rigorous methods. Table A5.4 Research gaps Intervention Evidence- to-decision domain Research gaps and illustrative research questions Birth and death notification Effectiveness Ⱥ What is the effect of birth and death notification on the quality and timeliness of birth and death reporting or on the accountability for responding to the data? Ⱥ Does notification by mobile devices lead to more timely and complete legal registration, in the case of births, increased coverage and timeliness of health and other social services (e.g. vaccination), or in the case of deaths, increased recording of the causes? Acceptability Ⱥ What is the acceptability of birth and death notification via mobile devices, rather than through standard practices of notification? Research should include how these interventions interact with the sociocultural norms and needs of different communities regarding births and deaths and their notification. Feasibility Ⱥ What are the legal, ethical, data security and policy requirements for allowing new groups of people or cadres of health worker to notify births and deaths? What types of modification to existing legal frameworks would be needed to implement birth and death notification by mobile devices at national scale? Ⱥ What are the most appropriate ways to train health workers and other people designated to use birth and death notification? Ⱥ In what ways do birth (and infant death) notification provide opportunities to link maternal health records with child health outcomes? Resource use Ⱥ See overarching research gaps in section 5.1 Gender, equity and rights Ⱥ How does this intervention increase or decrease health-related disparities? Are there population groups or settings that may not be able to benefit from this intervention, and how can this be addressed? W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 7 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Stock notification and commodity management Effectiveness Ⱥ What is the effect of stock notification and commodity management via mobile devices on improved availability/reduced stock-out of commodities at the point of care? Ⱥ What are the health system conditions that contribute to the effectiveness of this intervention (for example, supervision of health workers, effective transport of products, drug access/purchase policies)? Ⱥ Future research should also be conducted across a range of settings. Acceptability Ⱥ No research gaps identified Feasibility Ⱥ How can digital stock notification and commodity management systems be implemented so that they are aligned closely with both national ordering routines and local needs, and are also supported by well-functioning national and subnational commodity management? Ⱥ What can be learnt from practices in logistics management information systems used outside of the health sector that may be applicable to primary health care settings? Resource use Ⱥ What are the potential cost savings from introducing digital stock notification, for example through reducing the need for buffer stock and improving the accuracy of stock need forecasts? Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 8 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Client1-to- provider telemedicine Effectiveness Ⱥ What types of digital channel used in facilitating client-to-provider telemedicine are most effective (for example, transfer of images, voice, text, and other delivery channels)? Under which circumstances should these different channels be used? Ⱥ Future research should include the following outcomes: Ⱥ use of health services Ⱥ health behaviour, status and well-being Ⱥ health worker and client satisfaction Ⱥ unintended consequences, including the specific risks and safety concerns for implementing telemedicine different health domains or conditions. Acceptability Ⱥ How does this intervention influence health workers’ ability to communicate or explain information to clients, including issues of liability? Linked to this, in what ways does this intervention change interactions between clients/patients and health workers? Ⱥ Further research in low- and middle-income settings is especially needed. Feasibility Ⱥ What mechanisms can address identified implementation barriers, such as concerns about data privacy obtaining informed consent, and challenges in network connectivity that may compromise the quality of information exchanged (e.g. loss of quality of image files, interrupted connection)? Resource use Ⱥ What are the resources needed to implement client-to-provider telemedicine, and what is the cost-effectiveness of this intervention? This should include research on the cost-effectiveness of different delivery channels, such as voice- based consultations, image exchanges and other modalities to facilitate client- to-provider telemedicine for different health issues. Gender, equity and rights Ⱥ How does this intervention increase or decrease health-related disparities? Are there population groups or settings that may not be able to be able to benefit from this intervention, and how can this be addressed? 1 Although WHO’s Classification of digital health interventions v1.0 uses the term “client” (13), the terms “individual” and “patient” may be used interchangeably, where appropriate. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 9 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Provider- to-provider telemedicine Effectiveness Ⱥ What are the conditions that contribute to the effectiveness of provider-to- provider telemedicine? Ⱥ Future research should include the following outcomes: » health worker performance and adherence to recommended practice, quality of care provision » health behaviour, status and well-being » health worker and client satisfaction » unintended consequences, including the specific risks and safety concerns for implementing telemedicine different health domains or conditions. Acceptability Ⱥ How is provider-to-provider telemedicine perceived by health workers to influence inter-professional interactions and collaboration? Feasibility Ⱥ What are the potential barriers to implementing these interventions, and how can these be mitigated? Such barriers include, for example, challenges in connectivity and its resulting consequences on the quality of information exchange (e.g. loss of quality of image files, interrupted connections). Resource use Ⱥ What are the resources needed to implement provider-to-provider telemedicine, and what is the cost effectiveness of this intervention? This should include research on the cost-effectiveness of different delivery channels, such as voice-based consultations, image exchanges and other modalities, to facilitate provider-to-provider telemedicine for different health issues. Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 10 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Targeted client communication Effectiveness Ⱥ How does the frequency, dose, delivery channel and overall exposure to content of targeted client communication affect behaviour change and health outcomes? Ⱥ Future research on effectiveness should consider the following outcomes: » use of health services » health behaviour, status and well-being » satisfaction with services » knowledge and attitudes (for adolescent populations) » unintended consequences. Acceptability Ⱥ Most studies to date have asked people about their views were they to receive targeted communications via mobile devices, while some studies have evaluated people’s experiences within pilot projects or randomized trials. Future research should focus on the views of participants involved in national- scale targeted client communication programmes. Ⱥ What is the acceptability of different formats and delivery mechanisms across different sociocultural contexts and population groups, such as adolescents? Feasibility Ⱥ What strategies can be used to address privacy concerns and to mitigate any potential negative effects of transmitting sensitive health content, including ways to enforce consent and the ability to opt out of programmes? Ⱥ What ways can be used to maintain contact with clients who regularly change their phone numbers, or who have limited or shared access to mobile devices? Resource use Ⱥ What is the cost-effectiveness of different delivery channels, such as voice, text messages, USSD, and smartphone applications? Gender, equity and rights Ⱥ What strategies can be used to ensure equal access to and use of targeted client communication services for all groups, including people with poor access to mobile devices and/or poor network coverage, people who speak minority languages and people with low literacy or poor technological literacy and skills? Ⱥ Future research assessing the effectiveness of targeted client communication using mobile devices should make efforts to ensure that disadvantaged populations are included. Trials should avoid excluding, wherever possible, participants on the basis of mobile device ownership, literacy levels, language or participation in formal health care programmes. Other Ⱥ Where possible, research should take an integrated approach that includes outcomes across the continuum of care in pregnancy, childbirth and child health, as well as across sexual and reproductive health in general. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 11 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Health worker decision support Effectiveness Ⱥ What is the effectiveness of health worker decision support via mobile devices across different settings, health domains, levels of health care, and among health workers with different levels of training? Future research should focus on these outcomes: » health worker performance and adherence to recommended practice, quality of care provision » clients’/patients’ use of services » clients’/patients’ health behaviour, status and well-being » health worker and client satisfaction » unintended consequences. Acceptability Ⱥ How is decision support via mobile devices perceived by health workers and clients, and how does it influence their interactions in the provision of services? Feasibility Ⱥ What mechanisms can be used to validate the health content within decision support systems, to ensure that the recommended clinical practices are congruent with the best available evidence? Resource use Ⱥ See overarching research gaps in section 5.1 Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 Other Ⱥ What mechanisms can be used to ensure that decision support tools evolve with new clinical evidence and subsequent policy changes? The development of the clinical algorithms used within decision support systems is presently an inexact science. Further research is needed to identify best practice, to develop and refine these algorithms both in terms of their clinical effectiveness and their ease of use and acceptability for health workers and clients. The use of artificial intelligence for the development of decision support systems is an emerging area that may help to refine algorithms, but more research is needed on acceptability, feasibility and ethics. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 12 Intervention Evidence- to-decision domain Research gaps and illustrative research questions Digital tracking with decision support and targeted client communication Effectiveness Ⱥ What is the effectiveness of digital tracking across different settings and health domains? Research should focus on these outcomes: » health worker performance and adherence to recommended practices; quality of care provision » clients’/patients’ use of health services, including follow-up services » quality of data on the services provided » clients’/patients’ health behaviour, status and well-being » health worker and client satisfaction » unintended consequences. Acceptability Ⱥ What approaches can be used to minimize the dual burden on health workers of operating paper and digital systems? Feasibility Ⱥ What are the policy requirements for transitioning from paper to digital systems for client health records, including the establishment and institutionalized use of unique identification mechanisms? Ⱥ What are the implementation approaches and requirements for maintaining a longitudinal client record across the continuum of care and for ensuring linkages of records across different facilities? Ⱥ How should service delivery be planned for those individuals and communities who opt out of tracking when digital tracking systems are implemented at scale? Resource use Ⱥ What are the resources needed to implement and maintain digital tracking combined with health worker decision support and/or targeted client communication? Ⱥ Future research should also identify the potential savings from removing or reducing the costs of printing and assess the cost-effectiveness of these interventions. Modelling approaches such as the Lives Saved Tool (88) may be helpful. Gender, equity and rights Ⱥ How can digital tracking be implemented among marginalized populations, such as migrants and displaced populations, which may not be included within a unique identification system? Other Ⱥ What are the key feasibility, acceptability, resource use and equity considerations linked to incorporating emerging technologies that use biometric identification data to uniquely identify each client, including infants? This includes technologies such as facial recognition and fingerprint and optical scanning. W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 13 Intervention Evidence- to-decision domain Research gaps and illustrative research questions mLearning Effectiveness Ⱥ What are the health system conditions that contribute to the effectiveness of mLearning? Research should include these outcomes: » health worker skills and attitudes, including long-term effects on these outcomes » health worker performance and adherence to recommended practice; quality of care provision » client health behaviours » unintended consequences. Acceptability Ⱥ No research gaps identified Feasibility Ⱥ What are the potential barriers to implementing this intervention, including potential losses to the per diem remuneration received by health workers when shifting from face-to-face to mLearning modalities? Resource use Ⱥ What are the resources needed to implement mLearning, and what is the cost-effectiveness of these interventions? Research should consider the cost- effectiveness across different mLearning delivery channels. Ⱥ Resource use and cost-effectiveness was recognized as a cross-cutting research gap across all of the examined digital health interventions. Gender, equity and rights Ⱥ See overarching research gaps in section 5.1 W H O g u i d e l i n e r e c o m m e n dat i o n s o n d i g i ta l i n t e r v e n t i o n s f o r h e a lt h s y s t e m s t r e n g t h e n i n g page 14