Monitoring the implementation of digital health: an overview of selected national and international methodologies

Monitoring the implementation of digital health An overview of selected national and international methodologies The World Health Organization was established in 1948 as the specialized agency of the United Nations serving as the directing and coordinating authority for international health matters and public health. One of WHO’s constitutional functions is to provide objective and reliable information and advice in the field of human health. It fulfils this responsibility in part through its publication programmes, seeking to help countries make policies that benefit public health and address their most pressing public health concerns. The WHO Regional Office for Europe is one of six regional offices throughout the world, each with its own programme geared to the particular health problems of the countries it serves. The European Region embraces nearly 900 million people living in an area stretching from the Arctic Ocean in the north and the Mediterranean Sea in the south and from the Atlantic Ocean in the west to the Pacific Ocean in the east. The European programme of WHO supports all countries in the Region in developing and sustaining their own health policies, systems and programmes; preventing and overcoming threats to health; preparing for future health challenges; and advocating and implementing public health activities. To ensure the widest possible availability of authoritative information and guidance on health matters, WHO secures broad international distribution of its publications and encourages their translation and adaptation. By helping to promote and protect health and prevent and control disease, WHO’s books contribute to achieving the Organization’s principal objective – the attainment by all people of the highest possible level of health. Monitoring the implementation of digital health An overview of selected national and international methodologies Document number: WHO/EURO:2022-5985-45750-65816 © World Health Organization 2022 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial- ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition: Monitoring the implementation of digital health: an overview of selected national and international methodologies. Copenhagen: WHO Regional Office for Europe; 2022”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization. (http://www.wipo.int/amc/en/ mediation/rules/). Suggested citation. Monitoring the implementation of digital health: an overview of selected national and international methodologies. Copenhagen: WHO Regional Office for Europe; 2022. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Designed by: Pellegrini For further information please contact the WHO Data and Digital Health Unit (euhiudata@who.int) or the Regional Center for Studies on the Development of the Information Society of the Brazilian Network Information Center (https://cetic.br/en/contato/). iii Contents Acknowledgements iv Acronyms and abbreviations v Executive summary vi Introduction 1 Methodology 5 International and regional approaches to monitoring progress in digital health 9 World Health Organization 9 European Commission 11 Latin America and the Caribbean 14 Nordic eHealth Research Network 18 Organisation for Economic Co-operation and Development 20 Digital health monitoring initiatives in eight countries 25 Australia 25 Brazil 29 Costa Rica 32 Denmark 33 Italy 37 Netherlands 39 Republic of Korea 44 Uruguay 47 Main findings 51 Adoption of a functionality-based approach 53 Digital domains, actors and activities covered 53 Inclusion of policy-level domains 54 Data sharing and reuse 54 Measures of patients’ involvement in their own care and digital health inequalities 55 Data collection methods 55 New digital health domains 58 Measuring digital health maturity 58 Conclusions 61 References 65 iv Monitoring the implementation of digital health: an overview of selected national and international methodologies Acknowledgements This document was developed by the Data and Digital Health Unit, Division of Country Health Policies and Systems of the WHO Regional Office for Europe, Copenhagen, Denmark, and the Regional Centre for Studies on the Development of the Information Society of the Brazilian Network Information Centre (Cetic.br/NIC.br). The main authors were Elettra Ronchi, Senior Policy Consultant in data governance, privacy and digital health and Consultant, Data and Digital Health Unit, Division of Country Health Policies and Systems, WHO Regional Office for Europe; Luciana Portilho, Coordinator of the ICT in Health survey, Cetic.br/NIC.br; and Ana Laura Martinez, Coordinator of Technical Cooperation, Cetic.br/NIC.br. Alexandre Barbosa, Manager, Cetic.br/NIC.br; Fabio Senne, Survey Projects Coordinator, Cetic.br/NIC.br; and David Novillo Ortiz, Regional Adviser, Data and Digital Health Unit, Division of Country Health Policies and Systems, WHO Regional Office for Europe provided technical advice and direction during the production process. Special thanks also go to Natasha Azzopardi-Muscat, Director of the Division of Country Health Policies and Systems, WHO Regional Office for Europe, and Heimar Marin, Scientific Coordinator of the ICT in Health survey, Cetic.br/NIC.br, for their strategic guidance. The WHO Regional Office for Europe is grateful to the following individuals, who provided relevant information and validated the final versions of the sections about their countries: Vicki Bennett, Australian Institute of Health and Welfare, Canberra, Siaw-Teng Liaw, University of New South Wales, Sydney, and David Glance, University of Western Australia, Perth, Australia; Anna Maresso, Dimitra Panteli and Gemma Williams, European Observatory on Health Systems and Policies, Brussels, Belgium; Christian Nøhr, Aalborg University, Aalborg, Karsten Vrangbæk, University of Copenhagen, Copenhagen, and Kenneth Bøgelund Ahrensberg, Danish National Board of e-Health, Copenhagen, Denmark; Claudia Biffoli, Ministry of Health, Rome, and Fidelia Cascini, Universita Cattolica del Sacro Cuore, UNICATT, Rome, Italy; Seung-Mi Yoo, HIRA, Seoul, Republic of Korea; Madelon Kroneman, Nivel, Utrecht, and Roos van der Vaart, National Institute for Public Health and the Environment, Bilthoven, Netherlands; and Rosario Berterretche and Cecilia Muxi, Salud.uy Program, Montevideo, Uruguay. For further information please contact the WHO Data and Digital Health Unit (euhiudata@who.int) or Cetic.br/NIC.br (https://cetic.br/en/contato/). The following colleagues and experts provided technical review and feedback during the development of this document: WHO Regional Office for Europe Clayton Hamilton, Helen Caton-Peters, Jahan Nurmuhammedova and Ryan Dos Santos External experts Cátia Sousa Pinto, Inês Alexandra da Mota Lourenco and Miguel Ângelo Monteiro Santinhos (Shared Services for Ministry of Health, EPE, Portugal) and Michael Peolsson (Swedish eHealth Agency, Kalmar, Sweden) vAcronyms and abbreviations ADHA Australian Digital Health Agency AGESIC Agency for E-Government and the Information and Knowledge Society (Brazil) CCSS Social Security Fund (Costa Rica) Cetic.br Regional Centre for Studies on the Development of the Information Society CGI.br Internet Steering Committee (Brazil) CNES National Registry of Health Facilities (Brazil) DATASUS Unified Health System Informatics Department (Brazil) EC European Commission ECLAC Economic Commission for Latin America and the Caribbean eHealth electronic health EHR electronic health record EMR electronic medical record EU European Union GOe Global Observatory for eHealth (WHO) GP general practitioner HIE health information exchange HIS health information systems ICT information and communication technologies IT information technology KHIS Korean Health Information Service MBS Medicare benefits schedule mHealth mobile health MHR My Health Record NCM Nordic Council of Ministers NeRN National eHealth Research Network NIC.br Network Information Centre (Brazil) Nictiz Centre of Expertise for Standardization and eHealth (Netherlands) Nivel Netherlands Institute for Health Services Research OECD Organisation for Economic Co-operation and Development PAHO Pan American Health Organization PHR personal health record RNDS National Health Data Network (Brazil) SCA Statistical Conference of the Americas SNIS National Integrated Health System (Uruguay) SUS Unified Health System (Brazil) vi Monitoring the implementation of digital health: an overview of selected national and international methodologies Executive summary The use of information and communication technologies (ICT) for improving the delivery of health care, the efficiency of health systems and the management of health-care facilities has expanded rapidly in recent years. This trend was accelerated by the COVID-19 pandemic. The use of ICT in health, referred to as eHealth or digital health, also has the potential to improve health-care quality, safety and access (WHO, 2005; WHO/GOe, 2006). Digital health can play a crucial role in achieving universal health coverage and improving capacity-building for health-care professionals, as has been evident during the pandemic. Despite their clear potential, digital health programmes and interventions are often not monitored or evaluated. The lack of systematic monitoring and research to determine the factors that may facilitate or impede the adoption and use of digital health is problematic. The information produced by such monitoring can be used as strategic input for policy-making, academic research, resource allocation and decision-making. This study focused explicitly on efforts to monitor implementation rather than those aimed at evaluating impact. It consolidates the available information on approaches and indicators employed in the past 5 years to monitor digital health by the World Health Organization (WHO), the European Commission (EC), the Nordic eHealth Research Network, the Organisation for Economic Co-operation and Development (OECD), and the Working Group on the Measurement of Information and Communication Technology of the Statistical Conference of the Americas (SCA) of the Economic Commission for Latin America and the Caribbean (ECLAC). It also reviews national digital health monitoring activities in eight countries spanning three WHO regions over the same period. This overview contributes to the monitoring and evaluation of the WHO global strategy on digital health 2020–2025 and the measurement framework for the WHO European Programme of Work 2020–2025, which identifies digital health as a highly relevant indicator area for the WHO European Region, for which either suitable internationally agreed measures or adequate data at the international and regional levels are lacking. The review found that the international community had made progress in monitoring digital health. However, more should be done to strengthen the evidence base needed to monitor and shape the digital transformation of health-care systems. Existing metrics and indicators struggle to keep up with the rapid evolution of digital health. In particular, the measurement of governance and reuse of health data across health-care systems, and technical and operational readiness to share these data for statistical and research purposes (e.g. system-wide interoperability), are emerging as common and challenging priorities. vii The report concludes with a number of considerations directed towards improving monitoring activities. In particular, greater efforts and policy action are needed as follows. ▶ Improve the comparability of current indicators. In the short term, the challenge is to improve the comparability of the most commonly used indicators. Only a limited number of indicators can be compiled for monitoring and benchmarking across countries, and they will vary depending on the state of advancement of countries and priority policy dimensions. Exploiting the potential of current measurement initiatives, efforts to encourage progress in the further development and collection of comparable terminology and indicators should be supported, including efforts to ensure information and knowledge sharing for measuring national digital health maturity levels. ▶ Make progress in digital health and its impacts visible in national statistics. With few exceptions, in order to gain a general view of measurement strategies and the state of advancement of digital health in the case study countries, it was necessary to approach several national agencies and institutions in each country. Efforts are needed to make the available information more visible and accessible, including at international level. Additionally, the review identified the following areas as requiring further follow-up work and attention. ▶ Address the significant variability in how telehealth is monitored. The importance of telehealth cannot be overstated. The COVID-19 pandemic has caused health-care systems around the globe to rapidly, and in some cases radically, rethink the delivery of medical care and recognize the potential of telehealth. There is wide cross-country variation in measurements of telehealth capacity, which is generally not monitored systematically. As a quickly evolving area, telehealth can have different meanings in different contexts. There is a need to support the development of a common understanding of terms and formulate standard questions to gain insights and shared learning on the potential of telehealth. The usability of telehealth applications should be a key focus of measurement in this area. ▶ Design new indicators to monitor digital health inequalities. Most of the countries reviewed are increasingly offering their citizens remote access to care as well as access to their personal health-related data. Measurements are needed to monitor the type and level of access that citizens have to digital health services, the challenges they may experience, and the skills needed in the digital health era to address the potential risk of exacerbating inequalities among the most vulnerable. Two country cases worth taking as a reference in this regard are the Netherlands and Uruguay. ▶ Review new sources of health data and approaches to data collection. Measurement, with few exceptions, still tends to focus on specific digital health targets and broad quantitative measures of inputs or outputs rather than on the policies, processes and organizational factors involved. Measurements need to adapt to provide more granular insights, including on how information systems leverage the emerging new digital sources of health and behavioural data and the potential of health data reuse, including from electronic health records (EHRs). viii Monitoring the implementation of digital health: an overview of selected national and international methodologies Table ES1 presents examples of the indicators most commonly used in the eight countries reviewed, grouped according to 10 digital health dimensions. Table ES1. Most common dimensions and indicators from the surveys reviewed Dimension Examples of indicators Target(s) Examples of country surveys and data sources ICT infrastructure Use of computers and types of devices available Types and ranges of download speed of internet connections Network infrastructure Health-care facilities; nurses and physicians The surveys by Brazil and Uruguay collect detailed information on health-care facilities. Health professionals are asked to report on the frequency of use of devices. Italy collects data on the regional progress in the implementation of the network infrastructure. Governance of IT Presence of intranet and extranet Existence of websites and social media Existence of IT departments or areas People employed in IT departments People employed who are specialized in IT People with health degrees employed in IT departments People responsible for technical support Health-care facilities The main references for this dimension are the surveys by Brazil, Republic of Korea and Uruguay, whose questions were based on the ECLAC and OECD model surveys. EPR Use of EPRs Number of patients registered with EPRs Number of patients who have entered and viewed information in EPRs Patients who have retrieved data from EPRs Patients who feel that websites, apps and wearable electronic devices stimulate healthy behaviour Patients The surveys by the Netherlands and Uruguay enquire about the use, experience and evaluation of the EPR by patients. Denmark collects log data on the average number of users per month who retrieve data from their records on the sundhed.dk portal. The surveys by Brazil, Republic of Korea and Uruguay can be a reference for the type of patient data and online services available. ix Dimension Examples of indicators Target(s) Examples of country surveys and data sources EPR (contd) Types of patient data available electronically and their use Health-care facilities, nurses, and physicians Booking medical appointments Booking laboratory tests Viewing laboratory test results Viewing electronic medical records Interaction with medical teams Health-care facilities, patients Asynchronous communication between patients and professionals is available Health-care facilities EHR Availability of electronic systems to record patient information Methods used to keep medical records (electronic format or paper based) Functionalities available in electronic systems and their use Health-care facilities, nurses, and physicians Australia’s My Health Record system's data analytics provide information on the number and types of documents uploaded and viewed by different types of health-care providers broken down according to type of clinical information. The surveys by Brazil, Republic of Korea and Uruguay can be a reference for the type of functionalities available. Italy’s survey can be a reference for data collection about patient summaries and number and types of documents uploaded. The surveys by the Netherlands, Republic of Korea and Uruguay include questions on clinical terminology standards. Number of EHRs activated and used Number of patient summaries drawn up by GPs and paediatricians Health-care facilities, patients, and health professionals Number and types of health-care providers registered Number and type of documents uploaded Clinical terminology standards Health-care facilities Table ES1 contd x Monitoring the implementation of digital health: an overview of selected national and international methodologies Dimension Examples of indicators Target(s) Examples of country surveys and data sources Telehealth Types of available services, such as remote monitoring, distance learning and research in health care; telemedicine; teleconsulting; telediagnosis Use and frequency of available services Telehealth consultations Use of online services and teleconsultations Health-care facilities, nurses and physicians, and patients The surveys by Brazil, Italy, the Netherlands and Uruguay can be a reference. Uruguay's survey includes comprehensive questions about patients’ frequency of use and evaluation of telehealth. Brazil has a survey on users. The Italian National Institute of Health collects a wide range of data on telemedicine, including the pathology addressed and the main characteristics of the telemedicine application; safety; clinical efficacy; patient perception; economic impacts; organizational issues; and sociocultural, ethical and legal aspects. Australia draws information on telehealth use from the Australian Medicare benefits schedule. The Netherlands collects data on the use of teleconsultations and telemonitoring through its eHealth Monitor survey. Health data security and privacy Digital security tools used Existence of digital security or digital security policies Existence of digital security training programmes for employees Data backup strategy and frequency Strategies adopted in matters of digital security Health-care facilities The surveys by Brazil, Italy, the Netherlands, Republic of Korea, and Uruguay include relevant questions based on the OECD model survey. Results are shown by type of digital security tool. Italy collects data on citizens’ authentication and consent management systems. The Netherlands monitors users’ trust in digital applications in health care. Table ES1 contd xi Dimension Examples of indicators Target(s) Examples of country surveys and data sources Electronic medical prescriptions Health-care facilities that have electronic prescriptions to be viewed by any pharmacy Available list of patients’ prescriptions within and outside own organization Prescriptions sent/ mediated in electronic format Health-care facilities and GPs Denmark collects data recorded from the electronic dispensing systems of community pharmacies. Australia monitors the percentage of pharmacies registered on and that use My Health Record. Health information exchange Type of data available to send to or receive from other health-care facilities electronically Health-care facilities, nurses and physicians The surveys by Brazil, Republic of Korea and Uruguay enquire about the different types of information exchange based on the OECD model survey. Denmark has indicators for different types of messages and reports exchanged. In the Netherlands, data on information exchange with other providers are presented by type of health professional, e.g. GP, medical specialist and occupational physician. Availability of electronic systems with interoperability Type of data available for professionals from outside their own organization Health-care facilities Patient data exchanged between health-care providers Officials of health ministries Disruptive technologies Use of cloud services (email, office software, file storage or databases, processing capacity) Big data analytics, source of the data and type of analysis service provider Health-care facilities The main reference in this review is Brazil, which uses indicators based on a survey carried out by Eurostat. Workforce and digital skills Impacts on use of ICT while working at facilities Took a course or received training in health informatics Nurses and physicians Brazil and Uruguay collect data on different types of impacts. The Netherlands has indicators on job satisfaction by GPs and medical specialists. Brazil, Republic of Korea and Uruguay have information about training. Table ES1 contd xii Monitoring the implementation of digital health: an overview of selected national and international methodologies 1The development of digital health1 and the strengthening of health information systems (HIS) are strategic priorities for countries in achieving universal health coverage and equitable provision of health care. If underpinned by sufficient investment in governance and institutional and workforce capacity, digital health can support equitable and universal access to quality health services; improve the efficiency and sustainability of health systems; deliver better-quality, affordable and equitable care; and strengthen and extend health promotion, public health surveillance, disease prevention, diagnosis, management, rehabilitation and palliative care. WHO’s third and most recent Global Survey on electronic health (eHealth), conducted in 2015 (WHO, 2016a), revealed that, among participating countries in the WHO European Region, 70% (30 countries) had national eHealth policies or strategies, of which 90% (27 countries) included explicit references to objectives or key elements of universal health coverage. Digital health strategies and action plans are, however, not enough for change to be successful. Monitoring progress over time helps to ensure that efforts are effective and enables an understanding of what works and what does not. This requires repeated or continuous data collection to measure, for example, the rate of adoption and use of digital technologies against goals and to monitor the impacts of policy decisions and investments. Understanding can be bolstered by learning from other countries. However, for intercountry comparisons to be valid, countries need to adopt comparable indicators and standardized approaches to defining and measuring progress. In 2005 WHO launched the Global Observatory for eHealth (GOe), an initiative dedicated to the study of eHealth, its evolution, and its impact on health in countries (WHO/GOe, 2022). The first objective of the GOe was to determine a series of benchmarks at the national, regional and global levels for the actions needed to support the growth of eHealth. The aim was to provide governments with data that could be used as national benchmarks for their own policy development, as well as facilitating comparison of progress with that of other Member States. In 2005 the GOe conducted the first ever global survey to compile those data. In 2009 a study was conducted by Empirica, on behalf of the European Commission (EC), to collate and analyse existing eHealth monitoring and benchmarking sources in order to 1 The term digital health can have different meanings in different contexts and is often used interchangeably with eHealth. In this report, the terms are considered synonymous. Introduction Introduction 2 Monitoring the implementation of digital health: an overview of selected national and international methodologies identify best practices in data collection and to develop a framework for a European Union (EU)-wide eHealth benchmarking approach (Meyer et al., 2009). This effort was followed in 2012 by a multistakeholder project, conducted by the Organisation for Economic Co-operation and Development (OECD) in close collaboration with WHO and the EC, which resulted in the development of a draft guide to measuring information and communication technologies (ICTs) in the health sector (OECD, 2015) and a draft model survey. The objective was to facilitate cross- country data collection, comparisons, and learning on the availability and use of health ICT. Developing a list of common indicators for monitoring the availability, use and outcomes of health information technologies and formulating methodological recommendations to facilitate the comparison of the related statistics have also been among the central efforts of the Nordic eHealth Research Network (NeRN), established by the Nordic Council of Ministers (NCM) eHealth group in 2012 (Hyppönen et al., 2012). Similar efforts were made by the Working Group on the Measurement of Information and Communication Technology of the Statistical Conference of the Americas (SCA) of the Economic Commission for Latin America and the Caribbean (ECLAC) in 2013 (PAHO, NIC.br, 2019). Despite these substantial early international and regional efforts, little is known about the strategies for monitoring progress in availability and use of digital health technologies. Remarkable changes in the digital health landscape that have occurred since 2013 call for a fresh assessment and review of the frameworks and indicators used and their implementation at the international, regional and national levels. This study aims to update and consolidate the available information on monitoring activities, approaches and indicators employed by these organizations in the past 5 years. It also reviews national monitoring activities in selected countries spanning three WHO regions over the same period. This report describes the methodology of the study, the main findings and possible next steps in developing monitoring strategies. It contributes to the monitoring and evaluation of the WHO global strategy on digital health 2020–2025 (as endorsed by the World Health Assembly in decision WHA73(28)) (WHO, 2020) and to the measurement framework for the European Programme of Work 2020–2025 (WHO Regional Office for Europe, 2021b), which identifies digital health as an indicator area highly relevant for the WHO European Region and for which either suitable internationally agreed measures or adequate data are currently lacking. 3 4 Monitoring the implementation of digital health: an overview of selected national and international methodologies 5The documents included comprised reports: ▶ monitoring at national or international/regional level (single-institution or single-system reports were not included); and ▶ monitoring activities supported or carried out by international and regional organizations or by governments. In view of the focus and the characteristics of the measurement and benchmarking frameworks included in the review, it was decided not to conduct a systematic search of bibliographic databases and search platforms.2 Instead, the reports and documents examined were gathered from official websites and national research institutes. Eight countries spanning three WHO regions were selected for this review, all of which had reported eHealth/digital health monitoring and benchmarking activities (Table 1). The selection was limited to countries where experts linked to ministries of health and national agencies could be contacted for confirmation of the findings within the assigned timelines. The monitoring and benchmarking frameworks developed by the EC, ECLAC, NeRN, OECD and WHO were also examined. Given the scarcity of detailed documentation available online, the main approach employed was to interview and consult with representatives of national government and expert contacts in the eight selected countries and with representatives of the selected international, regional and national organizations, and to analyse the documents they provided. Therefore, this report is not an exhaustive account of all of the monitoring strategies adopted nationally, but represents a selection of the most well-known initiatives. The country reviews have been validated by the respective national authorities and local researchers. 2 Reports on ad hoc surveys by academic and research centres of excellence were not the focus of this review. This review, which was carried out between January and May 2022, consolidates and examines the available information on recent digital health monitoring activities at the international and regional levels, as well as at national level in selected countries. It looks at measurement strategies developed by national government agencies or independent bodies with a mandate to implement, monitor or govern digital health, with an explicit focus on efforts to monitor the implementation of digital health policies and programmes rather than those aimed at evaluating their impact. Methodology Methodology 6 Monitoring the implementation of digital health: an overview of selected national and international methodologies Table 1. List of countries and organizations included Geographical scope Selected case studies International/regional EC, ECLAC, NeRM, OECD, WHO WHO Region of the Americas Brazil, Costa Rica, Uruguay WHO European Region Denmark, Italy, Netherlands WHO Western Pacific Region Australia, Republic of Korea ©WHO 7 8 Monitoring the implementation of digital health: an overview of selected national and international methodologies 9World Health Organization Over the years, WHO has invested significant resources to strengthen digital health research and implementation. WHO developed its approach to international eHealth monitoring in 2005 (Scott & Saeed, 2008). The approach was most recently applied in the 2015 Global Survey on eHealth (WHO/GOe, 2016). WHO has also published a toolkit for assessment and planning of mobile health (mHealth) (WHO et al., 2015) and a handbook for monitoring and evaluating digital health interventions (WHO, 2016b). In 2019 the Pan American Health Organization (PAHO) and the Brazilian Network Information Centre (NIC.br) published methodological recommendations and case studies for measuring digital health (PAHO, NIC.br, 2019). More recent efforts by the WHO have centred on assisting countries to assess the maturity of their digital health projects. In the WHO Region of the Americas, the Information Systems for Health Maturity Assessment Tool (IS4H-MM) is an initiative that includes a method, a tool and questions for assessing organizational capacity related to governance, data management, digital transformation, innovation and knowledge management. The WHO European Region has developed a support tool to strengthen HIS, which includes guidance for assessing a full HIS and for subsequently developing an HIS strategy (WHO Regional Office for Europe, 2021a). Global Survey on eHealth The Global Survey on eHealth methodology was developed by the WHO Global Health Observatory on the basis of consultations with various strategic partners on the tools and policy actions needed to advance and implement digital health at the national, regional and global levels. The first survey, conducted in 2005, mapped the development of eHealth in Member States. The second, in 2009, was largely based on the previous survey but was expanded to include questions on how the privacy of the electronic health record (EHR) was addressed through legal and regulatory tools. This was aimed at ascertaining whether the regulatory environment was ready for a full and effective exploitation of eHealth solutions. The third survey, in 2015, explored developments in eHealth since the 2009 survey and the role it was playing in achieving universal health coverage (WHO, 2016a). International and regional efforts to monitor the availability and use of digital health technologies have been carried out since 2005 in Latin American and Caribbean countries by the EC, OECD and WHO and in the Nordic countries by NeRN. This section reviews the monitoring activities of these organizations since 2017. International and regional approaches to monitoring progress in digital health International and regional approaches to monitoring progress in digital health 10 Monitoring the implementation of digital health: an overview of selected national and international methodologies The 2015 survey also compared its findings with those of previous studies in order to understand the barriers to adoption and examine potential future trends. The survey was divided into eight thematic sections, each offering a different perspective on the contributions of digital health to universal health coverage (WHO, 2016a). The sections were: ▶ basis for eHealth ▶ EHR ▶ telehealth ▶ mHealth ▶ virtual learning in the health sciences ▶ legal frameworks for eHealth ▶ social media ▶ big data. The survey instrument was further revised and used to inform the development of the Digital Health Action Plan for the WHO European Region 2023–2030 and to steer the agenda for innovation in digital health in the Region. The revised survey is organized into three main sections and 12 subsections, or dimensions (Table 2). Table 2. Main sections and dimensions of the Regional Survey on Digital Health for the WHO European Region, 2022 Thematic section Dimensions Digital health foundations National digital health governance Funding and investment in digital health Monitoring and evaluation of digital health interventions Digital health literacy and capacity-building Regulatory frameworks Essential digital health tools EHRs Telehealth programmes mHealth service and programmes Health-related mobile applications (apps) Big data and advanced analytics for health Readiness for information sharing Standards and interoperability Access to, sharing and reuse of EHR data 11 European Commission The EC followed progress in the adoption and use of eHealth in acute care hospitals and general practice in the EU from 2008 onwards through a range of ad hoc surveys; these have been discontinued since 2019. Empirica (a Berlin-based research institute) played a leading role in a number of these surveys, which are briefly described below. The 2019 survey provided an overview of the situation in the EU, Norway and the United Kingdom with regard to the development of interoperable EHR systems. Use of ICT by individuals for booking appointments with health-care practitioners and for seeking health information is captured by the annual Eurostat Community surveys. Eurostat Community survey on ICT usage in households and by individuals  The  Community survey on ICT usage in households and by individuals  is an annual survey conducted since 2002 that collects data on households’ access to, and individuals’ use of, the internet. In the area of digital health, the survey collects information on the following. ▶ Individuals seeking online information about health: individuals who have used the internet in the previous 3 months to seek information about injury, disease, nutrition, improving health, etc. (collected since 2003). ▶ Individuals making appointments with practitioners via websites: individuals who have used the internet in the previous 3 months to make appointments with practitioners via websites (e.g. of hospitals or health-care centres; collected since 2012). Eurostat and national statistical offices have developed a methodological manual for surveys on ICT usage in enterprises and households to assist countries in their reporting and ensure methodological alignment (Eurostat, 2021). European Hospital Survey In 2013 the EC commissioned a European hospital survey to determine the extent to which eHealth had been adopted and was in active use in acute care hospitals (EC Joint Research Centre & Institute for Prospective Technological Studies, 2013). The survey covered the deployment and use of ICT infrastructure, ICT applications, health information exchange (HIE), security and privacy. The data collected were analysed by hospital size (i.e. number of beds) and ownership type (public, private not-for-profit, private). A number of eHealth uptake indicators were analysed at national level. This survey is not covered in greater detail, since it was not repeated. Benchmarking deployment of eHealth among general practitioners Between 2008 and 2018, the EC commissioned three studies on the use of ICT and eHealth applications by general practitioners (GPs) in the EU (Dobrev et al., 2008; Codagnone & Lupiáñez- Villanueva, 2013; Lupiáñez-Villanueva et al., 2018). International and regional approaches to monitoring progress in digital health 12 Monitoring the implementation of digital health: an overview of selected national and international methodologies The third of these studies aimed to: ▶ measure the use of ICT and eHealth applications by GPs in 27 EU Member States since 2013; ▶ analyse the main drivers of and barriers to eHealth adoption in primary health care; and ▶ compare the results with those of previous studies to examine how the levels of adoption, drivers and barriers had evolved since 2013. The 2018 survey used the same approach and the same questionnaire as the 2013 study. The questionnaire covered the sociodemographic and general characteristics of the GPs, as well as the availability and use of eHealth functionalities, and addressed attitudes to, perceived barriers to, and perceived impacts of ICT adoption. Questions on availability and use of eHealth functionalities were related to four categories of ICT in health care, as defined by OECD (2015): EHRs, HIE, telehealth and personal health records (PHRs). To gain a better understanding of the difference between availability and use of the different eHealth functionalities, the authors included additional variables, reflecting general measures of how well a functionality had been adopted, rated on a scale of 0 to 4. These variables were then used to develop composite indicators for each of the four dimensions (Table 3). Table 3. Dimensions used to monitor awareness and use of eHealth by GPs Dimension Indicator EHR awareness and use Awareness and use of 25 EHR functionalities HIE Awareness and use of 15 HIE functionalities Telehealth Awareness and use of 4 telehealth functionalities PHRs Awareness and use of 6 PHR functionalities Survey on the development of interoperable EHR systems in the EU (MonitorEHR study) This 2019 survey provided an overview of the development of interoperable EHR systems in the EU, Norway and the United Kingdom (Empirica, 2019). It was developed on the basis of the results of a literature review and was based on six dimensions (A–F; Table 4). Two further dimensions, G and H, were added for qualitative analysis and as a reaction to the COVID-19 pandemic. In contrast to previous large-scale studies, this survey did not aim to benchmark or rank countries. 13 Table 4. Dimensions and indicators used to measure EHR interoperability Dimension Indicator A Legal framework Legislation and national rules Policy activity B Organizational level and financial investments (measures to support) Citizen-centred data governance Institutionalization of standardization efforts Member State support for interoperable EHR systems and monitoring of EHR interoperability C Security and access Electronic identification Cybersecurity D Semantic interoperability A semantic clinical terminology standard is proposed and is mandatory National semantic interoperability efforts are aligned with international standards Use cases are developed E Technical interoperability Cross-border care Interoperability specifications and profiles for health information domains F Level of actual use of interoperable EHRs Actual use is high by type of institution/type of data Electronic data sharing among health professionals is high Level of structured and coded content of patient data is high G Use of EHRs and big data for early warning, surveillance and diagnosis (COVID-19) Level of standardization for public health reporting of infectious diseases Usage of EHR data H Lessons learned: barriers and success factors Barriers and success factors International and regional approaches to monitoring progress in digital health 14 Monitoring the implementation of digital health: an overview of selected national and international methodologies Latin America and the Caribbean In 2013 the Working Group on the Measurement of Information and Communication Technologies of the SCA of ECLAC undertook to adapt the OECD guide (OECD, 2015) to the Latin American and Carribean context. The aim was to harmonize ICT health statistics and develop methodological recommendations to facilitate the comparison of statistics between the various countries of the area. The work received valuable support from PAHO, the Technical Secretariat of the Plan of Action for the Information and Knowledge Society in Latin America and the Caribbean, and the Regional Centre for Studies on the Development of the Information Society (Cetic.br). The experience of Brazil in pilot-testing the adapted OECD framework was crucial and served as a basis for other countries in the region (PAHO, NIC.br, 2019). The adapted OECD framework was published in 2014 as the ICT in Health survey model and contained guidelines and a model survey for the region (PAHO, NIC.br, 2019). The model survey served as a basis for the countries of the region to prepare their own questionnaires according to local contexts, organization of health systems, policies and information needs. Brazil was the first country in the region to conduct a survey based on this work and continues to carry out an annual survey of health facilities and health-care professionals. Uruguay also systematically monitors digital health, once every 2 years. Other countries, such as Chile and Costa Rica, monitor the use of ICT, but not with the same periodicity. ©WHO 15 ICT in Health survey Table 5 shows the dimensions and indicators corresponding to the questionnaire prepared by SCA-ECLAC, as well as the main indicators collected by the countries of the region in their surveys. Table 6 shows the main common indicators for health-care professionals. Table 5. Dimensions and indicators used to monitor health-care facilities Dimension Indicators for health-care facilities Countries ICT infrastructure Use of computers and the internet ECLAC, Brazil, Costa Rica, Uruguay Types of devices available ECLAC, Brazil, Costa Rica, Uruguay Types and ranges of download speeds of internet connections ECLAC, Brazil, Costa Rica, Uruguay Governance (IT/data) Presence of intranet and extranet ECLAC Existence of specialists in ICT ECLAC, Costa Rica Number of specialists in ICT by area of education and sex ECLAC, Costa Rica Existence of IT departments or areas Brazil, Uruguay Number of people employed in IT departments Brazil, Uruguay People with health degrees employed in IT departments by degree areas Brazil, Uruguay Existence of websites Brazil, Uruguay Types of social media used ECLAC Electronic patient records Availability of electronic systems to record patient information Brazil, Uruguay Types of patient data available in electronic systems ECLAC, Brazil, Costa Rica, Uruguay Data insertion in medical records by patients ECLAC Booking medical appointments Brazil, Uruguay Booking laboratory tests Brazil, Uruguay Viewing laboratory test results Brazil, Uruguay Viewing electronic medical records Brazil, Uruguay Interaction with medical teams Brazil, Uruguay International and regional approaches to monitoring progress in digital health 16 Monitoring the implementation of digital health: an overview of selected national and international methodologies Dimension Indicators for health-care facilities Countries EHRs Types of methods used to keep medical information (electronic format or paper based) ECLAC, Brazil, Costa Rica, Uruguay Types of functionalities available in electronic systems ECLAC, Brazil, Uruguay Types of data that can be sent to or received from other health facilities electronically ECLAC, Brazil, Uruguay Access points of electronic systems in facilities Brazil, Uruguay Telehealth Telemedicine services by medical speciality ECLAC Remote patient monitoring Brazil, Uruguay Distance learning and research in health-care Brazil, Uruguay Teleconsulting and telediagnosis services Brazil, Uruguay Health data security and privacy Information security tools used ECLAC, Brazil Existence of information security policies Brazil, Uruguay Skills and competencies ICT training programmes ECLAC, Costa Rica Types of training programmes available ECLAC Number of employees that received ICT training programmes under the responsibility of health facilities by sex ECLAC Table 5 contd IT: information technology. 17 Table 6. Dimensions and indicators used to monitor health-care professionals Dimension Indicator(s) for health-care professionals Countries Access to ICT Types of devices available for use in health-care facilities Brazil, Costa Rica, Uruguay Frequency of use of computers in health-care facilities Brazil, Costa Rica, Uruguay Use of the internet in health-care facilities Brazil, Costa Rica, Uruguay Use of EHRs Types of patient data electronically available Brazil, Costa Rica, Uruguay Available electronic system functionalities Brazil, Uruguay Telehealth Use of telehealth services Brazil, Costa Rica, Uruguay Skills and competencies Took courses or received training in health informatics Brazil, Uruguay Impacts Perceived impacts of use of ICT on work practices and outcomes Brazil, Costa Rica, Uruguay ©WHO International and regional approaches to monitoring progress in digital health 18 Monitoring the implementation of digital health: an overview of selected national and international methodologies Nordic eHealth Research Network NeRN was established by the NCM eHealth group in 2012. The objective was to develop, test and evaluate a common set of indicators for monitoring eHealth in the Nordic countries, Greenland, the Faroe Islands and Aaland, for use by national and international policy-makers and scientific communities to support Nordic welfare. The approach has been to create a list of indicators, mainly based on survey questions used in the individual Nordic countries. The results of the network’s first mandate period (2012–2013) were published in the 2013 NCM report (Hyppönen et al., 2013). This report described a methodology for generating eHealth indicators by combining top-down and bottom-up approaches. It also outlined how the methodology was tested with four common Nordic indicators for measuring the availability of certain eHealth systems/functionalities and the use of particular functionalities (Table 7). Table 7. Nordic eHealth indicators Domain Indicator(s) Availability Proportion of public organizations offering clinicians access to send/transfer prescriptions electronically to be viewed by any pharmacy Proportion of public organizations offering clinicians electronic access to view patients’ current prescriptions within and outside their own organization (nationally or regionally) Proportion of public organizations offering patients electronic booking for any services Usage rate Proportion of all prescriptions that are sent or mediated in electronic format The results of the network’s second mandate period (2013–2015) were also published in an NCM report (Hyppönen et al., 2015). The publication extended the earlier list of common eHealth indicators, reported lessons learned, and provided recommendations to achieve efficient and easy-to-use benchmarking information. Benchmarking results were presented for 48 eHealth indicators out of an original total of 53 indicators. The 48 indicators included: seven comparable HIE indicators, nine HIE usage rate indicators, five PHR or patient portal availability indicators, 11 patient portal usage rate indicators, 12 usability indicators and four impact indicators. Results indicated that only 10 indicators were being collected by all of the participating Nordic countries (Table 8). HIE functionalities are the primary focus for all of the Nordic countries. Telemedicine indicators focused on the PHR (or patient portal) functionalities. Log data are the primary source of data about usage rates of eHealth services in the Nordic countries. Those data focus on two types of indicators: proportion of all data generated that is stored in electronic format in a national information system, and proportion of viewings or queries of the stored data via the national information system. In the second mandate period, NeRN also collaborated with OECD in the development and testing of the OECD model survey. 19 Table 8. Most commonly collected eHealth indicators in the Nordic region, 2014 Dimension Indicator(s) HIE Proportion of public organizations where detailed clinical notes from outside the organization are available for professionals Proportion of public organizations where patient summaries from outside the organization are available for professionals Proportion of public organizations where laboratory results from outside the organization are available for professionals Proportion of public organizations where imaging results from outside the organization are available for professionals Proportion of public organizations where immunization data from outside the organization are available for professionals Proportion of public organizations where lists of medications prescribed to patients outside the organization are available for professionals Proportion of public organizations where sending prescriptions to be dispensed in any pharmacy is available for professionals Proportion of prescriptions that are made electronically per year Proportion of public organizations where lists of medications prescribed to patients are available to patients Telemedicine (patient portal) Proportion of public organizations where asynchronous communication between patients and professionals is available The aim of the network’s third mandate period (2015–2017) was, above all, to identify common indicators that could be used to analyse and compare patients’ and citizens’ use and experiences of eHealth services (Hyppönen et al., 2017). In the most recent mandate period (2017–2019), the NeRN aimed to: ▶ examine the impact of policies and governance in Nordic countries; ▶ develop a theoretical framework to accommodate the shift of focus in national policies and contribute to furthering the development of indicators that can be monitored in all of the Nordic countries; ▶ develop a Nordic model survey to monitor citizens’ views on eHealth; and ▶ establish an understanding of the national and health-care sector-specific security strategies (Nøhr et al., 2020). International and regional approaches to monitoring progress in digital health 20 Monitoring the implementation of digital health: an overview of selected national and international methodologies Organisation for Economic Co-operation and Development OECD has led initiatives to support countries in strengthening their health information infrastructure since 2011. The impetus for this work was a call from OECD health ministers in October 2010 to make more effective use of health data to improve health sector performance and quality of care. In particular, OECD has undertaken significant work since 2013 to facilitate cross-country data collection, comparisons, and learning on the availability and use of digital health technologies. One of the key goals in recent OECD digital health surveys and studies has been to monitor health data governance and the secondary use of health data for health system measurement and quality improvement. OECD guide to measuring ICTs in the health sector In 2015 OECD published a guide to measuring use of ICT in the health sector (OECD, 2015), with the aim of providing a standard reference for statisticians, analysts and policy-makers in the field of health ICT. The guide was developed by an OECD Expert Group representing 30 countries, in cooperation with the EC, WHO and the then OECD Business and Industry Advisory Committee (now called Business at OECD). The guide has two primary components. The first is the model survey, composed of self- contained modules that ensure flexibility and adaptability to a rapidly changing environment (Table 9). The use of core modules (as an add-on to existing national surveys or as a stand-alone survey) allows measurements to be internationally comparable. Additional modules and new measures can be added to respond to evolving or country-specific policy needs. The second component is a methodological guide to aid implementation and promote the validity and comparability of resulting measurements. The model survey addresses four categories of broadly defined domains in which ICT supports care delivery: ▶ provider-centric electronic records ▶ patient-centric EHRs ▶ HIE ▶ telehealth. The survey adopts a functionality-based approach to defining key types of health ICT. This approach facilitates comparability, since differences in terminology across countries could impede meaningful benchmarking. It also supports technology neutrality (i.e. the questions neither require nor assume a particular technology) and is forward-looking (i.e. it does not hinder the use or development of technologies in the future). Part I of the survey is addressed to general, primary care and family practitioners in ambulatory settings; part II is intended for chief information officers and information technology (IT) administrators in acute care settings. 21 Table 9. Structure of the OECD health ICT model survey Part/section Dimension Part I General practitioners, primary care physicians Section A Contextual indicators (basic demographic data about respondents and their practice settings) Section B Availability and use of electronic records and HIE Section C Availability and use of functionalities that support patient engagement Section D Availability and use of telecommunications technologies to support health-care delivery Part II Chief information officers, IT administrators Section A Contextual indicators (basic demographic data about respondents and their organizations) Section B Availability and use of electronic records and HIE Section C Availability and use of functionalities that support patient engagement Section D Availability and use of telecommunications technologies to support health-care delivery ©WHO International and regional approaches to monitoring progress in digital health 22 Monitoring the implementation of digital health: an overview of selected national and international methodologies Survey of health data development, use and governance The OECD 2019–2020 survey of health data development, use and governance measured elements of national health data governance, including the implementation of national health data governance frameworks, and related regulations and policies. The survey included a detailed review of data development, quality, accessibility and sharing, as well as data security and privacy protection among the custodians of 13 key national health datasets. The survey was addressed to officials of health ministries or national health data authorities, who coordinated the completion of the questionnaire. Details of the survey and results are available online (Oderkirk, 2021). In July 2021 OECD conducted a survey of health data and governance changes during the COVID-19 pandemic. This survey examined the state of health data availability, timeliness, access and sharing, and the need for and benefits of improved and harmonized approaches to health data governance that had been adopted since March 2020. Survey of EHR system development, data use and governance In 2021 OECD also conducted a survey of EHR system development, data use and governance (Table 10). This survey followed up on a 2016 survey on the same topic and measured the governance of clinical data within EHR systems and the technical and operational readiness to use EHRs for statistical and research purposes. Respondents to the 2021 EHR survey were officials of health ministries or the national authorities responsible for EHR systems. Table 10. Main dimensions of the 2021 OECD survey of EHR system development, data use and governance Dimension Indicators EHR Patient EHR records available Patient EHR records exchanged between health-care providers Data extraction from EHR records possible Secure identification of patients and health-care providers Creation of databases from EHRs Governance of EHR data Laws and policies Data subject consent requirements Data protection and digital security Storage modalities Data quality auditing Standards and classification systems employed Reuse and linkage of EHR data For public health monitoring, monitoring health system performance, patient safety, physicians’ treatment decisions, research and clinical trials, COVID-19 management For machine learning, artificial intelligence Evaluation of usability of EHR data Barriers to reuse 23 24 Monitoring the implementation of digital health: an overview of selected national and international methodologies 25 Australia Responsibility for Australia’s health system is split between the Australian Federal Government and the governments of the states and territories. Among the Federal Government’s responsibilities is the management of a digital health strategy, which is overseen by the Australian Digital Health Agency (ADHA). ADHA and its predecessor, the National Electronic Health Transition Authority, created a platform for shared EHRs for every Australian, along with infrastructure and standards for unique health-care identifiers, secure messaging, electronic prescriptions, electronic pathology results, electronic discharge summaries, electronic referrals and letters. The Personally Controlled Electronic Health Record system was launched in 2010 and was later renamed My Health Record (MHR). MHR is central to Australia’s digital health strategy in that it provides secure information interchange at both the state and national levels. Even hospitals that do not maintain electronic medical records (EMRs) are able to interface with MHR and provide information that then becomes available to other health professionals, such as GPs. A range of health documents is supported by MHR and the systems that integrate with it. MHR also allows users to upload their own documents. The documents are stored and transmitted using HL7 clinical document architecture and cover a range of clinical functionalities, including: ▶ continuity of care — referrals, specialist letters, shared health summaries, discharge summaries; ▶ registers — immunizations and organ donors; ▶ prescriptions; ▶ advanced and end-of-life care — advanced care directives; and ▶ diagnostic documents — laboratory, pathology and imaging reports. Access to MHR is handled through the Australian Government site, which unifies digital access to several Federal Government services. Permissions for the records are customizable and under patient control. More than 98% of the content in MHR is machine-readable, including the Medicare benefits schedule (MBS), the pharmaceutical benefits scheme data, and a variety of rich clinical resources. Only 1–2% of the documents contained in MHR are in Portable Document Format (i.e. are PDFs). However, automatic transfer of data from MHR takes place only for electronic prescriptions. All other data have to be actively transferred or created by treating physicians. Digital health monitoring initiatives in eight countries Digital health monitoring initiatives in eight countries 26 Monitoring the implementation of digital health: an overview of selected national and international methodologies National digital health measurement initiatives ADHA developed a benefit realization plan in 2017 that defined success for the MHR system. The plan set out a strategy for managing the benefits of the MHR system, and making sure that benefit realization was “planned and budgeted, underpinned by a whole lifecycle process… appropriately resourced [and] backed by focused governance structures” (Australian National Audit Office, 2019). The plan stated that intermediate output measures would be tracked using MHR system data analytics. Intermediate output measures include the number of individuals and health-care providers registered and the number and types of documents uploaded and viewed by different types of health-care provider. Output measures for health-care providers are broken down by general practice, public and private hospitals, pharmacies, specialists, allied health, aged care, pathology, and diagnostic imaging (Table 11). Output measures for clinical information are broken down by the types of providers uploading and viewing them and by the types of documents: shared health summaries, discharge summaries, event summaries, specialist letters, MBS and pharmaceutical benefits scheme data, prescription records, medication viewings by types of providers, advance care documents, care plans, age care transfer forms, reports of patient observations and monitoring, organ donor status, child development reports, pathology reports, diagnostic imaging reports, immunization registers, screening status, and genomic information. The ADHA has also proposed adopting a digital maturity assessment process, similar to those in other countries such as the United Kingdom (NHS England, 2016). The aim is for health services to be able to assess their level of digital maturity (the extent to which they are supported by the effective use of digital technology) and be supported in improving it. Table 11. Intermediate output measures for MHR benefit measurement Dimension Target Data being collected Comments Individuals (consumers) registered (%) 98% by 2026–2027 Yes 90.1% in July 2018–2019 Health-care providers registered (%) Varies by type Yes Data are broken down by provider types Providers uploading clinical information (%) Varies by type Yes Data are broken down by provider and document type Clinical information uploaded (%) Varies by type Yes Data are broken down by document type System functionality implemented (%) Not specified Yes Data are broken down by document type and function Providers viewing information in MHR (%) Varies by type Yes Data are broken down by provider type 27 Dimension Target Data being collected Comments Consumers who have entered information (%) 30% by 2026–2027 Yes – Consumers viewing information in MHR (%) 52% by 2026–2027 Yes – MHR data provided for secondary use (%) 32% by 2026–2027 Not applicable Secondary use of data commences in 2020 Data on MHR adoption and usage ADHA publishes a range of statistics about how MHR is being used by health-care provider organizations and patients. The statistics include information about registrations, document uploads and prescription/dispensing documents. In November 2021 ADHA reported the metrics shown in Table 12. Table 12. Data on MHR adoption and usage Dimension Indicators reported Data reported in November 2021 Number of MHRs Number of MHRs (% of population) 23.2 million (90% of population) Number of MHRs per state 227 000–7.2 million Number of MHRs with data in them 22.1 million Percentage of MHRs with data 95% Number of documents uploaded to the system by consumers or health-care providers Number of documents uploaded to the system by consumers or health-care providers 514 million Number of documents uploaded to the system by consumers 415 000 Number of medical documents/ prescriptions uploaded by health-care providers, e.g. pharmacists and GPs 311 million Number of pathology reports uploaded 1.6 million Number of diagnostic imaging reports uploaded 17 million Number of diagnostic imaging reports viewed 179 000 Number of discharge summaries uploaded 13 million Number of discharge summaries viewed 44 000 Number of dispensing records uploaded 184 million Table 11 contd Digital health monitoring initiatives in eight countries 28 Monitoring the implementation of digital health: an overview of selected national and international methodologies Dimension Indicators reported Data reported in November 2021 Telehealth services Total number of telehealth services delivered 13/11/2020 to 12/1/2022: 91.36 million Pharmacy adoption and use Percentage of pharmacies registered 99% Percentage of pharmacies that have used MHR 99% GP adoption and use Percentage of GPs registered 99% Percentage of GPs who have used MHR 95% Public hospital adoption and use Percentage of public hospitals registered 97% Percentage of public hospitals that have used MHR 95% Source: ADHA (2021). Telehealth uptake statistics In March 2020 additional telehealth items were added to the Australian MBS to encourage physical distancing for those accessing medical, nursing and allied health services during the COVID-19 pandemic. The University of Queensland’s Centre for Online Health analysed the MBS service data and summarized telehealth uptake throughout Australia (Snoswell et al., 2020) (Table 13). All data were sourced from the Medicare item reports website published by the Australian Government (Services Australia, 2022). Table 13. Telehealth key statistics, July–September 2021 Dimension Indicators reported Results for July–September 2021 Number of telehealth consultations Total number of telehealth consultations 14.2 million Percentage of MBS services delivered by telehealth 28% Total number of telehealth consultations delivered by telephone 12.6 million Total number of telehealth consultations delivered by videoconference 1.6 million Number of telehealth consultations by health-care providers Total number of telehealth consultations by GPs 11.6 million Total number of telehealth consultations by specialists 1.5 million Percentage of total consultations delivered by telehealth by specialists 21% Table 12 contd 29 Dimension Indicators reported Results for July–September 2021 Number of telehealth consultations by health-care providers Percentage of total consultations delivered by telehealth by specialists 21% Total number of mental health telehealth consultations 3.1 million Percentage of total mental health consultations delivered by telehealth 35% Total number of telehealth consultations by nurse practitioners 200 000 Percentage of total consultations delivered by telehealth by nurse practitioners 24% Total number of telehealth consultations by allied health practices 2.8 million Total number of total consultations delivered by telehealth by allied health practices 64 000 Source: Snoswell et al. (2020). Brazil The Unified Health System (SUS) in Brazil is widely recognized as a universal coverage model that coexists with a private health-care system. In recent years, there has been greater adoption of digital health in the country, with an increased number of health-care facilities adopting electronic systems for recording patient information, and the use of telehealth in both public and private facilities. However, a number of health-care facilities, mainly in primary care, still do not have electronic systems or internet access (CGI.br, 2021a). Over the past few years, the Ministry of Health has developed actions aimed at the adoption and expansion of digital health, such as the National Health Information and Informatics Policy (Brazilian Ministry of Health, 2015), the Brazilian National Digital Health Strategy 2020–2028 (Brazilian Ministry of Health, 2020), and the Plan of Action, Monitoring and Evaluation of Digital Health for Brazil (2019–2023) (DATASUS, 2020a). Recent actions have sought to implement a project with a focus on two fronts: creation of a technological environment that functions as a network and takes a national approach to facilitate information sharing; and improvement of primary care activities to ensure the quality of health records. The Ministry of Health, through the SUS Informatics Department (DATASUS), has been coordinating the implementation of digital health projects to overcome challenges related to management, lack of adequate training and qualifications, infrastructure limitations, and poor access to equipment and supplies. In this effort, they depend on partnerships with, and the participation of, other ministries and public and private institutions that share the objective of promoting digital health in the country (DATASUS, 2020b). Table 13 contd Digital health monitoring initiatives in eight countries 30 Monitoring the implementation of digital health: an overview of selected national and international methodologies In 2020 the Conecte SUS programme was established to implement the digital health strategy, focusing on the computerization of health-care and the exchange of health information between all care units. The programme includes several subprogrammes, with an emphasis on three structural projects: the National Health Data Network (RNDS), which aims to develop a unified EHR through the exchange of information between units of care; Informatiza APS, which aims to support the computerization of primary care by providing necessary devices such as computers, tablets and laptops; and APS Connectivity, which aims to make internet access feasible for health units. In 2021 the ICT in Health survey showed greater use of the internet by primary health units (92%) and of electronic systems to record patient information (88% of all health-care facilities have electronic systems). The results also showed greater availability of patient data and functionalities in electronic format, especially those related to information exchange. However, some indicators have remained stable in recent years, including the speed of internet connections in public facilities, which continues to be low. National digital health measurement initiatives Monitoring and evaluation by DATASUS The Brazilian National Digital Strategy 2020–2028 monitoring and evaluation report from the Ministry of Health is developed and published by DATASUS. Its objective is to make public information on progress in implementing the strategy in public health-care facilities. For this, some metrics were established to measure the advancement of computerization of public health-care facilities. The dimensions and indicators are given in Table 14. Table 14. Indicators monitored in the Conecte SUS programme Dimension Indicator(s) Computerization of the three levels of care Percentage of active health-care facilities that participate in connection activation services via projects with the National Research Network Percentage of active health-care facilities waiting for suppliers to activate connections via projects with the National Research Network Percentage of active health-care facilities awaiting calls for connection activation via projects with the National Research Network Percentage of active health-care facilities with connectivity via projects with the National Research Network Percentage of computerized family health teams Interoperability with external systems Percentage of citizens who issued clinical documents – Conecte SUS Percentage of health professionals using Conecte SUS Professional Percentage of primary health units accredited to the RNDS Percentage of primary health units accredited to the RNDS that are accessing Conecte SUS Professional Governance Percentage of financial budgets executed by projects, initiatives and digital interventions of the Conecte SUS programme 31 ICT in Health survey The ICT in Health survey has been conducted annually since 2013 by Cetic.br, a department of NIC.br, which is linked to the Brazilian Internet Steering Committee (CGI.br). The target population is Brazilian health-care facilities and health professionals (nurses and physicians). The survey frame used for selecting the facilities is the National Registry of Health Facilities (CNES), which is maintained by DATASUS. The scope of the survey includes public and private legal entities that are registered in the CNES and that have at least one physician or nurse. For health-care facilities, the results are presented by type of facility, region, location (capital/ non-capital cities) and administrative jurisdiction. For professionals, age group strata are added. The survey provides more than 80 indicators on the following topics: ICT infrastructure and use in health-care facilities, IT management and governance, EHRs and exchange of information, online services provided to patients and telehealth, new technologies, and access to ICT and use by health-care professionals. The most recent edition of the survey incorporated indicators on the topic of data privacy and security and disruptive technologies. It asked health-care facilities whether they were adopting measures in relation to the Brazilian General Data Protection Law in order to verify adaptation to the new requirements. It also verified whether health-care facilities were offering courses and training on this topic for their employees. On the topic of disruptive technologies, the survey enquired about cloud services, big data analytics, use of blockchain, robotics and artificial intelligence. This new module was based on questions prepared for the 2018 version of Eurostat’s survey on the use of digital technologies in companies (Eurostat, 2018). Table 15 shows the available indicators, in addition to those already indicated above, in the ICT in Health survey in the section on Latin America. Table 15. Indicators monitored for health-care facilities Dimension Indicator(s) for health-care facilities ICT infrastructure Types of internet connections Governance (IT/ data) Main people responsible for computer technical support Online services available to patients Types of channels through which services, such as booking appointments and tests, are offered to patients (websites or applications) HIE Availability of electronic systems with interoperability Disruptive technologies Use of cloud services (email, office software, file storage or database, processing capacity) Big data analytics, source of data and types of analysis service providers Use of artificial intelligence, blockchain and robotics Health data security and privacy Existence of information security training programmes for employees Measures adopted concerning the Brazilian General Data Protection Law Digital health monitoring initiatives in eight countries 32 Monitoring the implementation of digital health: an overview of selected national and international methodologies Additionally, Cetic.br runs the ICT households survey, which includes indicators on the online search for health information and health services by individuals. The ICT Panel Covid-19 survey, which was carried out with internet users during the pandemic, investigated the use of telehealth, such as teleconsultations and health services (CGI.br, 2021b). Costa Rica Costa Rica is a middle-income Central American country with a population of around 5 million (INEC Costa Rica, 2022). It is a unitary state that is administratively divided into seven provinces, 82 cantons and 410 districts. In the 1990s the Costa Rican national health system underwent a reform by which all of the health-care functions formerly provided by the Ministry of Health were transferred to the Costa Rican Social Security Fund (CCSS), an autonomous public entity created in 1941. The Ministry of Health kept the ruling and regulatory functions, plus a reduced number of specific health promotion and disease prevention programmes, such as vector control and basic sanitation (PAHO, 2019). CCSS is the main pillar of the Costa Rican health-care system, covering 94% of the population and offering the three levels of health care, with a strong focus on primary care. There are 1064 basic comprehensive health-care teams, roughly one for every 3500–4000 inhabitants. They constitute the basis of the national health system and are spread throughout the country. The second level of care consists of 10 major clinics, 13 hospitals, and seven regional hospitals. The third level, providing specialized care at national level, consists of three general hospitals and five hospitals specializing in children, women, gerontology, rehabilitation and psychiatry. These hospitals are located in the capital city and its metropolitan area. The system has a total of 54 558 professionals, 19% of whom are administrative, 41% are nursing and support services personnel, 12% are doctors, 12% are other professionals in the medical sciences and 16% are general services staff. The ICT in Health initiative in Costa Rica prioritized the implementation of EMRs, with a goal of covering 100% of basic comprehensive health-care teams. By 2017, the date of the survey, 48% of hospitals had implemented EHRs, 19 million outpatient appointments had been managed, and 3.6 million emergency care actions and over 10 million medical care actions had been recorded via the system. Other ICT policy initiatives that provide policy context are the National Plan for Science, Technology and Innovation 2015–2021, which contemplates measures to adopt ICT in the health sector (Costa Rican Ministry of Science, Innovation, Technology and Telecommunications, 2015), and the Digital Government Programme, which has an eHealth section focused on improving the well-being of citizens and the implementation of eHealth (Costa Rican Ministry of Science, Innovation, Technology and Telecommunications, 2019). 33 National digital health measurement initiatives ICT in Health survey The Costa Rican ICT in Health survey followed the framework adopted in the Brazilian survey. To date, there has been one survey, in 2018. It consisted of a census of the basic health-care units under the jurisdiction of the CCSS, covering health-care facilities, physicians and nurses. The questionnaire was administered via the web, taking advantage of the electronic systems in place. Plans were made for surveys covering other health-care levels, as well as health institutions not belonging to the CCSS but, to date, they have not been implemented. All dimensions and indicators covered in this survey can be seen in the section on the Latin American regional framework. Denmark The Danish health-care system is characterized by extensive digitization, electronic communication between health-care providers, and systematic use of data and digitized work procedures. Increasing use of common IT standards has facilitated electronic communication among all health-care providers, including hospitals, GPs, specialists, laboratories, local authorities and home care services. Recent data indicate that all GPs maintain EHRs, and nearly all use them to exchange records (98%), order prescriptions at pharmacies (99%), receive laboratory test results from hospitals (100%), and send referrals to hospitals (97%), medical specialist clinics (100%) and psychologists (100%). Denmark is considered a global frontrunner, having developed a very high level of digital health maturity, and is seen as a role model for digital health innovation in Europe (European Commission, 2022). Since its introduction in 2007, the e-Journalen (e-record) system gives citizens and health- care professionals digital access to information on diagnoses, treatments and notes from EHR systems in all public hospitals. Clinicians at hospitals have access to e-Journalen directly through hospital EHR systems, while GPs and citizens can access the system through sundhed.dk. The health portal sundhed.dk, based on a federated IT architecture launched in 2003, integrates with local systems and consolidates relevant information from all parts of the health-care service. The portal serves as a central access point for doctors and citizens to make appointments and view images, laboratory results, clinical reports, medications, treatment plans and bills. A national infrastructure for telemedicine was established as early as 2012 as part of a planned large-scale implementation of telemedicine throughout the country. This infrastructure includes standards and relevant reference architectures spanning the entire health-care system, covering data measurement, videos, questionnaires and images. Digital health monitoring initiatives in eight countries 34 Monitoring the implementation of digital health: an overview of selected national and international methodologies National digital health measurement initiatives in place Denmark is well advanced in digital health measurement. Their comprehensive and varied approach provides opportunities for valuable research, quality improvement, and the development of new and innovative solutions to the benefit of the Danish citizens. sundhed.dk The sundhed.dk portal collates medical information and data for all Danish citizens aged 15 years or older (European Observatory on Health Systems and Policies & Petersen, 2019). It displays data from more than 120 different sources without storing or duplicating data. Accessibility to the portal across different end-user platforms (personal computers, tablets and mobile phones) is high and requests to display citizen health data are dealt with in a timely, efficient and secure manner (European Observatory on Health Systems and Policies & Petersen, 2019). Besides increasing the efficiency and effectiveness of health-care delivery, the information recorded and collected feeds automatically into national population-based databases and registries. The log data generate most of the available statistics on citizens’ and health professionals’ use of the system. Additionally, the Danish Centre for Health Informatics, Aalborg University, conducts annual surveys of health professionals and biennial surveys of citizens’ use of health IT. MedCom traffic statistics MedCom, an independent government-financed organization, was established in 1994 to develop nationwide communication standards for the most common messages between public hospitals and GPs, as well as private companies linked to the health-care sector, e.g. pharmacies. The messages cover the most frequent text-based clinical transactions in Danish health care, e.g. discharge letters, referrals, laboratory test orders, e-prescriptions and reimbursement from public health insurance. Monthly indicators for hospitals, GPs and municipalities were updated in 2022 to include the following: hospital messages sent, GP messages sent, specialist practice doctors’ messages sent, all municipality health and social care messages sent and notifications received, prescription renewals from municipalities to GPs, e-records looked up, hospital discharge letters sent, X-ray reports sent, X-ray referrals received, laboratory reports sent, and laboratory requisitions received (MedCom, 2021). In addition, MedCom produces monthly statistics on the use of the shared medication records by GPs. The indicator specifically monitors medication record updates. It does not include data from hospitals or specialists. MedCom also publishes statistics on the number of users of e-records, which include health information on citizens admitted to hospitals. 35 National Prescription Registry The National Prescription Registry receives data recorded in the electronic dispensing systems of community pharmacies. The Registry contains 46 variables that characterize each redeemed prescription, describing the patient, the drug dispensed, the health provider issuing the prescription and the dispensing pharmacy. Additional core variables are the civil personal register number (a unique personal identifier used in all Danish registries), the dispensing date (i.e. the date the prescription was redeemed) and the Nordic article number (a unique six-digit code designating each drug package). Complete documentation is provided by Statistics Denmark and the Danish Health Data Authority. Danish Health Data Authority In addition to the measurement initiatives mentioned above, the Danish Health Data Authority tracks progress against national digital health strategy action plans (Danish Ministry of Health, 2018). The latest strategy defines five focus areas for placing patient needs at the centre of care and digital health health and for making daily workflows easier for health-care professionals: ▶ patients as active partners ▶ knowledge on time ▶ prevention ▶ trustworthy and secure data ▶ progress and common building blocks. ©WHO Digital health monitoring initiatives in eight countries 36 Monitoring the implementation of digital health: an overview of selected national and international methodologies The latest benchmarking data from the Danish Health Data Authority include indicators and measurements for the first three focus areas (Table 16). Table 16. Benchmarking indicators from the Danish Health Data Authority, 2021 Dimension Indicators Indicator description Patients as active partners Possibility of using self-booking solutions in hospitals Total number of appointments nationwide that citizens can book themselves. No distinction is made between mandatory times (e.g. when the citizen has been given a time or has to choose a time for an examination) and voluntary times (when the citizen can book the time but does not have to do so, and can show up without booking). Possibility of SMS reminders for outpatient treatment or check-up in hospital Percentage of all created outpatient appointments that have links to text reminder functionality. Citizens’ use of their health records Average number of users per month who retrieved data from their health records on sundhed.dk. On-time knowledge Use of patient administrative modules in hospitals – real-time registration during hospitalization Proportion of patients registered in real time during hospitalization in relation to the total number of hospitalized patients. Medicine reconciliation — updating shared medication records in hospitals Proportion of updated medication records per discharge in relation to the total number of discharges. Hospital communication with municipalities — timely dispatch of discharge reports Proportion of discharge reports sent electronically and in a timely manner to the municipalities. Hospital communication with GPs — timely dispatch of discharge letters Percentage of discharge summaries sent electronically within 3 working days after discharge. Prevention Hospital communication with municipalities — rehabilitation plans Proportion of rehabilitation plans sent electronically to municipalities within 12 hours after discharge. Medication reconciliation — updating the shared medication records in general practice Proportion of up-to-date medication records in general practice measured as the proportion of times doctors click the Update button for changes in prescribing in shared medical records. Virtual hospital services Number of virtual telemedical hospital services as a proportion of the total number of hospitalizations in the region. 37 Italy In Italy, the Ministry of Health has general oversight over the implementation of the national digital health strategy. Other authorities tasked with this are the Ministry of Economy and Finance and the Agency for Digital Italy. In addition, the regions share responsibility for local implementation and act in an advisory capacity for new draft legislation. Since 2008, digital health has been a focus for the country’s health-care sector reform and was listed as one of the priority areas of the national Strategy for Digital Growth/Development 2014–2020. In 2017 a 3-year plan for IT in public administration was drawn up to guide the digital transformation of the country, including the health sector. It set out the fundamental architectural principles, the rules of usability and interoperability, and ICT expenditures. In 2020 Italy also launched its first National Strategy for Digital Skills and a related operational plan that includes more than 100 specific actions and set ambitious targets for 2025. During 2020 and 2021, there was a sharp acceleration in the adoption of major enabling platforms for digital public services, particularly in the health sector, including: ▶ establishment of the single reservation centre, known as CUP (Centro Unico di Prenotazione), a system that allows health-care bookings to be managed in a centralized manner; ▶ creation of the EHR (Fascicolo Sanitario Elettronico); ▶ implementation of ePrescription, dematerialization of medical reports and disease certificates, patient summary; and ▶ implementation of telemedicine. By 2021, most regions had adopted the EHR. Its use, however, remains uneven, with some regions reporting 100% usage and others only partial adoption. New reforms under the 2021 national recovery and resilience plan are expected to further boost the digitalization of health-care services and the modernization of public administration across the country. The investments in the national platforms of telemedicine (for the provision of services and its governance) and the EHR system will be completed in 2025. National digital health measurement initiatives in place The Agency for Digital  Italy and the Ministry of Health, in agreement with the regional administrations, have established a series of indicators to monitor the implementation and use of patient portals, regional EHR systems, e-prescriptions and other associated digital services in the national territory (Table 17). The indicators include data from each public health-care facility operating in the territory (Aziende Sanitarie Locali, Aziende Ospedaliere, Istituto di Ricovero e Cura a Carattere Scientifico, public foundations). Data are reported on a quarterly basis by each region and are made available on the Ministry of Health website. In order to monitor the progress achieved over time by each region, each indicator is associated with a percentage progression, which highlights the improvement achieved compared with the previous data collection. The Italian National Institute of Health (Istituto Superiore di Sanità) has recently started to map the adoption and use of telehealth by the regions with an online survey (Italian Ministry of Health, 2018). The survey includes seven domains based on a model for the assessment of telemedicine devised by Kidholm et al. (2012): description of the pathology addressed and the main characteristics of the telemedicine application; safety; clinical efficacy; patient perception; Digital health monitoring initiatives in eight countries 38 Monitoring the implementation of digital health: an overview of selected national and international methodologies economic impacts; organizational issues; and sociocultural, ethical and legal aspects. The Ministry of Health updated these results in 2021 through a national survey addressing telemedicine projects developed in each Italian region. The Agency for Digital  Italy and the Ministry of Health expected to further introduce a range of new indicators and data collection methodologies in the near future to monitor the digital transformation of the health-care sector in the light of the goals of the 2021 national recovery and resilience plan. Table 17. Indicators collected to monitor the implementation and use of regional EHR systems Dimension Indicator(s) Indicator(s) reported in 2021 Enabling componentsa Network infrastructure Percentage progress against goal in the implementation of the network infrastructure Authentication and consent management systems Percentage progress against goal for the implementation of citizens’ authentication and consent management systems Patient portals Percentage progress against goal in the implementation of patient portals PHRs Percentage progress against goal in the implementation of citizens’ PHRs Regional patient registries Percentage progress against goal in the implementation of regional patient registries Medical record management services provided by health-care facilities (e.g. health professional training services) Percentage progress against goal in setting up medical record management services Digital laboratory reports State of implementation of digital laboratory reports Number of structured and digitized laboratory reports produced by national public health facilities EHRs State of implementation of EHRs Number of regions in which there is at least one activated EHR Number of EHRs activated on a national basis Number of patients who consented to the uploading of their data in EHRs as a percentage of the region’s total patient population Use of EHRs by patients Number of patients who have accessed EHRs at least once in the previous 90 days as a percentage of the region’s total patient population who consented to EHRs 39 Dimension Indicator(s) Indicator(s) reported in 2021 EHRs Use of EHRs by patients Number of patients who have accessed EHRs at least once in the previous 90 days as a percentage of the region’s total patient population who consented to EHRs Use of EHRs by GPs and paediatricians Number of GPs and paediatricians using EHRs as a percentage of the total number of GPs and paediatricians reporting activated EHRs Number of patient summaries drawn up by GPs and paediatricians as a percentage of the total number of activated EHRs Use of EHRs by health-care facilities Number of medical care records uploaded in EHRs as a percentage of the total number of medical care records reported by health-care facilities Number of health professionals using EHRs compared with the total number of health-care workers in facilities EHR interoperability State of implementation of interregional interoperability of services Number of regions whose EHR services are partially or 100% linked to the national interoperability infrastructure a Specific goals and objectives, as well as timelines, may vary substantially among particular projects and programmes and across regions. Netherlands Digital transformation of the health-care system has long been a priority of the Dutch Ministry of Health, Welfare and Sport. In the Netherlands, eHealth is delivered through a highly developed decentralized system in which a variety of local institution-based EHR systems coexist, reflecting the fragmented health-care system. In 2011 proposed legislation to install a national system for the uniform exchange of medical data was not adopted. The result has been a digital system with limited functionality both geographically and across health settings and sectors. Table 17 contd Digital health monitoring initiatives in eight countries 40 Monitoring the implementation of digital health: an overview of selected national and international methodologies In 2019 about 10% of GPs and about 60% of medical specialists in the country reported not being able to exchange standardized data with hospitals within their region, although hospital and community pharmacies do exchange pharmaceutical data structurally. In addition, GPs often lack electronic connections with community nurses, nursing homes, home care organizations and dementia case managers (Nictiz & Nivel, 2019). On the other hand, virtually all GPs use EHRs and can order prescriptions (which since 2012 have to be electronic) and receive laboratory results electronically. Providers must allow patients access to their own files on request but online access options are not standardized across hospitals and may vary. Individual providers and the government are, nonetheless, actively seeking to promote interoperability and greater HIE (Bruins, 2019). The Dutch Digitization Strategy 2018–2022 (Dutch Ministry of Health, Welfare, and Sport, 2018) includes the development of new regulations on mandatory data exchange in several domains (e.g. medication, imaging) across multiple health-care domains, while easing regulations that impede data exchange. The Dutch Centre of Expertise for Standardization and eHealth (Nictiz), established in 2002, plays an important coordinating role in this ecosystem. Nictiz is an independent national competence centre responsible for setting standards and monitoring developments. It is continuously developing and refining national standards for electronic communications and supports the sector by finding functional IT solutions that can be used nationwide. The Association of Providers for Health Care Communication, representing the whole health-care sector and patients, was set up in 2011 to ensure the exchange of medical information through the national switch point. Patients must approve their participation in this exchange and have the right to withdraw. National digital health measurement initiatives The Netherlands started to systematically monitor the uptake of eHealth in 2013. From 2013 to 2019, Nictiz and the Netherlands Institute for Health Services Research (Nivel) mapped the progress of eHealth in the Netherlands and gathered data on the digital transition across health-care sectors and regions on an annual basis in the eHealth Monitor report. The report is based on the results of semi-structured interviews and questionnaires filled out by members of the Dutch Health Care Consumer Panel, members of the National Panel of the Chronically Ill and Disabled, physicians, nurses, mental health nurse practitioners, physiotherapists, managers and directors. In some years, a one-day workshop has also been held with members of the Dutch Health Care Consumer Panel. Since 2020, the Dutch National Institute for Public Health and Environment (Rijksinstituut voor Volksgezondheid en Milieu; RIVM) has has prepared the eHealth Monitor report in collaboration with Nivel and the National eHealth Living Lab. The report includes overviews of regional differences and qualitative measures based on opinion and attitude surveys. It covers use and implementation of eHealth in the full spectrum of Dutch health care, including general practice, hospitals and long-term care facilities, and among patients and the population. The term digital health is understood to mean the application of digital information and communication to support health and health care or improve health care. The same definition 41 has been drawn up by Nictiz for the term eHealth. The eHealth Monitor, therefore, is not limited to the use of specific digital applications by care providers and patients. It also aims to capture the reasons for differences in the nature and extent of digital care between regions, groups of patients and types of health-care providers. Moreover, it also contains examples of best practices and investigates which factors may promote or hinder the uptake of digital health applications. The 2021 eHealth Monitor report provided an overview of the use of various digital applications and developments since the previous measurement in 2019 (RIVM, 2022). The indicators reported are shown in Table 18. Digital health monitoring initiatives in eight countries ©WHO 42 Monitoring the implementation of digital health: an overview of selected national and international methodologies Table 18. Dimensions and indicators monitored by the eHealth Monitor, 2021 Dimension Indicator(s) Indicator(s) reported in the 2021 eHealth Monitor Telehealth Use of teleconsultations (screen contact, written consultations) Percentage of health-care providers who have used teleconsultations with patients Use of telemonitoring Proportion of patient population whose physicians have used telemonitoring Proportion of patient population who have used telemonitoring according to nurses Patient portalsa Use of patient portals Percentage of health-care providers who have access to patient portals from within their organization Patient experience with patient portals Experience of patients with chronic conditions with patient portals Personal health environmentsb Access to the personal health environment Percentage of patients who are able to access their personal medical information via personal health environments Use of personal health environments Percentage of patients who use the personal health environments Perceived usefulness of personal health environments The estimated proportion of patients who consider the personal health environment to be useful Digital applications to support nursing care (e.g. medicine dispensers) Use of digital applications to support nursing care (e.g. medicine dispensers) Percentage of nurses who indicate that they use digital applications to support care CDS tools Use of CDS tools Percentage of nurses who indicate that they use CDS tools in hospital care, elder care and GP care Use of CDS softwarec Percentage of physicians who indicate that they use CDS software HIE Exchange of health data Percentage of GPs, medical specialists and occupational physicians who indicate that they can exchange data with other health-care providers 43 Dimension Indicator(s) Indicator(s) reported in the 2021 eHealth Monitor Health-care workforce Quality of care and workload Percentage of GPs, medical specialists, patients, nurses and others who have experienced an increase or decrease in workload, quality of care, etc. as a result of eHealth applications Job satisfaction The percentage of GPs, medical specialists and others who have experienced an increase or decrease in job satisfaction as a result of eHealth applications Patient-centred care Promotion of healthy behaviours Percentage of users who feel that websites, apps and wearable devices stimulate healthy behaviour Other Confidence in eHealth Opinion of users on the usefulness of, and level of confidence in, digital applications in health care Digital skills Opinion of users about whether they would be able to participate in online care Perceived cost–effectiveness Percentage of health-care users who agree or disagree with the statement that digital applications in health-care can reduce health-care costs CDS: clinical decision support. a Patient portals offer patients (or their representatives) the opportunity to log in to websites or via apps of health-care organizations to view their own health files. Patient portals are generally managed by health-care organizations. b Personal health environments are personal digital environments in which patients (or their representatives) have control over their own data. Patients (or their representatives) can collect data from files of different health-care providers and share these data with other health-care providers. c CDS software uses artificial intelligence to support doctors in providing care. The software calculates expected health outcomes for individual patients or makes suggestions for treatment. These results are derived from collected (big) data, from which possible outcomes are analysed via algorithms. Table 18 contd Digital health monitoring initiatives in eight countries 44 Monitoring the implementation of digital health: an overview of selected national and international methodologies Republic of Korea In 2005 the Ministry of Health and Welfare of the Republic of Korea established a strategic plan for the digital transformation of the Korean Public Health Information System, with the aim of improving the provision of health care to citizens, reinforcing the coordination of care, advancing new national health policies and improving the efficiency of the public health-care system (Ryu et al., 2013). The cornerstone of the digital PHIS is standardized EHRs. In 2020 the Korean Health Information Service (KHIS) reported almost full digitization of patient data in hospital-level care. Adoption rates of EHRs in the country were 100% for tertiary teaching hospitals, 94.0% for general hospitals with 300 or more beds, 97.5% for general hospitals with fewer than 300 beds, and 90.5% for all hospitals. EHRs include a range of advanced functionalities, such as clinical decision support software, ability to alert providers of drug interactions and medication warnings. At the same time, lack of universal interoperability is cited as one of the many significant shortcomings of the EHRs currently in use. As a result, HIE across the health-care sector is relatively low (as of 2020, 78.6% for tertiary teaching hospitals 40.3% for general hospitals with 300 or more beds, 39.2% for general hospitals with fewer than 300 beds, and 18.7% for all hospitals). The 2020 Health Care Informatization Status Survey reported that mobile-based PHRs were provided by 2.4% of tertiary hospitals, and 4.4% of general hospitals with fewer than 300 beds, 61.9% of tertiary general hospitals and 22.6% of general hospitals with 300 or more beds. In 2021 the government launched the My Healthway project (Lee, 2022), a mobile-based public personal health record app service. The aim is to provide patients with a secure digital place to store health information, including vaccinations, prescriptions and emergency contacts. In addition, the Health Insurance Review and Assessment Service is set to launch a mobile-based personal health record app in 2022 to provide personal medical histories and prescription records.  The Republic of Korea is one of the few countries in which patient-facing telemedicine is explicitly banned. To contain community transmission during the COVID-19 pandemic, the Ministry of Health and Welfare temporarily approved the limited use of telemedicine with e-prescriptions. Monitoring activities and available statistics The KHIS has conducted three surveys on the adoption of eHealth, in 2015, 2017 and 2020. In addition, to gain a deeper understanding of how physicians were being affected by the digital transformation of the health-care system, in 2016 the Research Institute for Healthcare Policy of the Korean Medical Association carried out the Korean Physician Survey. The Survey used a structured questionnaire to ask 8564 physicians to evaluate their work environment and to report on the adoption and use of ICT, and their job satisfaction, lifestyle habits, health status, and future plans. The most recent KHIS survey, in 2020, covered tertiary teaching hospitals and special and general hospitals. Primary health-care facilities were excluded. The survey was designed to be comparable with the preceding two surveys and, as far as possible, with OECD indicators. 45 Tables 19 and 20 provide an overview of the main dimensions and indicators collected. HIS are typically classified into three main categories: management and administration, diagnostic and clinical care (provider-centric and patient-centric), and clinical research. Table 19. Structure of the 2020 survey of general hospitals A. Basis (data generation) B. Current status (data management) C. Care (primary use of data) D. Research and utilization (secondary use of data) A1. Information system A2. Informatization governance A3. Informatization standards A4. Education and EHR development B1. EHR status B2. EHR certification B3. Personal information protection B4. Information security C1. EHR upgrade C2. Care information exchanges C3. PHR C4. Smart hospital D1. Data utilization and sharing system D2. Data analysis environment D3. Basis for supporting data utilization Source: Ryu et al. (2013). ©WHO Digital health monitoring initiatives in eight countries 46 Monitoring the implementation of digital health: an overview of selected national and international methodologies Table 20. Main indicators collected in the 2020 survey of general hospitals Dimension Indicator(s) Provider-centric electronic systems (EMRs, order communication systems, picture archiving and communication systems, laboratory information systems, pharmacy management systems, referral management systems, HIE, telehealth) Availability of information systems by type Patient-centric electronic services and systems (websites, web-based PHRs, mobile PHRs) Availability of information systems and services by type Management information systems (administrative management, health insurance claims systems, enterprise resource planning systems, executive support systems, enterprise data warehouses) Availability of information systems by type Clinical research information systems (biobank information systems, clinical trial management systems, genome analysis information systems, clinical data management systems) Availability of information systems by type Governance (IT/ data) Existence of IT departments or areas Existence of personnel specialized in ICT Management and evaluation of data quality Adoption of electronic signatures Standards International standards adopted and applied Skills and competencies Types of training programmes available EHRs Use of EHRs Functionalities of EHR systems Types of patient data available in EHR systems EHR certification Health data security and privacy Existence of information security and privacy policies HIE Electronic sharing of clinical/patient medical information within and outside of organizations Sources: Jung (2021), Lee (2022). 47 Uruguay Uruguay is an upper-middle-income Latin American country with a population of 3  480  222 inhabitants (INE Uruguay, 2022), in a unitary State. The country stands out in the region for its low levels of socioeconomic inequality. In 2007 a reform of the health system was undertaken, which led to the National Integrated Health System (SNIS). The SNIS coordinates public and private health-care services within a comprehensive system that is financed by a common fund (the National Health Fund, FONASA). Legislation requires that institutions that provide health services, whether private or public and whether providing comprehensive or partial services, must participate in the SNIS. Most low- and medium-complexity health-care services are provided by institutions that are part of the SNIS. Highly complex medical services are provided via highly specialized medical institutes, which depend on the National Resources Fund (Fondo Nacional de Recursos), a universal insurance specifically designed to provide access to such procedures. The eHealth Uruguay initiative (Salud.uy) began in 2012 within the framework of the 2011–2015 Uruguayan Digital Agenda (Government of Uruguay, 2022), which included objectives and indicators related to the health sector, including the strategic use of ICT for improving the quality of medical services, and the implementation of EHRs integrated at national level. Subsequent editions of the Agenda have included goals in terms of innovation for social well-being and of integration of efforts among the education, health and social development policy sectors, taking advantage of the potential of digital technologies. The commitments and goals related to eHealth in the national digital agendas have included: ▶ 100% of comprehensive health providers to be in the National Electronic Medical Record in at least three organizational areas; ▶ 100% of public and private oncology services to be implementing electronic oncology medical records; and ▶ regulatory and technical requirements for implementing electronic medical prescriptions are enforced. Using timely survey data to monitor advances and barriers in the implementation of these initiatives was an integral part of the eHealth policy from the onset. Impact evaluation will also be implemented as soon as operational maturity is reached. National digital health measurement initiatives ICT in Health survey The Uruguayan ICT in Health survey follows the Brazilian ICT in Health survey model. It has been carried out every 2 years since 2014 by Salud.uy and the national Agency for E-Government and the Information and Knowledge Society (AGESIC) (Government of Uruguay, 2022). It collects data on three units of analysis: health-care facilities that provide comprehensive services within the SNIS, health-care professionals, and users. The results are presented by type of facility, capital/non-capital cities and public/private sector. Health-care facilities are surveyed in person, professionals are contacted via computer- assisted telephone interviews, and users are contacted on their cell phones (with a penetration rate of 90%). Digital health monitoring initiatives in eight countries 48 Monitoring the implementation of digital health: an overview of selected national and international methodologies Tables 21 and 22 present the available indicators (in addition to those already mentioned in the section on Latin America). Table 21. Indicators monitored for health-care facilities Dimension Indicator(s) for health-care facilities EHRs Types of clinical terminology standards Health data security and privacy Data backup strategies and frequency Use of advanced electronic signatures for electronic clinical documents Strategies adopted in matters of information security and cybersecurity Telehealth Availability of equipment for conducting teleconferences Table 22. Indicators monitored for users of the health-care system Dimension Indicator(s) for users of the health-care system Telehealth Use of online services available Use of teleconsultations Use of patient health records Use of the coronavirus UY app In 2020 a qualitative study was added to the survey to complement the quantitative findings by providing insight into the beliefs, attitudes and practices of both health-care users and professionals. ©WHO 49 50 Monitoring the implementation of digital health: an overview of selected national and international methodologies 51 The aim of this review was to map how digital health is being measured at the international, regional and national levels. Specifically, the review updates and consolidates the available information on recent monitoring activities, approaches and indicators employed by the EC, ECLAC, NeRN, OECD, PAHO and WHO in the past 5 years. It also reviews monitoring activities over the same period at national level in eight countries (Australia, Brazil, Costa Rica, Denmark, Italy, the Netherlands, Republic of Korea and Uruguay). The review confirms that efforts to improve measurement of the adoption and use of digital health technologies and approaches have been under way for more than a decade. Surveys are by far the most common data-gathering method, whether at international, regional or national level. Among the countries reviewed, they are commonly used in all three Latin American and Caribbean countries, Italy, the Netherlands and the Republic of Korea. A number of data-gathering approaches involving different organizations and institutions may coexist in countries. Denmark, for example, extensively harvests the log data generated from its national portal, sundhed.dk (European Observatory on Health Systems and Policies & Petersen, 2019). The National Board of Health IT uses the data to assess progress against the national digital strategy twice a year. In addition, MedCom collects data on clinical transactions and HIE, while the Danish Centre for Health Informatics, Aalborg University, conducts annual surveys of health professionals and a biennial survey of citizens’ use of health IT. Australia similarly uses a mixed approach that leverages the MHR system data analytics as well as drawing on administrative data and ad hoc surveys. In Italy, the Ministry of Health and the Agency for Digital Italy monitor the implementation and use of patient portals, the regional EHR systems, e-prescriptions and other associated digital services, while the National Institute of Health monitors the use of telehealth (Italian Ministry of Health, 2018). In order to gain a general view of the measurement strategy and progress of digital health, it was necessary to approach several national agencies and institutions in each country. The review also reveals the potential challenge of repeated survey efforts, particularly at the regional and international levels. Surveys are implemented infrequently (e.g. the WHO Global eHealth survey was last implemented in 2015) or discontinued (e.g. the European Hospital Survey). Ongoing measures of digital health progress are necessary, however, to support evidence-based policy and decision-making. At national level, surveys and their administration are generally tailored to local contexts and policy needs. The domains and indicators measured can vary substantially. Differences can be Main findings Main findings 52 Monitoring the implementation of digital health: an overview of selected national and international methodologies attributed to a range of issues, many of which have been examined and reported previously (Zelmer et al., 2017). Firstly, the domains and indicators monitored by countries depend on the state of advancement of digital health. Generally, measures of connectivity and digital health availability precede monitoring of use and user satisfaction, and monitoring of the level or intensity of use precedes the monitoring of data reuse, or of more advanced digital technologies and their impacts. Most countries that have achieved near-universal availability of EHRs no longer ask about the availability of internet connections and specific eHealth tools or services. Measurement is focused on the level of use of specific functionalities and health information exchange across the health-care sector. For example, in Denmark, MedCom monitors the most frequent text-based clinical transactions in Danish health care (e.g. discharge letters, referrals, laboratory test orders and e-prescriptions). The analysis also shows that, among the most advanced countries, there is general interest in measuring users’ experience and preferences, as well as the level of use of specific functionalities by patients to strengthen opportunities for self-care, self-management and patient participation. Improving measurement of patients’ access to their own health data is a focus in four of the countries reviewed. Digital health strategies in these countries aim to provide citizens with continuous electronic access to their health data. As an illustration, the annual eHealth Monitor produced by Nictiz and Nivel maps the state of advancement of eHealth in the Netherlands and collects data on both the distribution of digital HIE by region and the level of satisfaction of medical professionals and patients. The aim is not only to monitor the extent of use of digital technologies but also to assess whether there are differences across regions and to explain why digital tools and services may not be used and how their use can best be promoted. All of the countries reviewed had formulated national eHealth/digital health policies and developed strategies for implementation and monitoring. Three of the countries (Australia, Denmark and Italy) include indicators specifically aimed at tracking progress against their national digital strategy action plans. For example, the ADHA's 2017 benefits realization plan, which defined the parameters of success for the MHR system, includes intermediate output measures to be tracked using the MHR system’s data analytics. These measures include the number of individuals and health-care providers registered and the number and type of documents uploaded and viewed by different types of health-care provider. In Italy, to monitor the progress achieved by each region in the adoption and use of the EHR, the Ministry of Health, in agreement with the regional administrations, has established a series of indicators that are reported together with the percentage progression since the previous data collection. In Denmark, the 2021 benchmarking data from the Danish Health Data Authority include indicators and measurements for three of the main focus areas of the 2018–2022 national digital health strategy. Lastly, differences in the structure of health systems, such as how primary and acute care are organized, the extent of decentralization, the private/public mix of health-care organizations and services, and the size of the organizations offering the services, may affect the choice of indicators. The OECD guide to measuring ICTs in the health sector (OECD, 2015), for example, distinguishes between private and public, ambulatory and institutional, and primary and specialized care. For some of the Nordic country surveys, while private/public and primary/ specialized care distinctions are possible, the ambulatory/institutional distinction is not possible. In the Republic of Korea, the surveys covered both public and private hospitals without distinction as they are all included under the national health insurance and 90% of health-care providers are private. 53 These variations across countries were not unexpected and confirm the potential value of a modular approach to international measurements, as recommended by the OECD’s model survey framework. To be broadly useful, a model survey is composed of separate, self-contained modules that ensure flexibility and adaptability to a rapidly changing environment. Core modules can be added as digital health and information needs evolve, while supplemental modules can be used to address new policy needs or technological developments. At the same time, the review uncovered a number of significant promising commonalities that warrant further examination and are discussed in the next sections. Adoption of a functionality-based approach A general issue when comparing systems and services across countries is subtle differences in terminology. If physicians are asked a question about EHR or EMR use, differences in interpretation and approach across countries could impede meaningful benchmarking. Collecting and comparing data on functionalities rather than systems is one way of overcoming these differences. This approach also supports technology neutrality (i.e. the questions do not require or assume a particular technology), is forward-looking (i.e. does not hinder the use or development of new technologies in the future) and allows the assessment of digital health maturity. All of the countries included in this review have adopted some functionality-based approach in their data collection and survey efforts, that is, measures are focused on the types of clinical and other activities that are supported by digital systems rather than the availability of specific technologies. This reduces the effects of variations in terminology between countries and should be encouraged and supported. Digital domains, actors and activities covered With a few exceptions, the monitoring strategies of the countries reviewed in this report focus on both organizations (health-care facilities) and health-care professionals. Among health-care professionals, GPs are the target population of the surveys in most countries. In Latin American countries, the surveys also consider other types of physician, such as specialists. Nurses appear to be regularly surveyed only in Brazil, the Netherlands and Uruguay. Surveys aimed at citizens or users of health-care services are less common. They were reported primarily by Denmark, the Netherlands and Uruguay, although there is evidence of academic studies on usability and consumer attitudes towards the MHR in Australia as well. The Danish Centre for Health Informatics, Aalborg University, has conducted an annual survey of health professionals and a biennial survey of citizens’ use of health IT. Log data of citizens’ use of e-records are also largely exploited for these studies and accessed through national portals. In the Netherlands, the eHealth Monitor is based on the results of semi-structured interviews with physicians, nurses and mental health nurse practitioners and the results of a workshop with citizens. An initial mapping of surveys on citizens’ attitudes to eHealth in the Nordic countries was conducted in 2019 by NeRN. In Uruguay, the Salud.uy initiative and AGESIC conduct biennial surveys of health professionals, including nurses, as well as of citizens. In 2020 a qualitative study using focus groups was also carried out to consult health professionals and health-care users and to explore citizens’ understanding of and attitudes towards the newly implemented patient electronic records system. Main findings 54 Monitoring the implementation of digital health: an overview of selected national and international methodologies While most countries monitor provider-centric and patient-centric records, there is significant variability in how HIE and telehealth are currently monitored. In Denmark, for example, MedCom monitors all hospital messages sent, GP messages sent, specialist practice doctors’ messages sent, municipalities’ messages sent related to health and social care, e-records consultations, hospital discharge letters sent, X-ray reports sent, X-ray referrals received, laboratory reports sent, and laboratory requisitions received. At present, Italy monitors the state of interoperability of the national EHR and uses administrative data to monitor electronic prescriptions. In only a few cases, monitoring also considered processes and workflows (e.g. in Denmark and the Netherlands). There is wide variation in measurements of telehealth capacity, which is generally not monitored systematically in the countries reviewed here. Inclusion of policy-level domains In addition to information on the availability and use of ICT-based systems and applications, it is essential to promote data collection on the contextual information that is critical to the implementation of digital health policies. This includes not just the necessary physical conditions, such as infrastructure and connectivity, but also organizational characteristics and skills. All countries included some common policy questions, specifically on digital literacy, skills and competencies, privacy, and security. Unfortunately, the surveys generally do not contain information on organizational changes or the input (monetary and non-monetary) behind the registered level of deployment. The goal of benchmarking should ideally be to monitor both implementation of policy and organizational learning to allow evaluation. However, with the exceptions of Denmark and the Netherlands, national measurements appear to focus primarily on specific targets and broad quantitative measures of inputs or outputs and less on the policies, processes and organizational factors involved. Because of the methodology adopted, this review may have missed more qualitative ad hoc studies addressing these important aspects. Data sharing and reuse Efforts to measure governance and reuse of health data across health-care systems and technical and operational readiness to share the data for statistical and research purposes are also emerging as a common priority. Many countries have made progress in creating and improving electronic patient records and health-care databases in recent years. However, as highlighted in recent OECD surveys (Oderkirk, 2021), datasets from different parts of a country’s health system are still often unable to communicate with each other electronically. Linking health and behavioural data from a wide variety of sources can yield particularly powerful public health insights. For example, data from medical settings can be combined with data on social determinants of health, thereby offering new targets for personalized care and social intervention. Although the majority of countries that responded to a recent OECD survey (9 of 15 OECD countries) outlined examples of the use of digital health information to inform public health initiatives, just three countries (Canada, Estonia and the Netherlands) reported using non-traditional sources of data to improve public health (OECD, 2019). 55 Measures of patients’ involvement in their own care and digital health inequalities Involving patients in their own care by allowing them access to their own data is a trend seen in many countries. Most of the countries reviewed are increasingly offering their citizens access to their personal health-related data to strengthen opportunities for self-care, self-management, and patient participation. In Australia, Denmark and Uruguay, all citizens currently have access to their personal health data via e-portals. Monitoring in these countries includes the type and level of access citizens have to their own health data. Additionally, ad hoc surveys address possible inequalities and whether digital services for patients are inclusive and accessible. Inequalities in availability and use across regions are also monitored in Italy and the Netherlands. All countries attempt to monitor digital literacy and skills. Data collection methods Typically, nationwide indicators can be monitored using administrative records, censuses, sample surveys, assessments and tests. Recently, digitization of services and processes has led to the use of data provided by electronic records (such as EHRs and PHRs) to compile national indicators. Other unstructured big data sources are also being used to support decision-making in the sector. Data collection methods vary significantly across the countries reviewed, and depend on the maturity of the digital health system. Australia uses the MHR system data analytics to measure the number of individuals and health-care providers registered and the number and type of documents uploaded and viewed by different types of health-care providers. Since the Nordic countries all have close to 100% national distribution of the most significant digital health functionalities (e.g. HIE and patient-centric/patient portal functionalities), NeRN has developed a number of variables to measure practical use of eHealth systems using log data harvested from central servers. Among the Latin American countries considered, the implementation of sample surveys is still a key strategy. Although these countries are starting to use national data repositories to monitor indicators, sample surveys are still the main tool to track digital health availability and usage. Most of the countries rely on self-administered online questionnaires for data collection. Some countries, such as Brazil, use mixed modes that combine computer-assisted telephone interviews and computer-assisted web interviews. Table 23 presents a brief overview of the methodologies used in the countries reviewed. Main findings 56 Monitoring the implementation of digital health: an overview of selected national and international methodologies Table 23. Data collection methodologies reported by countries Country Methodology (institution, target population, sample, data collection method) Australia ADHA has monitored the success of the MHR since 2017. The data are collected using the MHR system data analytics. Measures include the number of individuals and health-care providers registered and the number and type of documents uploaded and viewed by different types of health-care providers. Data on health-care providers are broken down by general practice, public and private hospitals, pharmacies, specialists, allied health, and aged care. ADHA has also proposed adopting a digital maturity assessment process similar to ones proposed in other countries, such as the United Kingdom. Since 2020, Australian MBS service data have included telehealth uptake. These data can be obtained from the Medicare item reports website published by the Australian Government (Services Australia, 2022). Brazil The ICT in Health survey has been conducted annually since 2013 by Cetic.br, covering health-care facilities, nurses and physicians. The results are presented by type of facility, region, location, administrative jurisdiction and age group (professionals). Data collection is carried out by CATI and via the web. The survey provides more than 80 indicators and is based on the ECLAC adaptation of the model survey produced by OECD (OECD, 2015). DATASUS sporadically publishes monitoring data of the Digital Health Strategy Program on the public health- care facilities. Costa Rica The ICT in Health survey follows the Brazilian model and has been carried out once, in 2018. It consisted of a census of basic health-care units under CCSS jurisdiction, covering health-care facilities, physicians and nurses. The questionnaire was implemented via the web. Denmark Denmark has an advanced digital health-care system. Most of the statistics are produced from log data generated by the sundhed.dk portal and registries such as the National Prescription Registry. The Danish Centre for Health Informatics at Aalborg University conducts annual surveys of health professionals and biennial surveys of citizens’ use of health IT. MedCom reports on traffic statistics and the Danish Health Data Authority tracks progress against the national digital health strategy action plans. The data collected are aimed at monitoring the use of portals, information exchange, message standardization, and exchange traffic for hospitals, general clinics and municipalities, as well as progress against the national digital health strategy action plans. 57 Country Methodology (institution, target population, sample, data collection method) Italy The Agency for Digital Italy and the Ministry of Health have established a series of indicators to monitor the implementation and progress of digital health policies for public health-care facilities. For each indicator, the percentage progression is reported, thus highlighting the improvement since the previous data collection. Data are reported on a quarterly basis by each region. Also, the Italian National Health Service has recently started to map the adoption and use of telehealth by the regions with an online survey. Netherlands Since 2013, in the annual eHealth Monitor, Nictiz and Nivel map the progress of eHealth in the Netherlands and collect data on the state of the digital transition across health-care sectors and regions. The Monitor covers use in general practice, hospitals and long-term care facilities and among patients and the general population. The Monitor is based on the results of semi- structured interviews and questionnaires filled out by members of the Dutch Healthcare Consumer Panel, members of the National Panel of the Chronically Ill and Disabled, physicians, nurses, and mental health nurse practitioners. Additionally, a 1-day workshop is held, albeit not every year, with participants on the Dutch Healthcare Consumer Panel to discuss the results. Republic of Korea The KHIS has conducted three surveys on the state of adoption of eHealth, in 2015, 2017 and 2020, covering tertiary teaching hospitals, and specialist and general hospitals. The survey was designed to be comparable with the OECD indicators. In 2016 the Research Institute for Healthcare Policy of the Korean Medical Association carried out the Korean Physician Survey. Uruguay The ICT in Health survey follows the Brazilian model and has been carried out every 2 years since 2014 by the Salud.uy Program and AGESIC, covering health-care facilities, professionals and patients. The results are presented by type of facility, capital/non-capital cities and public/private sector. Health-care facilities are surveyed face to face, professionals are contacted via CATI and users are contacted on their cell phones. CATI: computer-assisted telephone interviews. Table 23 contd Main findings 58 Monitoring the implementation of digital health: an overview of selected national and international methodologies New digital health domains The COVID-19 pandemic has accelerated the need for greater interaction among health-care professionals, the use of new technologies and data analysis. Digital technologies can be used at several stages of this interaction: care provision, through supporting patient risk analysis, prognosis and decisions about medical care; short-term planning, such as in the organization of teams and resources, and in the management of hospitals, municipalities and states; and long-term planning, such as in the assessment of public policies and strategies (CGI.br, 2021a). Technologies such as cloud computing, big data analytics, artificial intelligence and robotics are contributing to the transformation of processes and the expansion of digital health. As new digital technologies that bring about disruptive innovation of health research and health-care delivery are adopted, monitoring their use in health-care facilities and by health professionals is increasingly relevant. Brazil, for example, adapted indicators developed by Eurostat to measure implementation among health-care facilities. Among the new data collected are: ▶ types of cloud computing services used ▶ performance of big data analytics ▶ sources used for big data analytics ▶ use of other new technologies, including blockchain, artificial intelligence and robotics. Measuring digital health maturity There is growing interest in the notion of digital health maturity, which provides a framework for coordination of digital activities in the pursuit of multiple aims, including improving population health outcomes and eliminating disparities, guiding resource allocation and controlling costs, enhancing patient experience, and improving the working conditions of health-care providers. To be successful, large-scale digital change programmes need to take digital maturity into account. Understanding their digital health maturity will allow countries to align their knowledge, skills and resources to systematically develop, implement and evaluate standards-based interoperable digital health systems and programmes to support and sustain their health priorities. However, new frameworks and indicators for assessing digital maturity in relation to these complex goals are needed. It is important to recognize that the digital health tools used to strengthen health systems at all levels of care are also potential sources of data for the measurement and monitoring of indicators of digital health maturity, including the impacts on care, quality improvement, health professional development and risk management (Liaw et al., 2021). 59 60 Monitoring the implementation of digital health: an overview of selected national and international methodologies 61 The review also shows how the indicators and methodological approaches have been under continuous development. The domains and indicators measured vary substantially between countries. These differences can be attributed to a variety of issues, many of which have been examined and reported previously, such as differences in the structure of health systems, the state of advancement of digital health, and policy and strategic priorities. Only a limited number of dimensions are currently available for monitoring across all of the countries included in this review, and the data are often collected using different surveys, approaches and methodologies. To exploit the potential of current measurement initiatives, progress is needed in the development and collection of comparable data and a common set of indicators, as well as sharing of information and knowledge for measuring national digital health maturity levels. Given the observed differences across countries, this study confirms the potential value of a modular approach to international measurements, so that measurement work can move forward in stages. To be broadly useful, a model survey would need to be composed of separate, self-contained modules that ensure flexibility and adaptability to a rapidly changing environment. Core modules can be added on as digital health and information needs evolve, while supplemental modules can be used to address new policy needs or technological developments. In this sense, the OECD and Latin American frameworks are valuable points of reference. Similarly, the efforts to disseminate them and build capacity for their implementation showcase the potential of multistakeholder collaboration, including research centres, international organizations and governments, in producing policy-relevant and internationally comparable digital health data. To facilitate benchmarking and learning and to generate trust, monitoring activities and data should be transparent and published. However, with a few exceptions, for this review it was necessary to approach several national agencies and institutions in each country to obtain a general view of measurement strategies and the state of advancement of digital health. A governance structure for coordination of national digital health monitoring should be established to ensure that valid data are collected regularly on progress in digital health implementation and to make the information easily visible and accessible. This study reveals the significant efforts made in the last few years to monitor the adoption and use of digital health approaches at both the national and international levels. Monitoring the adoption of digital health is clearly perceived as key to learning from initiatives and to providing evidence on which decision-makers can base digital health policy decisions. Conclusions Conclusions 62 Monitoring the implementation of digital health: an overview of selected national and international methodologies The following areas require follow-up work and attention. ▶ Address the significant variability in how telehealth is monitored. The importance of telehealth cannot be overstated. The COVID-19 pandemic has caused health-care systems around the globe to rapidly and, in some cases, radically rethink the delivery of medical care and recognize the potential of telehealth. There is wide cross-country variation in measurements of telehealth capacity, which is generally not systematically monitored. Telehealth can have different meanings in different contexts. Therefore, there is a need to support the development of a common understanding of terms and formulate standard questions to gain insights and broadly applicable learning on the potential of telehealth. The usability of telehealth applications should be a key focus of measurement in this area. ▶ Design new indicators to monitor digital health inequalities. Most of the countries reviewed are increasingly offering their citizens remote access to care as well as to their personal health-related data. There is a need to monitor the type and level of access that citizens have to digital health services, the challenges experienced by citizens, and the skills needed in the digital health era to address the potential risk of exacerbating inequalities among the most vulnerable. The Netherlands and Uruguay may be taken as references in this regard. ▶ Review new sources of health data and approaches to data collection. With few exceptions, measurement still tends to focus on specific eHealth targets and broad quantitative measures of inputs or outputs, rather than on the policies, processes and organizational factors involved. Measurements need to be adapted to provide more granular insights, including on how information systems leverage the emerging new digital sources of health and behavioural data and the potential of health data reuse. 63 64 Monitoring the implementation of digital health: an overview of selected national and international methodologies 65 ADHA (2021). My health record: statistics and insights. November 2021. Canberra: Australian Digital Health Agency (https://www.digitalhealth.gov.au/initiatives-and-programs/my- health-record/statistics, accessed 30 August 2022). Australian National Audit Office (2019). Section 4. Monitoring and evaluation. Paragraph 4.17. In: Implementation of the My Health Record System. Canberra: Australian National Audit Office (https://www.anao.gov.au/work/performance-audit/implementation-the-my-health- record-system, accessed 30 August 2022). Brazilian Ministry of Health (2015). Ordinance MS No. 589 of May 20, 2015. Institutes the National Health Information and Informatics Policy (PNIIS). Rio de Janeiro: Ministry of Health (https://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt0589_20_05_2015.html, accessed 30 August 2022). Brazilian Ministry of Health (2020). Brazilian National Digital Health Strategy 2020–2028. Rio de Janeiro: SUS Informatics Department, Ministry of Health (https://www.gov.br/saude/pt- br/assuntos/saude-digital/a-estrategia-brasileira/strategy_health_digital_brazilian.pdf, accessed 30 August 2022). Bruins B (2019). Derde Kamerbrief over elektronische gegevensuitwisseling in de zorg. [Third letter to Parliament about on electronic data exchange in health care]. The Hague: Ministry of Health, Welfare and Sport (https://www.gegevensuitwisselingindezorg.nl/publicaties/ brieven/2019/07/12/derde-kamerbrief-gegevensuitwisseling, accessed 30 August 2022) (in Dutch). Cetic.br, UNESCO Institute for Statistics (2020). Practical guide to implement surveys on ICT use in primary and secondary schools. São Paulo: Brazilian Internet Steering Committee (https://cetic.br/pt/publicacao/practical-guide-to-implement-surveys-on-ict-use-in- primary-and-secondary-schools/, accessed 30 August 2022). CGI.br (2021a). Survey on the use of information and communication technologies in Brazilian health care facilities: ICT in health 2021. São Paulo: Brazilian Internet Steering Committee (https://cetic.br/media/docs/publicacoes/2/20211130124545/tic_saude_2021_ livroeletronico.pdf, accessed 30 August 2022). CGI.br (2021b). COVID-19. ICT Panel. Web survey on the use of internet in Brazil during the new coronavirus pandemic. São Paulo: Brazilian Internet Steering Committee (https://cetic. br/media/docs/publicacoes/2/20210426095323/painel_tic_covid19_livro_eletronico.pdf, accessed 30 August 2022). Codagnone C, Lupiáñez-Villanueva F (2013). Benchmarking deployment of eHealth among general practitioners. Annex I. Technical compendium. Brussels: European Commission Directorate-General for Communications Network, Content and Technology(SMART 2011/0033; https://www.digitalhealthnews.eu/download/publications/3880-benchmarking- deployment-of-ehealth-among-general-practitioners-2013, accessed 30 August 2022). References References 66 Monitoring the implementation of digital health: an overview of selected national and international methodologies Costa Rican Ministry of Science, Innovation, Technology and Telecommunications (2015). Unidad de Planificación Institucional. Plan Nacional de Ciencia, Tecnología e Innovación 2015–2021 [National plan for science, technology and innovation, 2015–2021]. San José: Ministry of Science, Innovation, Technology and Telecommunications (http://www.conicit. go.cr/ver/sic/Biblioteca_virtual/Publicaciones/publica_cyt/prog_nac_cyt/Plan_NCTI-15- 21.pdf#.YxTIrHbMKSl, accessed 30 August 2022) (in Spanish). Costa Rican Ministry of Science, Innovation, Technology and Telecommunications (2019). Digital transformation strategy: the bicentennial of Costa Rica 2018–2022. San José: Ministry of Science, Innovation, Technology and Telecommunications (https://www.micitt.go.cr/wp content/uploads/2022/05/TransfDigitalCR-version-ingles-impreso-version-30-01-2020- FINAL.pdf, accessed 30 August 2022). Danish Ministry of Health (2018). A coherent and trustworthy health network for all: digital health strategy 2018–2022. Copenhagen: Ministry of Health (https://sundhedsdatastyrelsen. dk/-/media/sds/filer/strategi-og-projekter/strategi-digital-sundhed/digital-health- strategy-2018_2022.pdf, accessed 30 August 2022). DATASUS (2020a). Plano de ação, monitoramento e avaliação da estratégia de saúde digital para o Brasil 2019–2023 [Action plan, monitoring and evaluation of the digital health strategy for Brazil 2019–2023]. Rio de Janeiro: SUS Informatics Department, Ministry of Health (https://www.gov.br/saude/pt-br/assuntos/saude-digital/a-estrategia-brasileira/ PlanodeAoMonitoramentoeAvaliao.pdf, accessed 30 August 2022) (in Portuguese). DATASUS (2020b). 2º Relatório de monitoramento e avaliação estratégia de saúde digital para o Brasil 2020–2028: dezembro de 2021 [2nd Monitoring and evaluation report digital health strategy for Brazil 2020–2028: December 2021]. Rio de Janeiro: SUS Informatics Department, Ministry of Health (https://www.gov.br/saude/pt-br/assuntos/saude-digital/ monitoramento-e-avaliacao-da-esd/RelatrioMA2.pdf, accessed 30 August 2022) (in Portuguese). Dobrev A, Haesner M, Hüsing T, Korte WB, Meyer I (2008). Benchmarking ICT use among general practitioners in Europe. Final report, Bonn, April 2018. Luxembourg: Publications Office of the European Union (https://op.europa.eu/en/publication-detail/-/ publication/7d72981d-f924-4977-a032-37361bb8b4b3/language-en/format-PDF/source- search, accessed 30 August 2022). Dutch Ministry of Health, Welfare, and Sport (2018). Outcome-based health care 2018–2022. The Hague: Ministry of Health, Welfare, and Sport (https://www.government. nl/topics/quality-of-healthcare/documents/reports/2018/07/02/outcome-based- healthcare-2018-2022, accessed 30 August 2022). Empirica, Open Evidence (2019). eHealth, interoperability of health data and artificial intelligence for health and care. Lot I. Interoperability of electronic health records in the EU. SMART 2019/0056: a study prepared for the European Commission Directorate-General for Communications Networks, Content and Technology. Bonn: Empirica (https://empirica.com/ project/details/?projectid=291, accessed 30 August 2022). 67 EC (2022). The digital economy and society index [website]. In: Shaping Europe’s digital future, policies. Brussels: European Commission Directorate-General for Communications Networks, Content and Technology (https://digital-strategy.ec.europa.eu/en/policies/desi, accessed 30 August 2022). EC Joint Research Centre, Institute for Prospective Technological Studies (2013). European hospital survey. Benchmarking deployment of e-health services (2012–2013). Brussels: European Commission Directorate-General for Communications Networks, Content and Technology (SMART 2012/0036; https://digital-strategy.ec.europa.eu/en/library/european- hospital-survey-benchmarking-deployment-ehealth-services-2012-2013, accessed 30 August 2022). European Observatory on Health Systems and Policies, Petersen ME (2019). Achieving better health and wellbeing via the Danish E-Health portal sundhed.dk. Eurohealth. 25(2):20–3. Copenhagen: WHO Regional Office for Europe (https://apps.who.int/iris/ handle/10665/332595, accessed 30 August 2022). Eurostat (2018). Community survey on ICT usage and e-commerce in enterprises. Brussels: Statistical Office of the European Commission (https://ec.europa.eu/eurostat/statistics- explained/index.php?title=Digital_economy_and_society_statistics_-_enterprises, accessed 30 August 2022). Eurostat (2021). Digital economy and society: methodology. Brussels: Statistical Office of the European Union (https://circabc.europa.eu/ui/group/4f80b004-7f0a-4e5a-ba91- a7bb40cc0304/library/a45ba5dd-2a7c-42e9-9078-dadb28a37ee0/details, accessed 30 August 2022). Government of Uruguay (2022). Salud.uy [website]. In: Agencia de Gobierno Electrónico y Sociedad de la Información y del Conocimiento [E-Government and the Information and Knowledge Society Agency]. Montevideo: Government of Uruguay (https://www.gub.uy/ agencia-gobierno-electronico-sociedad-informacion-conocimiento/saluduy, accessed 30 August 2022) (in Spanish). Hyppönen H, Ammenwerth E, de Keizer N (2012). Exploring a methodology for eHealth indicator development. Stud Health Technol Inform. 80:338–42. PMID: 22874208. Hyppönen H, Kangas M, Reponen J, Nøhr C (2013). Nordic eHealth indicators: organisation of research, first results and the plan for the future (TemaNord 2013:522). Copenhagen: Nordic Council of Ministers. doi: 10.6027/TN2013-522. Hyppönen H, Kangas M, Reponen J, Nøhr C (2015). Nordic ehealth benchmarking (TemaNord 2015:539). Copenhagen; Nordic Council of Ministers. doi: 10.6027/TN2015-539. Hyppönen H, Koch S, Faxvaag A, Gilstad H, Nøhr C, Hardardottir GA et al. (2017). Nordic eHealth benchmarking: from piloting towards established practice (TemaNord 2017:528). Copenhagen: Nordic Council of Ministers. doi: 10.6027/TN2017-528. INE Uruguay (2022). Población [Population]. In: Demografía y Estadísticas Sociales [Demographics and Social Statistics] [website]. Montevideo: Instituto Nacional de Estadística Uruguay (https://www.ine.gub.uy/poblacion, accessed 30 August 2022) (in Spanish). References 68 Monitoring the implementation of digital health: an overview of selected national and international methodologies INEC Costa Rica (2022). Estimaciones y proyecciones de población [Population estimates and projections] [website]. San José: Instituto Nacional de Estadística y Censos de Costa Rica (https://www.inec.cr/poblacion/estimaciones-y-proyecciones-de-poblacion, accessed 30 August 2022) (in Spanish). Italian Ministry of Health (2018). Mappatura delle esperienze di telemedicina: sul territorio nell’anno 2018 [Mapping telemedicine experiences in the territory in 2018]. Rome: Ministry of Health (https://www.salute.gov.it/imgs/C_17_pagineAree_2515_2_file.pdf, accessed 30 August 2022) (in Italian). Jung J (2021). South Korea’s My HealthWay: a “digital highway” of personal health records, but to where? BMJopinion. 24 September 2021 (https://blogs.bmj.com/bmj/2021/09/24/ south-koreas-my-healthway-a-digital-highway-of-personal-health-records-but-to-where/, accessed 30 August 2022). Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A et al. (2012). A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care. 28:44–51. doi: 10.1017/S0266462311000638. Lee J-H (2022). Era of personal health records in Korea. Healthc Inform Res. 28(1):1–2. doi: 10.4258/hir.2022.28.1.1. Liaw ST, Zhou R, Ansari S, Gao J (2021). A digital health profile and maturity assessment toolkit: cocreation and testing in the Pacific Islands. J Am Med Inform Assoc. 28(3):494–503. doi: 10.1093/jamia/ocaa255. Lupiáñez-Villanueva F, Folkvord F, Faulí C (2018). Benchmarking deployment of eHealth among general practitioners: executive summary. Luxembourg: Publications Office of the European Union (https://data.europa.eu/doi/10.2759/511610, accessed 30 August 2022). MedCom (2021). Overall traffic monitoring 1994–2021. Odense: MedCom (https://statistik. medcom.dk/exports/medcom_monitorering_en.pdf, accessed 30 August 2022). Meyer I, Hüsing T, Didero M, Korte WB (2009). eHealth benchmarking (phase II): final report. Brussels: European Commission Information Society and Media Directorate (https://joinup. ec.europa.eu/collection/ehealth/document/eu-ehealth-benchmarking-phase-ii, accessed 30 August 2022). NHS England (2016). Digital maturity assessment. Leeds: National Health Service of England (https://www.england.nhs.uk/digitaltechnology/info-revolution/maturity-index/, accessed 30 August 2022). Nictiz, Nivel (2019). Electronic data exchange and communication between health care providers. Theme discussion 5: eHealth Monitor 2019. The Hague: Institute for Health Services Research (https://nictiz.nl/publicaties/theme-discussion-5-electronic-data-exchange/, accessed 30 August 2022). Nøhr C, Faxvaag A, Tsai CH, Hardardottir GA, Hyppönen H, Andreassen HK et al. (2020). Nordic eHealth benchmarking: towards evidence informed policies (TemaNord 2020:505). Copenhagen: Nordic Council of Ministers. doi: 10.6027/temanord2020-505. 69 Oderkirk J (2021). Survey results: national health data infrastructure and governance (OECD Health Working Papers, No. 12). Paris: Organisation for Economic Co-operation and Development. doi: 10.1787/55d24b5d-en. OECD (2015). Draft OECD guide to measuring ICTs in the health sector. Paris: Organisation for Economic Co-operation and Development (https://www.oecd.org/health/health-systems/ Draft-oecd-guide-to-measuring-icts-in-the-health-sector.pdf, accessed 30 August 2022). OECD (2019). Health in the 21st century: putting data to work for stronger health systems, OECD Health Policy Studies. Paris: Organisation for Economic Co-operation and Development (https://www.oecd.org/publications/health-in-the-21st-century-e3b23f8e-en.htm, accessed 30 August 2022). PAHO (2019). Perfil del sistema y servicios de salud de Costa Rica con base al marco de monitoreo de la Estrategia Regional de Salud Universal [Profile of the Costa Rican health system and services based on the monitoring framework of the Regional Universal Health Strategy]. San José; Pan American Health Organization (https://iris.paho.org/ handle/10665.2/38590, accessed 30 August 2022) (in Spanish). PAHO, NIC.br (2019). Measurement of digital health: methodological recommendations and case studies. São Paulo: Brazilian Internet Steering Committee (https://cetic.br/media/docs/ publicacoes/1/measurement%20of%20digital%20health.pdf, accessed 30 August 2022). RIVM (2022). E-health monitor 2021. Stand van zaken digitale zorg [State of digital healthcare]. Bilthoven: National Institute for Public Health and Environment (https://www.rivm.nl/ sites/default/files/2022-01/E-healthmonitor%202021%20Stand%20van%20zaken%20 digitale%20zorg_0.pdf, accessed 30 August 2022) (in Dutch). Ryu S, Park M, Lee J, Kim SS, Han BS, Mo KC et al. (2013). Web-based integrated public healthcare information system of Korea: development and performance. Healthc Inform Res. 19(4):314–23. doi: 10.4258/hir.2013.19.4.314. Scott RE, Saeed A (2008). Global eHealth – measuring outcomes: why, what, and how. A report commissioned by the World Health Organization’s Global Observatory for eHealth. In: Making the eHealth Connection, Bellagio, Italy, July 2008 (https://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.502.6475&rep=rep1&type=pdf, accessed 30 August 2022). Services Australia (2022). Medicare item reports. In: Medicare Statistics [website]. Canberra: Australian Government (http://medicarestatistics.humanservices.gov.au/statistics/mbs_ item.jsp, accessed 30 August 2022). Snoswell CL, Caffery LJ, Taylor ML, Haydon HM, Thomas E, Smith AC (2020). Telehealth and coronavirus: Medicare benefits schedule (MBS) activity in Australia. Brisbane: Centre for Online Health, University of Queensland (https://coh.centre.uq.edu.au/telehealth-and-coronavirus- medicare-benefits-schedule-mbs-activity-australia, accessed 30 August 2022). WHO (2005). Resolutions and decisions: annex. In: Fifty-Eighth World Health Assembly, Geneva, 16–25 May 2005 (document WHA58/2005/REC/1) Geneva: World Health Organization (https://apps.who.int/iris/handle/10665/20398, accessed 30 August 2022). WHO (2016a). Global diffusion of eHealth: making universal health coverage achievable. Report of the third Global Survey on eHealth. Geneva: World Health Organization (https://apps.who.int/iris/handle/10665/252529, accessed 30 August 2022). References 70 Monitoring the implementation of digital health: an overview of selected national and international methodologies WHO (2016b). Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. Geneva: World Health Organization (https://apps.who. int/iris/handle/10665/252183, accessed 30 August 2022). WHO (2020). Resolution 73(28). Global strategy on digital health. In: Seventy-third World Health Assembly, Geneva, 9–14 November 2020. Geneva: World Health Organization (https:// apps.who.int/gb/ebwha/pdf_files/WHA73/A73(28)-en.pdf, accessed 30 August 2022). WHO (2021). Global strategy on digital health 2020–2025. Geneva: World Health Organization (https://apps.who.int/iris/handle/10665/344249, accessed 30 August 2022). WHO, United Nations Foundation, UNDP/UNFPA/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction, Johns Hopkins University (2015). The MAPS toolkit: mHealth assessment and planning for scale. Geneva: World Health Organization (https://apps.who.int/iris/handle/10665/185238, accessed 30 August 2022). WHO/GOe (2006). Building foundations for eHealth: progress of Member States. Report of the Global Observatory for eHealth. Geneva: World Health Organization (https://apps.who.int/ iris/handle/10665/43599, accessed 30 August 2022). WHO/GOe (2016). Atlas of eHealth country profiles: the use of eHealth in support of universal health coverage: Based on the findings of the third Global Survey on eHealth 2015. Geneva: World Health Organization (https://www.who.int/publications/i/item/9789241565219, accessed 30 August 2022). WHO/GOe (2022). Global Observatory for eHealth [website]. In: World Health Organization. Geneva: World Health Organization (https://www.who.int/observatories/global-observatory- for-ehealth, accessed 30 August 2022). WHO Regional Office for Europe (2021a). Support tool to strengthen health information systems: guidance for health information system assessment and strategy development. Copenhagen: WHO Regional Office for Europe (https://apps.who.int/iris/ handle/10665/342126, accessed 30 August 2022). WHO Regional Office for Europe (2021b). Measurement framework for the European Programme of Work, 2020–2025: approach, targets, indicators and milestones. Copenhagen: WHO Regional Office for Europe (https://apps.who.int/iris/handle/10665/343314, accessed 30 August 2022). Zelmer J, Ronchi E, Hyppönen H, Lupiáñez-Villanueva F, Codagnone C, Nøhr C et al. (2017). International health IT benchmarking: learning from cross-country comparisons. J Am Med Inform Assoc. 24(2):371–79. doi: 10.1093/jamia/ocw111. 71 72 Monitoring the implementation of digital health: an overview of selected national and international methodologies THE WHO REGIONAL OFFICE FOR EUROPE The World Health Organization (WHO) is a specialized agency of the United Nations created in 1948 with the primary responsibility for international health matters and public health. The WHO Regional Office for Europe is one of six regional offices throughout the world, each with its own programme geared to the particular health conditions of the countries it serves. Albania Andorra Armenia Austria Azerbaijan Belarus Belgium Bosnia and Herzegovina Bulgaria Croatia Cyprus Czechia Denmark Estonia Finland France Georgia Germany Greece Hungary Iceland Ireland Israel Italy Kazakhstan Kyrgyzstan Latvia Lithuania Luxembourg Malta Monaco Montenegro Netherlands North Macedonia Norway Poland Portugal Republic of Moldova Romania Russian Federation San Marino Serbia Slovakia Slovenia Spain Sweden Switzerland Tajikistan Türkiye Turkmenistan Ukraine United Kingdom Uzbekistan MEMBER STATES World Health Organization Regional Office for Europe UN City, Marmorvej 51, DK-2100, Copenhagen Ø, Denmark Tel.:+4545337000; Fax:+4545337001 Email: eurocontact@who.int Web site: www.who.int/europe WHO/EURO:2022-5985-45750-65816