G U ID A N C E FO R C R E A T IN G I M P A C T FU L H E A LT H R E P O R T S Support tool to strengthen health information systems Guidance for health information system assessment and strategy development
Support tool to strengthen health information systems Guidance for health information system assessment and strategy development Abstract: This tool is an update of the 2015 Support tool to assess health information systems and develop and strengthen health information strategies. Both the global General Programme of Work and the European Programme of Work 2020–2025 emphasize the importance of supporting Member States for strengthening their health information systems (HISs) and this updated tool will be instrumental in this endeavour. The Tool has two main parts: guidance for performing an assessment of a full HIS and guidance for the subsequent development of an HIS strategy. The assessment methodology has been updated to reflect the current HIS context in the WHO European Region through achieving a better balance between data collection and actual data use and a better reflection of the growing importance of electronic health records and other digital solutions. Add-on modules provide support for more in-depth assessment of infectious disease surveillance, noncommunicable disease monitoring, health information for both programmes of work, human resources for health and health data governance. Finally, the guidance for HIS strategy development has been made more concise and practical. Keywords: HEALTH INFORMATION SYSTEMS, DECISION MAKING, HEALTH POLICY, HEALTH INFORMATION MANAGEMENT, MEDICAL INFORMATICS, DATA COLLECTION ISBN: 978-92-890-5574-1 © World Health Organization 2021 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition: Support tool to strengthen health information systems: guidance for health information system assessment and strategy development. Copenhagen: WHO Regional Office for Europe; 2021”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization (http://www.wipo.int/amc/en/mediation/rules/). Suggested citation. Support tool to strengthen health information systems: guidance for health information system assessment and strategy development. Copenhagen: WHO Regional Office for Europe; 2021. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Contents Acknowledgements ..............................................................................................................................................v Abbreviations ........................................................................................................................................................vi Executive summary ............................................................................................................................................vii Introduction ............................................................................................................................................................1 Supporting the implementation of WHO’s European Programme of Work ......................................2 What is new in this updated version of the tool? ....................................................................................4 Setting the scene: health information systems and their governance ......................................................6 HIS: what are we actually talking about? .................................................................................................6 HISs and related concepts: population health surveillance and monitoring, M&E, and health system performance assessment ........................................................................................9 HISs and governance .................................................................................................................................11 Step 1. Performing a health information system assessment: where are we now? ........................... 13 HIS assessment approaches and existing tools ..................................................................................13 Assessment approach for this support tool .........................................................................................15 Step 2. Developing a health information system vision: where do we want to go? ............................. 24 From external assessment to country-led strategy development ....................................................24 What should the HIS achieve? ..................................................................................................................24 What are important HIS values? ..............................................................................................................26 Step 3. Developing a health information system improvement plan: how are we going to achieve our vision? ............................................................................................................................................................29 Elements needed to build the HIS improvement plan ........................................................................ 29 Guiding principles for HIS development ................................................................................................ 36 Step 4. Monitoring progress and performing regular evaluations: are we on track? .......................... 37 Developing the M&E plan: what, how, and follow-up .......................................................................... 37 Compiling the HIS strategy .......................................................................................................................42 References ...........................................................................................................................................................43 Glossary ...............................................................................................................................................................48 Annex 1. Rationale for the add-on modules .................................................................................................50 Health information for GPW13 and EPW .............................................................................................. 50 Infectious disease surveillance ............................................................................................................... 54 Noncommunicable disease monitoring ................................................................................................ 60 Human resources for health .................................................................................................................... 66 Health data governance ........................................................................................................................... 71 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENTiv Annex 2. HIS assessment item sheet ............................................................................................................ 76 Introduction ..................................................................................................................................................76 CORE Data collection .................................................................................................................................78 CORE Analysis ............................................................................................................................................ 87 CORE Health reporting ............................................................................................................................. 90 CORE Knowledge translation .................................................................................................................. 92 CORE Governance and resources .......................................................................................................... 94 CORE Summary and scoring sheet ........................................................................................................97 Add-on GPW13-EPW .............................................................................................................................. 105 Add-on Infectious disease surveillance ...............................................................................................112 Add-on HRH .............................................................................................................................................. 124 Add-on NCD monitoring ......................................................................................................................... 133 Add-on Health data governance ........................................................................................................... 148 Annex 3. Template for HIS improvement roadmap ..................................................................................156 ACKNOWLEDGEMENTS v Acknowledgements This document was developed by the Data, Metrics and Analytics Unit in the Division of Country Health Policies and Systems of the WHO Regional Office for Europe. The main author is Marieke Verschuuren. Elsa Negro Calduch is the author of the add-on module on infectious disease surveillance, Graham Willis and Cris Scotter are the authors of the add-on module on human resources for health, Enrique Loyola and Ivo Rakovac are the authors of the add-on module on noncommunicable disease monitoring, and Tobias Schulte in den Baeumen is the author of the add- on module on health data governance. David Novillo Ortiz provided direction during the document production as well as technical advice during concept drafting, writing and review. Special thanks to Natasha Azzopardi-Muscat for her strategic guidance. For further information please contact the Data, Metrics and Analytics Unit (euhiudata@who.int). We thank the following colleagues and experts for their valuable feedback during the development of this document. WHO Adrienne Cox, Stefania Davia, Karapet Davtyan, Benson Droti, Jun Gao, Clayton Hamilton, Mark Landry, Kristina Mauer-Stender, Arash Rashidian, Tyrone Sy, Chelsea Maria Taylor, Tomas Zapata. External experts Petronille Bogaert, Damian Borbolla, Neville Calleja, Anna Korotkova, Hans van Oers, Lina Sofia Palacio Mejia, Nicole Rosenkötter, Andrea Schmidt, Catherine Staes. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENTvi CRVS Civil Registration and Vital Statistics EHR electronic health record EPHO Essential Public Health Operation EPW European Programme of Work GIS geographic information system GPW13 Thirteenth General Programme of Work HIS health information system HRH human resources for health HSPA health system performance assessment ICD International Statistical Classification of Diseases ICT information and communications technology IHR International Health Regulations (2005) IT information technology M&E monitoring and evaluation NCD noncommunicable disease NGO nongovernmental organization PRISM Performance of Routine Information System Management SDG Sustainable Development Goal SWOT strengths, weaknesses, opportunities, and threats UNDP United Nations Development Programme Abbreviations ExECUTIVE SUMMARY vii Executive summary It is essential for policy-makers to have a reliable and clear picture of how health is distributed in a given population, and what indicators contribute to or reduce opportunities to be healthy. Therefore, the surveillance of population health and well-being is the first of 10 essential public health operations (EPHO) defined by WHO. WHO has a long tradition of providing support to Member States to strengthen their health information systems (HISs). One of the tools for this is the Support tool to assess health information systems and develop and strengthen health information strategies, which was published in 2015. It was developed in response to a request from the WHO European Region’s Standing Committee of the Regional Committee for a practical tool to support Member States. This document is an update of the 2015 tool. Since its publication in 2015, the original version of this tool has been applied in a dozen Member States. Overall, it appeared from these assessments that there is a great need for capacity-building to support strategic assessment and development of HISs. The SARS-CoV-2 (COVID-19) crisis has underscored the critical need for all countries to strengthen their health data and information systems and circuits. Therefore, supporting Member States in improving health information remains an important focus of WHO’s work under the thirteenth general programme of work (GPW13) and the European Programme of Work 2020–2025 (EPW), and this updated tool will be instrumental in WHO’s efforts to strengthen national HISs while responding to the need for support as explicitly communicated by Member States. The support tool comprises two main parts: firstly, it provides guidance for performing an overall assessment of the full HIS, and secondly, it provides guidance for the subsequent development of an HIS strategy. In this updated version, the assessment methodology has been updated to better reflect the current context of HISs in the WHO European Region. Most notably, the balance between data collection and actual data use was improved by placing more emphasis on data analysis, reporting and dissemination. In addition, the assessment methodology now reflects the growing importance of electronic health records (EHRs) and other digital solutions in European HISs, and add-on modules on infectious disease surveillance, noncommunicable disease (NCD) monitoring, health information for GPW13 and EPW, human resources for health (HRH) and health data governance were added. Finally, the guidance for HIS strategy development was made more concise and practical. The common mode of application of this tool is an external HIS assessment by a WHO team and a subsequent country-led process of HIS strategy development, for which WHO can provide technical support, if desired. Nevertheless, national authorities and other users of this tool may also use the guidance provided in the tool to arrange a self-assessment of the HIS. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENTviii This tool begins with a chapter that sets the scene by defining HIS and HIS governance and clarifying the role of HIS assessments and HIS strategies therein. Next, a stepwise approach is followed, with the four steps described. Step 1. Perform an HIS assessment (answering the question: where are we now?) Step 2. Develop an HIS vision (answering the question: where do we want to go?) Step 3. Develop an HIS improvement plan (answering the question: how are we going to get there?) Step 4. Monitor progress and perform regular evaluations (answering the question: are we on track?) The HIS assessment is guided by an HIS assessment item sheet, which consists of a core module and several add-on modules. The aim of the core module is to provide an overview of the functioning of the entire national HIS. The aim of the add-on modules is to shed more light on specific parts or functions of the national HIS. The core module forms the basis of the HIS assessments, and one or several add-on modules can be added to it, according to the needs and preferences of the country being assessed. The HIS assessment results in a report with a summary of the situation in the country including an HIS maturity score, an analysis of strengths, weaknesses, opportunities, and threats (SWOT), and recommendations for improvement for the short, medium, and long term. The second step in the HIS strengthening process is defining an HIS vision. This will answer the question: where do we want to go? An HIS vision defines what the HIS should ultimately achieve (HIS goals) and how this should be achieved (HIS values). Next, the HIS improvement plan is defined. This will answer the question: how are we going to achieve our vision? The HIS improvement plan builds on the outcomes of the HIS assessment and consists of several building blocks: HIS improvement priorities, HIS improvement objectives and interventions, a mapping of ongoing and planned HIS strengthening activities, and a roadmap including a budget estimate. The fourth step is monitoring progress and performing regular evaluations. This will answer the question: are we on track? The monitoring and evaluation (M&E) plan provides guidance for this step by defining what will be measured to track progress (M&E framework), how it will be measured and reported on (M&E process), and how follow-up is arranged (M&E governance). Together, the HIS vision, the HIS improvement plan and the M&E plan form the HIS strategy. INTRODUCTION 1 Introduction This document is an update of the Support tool to assess health information systems and develop and strengthen health information strategies, which the WHO Regional Office for Europe published in 2015, following a request made in December 2013 by the Standing Committee of the Regional Committee to develop a practical support tool for Member States (1). The 2015 support tool comprises two main parts: firstly, it provides guidance for performing an overall assessment of the full national HIS, and secondly, it provides guidance for the subsequent development of an HIS strategy. Since its publication, the assessment part of the support tool has been applied in several Member States (2).1 Based on the experiences gained during these assessments, the tool has now been updated. Some experiences from Member States that took part in these assessments are presented in Box 1. Box 1. Experiences from Member States with the application of the assessment part of the support tool The WHO health information assessment was carried out in Uzbekistan in 2018. It gave us a comprehensive overview of the strong and weaker points in our system and provided a good starting point for the USAID-funded multifaceted development programme for the health information system in Uzbekistan, which runs from 1 September 2018 to 31 January 2021. Within the framework of this programme, at the request of the Ministry of Health, the WHO Regional Office for Europe together with the WHO Country Office in Uzbekistan developed the training titled “Surveillance of population health: generating and using health information for policy-making”, which took place in December 2020. This training built on the outcomes of the assessment and its programme was tailored to the situation in Uzbekistan. It was attended by 54 specialists from the Ministry of Health and other institutions across the country. Therefore, for us, the assessment was the kick-off of a series of health information strengthening activities. Dr. Zulkhumor Mutalova, Head, Department of Health Statistics and Accounts, Ministry of Health, Uzbekistan 1 The referenced publication describes the experiences of the first four applications of the assessment part of the support tool. After this initial pilot, the tool was applied in nine more Member States. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT2 Box 1 contd “I think the assessment is very important for our country’s efforts to build a comprehensive integrated system for the exchange and processing of health data. Keeping in mind that the assessment used the tool designed for this purpose, I believe that the results are credible and realistically show the current situation, but also give us guidelines that we will use to further develop and improve the HIS in our country. I also think that the analysis of the situation is comprehensive in terms of the fact that health stakeholders and policy documents were fully covered. We will also use the information we have in the assessment report in developing and improving a national e-health strategy.” Zhaklina Chagoroska, Advisor and National WHO Counterpart, Ministry of Health, North Macedonia “The Assessment Tool provided us with a clear, easy and objective methodology to analyse our HIS. This instrument was also useful for us to involve in the evaluation people and departments related to health information and not included in the Ministry of Health: Department of Statistics, Civil Registry, etc. This involvement caused them to reflect on and become aware of the need to have adequate information tools and use them properly. Consequently, at the end of the evaluation process, we had a roadmap and a set of recommendations to improve and strengthen the HIS. We hope to be able to get to work as soon as the pandemic is under control and is a bad memory and a lesson learned in our country.” Josep Romagosa Massana, Health Systems Information Technician and National WHO Counterpart, Ministry of Health, Andorra; Helena Mas Santuré, Secretary of State for Health, Ministry of Health, Andorra Supporting the implementation of WHO’s European Programme of Work The HIS assessments performed on the basis of the 2015 version of this tool revealed that key challenges faced by countries include limited use of existing health information for policy-making and poorly functioning intersectoral coordination mechanisms. Furthermore, there are concerns regarding access to these non-integrated systems and data quality. Overall, these assessments indicated a great need for capacity-building to support strategic development and assessment of HISs (2).2 See Table 1 for an overview of common strengths and challenges identified across the assessed Member States. In addition, a 2012 review of public health capacities and services in the WHO European Region found that while most countries have surveillance systems and registries in place for communicable diseases, environmental hazards and basic demographic and health status data, routine surveillance of risk factors for NCDs and broader determinants (including protective factors and inequalities) is generally poorly developed across the Region (3). 2 The referenced publication describes the experiences of the first four applications of the assessment part of the support tool. After this initial pilot, the tool was applied in nine more Member States. The common findings presented here and in Table 1 are based on all the assessments carried out using the 2015 version of the tool, that is, based on the findings from 13 Member States. Most of them are located in the eastern part of the WHO European Region. INTRODUCTION 3 In addition to these findings, the COVID-19 crisis has underscored the critical need for all countries to strengthen their health data and information systems and circuits. A lack of health data standardization (such as the definitions, calculations, and formats of the data), delays in receiving data, lack of integration and interoperability between the different data and information systems, and lack of trained people to manage and use the data have been identified as the main reasons for the inability to effectively leverage the volume and different types of available data from HISs in support of the pandemic response (4). Table 1. Common strengths and challenges based on the findings of assessments carried out in 13 Member States of the WHO European Region HIS area Strengths Challenges Data collection: data availability and usability • Functioning data collection systems • Most countries have implemented a generic Unique Personal Identification Number or are planning to do so • Limited analytical capacity • Data quality issues • Lack of data from private health-care facilities • Unclear roles and responsibilities for data exchange • Limited data linkage Data collection: digital health information systems and e-health • Growing interest for and promising developments in the area of digitalization of health information systems/e-health • Most countries work with an EHR system or are planning to implement it • Large parts of data flows still paper-based • Fragmentation and lack of interoperability • The EHR system implemented only in some health-care facilities Analysis and health reporting: indicators • Commonly clearly defined indicators are published at the national level • Growing interest at health care facility level for using indicators for quality control and performance improvement • Underlying selection criteria for published indicators are not always clear • Published indicators are not always clearly related to the health policy in the country • Lack of good data and indicators for benchmarking across health-care facilities and need for capacity-building in this area Knowledge translation, governance and resources • Understanding of policy- makers and managers on the need for sound health information for decision-making • Limited use of health information for decision-making • Lack of a clear HIS strategy • Central multisectoral coordination mechanisms missing or functioning poorly • Lack of resources and a general need for capacity-building • An HIS is often understood to be the same as an EHR system • Dependency on (temporary) donor funding Thus, there is still a clear need for capacity-building in the WHO European Region aimed at strengthening HIS in support of evidence-informed policy-making. The WHO Regional Office for Europe’s EPW 2020–2025 acknowledges this, stating that there is a need for a quantum leap in the ability to generate credible, reliable and actionable information, and that classic data collection SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT4 will need to be complemented by robust use of big data, online surveys, consensus panels and expert opinion to facilitate public health monitoring and forecasting and ensure that decisions are data-driven. Therefore, the strengthening of HISs continues to be a priority area of WHO’s work to further develop credible, timely and high-quality country health data. The EPW pays particular attention to leveraging the use of digital technologies, via the Empowerment through Digital Health initiative, one of the four flagship initiatives of the EPW. Health inequalities have been a persistent challenge in the WHO European Region. Therefore, the EPW places strong emphasis on leaving no one behind. The EPW also stresses the need to reinforce the leadership capabilities of health authorities by putting a particular focus on supporting capacities for effective health leadership and engagement with other policy sectors (5). This updated WHO tool will be instrumental in WHO’s efforts to strengthen HISs as part of the EPW work and answers the need for support as explicitly expressed by Member States. The EPW builds on the global GPW13 (6), and the GPW13 Impact Measurement Framework in turn largely seeks synergy with the United Nations Sustainable Development Goals (SDGs) (7). This tool supports Member States in the implementation of the EPW, GPW13 and the SDGs by providing guidance for optimizing health information for identifying priority areas for policy action, evaluating the effects of EPW-, GPW13- and SDG-related policies and interventions, and measuring progress towards the policies’ targets. What is new in this updated version of the tool? Like the 2015 version, this new version of the support tool provides guidance for HIS strengthening through assessing the full national HIS and the subsequent development of an HIS strategy. However, the assessment methodology has been updated to better reflect the current context of HISs in the WHO European Region. The 2015 support tool was largely built on existing resources developed by the former WHO Health Metrics Network (8), and primarily aimed at low- and middle- income countries. Consequently, in the 2015 version, there was a relatively heavy emphasis on data sources, and less on the actual use of the data through analysis, reporting and dissemination. In this update, a more balanced approach is applied. In addition, the tool now reflects the growing importance of EHRs and other digital solutions in European HISs. Another adaptation made to the assessment part of the tool is the inclusion of several add-on modules, which allow for a more in-depth assessment of specific parts or functions of the HIS. Add-on modules were added for the following HIS elements: infectious disease surveillance, NCD monitoring, health information for GPW13 and EPW, HRH and health data governance. All of these represent areas that are highly relevant for WHO as well as for national-level policy-making. The importance of the module on infectious disease surveillance has only increased in the light of the COVID-19 pandemic. The 2015 support tool provided detailed step-by-step guidance for the development of an HIS strategy. In this new version, a more condensed approach is used. By simplifying the application of this part of the tool, users can gain an overview of the process of HIS strategy development more quickly and adapt the methodology to better suit specific settings or address specific questions more easily, thus supporting a more flexible use of the support tool. INTRODUCTION 5 The simplified stepwise approach applied in this update is shown in Fig. 1 and reflected in the structure of this document. Performing HIS assessments and developing and implementing HIS strategies are part of HIS governance. Therefore, before step 1 is addressed, a chapter is included to set the stage and define HIS governance and clarify the role of HIS assessments and HIS strategies therein. Next, the steps as presented in Fig. 1 are followed. Step 1. Performing an HIS assessment: where are we now? Step 2. Developing an HIS vision: where do we want to go? Step 3. Developing an HIS improvement plan: how are we going to get there? Step 4. Monitoring progress and performing regular evaluations: are we on track? Fig. 1. Stepwise structure of this support tool As explained above, this support tool provides guidance for HIS strengthening through assessing the HIS and the subsequent development of an HIS strategy. Step 1 of this tool provides guidance for performing an HIS assessment. Subsequently, steps 2, 3 and 4 are related to the development of an HIS strategy. The outputs of these steps, namely the HIS vision, the HIS improvement plan, and the M&E plan, can be combined into a comprehensive HIS strategy document. The outcomes of the evaluation in step 4 should feed back into the HIS improvement plan (see Fig. 1). This support tool is primarily developed to help strengthen national HISs. However, to a large extent, it is also relevant for subnational-level HISs, and with some minor adjustments it can be used for performing assessments of and developing strategies for such HISs as well. The common mode of application of this tool is an external HIS assessment by a WHO team and a subsequent country- led process of HIS strategy development, for which WHO can provide technical support if desired. Nevertheless, national authorities and other users of this tool may also use the guidance provided in the tool to arrange a self-assessment of the HIS. 1. Perform an HIS assessment: where are we now? 2. Develop an HIS vision: where do we want to go? 3. Develop an HIS improvement plan: how are we going to get there? 4. Monitor progress and perform regular evaluations: are we on track? SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT6 HIS: what are we actually talking about? Definition of an HIS The aim of this tool is to support Member States in strengthening their HISs. But what does that mean? There are many different definitions of HISs, and in the application of the 2015 version of this tool, a lot of confusion was revealed to exist about what exactly the term HIS means. People often equate an HIS with a central database or an EHR system (which often will include information from the International Statistical Classification of Diseases (ICD)). However, an HIS is much more than databases or EHR systems, and digital solutions for health data collection are not a goal in themselves but tools for HIS strengthening and modernization. Box 2 contains several selected definitions of HISs. Box 2. Selected definitions of HISs HISs are systems providing information support to decision-making at all levels of the health system, which incorporate information generated by both population-based and institution- based data sources (8). HISs can be defined as infrastructures for the monitoring of health activities, population health outcomes, and policies with a significant impact on health. They encompass the people, institutions, legislation, interinstitutional relationships, values, technologies, and standards that contribute to the different stages of data processing. These stages include the collection, analysis, storage, transmission, display, dissemination, and further utilization of data and information from various sources. The goal of health information systems is to allow all professional and lay users within and outside the health sector to use, interpret, and share information and to transform it into knowledge (9). An HIS refers to a system involving producers, users, and other factors contributing to the production and use of health information. Health information generated by an HIS supports evidence-informed decision-making at every level of a health system (10). The HIS provides the underpinnings for decision-making and has four key functions: (i) data generation, (ii) compilation, (iii) analysis and synthesis, and (iv) communication and use. The HIS collects data from health and other relevant sectors, analyses the data and ensures their overall quality, relevance and timeliness, and converts the data into information for health- related decision-making (11). An HIS is the total of resources, stakeholders, activities and outputs enabling evidence- informed health policy-making (12). Setting the scene: health information systems and their governance SETTING THE SCENE: HEALTH INFORMATION SYSTEMS AND THEIR GOVERNANCE 7 A common feature of these definitions is that they depict an HIS as a complex, multilevel system, aimed at producing health intelligence to inform decision-making. Such a comprehensive HIS approach also underlies this support tool. In line with the model by Verschuuren and van Oers (see Fig. 2) (13), which is based on the well-known data, information, knowledge and wisdom hierarchy (14), this support tool discerns the following HIS functions or domains: y data collection y analysis y health reporting y knowledge translation y governance and resources. Fig. 2. Population health monitoring model combining health information system outputs and activities Note: The levels in the data, information, knowledge and wisdom hierarchy, also known as the knowledge hierarchy or information pyramid, can be seen as the outputs an HIS generates by performing the activities depicted on the right side of the figure. Thus, when these HIS activities are performed, the information pyramid can be climbed to reach evidence-informed policy-making at the top. The scope of HISs and how HISs relate to digital solutions and e-health The function of the HIS goes far beyond collecting data; it starts with defining a conceptual approach, after which data are collected, analyses are performed, and knowledge is generated and actively brought into policy and practice. HISs play an important role in health system governance. This is elaborated further in the last section of this chapter. By health system we mean a health system in the broadest sense: next to primary, secondary, and tertiary health-care services, this includes public health and preventive services, as well as intersectoral action to address the wider determinants of health. This implies that a well-functioning HIS needs data not only from the health sector, but from other domains as well (see Fig. 3 (15)). All these different data sources Evidence-informed policy-making Knowledge translation Health reporting Analysis/contextualization Data collection Health information strategy & conceptual approach The ability to make well informed policy decisions Wisdom Knowledge Information Data Health information system SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT8 can either be paper-based or electronic. They can also be related to or integrated with wider digital applications, such as cloud-based governmental data repositories or e-health systems that combine an EHR function with other functions, such as planning of consultations and diagnostic tests, prescription of medicines and billing. Although it follows from this that there is an overlap – and options to create synergies – between this HIS tool and tools for e-health and information and communications technology (ICT) assessments and strategies, the main focus of this tool in terms of digital applications is on their health data collection functions and to what extent the thus- collected data can be and are being used for decision-making. This does not mean that this tool disregards the wider context in which digital solutions are operating, only that this is not a topic of assessment per se. Similarly, the tool does not focus on technical software specifications of digital information systems, but on their usage and usability in practice. Fig. 3. The different data sources that constitute health information The HIS supports decision-making at various levels HISs are aimed at supporting evidence-informed policy or decision-making. Here we can discern different levels of decision-making. This includes national or subnational strategic policy-making and planning by health-care authorities, as well as the use of health information at health-care facility or provider level, for example in benchmarking or performance improvement, quality control and resource planning. This implies that data derived from the health-care delivery process can and should be used for informing both health-care practice and health policy. This is reflected in the concept of the learning health care system, see Fig. 4 (16). This figure illustrates that it should be possible to use data that were originally collected for individual patient care also for monitoring, research, and ultimately policy-making. This is also called the secondary use of health data, or the use of health data for secondary purposes. Therefore, regulatory frameworks that facilitate safe use of health data for secondary purposes are vital for a well-functioning HIS. Data sources: from other domains Police data, e.g. on injuries, violence Policy-making and planning Environmental data Data from the social domain Etc. Data sources: other sources on health Vital statistics, causes of death, demographics Disease registries (e.g. cancer, diabetes) Health interview and examination surveys Etc. Data sources: health care Clinical health records Administrative health-care records Patient-reported information Etc. Health information SETTING THE SCENE: HEALTH INFORMATION SYSTEMS AND THEIR GOVERNANCE 9 Fig. 4. The learning health-care system concept: using health-care-generated data for improving health services and informing health policy HISs and related concepts: population health surveillance and monitoring, M&E, and health system performance assessment It is essential for policy-makers to have a reliable and clear picture of how health is distributed in a given population, and what indicators contribute to or reduce opportunities to be healthy. Therefore, surveillance of population health and well-being, aimed at feeding information and intelligence to health needs assessments, health impact assessments and to planning for health services, is the first of 10 EPHOS defined by WHO (17).3 Instead of population health surveillance, the term “population health monitoring” is also used to describe the same public health function. Population health surveillance or monitoring can be defined as regular and institutionalized production and dissemination of information and knowledge about the health status of a population and its determinants, aimed at informing policy-making (13). HIS and population health surveillance are closely related. An HIS may be considered the necessary infrastructure for carrying out all the steps necessary to climb the information pyramid in a regular and timely manner, to reach evidence-informed policy-making at the top (see Fig. 2) (13). HISs and population health surveillance overlap with the concept of M&E. M&E refers to tracking performance of specific interventions, programmes, or policies. M&E activities also provide guidance on future intervention activities and are an important part of accountability to funding agencies and stakeholders (18). Often, specific health programmes (such as programmes for tuberculosis or HIV/AIDS) generate their own data to track performance. These data should be seen as part of the HIS, as they can also be used for secondary purposes such as general population health surveillance or research. Data from regular HIS data sources, such as mortality data from civil registration and vital statistics (CRVS) registries or health-care utilization data derived from facilities, can also be used for M&E frameworks for specific programmes or policies. 3 Different definitions of surveillance exist, and people often associate it with infectious disease surveillance. However, the concept can also be used in a broader sense, as illustrated by WHO’s EPHO1. In this support tool, when we speak of surveillance, we refer to this broad definition. Health policy Health services Research Practice feedbackKnowledge Data SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT10 Another concept related to population health surveillance is health system performance assessment (HSPA). HSPA can be defined as an assessment of a health system as a whole, using a limited number of indicators linking outcomes with functions or strategies. It is country-specific, embedded in a national or subnational policy process, and linked to national health plans or strategies where possible (19). HSPA could be deemed an element of population health surveillance, but population health surveillance is more comprehensive than just HSPA. Population health is influenced by the health-care system and its functioning, but also by many other determinants of health, such as lifestyle-related aspects and environmental and social factors (see Fig. 5) (20). The latter are also referred to as the wider determinants of health. Population health surveillance aims to understand all these health determinants and their effects on population health. Fig. 5. The main determinants of health Age, sex and constitutional factors Ge ne ral so cio -ec onom ic, cultural and environmental conditions So cia l an d community networks Ind ivid ual lifestyle factors Living and working conditions Work environmental Unemployment Water and sanitation Health-care services Education Agriculture and food production Housing In line with the above, in this support tool, generating information for HSPA is regarded as one of the functions of the HIS, next to generating information on other health determinants and health outcomes. This implies that an HIS is much broader than a health-care information system, and that even though administrative and clinical health-care data are important sources for HISs, many other data sources are needed to derive a comprehensive picture of population health. These include CRVS, disease registries, health interview and examination surveys, data from preventive programmes, and sources from outside the health domain, such as police data on violence and injuries, environmental data, and data from the social domain (for example, on unemployment or disability; see Fig. 3). Source: Institute for Future Studies, 1991 (20). SETTING THE SCENE: HEALTH INFORMATION SYSTEMS AND THEIR GOVERNANCE 11 HISs and governance Governance can be defined as the process of decision-making and the process by which decisions are implemented (21). Governance and accountability are closely related. The essence of accountability is answerability; being accountable means having the obligation to answer questions regarding decisions and/or actions (22). HISs play an important role in the governance and accountability of health systems. WHO defines five broad actions or functions for health system governance, see Table 2 (23). One of these is “generating intelligence: information and analysis for decision-making”. This function is performed by the HIS. Consequently, HISs are an intrinsic part of health system governance and contribute to their accountability. However, HISs themselves also are subject to accountability, and those responsible for the HIS should be able and willing to explain the decision-making processes on the scope of health information collected and the spending of – often public – HIS funding. Although guidance on good governance is available for specific elements of the HIS, such as data governance or related aspects, such as multisectoral governance, not much documentation is available that specifically addresses HIS governance as such. However, the above-mentioned five functions of health system governance can also be used as a framework for defining the main functions that an HIS governance mechanism should cover. Table 2 illustrates how these functions translate to an HIS context and how this links with this support tool. Thus, performing a multistakeholder HIS assessment and subsequently developing an HIS strategy supports transparent and evidence- informed HIS decision-making and contributes to good HIS governance, which in turn contributes to good health system governance. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT12 Table 2. Five functions of health system and HIS governance Five functions of health system governancea Translation into five functions of HIS governance and link with this support tool Formulating policy and strategic plans Formulating HIS policy and strategic plans This support tool provides guidance for the development of an HIS strategy Generating intelligence: information and analysis for decision-making Generating intelligence: information and analysis for decision-making on further development of the HIS This support tool provides guidance for the development of an evaluation mechanism to keep track of progress towards the goals set in the HIS strategy Putting in place levers or tools for implementing policy – including design of health system organizational structures and their roles, powers and responsibilities; design of regulation; standard-setting; incentives; enforcement and sanctions Putting in place tools for implementing the HIS strategy This support tool provides guidance for the development of an HIS strategy and tools for its implementation, most importantly, in the form of an HIS improvement roadmap, which specifies which stakeholder needs to do what and when. The M&E plan also stimulates the implementation of the HIS strategy (see below) Collaboration and coalition-building across sectors and with external partners Collaboration and coalition-building across HIS stakeholders and across sectors This support tool emphasizes the need for broad stakeholder involvement and collaboration as a prerequisite for successful HIS strengthening Ensuring accountability by putting in place: governance structures, rules and processes for health sector organizations; mechanisms for independent oversight, monitoring, review and audit; transparent availability and publication of policies, regulations, plans, reports, accounts, etc; and openness to scrutiny by political representatives and civil society Ensuring accountability by putting in place governance mechanisms for HIS reform This support tool provides guidance for the development of an HIS strategy including accountability mechanisms, most importantly, a comprehensive M&E plan, including an M&E framework (milestones and indicators), M&E process and M&E governance mechanisms. The importance of transparency is stressed throughout the tool aWHO, 2014 (23). STEP 1. PERFORMING A HEALTH INFORMATION SYSTEM ASSESSMENT: WHERE ARE WE NOW? 13 Step 1. Performing a health information system assessment: where are we now? The first step in the HIS strengthening process is performing an HIS assessment. This will answer the question: where are we now? The HIS assessment methodology applied in this support tool allows for rapid yet comprehensive mapping of the main HIS strengths, weaknesses, opportunities, and threats, and results in a maturity score for each of the main HIS domains (data collection; analysis; health reporting; knowledge translation; and governance and resources). Performing an HIS assessment is the first step of the HIS strengthening process described in this tool. The objectives of HIS assessments are to identify priority areas for HIS strengthening, mobilize technical and financial support for the HIS strengthening process, and serve as a baseline for monitoring progress. Another objective of HIS assessments is to enhance knowledge about the HIS and inform HIS stakeholders about aspects of the HIS with which they are less familiar (24). Often the HIS assessment process will also result in different HIS stakeholders getting to know each other and getting more insight into each other’s points of view. This will contribute to enhanced national networks and improved collective knowledge about the HIS and its functioning, which in turn will contribute to a successful implementation of the HIS strategy. Therefore, broad stakeholder involvement is an important prerequisite for a successful HIS strengthening process. The box on page 17 (In focus: ensuring broad stakeholder involvement) explains how to achieve this. HIS assessment approaches and existing tools Different HIS assessment approaches exist, each with their own advantages and disadvantages. Which approach is best will depend on the specific circumstances of the assessment, and will be influenced, among other things, by the goal of the exercise, the extent to which previous assessments can be built on, and the available time and resources. Although there are many different variants, HIS approaches can roughly be categorized according to two categories: comprehensive versus partial, and self-evaluation versus external evaluation. In a comprehensive assessment approach, the entire HIS is covered, while in a partial approach only part of the HIS is assessed; for example, HIS data quality or the M&E component of specific health programmes such as for AIDS or NCDs. Assessment approaches based on self-evaluation are carried out by people working in or familiar with the HIS, while external evaluations are carried out by people who do not have direct links to the HIS. External assessors can either be formal assessors; for example, WHO staff who are performing an assessment at the request of the ministry of health, or peer assessors. In the latter case, health information experts from one country assess the HIS of another country in a more informal setting. Such a peer assessment approach was applied for example in the EU-funded Joint Action InfAct (25). Table 3 lists the main advantages and disadvantages of these different SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT14 HIS assessment approaches adapted from the WHO indicators handbook (26), as well as some examples of existing tools. Table 3. Different HIS assessment approaches: advantages and disadvantages and examples of existing tools Assessment approach Advantages Disadvantages Examples Comprehensive versus partial Comprehensive • Results in an overview of the entire HIS, allowing for optimally identifying options for synergies across stakeholders and domains • Resource-intensive • May result in a long list of action points, which can be overwhelming and may complicate priority-setting and concrete follow-up • WHO and Measure Evaluation. Routine Health Information System Rapid Assessment Tool (27) • WHO SCORE Assessment Instrument (28) Partial • Less resource-intensive • More focused outcomes, which will be beneficial for setting priorities for follow-up • Gives insight into only part of the HIS, which means that potentially more pressing problems stay out of sight, and fewer options for synergies across stakeholders and domains can be identified • UNAIDS. 12 Components Monitoring and Evaluation System Strengthening Tool (29) • WHO. Data quality review, modules 1–3 (30–32) Self-assessment versus external assessor Self-evaluation • Creates support and ownership for the process and outcomes of the assessment • Less likely that issues will be overlooked, as assessment is carried out by people who know the HIS well • Assessors may not be able to judge the HIS objectively • Often time-consuming and complex to implement • Less likely to generate results that can be compared over time or between countries • PAHO/WHO. Information Systems for Health. Standard Assessment Method (33) • MEASURE Evaluation. Demand and Readiness Tool for Assessing Data Sources in Health Information Systems (HIS DART) (34) External evaluation • More objective assessment approach. • Often based on existing sources, such as international databases, to minimize reporting burden on countries, and therefore easier to implement • Enables comparisons between countries and over time • The assessment process and outcomes may not be accepted by the HIS stakeholders • It may prove difficult to retrieve all relevant documentation because assessors do not know all the ins and outs of the HIS • WHO. Health Information Systems Performance Index (HISPIX) (26) • Joint Action on Health Information InfAct. Peer HIS assessment approach (35) STEP 1. PERFORMING A HEALTH INFORMATION SYSTEM ASSESSMENT: WHERE ARE WE NOW? 15 WHO has a long tradition of supporting its Member States in assessing and strengthening their health information systems, and as such, there is a comprehensive body of WHO tools and guidance documents, both with global and regional perspectives, some of which are listed in Table 3. However, other organizations also have extensive experience in providing support to countries for HIS assessment and improvement. One of these is MEASURE Evaluation, which is funded by the United States Agency for International Development. MEASURE Evaluation is dedicated to improving health through improving data and has developed a wide array of tools for this purpose, such as the online HIS Strengthening Resource Center, which includes a database of health information system assessment tools (36). Assessment approach for this support tool General aspects This support tool to strengthen HIS applies a comprehensive assessment approach, evaluating the entire HIS. Setting the scene: health information systems and their governance and the HIS assessment item sheet, below, have more information about the scope of the tool. Although this tool can also be used for a self-evaluation of the HIS, the common approach applied when using the tool is an external evaluation, carried out by a WHO team at the request of the ministry of health. See Box 3 for an overview of the different steps in the assessment approach commonly used by WHO in the application of this tool. As illustrated by Box 3, the approach used for this tool is a mix of the different HIS assessment approaches described in the previous paragraph, combining their advantages. The assessment is carried out by an external team to achieve an objective evaluation. However, the assessment is not solely based on public sources, but, to a large extent, on input from HIS experts, which is gathered through interviews. This ensures that no important issues are overlooked, while creating support for the assessment process and outcomes. The multistakeholder meeting at the end of the country visit is also an important means of creating support. The box on page 17 (In focus: ensuring broad stakeholder involvement) provides guidance on how to ensure broad stakeholder involvement representing the main HIS perspectives. Strong involvement of the ministry of health is essential for ensuring commitment for follow-up, including making available funds and other resources that are necessary for HIS strengthening activities, and institutionalization of the HIS improvement process. The assessment approach tries to find the right balance between speed and efficiency on the one hand and comprehensiveness on the other. The assessment results in an overview of the entire HIS and its functioning, and identifies elements in the system that are currently functioning suboptimally. The outcomes of the assessment support the national authorities in setting priorities for the improvement of the HIS and pinpointing specific areas that require further developmental work and capacity-building. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT16 Box 3. Steps in the assessment approach commonly used when applying this tool y The ministry of health asks WHO to carry out an HIS assessment. y A WHO team performs a preparatory desk review based on public sources to get a basic understanding of the organization and general functioning of the HIS. y The WHO team together with the Country Office and/or national counterpart prepare the agenda for a country visit of 4–5 days. y During the country visit, the WHO team holds semi-structured interviews with the main HIS stakeholders in the country. These interviews are guided by and structured according to a standard HIS assessment item sheet. y The country visit concludes with a multistakeholder meeting in which the main findings of the assessment are presented and discussed. y After the country visit, the WHO team writes a summary report for the ministry of health consisting of: 1. a summary of the HIS assessment item sheet, including a scoring of the maturity level of the HIS; 2. a SWOT analysis; 3. concrete recommendations for further improvement of the HIS. y After the report is submitted, concrete follow-up actions are agreed (for example, organizing a capacity-building event or performing a more detailed follow-up assessment of specific parts of the HIS) and, if appropriate, included in the biennial collaborative agreement between WHO and the Member State concerned. STEP 1. PERFORMING A HEALTH INFORMATION SYSTEM ASSESSMENT: WHERE ARE WE NOW? 17 In focus: ensuring broad stakeholder involvement In every assessment approach, involving the right HIS stakeholders is crucial for the success of the exercise. What do we mean by “stakeholders” and which ones should be involved? A stakeholder is anybody who can affect or is affected by the assessment. They can be internal or external (37). MEASURE Evaluation defines an HIS stakeholder as anyone who has the power to influence the implementation of the follow-up of the assessment (38). Stakeholders can be individuals, groups, or organizations. When considering whether all the appropriate HIS stakeholders have been included, it is helpful to apply different HIS perspectives: y Previous assessments have shown that while a lot of health information is being produced, it often does not fully meet the needs of the users. Therefore, it is important to include both producers and users of health information. Producers are those stakeholders who collect, analyse, and/or disseminate health information or provide support for it. They may include statistical offices, national and subnational public health authorities, health care insurance companies, health-care facilities, and academia, as well as operators and suppliers of digital information systems such as EHR systems. Users of health information may include policy-makers, health-care facility managers, umbrella organizations for health-care professionals, patient organizations, and nongovernmental organizations (NGOs). y Previous assessments have shown that there are often considerable gaps between the perceptions of developers and end-users of digital information systems such as EHR systems and electronic disease registries. End-users are the health-care staff (doctors, nurses, administrators) and sometimes patients who need to feed the information into these digital systems. While the developers may be satisfied with the systems from a technical point of view, health-care staff in practice often feel that the functionality and user-friendliness of the systems are insufficient, and that the systems do not meet their needs. In fact, digital information systems are often regarded as a burden complicating the delivery of health services rather than supporting it. Therefore, it is recommended that both developers and end-users should be involved in the assessment. This will improve understanding of the barriers experienced by end-users in the implementation of digital information systems such as EHR systems. y A common finding in previous assessments is that data from the private sector are lacking. However, these data are essential to obtaining a complete picture. Therefore, involving both the public and private health-care sector will help identify the underlying reasons for missing or incomplete data and insufficient data exchanges between the sectors. HIS assessment item sheet Part of this support tool is an HIS assessment item sheet. The aim of this sheet is to guide the semi-structured interviews that take place during the country visit (see Box 3) and summarize the input gathered during these interviews. Core module and add-on modules The assessment sheet consists of a core module and several add-on modules. The core module consists of five domains. This is explained further in the next section. The aim of the core module is to obtain a generic overview of the functioning of the entire national HIS. The aim of the add-on modules is to look in more detail at specific parts or functions of the national HIS. See Fig. 6 for an overview of the structure of the HIS assessment item sheet. At the time of writing, the assessment sheet contains add-on modules on health information for WHO’s GPW13 and EPW, human SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT18 resources for health, infectious disease surveillance, NCD monitoring and health data governance. Additional topics may be included in the future. The add-on modules allow for a more flexible use of the tool, allowing Member States to zoom in on specific parts of the HIS that are of particular importance to them. However, it is emphasized that the main module assessing the entire HIS remains the core of the assessment approach. More detailed analyses of specific parts of the HIS can be added to it, but they should not replace the core module. The common assessment approach for this tool is described under General aspects and Box 3 relates to the application of the core module. If add-on modules are also applied, the duration and agenda of the country visit will need to be amended accordingly, and staff with specific expertise may need to be added to the assessment team. Descriptions of the rationale for the different add-on modules can be found in Annex 1, covering issues such as how the topic of the add-on module fits into the wider HIS, the policy relevance of the module, and how the assessment items were defined and selected. The next paragraphs describe the domain structure of the core module and the approach applied in the core module for guiding the interviewers during the assessment. Although the domain structure of the add- on modules may be somewhat different to accommodate the specificities of the HIS function covered, the add-on modules in principle apply the same approach for guiding the interviewers as the core module. Fig. 6. Structure of the HIS assessment item sheet Part 1: gathering the information Part 2: summarizing the information Assessment items in the form of overarching and probing questions to support the assessors during the interviews, covering the following modules and domains: Summary and scoring template, describing what a fully matured HIS looks like for the following modules and (sub)domains: Core module Core module If applicable: add-on modules Add on modules 1. Data collection 1a. Data collection: data sources 2. Analysis 3. Health reporting 4. Knowledge translation 5. Governance and resources 1b. Data collection: data infrastructure, management and resources Health information for GPW13 and EPW Noncommunicable disease monitoring Infectious disease surveillance Health data governance Human resources for health 4. Knowledge translation 2. Analysis 5. Governance and resources 3. Health reporting STEP 1. PERFORMING A HEALTH INFORMATION SYSTEM ASSESSMENT: WHERE ARE WE NOW? 19 Core module: structure The core module comprises five domains: data collection; analysis; health reporting; knowledge translation; and governance and resources (see Fig. 6). Data collection covers available data collections, efficiency of data flows, and the quality and usability of existing data collections. This also includes the availability, usability, and interoperability of digital data collection systems. Analysis covers the availability, comprehensiveness and use of indicator sets, and health reporting looks at the availability, comprehensiveness and use of health reports for policy-making and planning. Knowledge translation assesses the extent to which stakeholders are familiar with available health information and knowledge products, and which knowledge translation tools and mechanisms are being used. The final domain, governance and resources, looks at HIS governance mechanisms and general HIS resources, which include the legal framework, financial resources and ICT infrastructure. (Note: human resources are addressed as part of the other four dimensions.) Each domain consists of assessment items that are phrased as general or main questions. Main questions are subdivided into probing questions and for each probing question, a description is given of what the situation would be like in a fully matured HIS, or the ideal scenario. This part of the assessment tool is meant to support the assessors in the interviews they have with the HIS stakeholders. It will help them with structuring the interviews and tailoring them according to the specific expertise of the interviewee, and ensuring that no important items are missed. See Table 4 for an excerpt from the HIS assessment item sheet to illustrate the approach of the sheet. See Annex 2 for the full HIS assessment item sheet. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT20 Table 4. Excerpt from one of the domains of the first part of the HIS assessment item sheet, showing how an assessment item is phrased as one overarching and several probing questions Item ID Main question Probing questions Expectations Analysis_1 Is a core set of health indicators defined in the country? 1) Is the core set linked to a specific health policy (process) and/or to specific health goals or targets? (1) The core set is linked to a specific health policy (process) and/or to specific health goals or targets 2) How were core indicators selected? (2) National minimum core indicators have been transparently identified for national and subnational levels. The selection of indicators is also informed by international indicator sets 3) Which categories does the indicator set cover? (3) Indicators cover all categories of health indicators (e.g. determinants of health; health system inputs, outputs and outcomes (health systems performance assessment); health status; health inequalities) (Examples: Joint Monitoring Framework, SDG, NCD). If possible it includes also relevant indicators from other policy sectors (e.g. social affairs, education) 4) How is the indicator defined and calculated? (4) An indicator definition exists and the method for its calculation is documented. If applicable: the numerator and denominator of the indicator are clearly defined 5) Are metadata available and harmonized within the country and across countries? (5) Metadata that are regularly updated and exist for each indicator and are publicly available. Metadata include the headings: definition, calculation/ method, available dimensions/ subgroups (e.g. age, gender, geographical information, nationality, migration, social status (education, employment status, income)), rationale, data sources 6) If individual data on social status are not available: is a geographically based deprivation index (or similar) available? (6) Alternatively, for social status, a deprivation index is available to perform comparisons on ecological level STEP 1. PERFORMING A HEALTH INFORMATION SYSTEM ASSESSMENT: WHERE ARE WE NOW? 21 Core module: summary and scoring template The second part of the assessment item sheet is a template for summarizing the findings and scoring the maturity level of the HIS. This summary and scoring template will form the basis of the summary report, and will be complemented by the SWOT analysis and recommendations for improvement (see Box 3). The box on page 23 (In focus: conducting a SWOT analysis) provides tips for performing a SWOT analysis. The summary template comprises a block for each of the HIS assessment item sheet domains: analysis, health reporting, knowledge translation, and governance and resources, and two blocks for the more elaborate data collection domain (see Fig. 6). Each of these subdomains is divided into 4–6 themes, accompanied by a description of what the situation would be like in a fully matured HIS. These descriptions are at a higher level of aggregation than the descriptions provided for each of the probing questions in the first part of the HIS assessment item sheet (see Table 4 for examples of the latter). In the template, a description of what the situation is like in the country can be added for each of the themes, as well as a score (see Table 5). The following scoring system is applied: y situation in the country comparable to full maturity = 4 y many elements of the HIS are fully matured but some work is still needed = 3 y some elements of the HIS are fully matured but substantial work is still needed = 2 y situation in the country is still very deviant from full maturity = 1. This structured summary approach, including a quantitative score, enables the comparison on HIS assessments over time and between countries. However, it is noted that when the HIS assessment is carried out at the request of the ministry of health, it is up to the ministry to decide whether or not the summary report of the assessment will be made public. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT22 Table 5. Part of the summary and scoring template Data collection: data sources Description of situation in HIS with full maturity Description of current situation in the country Country score (maximum of 24) 1 Vital statistics: registration of births and deaths and associated medical information is complete and up-to-date. Quality of cause-of-death information is high and coding is done in line with international standards and classifications 2 Health service records: a centralized electronic health record system is in place. Tailored aggregated data sets for secondary purposes can be easily extracted. Coverage and quality of the data collected in the EHR system is high. International classifications for coding diagnoses and interventions are integrated. Health insurance data have high coverage and quality and include ICD-10 codes 3 Disease registries: there is a national-level population-based cancer registry operating according to international standards. Registries for other major chronic diseases are in place; if not, robust morbidity estimates from other sources are available. An electronic surveillance system for infectious diseases is in place with real-time data. Information on notifiable diseases according to country-specific legislation and international obligations (International Health Regulations) is available 4 Health surveys: there is a long-term operational plan for regular conduction of national health interview and health examination surveys. The methodology applied is in accordance with international standards and requirements. Specific effort is taken to make sure that hard-to-reach groups are adequately repesented. Health and statistical authorities work together on survey design, implementation and data analysis and dissemination 5 Health-care resources: there is a national human resources database with complete coverage, including the annual number of graduates. There is a national database of public and private sector health facilities with complete coverage. Each health-care provider and facility has been assigned a unique identifier code 6 Health expenditure data: financial records are available on general government expenditure on health and its components. Expenditure data are being collected in accordance with the System of Health Accounts methodology Total score STEP 1. PERFORMING A HEALTH INFORMATION SYSTEM ASSESSMENT: WHERE ARE WE NOW? 23 In focus: conducting a SWOT analysis The completed summary and scoring template forms the basis for the assessment summary report. It is complemented by a SWOT analysis and recommendations for improvement. What is a SWOT analysis? The SWOT analysis is a simple yet powerful tool for informing strategic plans, examining strengths, weaknesses, opportunities and threats within the plans. When performing a SWOT analysis, it is important to realize that strengths and weaknesses are internal HIS factors, and opportunities and threats are developments that are external to the HIS. As such, generally speaking, HIS stakeholders are able to directly influence the strengths and weaknesses, but not the opportunities and threats. However, HIS stakeholders can take action to ensure that the benefits of opportunities are reaped, and the risks of threats are mitigated. Therefore, it is important to look not only at internal HIS factors, but also at external developments. See Table 6 for a hypothetical example of a SWOT analysis of an HIS. Table 6. Example of a SWOT analysis of an HIS Strengths • Complete vital statistics and high quality of causes-of-death statistics • Centralized EHR system in place • Core set of indicators for the national health strategy has been defined and regularly reported on Weaknesses • Limited interoperability between databases • No institutionalized mechanisms for using health information for policy-making and planning • Lack of data from private health-care providers Opportunities • New five-year national health strategy under development • Growing demand from societal organizations for health information • Major investment from donor organization in ICT infrastructure for health-care facilities Threats • Political instability • Economic recession resulting in austerity measures and diminished public spending • Higher salaries in other (private) sectors and/or abroad SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT24 The second step in the HIS strengthening process is defining an HIS vision. This will answer the question: where do we want to go? An HIS vision defines what the HIS should ultimately achieve (HIS goals) and how this should be achieved (HIS values). The HIS vision is the first element of the HIS strategy. From external assessment to country-led strategy development The previous chapter covered the first step in the HIS strengthening process, which is carrying out an HIS assessment. This gave us an overview of the current performance of the HIS: where are we now? The next step is to define where we want to go: what do we want the HIS to achieve, how do we want it to perform better than it is currently doing? Defining such an HIS vision is the first step in the development of an HIS strategy. As explained in the previous chapter, the common modus for application of the first part of this tool is an external assessment carried out by a WHO team at the request of the ministry of health. It is recommended that the ministry of health (or another ministry or entity with HIS authority) take the lead in the development of the HIS strategy, as leadership and ownership are important prerequisites for effective HIS development. As for the HIS assessment, broad stakeholder involvement is also crucial in the formulation of the HIS vision and subsequent steps of HIS strategy development (see Guiding principles for HIS development). It is recommended that the HIS vision define both what the HIS should achieve and which HIS values should be respected while working towards achieving the set goals. These two components of the HIS vision are explained below. The box on page 28 (In focus: writing a good vision statement) offers tips on how to write a good vision statement. What should the HIS achieve? The HIS vision defines what the HIS should ultimately achieve. It provides guidance for priority- setting for HIS reform activities and resource allocation by fixing a point on the horizon. When developing an HIS vision statement, it is important to ensure that it is focused on outcome rather than output. What does it mean? One could simply say that the aim of an HIS is to produce health information. However, this approach would ignore the reasons for which this health information is being produced, and thus be of limited use for genuinely improving HIS performance. Health information is the output of an HIS, but producing health information is not the end goal of an HIS; health information is a means to reach certain outcomes or goals, for example supporting the implementation and evaluation of the national health strategy, or enabling evidence-informed resource planning by health-care facility managers. Step 2. Developing a health information system vision: where do we want to go? STEP 2. DEVELOPING A HEALTH INFORMATION SYSTEM VISION: WHERE DO WE WANT TO GO? 25 How can the outcomes or goals of the HIS be defined? Generally speaking, the purpose of the HIS is to generate information to enable decision-makers at all levels of the health system to identify problems and needs, make evidence-based decisions on health policy and optimally allocate scarce resources (11). To make this more concrete for the purpose of defining an HIS vision statement, it can be useful to look at the attributes of a well-functioning HIS. WHO in 2007 defined this as follows (53): A well-functioning health information system is one that ensures the production, analysis, dissemination and use of reliable and timely health information by decision- makers at different levels of the health system, both on a regular basis and in emergencies. It involves three domains of health information: on health determinants; on health systems performance; and on health status. From this definition, different attributes for a well-functioning HIS can be discerned for use in operationalizing the goals of an HIS. First, an HIS should be able to support different types of decision-making processes: the definition above discerns regular, longer-term health policy- making and planning processes and more acute detection and management of events that threaten public health. Second, an HIS should inform decision-making at various levels in the health system, which means that it should serve multiple users. These can be policy-makers, planners, managers, health-care providers, communities, and individuals (26). A third perspective that can be used for operationalizing the expected outcomes of the HIS is to look at the various domains that the HIS should cover. The definition above distinguishes health determinants, health systems performance and health status. The following is a slightly more detailed description the different kinds of information needed by health planners and decision-makers: y health determinants (socioeconomic, environmental, behavioural and genetic factors, and the contextual environments within which the health system operates); y inputs to the health system and related processes (policy and organization, health infrastructure, facilities and equipment, costs, human and financial resources and health information systems); y the performance or outputs of the health system (availability, accessibility, quality and use of health information and services, responsiveness of the system to user needs, and financial risk protection); y health outcomes (mortality, morbidity, disease outbreaks, health status, disability and well- being); and y health inequities (determinants, coverage of use of services, and health outcomes, and including key stratifiers such as sex, socioeconomic status, ethnic group and geographical location) (26). Thus, the following three questions can be a useful framework for defining what the HIS should ultimately achieve. 1. What types of decision-making is the HIS supposed to support? 2. What users is the HIS aiming to serve? 3. What domains should be covered by the HIS? SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT26 What are important HIS values? Vision statements often also include values alongside concrete outcomes or goals. Such values describe the norms according to which we want the HIS to function: we do not only want the HIS to simply achieve its goals, but we also have certain expectations about what basic points of departure should be taken into account while working towards them. Here, we might look at the principles of good governance for inspiration. A well-known set of characteristics of good governance is the one developed by the United Nations Development Programme (UNDP). The Institute on Governance in Canada clustered the UNDP characteristics into five principles (see Box 4) (39). Although not all of these principles are equally applicable to an HIS context, some important HIS values can be deduced from them, namely: y inclusiveness y responsiveness y efficiency y transparency y lawfulness y equity and equitability. Many different stakeholders are involved in an HIS, each with their own roles, tasks, and interests. Therefore, broad and active stakeholder involvement is crucial for a well-functioning HIS, and HIS operations should be based on an inclusive, participatory approach. We also want HISs to be responsive and able to respond to the sometimes changing needs of its end-users. HISs should be efficient, which means that existing data and information should be used optimally and overlaps in data collections and reporting activities should be prevented. Efficient HIS operations also relate to being able to pursue long-term goals and resist thinking along the lines of political issues-of-the-day, thus enabling sustainable monitoring activities and preservation of knowledge and expertise. We also want HIS activities and HIS governance to be transparent. According to the UNDP principles, transparency requires a free flow of information (see Box 4). In an HIS context, two types of information can be discerned: process information (for example, the HIS strategy, HIS strategy evaluation reports, meeting notes of HIS governing bodies) and actual health information itself. Both types of information should be made publicly available whenever possible, even though some restrictions related to privacy and data protection legislation may need to be taken into account when publishing health data. This brings us to the next HIS value: lawfulness. Processing health data and information is subject to relevant international and national legislation, most importantly related to privacy and data protection. When operating an HIS, compliance with prevailing legislation should be ensured. Lastly, we want HISs to be equitable. Health information inequalities exist both within and between countries. Overcoming them and ensuring that there is also adequate information available on vulnerable and hard-to-reach populations should be high on the HIS agenda (12). STEP 2. DEVELOPING A HEALTH INFORMATION SYSTEM VISION: WHERE DO WE WANT TO GO? 27 Box 4. The five good governance principles based on the UNDP principles 1. Legitimacy and voice Participation – all men and women should have a voice in decision-making, either directly or through legitimate intermediate institutions that represent their intention. Such broad participation is built on freedom of association and speech, as well as capacities to participate constructively. Consensus orientation – good governance mediates differing interests to reach a broad consensus on what is in the best interest of the group and, where possible, on policies and procedures. 2. Direction Strategic vision – leaders and the public have a broad and long-term perspective on good governance and human development, along with a sense of what is needed for such development. There is also an understanding of the historical, cultural and social complexities in which that perspective is grounded. 3. Performance Responsiveness – institutions and processes try to serve all stakeholders. Effectiveness and efficiency – processes and institutions produce results that meet needs while making the best use of resources. 4. Accountability Accountability – decision-makers in government, the private sector and civil society organizations are accountable to the public, as well as to institutional stakeholders. This accountability varies depending on the organization and whether the decision is internal or external. Transparency – transparency is built on the free flow of information. Processes, institutions and information are directly accessible to those concerned with them, and enough information is provided to understand and monitor them. 5. Fairness Equity – all men and women have opportunities to improve or maintain their well-being. Rule of law – legal frameworks should be fair and enforced impartially, particularly the laws on human rights. Source: Graham et al., 2003 (39). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT28 In focus: writing a good vision statement Many tips and tricks for writing a good vision statement can be found online. They mainly focus on businesses, but to a large extent the recommendations are relevant for other contexts as well. Although there is no golden standard for writing a good vision statement, many compelling visions share these characteristics: y concise y clear y future-oriented y stable y challenging y abstract y desirable or able to inspire. This highlights the importance of striving for a vision that: y is brief (so that it can be remembered and repeated easily); y contains a prime goal to be achieved; y can encompass all organizational interests; y is not a one-time, specific goal that can be achieved and then discarded; y provides a source of motivation for employees to do their best by including a degree of difficulty or stretch (for example, to achieve national or international status); y offers a long-term perspective for the organization and indicates the future environment in which it will function; y is unlikely to be affected by market or technology changes; and y is viewed as desirable by employees (40–42). Guidance for a good business or company vision statement recommends that the statement should be short, almost catchphrase-like. Because it may be difficult to capture the essence of what the HIS should achieve with a sufficient level of specificity in one or two sentences;, more sentences can be used if necessary to define the HIS vision. However, it remains important that the vision statement must be easy to read and understand and must be formulated as concisely as possible. STEP 3. DEVELOPING A HEALTH INFORMATION SYSTEM IMPROVEMENT PLAN: HOW ARE WE GOING TO ACHIEVE OUR VISION? 29 Step 3. Developing a health information system improvement plan: how are we going to achieve our vision? The third step in the HIS strengthening process is developing an HIS improvement plan. This will answer the question: how are we going to achieve our vision? The HIS improvement plan consists of several building blocks: HIS improvement priorities, HIS improvement objectives and interventions, a mapping of ongoing and planned HIS strengthening activities, and a roadmap including a budget estimate. The HIS improvement plan is the second element of the HIS strategy. Elements needed to build the HIS improvement plan The HIS improvement plan is the linking pin between the outcomes of the HIS assessment, which has resulted in an overview of strengths and weaknesses (that is, the problems that need to be tackled), and the HIS vision, which defines what the HIS should ultimately achieve, and lays out the plan for getting from A to B. In the development process, different steps can be discerned that will result in several building blocks for the improvement plan. 1. setting priorities for HIS improvement 2. defining HIS improvement objectives and interventions 3. mapping ongoing and planned HIS strengthening projects and activities 4. defining a roadmap including a budget estimate. These steps and their outputs are described below. 1. Setting priorities for HIS improvement Expected output: this step will result in an overview of priority HIS problems. The HIS assessment has resulted in an overview of overall strengths and weaknesses of the HIS and a maturity score for each of the main HIS domains (see Step 1. Performing a health information system assessment: where are we now?). Commonly, multiple HIS areas that need strengthening are identified, which makes it difficult to tackle all issues at once. Therefore, the first step in developing the improvement plan is setting priorities: which HIS problems are deemed the most urgent and need to be addressed first? If a national health strategy is in place or under development, it is advisable to align the priorities for HIS improvement with the priorities of the health strategy. After all, the overall aim of an HIS is to support decision-making (see Step 2. Developing a health SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT30 information system vision: where do we want to go?). Therefore, a logical approach is to make sure that the HIS improvement efforts will result in better implementation and evaluation of the national health strategy. For example, one of the priorities of the health strategy may be prevention of NCDs. If the HIS assessment has shown that there are issues with health interview surveys, such as problems with the periodicity of the surveys, or with coverage of specific vulnerable groups in the sampling frame, solving these issues may be prioritized to ensure that the ministry of health has regular, representative data on risk factors in the population. If no health strategy is in place, international policy frameworks such as the SDGs or WHO’s EPW can be used as a proxy. See Box 5 for the priorities and flagships of WHO’s EPW (5).4 Alternatively, if only a narrow health strategy is in place, for example, focusing only on hospital care, such international policy frameworks may be used to complement priority-setting for HIS improvement. Box 5. Priorities and flagships of WHO’s EPW Core priority 1. Moving towards universal health coverage Support Member State efforts to: • put people at the centre of services • ensure and enhance financial protection • face post-COVID-19 recovery health workforce challenges • ensure access for all to medicines, vaccines and health products • improve governance and stewardship. Core priority 2. Protecting against health emergencies • learn lessons: expand the ongoing in-action review of the COVID-19 crisis into a formal review of the Region’s response to recent health emergencies; • support country preparedness and response capacity; and • reinforce regional preparedness and capacity to respond, and produce the public goods required to manage crises. Core priority 3. Promoting health and well-being • supporting local living environments that enable health and well-being; • promoting safer, healthier and better lives; • improving patient safety and tackling antimicrobial resistance; • developing strategic intelligence on levels and inequalities of health and well-being; and • reviewing major well-established programmes within WHO Regional Office for Europe’s technical portfolio, assessing their need for improved efficiency through innovation in terms of digitalization, technology and organization. Flagship initiatives: • The Mental Health Coalition • Empowerment through Digital Health • The European Immunization Agenda 2030 • Healthier behaviours: incorporating behavioural and cultural insights. 4 At the time of writing, the measurement framework for the EPW was still under development. STEP 3. DEVELOPING A HEALTH INFORMATION SYSTEM IMPROVEMENT PLAN: HOW ARE WE GOING TO ACHIEVE OUR VISION? 31 2. Defining HIS improvement objectives and interventions Expected output: an overview of HIS objectives and interventions that will form the basis of the roadmap defined in step 4. Therefore, the outputs of this step are an intermediate product. After setting priorities for HIS improvement, the next step is defining concrete objectives and interventions: what exactly do we want to achieve when tackling priority HIS problems, and how are we going to do that? Table 7 below provides examples of priority HIS problems with accompanying improvement objectives and interventions. Table 7. Examples of HIS improvement objectives and interventions Example 1 Priority HIS problem Lack of data from private health care providers HIS improvement objective Increase availability of service records from private health-care providers HIS improvement interventions Organize a meeting with private health-care providers to understand the reasons for the lack of data delivery Investigate models for private sector involvement implemented in other countries to identify international best practices Example 2 Priority HIS problem Insufficient coordination and collaboration among HIS stakeholders HIS improvement objective Facilitate both formal and informal HIS stakeholder coordination and collaboration HIS improvement interventions Establish a multistakeholder HIS coordination group with a formal mandate Organize a series of workshops on specific topics where HIS stakeholders can learn more about each other’s expertise, roles, and tasks, and where personal relationships can be established Example 3 Priority HIS problem Serious flaws in causes-of-death statistics HIS improvement objective Improve coverage and quality of causes-of-death statistics HIS improvement interventions Investigate the reasons for insufficient coverage of death registrations by the civil registry Organize training for medical doctors on how to fill in the death registration forms according to WHO guidelines Make resources available for expanding and training the number of coders at the statistical institute A useful framework for determining which interventions are needed to tackle an HIS problem is the Performance of Routine Information System Management (PRISM) conceptual model developed by MEASURE Evaluation (see Fig. 7). This model presents the relationship of HIS inputs and SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT32 processes with the outputs, outcomes, and impact of the HIS, as well as the HIS determinants: the technical, organizational, and behavioural factors that influence the outputs and outcomes of HIS improvement interventions. Use of the PRISM Conceptual Model helps to visualize the critical elements needed to address the problems identified by the HIS assessment. For example, technical interventions to introduce ICT solutions for real-time access to data to support prompt decision-making require coupling the ICT solutions with organizational interventions to establish good management, interoperability, and maintenance of the ICT systems, and making a conscious effort to develop the skills of health and ICT staff (43). Fig. 7. The PRISM conceptual model by MEASURE Evaluation Source: MEASURE Evaluation, 2018 (43). There are many tools for interventions to improve the HIS. WHO’s SCORE for Health Data Technical Package brings many of them together and provides a comprehensive resource for essential HIS strengthening interventions (44,45). See Box 6 for more information. More examples of HIS improvement interventions can be found on the MEASURE Evaluation website (46). HIS DETERMINANTS Technical factors y Complexity of reporting forms, procedures y HIS design y Computer software y Information technology complexity Organizational factors y Critical management functions and information needs y Governance y Planning y Training y Supervision y Quality y Finance Behavioural factors y Level of knowledge of content of HIS forms y Data quality checking skills y Problem- solving for HIS tasks y Competence in HIS tasks y Confidence levels for HIS tasks y Motivation HIS processes y Data collection y Data transmission y Data processing y Data analysis y Data quality check y Feedback Improved health system performance Improved health status Improved HIS performance y Data quality y Information use Availability of resources INPUTS PROCESSES OUTPUTS OUTCOMES IMPACT Promotion of a culture of information STEP 3. DEVELOPING A HEALTH INFORMATION SYSTEM IMPROVEMENT PLAN: HOW ARE WE GOING TO ACHIEVE OUR VISION? 33 Box 6. WHO’s SCORE HIS strengthening tools SCORE stands for survey (population and health risks), count (births, deaths and causes of death), optimize (health services data), review (progress and performance), and enable (data use for policy and action). For each of these segments underlying key elements are defined, and for each of these key elements an overview of existing tools and standards is provided. The key elements of the survey segment are: y system of regular population-based health surveys y surveillance of public health threats y regular population census. Tools to support implementation and improvement of these three HIS elements include manuals for conducting surveys, guidance for setting up an early warning, alert and response system, and standards for population censuses. These tools can be found on the WHO website at: https://score.tools.who.int/tools/survey-populations-and-health-risks/. The key elements of the count segment are: y full birth and death registration y certification and reporting of causes of death. Tools to support implementation and improvement of these two HIS elements include a handbook on CRVS management, operations and maintenance, a CRVS eLearning course, and the ICD-11 and related manuals and training. These tools can be found on the WHO website at: https://score.tools.who.int/tools/count-births-deaths-and-causes-of-death/. The key elements of the optimize segment are: y routine facility and community reporting system with patient monitoring y regular system to monitor service availability, quality and effectiveness y health service resources: finance and health workforce data. Tools to support implementation and improvement of these three HIS elements include the International Classification of Health Interventions, guidance for performing facility surveys, the System of Health Account manual, and the National Health Workforce Accounts handbook. These tools can be found on the WHO website at: https://score.tools.who.int/tools/ optimize-health-service-data/. The key elements of the review segment are: y regular analytical progress and performance reviews, with equity y institutional capacity for analysis and learning. Tools to support implementation and improvement of these two HIS elements include health inequality monitoring resources, a toolbox to help countries examine the geographic aspects of their health system, and a data quality review toolkit. These tools can be found on the WHO website at: https://score.tools.who.int/tools/review-progress-and-performance/. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT34 Box 6 contd The key elements of the enable segment are: y data and evidence drive policy and planning y data access and sharing y strong country-led governance of data. Tools to support implementation and improvement of these three HIS elements include a guide with a set of tools for improving the demand for and use of data to inform health policy and decision-making, practical guides for making data meaningful for non-statisticians, recommendations on digital interventions for health system strengthening, and a Health Information Systems Interoperability Maturity Toolkit. These tools can be found on the WHO website at: https://score.tools.who.int/tools/enable-data-use-for-policy-and-action/. In addition to these tools and standards, which to some extent focus on low- and middle- income countries, the WHO Regional Office for Europe aims to develop a set of additional, complementary tools and guidance documents for strengthening HIS that are specifically geared towards the WHO European Regional context. 3. Mapping ongoing and planned HIS strengthening projects and activities Expected output: overview of ongoing and planned HIS strengthening activities After setting priorities and defining HIS improvement objectives and interventions, the next step is to map ongoing and planned HIS strengthening projects and activities, to identify initiatives that can contribute to the objectives. For this purpose, it is suggested that any activity whose purpose is to improve architecture and procedures for data capture, storage, flow and sharing, enhance analysis, reporting and knowledge translation capacities, and reinforce HIS governance and resources should be listed. For example, a new statistical law may be in the making that provides an opportunity for improving the legal framework for the secondary use of data for population health surveillance, or a skill-building workshop on making data visualizations may be planned for health information experts. There may also be a project at the initiative of a group of hospitals on improving the interoperability of their digital information systems, which could be used as a pilot project for developing national interoperability standards. Mapping such projects and activities will make clear where synergies with the set HIS improvement objectives and interventions can be created, and will thus prevent investing in activities that overlap with existing initiatives. Some examples of potential ongoing HIS strengthening activities are listed in Box 7. Routine activities such as data entry, cleansing and analysis and the production of reports resulting from the analysis of service and survey data should not be considered strengthening in nature and should not be included in the mapping (8). STEP 3. DEVELOPING A HEALTH INFORMATION SYSTEM IMPROVEMENT PLAN: HOW ARE WE GOING TO ACHIEVE OUR VISION? 35 Box 7. Examples of HIS strengthening activities y The development of: △ new or improved records, registers, reports and data flow procedures; and △ new computer applications for data entry, database management and report generation. y The development and implementation of new procedures for facilitating the use of existing data in support of planning, M&E of health programmes and services y The design and conduct of: △ new or strengthened training (basic and in-service) curricula and materials aimed at introducing new or improving the performance of existing data management functions at various levels of the health system; and △ new surveys or survey modules, M&E processes and investigative procedures intended to be routinely or periodically applied in the future. y Any special data analysis effort and generation of information products using procedures that can be repeated in the future. y Revision and/or formulation of new legislation and regulations on health event notification and service reporting requirements. y Establishment of statistical and information coordinating committees. Source: Health Metrics Network, 2009 (8). 4. Defining a roadmap and a budget Expected output: HIS improvement roadmap including a budget estimate In step 2 we have defined HIS improvement interventions. To support the implementation of these interventions, it is advisable to break them down into concrete activities and define the main outcome that these activities should bring. For each of the defined activities, a start and end date can be added, as well as the responsible agency, thus creating an HIS improvement roadmap. A template for such a roadmap, partly filled in with examples, is provided in Annex 3. The template also includes columns for entering the types of resources needed and, on that basis, an estimate of the budget needed per activity. Possible resource types include funds for development work, technical support, equipment, materials and documentation, and training. While most of the activities will be one- off development work, some may be recurring and eventually become additional routine activities and expenses, such as salaries of new staff, Internet access expansion costs, or maintenance costs for a new database. Making this distinction between one-off developmental and recurrent costs enables the stakeholders involved in the implementation of the HIS improvement plan to estimate both the costs necessary for the implementation of the activities in the plan over the next 2–5 years and the longer-term incremental regular costs for operating the HIS as a result of the structural improvements that will have been implemented in the system. The cost summary thus created will allow national authorities to secure the necessary budget for the planned HIS improvement trajectory. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT36 Finalizing the four steps as described above will result in an overview of priority HIS problems, an overview of planned and ongoing HIS strengthening activities, and a roadmap. Together, these three elements form the HIS improvement plan. Guiding principles for HIS development HISs are highly diverse, and policy priorities and available resources in countries may vary. Therefore, national HIS improvement plans are likely to differ to a considerable extent. Nevertheless, there are common principles that can be used in the development of an HIS strategy to help ensure the maximum impact of HIS improvement interventions and activities. These guiding principles are listed below. The first five principles are based on the principles of the Paris Declaration on aid effectiveness (48), which are also used by HMN and by MEASURE Evaluation (11,43). 1. Leadership and ownership. The process of developing and then implementing the HIS strategy should be led and owned by the country or organization responsible for the HIS. In most countries, the ministry of health is the first entity responsible for the national HIS. 2. Stakeholder involvement and consensus-building. The HIS strategy should be developed with the engagement of stakeholders, to ensure broad-based consensus and stakeholder buy-in. 3. A focus on the needs of Member States. The activities and interventions in the strategy should be relevant to the country context and address the priority needs of the country or organization, including its subunits. 4. Building on what exists. Wherever possible, the implementation process of the HIS strategy should build upon existing initiatives, systems, and knowledge. Strengthening HIS should not take place in a vacuum but should be linked to and build upon similar initiatives, especially national and international strategies for the development of statistics. 5. Sustainability and flexibility. The HIS improvement interventions and activities should lead to the sustainability of the HIS, so that the system can satisfy the present information needs and evolve as those needs change. 6. Registration at the source. The HIS strengthening activities should support the optimization of registration at the level at which the data are first generated to increase confidence in the completeness, quality and reliability of the data at higher levels in the health system (8). 7. International standards. The HIS strengthening activities should pursue implementation of and compliance with international standards of data and statistics quality and governance (8). 8. Integration and interoperability. Strong HISs are integrated and interoperable (48,49). The HIS strengthening activities should, therefore, contribute to an integrated and interoperable HIS, that is, an HIS in which health data can be exchanged, triangulated across data sources, and used across multiple disciplines, sectors and domains, and in which indicators are aligned across the entire HIS (12). 9. Digital solutions. The digital revolution should be used to help improve the availability, completeness, timeliness, quality, and use of data for decision-making, and minimize the burden of data collection (50), but it is important to be realistic about what technology can and cannot do, and about what is needed in terms of maintenance and training for well-functioning and sustainable digital solutions. STEP 4. MONITORING PROGRESS AND PERFORMING REGULAR EVALUATIONS: ARE WE ON TRACK? 37 Step 4. Monitoring progress and performing regular evaluations: are we on track? The fourth step in the HIS strengthening process is monitoring progress and performing regular evaluations. This will answer the question: are we on track? The M&E plan provides guidance for this step by defining what will be measured to track progress (M&E framework), how this will be measured and reported on (M&E process), and how follow-up is arranged (M&E governance). The M&E plan is the third and final element of the HIS strategy. In the previous chapter we have defined HIS improvement objectives, interventions, and activities. In this chapter we will define the M&E plan to monitor whether the activities agreed in the HIS improvement plan are being performed and whether the objectives are being met. The M&E plan consists of an M&E framework, defining what will be measured to track progress, and an M&E process, defining how progress will be monitored and reported on. Finally, the M&E plan defines the M&E governance, meaning that the plan stipulates how follow-up of the outcomes of the M&E activities will be arranged. Developing the M&E plan: what, how, and follow-up Defining the M&E framework: what are we going to measure to track progress? The M&E plan builds on the roadmap, which brings together HIS improvement objectives, interventions, and activities. As the first element of the M&E framework, it is recommended to define indicators for monitoring whether the agreed HIS improvement objectives are being met. Looking at the HIS improvement objective example about increasing the availability of service records from private health care providers elaborated in Annex 3, we see that the accompanying HIS interventions relate to organizing a stakeholder meeting and doing research to provide the ministry of health with practice- and science-informed policy recommendations. It is expected that the ministry of health will adapt its policies based on these recommendations, but that does not necessarily guarantee that the availability of data from the private sector will improve to the desired level. To establish whether the interventions indeed result in increased availability of service records, indicators such as those presented in Table 8 can be used. For each indicator, it is recommended to define a baseline and a target value, as well as the desired frequency of data collection or reporting (see Table 8). In addition, for each of the HIS improvement activities in the roadmap, the desired outputs and a time frame have been defined. If we proceed with the private health-care providers example in Annex 3, we observe, for example, that a literature overview and interview transcripts need to be ready by March 2021, and a scientific report by May 2021. These outputs and their deadlines can also be included in the M&E framework, as they allow for tracking SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT38 whether the agreed activities have indeed been performed on time and whether they have resulted in the envisaged outputs. Different types of indicators exist, each with their own benefits and caveats. The box on page 39 (In focus: using different types of indicator) investigates the differences between outcome and process indicators and between quantitative and qualitative indicators. Table 8. Examples of indicators for measuring progress towards the HIS improvement objectives HIS improvement objective Indicators Baseline Target Frequency of reporting 1. Increase availability of service records from private health-care providers % of licensed private providers submitting HIS reports to the ministry of health 50% (2019) 80% (2022) Annually Completeness of HIS reports submitted to the ministry of health by licensed private providers 40% complete (2019) 80% complete (2022) Annually Private providers are represented in the multistakeholder HIS coordination mechanism No (2019) Yes (2020) Annually 2. Facilitate both formal and informal HIS stakeholder coordination and collaboration Attendance rate for the regular multistakeholder coordination group meetings - At least 90% of coordination group members are present Half-yearly Evaluation of multistakeholder workshops - At least 65% of participants indicate that the workshops contributed to better networking opportunities Half-yearly 3. Improve coverage and quality of causes- of-death statistics % of deaths covered by the civil registry 87% (2019) 95% (2023) Annually % of death registrations including cause-of-death information 80% (2019) 95% (2023) Annually % of death registrations with ill-defined causes of death 20% (2019) 5% (2023) Annually Note: the baseline and target values used in this table are fictional. Source: Health Metrics Network, 2009 (8). STEP 4. MONITORING PROGRESS AND PERFORMING REGULAR EVALUATIONS: ARE WE ON TRACK? 39 In focus: using different types of indicators There are different kinds of indicators, each with its own usages and caveats. Here we will investigate the differences between outcome and process indicators and between quantitative and qualitative indicators. Process and outcome indicators Process indicators measure a programme or policy’s activities and outputs (direct products or deliverables of the activities). Together, measures of activities and outputs indicate whether the programme is being implemented as planned. Outcome indicators measure whether a programme or policy is achieving the expected effects and/or changes in the short, intermediate, and long term. Some programmes or policies refer to their longest-term or most distant outcome indicators as impact indicators (51). Some of the main characteristics of process and outcome indicators include (adapted from Smith et al., 2009 (52)). y Process indicators: △ readily measured; △ easier to interpret and provide clear pathways for action; △ may have little value to potential users of the indicators unless they understand how they relate to health outcomes; and △ may be more easily manipulated. y Outcome indicators: △ relatively easy to measure some outcomes validly and reliably (for example, death) but others are notoriously difficult (for example, wound infection); △ may be difficult to interpret, as they stem from many factors that are difficult to disentangle; △ often more meaningful to potential users of the indicators; and △ not easily manipulated. Quantitative and qualitative indicators Quantitative indicators are indicators that are expressed as numbers. There are different ideas of what constitutes a qualitative indicator. First, to some, qualitative indicators are indicators measured and/or reported in words rather than in numbers. An example of such an indicator is an indicator that asks whether a specific policy is in place. Another common definition of qualitative indicators is that they are subjective (that is, about people’s opinions, attitudes, or beliefs) rather than objective. For example, an often-used qualitative measure in population health surveillance is self-reported health status. For the computation of this indicator, respondents are asked to rate their own health, typically on a four- or five-point scale. A downside of qualitative subjective indicators is that it can be difficult to use them in international comparisons because they often relate to concepts that have different meanings in different cultural contexts. This can lead to difficulties in translating survey questions. Cultural contexts may also influence the way people answer survey questions. As a result, it is often difficult to interpret international comparisons of qualitative subjective indicators. Not everything that is relevant can be captured by objective indicators and expressed as numbers. Therefore, in many cases, both quantitative and qualitative indicators are needed to provide a comprehensive picture of the situation. Different types of indicator complement each other There are no good or bad types of indicator. Each type has its specific advantages and usages. What will be the most appropriate indicators to use will depend on the situation at hand. Often a mix of different indicators will be the most informative and, thus, preferred option. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT40 Defining the M&E process: how are we going to measure progress? Who will do the monitoring and reporting? When defining the M&E process, one of the questions that needs to be answered is who will do the necessary data collection, data analysis and reporting for the M&E framework. If more practical, data collection can be divided over multiple stakeholders, but it is recommended to make one stakeholder or agency responsible for analysis and reporting, to help ensure a uniform approach. In most countries, the ministry of health is the first entity responsible for the national HIS. For transparency reasons, it is therefore recommended to appoint a stakeholder or agency responsible for data analysis and reporting outside the ministry. Ideally, this stakeholder or agency has a good understanding of the various elements of the HIS, as this will contribute to balanced reporting on the M&E framework. Of course, the agency appointed to do the monitoring and reporting needs to receive sufficient funding for carrying out this task. Moreover, for several of the indicators defined in the M&E framework, routine data collection probably cannot be used, and additional effort will be needed to gather the necessary data. See the indicator examples in Table 8 above: the indicators on causes- of-death statistics will probably be collected routinely already, but this may not apply to other indicators such as “% of licensed private providers submitting HIS reports to the ministry of health” or “completeness of HIS reports submitted to the ministry of health by licensed private providers”. The resources that will be made available for M&E should also cover the additional data collection efforts needed for the calculation of such indicators. Which reporting frequency and dissemination tools will be used? Another aspect that needs to be elaborated is the manner of reporting on the outcomes of the M&E framework monitoring. This can take the form of an online tool, factsheet, policy brief, or a report, or a combination of reporting formats. The frequency of reporting (for example, once a year, or a limited report yearly and a more elaborate report once every three years) and the target audience will also need to be determined. It is recommended that the M&E reports be made public and accessible to everyone. Nonetheless, targeted products and communication activities can be deployed, ensuring that specific groups such as the cabinet of ministers, parliament, health authorities, or important donors are informed about the progress reports. Defining M&E governance: how is follow-up of the outcomes of the M&E activities arranged? The first two elements of the M&E plan described above (the M&E framework and the M&E process) will result in regular reports on the progress of HIS improvement according to a uniform structure that will enable tracking of progress over time. The next element of the M&E plan that needs to be defined is how the follow-up of the findings in the progress reports will be arranged. We refer to this set of mechanisms and procedures as M&E governance. STEP 4. MONITORING PROGRESS AND PERFORMING REGULAR EVALUATIONS: ARE WE ON TRACK? 41 Multistakeholder guidance committee It is recommended to establish a multistakeholder guidance committee that has a central role in the follow-up process. The committee should include at least the stakeholders who were involved in the HIS assessment (see Step 1. Performing a health information system assessment: where are we now?). The guidance committee can advise the ministry of health – or another entity in charge of the HIS improvement process – about whether and how the HIS improvement plan should be adapted based on the findings in the progress reports, identify barriers in the implementation of the HIS improvement plan in daily practice and advise on how to overcome them, and identify emerging needs that are not yet captured in the HIS improvement plan. Formal terms of reference for the guidance committee should be established, defining, for example, the frequency of meetings, the role of the committee (whether it is strictly advisory or whether the group also has decision-making powers), procedure for nomination of the chair (preferably an independent chair), and how the committee’s recommendations will be reported on and how the ministry will respond to it (for example, an official response in writing). Indicators for measuring M&E performance In addition to indicators linked to the HIS improvement objectives as described above, it is also worth including some indicators and targets in the M&E plan that measure performance related to the M&E plan itself: are the M&E activities going as agreed? Are we meeting the outputs and standards that we have set for ourselves? See Table 9 for examples. Table 9. Examples of indicators and targets for measuring performance of the M&E process Indicator Target Terms of reference for the multistakeholder guidance group In place, publicly available Meetings of multistakeholder guidance group Take place twice a year Meeting reports of the multistakeholder guidance group Published twice a year, publicly available HIS improvement progress reports based on the M&E framework Published once a year, publicly available Completeness of HIS improvement progress reporting 100% of the outputs and indicators in the M&E framework are reported on The ministry of health and the multistakeholder guidance committee discuss the HIS improvement progress report and its implications for the HIS improvement plan Discussions take place once a year Publication of meeting reports of the multistakeholder guidance group Within 4 weeks after the meeting Official response from the ministry of health to the HIS improvement progress reports and the recommendations based on those from the multistakeholder guidance group Once a year, publicly available The ministry of health’s communication strategy related to the HIS improvement progress reports and subsequent adaptations to the HIS improvement plan In place Mechanism to collect feedback on the HIS improvement progress reports from HIS stakeholders not represented in the multistakeholder guidance group and from civilians In place SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT42 Compiling the HIS strategy After the M&E plan has been defined, all the elements for compiling the HIS strategy are in place. Defining the M&E plan was step 4 in this tool. Step 1 provides guidance for performing an HIS assessment, and steps 2, 3 and 4 all contribute to the development of an HIS strategy. The outputs of these three steps can be combined into a comprehensive HIS strategy document. Box 8 summarizes these outputs to provide a brief overview of how the HIS strategy can be assembled. Box 8. Elements of the HIS strategy document 1. HIS vision The HIS vision defines what the HIS should ultimately achieve (the HIS goals) and how it should be achieved (the HIS values). 2. HIS improvement plan The HIS improvement plan has been developed by following several steps, which result in: y an overview of priority HIS problems y an overview of ongoing and planned HIS strengthening activities y an HIS roadmap. These three elements together form the HIS improvement plan. For each of the prioritized HIS problems, the HIS roadmap includes: y the HIS improvement objectives; y the HIS improvement interventions; and y the HIS improvement activities, including expected outputs, timeline, responsible stakeholder or agency, and budget estimate. The interventions and activities in the HIS roadmap have been cross-referenced with: y the outcomes of the mapping of ongoing and planned HIS strengthening activities, to identify opportunities for synergies and prevent overlaps; and y the guiding principles for HIS development, to ensure that the HIS improvement interventions and activities have a maximum impact. 3. M&E plan The M&E plan defines what will be measured to track progress (M&E framework), how it will be measured and reported on (M&E process), and how follow-up is arranged (M&E governance). The M&E framework includes: y indicators to measure progress towards each of the HIS improvement objectives (to monitor whether the set objectives are being met); and y outputs and timelines as defined in the roadmap, see under HIS improvement plan (to monitor whether we are achieving what we planned to do). The M&E process defines how progress will be measured. It elaborates: y who will be doing the monitoring of and reporting on the M&E framework y what reporting frequency and tools will be used. The M&E governance defines how the follow-up of the findings in the progress reports will be arranged. As part of M&E governance, it is recommended to: y establish a multistakeholder guidance committee y define indicators to measure M&E performance. REFERENCES 43 References 1. Support tool to assess health information systems and develop and strengthen health information strategies. Copenhagen: WHO Regional Office for Europe; 2015 (https://apps.who.int/ iris/handle/10665/172761, accessed 29 June 2021). 2. Verschuuren M, Diallo K, Calleja N, Burazeri G, Stein C. First experiences with a WHO tool for assessing health information systems. Public Health Panorama. 2016;2(3):379–82 (https://apps. who.int/iris/handle/10665/325429, accessed 29 June 2021). 3. Review of public health capacities and services in the WHO European Region. Copenhagen: WHO Regional Office for Europe; 2012 (https://apps.who.int/iris/handle/10665/340436, accessed 29 June 2021). 4. Azzopardi-Muscat N, Kluge HHP, Asma S, Novillo-Ortiz D. A call to strengthen data in response to COVID-19 and beyond. J Am Med Inform Assoc. 2021;28(3):638–9. doi: 10.1093/jamia/ ocaa308. 5. European programme of work, 2020–2025: united action for better health in Europe. Copenhagen: WHO Regional Office for Europe; 2020 (EUR/RC70/11 Rev.4; https://apps.who.int/ iris/handle/10665/339209, accessed 29 June 2021). 6. Thirteenth General Programme of Work 2019−2023. Geneva: World Health Organization; 2019 (https://apps.who.int/iris/handle/10665/279451, accessed 29 June 2021). 7. Sustainable Development Goals. New York: United Nations; 2015 (https://sdgs.un.org/goals, accessed 29 June 2021). 8. Health Metrics Network. Guidance for the health information systems (HIS) strategic planning process: steps, tools and templates for HIS systems design and strategic planning, version 6. Geneva: World Health Organization; 2009 (https://www.measureevaluation.org/his- strengthening-resource-center/resources/Guidanceforthe-healthInformationSystemsHISStrategi cPlanningProcess.pdf, accessed 29 June 2021). 9. Gissler M, Dumitrescu A, Addor V. Improving the performance of National Health Information Systems: the 2002–2003 reform in Finland from an international perspective. Copenhagen: WHO Regional Office for Europe; 2006 (https://apps.who.int/iris/handle/10665/107685?locale- attribute=ar&, accessed 29 June 2021). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT44 10. Defining health information systems. In: Health information systems strengthening resource center [website]. Chapel Hill (MD): MEASURE Evaluation; 2021 (https://www.measureevaluation. org/his-strengthening-resource-center/his-definitions.html, accessed 29 June 2021). 11. Framework and standards for country health information systems, second edition Geneva: Health Metrics Network and World Health Organization; 2008 (https://apps.who.int/iris/ handle/10665/43872, accessed 29 June 2021). 12. Rechel B, Rosenkötter N, Verschuuren M, van Oers H. Health information systems. In: Verschuuren M, van Oers H, editors. Population health monitoring: climbing the information pyramid. Basingstoke: Springer Nature; 2019:11–34. 13. Rechel B, Rosenkötter N, Verschuuren M, van Oers H. Introduction. In: Verschuuren M, van Oers H, editors. Population health monitoring: climbing the information pyramid. Basingstoke: Springer Nature; 2019:1–9. 14. Ackoff R. From data to wisdom. J Appl Sys Anal. 1989;16:3–9. 15. Verschuuren M, van Oers H. Population health monitoring: an essential public health field in motion. Bundesgesundheitsbl. 2020;63(9):1134–42. doi: 10.1007/s00103-020-03205-9. 16. Jansen T, Coppen R, Urbanus T, Bos N, Verheij R. Naar een lerend zorgsysteem voor de ambulancezorg: haalbaarheidsstudie hergebruik en gegevenskoppeling routine zorgdata [Towards a learning care system for ambulance care: feasibility study reuse and data linkage routine care data]. Utrecht: Nivel; 2019 (in Dutch; https://nivel.nl/sites/default/files/bestanden/ Haalbaarheidsstudie_ambulancegegevens.pdf, accessed 29 June 2021). 17. EPHO1: surveillance of population health and wellbeing. Copenhagen: WHO Regional Office for Europe; 2020 (https://www.euro.who.int/en/health-topics/Health-systems/public-health- services/policy/the-10-essential-public-health-operations/epho1-surveillance-of-population- health-and-wellbeing, accessed 29 June 2021). 18. HIV/AIDS. Category 6: Monitoring and evaluation. Copenhagen: WHO Regional Office for Europe (https://www.who.int/hiv/topics/vct/sw_toolkit/monitoring_and_evaluation/en/, accessed 29 June 2021). 19. Fekri O, Macarayan ER, Klazinga N. Health system performance assessment in the WHO European Region: which domains and indicators have been used by Member States for its measurement? Copenhagen: WHO Regional Office for Europe; 2018 (https://apps.who.int/iris/ handle/10665/326260, accessed 29 June 2021). 20. Dahlgren G, Whitehead M. Policies and strategies to promote social equity in health. Background document to WHO: strategy paper for Europe. Bellville: Institute for Future Studies; 1991 (https://core.ac.uk/download/pdf/6472456.pdf, accessed 29 June 2021). 21. What is good governance? Bangkok: United Nations Economic and Social Commission for Asia and the Pacific; 2009 (https://www.unescap.org/sites/default/files/good-governance.pdf, accessed 29 June 2021). REFERENCES 45 22. Brinkerhoff D. Accountability and health systems: overview, framework, and strategies. Bethesda (MD): Partners for Health Reformplus, Abt Associates; 2003 (https://www.who. int/management/partnerships/accountability/AccountabilityHealthSystemsOverview.pdf, accessed 29 June 2021). 23. Health systems governance for universal health coverage: action plan. Geneva: World Health Organization; 2014 (https://apps.who.int/iris/handle/10665/341159, accessed 29 June 2021). 24. Health Metrics Network. Assessing the national health information system: an assessment tool, version 4.00. Geneva: World Health Organization; 2008 (https://apps.who.int/iris/ handle/10665/43932, accessed 29 June 2021). 25. WP 5: status of health information systems in MS and regions. Berlin: Information for Action; 2020 (https://www.inf-act.eu/wp5, accessed 29 June 2021). 26. Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies. Geneva: World Health Organization; 2010 (https://apps.who.int/iris/ handle/10665/258734, accessed 29 June 2021). 27. Routine health information system rapid assessment tool. Chapel Hill (NC): MEASURE Evaluation; 2020 (https://www.measureevaluation.org/resources/tools/rhis-rat/routine-health- information-system-rapid-assessment-tool, accessed 29 June 2021). 28. SCORE assessment instrument. Geneva: World Health Organization; 2021 (https://www.who. int/publications/i/item/assessment-methodology-2020, accessed 29 June 2021). 29. 12 components monitoring and evaluation system strengthening tool. Geneva: Joint United Nations Programme on HIV and AIDS; 2009 (https://www.unaids.org/sites/default/ files/sub_landing/files/2_MERG_Strengthening_Tool_12_Components_ME_System.pdf, accessed 29 June 2021). 30. Data quality review: module 1: framework and metrics. Geneva: World Health Organization; 2017 (https://apps.who.int/iris/handle/10665/259224?search-result=true&query=Data+quality+re view%3A+module+1%3A+framework+and+metrics&scope=&rpp=10&sort_by=score&order=desc, accessed 29 June 2021). 31. Data quality review: module 2: desk review of data quality. Geneva: World Health Organization; 2017 (https://apps.who.int/iris/handle/10665/259225?search-result=true&query=Data+quality+re view%3A+module+1%3A+framework+and+metrics&scope=&rpp=10&sort_by=score&order=desc, accessed 29 June 2021). 32. Data quality review: module 3: data verification and system assessment. Geneva: World Health Organization; 2017 (https://apps.who.int/iris/handle/10665/259226?search-result=true& query=Data+quality+review%3A+module+1%3A+framework+and+metrics&scope=&rpp=10&so rt_by=score&order=desc, accessed 29 June 2021). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT46 33. Information systems for health standard assessment method IS4H-MM version 1.0 – March 2018. Washington (DC): Pan American Health Organization and World Health Organization Regional Office for the Americas; 2018 (https://www.paho.org/ish/images/docs/maturity. pdf?ua=1, accessed 29 June 2021). 34. Greenwell F, Wambugu S. Demand and readiness tool for assessing data sources in health information systems (HIS DART). Chapel Hill (NC): MEASURE Evaluation; 2018 (https://www. measureevaluation.org/resources/publications/tl-18-14, accessed 29 June 2021). 35. Verschuuren M, Calleja N, Bogaert P. WP5 task 1: health information system assessment manual: objectives, process & guidelines, and roles & tasks. Berlin: Information for Action; 2018 (https://www.inf-act.eu/sites/inf-act.eu/files/2020-01/UPDATED_%20InfAct%20assessment%20 manual_1.0_Final.pdf, accessed 29 June 2021). 36. HIS assessment tools. Chapel Hill (NC): MEASURE Evaluation; 2021 (https://www. measureevaluation.org/his-strengthening-resource-center/his-tools, accessed 29 June 2021). 37. Stakeholder analysis, project management, templates and advice. In: stakeholdermap.com [website]. Reading: Stakeholdermap.com; 2008 (https://www.stakeholdermap.com/stakeholder- definition.html, accessed 29 June 2021). 38. Performance of routine information system management (PRISM) user’s kit: preparing and conducting a PRISM Assessment. Chapel Hill (NC): MEASURE Evaluation; 2018 (https://www. measureevaluation.org/resources/publications/ms-18-140/, accessed 29 June 2021). 39. Graham J, Amos B, Plumptre T. Principles for good governance in the 21st century. Ottowa: Institute On Governance; 2003 (Policy Brief No. 15; https://iog.ca/docs/2003_August_ policybrief15.pdf). 40. Vision statement. In: Wikipedia [website]. San Francisco: Wikimedia; 2021 (https:// en.wikipedia.org/wiki/Vision_statement, accessed 29 June 2021). 41. Kantabutra S. What do we know about vision?, J Appl Bus Res. 2008;24(2):127–38. doi: 10.19030/jabr.v24i2.1359. 42. Kantabutra S, Avery GC. The power of vision: statements that resonate. J Bus Strategy. 2010;31(1):37–45. doi: 10.1108/02756661011012769. 43. Performance of Routine Information System Management (PRISM) user’s kit: moving from assessment to action. Chapel Hill (NC): MEASURE Evaluation; 2018 (https://www. measureevaluation.org/resources/publications/ms-18-142/, accessed 29 June 2021). 44. SCORE for health data technical package: essential interventions. Geneva: World Health Organization; 2020 (https://apps.who.int/iris/handle/10665/334005?search-result=true&query=E ssential+Interventions&scope=&rpp=10&sort_by=score&order=desc, accessed 29 June 2021). REFERENCES 47 45. SCORE for health data technical package: tools and standards for SCORE essential interventions. Geneva: World Health Organization; 2020 (https://apps.who.int/iris/ handle/10665/334006?search-result=true&query=Essential+Interventions&scope=&rpp=10&so rt_by=score&order=desc, accessed 29 June 2021). 46. Common set of HIS interventions, health information systems strengthening resource center. Chapel Hill (NC): MEASURE Evaluation (https://www.measureevaluation.org/his-strengthening- resource-center/his-interventions/common-set-of-his-interventions, accessed 29 June 2021). 47. Paris declaration on aid effectiveness. Paris: Organisation for Economic Co-operation and Development; 2005 (https://doi.org/10.1787/9789264098091-fr, accessed 29 June 2021). 48. Michelsen K, Brand H, Achterberg P, Wilkinson J. Promoting better integration of health information systems: best practices and challenges. Copenhagen: WHO Regional Office for Europe; 2015 (https://apps.who.int/iris/handle/10665/152819, accessed 29 June 2021). 49. What are the characteristics of a strong health information system? Chapel Hill (NC): MEASURE Evaluation, United States Agency for International Development; 2018 (https://www. measureevaluation.org/resources/publications/fs-18-294, accessed 29 June 2021). 50. Strengthening health information systems; New Delhi: WHO Regional Office for South- East Asia; 2017 (https://apps.who.int/iris/bitstream/handle/10665/259716/UHC_tech_brief. pdf?sequence=1&isAllowed=y, accessed 29 June 2021). 51. Indicators. CDC approach to evaluation. Atlanta (GA): Centers for Disease Control and Prevention; 2016 (https://www.cdc.gov/eval/indicators/index.htm, accessed 29 June 2021). 52. Smith P, editor. Performance measurement for health system improvement: experiences, challenges and prospects. Cambridge: Cambridge University Press; 2009. 53. Everybody's business -- strengthening health systems to improve health outcomes : WHO's framework for action. Geneva: World Health Organization; 2007 (https://apps.who.int/iris/ handle/10665/43918, accessed 2 July 2021). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT48 Term Definition Health information system (HIS) A complex, multilevel system aimed at producing health intelligence to inform decision-making, with the following main functions or domains: • data collection • analysis • health reporting • knowledge translation • governance and resources Health system All the activities whose primary purpose is to promote, restore and/or maintain health; (ii) the people, institutions and resources, arranged together in accordance with established policies, to improve the health of the population they serve, while responding to people’s legitimate expectations and protecting them against the cost of ill-health through a variety of activities whose primary intent is to improve health (1). Health system performance assessment An assessment of a health system as a whole, using a limited number of indicators linking outcomes with functions or strategies. It is country-specific, embedded in a national or subnational policy process, and linked to national health plans or strategies wherever possible (2). HIS assessment A formal and structured process of evaluation of the functions of the HIS, either quantitative or qualitative, depending on what is appropriate and feasible (1). HIS resources HIS resources encompass everything that is needed to operate the HIS, and range from small items to large structures and from very concrete to less tangible. HIS resources include general resources that are needed for all parts of the HIS (for example, human resources (volume and capacity), ICT infrastructure, legal framework) and specific resources that are linked to specific phases in the population health surveillance process (for example, databases, indicator sets, quality criteria for health reporting, knowledge translation tools) (3). HIS stakeholder Anybody who can affect or is affected by the HIS assessment. Stakeholders can be individuals, groups, or organizations. They can be internal or external (4). HIS strategy A method or plan of action chosen to bring about a desired future for the HIS (5). It is designed to achieve a long-term aim Note. This tool provides guidance for the development of an HIS strategy document that consists of: • an HIS vision • an HIS improvement plan • an M&E framework. Glossary GLOSSARY 49 Term Definition Monitoring and evaluation (M&E) A system (6) to provide: • information on what an intervention is doing, how well it is performing and whether it is achieving its aims and objectives; • guidance on future intervention activities; and • an important part of accountability to funding agencies and stakeholders. Plans for M&E should be made at the beginning of an intervention development process Population health surveillance or monitoring Population health surveillance or monitoring can be defined as regular and institutionalized production and dissemination of information and knowledge about the health status of a population and its determinants, aimed at informing policy-making (3). References 1. Health systems strengthening glossary. Geneva: World Health Organization; 2011 (https://www. who.int/healthsystems/Glossary_January2011.pdf, accessed 29 June 2021). 2. Fekri O, Macarayan ER, Klazinga N. Health system performance assessment in the WHO European Region: which domains and indicators have been used by Member States for its measurement? Copenhagen: WHO Regional Office for Europe; 2018 (https://apps.who.int/iris/ handle/10665/326260, accessed 29 June 2021). 3. Verschuuren M, van Oers H, editors. Population health monitoring: climbing the information pyramid. Basingstoke: Springer Nature; 2019. 4. Stakeholder analysis, project management, templates and advice. In: stakeholdermap.com [website]. Reading: Stakeholdermap.com; 2008 (https://www.stakeholdermap.com/stakeholder- definition.html, accessed 29 June 2021). 5. Strategy: definition and features. In: ManagementStudyGuide.com [website] (https://www. managementstudyguide.com/strategy-definition.htm, accessed 29 June 2021). 6. HIV/AIDS. Category 6: monitoring and evaluation. Copenhagen: WHO Regional Office for Europe (https://www.who.int/hiv/topics/vct/sw_toolkit/monitoring_and_evaluation/en/, accessed 29 June 2021). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT50 Health information for GPW13 and EPW What is the role of health information for GPW13 and EPW in the wider HIS? The impact measurement frameworks for WHO’s GPW13 and EPW bring together indicators from different parts and functions of the health-care system and from different domains (1). As such, it is a cross-cutting theme for the HIS. The framework spans several domains, including public health, health care, environment (for example, air quality, access to safely managed drinking water services, primary reliance on clean fuels), and accidents and violence. Due to the broad scope of the measurement framework, it builds on a variety of data sources, such as vital statistics, health-care records, preventive programmes, infectious disease surveillance mechanisms, and health interview surveys. Typical problems with implementing such comprehensive, intersectoral monitoring frameworks relate to the absence of adequate coordination mechanisms, insufficient overview of data availability across domains, and data gaps. In addition, it may prove challenging and time-consuming for large groups of experts from various domains to reach a consensus on which indicators to use. As regards reporting on intersectoral monitoring frameworks, a common obstacle is that there is limited evidence on the effectiveness of integrative interventions, which makes it difficult to provide solid recommendations on policy options (2). It may also prove challenging to write monitoring reports in such a way that a wide range of users with different backgrounds and from various domains can relate to them and find them meaningful and useful. Indicators related to international policies such as GPW13 and EPW are commonly used by policy- makers, both at national and subnational levels. However, other societal stakeholders may also use them to guide their strategies or substantiate advocacy efforts. Policy relevance of this module At the global level, GPW13 defines WHO’s strategy for the five-year period of 2019–2023 (3). It translates WHO’s mission to promote health, keep the world safe, and serve the vulnerable into the following targets (the so-called triple billion targets): y one billion more people to benefit from universal health coverage y one billion more people better protected from health emergencies y one billion more people enjoying better health and well-being. Annex 1. Rationale for the add-on modules ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 51 The GPW13 measurement framework aims to measure and monitor the strategy’s impacts on population health, both at national and international level. It measures progress at three levels (see Fig. A1): y 1. 46 outcome indicators y 2. the triple billion targets y 3. healthy life expectancy as an overarching measure of the health of populations. Fig. A1. The WHO Impact Measurement Framework The measurement framework is aligned with the SDGs to reduce the burden of data collection on countries and accelerate progress towards the SDGs. Of the 46 outcome indicators, 39 are SDG indicators. They have been supplemented with seven non-SDG indicators that address priorities identified by Member States, including antimicrobial resistance, polio, three additional indicators for NCDs (hypertension, obesity, and trans fat policy), and two emergencies-related indicators. The triple billion indices use a subset of 46 outcome indicators. The EPW builds on GPW13 and sets the priorities for the WHO European Region for the period 2020–2025 (4). At the time of writing, a measurement framework for the EPW that is aligned with the frameworks for GPW13, the SDGs and the Joint Monitoring Framework (5) is under development. Thus, by applying the HIS assessment add-on module on health information for GPW13, countries will not only strengthen their ability to measure progress towards the targets of GPW13, but also enhance their monitoring and reporting capacities for the SDGs and EPW. Healthy life expectancy 46 outcome indicators Triple billion targets Universal health coverage Health emergencies protection Healthier populations 1 2 3 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT52 Structure and content of this module GPW13 and EPW apply a flexible programmatic indicator approach. Not every country will track every indicator. In line with this approach and the overall aim of the WHO support tool to strengthen national HIS, the main aim of the add-on module on health information for GPW13 and EPW is to assess the ability of countries to monitor progress towards their national priority GPW13 milestones, using the related outcome indicators. The GPW13 and EPW add-on module is based on the structure of the core module of WHO’s support tool to strengthen HIS, but has been adapted for the specific purpose of assessing the countries’ ability to comply with the GPW13 results framework and the EPW measurement framework. However, the basic components (data collection, analysis, health reporting, knowledge translation, and governance and resources) can still be discerned. As the GPW13 framework and the EPW measurement framework are largely aligned with the SDGs, the add-on module first looks at the overarching organization and functioning of SDG monitoring and reporting in countries. After that, it addresses availability, usability, and actual use of data for the GPW13 and EPW indicators. How can the outcomes of this module be used to improve health information? The GPW13 impact framework and the EPW measurement framework are broad, covering various aspects of public health and health care, and building on a variety of data sources. Therefore, it is likely that interlinkages can be created between the outcomes of the add-on module on health information for GPW13 and EPW and the outcomes of the overall assessment of the wider HIS based on the core module of the WHO support tool to strengthen HISs. For example, one of the findings of the add-on module may be that there is no adequate intersectoral coordination mechanism, while one of the findings of the overall assessment may be that coordination of the wider HIS needs improvement. It would then be insufficient to independently develop two separate coordination mechanisms, without giving thought to how these mechanisms could collaborate or even merge. Alternatively, one of the findings of the add-on module may be that there are insufficiencies in the process of data delivery to WHO and other international organizations. The recommendation would then be to tackle these insufficiencies, not only related to data delivery for the GPW13 and EPW indicators, but also for other indicators that are to be provided by the country to international organizations, as there may exist common underlying problems that do not only affect the data deliveries for GPW13 and EPW, but other international data deliveries as well. The add-on module on health information for GPW13 and EPW will contribute to the strengthening of national capacities for intersectoral monitoring and reporting. This will not only be beneficial for implementing GPW13 and EPW, but also for other health policies with intersectoral components or intersectoral policies with health components, such as policies aimed at reducing risk factors for NCDs that include interventions in the physical and social environment, or policies aimed at reducing the health impacts of climate change. Typically, the ministry of health has a leading role in the HIS assessments carried out using the WHO support tool to strengthen HIS and the follow-up of the assessment outcomes. For follow-up of the outcomes of the add-on module on health information for GPW13 and EPW, it would also be important to include other relevant ministries. ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 53 References 1. Thirteenth General Programme of Work (GPW13). Methods for impact measurement. Geneva: World Health Organization; 2020 (https://apps.who.int/iris/handle/10665/341371, accessed 29 June 2021). 2. Verschuuren M, van Oers H. Population health monitoring: an essential public health field in motion. Bundesgesundheitsbl. 2020;63(9):1134–42. doi: 10.1007/s00103-020-03205-9. 3. Thirteenth General Programme of Work 2019−2023 [website]. Geneva: World Health Organization; 2019 (https://www.who.int/about/what-we-do/thirteenth-general-programme-of- work-2019---2023, accessed 29 June 2021). 4. European Programme of Work, 2020–2025: united action for better health in Europe. Copenhagen: WHO Regional Office for Europe; 2020 (EUR/RC70/11 Rev.4; https://apps.who.int/ iris/handle/10665/339209, accessed 29 June 2021). 5. Briefing note on the expert group deliberations and recommended common set of indicators for a joint monitoring framework In: 68th session of the Regional Committee for Europe. Copenhagen: WHO Regional Office for Europe; 2018 (https://apps.who.int/iris/ handle/10665/338926, accessed 29 June 2021). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT54 Infectious disease surveillance What is the role of infectious disease surveillance in the wider health information system? Infectious disease surveillance is an important epidemiological tool for monitoring the health of a population. Infectious disease surveillance has three main goals: to describe the burden and epidemiology of a disease, to monitor trends, and to identify outbreaks and unusual events of public health importance (1,2). Infectious disease surveillance can be classified, based on its features, as passive or active. Passive surveillance gathers infectious disease data from potential reporting health-care facilities and laboratories. Health authorities do not stimulate reporting and do not provide feedback to individual health workers. An example of passive surveillance is the routine reporting of notifiable diseases. In contrast, active surveillance requires staff to engage actively in surveillance and take action to receive infectious disease notifications. For instance, active investigation of individual cases of measles becomes necessary when the aim is to eliminate the disease. Sometimes, when collection of high-quality data is needed to monitor a particularly important disease, sentinel surveillance is incorporated into health-care-based surveillance systems. Diseases under sentinel surveillance will depend on national or subnational disease priorities. Examples of diseases monitored by this type of surveillance include severe acute respiratory infections, bronchiolitis, meningitis, and severe diarrhoeal disease. The three types of surveillance, that is, passive, active and sentinel surveillance, involve systematic collection, monitoring, analysis and interpretation of structured data produced by a number of well- identified, mostly health-based, formal sources. The traditional surveillance systems which report standardized and structured infectious disease information to public health officials are known as indicator-based surveillance (3). Laboratories, general practitioners, and hospitals are the most common sources of information for indicator-based surveillance systems. Laboratory data sources can be classified by geographical coverage, administrative status (public, private) or type of population covered (human, animal, environmental). General practitioners in primary care, based in a private or public health-care facility or a community health service, participate in indicator-based surveillance systems by notifying cases of infectious diseases, traditionally through mandatory notification (or on a voluntary basis). Hospitals report data on hospital morbidity and mortality, health-care-associated infections and antimicrobial resistance. Laboratories for microbiological analysis represent a major source of data by directly reporting to a network dedicated to surveillance activities or indirectly reporting through health-care facilities. The integration of both clinical and laboratory reporting requires considerable laboratory capacity and advanced IT infrastructure (4). Event-based epidemiological surveillance is the second of the two main types of surveillance used to identify and track infectious diseases and other public health events. As opposed to indicator- based surveillance, event-based surveillance relies on sources of information beyond traditional health system sources such as reports, stories, rumours, and other information about health events that could pose a serious risk to public health. Such information may be described as unstructured because the information obtained is non-standardized or subjective. Other sources of event-based data include syndromic surveillance; mortality monitoring or veterinary, environmental and food safety data sources. Event-based surveillance and indicator-based surveillance complement one another. ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 55 In consideration of the growth in international travel and trade, and the emergence or re-emergence of international disease threats and other public health risks, in 2005 the World Health Assembly adopted the International Health Regulations (IHR), an instrument of international law that is legally binding on 196 countries. The need for building capacity to develop monitoring processes, timely detect potential threats to public health and maintain core capacities for infectious disease surveillance, as required by the IHR, is accelerating the adoption and meaningful use of efficient information systems for infectious disease surveillance. Fortunately, increased global health concerns have also been accompanied by a rapid advancement of health-care technologies. Health-care stakeholders have been intensifying their digital transformation efforts as a result of these challenges. The use of information systems for infectious disease surveillance, which deal with the management, analysis and presentation of large amounts of surveillance data, offers the potential for a truly integrated epidemiological surveillance based on a communications infrastructure, data standardization and policies on data access and sharing. These approaches have proven to be the most efficient way to meet international standards, as they can detect outbreaks much earlier and even identify public health hazards that would have previously gone unnoticed. Additionally, a geographic information system (GIS) provides an excellent means of collecting and managing epidemiological surveillance and programmatic information. When mapped together, such information creates a powerful tool not only for monitoring of surveillance results but also for operational planning and targeting of interventions and resources to areas and communities in need. In some countries, the use of GISs within the health sector may still be relatively new or even non-existent. Information systems for infectious disease surveillance can function with varying degrees of success. Sometimes numerous information systems focusing on a specific disease or programme are deployed under different initiatives, working in isolation, without proper mechanisms for sharing and generating valid surveillance information for public health action. Moreover, the systems might gather information at the first level of health-care facilities, but there is no systematic and continuous data reporting and data sharing. Another challenge involves ensuring quality and effectiveness of infectious disease surveillance in a decentralized environment. Surveillance systems might vary geographically and level-to-level in terms of functionality, data collection, data format, reporting and record-keeping methods. A centralized infectious disease database might not be available, and data are dispersed among different levels and institutions with limited standardization and capacity for monitoring data quality. In terms of data ownership, surveillance systems tend to be the responsibility of the government. Infectious disease surveillance data are often collected, and as long as it complies with data protection legislation, this practice is ethically and legally justified as an element of the government’s responsibility to protect the public’s health. Privacy rules apply to the disclosure of certain health information. The use of event-based surveillance systems data raises new ethical and legal challenges that might affect the ability to implement this type of surveillance. These issues need to be addressed as more informal surveillance systems become integrated into conventional health information systems (4). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT56 Policy relevance Infectious disease surveillance provides policy-makers with information about the burden and epidemiology of a disease, and gives them situational awareness during an outbreak or a public health emergency, allowing them to make informed decisions. The IHR requires countries to establish and maintain core capacities for infectious disease surveillance of public health threats of national and international importance. The IHR calls upon Member States to develop and enhance their capacities for surveillance, reporting, notification, verification, response and collaboration (5). These requirements create the need for new technologies to reinforce the capacity of traditional infectious disease surveillance systems. The early warning functions of surveillance are fundamental for national, regional and global health security. Therefore, information systems for infectious disease surveillance should include early warning surveillance data and laboratory findings (6) in order to assess public health events and report to WHO those that may constitute a public health emergency of international concern within 24 hours, as required by the IHR (7). The add-on module for infectious diseases includes items to assess specific components related to IHR procedures regarding public health emergencies of international concern. Additionally, questions on certain specific infectious disease indicators that are needed to track progress towards the implementation of the SDGs across the WHO European Region have also been incorporated. Structure and content of this module This add-on module aims at obtaining a specific overview of the functioning of the infectious disease surveillance and its integration in the wider health information system designed to manage a wide array of data. The assessment worksheet for the add-on module on infectious diseases is based on the structure of the core module of this support tool and includes its basic components. The first part of the assessment will provide an overview of the infectious disease surveillance systems in place, and the universal functions of data collection, data management, analysis and reporting that are common regardless of the disease under surveillance (5). The infectious disease add-on module also includes questions to assess health reporting, knowledge translation and support functions such as governance, supervision, training, evaluation, communications and other resources. The early warning functions of surveillance are fundamental to national, regional and global health security. An HIS should include early warning surveillance data and laboratory findings (6,8). A set of indicators (9,10) has been included to assess for compliance with IHR with regards to implementing effective surveillance to detect public health threats of national and international importance. The items included in the add-on module have taken into consideration a combination of the following assessment tools and guidance documents: ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 57 Box 1. Guidance documents y Protocol for the Assessment of National Communicable Disease Surveillance and Response Systems. Guidelines for Assessment Teams. Geneva: World Health Organization; 2001. y Updated Guidelines for Evaluating Public Health Surveillance Systems. Morbidity and Mortality Weekly Report. Atlanta: Centers for Disease Control and Prevention. July 27, 2001 / 50(RR13);1–35. y Communicable disease surveillance and response systems - Guide to monitoring and evaluating. Geneva: World Health Organization; 2006. y A guide to establishing event-based surveillance. Manila: WHO Regional Office for the Western Pacific; 2008. y Early detection, assessment and response to acute public health events: implementation of early warning and response with a focus on event-based surveillance. Geneva: World Health Organization; 2014. y Support tool to assess health information systems and develop and strengthen health information strategies. Copenhagen: WHO Regional Office for Europe; 2015. y Data quality monitoring and surveillance system evaluation: a handbook of methods and applications. Stockholm: European Centre for Disease Prevention and Control; 2014. y Joint external evaluation tool: International Health Regulations (2005). Geneva: World Health Organization; 2016. y Briefing note on the expert group deliberations and recommended common set of indicators for a joint monitoring framework. 68th session of the Regional Committee for Europe. WHO Regional Office for Europe; 2018. The surveillance attributes related to the quality of the infectious disease surveillance system (6,8,11,12) assessed are shown in Box 2. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT58 Box 2. Surveillance attributes for assessment y Stability – refers to the ability of a surveillance system to collect, manage, and provide data without failure and to be operational when needed. y Simplicity – refers to the system’s structure and ease of operation. y Timeliness – the period between data collection and its availability to a higher level, or its publication; y Periodicity – the frequency with which an indicator is quantified; y Representativeness – the extent to which data adequately represent the population and relevant subpopulations; y Data quality –> Completeness – Completeness has two dimensions: external, which refers to completeness of surveillance data or how the system reflects the true number of cases (that is, percentage of reporting sites submitting weekly or monthly surveillance reports on time to the next higher level), and internal, which refers to whether there are missing data fields or data items. A key data quality indicator of passive surveillance will be external completeness because it relies on sources to take the initiative to report data. An active surveillance system is based on the public health officials’ initiative to contact relevant sources to report data. The indicator in this case will be internal completeness. y Data quality –> Validity – Internal validity is the ability to capture errors within information (that is, coding errors), whereas external validity relates to whether data are correct when compared to an external database (that is, comparability of data entered into computer databases to original paper-based records). y Usefulness – Usefulness implies that the surveillance system is used for action. Assessing usefulness consists in making an inventory of actions that have been taken as a result of information generated by the infectious disease surveillance system and that have contributed to infectious disease control and prevention. How can the outcomes of this module be used to improve health information? This add-on module can be used to assess HIS capacities for all infectious diseases included in a surveillance system or only for specific diseases. The assessment group may wish to adjust the assessment tool to accommodate its situation, applying the same set of assessment criteria. Nevertheless, it is recommended that during the assessment of the infectious disease surveillance component, all diseases under surveillance should be examined to determine whether these are still a priority. Otherwise, countries may lack surveillance in critical areas. The outcomes of this module can assist decision-makers with identifying possible challenges that may adversely affect the capacity of the infectious disease surveillance system to meet its goals, namely, to describe disease epidemiology, to monitor trends and to detect acute public health events. Poor infectious disease surveillance capacity and uncoordinated data management result in a dearth of infectious disease information which might be critical to controlling an epidemic and protecting public health. Appropriately organized and coordinated infectious disease surveillance systems and attention to data has correlated with overall success at controlling COVID-19 outbreaks in some countries. ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 59 Existing infectious disease surveillance systems should be integrated within health information systems by building on principles of public health informatics, and agreements on data access, with the ultimate aim of optimizing the usefulness of surveillance information and its utilization for action. References 1. Aziz HA. A review of the role of public health informatics in healthcare. J Taibah Univ Medical Sci. 2017;12(1):78–81. doi:10.1016/j.jtumed.2016.08.011. 2. Murray J, Cohen AL. Infectious disease surveillance. Int Encycl Public Health. 2017:222–9. doi: 10.1016/B978-0-12-803678-5.00517-8. 3. Early detection, assessment and response to acute public health events: implementation of early warning and response with a focus on event-based surveillance. Geneva: World Health Organization; 2014 (https://apps.who.int/iris/bitstream/handle/10665/112667/WHO_HSE_GCR_ LYO_2014.4_eng.pdf?sequence=1, accessed 29 June 2021). 4. Edelstein M, Lee LM, Herten-Crabb A, Heymann DL, Harper DR. Strengthening global public health surveillance through data and benefit sharing. Emerg Infect Dis. 2018;24(7):1324–30. doi: 10.3201/eid2407.151830. 5. Protocol for the assessment of national communicable disease surveillance and response systems. Geneva: World Health Organization; 2001 (https://apps.who.int/iris/ handle/10665/66787, accessed 29 June 2021). 6. Support tool to assess health information systems and develop and strengthen health information strategies. Copenhagen: WHO Regional Office for Europe; 2015 (https://apps.who.int/ iris/handle/10665/172761, accessed 29 June 2021). 7. International Health Regulations (2005), second edition. Geneva: World Health Organization; 2008 (https://apps.who.int/iris/handle/10665/43883, accessed 29 June 2021). 8. Communicable disease surveillance and response systems: guide to monitoring and evaluating. Geneva: World Health Organization; 2006 (https://apps.who.int/iris/ handle/10665/69331, accessed 29 June 2021). 9. Joint external evaluation tool: International Health Regulations (2005). Geneva: World Health Organization; 2016 (https://apps.who.int/iris/bitstream/handle/10665/204368/9789241510172_ eng.pdf?sequence=1, accessed 29 June 2021). 10. A guide to establishing event-based surveillance. Manila: WHO Regional Office for the Western Pacific; 2008 (https://iris.wpro.who.int/bitstream/handle/10665.1/10421/9789290613213_eng.pdf, accessed 29 June 2021). 11. Updated guidelines for evaluating public health surveillance systems. Atlanta (GA): Centers for Disease Control and Prevention Morbidity and Mortality Weekly Report; 2001:1–35 (50(RR13); https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5013a1.htm, accessed 29 June 2021). 12. Data quality monitoring and surveillance system evaluation: a handbook of methods and applications. Brussels: European Centre for Disease Prevention and Control; 2014 (https://data. europa.eu/doi/10.2900/35329, accessed 29 June 2021). Noncommunicable disease monitoring What is the role of health information on NCDs in the wider health information system? SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT60 NCDs represent a major public health challenge in the WHO European Region, where a significant fraction of all health losses (morbidity, disability and mortality) is caused by this group of diseases (1). NCDs comprise a large group of diverse diseases, but five major groups, that is, cardiovascular diseases, cancer, diabetes, chronic respiratory diseases and mental disorders, are responsible for the largest fraction of mortality, morbidity and disability (2). Furthermore, even though they may have multiple causes, the burden of the main NCDs is associated with eight groups of common behavioural and metabolic risk factors, that is, tobacco use, harmful alcohol consumption, unhealthy diet, physical inactivity, overweight and obesity, raised blood pressure, raised blood glucose and raised blood cholesterol, as well as deprived physical and socioeconomic environments and poor quality of health-care services (3). Given the synergies between risk factors, it is important to assess NCDs in an integrated manner. However, in general, country health information for NCDs has been collected with systems for specific diseases and risk factors independently and often in isolation, making it difficult to use and compile comprehensive information for understanding the complex causes and pathways of NCDs in specific population groups. Adequately addressing the complexity of NCD-related challenges requires comprehensive and integrated health information from different processes, from both population and individual levels, for the purposes of health situation and trend analyses (public health surveillance), health policy planning, target setting and monitoring (governance), and evaluation of the effectiveness and quality of NCD-related policies, programmes and health interventions (4). Due to their chronic and insidious nature, NCDs may be more likely to be diagnosed and monitored at the point of contact with the health services, or later by the civil registration at the time of recording of death. This passive approach leaves little or no opportunity to monitor underlying risk factors across the life-course and collect information required to prevent or modify pre-clinical stages of the NCDs, so monitoring and surveillance of risk factors at population level are needed as a foundation for complementing NCD public health surveillance. This also depends on opportunities and use of primary health-care services for early assessment of individuals, or on surveys of risk factors and medical history from nationally representative populations. In addition, assessment of the overall quality of NCD health care requires continuity of individual monitoring of disease and risk factors (enabling assessment of specific NCD diagnosis and treatment cascades) and adequacy or adjustments of treatment/procedures, thus depending on appropriate health records that can be accessed at different levels of the health services. Health promotion and behavioural modifications for reducing NCDs and their risk factors are well-known cost-effective policy interventions, which should, therefore, be included in comprehensive universal health coverage, and levels and inequities of their coverage should be monitored. Furthermore, health system response, including NCD data assessment related to disease care cascade management, is essential for assessing universal health coverage. This includes information on diagnosis, treatment and management coverage as well as outcome monitoring. Access and adherence to treatment and availability and needs of medicines, technologies and procedures are other key aspects of NCD performance management that also provide evidence for coverage and effectiveness of interventions. There are many information subsystems collecting and compiling needed data for NCD planning and monitoring, but there are also important fragmentations, partly due to the use of different standards and tools and lack of unique personal identifiers, that preclude their integration. Although European countries have developed many health information subsystems, including mortality and disease registries, hospital morbidity records, results from laboratory, pharmacy records, and disease and risk factors surveys, that are required for integrated NCD monitoring, management and evaluation, they often lack the necessary quality and accessibility to allow a comprehensive ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 61 approach to NCD health information. Today, the availability of and access to newer methods, tools and approaches, including those based on ICTs, provide opportunities for addressing some of these limitations (5). Policy relevance By 2017, the main NCD causes accounted for 68% of all premature deaths (those in the 30–69 years of age group) in WHO European Member States. This also meant that, according to the WHO indicator of the risk of premature death, a 30-year-old would have a 16% chance of dying from a main NCD before reaching 70 years of age, with the figures for males (21%) being twice as high as those for females (11%) (6). Increasing population ageing, changes in lifestyle, health-related behaviours, metabolic factors and harmful effects of changing environmental and socioeconomic conditions, are all contributing to the NCD epidemic worldwide and, in particular, among the populations of some Member States in the Region, where the burden of disease is among the largest globally (7). The impact of NCDs goes beyond health effects (for example, morbidity, disability and mortality and the burden on health-care systems), but it has also been identified as a major cause of social and economic loss. Furthermore, NCDs and their related risk factors also contribute to the occurrence of other health problems, such as injuries (for example, alcohol consumption and motor vehicle accidents (8)) and violence (for example, suicides), or, in turn, are affected by other health problems such as certain infectious diseases (for example, human papilloma virus or hepatitis B and development of cervical and liver cancers). More recently, the COVID-19 pandemic has become a painful reminder of the interlinkages between NCDs and communicable diseases, as the severity of COVID-19, the risk of hospitalization and poorer health outcomes are exacerbated by the presence of NCDs and their risk factors, and failures to prevent and control NCDs and their risk factors, such as tobacco use, alcohol consumption, obesity, chronic respiratory disease, diabetes, cancer and hypertension, are resulting in increased morbidity and mortality (9). In addition, there are indications that the prevalence of NCDs and their risk factors increased during the pandemic. Given this complex multifactorial nature and pathways of NCDs, the response of the health system to their monitoring, prevention and control also demands intersectoral involvement and actions. Recognizing the above challenges and the slow progress made in the prevention and control of NCDs over the past two decades, WHO and its Member States have agreed on several policy frameworks to increase action for reducing the causes and consequences of NCDs, through well-known and proven public health measures and strategies, aiming to achieve several health targets. Such targets and their frameworks have been defined in the WHO Global Action Plan for the Prevention and Control of Noncommunicable Diseases and the United Nations SDGs, to be reached by the years 2025 and 2030, respectively (10,11). WHO and the United Nations were asked to provide regular assessments of the progress towards achieving the targets. Thus, Member States have approved 25 outcome indicators of the Global Monitoring Framework for NCDs to monitor NCD prevention and control and 19 complementary Progress Monitor indicators from the NCD Country Capacity Surveys to monitor NCD policies and the capacity to address them (12,13). This information also contributes and provides evidence to the EPW (14), in particular, to improving health governance, developing intelligence on inequalities of health and well-being and improving health monitoring with the use of effective e-health tools (for example, EHR, mobile phones, social media and big data) (15). In order to collect and generate comparable country data and indicators for monitoring the situation and trends at the WHO European Region level, the use of information standards (for example, indicators definitions, measurement methods and metadata) is essential for reporting on the international indicators agreed by WHO and the United Nations (16). This also SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT62 increases comparability across settings and geographies, and facilitates comparative studies, thus increasing the usability of routinely collected data for monitoring population health and universal health coverage. Therefore, in the context of the NCD burden and complexity, the assessment of the different health information subsystems required for national NCD-integrated monitoring, surveillance, evaluation and management is highly relevant and necessary for improving the interoperability of seamless collection and use of data in European countries. Organization and content of the module The add-on module on NCD health information follows a similar structure of the core/basic module, that is, data collection, analysis, knowledge sharing and governance sections, which include aspects of inputs, processes and outputs of each topic. For the data collection section, this involves determining the availability of comprehensive NCD health information and its different sources of data, according to their hierarchy and properties (for example, administrative vs survey- based collection, multiple vs single data elements, with broad vs subpopulation (for example, public sector) coverage, annual vs periodic frequency of collection, and validity, coverage, completeness and quality measured vs not measured, data series revised/adjusted yearly vs not, easy vs difficult/ limited accessibility and usability, usage or not of diagnoses/definitions according to international standards, include or not include public and private information) (17). Several sources may be the same as those addressed in the core module, but the interest is in another type of issue or NCD data-specific content. Specific information involves mortality and population statistics (NCD deaths by cause), disease registries (by NCD cause), surveys of disease and risk factors by NCD cause and type, hospital discharges/morbidity (by NCD cause and procedure), EHR (by NCD cause and treatment/procedures used), NCD- and risk factor-specific health system policies, resources and materials (for example, availability of NCD-related medicines and equipment). For the analysis section, the focus is on definition, measurement and availability of core NCD health indicators (for example, NCD morbidity, mortality, prevalence of risk factors, determining multiple causes and risk factors, cardiometabolic risk assessments, access to information on diagnoses, counselling or treatment by health professionals, availability of and access to guidelines for diagnosis and treatment of diseases, and referral information, drug treatments and medical equipment, capacity to establish disease cascade paths, capacity to determine health inequalities among age/gender/socioeconomic population subgroups) (18). The section on health reporting mainly addresses the access, capacity and use of data for designing, conducting and preparing reports that provide information and evidence, and are strategic for policy-making. These include health situation and trend analysis, health profiles, socially determined health inequality assessments, evaluation of progress and effectiveness of NCD plans, programmes and interventions. It also involves the use of reports for other aspects of policy planning, such as development and measurement of NCD targets and their indicators, and assessment of trends. Finally, the section on governance involves the legal aspects, structures, resources and funding aspects for sustainable monitoring, surveillance and evaluation of NCDs, and processes to guarantee data and information access for stakeholders and the public at large. All of these are placed within the context of a national plan or policy with appropriate, measurable and feasible ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 63 targets, a process of transparency and accountability with an open data policy, including access to microdata for research purposes. How can the outcomes of this module be used to improve health information? A key aspect of NCD information is the need to integrate information from different sources. This may involve identifying prerequisites for linkage and integration of data sources, developing a map of essential information for NCDs, establishing a set of agreed core health indicators (including definitions and metadata) and ensuring interoperability of information systems (with harmonization of technical standards and development and agreement of mechanisms of governance (19)). Alongside other health processes, it is important to develop analytical capacity and establish tools for setting and monitoring NCD-related health targets and evaluation processes. In order to comprehensively report on universal health coverage, including unmet health-care needs, in general, there is a need for more objective health measures of risk factors (for example, measured vs self-reported weight and height or hypertension status, or nicotine measurement vs self-reported tobacco use prevalence) and further stimulating the implementation of health examination surveys in the region, taking advantage of the rich history and long list of existing population surveys, and using innovative approaches (for example, subsampling or implementing module subunits in different years in a period). Overall, there also is a need to further develop knowledge and capacity on the use of e-health for NCD monitoring, surveillance and evaluation, including development and use of innovative tools such as application of artificial intelligence, as well as identification, assessment of the usefulness and use of different sources of big data, including real-life and digital trails, for complementing public surveillance and monitoring beyond traditional means. Examples could include monitoring and preventing digital marketing of unhealthy products to children (20), and monitoring of population mobility and risk of diseases (based on mobile phone signals in a COVID-19 tracing application (21)). References 1. Noncommunicable diseases country profiles 2018. Geneva: World Health Organization; 2018 (https://www.who.int/publications/i/item/ncd-country-profiles-2018, accessed 29 June 2021). 2. Bennett JE, Stevens GA, Mathers CD, Bonita R, Rehm J, Kruk ME et al. NCD Countdown 2030: worldwide trends in non-communicable disease mortality and progress towards Sustainable Development Goal target 3.4. Lancet. 2018;392(10152):1072–88. doi: 10.1016/ S0140-6736(18)31992-5. 3. Livingston G, Huntley, J, Sommerlad A, Ames D, Ballard C, Banerjee S et al. Dementia prevention, intervention, and care: 2020 report of the Lancet Commission. Lancet. 2020; 396(10248):413–46. doi: 10.1016/S0140-6736(20)30367-6. 4. Jakab M, Farrington J, Borgermans L, Mantingh F, editors. Health systems respond to NCDs: time for ambition. Copenhagen: WHO Regional Office for Europe; 2018 (https://www.euro.who. int/__data/assets/pdf_file/0009/380997/Book-NCD-HS.pdf, accessed 29 June 2021). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT64 5. Ballicer RD, Oroz-Luengo M, Cohen-Stavi C, Loyola E, Mantingh F, Romanoff L et al. Using big data for non-communicable disease surveillance. Lancet Diabetes Endocrinol. 2018;6(8):585–98. doi: 10.1016/S2213-8587(17)30372-8. 6. Global health estimates 2016. Geneva: World Health Organization; 2018 (http://www.who.int/ healthinfo/global_burden_disease/en/, accessed 29 June 2021). 7. NCD Countdown 2030 Collaborators. NCD Countdown 2030: pathways to achieving Sustainable Development Goal target 3.4. Lancet. 2020;396(10255):918–34. doi: 10.1016/ S0140-6736(20)31761-X. 8. Part 2: alcohol-attributable burden of disease and injury in EU+ countries. In: Status report on alcohol consumption, harm and policy responses in 30 European countries 2019. Copenhagen: WHO Regional Office for Europe; 2020:9–24 (https://www.euro.who.int/__data/assets/pdf_ file/0019/411418/Alcohol-consumption-harm-policy-responses-30-European-countries-2019.pdf, accessed 29 June 2021). 9. Editorial: COVID-19: a new lens for non-communicable diseases. Lancet. 2020;396(10252):649. doi: 10.1016/S0140-6736(20)31856-0. 10. Global action plan for the prevention and control of noncommunicable diseases 2013–2020. Geneva: World Health Organization; 2013 (https://apps.who.int/iris/handle/10665/94384, accessed 29 June 2021). 11. Take action for the Sustainable Development Goals. New York: United Nations; 2020 (https:// www.un.org/sustainabledevelopment/sustainable-development-goals/, accessed 29 June 2021). 12. NCD global monitoring framework. Geneva: World Health Organization; 2013 (https://www. who.int/teams/ncds/surveillance/monitoring-capacity/gmf, accessed 29 June 2021). 13. Assessing national capacity for the prevention and control of noncommunicable diseases: report of the 2019 global survey. Geneva: World Health Organization; 2020 (https://apps.who.int/ iris/handle/10665/331452, accessed 29 June 2021). 14. European programme of work, 2020–2025: united action for better health in Europe: virtual session 14–15 September 2020. Copenhagen: WHO Regional Office for Europe; 2020 (https://apps.who.int/iris/bitstream/handle/10665/333908/70wd11e-rev4-EPW-200673. pdf?sequence=1&isAllowed=y, accessed 29 June 2021). 15. Gall C, Suzuki E. Big data: a new dawn for public health? In: Health in the 21st century: putting data to work for stronger health systems Paris: Organisation for Economic Co-operation and Development; 2019 (https://www.oecd-ilibrary.org/sites/e3b23f8e-en/1/2/5/index.html?itemId=/ content/publication/e3b23f8e-en&_csp_=c23ceb96c5ce2feb951a759a657e4b9f&itemIGO=oecd &itemContentType=book, accessed 29 June 2021). 16. Health Metrics Network. Framework and standards for country health information systems. Geneva: World Health Organization; 2008 (https://apps.who.int/iris/handle/10665/43872, accessed 29 June 2021). ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 65 17. SCORE for health data technical package: global report on health data systems and capacity, 2020. Geneva: World Health Organization; 2021 (https://cdn.who.int/media/docs/default-source/ world-health-data-platform/score/who_2021-01-31_global-report-score_tb_v2.pdf?sfvrsn=cf86a 4fb_3&download=true, accessed 29 June 2021). 18. STEPwise approach to noncommunicable disease risk factor surveillance (STEPS). Geneva: World Health Organization; 2017 (http://www.who.int/ncds/surveillance/steps/riskfactor/en/, accessed 29 June 2021). 19. Vayena E. Value from health data: European opportunity to catalyse progress in digital health. Lancet. 2021;397:652–3. doi: 10.1016/S0140-6736(21)00203-8. 20. Monitoring and restricting digital marketing of unhealthy products to children and adolescents. Report based on the expert meeting on monitoring of digital marketing of unhealthy products to children and adolescents, Moscow, Russian Federation, June 2018. Copenhagen: WHO Regional Office for Europe; 2019 (https://www.euro.who.int/__data/assets/ pdf_file/0008/396764/Online-version_Digital-Mktg_March2019.pdf, accessed 29 June 2021). 21. Murray CJL, Alamro NMS, Hwang H, Lee U. Digital public health and COVID-19. Lancet Public Health. 2020;5:e469–70. doi: 10.1016/S2468-2667(20)30187-0. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT66 Human resources for health What is the role of HRH in the wider HIS? This module covers health information for the health workforce. The HRH workforce is responsible for organizing and delivering health services to meet the needs of the population and achieving the goals and objectives of national health policy. While there is a direct link between workforce numbers and population health outcomes, almost all countries are constrained, in one way or another, from achieving the ideal workforce in terms of numbers, skills and distribution. Understanding the health workforce, stocks, distribution and flows is key to formulating a health strategy and plan. The data sources discussed in the core module will be explored in greater detail from the perspective of HRH. In particular, the key data sources include: y population-based census and surveys (HIS assessment sheet item Data sources_1) y population health needs (HIS assessment sheet item Data sources_2, 3, 4 and 5) y active health workforce (HIS assessment sheet item Data sources_8) y health education and training (HIS assessment sheet item Data sources_8) y health financing (HIS assessment sheet item Data sources_9) y health service utilization (HIS assessment sheet item Data sources_3). Specific HRH data sources include: y country-level labour and employment surveys and statistics to understand the overall numbers in the health and non-health sectors, health labour market flows, average wages, and competition for staff; y health employer surveys and statistics to understand the health workforce, for example, the distribution between the public and private sectors, length of employment, pay rates, and working conditions; y the health education sector to understand the health workforce training and supply situation; and y health professional organizations or bodies to understand the pressures on individual workforce groups, such as undersupply, working conditions or scope of practice. Common problems and challenges are: y difficulty of obtaining data from the private sector, for example, ownership issues, dual practice (working across both sectors) leading to double counting, data that are not validated leading to quality issues; y lack of data on health workforce education and training, for example, data on the training of individual workforces may be held separately by the respective training institutions, and not aggregated; y lack of data on population health needs (which drive the requirements for the health workforce) and how the existing workforce is meeting these needs. This is often simply not collected; ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 67 y lack of data on how the population, its composition and distribution, and health needs may change in the future; and y lack of focus on critical data quality issues. In particular, in projecting workforce supply and future health requirements, small errors in some parameters can make a big difference in the final result (this is called data sensitivity). If these parameters are of low quality, then the calculated numbers will have high uncertainty. Such parameters are often not identified. HRH information is typically used by senior policy-makers and health workforce planners. However, good HRH information has wide relevance across all sectors of government. Policy relevance of this module Health workers play a critical role in health systems by ensuring sustainability, resilience and the delivery of high-quality services. An effective, well-motivated, appropriately skilled and well- managed health workforce is at the very heart of this goal. This is evidenced by: y WHO’s GPW13, which defines the strategy for 2019–2023, focusing on measurable impacts on people’s health at the country level (1); y the EPW, which sets out a vision of how the WHO Regional Office for Europe can support countries in the Region in better meeting citizens’ expectations about health (2); y the United Nations SDGs (3) and the objectives of universal health coverage (4), which are the main goals in GPW13 and the EPW. The WHO report on health employment and economic growth (5) contains several case studies that emphasize the positive impact of health policies on economic growth through six key pathways: health system, economic output, social protection, social cohesion, innovation and diversification, and health security. Structure and content of this module For simplicity, the HRH module follows the structure of the core module. The general approach and, in particular, Data Collection and Analysis/Contextualization are based on a large body of work from the Centre for Workforce Intelligence, which was the British authority on workforce planning for the period from 2010 to 2016, providing advice and information to the health and social care system (6). Effective health human resources planning requires an understanding of the situation today and how it may unfold in the future in order to anticipate changes in population health needs and the delivery of health services. Having a sustainable and appropriately skilled workforce will help improve the quality of care and reduce swings of undersupply or oversupply. To do this, the HRH information in the HIS needs to support understanding of the situation today regarding: y population age and gender y health needs of the population y active health workforce stock that is delivering services to meet population health needs SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT68 y health education and training of the future workforce y effort spent by the health workforce on meeting health needs. Knowledge is also needed of: y health finances including the cost of health education, and expenditure on the workforce; y health labour market including the employment situation, the flows into and out of the health workforce, and the sectors competing for staff; and y health working conditions. The above areas are addressed in the Data collection assessment. The next stage involves thinking about the future using the following analysis. y Determine the gap between the effort that is being exerted by each health workforce, and what is required to meet today’s population health needs. y Consider how these gaps (if any) have arisen, the pressures on the different workforce groups, and how these elements could change in the future. y Determine the country’s future population size and distribution. For example, the population could increase, the gender balance could change, it could become older, and the distribution between urban and rural regions could shift. All of this has implications for the kind of health services required, and changes in demand for these services compared to today. y Determine the future health needs of the population. For example, an ageing population could mean a greater number of people with long-term conditions. Although the future may look different than the situation today, the country should start by thinking about its expected future. This is the scenario where the current situation unfolds as an impartial observer would expect: plans in place today or scheduled for the future unfold as expected, and there are no sudden shifts or shocks. y Determine the future supply of each health workforce group, assuming that the workforce is structured in the same way it is today, training and workforce inflows and outflows continue at today’s rates, and service delivery remains unchanged. y Finally, the analysis should allow the country to estimate the gap or mismatch between future workforce supply and population health requirements, at individual workforce group level. This will allow the country to develop effective HRH policies, strategies and plans. It is important for the country to have a comprehensive set of HRH indicators to support policy development and implementation. For example, the National Health Workforce Accounts (7) will provide a large portion of the information required. However, additional information is also needed for effective policy-making and planning. The National Health Workforce Accounts does not address population age, gender and health needs; workforce effort to meet these needs; estimation of future changes; or projections of future supply to meet changing population requirements. Following this analysis, health reports should be produced to inform the policy, strategy and decision-making processes. In particular, these should draw attention to population health needs ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 69 and trends, the current and likely future pressures on the health workforce, and any gaps or mismatches in workforce numbers, skills or competences. It may be necessary at this point to return to data collection and analysis to look at specific workforce issues in greater detail. The knowledge translation section considers whether the information provided is understood, being used effectively and whether it meets the needs of decision-makers. This is important as the approach described considers how the future may unfold, in order for workforce plans to take account of changing population needs. Because decision-makers are often uncomfortable when faced with uncertainty, this approach should be presented in a way that can be understood and used effectively. Finally, a robust governance structure needs to be in place for the implementation of HRH indicators, and regulations of the health workforce and labour market to support effective policy- making and planning. How can the outcomes of this module be used to improve health information? This HRH module supports the development of an effective set of HRH indicators, and the information required to produce health workforce strategies, policies and plans. Much of the information required in an HRH information system is available but often fragmented and spread across many systems and databases. This module describes the core elements and how they are to be used to prioritize data collection. It is recommended as a starting point to: y work with the country to confirm the scope of HRH strategic planning and the key stakeholders, as discussed in Data collection 1; y develop an agreed set of HRH indicators, as discussed in Analysis 1; y identify the key data owners for the areas discussed in Data collection 2 to 9; and y develop metadata standards as discussed in Data collection 10. Following this, the analysis section may highlight areas where information is lacking or of poor quality. These issues should be prioritized, and processes should be put in place to improve health information and close any gaps. Additionally, the health reporting section may suggest further areas of improvement where there is a mismatch between the information needs of decision- makers, what is being provided to them, and how it is being used. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT70 References 1. Thirteenth General Programme of Work 2019–2023 [website]. Geneva: World Health Organization; 2019 (https://apps.who.int/iris/handle/10665/279451, accessed 29 June 2021). 2. European programme of work, 2020–2025: united action for better health in Europe. Copenhagen: WHO Regional Office for Europe; 2020 (EUR/RC70/11 Rev.4; https://apps.who.int/ iris/handle/10665/339209, accessed 29 June 2021). 3. Transforming our world: the 2030 Agenda for Sustainable Development. New York: United Nations; 2015 (https://sdgs.un.org/2030agenda, accessed 29 June 2021). 4. Global health and foreign policy: a healthier world through better nutrition. Resolution adopted by the General Assembly on 13 December 2018. New York: United Nations; 2019 (https:// digitallibrary.un.org/record/1660216?ln=en, accessed 29 June 2021). 5. Health employment and economic growth: an evidence base. Geneva: World Health Organization; 2017 (https://apps.who.int/iris/handle/10665/326411, accessed 29 June 2021). 6.Willis G, Cave S, Kunc M. Strategic workforce planning in healthcare: A multi-methodology approach. Eur J Oper Res. 2018;267(1):250–63. doi: 10.1016/j.ejor.2017.11.008. 7. National health workforce accounts (NHWA). Geneva: World Health Organization; 2017 (https:// apps.who.int/iris/handle/10665/259360, accessed 29 June 2021). ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 71 Health data governance What is the role of health data governance in the wider HIS? Public health systems and health-care service providers have become increasingly interested in using the best available evidence for public health policy-making as well as clinical and operational decisions. This has led to the design of health information management systems and the collection and evaluation of ever-larger data sets around all aspects of care delivery, performance and cost– benefit analysis (1). As health systems continue to adopt new technologies to enable new or improved approaches to public health and clinical care, the size of health data sets will continue to grow. New technologies and health information systems are set up to: y improve quality, safety, efficiency, and coordination of health-care services y engage patients and family in health-care decisions and priority setting y improve population health and reduce health disparities y maintain privacy and security of patient health information. It is important to look at the ultimate goal of health data governance. Data governance is an approach to managing data that allows organizations to balance the need to collect and secure information while maximizing value from that information (2). In this regard, it is the aim of data governance, broadly speaking, to strive for a concordance between the various issues at stake (3). Given these aims and ambitions, data governance can be defined as an organizational approach to data and information management that is formalized as a set of policies and procedures that encompass the full life cycle of data, from acquisition to use to disposal (4). This includes establishing decision-making authority, policies, procedures, and standards regarding data security and privacy protection, data inventories, content and records management, data quality control, data access, data security and risk management, data sharing and dissemination, as well as ongoing compliance monitoring of all of the above-mentioned activities. Policy relevance of the module Data governance as an approach, but also as a methodological toolkit, is increasingly important in public health and health information systems, as health data are nowadays viewed as a strategic asset. Like any other organizational asset (for example, people, machinery or processes), health data require ongoing monitoring and control. Data governance provides a formal structure for data management so that organizations and policy-makers can enhance the public health value of data (5). In line with the policy-making cycle, this module enables stakeholders to assess the maturity of the health data governance in the public health system and to conduct, or rather initiate, a gap analysis that enables public health systems to achieve a higher data governance maturity level (6). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT72 The module specifically targets public health professionals who, openly or even unknowingly, serve as data governance ambassadors in their public health systems. Public health professionals leading data governance initiatives within their organizations must be able to: y develop policies and procedures that support data governance efforts; y educate all members of the organization about the importance of data governance, and how they can support data governance; y leverage the allocation of relevant resources, including clinical, financial, and administrative data, to support key organizational initiatives; and y measure the return on investment on data governance initiatives and advocate for the continuous improvement of health data governance. The module also aims to help public health systems to identify the early adopters in the system, those data-minded and data-driven professionals who see and subsequently shape the policies that enable health information systems to maximize the value of data, while preserving the privacy, integrity and availability of public health data (7). With the stringent regulation of data protection, and the increasing level of regulation in areas such as critical infrastructure protection and cybersecurity, compliance with the law and protection of trust in health information systems are of utmost importance for the achievement of the long-term goals of data governance. As illustrated by the graphic in Fig. A2, data governance consists in mitigating risks and maximizing benefits, ideally, through a set of actions that serve both endpoints. In that respect, data governance is a toolbox rather than an individual tool, and the different tools must be applied and orchestrated jointly, in a value-driven and transparent process. Fig. A2. Data use decisions should be taken by weighing societal benefits and risks within a data governance framework that maximizes benefits and minimizes risks Source: provided by the Organisation for Co-operation and Development, 2015 (1). Benefits and risks of proposed data uses are evaluated:1. Health information system 2. Legal framework 3. Public communication 4. Certification or accreditation of processors 5. Project approval process 6. Data de-identification 7. Data security and management 8. Data governance review cycles Benefits: rights to health, societal values toward health, health-care quality & efficiency, scientific discovery and innovation, benefits to individuals Informed decisions to process personal health data are taken Risks: rights to privacy, societal trust in government/institutions, societal values toward privacy and sharing data, harms to individuals ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 73 Structure and content of the module Building on the conceptual approach depicted in Fig. A2, the module follows a clear and concise structure as it looks at the foundations of data governance, elaborates on the risks and the regulatory requirements, while also viewing data as an asset or an opportunity for health information systems. The module is constructed in a way that enables it to serve public health systems with a heterogeneous level of maturity in data governance. Data governance is multidimensional by nature, as it integrates aspects of data quality, data protection, data security and data use in such a way that these endpoints are not targeted in an either/or manner, but in a way that maximizes these endpoints (3). In this regard, the module reflects these dimensions by addressing the different aspects of data governance, including the organizational structure, roles, processes, policies, procedures, standards, and metrics, established to manage health data as a strategic asset in health-care systems and public health (8). In other words, health data are both a risk and an opportunity, and any assessment of the maturity of a health data governance system must explore these two dimensions of the concept. None of these aspects, not even data protection or IT security, are absolute and they need to be balanced against each other, ideally creating a Pareto principle-based data governance framework. In line with the Pareto principle, also known as the 80/20 rule, 80% of the outcomes would come from just 20% of the actions; consequently, any data governance programme, in particular, in a resource- scarce environment, would focus on the 20% of actions that provide for 80% of the risk mitigation and opportunity outcomes. The module centres on the aspect of trust (data protection, data security, data ethics, policies) and the aspect of value (standardization, metrics, alignment with international/multinational initiatives, data analytics), and combines them with cross-cutting, horizontal issues such as decision-making processes, communication and resource allocation (2). The module looks specifically at the allocation of resources, which is crucial to a sustainable implementation of a data governance initiative and the availability of training and awareness opportunities in public health systems. How can the outcomes of the module be used to improve health information? Data governance is critical to public health systems as those systems are increasingly data-driven and data-dependent. However, the most critical aspect is still public trust in health information systems and in the professionals establishing the policy framework for health data governance. Only with this level of trust will public health systems be able to move forward and take advantage of the opportunities that big data and other new information management approaches offer to public health. Consequently, data governance is a key tool for improving the accuracy and completeness of health information. Depending on the pre-existing maturity level, the module can serve either as a checklist or as a blueprint for a gap analysis (6). The module works on the assumption that very few countries have established a sophisticated health data governance framework so far and that the concept is new to the majority of WHO Member States. Consequently, the expectation is not for health governance to be institutionalized in a holistic manner, but rather for certain elements of data governance, such as data protection, to be more mature than other elements. Therefore, one of the key challenges prior to the implementation of the module is to identify the appropriate stakeholders and to duly reflect SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT74 the pre-existing level of maturity. This preparatory process may conclude that several institutions need to be involved in the execution of the module, potentially not only the ministry of health or the national institute of public health, but also the ministry of justice, the data protection authority or the national institution in charge of information technology (IT) security and critical infrastructures. Users may even explore whether civil society actors or NGOs could make a valuable contribution to the assessment. The module intends to facilitate an assessment of the status quo, to identify the gaps that need to be closed in subsequent steps of creating and implementing tools and policies. The user will develop a conceptual overview of the strengths and weaknesses of health data governance practices in the country, and will be able to spin off a roadmap from the module that enables the country to achieve a higher maturity level. By using the checklist, the user will also be encouraged to gain an in-depth understanding of the national policies and guidelines applicable to data governance, and determine whether (and, if so, how) these policies and guidelines are embedded in international frameworks and best practices. Both international and regional collaboration is key to developing data governance as the exchange of best practices is cross-fertilizing the promotion of the underlying concepts of data governance. Users are encouraged to repeat the process periodically, for example, annually, to assess whether the country is improving in terms of its maturity level, and whether data governance has become an integral, and therefore sustainable, part of the health information system. Such a follow-up process can also help the country to reassess the allocation of resources, and to review the path towards a multistakeholder institutionalization of the topic. References 1. Health data governance: privacy, monitoring and research. Paris: OECD Publishing, 2015 (OECD Health Policy Studies; https://doi.org/10.1787/9789264244566-en, accessed 29 June 2021). 2. Hernández-Pérez T, Rodríguez-Mateos D, Novillo-Ortiz D. Boosting health data governance framework. Copenhagen: WHO Regional Office for Europe; 2021, in press. 3. Winter JS, Davidson E. Investigating values in personal health data governance models. In: 23rd Americas Conference on Information Systems (AMCIS). Boston: AMCIS; 2017 (https://core.ac.uk/ download/pdf/231905138.pdf, accessed 29 June 2021). 4. Data Governance Checklist, 2011. Washington (DC): US Department of Education, Privacy Technical Assistance Center; 2011 (https://nces.ed.gov/forum/pdf/data_governance_checklist. pdf, accessed 29 June 2021). 5. von Grafenstein M, Wernick A, Olk C. Data governance: enhancing innovation and protecting against its risks. Intereconomics. 2019;54:228–32. doi: 10.1007/s10272-019-0829-9. 6. National IS4H data governance framework: information systems for health toolkit. Washington (DC): Pan American Health Organization, WHO Regional Office for the Americas; 2018 (https:// www.paho.org/ish/images/toolkit/IS4H-NDGF-EN.pdf?ua=1, accessed 29 June 2021). ANNEx 1. RATIONALE FOR THE ADD-ON MODULES 75 7. Hassan S, Chindamo P. Governance in practice: effective data governance -- from strategy through to implementation. Governance Directions. 2017;69(4):207–10. doi: 10.3316/ ielapa.797284650927055. 8. Kariotis T, Ball M, Greshake Tzovaras B, Dennis S, Sahama T, Johnston C et al. Emerging health data platforms: from individual control to collective data governance. Data Policy. 2020;2:E13. doi: 10.1017/dap.2020.14. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT76 Introduction Background This is the revised 2021 assessment item sheet belonging to the WHO Support tool to strengthen health information systems: guidance for health information system assessment and strategy development. This tool was originally published in 2015 as Support tool to assess health information systems and develop and strengthen health information strategies. In 2015–2019, the support tool was applied in a dozen Member States in the WHO European Region. Based on the experiences during those assessments, the tool including this assessment item sheet has now been updated. Overall structure This assessment sheet consists of a core module, including a summary and scoring sheet and several add-on modules. The aim of the core module is to obtain a generic overview of the functioning of the entire national HIS. The aim of the add-on modules is to zoom in on specific parts or functions of the national HIS in greater detail. This assessment sheet includes add-on modules on health information for WHO’s GPW13 and EPW, infectious disease surveillance, NCD monitoring, HRH, and health data governance. Organization of the core module The core module comprises five domains: data collection; analysis; health reporting; knowledge translation; and governance and resources. This structure is based on the definition of an HIS by Verschuuren and van Oers in Population health monitoring (see Fig. A2.1). They define an HIS as “the total of resources, stakeholders, activities and outputs enabling evidence-informed health-policy- making”. Each domain consists of assessment items that are phrased as general or main questions. Main questions are subdivided into probing questions, and a description of the expectations (i.e. the situation in a fully mature HIS) is given for each probing question. The findings of the assessment for the core module can be summarized in the summary and scoring worksheet (light green worksheet). Domains of the core module • Data collection includes assessment of available data collections, (efficiency of) data flows and quality and usability of existing data collections. • Analysis includes assessment of availability and use of indicator sets. • Health reporting includes assessment of availability and use of health reports. • Knowledge translation includes assessment of the extent to which stakeholders are familiar with available health information and knowledge products, and which knowledge translation tools and mechanisms are being used. • Governance and resources includes assessment of HIS governance and general HIS resources (legal framework, financial resources and ICT infrastructure). Note: human resources are addressed as part of the other four dimensions. Note: to facilitate the work of the assessors when conducting an assessment and applying this item sheet, we have created an Excel file that you can download at: https://apps.who.int/iris/handle/10665/342048 Annex 2. HIS assessment item sheet INTRODUCTION 77 Health policy HIS activities HIS stakeholders HIS outputs HIS resources ICT infrastructure Legal framework Financial resources Human resources (volume and capacity) Data collection e.g. Statistical institutes, other data collectors and data owners, e.g. public health institute, health insurance funds e.g. Spreadsheet or table with raw data e.g. Databases, quality control mechanisms, classifications, data processing software e.g. Indicator sets, analysis methods, data analysis software e.g. Quality criteria and toolkit for reporting, evidence resources e.g. Knowledge translation tools & mechanisms, evidence resources e.g. Indicators, comparisons of indicators in time, space and between groups e.g. Integrated health report, comprehensive websites e.g. Targeted key messages, policy briefs, policy dialogues e.g. Public health institutes, other knowledge institutes e.g. Public health institutes, universities, other knowledge institutes e.g. Knowledge translation/brokering institutes, ministry of health, other authorities with public health information requirements Analysis/ contextual- ization Health reporting Knowledge translation Data Information Knowledge EiP Evidence- informed policy- making Fig. A2.1. Organization of the core module Source: Rechel B, Rosenkoetter N, Verschuuren M, van Oers H (2019). Health information systems. In: Verschuuren M, van Oers H, editors. Population health monitoring. Cham: Springer (https://doi.org/10.1007/978-3-319-76562-4_2). SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT78 CORE Data collection Item ID Question Probing question Expectations Data sources Data sources_1 Are regular censuses performed in the country? 1) What is the frequency at which censuses are performed? 1) Censuses are planned and conducted at fixed, regular intervals, at least once every 10 years, in line with the United Nations recommendation on the frequency of population censuses 2) Is the census based on surveys and/or administrative data sources? 2) In line with Eurostat requirements for the 2021 census, the census is primarily based on administrative data sources 3) Are time series revised backwards? If so, for how many years? 3) If time series are revised backwards, a communication strategy is in place on how to communicate to the ministry of health and other health information users about the reasons for the retroactive amendment of indicators and the impact of the revision on the indicator values 4) Are different indicator values used in parallel, based on different population figures/different denominators? 4) If different indicator values are used in parallel, a communication strategy is in place on how to communicate to the ministry of health and other health information users why different versions of the same indicator are being calculated and reported, and how these should be interpreted 5) (Only necessary if civil registration covers less than 95% of deaths) Are questions on mortality included in the census? 5) If questions on mortality are included, results are used to estimate child mortality and household deaths in the past 12/24 months, including sex of deceased and age at death Data sources_2 What is the status of registration of vital statistics? 1) Are any births unregistered and, if so, what share of births – and which subgroups of the population – does this concern? 1) The coverage of registered births is (close to) 100% 2) What kind of medical information is regularly collected on births? 2) Information on birth weight, gestation period/ prematurity, birth order (for multiple births), method of delivery, complications during delivery, stillbirth and date of the mother’s most recent delivery is part of the regular civil registration and vital statistics data collection 3) Are any deaths unregistered and, if so, what share of deaths – and which subgroups of the population – does this concern? 3) The coverage of registered deaths is (close to) 100% 4) What is the coverage of cause-of-death information recorded on the death registration form? 4) The coverage of cause-of-death information recorded on the death registration form is (close to) 100% CORE DATA COLLECTION 79 Item ID Question Probing question Expectations Data sources contd 5) What is the quality of the cause-of-death information recorded on the death registration form? 5) Cause-of-death information recorded on the death registration form is of high quality: • medical doctors are trained (as part of the regular curriculum and/or postgraduate training) to fill in the death registration forms; • clear rules and a legal framework that define under which circumstances an autopsy needs to be performed to establishing the cause of death are in place; • ICD-10 is used to code causes of death; • coding is performed by dedicated, specifically trained staff; and • the proportion of all deaths coded to ill-defined causes is low Data sources_3 What is the status of health service records? 1) What kind of record- keeping systems are used in hospitals and in primary health care/by general practitioners? 1) A centralized EHR system is in place. If various electronic information systems are used, interoperability between these systems is ensured See also items Data infrastructure_1 and Data infrastructure_3 2) Can data for secondary purposes – such as quality control, planning and policy- making – easily be extracted from these systems? 2) Tailored aggregated datasets can be extracted easily by administrators, managers and health-care staff See also item Data infrastructure_1, probing question 5 3) What is the coverage and quality of information on medical procedures registered in the health service records? 3) There is high coverage of registration of care provided – including diagnostic tests, treatments (surgery, drugs, other), medical devices, type of care (inpatient, ambulatory, emergency) and length of stay – in the health service records, and appropriate international classifications are used (such as International Classification of Health Interventions). Biases affecting these data are limited and known 4) What is the coverage and quality of information on diagnoses registered in the health service records? 4) Coverage of diagnoses registered in the health service records is high. For hospital discharge records ICD-10 is used to register diagnoses, and in primary health care the International Classification of Primary Care is used. Biases affecting these data are limited and known 5) What is the coverage and quality of health insurance records? 5) Health insurance records are complete, and information on care received is linked to ICD-10 coded information on diagnoses. Biases affecting these data are limited and known. Data are available for secondary purposes such as population statistics and research Data sources_4 Which disease registries are operated? 1) Is a cancer registry in operation? 1) A national-level population-based cancer registry is in operation, according to international standards SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT80 Item ID Question Probing question Expectations Data sources 2) Are any other disease registries in operation, such as for diabetes, cardiovascular disease or dementia? 2) Dedicated registries for major chronic diseases are in place; if not, robust morbidity estimates from other sources are available Data sources_5 How is infectious disease surveillance organized? 1) What is the mechanism for identifying notifiable infectious diseases and reporting them to the relevant authorities? 1) An electronic notification system is in place, with real-time data, allowing authorities to respond immediately 2) Is information on notifiable diseases according to country-specific legislation and international obligations (International Health Regulations (2005)) available? 2) Information is available on epidemic-prone diseases like cholera, diarrhoea with blood, measles, meningitis, plague, viral haemorrhagic fevers, yellow fever, severe acute respiratory syndrome and bird flu; and on diseases targeted for eradication/ elimination (such as poliomyelitis, neonatal tetanus and leprosy). A list of priority diseases, conditions and case definitions exists. The available information is complete and timely 3) Is information from the infectious disease surveillance system available for population health monitoring? 3) Information from the infectious disease surveillance programme (e.g. number of measles cases, number of cases of influenza) is readily available for use in population health reports, where they it can be placed into a broader context Data sources_6 Which (preventive) health programmes are implemented? 1) How is the coverage and quality of information from the vaccination programme? 1) There is a central, national database with programme-based data on all vaccinations in the vaccination programme, with full coverage. 2) What is the coverage and quality of information from screening programmes? 2) For each official screening programme, there is a central, national database with programme-based data, with full coverage 3) What is the coverage and quality of information from (vertical) programmes, such as on HIV/AIDS, tuberculosis and diabetes? 3) Such health programmes include a M&E component, and programme-based data are collected in a central database, with full coverage Data sources_7 Are regular health surveys conducted? 1) Are regular health interview surveys carried out? 1) A long-term operational plan is in place, including financing from the public budget, for regular conducting of national health interview surveys. The methodology applied in these surveys is in accordance with international standards and, if applicable, international data delivery requirements. Specific efforts are made to make sure that hard- to-reach groups are adequately represented in the sample (e.g. people in long-term care facilities, non- native speakers, homeless people) CORE DATA COLLECTION 81 Item ID Question Probing question Expectations Data sources 2) Are regular health examination surveys carried out? 2) A long-term operational plan is in place, including financing from the public budget, for regular conducting of national health examination surveys. The methodology applied in these surveys is in accordance with international standards. Specific effort is made to make sure that hard-to-reach groups are adequately represented in the sample (e.g. people in long-term care facilities, non-native speakers, homeless people) 3) Are any additional regular health or health-related surveys performed? 3) The statistical office carries out regular household budget surveys and surveys on living conditions, in accordance with international standards and requirements. Other regular surveys may be carried out, such as surveys focused on specific groups (e.g. Health Behaviour in School- aged Children, Childhood Obesity Surveillance Initiative, Survey of Health, Ageing and Retirement in Europe) or specific topics (e.g. nutritional surveys) 4) Do health and statistical authorities work together on survey design, implementation and data analysis and use? 4) Cooperation mechanisms exist (e.g. between the public health institute, statistics office and universities) Data sources_8 What data sources on health-care resources exist? 1) What data sources exist on human resources? 1) A national human resources database tracks the number of health professionals by major professional category working in either the public or the private sector, with complete coverage. A national database tracks the annual numbers graduating from all health training institutions, with complete coverage. Each individual health-care provider in the national human resources database has been assigned a unique identifier code, which stays with them for their practising career; this permits data on the same provider to be merged 2) What data sources exist on facilities? 2) A national database of public and private sector health facilities is in place, with complete coverage. Each health facility has been assigned a unique identifier code that permits data on facilities to be merged Data sources_9 What data sources on health expenditure exist? 1) Are data available for both public and private expenditure? 1) Financial records are available on general government expenditure on health and its components (e.g. by the ministry of health, other ministries, social security, regional and local governments and extrabudgetary entities) and on private expenditure on health and its components (e.g. household out-of-pocket expenditure, private health insurance, NGOs, firms and corporations) 2) Are data collected in accordance with the System of Health Accounts methodology? 2) Data are collected in accordance with the System of Health Accounts methodology SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT82 Item ID Question Probing question Expectations Data sources Data sources_10 What data sources from other domains are available? 1) What data sources from other domains are available? 1) Various data sources from other domains that are relevant for population health are available, such as social security data and data on, for example, the (mental) health of students from schools and universities, retirement statistics, injuries/accidents, crime records, road accidents, air pollution, ambient noise, the living environment (e.g. green spaces, walkability), threats related to food safety and threats of chemical or radiologic/nuclear origin Data infrastructure Data infrastructure_1 What is the status of adoption of EHR systems? 1) Are EHR systems being used by health-care facilities/ providers? 1) Health-care facilities and providers only use electronic patient records; there is no parallel paper record keeping 2) How advanced or extensive are these EHR systems? 2) The EHR systems are comprehensive and compile all information related to the care for an individual patient. This means that the systems, alongside information on the health status/ diagnosis of the patient, hold information on, for example, lab results, diagnostic imaging, the care process (e.g. referrals), (surgical) interventions, drug prescriptions and billing/reimbursement. In addition, the systems include professional standards and clinical decision support tools. eSignatures are used for authorization 3) (If one centralized or a limited number of EHR systems are in use) Were the needs of end-users (i.e. various health-care workers) taken into account in the development of the EHR system? 3) The needs of the health-care workers that will be working with the system in daily practice were taken into account in its development 4) (If one centralized or a limited number of EHR systems are in use) Were patients/patient organizations involved in the development of the EHR system? 4) Patients/patient organizations were consulted during development of the system 5) (If one centralized or a limited number of EHR systems are in use) Was secondary use of health data taken into account in the development of the EHR system? 5) The EHR system provides business intelligence analysis information for quality control and to enhance performance; data extractions for regular official statistics at the population level have been automated; and these fulfil the needs of the statistics office, ministry of health and public health agency See also item Data sources_3, probing question 2 6) Can patients access their own data in the EHR system? 6) Patients can access their own data, empowering them by allowing them to check these for completeness and correctness CORE DATA COLLECTION 83 Item ID Question Probing question Expectations For interoperability, see item Data infrastructure_3 For ICT infrastructure, see item Governance and resources_8 Data infrastructure_2 What is the status of adoption of other electronic information systems in the national HIS? 1) Is an electronic system for registration of births in place? 1) An electronic system for registration of births is in place 2) Is an electronic system for registration of deaths in place? 2) An electronic system for registration of deaths is in place 3) Is an electronic system in place for notification and registration of notifiable infectious diseases? 3) See item Data sources_5, probing question 1 4) Are electronic information systems in place for medicines and medical devices? 4) Electronic information systems are in place for quality control, pharmacovigilance/side-effects of medicines and medical devices, market access and stock keeping Data infrastructure_3 Are any interoperability standards defined and used? 1) Are any commonly agreed interoperability requirements or standards in place for the information systems in the HIS and wider information systems? 1) The standards that health-care providers and facilities need to use to communicate between organizations and to report to authorities are defined 2) Are any accreditation standards in place for digital services and applications that focus on ensuring interoperability with other services and applications? 3) Which organizations or bodies are responsible for development of health sector and broader national standards? 4) Which organizations or bodies are responsible for undertaking conformance, compliance and accreditation of products and services – including ICT – used in the health sector? 2–4) Specific offices/agencies are in charge of defining official interoperability standards, compliance with these standards and accreditation of products and services 5) What is the level of adoption of interoperability standards among existing health services and applications? 5) An overview of the level of adoption of health-care standards is available (for example, through a survey); the level of (planned) adoption is high SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT84 Item ID Question Probing question Expectations Data infrastructure_4 Is a unique personal identification number (UPIN) in use? 1) Is a UPIN issued at birth for each citizen? 1) A UPIN is issued at birth for each citizen, and this is used across government services, including health services For health-care providers and facilities, see item Data sources_8 Data management Data management_1 What metadata standards are in use? 1) Are common standards in use? 2) Are these standards aligned with international standards? 1–2) Common standards are used for metadata for official (health) statistics, which are aligned with international metadata standards 3) Are the metadata made available? 3) Metadata are structurally available for all official statistics and can be easily located and accessed by users of these statistics 4) If adjustments such as standardization or weighing are used, how are these selected? 4) In-country adjustments use transparent, well established methods. If a weighing factor is applied, methods and variables used are clearly described Data management_2 What quality control mechanisms are applied for the data sources listed above? 1) What kind of automated control mechanisms are built into the EHR/information systems? 1) Automated logic checks are built into the systems 2) What kind of manual quality control checks are performed? 2) Regular manual checks are performed according to well established and well documented protocols to check for completeness and correctness of the data 3) Are audits performed to check the completeness and correctness of data? 3) Regular audits are performed – e.g. by the health insurance company or ministry of health – to check the quality of the diagnosis-related group information submitted by health-care facilities Data management_3 How can the data sources listed above be accessed and used for secondary purposes? 1) Are publicly funded data sources publicly available and published as open data? 1) Publicly funded data sources are publicly available and published as open data (provided that the necessary data protection safeguards have been taken into account) CORE DATA COLLECTION 85 Item ID Question Probing question Expectations 2) Is there an electronic platform for exchanging data between (semi-) governmental organizations? 2) An electronic data exchange platform for the safe and efficient exchange between (semi-) governmental organizations is in place. All government departments and other relevant agencies are connected to the platform 3) Is a central data warehouse in place? 3) An integrated data warehouse is operated at the national level, containing data from all data sources (both population-based and facility-based sources, including all key health programmes). The data warehouse has a user-friendly user interface, which is accessible to various user audiences and which allows for the tailored extraction of data and indicators 4) Are opportunities available to link data sources safely at the subject level and perform comprehensive analyses – for example, through a closed controlled working environment operated by the statistical office, or through anonymization and linkage by a trusted third party See also item Data infrastructure_5 4) Opportunities are available to link data sources safely at the subject level and perform comprehensive analyses – for example, through a closed controlled working environment operated by the statistical office, or through anonymization and linkage by a trusted third party See also item Data infrastructure_5 Data management_4 Are international data delivery requests being met? See also item Analysis_5 1) Are data collection methods and analytical approaches (e.g. calculation of indicators) in line with international standards and recommendations? 1) Data collection methods are in line with international standards and recommendations 2) Is the country able to meet all data delivery requirements from the international organizations of which it is a member/with which it is collaborating? 2) The country can fulfil all health information requests from international organizations 3) Does the country participate in international health information projects or activities? 3) Which actors are involved in international projects or activities is known. Developments with regard to health information at an international level are routinely monitored and shared by experts in the HIS See also item Governance and resources_4 Resources for data collection Resources for data collection_1 Is an adequate legal framework in place for HIS data collections? 1) Is there a legal basis for the HIS data collections listed above? 1) There is a legal basis for the most important HIS data collections 2) Which data elements are defined in the law? 2) Specific data standards (e.g. disaggregation levels and ICD-10 codes) are defined SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT86 Item ID Question Probing question Expectations 3) Is an adequate legal framework in place for linking and sharing the data collections listed above? 3) Criteria for data privacy, secondary processing, sharing of information and data linkage are specified. A legal framework that is not too restrictive is in place – i.e. one that strikes the right balance between data protection and making health data available for the public good. Accessibility of essential data sources for the most important HIS stakeholders (statistical office, ministry of health) is regulated by law 4) Are requirements for data storage defined in the law? 4) Requirements for data storage in the HIS are defined Resources for data collection_2 Are sufficient human resources available for maintaining and operating HIS data collections? 1) Do HIS stakeholders have adequate tools to maintain and operate HIS data collections? 1) HIS stakeholders have adequate tools (e.g. database and data management software) For ICT infrastructure, see item Governance and resources_8 2) Do HIS stakeholders have adequate manpower to maintain and operate HIS data collections? 2) HIS stakeholders have adequate manpower, and staff turnover is limited 3) Do HIS stakeholders have adequate capacity to maintain and operate HIS data collections? 3) HIS stakeholders have adequate capacity – i.e. staff with the right technical skills and expertise. Regular training is provided/funds are available for regular training CORE ANALYSIS 87 CORE Analysis Item ID Question Probing question Expectations Analysis_1 Is a core set of health indicators defined? 1) Is the core set linked to a specific health policy (process) and/or to specific health goals or targets? 1) The core set is linked to a specific health policy (process) and/or to specific health goals or targets 2) How were core indicators selected? 2) National minimum core indicators were transparently identified for national and subnational levels. Selection of indicators is also informed by international indicator sets 3) Which categories does the indicator set cover? 3) Indicators cover all categories of health indicators, such as determinants of health; health system inputs, outputs and outcomes (health systems performance assessment); health status; and health inequalities. If possible, the set includes relevant indicators from other policy sectors (e.g. social affairs, education) 4) How are the indicators defined and calculated? 4) Indicator definitions exist and the method for their calculation is documented. If applicable, the numerator and denominator of the indicators are clearly defined 5) Are metadata available and harmonized within the country and across countries? 5) Regularly updated metadata exist for each indicator and are publicly available. Metadata include the categories definition, calculation/method, available dimensions/subgroups (e.g. age, gender, geographical information, nationality, migration, social status – e.g. education, employment status, income), rationale and data sources 6) (If individual data on social status are not available) Is a geographically based deprivation index (or similar) available? 6) As an alternative indicator of social status, a deprivation index is available to perform comparisons at the ecological level Analysis_2 What kind of analyses are performed on the core indicators? 1) Are subnational comparisons made? 1) Core indicators can be broken down according to relevant subnational entities (e.g. regions, municipalities). Subnational disaggregations are produced regularly 2) Are international comparisons made? 2) Definitions of national core indicators are aligned with international definitions to allow international comparisons; these are produced regularly. If different definitions and/or data sources are used for national indicator values and for international comparisons, the reasons for this are clearly explained in the indicator metadata 3) Are historical time trends made? 3) Historical trend data are available for the core indicators and time trends are produced regularly 4) Are future projections made? 4) Periodic population projections are made. These are used to make demographic projections for key indicators. If adequate trend data are available, combined demographic and epidemiological projections are made for key indicators SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT88 Item ID Question Probing question Expectations 5) Are comparisons between subgroups made/are analyses of health inequalities performed? 5) Data for the core indicators can be disaggregated according to age, sex, socioeconomic status and other relevant stratifiers (e.g. ethnicity). Disaggregated indicator values are produced regularly See also probing question 1 on subnational comparisons Analysis_3 Are the core indicators reported/ published regularly? 1) How and how often are they reported/published/updated? 1) Core indicators are regularly reported in standardized tables, in health reports and in basic tools for spatial comparisons and comparisons over time 2) What is the frequency at which data for the core indicators become available? 2) The datasets used to calculate the core indicators are updated regularly, and the frequency of these updates is in line with policy needs 3) Are regular publication dates/ periods available for each indicator? 3) A publication schedule is available Analysis_4 Alongside a core set of health indicators, are other formalized health indicator sets in use? 1) Are indicator sets to monitor implementation and impact of specific programmes in use? 1) Indicator sets for the monitoring of specific programmes (e.g. on NCDs, infectious diseases) are in use. These have been formally established and are regularly reported on 2) Are indicator sets to monitor performance of (specific parts of) the health-care system in use? 2) An indicator set for health system performance assessment and/or indicator sets to monitor specific part of the health-care system (e.g. primary health care, hospital care, care for elderly people) are in use. These are formally established and regularly reported on Analysis_5 Are the indicators used aligned with international indicator sets – in particular WHO’s Impact Framework of GPW13? See also item Data management_4 1) Are indicators used aligned with the Impact Framework of WHO’s GPW13? 1) Indicators used are aligned with the Impact Framework of WHO’s GPW13: the indicators in the Impact Framework are used to guide national health policy and WHO’s data delivery requests can be met 2) Are indicators aligned with the SDGs? 2) Indicators used in the country are aligned with the SDGs and are used to guide national (intersectoral) health policy. At a minimum, the country can meet the SDG-related data requests in the WHO Regional Office for Europe’s Joint Monitoring Framework 3) Are indicators used in the country aligned with the European core health indicators? 3) Indicators used in the country are aligned with the European core health indicators Analysis_6 Is the country investing in Big Data and Artificial Intelligence (AI) research and development? 1) Is a big data/AI strategy related to (public) health in place? 1) A strategy for big data/AI, either standalone or as part of the digital health/e-health strategy or another national digital health initiative, is in place 2) Is action on big data/AI included in the national budget? 2) The national health and/or research budget includes dedicated funds for action on big data/AI CORE ANALYSIS 89 Item ID Question Probing question Expectations 3) Are strategies to promote research into the usability of big data for (public) health purposes in place? Are strategies to promote development and use of AI for (public) health in place? 3) A research strategy around big data/AI and data science is in place that includes applications for (public) health 4) Are ethical big data/AI requirements, standards and best practices listed and respected? 4) A set of ethical principles is defined, together with policies and regulations. Best practices are actively shared 5) Are infrastructure requirements for big data/AI and data science defined? 5) The following infrastructure requirements are clearly defined: computing capacity, storage capacity, networking infrastructure, security policies 6) Is the country working on transforming the health information workforce to be fit for the new big data/AI era? 6) The country has a plan to train health information professionals in data science and big data/AI, including adequate funding Analysis_7 Do HIS stakeholders have adequate resources to analyse and report on indicators regularly? 1) Do HIS stakeholders have adequate tools for regular analysis and publication of indicators? 1) HIS stakeholders have adequate tools for analysis (e.g. computers, servers, analysis software) and publication (e.g. module for interactive dashboard) 2) Do HIS stakeholders have adequate manpower for regular analysis and publication of indicators? 2) HIS stakeholders have adequate manpower, and staff turnover is limited 3) Do HIS stakeholders have adequate capacity for regular analysis and publication of indicators? 3) HIS stakeholders have adequate capacity – i.e. staff with the right skills and expertise (such as statisticians, epidemiologists, GIS experts, data visualization experts, communication experts). A multidisciplinary team works on publication of the core indicators. Regular training is provided/funds are available for regular training on analysis skills SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT90 CORE Health reporting Item ID Question Probing question Expectations Health reporting_1 Are the health reports produced aimed at informing policy- making? This may concern formulation of new policy/agenda setting, monitoring/ evaluation of existing policy and/or planning of resources 1) Are such health reports being produced on a regular basis, and by whom? A health report can either be a more traditional report in paper/ PDF format, or a web- based report/website 1) Regular health reports are produced by the national public health agency or comparable institution, independent of the ministry of health. Scientific standards and common transparency requirements are followed in the production of the reports 2) How comprehensive are these reports? 2) Regular health reports apply a broad and integrative approach, meaning that they address population/ public health, health care, and health-related topics from other domains. Data are accompanied by contextual information/explanations. The reports also include options for action, such as overviews of (cost-) effective interventions and policy options 3) Are foresight and scenario exercises included in the reporting efforts? 3) Foresight and scenario exercises are performed at regular intervals (e.g. once every 4–5 years) to inform long-term strategic health policy-making 4) What format do the health reports use? 4) Regular health reports are written in easy-to- read language, use a combination of text and informative visualizations, and include key messages. Web-based reports use interactive visualization tools that generate tables, graphs/charts, maps and infographics. It is possible to download the visualizations and the data on which they are based. Tailored summaries/factsheets are available for different target audiences 5) Are these reports publicly available? 5) Regular health reports are publicly available. Online products are freely available, or, if necessary, after registration. Printed reports can be ordered online 6) What kind of communication and dissemination strategies are used for these reports? 6) Comprehensive communication and dissemination strategies are in place, including mass media, social media, online health (information) platforms, newsletters, email messages, presentations and lectures. Active after-care is also part of the communication and dissemination strategy, including follow-up on social media. Experts talking to the mass media have received relevant training 7) Is it known to what extent policy- makers and other users (such as media, patient organizations, NGOs, professional organizations) actually use the reports? 7) User surveys are conducted regularly. Website statistics are monitored and analysed regularly. Reports about the results of the user surveys and website statistics are publicly available CORE HEALTH REPORTING 91 Item ID Question Probing question Expectations Health reporting_2 What mechanisms exist for using health reports in the policy-making process? 1) What is the mechanism for using health reports in the health policy-making process? 1) There is a formal, public and transparent procedure for using health reports in the policy-making process (e.g. once every X years the public health institute makes a health report for the ministry of health, at its request, and the ministry formally and publicly reports on how it has used the information in this report). Parliament is informed by the ministry of health when formal health reports are published 2) Are the reports also used to inform intersectoral policy- making, and do other policy sectors also include information on health in their reporting efforts/ use health information for informing their policies (a Health in All Policies approach)? 2) An intersectoral governmental body that discusses (how to use) the health reports is in place, and its decisions are formally and publicly reported. Health is a standard dimension in reports of other policy sectors Health reporting_3 Are health reports produced at the health-care facility and provider levels? 1) Do managers and medical staff use health reports to monitor and improve performance (e.g. quality control, patient safety)? 1) Managers and medical staff regularly use health reports to monitor and improve performance. Such reports are discussed jointly (e.g. at the department or team level) and ways to improve are decided together. There is an open attitude among health-care staff towards measuring and monitoring performance. Health-care staff feel safe to discuss (suboptimal) quality of care and performance 2) What kind of indicators are used for these reports? 2) Indicators that are acknowledged by (international) peers as valid and useful are used. Indicators used include patient-reported outcomes (PROMs) and patient-reported experiences (PREMs) Health reporting_4 Do HIS stakeholders have adequate resources for producing and publishing regular health reports? 1) Do HIS stakeholders have access to adequate tools for health reporting? 1) HIS stakeholders have access to adequate tools for producing health reports (e.g. quality criteria/toolkit, evidence resources) and publishing health reports (e.g. software for creating interactive graphs and options for integrating videos in online reports) 2) Do HIS stakeholders have adequate manpower for producing and publishing regular health reports? 2) HIS stakeholders have adequate manpower, and staff turnover is limited 3) Do HIS stakeholders have adequate capacity for producing and publishing regular health reports? 3) HIS stakeholders have adequate capacity, i.e. staff with the right skills and expertise (such as statisticians, epidemiologists, GIS experts, data visualization experts, writers/editors, communication experts). A multidisciplinary team works on producing the health reports. Regular training is provided/funds are available for regular training on reporting skills SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT92 CORE Knowledge translation Item ID Question Probing question Expectations Knowledge translation_1 Do relevant stakeholders know which information and knowledge products are available, and are they able to use them? 1) Are information products regularly demanded by users like senior managers and policy-makers? 1) The regular information and knowledge products produced within the HIS and their publication schedules are well known by policy- makers, senior managers and other actors such as media representatives. Senior managers and policy-makers demand complete, timely, accurate, relevant and validated HIS information, and know how to interpret and use it 2) Are support mechanisms available to train relevant actors on how to interpret and use the products? 2) Training or information courses on the products and their use are offered regularly Knowledge translation_2 Do policy-makers (and other relevant stakeholders) have access to all information and knowledge necessary to support policy-making? 1) Do the information and knowledge products produced within the HIS meet the needs of policy-makers? 1) Regular exchange sessions are held to identify the information needs and assess the timeliness and usefulness of the formats with policy-makers and other relevant users. The outcomes of these sessions and implemented changes are documented and reported 2) Is it possible for policy- makers (and other relevant stakeholders) to monitor the targets of the health strategy? See items Analysis_1 to Analysis_4 2) A set of indicators to monitor progress towards the targets of the health strategy has been defined and is reported on regularly See items Analysis_1 to Analysis_4 3) Do policy-makers and other relevant stakeholders have enough information to determine which interventions and policies to implement? 3) Regular health reports include options for action, such as overviews of (cost-)effective interventions and policy options. See item Health reporting_1 Exchange and integrated knowledge translation approaches are applied to make sure that the information and knowledge produced meet the needs of policy-makers. See item Knowledge translation_3 4) What kind of communication mechanisms are in place if there are questions about information and knowledge products or ad hoc requests for health information? 4) A rapid response team/mechanism is in place to respond to quickly ad hoc questions (e.g. when the ministry of health is looking for health information to answer questions from parliament). After-care is a structural element in the communication and dissemination plans for health information and knowledge products See item Health reporting_1, probing question 6 A user survey is conducted regularly to identify the usability of health information and knowledge products See item Health reporting_1, probing question 8 Knowledge translation_3 What kind of knowledge translation tools and mechanisms are used? 1) Are specific tools to stimulate uptake of information and knowledge in policy-making used? 1) Producers of reports use tools specifically aimed at stimulating uptake of information and knowledge in policy-making, such as policy briefs and policy dialogues CORE KNOWLEDGE TRANSLATION 93 Item ID Question Probing question Expectations 2) Alongside the more traditional push and pull mechanisms, are exchange and integrated approaches also applied for knowledge translation? In exchange approaches, information analysts and relevant users of the HIS work in partnership, often facilitated by knowledge brokers, to collect the necessary evidence. In integrated approaches, a knowledge translation infrastructure is institutionalized and represents clear objectives for action, regular assessments of the relevance of its efforts and incorporation of elements of push, pull or exchange efforts 2) Exchange and integrated approaches for knowledge translation are applied 3) To what extent are the applied knowledge translation tools and mechanisms institutionalized? 3) The applied knowledge translation tools and mechanisms are institutionalized: they are a structural element of the health policy-making process Knowledge translation_4 Do HIS stakeholders have adequate resources for knowledge translation? 1) Do HIS stakeholders have adequate manpower for knowledge translation? 1) HIS stakeholders have adequate manpower for knowledge translation 2) Do HIS stakeholders have adequate capacity for knowledge translation? 2) HIS stakeholders have adequate capacity. Staff have been trained in knowledge translation concepts, tools and skills, and adequate budget id available for training to keep staff capacity up to date SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT94 CORE Governance and resources Item ID Question Probing question Expectations Governance and resources_1 Is the legislation providing the legal framework for the HIS up to date? Ideally, this legal framework should also cover an evidence-informed policy cycle For the legal framework for data collection, see item Resources for data collection_1 1) Does the country have health legislation? 1) The legislation exists and is enforced. The legislation defines the tasks of the public health authority. The legislation covers WHO’s essential public health operations 2) Does the public health service law include population health monitoring and maintenance of an HIS as a mandatory task of the public health authority? 2) Population health monitoring and the maintenance of the HIS are part of the legislation 3) Does the public health service law also define tasks that cover the whole policy cycle? 3) The public health service law defines tasks and roles that cover the whole policy cycle (problem definition, agenda setting, policy formulation, decision-making, policy implementation, policy evaluation) 4) Does the country have electronic HIS legislation that governs how health information is stored, accessed and shared across geographical and health sector boundaries? 4) A legislative framework for the electronic HIS is in place. The legislation defines which (international) classifications must be used (e.g. ICD-10, System of Health Accounts). National interoperability standards and other requirements are developed. Compliance, conformance and accreditation of electronic health information products and services are defined and implemented. The legislation ensures equal access for all citizens to their own health data 5) What policies exist to stimulate and manage innovations in the field of electronic health information systems, such as who is responsible for introducing change and innovation, how risks are managed and how to evaluate appropriateness, feasibility and utility? 5) Policies are in place that stimulate and manage innovations in the field of electronic health information systems. Best practices and evaluation reports are collected in a central place and made publicly available Governance and resources_2 Does the country have a comprehensive HIS strategic plan that is in active use and implemented? 1) Is an HIS strategic plan in place? 1) A formal comprehensive HIS strategic plan is in place, either as a standalone strategy or as a clearly discernible element of a wider health/health systems strategy. The most important HIS stakeholders, including other relevant ministries beside the ministry of health, were involved in the development of the strategic plan and are involved in its M&E (see probing question 3). The HIS strategic plan defines an HIS vision and clear goals CORE GOVERNANCE AND RESOURCES 95 Item ID Question Probing question Expectations 2) Does the HIS strategic plan include aspects of electronic health information services, or does the country have a separate strategy for this? 2) The HIS strategic plan includes aspects of electronic health information services. If this is not the case, a separate strategic plan/policy for electronic health information services exists See item Governance and resources_3 3) What kind of M&E mechanisms for the strategic plan are in place? 3) The strategic plan defines mechanisms for monitoring progress towards the goals (e.g. what indicators will be used, who will collect data for these indicators and report on them, how often progress will be assessed) 4) How well integrated is the HIS strategic plan between the national, state, regional and local levels (and different policy sectors)? 4) The strategic plan lays out how cooperation and integration of work between different levels is performed Governance and resources_3 Is an HIS coordination mechanism/body in place? 1) What kind of HIS coordination mechanism is in place? 1) A formal multistakeholder coordination mechanism/body for the HIS is in place. The ministry of health is actively involved. The coordination mechanism has clear terms of reference that are publicly available. Meeting reports including lists of decisions taken are formally established and made publicly available. The coordination body is chaired by an independent expert 2) Does the HIS coordination mechanism/body have a full overview of health information needs and of what health information is available? 2) The HIS coordination body has a full overview of health information needs and available health information. A strategy for overcoming discrepancies between needs and available information is in place. See item Governance and resources_2 The role of the HIS coordination body in this is clear (e.g. the HIS coordination body has an advisory function for the ministry of health regarding the establishment of new data collections or the adaptation of existing data collections) Governance and resources_4 Are mechanisms in place for monitoring performance and outcomes of the HIS and its various subsystems? 1) Is a mechanism in place that assures the quality of the HIS and its knowledge/information products? 1) A mechanism is in place. Quality assurance procedures for the various elements that are evaluated are described (e.g. acceptability, data quality, flexibility, usability, simplicity, output). Outcomes of quality assessments are publicly reported See item Data management_2 2) Who is responsible for HIS quality assurance? 2) Tasks and responsibilities regarding quality assurance processes are clearly defined. If an HIS coordination body is in place, this has a role in (discussing the outcomes of) HIS quality assessment See item Governance and resources_3 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT96 Item ID Question Probing question Expectations Governance and resources_5 Are mechanisms in place for monitoring performance and outcomes of EHRs? 1) Are important aspects of EHRs monitored and evaluated? 1) A routine M&E process is in place, looking at aspects such as functionality, stability, usability, safety, efficacy and effectiveness, as well as completeness and quality of the data entered into the system 2) Who is responsible for monitoring and evaluating the performance of EHRs? 2) Related to performance of the HIS regarding the secondary use of data, regular and institutionalized coordination mechanisms are in place, including the primary users of the system (health-care facilities), software developers (if applicable) and secondary users (e.g. ministry of health, public health institute) Governance and resources_6 How are the different entities and activities (including staff, data collection, infrastructure and training) in the HIS financed? 1) Is how the different HIS entities, activities and resources, including investments in electronic health information systems, are financed known? 1) The budgets/financing streams for the institutions involved and their activities are transparent. Core HIS activities are financed from the public budget 2) Is the financing sustainable? 2) The financing is sustainable and is not primarily based on third-party money/donor or project funds Governance and resources_7 What is the status of the ICT infrastructure in the national HIS? 1) Is the availability of hardware sufficient? 2) Is the availability of software sufficient? 3) Are enough skilled ICT staff available? 1–3) Adequate ICT infrastructure (e.g. computers, internet access, servers) and adequate ICT support are in place at the national level, at relevant subnational levels and at the hospital/provider level 4) What is the capacity and coverage of data connectivity and networking across the country, including metropolitan, regional, rural and remote areas? 4) Sufficiently fast internet is available throughout the country, including in remote rural areas 5) What is the capacity and coverage of mobile connectivity and networking (e.g. mobile phone coverage) across the country, including metropolitan, regional, rural and remote geographical areas? 5) There is coverage of mobile connectivity and networking throughout the country, including in remote rural areas 6) What is the level of ICT skills of people working in the HIS? Note: this relates to people other than dedicated ICT staff – e.g. civil servants, scientific staff, medical staff 6) People working in the various institutions with a role in the HIS (e.g. ministry of health, public health institute, statistical office, health-care facilities) have adequate ICT skills and access to training if needed CORE SUMMARY AND SCORING SHEET 97 CORE Summary and scoring sheet Summary of the HIS assessment: HIS maturity score framework Scoring system The situation in the country is comparable to full maturity = 4 Many elements of an HIS with full maturity are in place but some work is still needed = 3 Some elements of an HIS with full maturity are in place but substantial work is still needed = 2 The situation in the country is still very deviant from full maturity = 1 Data collection: data sources Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 1 Vital statistics: registration of births and deaths and associated medical information is complete and up to date. Quality of cause- of-death information is high, and coding is done line with international standards and classifications 0 2 Health service records: a centralized EHR system is in place. Tailored aggregated datasets for secondary purposes can be extracted easily. Coverage and quality of the data collected in the EHR system are high. International classifications for coding diagnoses and interventions are integrated. Health insurance data have high coverage and quality, and include ICD-10 codes 0 3 Disease registries: a national-level population- based cancer registry is operating according to international standards. Registries for other major chronic diseases are in place; if not, robust morbidity estimates from other sources are available. An electronic surveillance system for infectious diseases is in place, with real- time data. Information on notifiable diseases according to country-specific legislation and international obligations (IHR) is available 0 4 Health surveys: a long-term operational plan is in place for regular conducting of national health interview and health examination surveys. The methodology applied is in accordance with international standards and requirements. Specific effort is made to ensure that hard-to-reach groups are adequately represented. Health and statistical authorities work together on survey design and implementation, and on data analysis and dissemination 0 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT98 Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 5 Health-care resources: a national human resources database is in place, with complete coverage, including the annual numbers of graduates. A national database of public and private sector health facilities is also in place, with complete coverage. Each health-care provider and facility has been assigned a unique identifier code 0 6 Health expenditure data: financial records are available on general government expenditure on health and its components. Expenditure data are collected in accordance with System of Health Accounts methodology 0 TOTAL 0 Data collection: data infrastructure, management and resources Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 1 Electronic information systems: health-care facilities and providers only use electronic patient records; there is no parallel paper record keeping. The EHR systems compile all information related to the care of an individual patient, and include additional functionalities supporting the health-care process (e.g. drug prescription module, clinical decision support tools, eSignatures). The needs of end-users have been taken into account in development of the EHR systems. Patients have access to their own data. Electronic information systems for other essential HIS data collections are in place (e.g. vital statistics, infectious disease surveillance, medicines, medical devices) 0 2 Interoperability: commonly agreed interoperability standards for the HIS and wider information systems are structurally implemented. An office or agency is in charge of defining interoperability standards. A Unique Personal Identification Number (UPIN) is issued at birth for each citizen, and this is used structurally across government services, including health services. Opportunities are available to link data sources at the subject level 0 CORE SUMMARY AND SCORING SHEET 99 Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 3 Metadata and quality control: common standards are structurally used for metadata for official (health) statistics, and these are aligned with international metadata standards. Structural quality checks are performed according to well documented protocols on the major HIS data sources (e.g. automated checks in the EHR system, manual checks, audits) 0 4 Accessibility and usability of data sources for secondary purposes: publicly funded data sources are publicly available and published as open data. Other (semi-public) data sources can be used for secondary purposes, either free of charge or for a reasonable fee covering the costs needed to produce tailored data extractions. An integrated data warehouse is operated at the national level, containing data from all relevant HIS data sources. If a central data warehouse is not in place, a data exchange platform for safe and efficient exchange between (semi-)governmental organizations is in place 0 5 Legal framework: there is a legal basis for the most important HIS data collections. Specific data standards (e.g. disaggregation levels, ICD- 10 codes) are defined in the law. Criteria for data privacy, secondary processing, sharing of information, data linkage and storage are specified. The legal framework is workable and not too restrictive. Accessibility of essential data sources for the most important HIS stakeholders is regulated by law 0 6 Resources: HIS stakeholders have adequate tools for data collection (e.g. database and data management software), and adequate manpower with limited staff turnover. HIS stakeholders have adequate capacity, i.e. staff with the right technical skills and expertise. Regular training is provided/funds are available for regular training 0 TOTAL 0 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT100 Analysis Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 1 Indicator sets: a core set of policy-relevant health indicators has been selected transparently. The core set covers all main categories necessary for informing health policy – i.e. health status, demographics, (wider) health determinants (including indicators on poverty, employment, education, environment) and health systems 0 2 Indicator updates: indicator values are regularly computed, analysed and publicly reported, according to an official publication schedule, including comprehensive metadata. The datasets used to calculate the core indicators are updated regularly, and the frequency of these updates is in line with policy needs 0 3 Comprehensiveness of analysis: regular analysis of indicator values includes historical time trends, international comparisons and disaggregations according to relevant subnational entities (e.g. regions, municipalities). Periodic future projections are made for key indicators based on population projections and epidemiological trend data 0 4 Health inequalities: regular analysis includes disaggregation of indicator values according to various dimensions of health inequalities, including gender, socioeconomic status, ethnicity and geographical differences 0 5 Alignment with international indicator sets: indicators used in the country are aligned with international reporting frameworks, most importantly with the SDGs and the Impact Framework for WHO’s GPW13. Collection methods for the data underlying the indicators are in line with international standards and recommendations. The country can fulfil all health information requests from international organizations. Experts and government representatives participate in international health information networks and projects 0 6 Resources: HIS stakeholders have adequate tools for analysis (e.g. computers, servers, analysis software) and publication (e.g. module for interactive dashboard). HIS stakeholders have adequate manpower, with limited staff turnover and adequate capacity. A multidisciplinary team works on the publication of the core indicators. Regular training is provided/funds are available for regular training on analysis skills 0 TOTAL 0 CORE SUMMARY AND SCORING SHEET 101 Health reporting Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 1 Health reports aimed at informing policy- making: regular health reports are produced by the national public health agency or comparable institution, independent of the ministry of health. The reports integrate data and information from public health, health care and other relevant policy domains, and include options for action. Scientific standards and common transparency requirements are followed. Health reports are written in easy- to-read language, and include informative visualizations and key messages 0 2 Dissemination and communication: regular health reports are publicly available. In the case of web-based applications, it is considered possible to download the visualizations and the data on which they are based. Tailored summaries/factsheets are available for different target audiences. Comprehensive communication and dissemination strategies are in place. User surveys are regularly, and website statistics are monitored and analysed regularly 0 3 Mechanisms for using health reports in the policy-making process: a formal, public and transparent procedure for using health reports in the policy-making process is in place. Parliament is informed by the ministry of health when formal health reports are published. An intersectoral governmental body to discuss (how to use) the health reports is in place, and its decisions are formally and publicly reported. Health is a standard dimension in reports of other policy sectors 0 4 Use of health reports at the health-care facility and provider level: managers and medical staff use health reports regularly to monitor and improve performance. Such reports are discussed jointly (e.g. at the department or team level). There is an open attitude among health-care staff towards measuring and monitoring performance. Health-care staff feel safe to discuss (suboptimal) quality of care and performance. For these reports, indicators that are acknowledged by (international) peers as valid and useful are used. These include patient-reported outcomes and patient-reported experiences 0 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT102 Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 5 Resources: HIS stakeholders have access to adequate tools for producing health reports (e.g. quality criteria/toolkit, evidence resources) and publishing health reports (e.g. software for using interactive graphs and options for integrating videos in online reports). HIS stakeholders have adequate manpower with limited staff turnover. HIS stakeholders have adequate capacity. A multidisciplinary team works on producing the health reports. Regular training is provided/funds are available for regular training on reporting skills 0 TOTAL 0 Knowledge translation Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 1 Familiarity with information and knowledge products: the regular information and knowledge products produced within the HIS and their publication schedules are well known by policy- makers, senior managers and other actors such as media representatives. Senior managers and policy-makers demand complete, timely, accurate, relevant and validated HIS information and know how to interpret and use it. Training or information courses on the products and their use are offered regularly 0 2 Alignment with policy-makers’ needs: exchange and integrated knowledge translation approaches are applied to make sure that information and knowledge produced meet the needs of policy-makers and other relevant users. A rapid response team/ mechanism is in place to respond quickly to ad hoc questions (e.g. when the ministry of health is looking for health information to answer questions from parliament) 0 3 Knowledge translation tools and mechanisms: producers of information and knowledge products use tools that are specifically aimed at stimulating uptake of information and knowledge in policy-making, such as policy briefs and policy dialogues. A knowledge translation infrastructure is in place that represents clear objectives for action and incorporates elements of push, pull or exchange efforts. The applied knowledge translation tools and mechanisms are institutionalized and a structural element of the health policy-making process 0 4 Resources: HIS stakeholders have adequate manpower for knowledge translation and adequate capacity. Staff have been trained in knowledge translation concepts, tools and skills and adequate budget is available for training to keep staff capacity up to date 0 TOTAL 0 CORE SUMMARY AND SCORING SHEET 103 Governance and resources Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 1 HIS legislative framework: legislation providing the legal and regulatory framework for the HIS exists, is up to date and is enforced. The legislation defines tasks related to population health monitoring, covering the whole policy cycle (problem definition, agenda setting, policy formulation, decision- making, policy implementation, policy evaluation). A legislative framework is in place for electronic information systems. National interoperability standards and other requirements are developed. Compliance, conformance and accreditation of electronic health information products and services are defined and implemented 0 2 HIS strategic plan: a formal comprehensive HIS strategic plan is in place, either as a standalone strategy, or as a clearly discernible element of a wider health/health systems strategy. The HIS strategic plan defines an HIS vision and clear goals, and defines mechanisms for monitoring progress towards these goals. HIS stakeholders were involved in its development and are involved in its M&E. The HIS strategic plan includes aspects of electronic health information services, or a separate strategic plan/policy for electronic health information services exists 0 3 HIS coordination: a formal multistakeholder coordination mechanism/body for the HIS is in place. The ministry of health is actively involved. The coordination mechanism has clear terms of reference. Meeting reports are formally established and made publicly available. The HIS coordination body has a full overview of health information needs and available health information 0 4 Monitoring HIS performance: mechanisms for monitoring HIS performance are in place. Quality assurance procedures for the different elements that are evaluated are described (e.g. acceptability, data quality, flexibility, usability, simplicity, output). A routine monitoring and evaluation process is in place for monitoring performance of the EHR systems, looking at aspects such as functionality, stability, usability, efficacy and effectiveness, as well as completeness and quality of the data entered into the system. Outcomes of quality assessments are publicly reported 0 SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT104 Description of the situation in an HIS with full maturity Description of the current situation in the country Country score Recommendations 5 HIS financing: the budgets and financing streams for the various entities and activities in the HIS are clear and transparent. The financing is sustainable and is not primarily based on third-party money/donor or project funds 0 6 HIS ICT infrastructure: adequate ICT infrastructure (e.g. computers, internet access, servers) and adequate ICT support are in place at the national level, at relevant subnational levels and at the hospital/provider level. People working in the various institutions with a role in the HIS (e.g. ministry of health, public health institute, statistical office, health-care facilities) have adequate ICT skills and access to training if needed 0 TOTAL 0 ADD-ON GPW13-EPW 105 Add-on GPW13-EPW Item Question Probing question Expectations for GPW13 Expectations for EPW Governance and overall data management GPW13_1 How are the SDGs being used at the national level? 1) Are the (health-related) SDGs used in national policies and strategies? Is this set out in a law? 1) The SDGs are formally acknowledged as policy goals at the national level (either in the form of a law or another form of formal regulation) 1) The SDGs are formally acknowledged as policy goals at the national level (either in the form of a law or another form of formal regulation) 2) Are national progress reports for the (health- related) SDGs produced? 2) Regular progress reports are produced according to an officially established publication schedule and made publicly available 2) Regular progress reports are produced according to an officially established publication schedule and made publicly available 3) Which ministry is responsible for national implementation of the SDGs? 3) One ministry (e.g. the ministry of foreign affairs) has been officially appointed to coordinate national SDG implementation, and this has been communicated clearly to other ministries and stakeholders 3) One ministry (e.g. the ministry of foreign affairs) has been officially appointed to coordinate national SDG implementation, and this has been communicated clearly to other ministries and stakeholders 4) How is the SDG work organized at this ministry (e.g. central coordinator, taskforce)? 4) Within the ministry, there is a clear organizational structure working on implementation of the SDGs. This structure has a clear mandate and adequate and sustainable resources 4) Within the ministry, there is a clear organizational structure working on implementation of the SDGs. This structure has a clear mandate and adequate and sustainable resources 5) Is an intersectoral coordination mechanism for implementation of the SDGs in place? 5) An interministerial coordination mechanism is in place, including all relevant ministries 5) An interministerial coordination mechanism is in place, including all relevant ministries GPW13_2 How is data collection for the SDGs organized? 1) Is a central body responsible for data collection for the (health- related) SDG indicators and data deliveries to international organizations? 1) A central body (e.g. the national statistics office) is responsible for data collection for the SDG indicators, and the same body is responsible for data deliveries to international organizations 1) A central body (e.g. the national statistics office) is responsible for data collection for the SDG indicators, and the same body is responsible for data deliveries to international organizations SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT106 Item Question Probing question Expectations for GPW13 Expectations for EPW 2) How are collaborations managed between the central body and the other institutions that maintain databases from which data for the SDGs are drawn? 2) There is a formal coordination structure for regular consultations between the relevant institutions 2) There is a formal coordination structure for regular consultations between the relevant institutions 3) Is data production and dissemination of statistics for the SDGs integrated into the national statistics strategy and workplans? Is there a legal basis for these activities? 3) Data production and dissemination of statistics for the SDGs are integrated into the national statistics strategy and workplans and explicitly mentioned. The national statistics strategy and workplans have a legal basis 3) Data production and dissemination of statistics for the SDGs are integrated into the national statistics strategy and workplans and explicitly mentioned. The national statistics strategy and workplans have a legal basis 4) Is an overview of data availability for the (health- related) SDG indicators available? (Note: has the country checked and formally approved the SCORE exercise from WHO headquarters?) Are there important data gaps – i.e. are data missing for any main indicators? 4) An overview of data availability for the SDG indicators is in place, and is updated regularly. The overview is publicly available. There are no main data gaps – i.e. overall data availability for the SDG indicators is high 4) An overview of data availability for the SDG indicators is in place, and is updated regularly. The overview is publicly available. There are no main data gaps – i.e. overall data availability for the SDG indicators is high 5) Is an action/ improvement plan for overcoming data gaps for the SDG indicators in place? If so, who is responsible for its execution? 5) An action plan for overcoming data gaps for the SDG indicators is in place, developed together with the interinstitutional coordination mechanism (see probing question 2). This has been approved officially by the ministry responsible for SDG implementation. The institute responsible for data collection for the SDGs (see probing question 1) is responsible for execution of the action plan 5) An action plan for overcoming data gaps for the SDG indicators is in place, developed together with the interinstitutional coordination mechanism (see probing question 2). This has been approved officially by the ministry responsible for SDG implementation. The institute responsible for data collection for the SDGs (see probing question 1) is responsible for execution of the action plan 6) Are data for the (health- related) SDG indicators regularly provided to the United Nations/custodian agencies? 6) Data for the (health- related) SDG indicators are regularly provided to the United Nations/custodian agencies according to the official delivery schedules 6) Data for the (health- related) SDG indicators are regularly provided to the United Nations/custodian agencies according to the official delivery schedules 7) Are national data for the SDG indicators published regularly online, e.g. on the website of the statistical institute? 7) National data for the SDG indicators are published regularly online and are freely accessible for everyone 7) National data for the SDG indicators are published regularly online and are freely accessible for everyone ADD-ON GPW13-EPW 107 Item Question Probing question Expectations for GPW13 Expectations for EPW GPW13_3 How is WHO’s GPW13 and EPW used at national level? 1) Are the GPW13 outcome indicators/EPW indicators, milestones and targets used in national policies and strategies? Is this set out in a law? 1) The GPW13 outcome indicators, milestones and targets have been formally integrated into national health policy 1) The EPW indicators have been formally integrated into national health policy 2) Has a national selection of the GPW13 outcome indicators/EPW indicators and milestones been defined? If so, how has this been documented? 2) A well documented national selection of the GPW13 46 outcome indicators and milestones has been defined 2) A well documented national selection of EPW indicators and milestones has been defined GPW13_4 How is data collection for WHO indicators organized? Note: This section addresses collection and delivery of data for WHO indicators in general 1) Is a central body responsible for data collection for WHO indicators and data deliveries to WHO? Is this the same body that is responsible for data deliveries to the United Nations for the SDGs (see item GPW13_2)? 1) A central body (e.g. the national statistics office) is responsible for data collection for WHO indicators and data deliveries to WHO 1) A central body (e.g. the national statistics office) is responsible for data collection for WHO indicators and data deliveries to WHO 2) Are (the improvement of) data deliveries to WHO and other international organizations part of the national statistical strategies and workplans? Is there a legal basis for these activities? 2) Data deliveries to WHO and other international organizations and, if applicable, improving completeness, quality and timeliness of these, are integrated into the national statistics strategies and workplans and explicitly mentioned. The national statistics strategies and workplans have a legal basis 2) Data deliveries to WHO and other international organizations and, if applicable, improving completeness, quality and timeliness of these, are integrated into the national statistics strategies and workplans and explicitly mentioned. The national statistics strategies and workplans have a legal basis 3) Are data for official indicators regularly provided to WHO? 3) Data for WHO indicators are regularly provided to WHO according, to agreed delivery schedules 3) Data for WHO indicators are regularly provided to WHO according, to agreed delivery schedules Availability and usability of data and indicators for the GPW13 impact measurement framework (with a focus on the 46 outcome indicators) and the EPW GPW13_5 What is national data availability for the (national selection of the) GPW13 outcome indicators and EPW indicators? 1) Are data available for the (national selection of the) outcome indicators and milestones? 1) Data are available for all indicators/milestones that have been selected nationally 1) Data are available for all indicators that have been selected nationally 2) Is an overview of data availability for the (national selection of the) outcome indicators in place? 2) An overview of data availability for the (national selection of the) GPW13 46 outcome indicators is in place; it is regularly updated and is publicly available 2) An overview of data availability for the (national selection of the) EPW indicators is in place; it is regularly updated and is publicly available SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT108 Item Question Probing question Expectations for GPW13 Expectations for EPW 3) What are the main problems with data availability? Where are the main data gaps? (Note for assessor: identify issues – if applicable – related to specific types of data sources (e.g. health surveys, EHRs); this will facilitate cross- referencing with the outcomes of the overall HIS assessment.) 3) There are no significant data gaps, and all indicators can be disaggregated according to the necessary stratifiers for identifying health inequalities 3) There are no significant data gaps, and all indicators can be disaggregated according to the necessary stratifiers for identifying health inequalities 4) Is an action/ improvement plan for overcoming data gaps for the (national selection of the) outcome indicators in place? If so, who is responsible for its execution? 4) An action plan for overcoming data gaps for the (national selection of the) GPW13 46 outcome indicators is in place, officially approved by the ministry of health. The institute responsible for data collection for WHO indicators (see item GPW13_4) is responsible for execution of the action plan 4) An action plan for overcoming data gaps for the (national selection of the) EPW indicators is in place, officially approved by the ministry of health. The institute responsible for data collection for WHO indicators (see item GPW13_4) is responsible for execution of the action plan GPW13_6 Are there any issues with the usability of the data to compute regular figures for the (national selection of the) GPW13 outcome indicators / EPW indicators? 1) Are there any issues related to accessibility of the data? Can the necessary data (easily) be used for statistical purposes? If not, what are the main reasons? 1) All the data necessary for computing the (national selection of the) 46 outcome indicators are accessible and can be used for population health statistics 1) All the data necessary for computing the (national selection of the) EPW indicators are accessible and can be used for population health statistics 2) Are there any issues with the frequency of data updates in relation to the requirements/duration of the GPW13 framework and EPW (e.g. the data are only updated once every five years)? 2) Data for the (national selection of the) 46 outcome indicators are available according to the frequency recommended/ requested by WHO 2) Data for the (national selection of the) EPW indicators are available according to the frequency recommended/requested by WHO 3) Are there any issues with the quality of available data for the (national selection of the) outcome indicators? (Note for assessor: identify issues – if applicable – related to specific types of data sources (e.g. health surveys, EHRs); this will facilitate cross- referencing with the outcomes of the overall HIS assessment.) 3) Data for the (national selection of the) 46 outcome indicators are complete, collected according to the relevant international standards (e.g. classifications) and robust 3) Data for the (national selection of the) EPW indicators are complete, collected according to the relevant international standards (e.g. classifications) and robust ADD-ON GPW13-EPW 109 Item Question Probing question Expectations for GPW13 Expectations for EPW GPW13_7 The GPW13 outcome indicators / EPW indicators are existing SDG and WHO indicators. Are these indicators already in use? 1) To what extent are these indicators already in use? Are they included in (official) national indicator sets? 1) The (national selection of the) 46 outcome indicators are already in use and integrated with official national indicator sets. This means that well established data collection and indicator computation methods and schemes are in place 1) The (national selection of the) EPW indicators are already in use and integrated with official national indicator sets. This means that well established data collection and indicator computation methods and schemes are in place 2) Which institutions are computing the indicators, and at what frequency? 2) One central body (e.g. the national statistical office) is responsible for computing the indicators. If the indicators are computed by multiple institutions, one institution has central oversight. Indicators are computed according to the frequency recommended/ requested by WHO 2) One central body (e.g. the national statistical office) is responsible for computing the indicators. If the indicators are computed by multiple institutions, one institution has central oversight. Indicators are computed according to the frequency recommended/ requested by WHO 3) Are the indicators published regularly; if so, on which platforms/in which products? See also item GPW 13_10, probing question 2 3) Updated indicator values are published regularly in publicly accessible platforms/reports 3) Updated indicator values are published regularly in publicly accessible platforms/reports GPW13_8 Are there any issues with the usability of the existing GPW13 outcome indicators / EPW indicators in use? 1) Are the national indicators well aligned with the SDG/WHO definitions? 1) The national indicators are well aligned with the SDG/WHO definitions, and can be disaggregated according to the necessary stratifiers for identifying health inequalities 1) The national indicators are well aligned with the SDG/WHO definitions, and can be disaggregated according to the necessary stratifiers for identifying health inequalities 2) Are detailed and up-to- date metadata available for the indicators? 2) Detailed and up-to-date metadata are available in accordance with international metadata standards. The metadata are updated regularly 2) Detailed and up-to-date metadata are available in accordance with international metadata standards. The metadata are updated regularly SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT110 Item Question Probing question Expectations for GPW13 Expectations for EPW Analysis, reporting and knowledge translation GPW13_9 What kind of analyses are used for the (national selection of the) GPW13 outcome indicators / EPW indicators, and how are these published? 1) Are comparisons in time (historical time trends, future projections) and space (international comparisons, subnational comparisons) made for the indicators? Are the indicators disaggregated according to common stratifiers for assessing health inequalities? See also item GPW 13_10, probing question 1 1) Comprehensive analyses, including time trends, future projections, international and subnational comparisons and analyses of health inequalities are performed for the (national selection of the) 46 outcome indicators 1) Comprehensive analyses, including time trends, future projections, international and subnational comparisons and analyses of health inequalities are performed for the (national selection of the) EPW indicators 2) How/for what purposes are the health reports used (e.g. national SDG progress reports, regular reports on progress towards national health goals, in the framework of an annual statistical workplan)? See also item GPW 13_7, probing question 3 2) The rationale/aim of the reports is clear, as are their intended audiences 2) The rationale/aim of the reports is clear, as are their intended audiences 3) Are these health reports publicly available? 3) The reports are publicly available 3) The reports are publicly available 4) What (formal) mechanisms exist for using these indicator/health reports in the health policy- making process? 4) A formal, public and transparent procedure for using the health reports in the policy-making process is in place (e.g. once every X years the public health institute makes a health report for the ministry of health, at its request, and the ministry formally and publicly reports on how it has used the information in this report). Parliament is informed by the ministry of health when formal health reports are published 4) A formal, public and transparent procedure for using the health reports in the policy-making process is in place (e.g. once every X years the public health institute makes a health report for the ministry of health, at its request, and the ministry formally and publicly reports on how it has used the information in this report). Parliament is informed by the ministry of health when formal health reports are published 5) Are the reports also used to inform intersectoral policy-making? 5) An intersectoral governmental body to discuss (how to use) the health reports is in place, and its decisions are being formally and publicly reported 5) An intersectoral governmental body to discuss (how to use) the health reports is in place, and its decisions are being formally and publicly reported ADD-ON GPW13-EPW 111 Item Question Probing question Expectations for GPW13 Expectations for EPW GPW13_10 Are policy-makers able to use the available indicator analyses and health reports? 1) Do available analyses meet the needs of policy-makers? See also item GPW 13_9, probing question 1 1) The available analyses provide the evidence base that policy-makers need 1) The available analyses provide the evidence base that policy-makers need 2) Do policy-makers and other relevant stakeholders know which health reports are available, and do they consider them to be easily accessible? 2) The reports and their publication schedules are well-known by policy-makers and other stakeholders, and these users know how to access the reports 2) The reports and their publication schedules are well-known by policy-makers and other stakeholders, and these users know how to access the reports 3) Do policy-makers and other relevant stakeholders know how to use the health reports? 3) Policy-makers and other stakeholders know how to interpret and use the reports 3) Policy-makers and other stakeholders know how to interpret and use the reports 4) Are specific tools to stimulate uptake in policy- making of the information and knowledge derived from the indicator analyses used, such as policy briefs and policy dialogues? 4) The producers of the reports use tools specifically aimed at stimulating uptake of information and knowledge in policy-making, such as policy briefs and policy dialogues 4) The producers of the reports use tools specifically aimed at stimulating uptake of information and knowledge in policy-making, such as policy briefs and policy dialogues Resources GPW13_11 Are enough resources available for data collection for, analysis of and reporting on the (national selection of the) 46 GPW13 outcome indicators / EPW indicators? 1) Do HIS stakeholders have adequate financial resources for data collection for, analysis of and reporting on the indicators? 1–4) HIS stakeholders have adequate financial, human and technical resources for data collection for, analysis of and reporting on the (national selection of the) 46 outcome indicators 1–4) HIS stakeholders have adequate financial, human and technical resources for data collection for, analysis of and reporting on the (national selection of the) EPW indicators 2) Do HIS stakeholders have access to adequate tools for data collection for, analysis of and reporting on the indicators? 3) Do HIS stakeholders have adequate manpower for data collection for, analysis of and reporting on the indicators? 4) Do HIS stakeholders have adequate expertise and adequate means for capacity-building for data collection for, analysis of and reporting on the indicators? SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT112 Add-on Infectious disease surveillance Item Question Probing question Expectations Data collection InfDisSurv_1 Does the country have any infectious disease/pathogen surveillance systems in place? 1) Please describe the infectious disease/pathogen surveillance system in place 1) The country has nationwide mandatory and regular passive notification of priority infectious diseases. In addition, active sentinel surveillance is conducted for selected infectious diseases targeted for eradication or elimination (such as measles). The surveillance system must cover 80% of level 3 administrative units in the country to be considered “nationwide” 2) Please describe sources used for infectious disease surveillance and mechanisms of collecting data (paper, fax, electronic, phone). If no electronic reporting systems for infectious diseases exist, are there plans to implement electronic reporting in the future? 2) Infectious disease surveillance reports are submitted from physicians at the community and primary care level, hospitals, laboratories and private health-care facilities. The country has an interoperable, interconnected electronic reporting system in place, which is sustained by the government. Laboratory data are reported electronically (from reporting laboratories or laboratory networks) 3) What is the proportion of reporting sources, at subnational and local levels, with a standardized reporting system? 3) The proportion of subnational/local reporting facilities with a standardized reporting system is high. The surveillance system should cover 80% of level 3 administrative units in the country to be considered “nationwide” 4) Is a national database of infectious diseases available? 4) An infectious diseases database that facilitates storage, management, extraction, querying and sharing of data among stakeholders is in place. Laboratory data are received electronically (from reporting laboratories or laboratory networks) and linked to epidemiological and clinical data 5) Does the country conduct sentinel surveillance? If so, for which syndromes, infectious diseases or pathogens? 5) Examples are given of sentinel surveillance system in place (e.g. for influenza-like illness and severe acute respiratory infections) 6) Are any special arrangements in place to cover high-risk yet hard-to- reach populations/settings such as prisons and refugee camps? Please provide examples 6) Where routine notification procedures are not feasible, are disease surveillance programmes are in place, such as syndromic surveillance, that specifically target high-risk, vulnerable and hard-to-reach populations. Examples of such programmes are provided 7) Is there a GIS database integrated with the infectious disease surveillance system? Please list practical examples of the different uses of GIS resources within the ministry of health or other agencies and partners 7) Concrete examples of GIS systems and their uses are given (e.g. assessing spatial distribution of a disease /mapping, monitoring results of disease surveillance, planning/ targeting interventions) ADD-ON INFECTIOUS DISEASE SURVEILLANCE 113 Item Question Probing question Expectations InfDisSurv_2 Is an early warning system in place? 1) Please describe the early warning system components and methods for data collection, reporting, analysis and dissemination 1) All potential sources of event-based surveillance data are listed, health risks have been prioritized and specific procedures to detect any abnormal occurrence from the normally observed frequency of phenomena as early as possible are in place. Responsibilities for notification and investigation of immediately reportable diseases and events are established and documented. Standardized case definitions and standard operating procedures (SOPs) are available. Examples are provided 2) Are any innovative data sources (big data, Internet searches, social media) in use for early warning/ syndromic surveillance? 2) Internet tools such as social media data, search engine data and others are used to track epidemics and to create patterns and rules for an early prediction 3) Please describe specific indicators and thresholds developed to alert for action 3) Specific indicators and thresholds have been developed to alert for action. These may report on absolute numbers (e.g. one case of polio or one case of rabies) or absolute rates over the past 52 weeks (e.g. more than two cases per 100 000 population for meningococcal meningitis). Trends and signals are analysed and assessed in real time 4) Is event-based surveillance of communicable diseases in place? If not, are there plans to develop event- based surveillance? 4) Mechanisms exist to capture unusual or public health events from non-routine sources in the health system. Reporting sources have been identified and prioritized, and event definitions exist. An event assessment team/unit responsible for assessing each reported event and triggering an immediate response is in place. Both health-care workers and the community are involved in event- based surveillance. Capacity to conduct a preliminary outbreak investigation exists at the subnational level 5) Please describe data sources used by event-based surveillance systems and mechanisms of collecting data (paper, fax, electronic, phone) 5) Data sources for event-based surveillance include reports and rumours from the community, health-care workers, media representatives, hotline or other informal sources. A rumour log or database for registration of suspected public health events from informal sources exists and works 24 hours per day, 7 days a week at the national and subnational levels 6) How many events were detected, verified, assessed and investigated at national, intermediate and local levels during the last 12 months? 6) Specific examples are given of detected, assessed and verified events and actions triggered. An event assessment team/unit responsible for assessing each reported event and triggering an immediate response is in place at the national and subnational levels SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT114 Item Question Probing question Expectations 7) Is event-based surveillance integrated with the regular indicator- based surveillance system? 7) Epidemic intelligence capacity exists to collate and integrate event-based and indicator-based (from traditional systems) surveillance. Combined data from both types of surveillance are analysed and interpreted to detect acute public health events and/or risks. Feedback is provided to all reporting sites and other stakeholders 8) Does the country conduct syndromic surveillance? Please describe it. If not, are there plans to develop syndromic surveillance? 8) Syndromic surveillance is conducted. Prioritization of syndromes under surveillance and objectives of the system are documented (i.e. providing early warning of seasonal illness, epidemiological information for seasonal outbreaks or specific public health incidents). Surveillance is conducted for at least three core syndromes indicative of potential public health emergencies according to international standards (severe acute respiratory syndrome, acute flaccid paralysis, acute haemorrhagic fever, acute watery diarrhoea with dehydration and jaundice with fever) 9) Which data sources are used and how are data reported? Are data collected specifically for syndromic surveillance? 9) Data sources used sufficiently inform and identify syndromes in the population. These may include EHRs, school absenteeism, retail sales and over-the-counter medication. Data are electronically reported and available in real time 10) Is the syndromic surveillance system integrated with the regular indicator-based surveillance system? 10) The syndromic surveillance system is integrated with the traditional indicator-based surveillance system. The two systems are interoperable, and able to exchange data and subsequently present that data to users 11) Does the current syndromic surveillance system support the early warning function? Please provide examples 11) Examples are given of how the syndromic surveillance system has been useful for threat detection in the past. Event data are appropriately stored, managed, analysed, interpreted and disseminated InfDisSurv_3 Is a surveillance system in place for priority zoonotic diseases/ pathogens? 1) Please describe the surveillance system in place for priority zoonotic diseases/pathogens. Otherwise, are plans in place to develop zoonotic disease/pathogen surveillance? 1) Zoonotic surveillance systems are in place for five or more zoonotic diseases/ pathogens of greatest public health concern. Information exchange between animal/ wildlife surveillance units, human health surveillance units and other relevant sectors regarding potential zoonotic risks and urgent zoonotic events is timely and systematic. Relevant documentation, including a list of zoonotic priority pathogens for public health, is provided 2) Are priority zoonotic disease/ pathogen surveillance data integrated with the indicator-based surveillance system? 2) Priority zoonotic disease/pathogen surveillance data are integrated with the regular indicator-based surveillance system ADD-ON INFECTIOUS DISEASE SURVEILLANCE 115 Item Question Probing question Expectations InfDisSurv_4 Does the country conduct antimicrobial resistance surveillance? 1) Please describe the routine surveillance system for resistance detection 1) Designated laboratories detect and report all priority resistant pathogens. Records of tests conducted and results are available (by setting, population etc.). Detection and reporting of antimicrobial resistance is done to international recommended standards by both veterinary and human sectors 2) Is surveillance of infections caused by antimicrobial-resistant pathogens being performed? 2) Designated sentinel sites conduct surveillance of infections caused by all priority antimicrobial-resistant pathogens 3) Are surveillance data integrated with the regular indicator-based surveillance system? 3) Surveillance data are integrated with the regular indicator-based surveillance system Analysis InfDisSurv_5 How is analysis of infectious disease surveillance data performed? 1) Which stakeholders are involved in the design and implementation of surveillance strategies, and infectious disease surveillance data analysis? 1) Health and statistical authorities work together on the design and implementation of surveillance strategies, and infectious disease surveillance data analysis. Cooperation mechanisms exist between the public health institute, statistics office, universities and others. Examples are provided 2) What tools are in use for analysis of infectious disease data? Please provide examples 2) Examples are given of electronic platforms that integrate, synthesize and visualize information pertaining to disease surveillance. Computer-generated customized reports, tables, charts, maps and metadata are implemented 3) Are data collection methods and analytical approaches (e.g. calculation of indicators) in line with international standards and recommendations? Please provide examples 3) Data collection methods are in line with international standards and recommendations, and the country fulfils all health information requests from international organizations 4) Have pre-defined epidemic threshold values been established for the priority infectious diseases? Please provide examples 4) Pre-defined action thresholds for selected indicator diseases (epidemic-prone, vaccine-preventable, others of public health importance) are established. Examples are provided 5) Are appropriate denominators calculated and used? What is the source of the denominators? Please provide examples 5) Average population of a defined geographical area for a calendar year is obtained from appropriate sources. Catchment population is calculated when necessary. Examples are provided SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT116 Item Question Probing question Expectations 6) Is analysis of surveillance data over time performed routinely for selected indicator diseases (i.e. epidemic-prone, vaccine-preventable and other relevant infectious diseases)? Please describe specific data outputs 6) Patterns of disease occurrence over time are displayed, and trend analysis is performed (according to documented SOPs). Crude numbers or standardized rates of disease, as appropriate, are calculated for selected diseases by time period. Epidemic curves are available for diseases and outbreaks. Tables and graphs, along with interpretations, are produced. Examples are provided for diseases under surveillance 7) Is analysis of surveillance data conducted routinely by person and cause/risk factor/mode of transmission (for selected indicator diseases such as epidemic-prone, vaccine-preventable and other relevant infectious diseases)? Please describe specific data outputs 7) Data analysis is performed by person (e.g. age, sex, race), biological characteristics (e.g. immune status), acquired characteristics (e.g. marital status), activities (e.g. occupation, leisure activities, use of medications/tobacco/ drugs) or the conditions in which they live (e.g. socioeconomic status, access to medical care), according to documented SOPs. Standardized rates of disease are calculated for selected diseases by person. Tables and graphs, along with interpretations, are produced. Examples are provided 8) Is analysis of surveillance data routinely conducted by place (for selected indicator diseases such as epidemic-prone, vaccine-preventable and other relevant infectious diseases)? Please describe specific data outputs 8) Occurrence of infectious disease is described by relevant geographical location (i.e. place of diagnosis or report, birthplace, site of employment, school district, hospital unit or recent travel destinations) according to documented SOPs. Tables and maps, along with interpretations, are produced. Examples are provided 9) Are routine infectious disease surveillance data used for analytical studies? Please provide examples 9) Infectious disease data are used to quantify the association between exposures and outcomes and to test hypotheses about causal relationships. Examples are shown Health reporting InfDisSurv_6 Is there capacity for reporting and publication of infectious disease surveillance information? 1) Are surveillance reports for each of the infectious diseases produced regularly, and by whom? 1) Regular surveillance reports are produced by the national public health agency or comparable institutions, independent of the ministry of health. Scientific standards and common transparency requirements are followed in the production of the reports 2) Is information from the infectious disease surveillance system available for population health monitoring? 2) Information from the infectious disease surveillance programme (e.g. number of cases of measles or influenza) is readily available for use in population health reports, where it can be placed in a broader context 3) How comprehensive are infectious disease surveillance reports? 3) Epidemiological reports are written in easy- to-read language, use a combination of texts and informative visualizations and include key messages, analysis outputs, interpretation of results and options for actions (i.e. assessment of preventive measures) ADD-ON INFECTIOUS DISEASE SURVEILLANCE 117 Item Question Probing question Expectations 4) What format do the infectious disease surveillance reports use? 4) Regular health reports use standard reporting formats and tools, preferably publicly available web-based reports that use interactive visualization tools that allow generation of tables, graphs/charts, maps and infographics. It is possible to download the visualizations and the data on which they are based. Tailored summaries/factsheets are available for different target audiences 5) Are infectious disease surveillance reports publicly available? 5) Infectious disease surveillance reports are publicly available and readily accessible 6) What kind of communication and dissemination strategies are used for these reports? 6) Comprehensive communication and dissemination strategies are in place, including mass media, social media, online health (information) platforms, newsletters, email messages, presentations and lectures. Active feedback is also part of the communication and dissemination strategy, including follow- up of social media. Experts talking to the mass media have received relevant training 7) Is it known to what extent policy- makers and other users (such as media, patient organizations, NGOs and professional organizations) actually use infectious disease surveillance reports? 7) User surveys are conducted regularly. Website statistics are monitored and analysed regularly, and results are publicly available 8) Is event-based surveillance information communicated effectively? 8) Event-based surveillance information is routinely reported (and can be integrated with indicator-based infectious disease surveillance bulletins). The reports should contain the following aggregated data: number of events reported, number of events assessed, number of events confirmed, number of events not confirmed (i.e. false rumours), and source of report (e.g. health-care facility, community leader). When responses to events are undertaken, brief descriptions are included in the surveillance bulletin to highlight the link between reporting and response. Feedback is given to all reporting sources, people and organizations involved in event response 9) Is antimicrobial resistance surveillance information effectively communicated? 9) Surveillance information is routinely reported (and can be integrated with indicator- based infectious disease surveillance bulletins). The report should contain at least the following: proportion of antimicrobial- resistant pathogens among specimens or isolates, results from participation in international external quality assessment rounds of the national reference laboratory, incidence of infections caused by antimicrobial-resistant pathogens at sentinel sites (community and hospital-acquired), proportion of facilities adhering to best practices for health-care-associated infections (if known) and percentage of antibiotics administered appropriately (if surveyed) SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT118 Item Question Probing question Expectations InfDisSurv_7 Are infectious disease information requirements from international organizations met? 1) What proportion of outbreaks is notified to WHO, the United Nations Food and Agriculture Organization (FAO) or the World Organisation for Animal Health (OIE) within 24 hours? 1) IHR national focal points, OIE delegates and World Animal Health Information System national focal points are notified within 24 hours of all events that could constitute a public health emergency of international concern (PHEIC). The country has demonstrated ability to identify a potential PHEIC and file a report to WHO within 24 hours; similarly to the OIE for a relevant zoonotic disease. Mechanisms are in place for effective risk communication during a public health emergency 2) Does the country participate in international infectious disease/ virological surveillance projects or activities? Please provide examples 2) Examples are provided of collaborations with international infectious disease/ virological surveillance projects (such as the FluNet global web-based tool for influenza virological surveillance) 3) Are the recommended indicators for monitoring progress towards the SDGs in the WHO European Region available? 3) The following SDG indicators are reported (disaggregated by time, person and place): number of new HIV infections per 1000 uninfected population, by sex, age and key populations; tuberculosis incidence per 100 000 population; hepatitis B incidence per 100 000 population InfDisSurv_8 What mechanisms exist for using infectious disease/pathogen surveillance reports in the policy- making process? 1) What is the mechanism for using infectious disease/pathogen surveillance reports in the health policy-making process? 1) There is a formal, public and transparent procedure for using infectious disease reports in the policy-making process (e.g. once every X years the public health institute makes a health report for the ministry of health, at its request, and the ministry formally and publicly reports on how it has used the information in this report). Parliament is informed by the ministry of health when formal infectious disease reports are published 2) Are the reports also used to inform intersectoral policy-making, and do other policy sectors also include information on infectious disease /pathogens for informing their policies? 2) An intersectoral governmental body that discusses (how to use) the health reports is in place, and its decisions are formally and publicly reported. A “one health” approach is taken for design and implementation of public health programmes, policies, legislation and research Knowledge translation InfDisSurv_9 Do relevant stakeholders know which infectious disease/pathogen information is available and are they able to access and use it efficiently? 1) Is infectious disease/pathogen information regularly demanded by users like senior managers and policy-makers? 1) Infectious disease/pathogen information and its publication schedules are well known by policy-makers, senior managers and other actors such as media representatives. Senior managers and policy-makers demand complete, timely, accurate, relevant and validated infectious disease/pathogen information and know how to interpret and use it 2) Are any support mechanisms available for training relevant actors on how to interpret and use the products? 2) Training or information courses on the information and knowledge products and their use are offered regularly ADD-ON INFECTIOUS DISEASE SURVEILLANCE 119 Item Question Probing question Expectations 3) Does the infectious disease/ pathogen information produced within the HIS meet the needs of the policy-makers? 3) Regular exchange sessions take place to identify the information needs and to assess the timeliness and usefulness of the formats with policy-makers and other relevant users. The outcomes of these sessions and implemented changes are documented and reported. Exchange and integrated knowledge translation approaches are applied to make sure that information and knowledge produced meet the needs of policy-makers 4) What kind of communication mechanisms are in place if there are questions about infectious diseases or pathogens, or ad hoc requests for this type of information? 4) A rapid response team/mechanism is in place to respond quickly to ad hoc questions (e.g. when the ministry of health is looking for health information to answer questions from parliament). After-care is a structural element in the communication and dissemination plans for health information and knowledge products. A regularly conducted user survey is applied to identify the usability of health information and knowledge products Governance and resources Legislation InfDisSurv_10 What legal and regulatory framework supports implementation of infectious disease surveillance? 1) Have infectious diseases and pathogens been prioritized for surveillance? When was the list of priority diseases last updated? 1) Evidence exists of prioritization of infectious diseases and pathogens for surveillance. A list of priority diseases and case definitions is available. The list is updated as necessary and includes epidemic-prone diseases like cholera, diarrhoea with blood, measles, meningitis, plague, viral haemorrhagic fevers, yellow fever, severe acute respiratory syndrome and diseases targeted for eradication/elimination 2) Are objectives for disease surveillance clearly stated and documented? 2) Clear objectives for surveillance of infectious diseases and pathogens have been defined (as evidenced by relevant documentation) 3) What is the legal and regulatory framework for implementation of infectious disease/pathogen surveillance activities? 3) A national legal framework (laws and regulations) for infectious disease/pathogen surveillance is available and updated regularly 4) Is the legal and regulatory framework for implementation of infectious disease/pathogen surveillance activities in line with relevant international standards? 4) A national legal framework (laws and regulations) for infectious disease/pathogen surveillance is in line with relevant international standards 5) What is the level of compliance with the available legal and regulatory framework as regards infectious disease/pathogen surveillance and potential challenges for implementation? 5) Compliance with the surveillance legislation is satisfactory and no major issues are reported SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT120 Item Question Probing question Expectations 6) Are surveillance standards and guidelines available? What proportion of surveillance units has standards and guidelines for infectious disease surveillance? 6) Surveillance standards and guidelines for priority infectious diseases and pathogens are available to all reporting sites. Standard case definitions are available to all reporting units 7) Is a national comprehensive plan to strengthen surveillance and laboratory capacity of antimicrobial- resistant pathogens in place? 7) A comprehensive plan to combat antimicrobial resistance is agreed and implemented at the national level and is updated regularly. In addition, yearly reporting on progress towards implementation takes place at the international level 8) Is there a memorandum of understanding or other agreement between public health and security authority entities at the national level with regard to detection, investigation and response to public health emergencies? 8) SOPs or emergency response plans that include security authorities are in place to link public health and security authorities (e.g. law enforcement, border control, customs) during a suspected or confirmed public health emergency if required 9) Is a legal and regulatory framework in place for public health, animal health and security authorities to make decisions on coordination, communication and reporting of a potential PHEIC to WHO, FAO and OIE? 9) A mechanism for multisectoral/ multidisciplinary coordination, communication and partnerships to make decisions on reporting of a potential PHEIC is available and documented by legislation, protocols and/or guidelines Policies, planning and evaluation InfDisSurv_11 Is a plan of action to enhance the infectious disease/pathogen surveillance system in place? 1) Are strategic and operational plans for implementing and strengthening infectious disease/ pathogen surveillance in place? 1) Strategic and operational plans for implementing and strengthening communicable disease surveillance and response systems are in place. These are updated every 3–5 years for strategic plans and annually for operational plans 2) How many activities have been implemented according to the plans? 2) Annual activity reports are available. Most activities have been implemented according to the plans InfDisSurv_12 How are the infectious disease/pathogen surveillance strategy and its coordination conducted? 1) Is there a M&E system? 1) All reporting sites are routinely monitored. The infectious disease/pathogen surveillance system has been evaluated in the last 5 years 2) Are roles and responsibilities well defined and documented? 2) Roles and responsibilities are well defined and documented at each level of the infectious disease/pathogen surveillance system 3) Is a coordinating body in place? 3) A surveillance unit is in place at the national level for coordination of infectious disease/ pathogen surveillance activities 4) Are any mechanisms in place to link public health and security authorities during a suspected or confirmed public health emergency? 4) SOPs or emergency response plans that include security authorities are in place. Mechanisms exist that link public health and security authorities (e.g. law enforcement, border control, customs) during a suspected or confirmed public health emergency if required. Information reports are shared regularly with security authorities ADD-ON INFECTIOUS DISEASE SURVEILLANCE 121 Item Question Probing question Expectations InfDisSurv_13 Are regular advanced training options available for people with roles in infectious disease/pathogen surveillance? 1) What proportion of surveillance units has completed training modules? 1) A high proportion of surveillance units complete training modules every year 2) What proportion of staff/ health-care workers is trained on surveillance of infectious diseases? 2) A high proportion of staff/health-care workers completes training modules on infectious disease/pathogen surveillance every year 3) What proportion of epidemiologists is trained on infectious disease/pathogen surveillance in each district? 3) All epidemiologists are trained on infectious disease/pathogen surveillance 4) What proportion of staff/ health-care workers has received a refresher course on surveillance of infectious diseases? 4) A high proportion of staff/health-care workers has undergone refresher training modules on infectious disease/pathogen surveillance in the past two years 5) Has training been conducted jointly (at an intermediate (regional) or national level) with both public health and security authorities on topics related to infectious disease/ pathogen information sharing and joint investigations of public health emergencies? 5) The country has organized and conducted training jointly with public health and security authorities on topics related to infectious disease/pathogen information sharing and joint investigations of and responses to public health emergencies InfDisSurv_14 Have the attributes of the infectious disease surveillance system been evaluated? 1) What is the frequency of surveillance reports (immediate, weekly, monthly)? Please provide examples 1) The frequency of notifications is appropriate, based on the surveillance objectives and the epidemiology of the disease. Examples are provided 2) What proportion of surveillance units submitted surveillance reports (immediate, weekly, monthly) to the next level up on time? 2) A high proportion of surveillance units submitted surveillance reports (immediate, weekly, monthly) to the next level up within the expected time frame. (1: highly adequate if 90% or more; 2: adequate if 75–89%; 3: present but not adequate if 25–74%; 4: not adequate at all if less than 25%.) Poor timeliness has triggered corrective actions (documented) 3) How is absence of underreporting (external completeness) of diseases to the infectious disease surveillance system evaluated? 3) Evaluation of external completeness is planned and implemented at least for priority diseases requiring a high level of completeness. Possible methods for evaluation of external completeness include reviews of a sample of all surveillance data, reviews of medical records and capture– recapture studies. Underreporting has triggered corrective actions (documented) 4) What procedures are in place to assess internal completeness? What is the proportion of surveillance reports with no missing required information (variables)? 4) A high proportion of completeness of variables is reported. (1: highly adequate if 90% or more; 2: adequate if 75–89%; 3: present but not adequate if 25–74%; 4: not adequate at all if less than 25%.) A mechanism is in place to monitor internal completeness. Low internal completeness has triggered corrective actions (documented) SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT122 Item Question Probing question Expectations 5) How representative is the surveillance system? What is the geographical coverage? Do all clinical settings participate? Is the prevalence of urban versus rural reporting sites evenly distributed? Are minority populations reached by the surveillance system? 5) Cases notified are representative of the population under surveillance for a defined geographical area, based on surveillance objectives 6) Is the external validity of the data generated by any infectious disease surveillance system evaluated? What methods are used? Please provide examples 6) Designated staff perform validity checks. Possible external data sources that can be used to validate surveillance data are listed. Retrospective (i.e. concordance evaluation with medical records) or prospective evaluation methods are used. Methodology for validity checks and results (e.g. duplicate records, case misclassification, invalid data collection formats) are described thoroughly. Examples are provided. The procedures are documented with relevant protocols and SOPs 7) Is the internal validity of the data generated evaluated? What methods are used? Please provide examples 7) Validity checks such as cross-tabulations and distribution analysis of variables are routinely performed to identify outliers and inconsistent values. A descriptive analysis of data is compared to expected values, based on available background surveillance data, to identify any deviations. Examples are provided. The procedures are documented with relevant protocols and SOPs 8) Are data flow and operating procedures sufficiently clear to all relevant stakeholders? 8) Implementers and users of the system rate the procedures as sufficiently simple and clear. The amount and type of data collected, managed and analysed is appropriate. The number of organizations involved in receiving case reports from a surveillance unit and the time spent on system maintenance is kept to a minimum while still meeting surveillance objectives 9) Is the surveillance system able to collect, manage and provide data properly and consistently over time without failure? 9) Data and methodology are consistent over time and available when needed. The system is fully operational at all times. Dedicated resources (human and material) for running and maintaining the surveillance activities are secured in the long term 10) Has the usefulness of the data generated by any infectious disease surveillance system been evaluated? What methods were used? Please provide examples 10) An inventory of actions taken as a result of information generated by the infectious disease surveillance system has been made. Usage of the system is documented (users of information, actions). Examples of actions are given and might include timely response to health hazards, informing target groups for vaccination, hypotheses to stimulate research, hospital bed capacity increases ADD-ON INFECTIOUS DISEASE SURVEILLANCE 123 Item Question Probing question Expectations Infrastructure InfDisSurv_15 Are sufficient resources available for maintaining and operating the infectious disease/ microbiological datasets? 1) Do stakeholders have adequate tools for maintaining and operating the infectious disease/ microbiological datasets? 1) Infectious disease stakeholders have adequate tools (e.g. database and data management software) 2) Do stakeholders have adequate manpower for maintaining and operating the infectious disease/ microbiological data collections? 2) Infectious disease stakeholders have adequate manpower, and staff turnover is limited 3) Do stakeholders have adequate capacity for maintaining and operating the infectious disease/ microbiological data collections? 3) Infectious disease stakeholders have adequate capacity – i.e. staff with the right technical skills and expertise 4) Did reporting sites have appropriate surveillance forms or other means of notification – such computers, internet connection – at all times during the last six months? 4) Reporting sites have appropriate surveillance forms and all necessary material resources (computers, internet connection, etc.). There are no structural problems with the availability of reporting forms/other resources InfDisSurv_16 Do infectious disease surveillance stakeholders have adequate resources for producing and publishing infectious disease reports regularly? 1) Do stakeholders have access to adequate tools for infectious disease/pathogen surveillance reporting? 1) Infectious disease surveillance stakeholders have access to adequate tools for producing infectious disease/pathogen surveillance reports (e.g. quality criteria/toolkit, evidence resources) and publishing health reports (e.g. software for using interactive graphs and options for integrating videos in online reports) 2) Do stakeholders have adequate manpower for producing and publishing regular infectious disease/pathogen surveillance reports? 2) Infectious disease surveillance stakeholders have adequate manpower, and staff turnover is limited 3) Do infectious disease surveillance stakeholders have adequate capacity for producing and publishing regular infectious disease/pathogen surveillance reports? 3) Stakeholders have adequate capacity, i.e. staff with the right skills and expertise (such as statisticians, epidemiologists, GIS experts, data visualization experts, writers/editors, communication experts). A multidisciplinary team works on producing the infectious disease/pathogen surveillance reports. Regular training is provided/funds are available for regular training on reporting skills FAO: Food and Agriculture Organization; OIE: World Organisation for Animal Health; PHEIC: public health emergency of international concern; SOP: standard operating procedures. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT124 Add-on HRH ID Question Probing question Expectations Data collection HRH_1 What is the required scope of data collection for HRH strategic planning? 1) Is the private sector included in strategic health workforce planning, or is the focus solely on the public sector? 1) A clear definition of the scope of HRH and strategic planning is in place – e.g. a list of what is in and out of scope. This may be held in ministry of health documentation 2) Does the scope include both the unregulated and the regulated health sector? The unregulated health sector typically means the social care sector, as the health sector will be regulated. If this is the case, is there a formal definition of what is meant by “unregulated” and documentation on how data are to be collected and quality assessed? 2) If included, a formal definition of what is meant by “unregulated” and documentation on how data are to be collected and quality assessed are in place 3) Are HRH data collected and aggregated across public and private sector health providers (if within the scope)? 3) A master list of private sector health education and service providers is available, and information is collected on a regular basis 4) Is a formal definition of all staff groups within the scope available? 4) A formal and agreed definition of health occupations (also referred to as workforce groups) is in place, including medical doctors and nursing and midwifery professionals. This may be held in a health workforce registry or database. The International Standard Classification of Occupations (4-digit)a is the recommended standard to follow, and is widely used in HRH planning. Note: It is important to have a clear understanding of the level of detail needed. Health workforce planning to medical specialty level should be a separate activity from a wider health workforce strategy. 5) Are public health and social care included in strategic health workforce planning? 5) Formal and agreed definitions of the health, public health and social care workforces, and the specific workforce groups in each area, are agreed 6) Is a formal statement available of the scope of practice of the workforce groups defined above? 6) Formal and agreed definitions of professional roles exist – for example, from a regulatory body or licensing organization. These should be consistent between the public and private sectors ADD-ON HRH 125 ID Question Probing question Expectations 7) Are informal data sources and expert opinions used for data that may be too difficult or expensive to collect? 7) A list exists of people who can be asked, individually or working as a group, to provide expert knowledge across areas such as: • population health needs • health workforce labour markets • how health needs may change in the future See also item HRH_13 8) Are HRH data sources managed appropriately? 8a) A list of all HRH data sources includes the data owners, how the data may be used, how to get permission to access the data (if not open access), how the data are maintained, how often they are updated, when they were last updated, known gaps and issues, and an assessment of data quality (this should be an assessment by the country, not the owners of the data) 8b) Effective data management is in place for this list, covering data entry, maintenance, training, skills and resources 9) Has the completeness of the information available to date on indicators and plans for improvement been assessed? 9a) An assessment has been made of what information is available, data quality and any gaps 9b) A plan is in place to improve data quality and close any gaps HRH_2 Do the data sources provide sufficient understanding of today’s population? 1) Are country-level population data available that facilitate making of forecasts of sufficient detail for health workforce planning? 1) The data should be provided by item Data sources_1 in the Core module and include the current population by gender in 5-year age bands. Birth and death rates are needed for population forecasting HRH_3 Do the data sources provide sufficient understanding of today’s population health needs? 1) Are data available to facilitate understanding of which areas of health are driving demand for health workforce activity? 1) The data should be provided by items Data sources_2, 3, 4 and 5 in the Core module. Aggregated data on the primary causes of death, disability and premature death across the population, and the key drivers of death and disability are available. Potential data sources include: • population health surveys • national vital statistics • global surveys – for example, by WHO, the World Bank and the Institute for Health Metrics and Evaluation 2) Are health needs categorized into areas that are distinct and non- overlapping, and that cover all health workforce activities? 2) A list exists of categories of demand for health services, in terms of the different kinds of population needs that workforce activity is addressing – e.g. long-term conditions, infectious diseases, maternal and perinatal, oral health SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT126 ID Question Probing question Expectations HRH_4 Do the data sources provide sufficient understanding of today’s active health workforce stock? 1) Are data available on the current health workforce stock? 1) The data should be provided by item Data sources_8 in the Core module. As described, coverage should be complete of the number of health workforce professionals across all workforce groups, working in the public and private sectors. If possible, this should cover both salaried and self-employed health workers. Potential data sources include: • health workforce registries or databases • health facilities databases • aggregate data from health facilities – e.g. employment records from hospital and care facilities • district or regional HISs • professional bodies and their registers of qualified members • labour force surveys • insurance or pension fund registries 2) Are data available on whether health workers work full time or part time? 2a) For employed health workers, the data should include the number of people (the headcount), and their full-time equivalent to show whether they are working full time or part time 2b) A clear definition of how the full-time equivalent is calculated should be available for each workforce group, including the average annual hours worked in a full-time job 3) Are data available on the age profile of the workforce? 3) Data are available for each employed workforce on their age and gender, in 5-year bands 4) Is information on the health workforce stock held in a single database or multiple databases? If the latter, can the data be aggregated? 4a) A list of the databases or data sources that hold this information is available 4b) It is common for health workforce data to be held across multiple databases, including employer or hospital-level data. If this is the case, a process should be in place to aggregate the data, taking into account data quality and any gaps. These should be highlighted HRH_5 Do the data sources provide sufficient understanding of health workforce education and training? 1) Are data available on the flow of health workers from training into the workforce? 1a) A master list of accredited health workforce and training institutions is available, which includes accreditation of private training institutions and continuing education ADD-ON HRH 127 ID Question Probing question Expectations 1b) Data are available on each workforce (within the scope) sufficient to calculate future workforce stocks. The information needs to include, as a minimum: • duration of education and training • yearly intake or admission of students into training • training fill rate (proportion to which a training course is filled) • exit or drop-out from training • numbers in training, by year of training • graduation rate from education and training programmes Potential data sources include: • national statistical service • ministries of labour, employment and education • ministry of health databases on education and statistics • education and training institutions HRH_6 Do the data sources provide sufficient understanding of the effort spent by the health workforce on meeting health needs? 1) See item HRH_3 above: if categories of population health need exist, are data available to estimate the service delivery effort for each workforce group, across one or more categories? 1) Item Data sources_3 in the Core module should cover data on medical and health procedures and services, by workforce group. Potential data sources include: • health service records • health insurance records • professional surveys and census records • health-care provider records Note: 100% of time spent in delivering services, for each workforce, should be attributable to one or more of these groups. This is trivial for groups like dentists (100% on oral health) and midwives (100% on maternal and perinatal health) but much harder for family doctors/ general practitioners and hospital doctors/ medical specialists. Nevertheless, the country needs this information for effective HRH planning HRH_7 Do the data sources provide sufficient understanding of health finances? 1) Are data available on the cost of education and training? 1a) Data are available on total expenditure in higher education and on health workforce education 1b) Data are available on the average cost of training, per student, for each workforce, and the average tuition fees (if appropriate) 1c) The above information should cover both the public and private sectors. Potential data sources include: • integrated financial management systems –e.g. held by the ministry of finance, labour, education or health • government financial statistic departments • national health accounts • education and training provider financial records SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT128 ID Question Probing question Expectations 2) Are data available on health workforce salaries? 2) Data are available on health workforce salaries, including starting or entry-level salaries, and the average salary for each workforce group. This needs to cover both salaried and self-employed workers. Potential data sources include: • systems and records as listed above • payroll data • income tax data • general labour force and health workforce surveys 3) Are data available on the average salaries in sectors that may be in competition for students for training, or the trained health workforce, including other countries? countries? 3) Data are available on the average salaries of comparable or competing sectors for health workforce staff HRH_8 Do the data sources provide sufficient understanding of the health labour market? 1) Are data available on entries to and exits from the health labour market, and labour market shortages? 1a) Data are available on the number of graduates who enter each workforce each year 1b) Data are available on the number of qualified or registered health workers, the number of employed health workers and the number of unemployed health workers 1c) Data are available on the number of trained health workers who join each workforce each year, including those training domestically and those trained in another country 1d) Data are available on the number of staff who exit each workforce group each year, distinguishing between voluntary exits and involuntary exits (e.g. retirement or ill health) 1e) Data are available on health workforce vacancies, for each workforce group 1f) Data are available on health workforce migration to and from other countries 1g) Data are available on health workforce flows between regions – e.g. urban and rural HRH_9 Do the data sources provide sufficient understanding of health working conditions? 1) Are sufficient data available to understand the working conditions of the health workforce, to identify which may be under most pressure? 1a) Data are available on the job satisfaction of health workers. Potential data sources include: • labour force surveys • member surveys of health professional bodies 1b) Data are available on workforce sickness and absence rates. Potential data sources include: • health workforce registries or databases • health facilities databases • district or regional HISs • social insurance databases 1c) Data are available on attacks on health workers. Potential data sources include: • labour force surveys • member surveys of health professional bodies • police records ADD-ON HRH 129 ID Question Probing question Expectations HRH_10 Are metadata standards available to support data collection? 1) Are metadata standards defined and agreed? 1a) Metadata standards are in place to support data collection for an agreed set of HRH indicators (see item HRH_11) and the wider data required 1b) The metadata standards should include data quality standards, managing the collection of data from multiple sources, identifying key data gaps and the process of data validation Analysis HRH_11 Is a defined set of HRH indicators in place? 1) Do these indicators follow (international) standards? 1) The indicators are derived from the WHO Minimum Data Set for Health Workforce Registryb and follow a documented standard, for example the National Health Workforce Accountsc 2) Are HRH indicators used to support completion of international health-care data requests? 2a) The indicators are used to support data requests, such as from the Organisation for Economic Co-operation and Development, the Statistical Office of the European Union (Eurostat) and WHO Regional Office for Europe Joint Data Collection on non-monetary health- care statistics 2b) There is documented evidence of completed data returns, with any gaps highlighted 3) Are the indicators updated on a regular basis? 3) Documented processes are in place for data gathering, data quality assessment, data validation and updating of the HRH indicators 4) Are these indicators clearly linked to health policy, planning and decision- making demands? 4) Documented evidence of the indicators being used in the HRH decision-making process is available 5) Are the indicators used to make comparisons with other countries and geopolitical areas? 5) Documented evidence of country comparisons being used in the HRH decision- making process is available 6) Are these indicators monitored on a regular basis to track progress? 6) A documented review and monitoring process is in place HRH_12 Are analyses performed to determine any current gaps between health workforce supply and requirements? 1) Is health workforce supply reviewed against population health requirements? 1) Estimates are available of the size of any gaps between current workforce numbers and the desired requirements SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT130 ID Question Probing question Expectations HRH_13 Is an approach used to elicit judgements about the future? 1) Is an approach used to estimate how much key parameters may change in the future (or to estimate current situations where no – or no appropriate – data are available)? 1a) A formal and rigorous method is used – e.g. structured expert judgement, also called expert elicitationd 1b) The method description includes defining the expertise required, selecting suitable experts, providing training, conducting the elicitation and reporting the results 1c) The method captures the estimated value, the uncertainty and the reasoning of the experts who made this judgement HRH_14 Are analyses performed to determine the future population size and age? 1) Are population health projections available for the next 10 years, by age and gender, or are they produced? 1a) Agreed country-level population forecasts are available – e.g. from a national or international statistical body, used in national planning. These are updated regularly, at least on an annual basis 1b) A published and peer-reviewed forecast methodology is in place HRH_15 Are analyses performed to determine the future health needs of the population? 1) Has it been estimated how future health needs might change from today, for the expected future? 1a) Estimates are available of the percentage change in the categories of health needs, as defined in Data sources_3 in the Core module 1b) A documented method for how these estimates have been produced is documented – e.g. as discussed in item HRH_13 HRH_16 Are analyses performed to determine the future supply of health workers? 1) Have projections of future health workforce supply been produced? 1a) Workforce supply projections are available for all workforce groups 1b) A documented method is available for how these estimates have been produced – e.g. using stock and flow modellinge 1c) An analysis has been performed on the parameters used in the modelling to assess their data quality, and to identify those that have the greatest impact on the calculated results (high sensitivity) 1d) A documented process is in place to improve the data quality for those parameters which have high sensitivity and low data quality HRH_17 Are analyses performed to determine any future gaps or mismatches between health workforce supply and population health needs? 1) Has an estimate been made of how the health workforce needs to change in order to meet the projected population and future health needs? 1a) Estimates of the health workforce required to meet future population health needs and changing population size and age are available 1b) A documented method for how these estimates have been produced is available 1c) Health workforce groups with the largest gaps or mismatches have been identified ADD-ON HRH 131 ID Question Probing question Expectations Health reporting HRH_18 Are HRH reports produced to inform the policy, strategy and decision-making processes? 1) Has an HRH workforce planning and decision-making cycle been defined? 1) A documented process and timescale are in place for HRH planning and decision-making 2) Are HRH reports produced to inform HRH planning and decision-making? 2) Documented processes and mechanisms are in place for the timely use of HRH reports 3) Do the reports provide a comprehensive picture of key HRH issues? 3a) The reports clearly identify population health needs and trends, pressures on the health workforce, and gaps and mismatches in workforce numbers, skills and competences 3b) The reports address issues of ethnicity, diversity and potentially disadvantaged or under-represented groups 3c) The reports present decision-makers with the inherent uncertainty in modelling health workforce supply and future requirements, and do not simply provide forecast estimates Knowledge translation HRH_19 Are the HRH knowledge products used effectively by policy and decision-makers? 1) Are decision-makers aware of the HRH knowledge products available? 1) Published schedules of products are available, along with communications on their availability and use 2) Do decision-makers have the necessary knowledge to use HRH knowledge products effectively? 2) Regular training and literature on the HRH knowledge products and their application to policy, strategy and planning are available. Feedback is captured after these sessions and used to improve both the products and the training 3) Have decision-makers been asked what HRH knowledge products they need to make effective decisions? 3) Key decision-makers and stakeholders have been identified and interviewed to gain insight into their work, the information they require and the application of HRH knowledge products to this work 4) Have the HRH knowledge products been used to inform decision-making? 4) The HRH knowledge products are cited in the decision-making process and reports Governance and resources HRH_20 Are sufficient resources available to support the implementation of HRH indicators? 1) Is a plan in place to strengthen the collection of data to support HRH indicators? 1) Support is available from the ministry of health and other relevant departments for implementation of an HRH indicator set and collection of HRH information 2) Are sufficient resources available for implementation of the indicators? 2) Sufficient resources are available to support data collection and validation, including establishing metadata standards 3) Are sufficient resources and expertise available to analyse and report on the HRH indicators? 3) A skills assessment has been done, knowledge gaps identified and plans put in place to close them SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT132 ID Question Probing question Expectations HRH_21 Is a national policy and strategy in place on the generation and use of HRH information to support policy- making and planning? 1) Are policies and procedures available to ensure that health workforce priorities are considered in broader government action across the health sector and in related sectors, such as education? 1a) A governance structure, with representation from relevant sectors and stakeholders, is in place, along with a working group (or equivalent) to support implementation 1b) Evidence of joined-up thinking and planning should be clear HRH_22 Are regulations in place that support effective HRH policy-making and planning? 1) Is private sector health provision regulated? 1) Evidence of policies to engage and regulate the private sector should be available, including education and training and dual working. Lack of regulation may lead to training of health workers that does not match their defined scope of practice, or the private sector being in competition with the public sector for staff 2) Is the international flow of health workers regulated? 2) Evidence of policies to regulate the international labour market should be available. Lack of regulation may mean the loss of trained staff to other countries or missed opportunities to attract health workers into the country 3) Are regulations in place that support the health workforce? 3) Evidence of policies to improve the well- being and skills of health workers should be available. These might include regulations on working hours and conditions, social protection, minimum wages and attacks on health workers. Lack of regulation may mean potential staff not joining key health professions, or trained staff leaving their profession a International Standard Classification of Occupations 2008 (ISCO-08): structure, group definitions and correspondence tables. Geneva: International Labour Organization; 2008 (https://www.ilo.org/global/publications/ilo-bookstore/order-online/books/WCMS_172572/lang-- en/index.htm). b Minimum data set for health workforce registry. Geneva: World Health Organization; 2015 (https://www.who.int/hrh/statistics/ minimun_data_set/en/). c National health workforce accounts (NHWA) [website]. Geneva: World Health Organization; 2020 (https://www.who.int/hrh/statistics/nhwa/ en/). d For an explanation of these approaches see European Food Safety Authority. Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal. 2014;12(6):3734. doi:10.2903/j.efsa.2014.373. e For example, Willis G, Cave S, Kunc M. Strategic workforce planning in health care: a multimethodology approach. Eur J Oper Res. 2018;267(1):250–63. ADD-ON NCD MONITORING 133 Add-on NCD monitoring ID Question Probing question Expectations Data collection NCD_1 What is the status of registration of NCD deaths in the vital statistics of the country? Questions marked with an asterisk (*) in this section are either already included in the core module or are related to it. They are provided here for complementarity of the NCD module sections 1) Is the coverage of deaths registration complete and are there important differences nationally? * Core section Data sources_2 1) Death registration is high in Europe, but some country regions have more limited access. Knowing such differences makes it possible to make adjustments. Nearly 100% completeness in all country regions is expected in a mature registration system 2) What is the proportion of deaths with medical certification of death? * Core section Data sources_2 2) A high, nearly 100% proportion of medically certified deaths provides an indication of the maturity and quality of the mortality registration system. It also speaks indirectly of an organized system of physician training to address needs 3) What version of ICD is used for classification of causes of death: a complete list or a short list? What is the proportion of ill-defined deaths or unknown causes * Core section Data sources_2 3) Use of older ICD versions and short/ aggregated lists limit some analyses of NCD causes of death and, thus, of their relative importance. Full implementation of ICD- 10/11 for coding medically certified deaths should be the goal, while a proportion of <10% of deaths coded with ill-defined causes or unknown causes would indicate an optimum/ high quality for the use of death data 4) Is the coding of causes of death carried out manually or is it automated using computer- assisted programs? 4) To improve quality, international comparability and use of mortality data, minimizing coding variations and errors, national information systems are implementing coding based on standardized and expert-developed software, such as the Automated Classification of Medical Entities, the Mortality Medical Indexing, Classification, and Retrieval and the Translation of Axis (all developed by the United States National Center for Health Statistics) or IRIS (a language-independent coding system developed by European Union Member States, the United States and others in collaboration with Eurostat, which, according to the WHO's Family of International Classifications, became the international standard). In addition to the software and its rules, this requires having a national dictionary to process context- specific data SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT134 ID Question Probing question Expectations 5) Is the mortality data system capturing more than one cause death, and if so, are data recorded being used for informing NCD policy? 5) Availability and use of both the underlying cause of death (following the WHO ICD-10 and other editions) and multiple causes of death (all causes appearing in the certificate, and classified according to record axis rules) indicates an improved system and a potential for more comprehensive analyses of NCDs, as diseases and their risk factors tend to cluster in individuals. This is relevant information for understanding disease and classified patterns, for preparing reporting assessments to inform policy to develop appropriate public health and individual level packages of health interventions 6) In addition to cause, age and sex, what other data are recorded from the death certificate? Can data be linked to other information systems? 6) Geographical, social and economic stratifiers allow identifying differences between population subgroups, information necessary for priority setting, various situation and trends analyses and health planning, and is shown in reports and plans. Integrated information allows improved monitoring and reporting with specific topic data or more comprehensive health situation assessments 7) Is there an electronic system for collecting national data? How often it is compiled at the central level? 7) The availability of a centralized electronic system for national data collection indicates the capacity for more accurate and agile compilation and distribution of data 8) How often and by whom is the quality of the data evaluated? 8) Information is aimed at determining if this is an independent and regular annual process that maintains/improves data quality 9) Are time series data re-run every year and published with appropriate notes for changes? 9) This indicates needed attention given to the quality of data to assess trends, including adjustments due to late reporting or changes of target population and informing users of the process to avoid confusion from data and value changes from year to year. Adjusted available data and reports, including notes on data changes, are expected 10) Are data delivered to WHO or other agencies for international comparisons? 10) Commitment and regular submission of data to WHO and other international agencies, with high data quality standards, are a good indication of a mature system. Participating in the data correspondent network with data review/changes in codification and classification helps to improve quality ADD-ON NCD MONITORING 135 ID Question Probing question Expectations NCD_2 What is the status of health service records with regard to NCDs in the country? 1) What is the coverage of NCD information in health service records, national or otherwise? 1) An important indication of the usefulness and representativeness of non-fatal NCD health events information in the health system is that the higher and broader the coverage, the better. It provides indirect information on the burden on health impacts and on health services 2) Do health service records include separate primary health care from hospital data? * Core section Data sources_3 2) Health service data are expected to be separate sets as the representativeness and type of demands vary considerably, but both are essential for determining their complementarity in coverage and functions 3) Do health services records involve the private sector? What is the coverage? 3) Involvement of the private sector provides information on the comprehensiveness and completeness of data coverage. More mature systems tend to have high coverage 4) How often are data reported to and compiled at national level? Are they collected and reported in electronic or paper format? * Core section Data sources_3 4) Data are expected to be collected by electronic means at the source, though it may be paper-based and then electronic in more remote areas. Thus, annual data are compiled at national levels after a few months’ delay, becoming available for health analyses 5) Are inpatient and outpatient data reported separately? 5) Having separate health services data will make it possible to carry out more appropriate and comprehensive assessments, which is an indication of mature systems 6) How complete is the information provided with regard to consultations, admissions, discharges, procedures, other? * Core section Data sources_3 6) Provide indication of the comprehensiveness and effort required by the health system to address NCDs and the risk factors. Mature systems will involve more parameters to assess the effectiveness and quality of processes 7) Are primary health-care facilities’ information systems integrated and able to offer cardiovascular risk stratification for the management of patients at high risk of heart attack and stroke? 7) Integrated parts of the information system on clinical, laboratory and risk factor information are able to produce a >30% or higher probability of cardiometabolic risk among people older than 40 years, which are essential for monitoring a WHO global monitoring framework target indicator on health capacity and population levels of cardiometabolic risk. Information generated is regularly used for assessing and managing individuals with specific health interventions and counselling. Thus, more mature systems are capable of collecting different pieces of information, integrating them, calculating risk scores and determining the proportion of the high-risk population to meet this requirement 8) Is there a regular review process to assess the quality of the data? * Core section Data sources_3 8) It is important to determine who conducts the review, how often it is conducted, and what indicators are used. Health service records tend to have high coverage and potential biases are known. An independent group using international standard indicators in a well-defined process indicates maturity SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT136 ID Question Probing question Expectations NCD_3 Which NCD registries are maintained in the country? 1) What diseases and risk factors, such as cancers, diabetes, injuries, acute coronary events and others, are included in the registries? * Core section Data sources_4 1) This indicates the effort dedicated to NCDs and other diseases or health problem registries and how challenging their operation is in a country, as only a few have genuinely documented well-established and integrated processes 2) What is the registries’ coverage, national or otherwise? Are they population-based, hospital-based or other? 2) Information relevant for understanding the potential biases or limitations of data in terms of representativeness, with mature systems being national and subnational population- based systems generating regular information and reporting every 1–2 years 3) Do these registries include public and private health sectors? 3) Information on the completeness coverage and the representativeness of the data, expecting to have all possible sources in a more mature system 4) What kind of information is collected and how complete is it? 4) Information on the appropriateness of the data for NCD analysis. Mature systems would collect comprehensive, complete and representative data 5) If cancer is one of them, what indicators and classifications are used for them? 5) These data indicate the level of development of the registry and its resources and of the potential quality of the information generated for decision-making. In addition to WHO ICD-O-10 classification, the capacity to generate data on disease survival and on types, sites and stages (tumour, nodes, metastases classification) of main cancers is expected in a mature system 6) What kind of indicators are generally available from these registries? 6) Depending on the capacity to generate robust and diverse indicators, such as prevalence and incidence, disease staging and survival times since diagnosis or interventions. In a mature system, this informs of the effectiveness to meet objectives and functions of the system 7) How often and by whom is the quality of the registries evaluated? 7) It is important to determine who conducts the review, how often reviews are conducted and what indicators are used. An independent group using international standard indicators in a well-defined process indicates maturity ADD-ON NCD MONITORING 137 ID Question Probing question Expectations NCD_4 How are NCD screening programmes organized in the country? 1) What NCDs are included in the screening programmes (for example, breast, cervical, colon cancers, other types)? * Core section Data sources_6 1) Insight on the health information needs and use of information resources for screening diseases. They should involve only diseases or health problems for which there is an internationally accepted validated process for diseases where there is a demonstrable benefit of screening 2) Are data on the coverage of screening programmes being regularly collected? * Core section Data sources_6 2) There are specific national databases and indicators showing that programmes are reaching regular levels that have the potential to impact early/preclinical detection and action to reduce the occurrence of disease 3) Are these population-based or opportunistic screening programmes? 3) In general, population-based programmes are more mature and generate better-quality data, which includes closer representation of the target population 4) Are there regular assessments of the quality of the screening programmes? 4) Timely and regular annual assessments and reports with recommendations are available from population-based NCD programmes from a mature system NCD_5 What regular surveys on main NCDs (cardiovascular disease, cancer, diabetes, chronic respiratory disease or mental health) and their key risk factors (tobacco use, alcohol consumption, unhealthy diet, insufficient physical activity, overweight and obesity, high blood pressure, raised blood sugar, raised blood cholesterol) are conducted in the country? 1) What surveys on main NCDs (main groups or individual diseases) and/or key risk factors (behavioural and biological) have been carried out in the country in the past 10 years (provide the year the last one of each type was conducted? 1) This provides an indication on the effort and coverage of information on the recognized main NCDs and their key risk factors in the country. However, if results come from independent samples, it only makes it possible to determine the frequency of NCDs and risk factors and a limited profile of their occurrence in the population. It may also suggest more limited governance and coordination and an opportunity to improve their level. Frequency and regularity will allow assessment of their capacity to detect change and time trends of survey indicators. Addressing these aspects indicates a mature system 2) Are they integrated (5 disease groups x 5 groups of risk factors) NCD surveys? [Integrated means having all the information collected from each participant.] If not, indicate what diseases and risk factors are covered 2) In contrast to single-disease or risk factor, surveys should be aiming for the integration of measuring main NCDs and common risk factors, which helps to determine their frequency and patterns at individual and population levels, information for designing appropriate packages of health interventions. Integrated disease and risk factor monitoring and assessment reports would provide an indication of the maturity SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT138 ID Question Probing question Expectations 3) What are their target populations, including age/sex and any specific target groups (e.g. migrants)? 3) NCDs and risk factors are generally monitored among adult populations (18–69 years). However, NCDs also being to develop early in life, as surveys of schoolchildren and adolescents have shown. Recently, the burden of NCDs among the elderly population and the potential for modification has been recognized and has been receiving more attention. Thus, the availability of integrated reports on NCDs along the life-course indicates a more mature system 4) Are the surveys population- based or health services-based with national or subnational representativeness? * Data sources_7 4) This helps to determine survey scope and population representativeness and generalization of results, with the population-based ones with subnational representativeness providing a better approach 5) Do the surveys follow a probability complex sample design? Is there national institutional capacity to design, conduct and monitor such NCD surveys? * Data sources_7 5) This provides an indication of the scientific validity of the data collected, their representativeness of the target population and the robustness of indicators generated to support the interpretation of the results. The availability of a national team (often interinstitutional with academia, national statistics office and a public health institute), with the capacity to design and monitor such surveys, provides an indirect indication of the quality of the collections, analysis and reports 6) Are the surveys regular? How often they are conducted (years)? 6) This information is essential for determining the feasibility of trend assessments. It is also an indication of the relative importance given to NCDs in the country, as well as planning and availability of resources. Technical and financial sustainability and a plan for future surveys indicates a mature system 7) Do the surveys involve objective or only self-reported measures of health, for example, health examination or only health interview surveys? 7) Given the potential biases from culture and personal experience on self-perception and memory of health status, the use of more objective measures to provide valid quality estimates of NCDs should be promoted. Conduct of surveys with more objective health measurements would indicate a more mature system 8) How are the surveys conducted, for example, face-to- face, by telephone or using other means? 8) The means for conducting surveys may provide indications of their quality and reliability, the preferred means being face-to- face, which reduces some potential issues but also increases cost. A balance of approaches, with limited phone use, may be achieved in a more mature system as quality and reliability of data increase ADD-ON NCD MONITORING 139 ID Question Probing question Expectations NCD_6 What is the status of adoption of electronic health record (EHR) systems in the country? 1) Is there a national legislation on EHR, what does it cover and is there a plan for implementation? * Core section Data infrastructure_2 and_3 1) A national legal framework setting the content of the EHR and a detailed approach for its development and implementation, including the roles of professionals and interoperability of different health information systems, making it possible to analyse NCD data. An approved and implemented legislation and a development plan including different stakeholders and their roles would be expected 2) What is the national coverage of the EHR system and does it include the private sector? * Core section Data infrastructure_2 2) This information is essential for understanding the scope of NCDs in the national system, including the populations that are represented in the data. A relatively high coverage, including the private sector, and a regular report on this matter, indicates a more mature system is in place 3) Is the EHR a national, unified, standardized and interoperable system? * Core section Data infrastructure_2 and _3 3) EHR systems often evolve from diverse initiatives and resources, making it difficult to have a national system. A documented plan and use of standards and interoperable designs and system language are essential for addressing the limitations, and further indicate a mature system 4) What subsystems, such as diagnoses, procedures, laboratory, pharmacy or appointments, are included in the EHR, feeding into NCD assessments? 4) Integrated NCD information from the EHR may allow comprehensive and closer monitoring and management of NCDs and their risk factors, also helping to identify quality-of-care issues and required practice adjustments. Availability of a system compiling data and regular monitoring reports would indicate a mature process 5) Is it possible to extract full individual NCD records over time to determine progression, effectiveness of treatment, and quality of care? Is this performed regularly? 5) EHRs may become an essential tool for NCD management and follow-up of individuals across referral levels of the health system and along continuity of care, independent of their geographical location. Here, the aim is not for individual assessments and case management but for the capacity to assess the overall performance of the health system at monitoring and addressing health-care needs and universal health coverage 6) Who can review the EHRs and how can they be accessed? 6) This provides an indication of the accessibility, transparency and potential accountability of the EHR, where all stakeholders should have the opportunity to check data. Within a framework of privacy and confidentiality, the more open and accessible the system, the better 7) Is there a regular review process to assess the quality of NCD data in the EHR? 7) The quality of EHR contents, processes as well as effectiveness of their use to answer disease management issues and necessary adjustments. Timely and regular annual assessments and reports with recommendations are available from a mature system SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT140 ID Question Probing question Expectations NCD_7 What data sources on NCD health-care resources exist in the country? 1) Is there a national information system or survey with data on the availability of a basic list of NCD medicines in health-care facilities? It includes: insulin, aspirin 100mg, metformin, thiazide diuretics, angiotensin- converting enzyme inhibitors, calcium channel blockers, beta- blockers, statins, oral morphine, steroid inhaler, bronchodilator, sulfonylureas, benzathine penicillin, nicotine replacement therapy. * Data sources_8 and Data infrastructure_2 1) Availability of essential medicines in over 80% of health settings is a WHO global monitoring framework target indicator. Availability of a basic list of drugs is an essential intervention for the prevention and management of NCDs, aimed at reducing the prevalence of risk factors, more severe disease and disability or death. A mature NCD HIS system should be able to generate data and allow assessments of trends and needs of such lists of essential medicines on a regular annual basis 2) Is there a national information system or survey on the availability of basic NCD technologies for early detection, diagnosis and monitoring in health-care facilities? Includes: measurement of weight and height, blood glucose, glucose tolerance test, HbA1c, dilated fundus exam, foot vibration perception, urine strips for glucose and ketones, blood cholesterol, blood pressure, urine strips for albumin, peak flow and spirometry. * Data sources_8 and Data infrastructure_2 2) Availability of basic technologies for diagnosis and care in over 80% of health settings is another WHO global monitoring framework target indicator. Availability of a basic list of health technologies is an essential intervention for early detection, prevention and management of NCDs, aimed at reducing the prevalence of risk factors, more severe disease and disability or death. A mature NCD HIS system should be able to generate data and allow for assessments of trends and needs of such health technologies on a regular annual basis 3) Is there a national information system or survey on the availability of basic NCD treatment procedures? Includes: retinal photocoagulation, renal replacement therapy by dialysis or by transplantation, coronary bypass, stenting, and thrombolytic therapy for acute myocardial infarction. * Data sources_8 and Data infrastructure_2 3) This complements information on the resolutive capacity of the health-care system and access to basic technologies for addressing NCD morbidity and disability, an aspect that is a determinant of survival and quality of life. A mature NCD HIS system should be able to generate data and allow assessments of trends and needs of such medical procedures on a regular annual basis 4) Do systems or surveys cover primary health care and hospitals of both public and private sectors? 4) This information indicates the representativeness of information from primary health care and hospital settings, the optimum being full coverage of health-care levels and sectors ADD-ON NCD MONITORING 141 ID Question Probing question Expectations NCD_8 Is there a central repository of national and subnational NCD-related policies and legislation in the country? 1) Are there data in an information system on NCD- related policies and progress on their implementation levels, including tobacco, alcohol, salt, sugary beverages and fat product content in the food industry, environment, work environment, health services regulations and guidelines? 1) Although with some limitations, information on NCD-specific policy availability and implementation is relevant for assessing and understanding progress on the implementation and improvement of prevention and reduction of prevalence of NCDs and their risk factors. Monitoring and reporting of NCD-related policies and indicators address the country’s commitment to WHO and the United Nations on the prevention and control of NCDs. The availability of a time-bound national searchable document repository on NCD policies would be an expected goal of NCD HIS 2) Is the country implementing national information subsystems collecting data on cost-effective policies for reducing main NCDs or their risk factors? What are the levels of their implementation? 2) There is a WHO list of 19 progress monitor targets indicators (on governance, reduction of tobacco affordability through fiscal policies and increased warnings, alcohol regulations on age and taxes, on reducing salt and trans fat consumption and preventing marketing of products to children, on improving physical environments and increasing physical activity and on availability of basic NCD medicines and technologies in health settings) that is reported every two years to WHO and the United Nations. A national system allowing for the assessment of the situation and trends of basic NCD policy implementation indicators would be an expected goal of NCD HIS 3) Is there an information system collecting health system data on which government-approved, evidence-informed national guidelines are available for managing main NCDs? What is their implementation level in the country? 3) This indicates whether there are national NCD guidelines prepared, distributed and when were the latest implemented and functioning at national levels. Availability of a system would be an expected goal NCD_9 What data sources of interest to NCDs from other domains are available in the country? 1) Are there other population- based surveys of use jointly with NCD data? For example, Surveys on Living Conditions, Labour Force and Employment. * Core section Data sources_10 1) Social and economic data contained in other population-based surveys may be used jointly with other health data for health situation analysis, such as those on health inequalities or vulnerable groups. Monitoring and assessment reports using linked data with some disaggregations, involving different stakeholders, would be the expected result of an integrated effort 2) Are there other non-health- based surveys of use jointly with NCD data? For example, on alcohol, tobacco, sugary drink sales and taxes, or on determination of salt and trans fat content in food products 2) These types of surveys are essential for complementing information on health-related behaviours and understanding their levels, patterns and determinants. Monitoring and assessment reports using linked data with some disaggregations, carried out by specific stakeholders, would be the expected result of an integrated effort SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT142 ID Question Probing question Expectations 3) Are there other information systems and tools for use jointly with NCD data? For example, GIS and socioeconomic (household crowding, housing quality, economic/social deprivation) and environmental (e.g. green spaces, proximity to roads and other pollution sources) data 3) Additional information systems with national or broader coverage and diverse granularity (as per spatial subunit measurements) allow for combining of information with health information on NCDs, to assess their ecological risk associations, further informing and providing evidence for decision-making. Monitoring and assessment reports using geographically linked data with some socioeconomic disaggregations, carried out by specific stakeholders, would be the expected result of an integrated effort 4) Are there national experiences on the use of e-health for NCD monitoring, surveillance, management and evaluation, including social media, digital trails or other big data sources? 4) This provides an indication of innovative thinking on sources of NCD data and developmental progress on the search for feasible tools for improving monitoring and surveillance of NCDs and their risk factors. Research and implementation reports and assessment of implementation process, involving different stakeholders, would be expected NCD_10 What is the status of an NCD-related electronic information system in the national HIS? 1) Is there an independent national data collection system and database for NCDs in the ministry of health? Is it available to other stakeholders and users of the national HIS? * Core section Data infrastructure_2 1) A national integrated information system of NCDs with data and indicators from different health-related domains is in place and available to different users, including policy- makers, stakeholders and the public at large. The system has transparent and open access to policy-makers, stakeholders and general users 2) Is the national system connected to other information sources, making it possible to compile different data to generate specific NCD indicators? * Core section Data infrastructure_2 2) A national NCD information system considers the potential for merging or linking records from different entities for improved indicator generation and analysis. NCDs and other sources share interoperable systems that allow for data linkages at different levels 3) Do these NCD indicators follow international standards on definition, measurement methods and metadata? *Core section Data infrastructure_2 3) The national NCD-related electronic system uses data and indicator standards, agreed definitions, measurement methods and metadata that provide an indication of validity, international comparability and overall quality NCD_11 Is there a dedicated responsible unit for NCD surveillance in the country? 1) Is there a unit/department in the ministry of health dealing with the overview of the design, management and operation of the NCD-related health information systems? Is it part of the unit with overall responsibility for health information, digital health, NCDs, or another unit (specify which)? * Core section 1) There is a national department/agency institutional at the MoH with the capacity to design, manage and operate the NCD information system in place. It has sufficient and diverse human resources to deal with all operations, including data collection, monitoring, analysis and reporting, and an independent sustainable budget from national resources ADD-ON NCD MONITORING 143 ID Question Probing question Expectations 2) Are the NCD data from different sources regularly used and/or compiled into a single database for public health monitoring, surveillance and evaluation, and disease management purposes? 2) There is a single NCD database that provides an indication of the NCD integrated data quality collection process, with its monitoring, surveillance, evaluation and health-care management purposes in place NCD_12 Is there an adequate legal framework for the NCD data collections? 1) Is there a specific legal framework for NCD surveillance and monitoring in the country? * Core section Resources for data collection_1 1) A developed and implemented legal framework guarantees the need, process and management of NCD surveillance and monitoring and the provision of the required resources. It also establishes the roles and responsibilities of the different participants in the process 2) Is there a plan in place to strengthen the collection of data to support NCD indicators? 2) Having a plan with objectives, targets and follow-up for monitoring provides an indication of the direction to strengthen NCD data collections. The plan should involve human and financial resources and timelines needed in the process 3) Is there independent dedicated funding for surveillance, M&E functions, including training of human resources? 3) An NCD-dedicated budget with regular funding is specified, including training activities of the NCD information system team. This guarantees the sustainability of the process and updating of capacities to meet different and future requirements 4) Are there sufficient and trained human resources available to maintain and operate the NCD data collections? 4) Managed by a trained team, the NCD information system operates smoothly and meets its objectives. Based on requests and feedback from users, the system can be adjusted according to new and resourced demands Analysis NCD_13 How often are NCD data compiled and disseminated to stakeholders and other users? 1) Are data and indicators from different information systems regularly available for integration in accordance with similar standards? Do they include data from other sectors (for example, alcohol or tobacco sales, taxes or price levels)? 1) Information is seamlessly and regularly compiled in a data warehouse, allowing to conduct more comprehensive NCD assessments. The expected effort to get information from different sources together is minimized using similar standards across subsystems 2) Are big data (e.g. web scraping, social media, digital trails, mobile phone or other portable device data) other than EHRs regularly included in data compilation or assessments? 2) There have been some national experiences with regard to using big data sources or there are plans to implement some. Indicate which ones SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT144 ID Question Probing question Expectations NCD_14 What is the feasibility of carrying out suitable health situation and trends analyses? 3) Is there a list of key standard indicators on NCDs and their risk factors, including different domains available for diverse health situation and trends analyses, that is collected regularly? Does it include definitions, measurement methods and metadata? 3) This indicator list is crucial for determining if it is possible to carry out different NCD and risk factor situation and trend assessments and reporting, without limiting comparisons from changes of definitions or calculation methods. Including the relevant domains (for example, specific NCD health impacts, dynamic of risk factors, policies, programmes and interventions) for comprehensive assessments increases the value for monitoring and policy 4) Are those available indicators disaggregated by age, sex, socioeconomic strata and geographical location? 4) Disaggregation of NCDs and risk factors by relevant geographic, social, economic and environmental determinants and other stratifiers is essential for measuring NCDs among different risk and vulnerable groups, monitoring patterns and trends associated with health programmes, and it is used for preparing overall and specific reports NCD_15 What is the status of data dissemination and use of NCD information? 5) Is there a national data warehouse or a national health portal/observatory for accessing and use of NCD core data? Are the data adequately disaggregated for different analyses? Is it possible to generate and visualize data in tabular and graphical formats? Are those data available to the general public? 5) There is a national health observatory in place to enable sharing of aggregated data across national, subnational and district levels to support data analysis and use. There is an open data policy for all users. Portal content should be appropriate for different stakeholder groups and use innovation in visualization and analysis, including tools for tables, graphs and dashboards, as well as mapping and spatial analysis for social and environmental data (e.g. GIS) 6) Are integrated data used for developing, measuring and monitoring national NCD targets? 6) There is a national health or NCD plan using data generated by NCD sources for setting priorities and targets, including indications of how they are measured Health reporting NCD_16 Do NCD reports contribute to informing and providing evidence for policy, strategy and decision- making processes? 1) Are NCD reports produced to inform policy, strategy and decision-making processes, regularly and/or on demand? 1) National and other relevant policy documents are fed with data and analyses for priority setting, establishing targets and determining progress on NCDs, their risk factors and specific health policies 2) What kinds of NCD-related reports are produced to inform and provide evidence for policy? 2) Different reports on specific aspects of NCDs and their risk factor situation and trend assessments, reviews of health inequalities, monitoring of NCD plans or strategy progress, evaluation of effectiveness of health policy interventions, and public health profiles are available for stakeholders and public reviews 3) What kind of disaggregations are used in those NCD reports? 3) At least age and sex are expected in health analysis disaggregations, while other socioeconomic, geographical and environmental stratifiers will add a context and source for explaining NCD-related differences in different types of analyses ADD-ON NCD MONITORING 145 ID Question Probing question Expectations Knowledge translation NCD_17 Do knowledge products on NCDs reach and are they properly used by decision- makers and other relevant stakeholders? 1) Are decision-makers and other relevant stakeholders aware of the NCD knowledge products available? 1) Discussion and agreement with stakeholders on the kinds of reports needed for policy development and evaluation is established. Published schedules of different products to inform policy are available, along with communications on their availability and use 2) Do policy-makers have the appropriate knowledge to understand the content and implications of the NCD reports? 2) Knowledge products are developed in an understandable and simplified manner. Additional technical training materials allow further understanding, according to different users’ level of knowledge 3) Are NCD knowledge products presented and discussed in a proper forum with policy- makers and other stakeholders, to clarify and explain the implications of the report? 3) Appropriate presentations with launching of reports and policy dialogues events allow for a better understanding and proper use of data for policy-making. Further availability through Q&A and information desk functions 4) Do policy-makers request additional or specific NCD knowledge reports to address their needs for making decisions? Are their requests satisfied? 4) Special reports, policy briefs and policy dialogues to assess and discuss NCD policy issues are available and published following demands from policy-makers, stakeholders and the public 5) Are NCD knowledge products easily accessible and being used to inform policy-making for national NCD Action Plans, including determining of priorities, target setting and evaluation of progress? 5) Health data and knowledge products are accessible to decision-makers at all levels, including subnational decision-makers and local communities, with appropriate disaggregation for equity dimensions; and to all constituencies, including the public. Different formats (printed or electronic) are available for use Governance and resources NCD_18 Is there an adequate framework and resources for NCD data collections, monitoring, surveillance, evaluation and reporting functions? 1) Is there a framework for NCD data collections, including content areas and a plan in place to strengthen the collection, use and dissemination of data to support NCD and their risk factor indicators? * Core section Data sources_2 and Governance and resources_1 1) The country health information system operates within a sound policy and institutional environment. This includes sound governance policies and frameworks for data as well as multistakeholder coordination mechanisms, with defined roles and responsibilities for the different stakeholders SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT146 ID Question Probing question Expectations 2) Is there a unit/department at the ministry of health or equivalent with responsibility for NCD health information functions, including data collections, monitoring, surveillance and reporting activities? Are responsibilities clearly identified and documented? Who else is responsible for NCD surveillance activities? Are these external/ independent agencies of the government unit? * Core section Governance and resources_1 2) This information helps to understand the operation of health information, monitoring, surveillance and reporting activities, who is responsible for its different functions, and whether these are held by a single institution or multiple institutions. It helps with identifying areas or people at critical points. It defines the role of other, more independent actors in the area of NCD monitoring and surveillance, if available 3) What are the specific major sources of funding for NCD monitoring, surveillance, evaluation and reporting activities (for example, government revenues, earmarked taxes, donors or health insurance)? Is the operational funding sufficient for addressing main functions? * Core section Governance and resources_1 3) This indicates the potential adequacy and sustainability of resources for continuous function and operation of the systems. National funding supported within the government budget would be the goal for basic operations, while additional funding, for example, from earmarked taxes, would be a supplementary source for specific needs, usually linked to a specific disease and/or its risk factor (for example, alcohol or tobacco excise taxes that may be used to assess prevalence, monitor alcohol biomarkers or cotinine for tobacco underreporting) 4) Does the country have an integrated NCD policy/strategy or a plan of action addressing main NCDs and their risk factors as well as early detection, treatment and care? * Governance and resources_1 4) In addition to the need for national comprehensive NCD information, an integrated policy framework is essential for orienting actions towards data collections and the access of data to different stakeholders, understanding the pathways and linkages of diseases and common risk factors, with the United Nations Development Assistance Framework having a role in coordinating national actions to prevent and control NCDs. A set of national time-bound target indicators is included in the NCD plan and regularly monitored. This is also a country commitment that is reported to WHO 5) Is the national NCD plan/ policy operational or still in development? 5) Current or recent implementation of recent plan/policy indicates political priority and resources available, and maturity of a system 6) To what extent is NCD surveillance and monitoring included in the national NCD plan/policy? * Core Governance and resources_1 and Core section Governance and resources_3 6) These surveillance and monitoring activities/functions are expected to be an integral part of the policy, including specific responsibilities, functions, roles, resources, targets/objectives and outputs/outcomes ADD-ON NCD MONITORING 147 ID Question Probing question Expectations 7) Is there a set of time-bound national targets for NCDs based on the 9 voluntary global targets from the WHO Global Monitoring Framework for NCDs? 7) An agreed set of national time-bound targets is established, including their indicators and monitoring processes. Reports are regularly available for policy planning and assessment of progress and public awareness 8) Does the country have a specific NCD policy/strategy or a plan of action addressing each of the main NCDs, their common risk factors as well as early detection, treatment and care? 8) A more manageable yet comprehensive disease- or risk factor-specific document dealing with each main NCD helps provide specifics on the processes and needs. This, however, does not suggest preference over an integrated policy but rather additional, more developed and expanded specific policies for addressing each risk factor, which would indicate further development of an area, a matter that would not be manageable to address in a national integrated policy or plan 9) Is the disease- or risk factor- specific national NCD plan/ policy operational or still in development? 9) Current or recent implementation of a recent plan/policy indicates political priority and resources available 10) Is there a national multisectoral commission, agency or mechanism in place to oversee NCD engagement, policy coherence and accountability of sectors beyond health? What institutions are its members? * Core section Governance and resources_3 10) There should be a multistakeholder mechanism to ensure independence of analysis and reporting, coherence and coordination of action, transparency and accountability of results, participatory processes. It is essential to officially nominate such a commission/agency/mechanism to oversee NCD engagement 11) Is such a multisectoral commission/mechanism properly funded to carry out its activities? What are the funding sources? * Core section Governance and resources_3 11) It is important to determine whether a regularly available budget is provided and whether it is sufficient for addressing diverse working aspects. Determining the funding sources, whether governmental, tax-based, external/international or mixed, is also an indication of potential sustainability of the agency and governmental commitment 12) To what extent are data collected by the commission used to inform the work? How regularly does this happen? * Core section Governance and resources_3 12) Regular reports on the work of the commission are prepared for policy-makers, published on a specified structure and time basis and discussed with policy-makers and stakeholders, while ad hoc reports are also made available to the target groups 13) Are data and information open and accessible to all interested users? * Core section Governance and resources_1 13) There is a national open data policy framework in place. It offers well-documented microdata, including relevant metadata (with appropriate safeguards to ensure confidentiality), to bona fide researchers for research and analysis that can contribute to evidence for policy-making and planning SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT148 Add-on Health data governance Item Question Probing question Expectations Health data governance - general aspects HDG-1 Is there a national/ regional health data governance or e-health framework in place? 1) Is there a national/regional health data governance / e-health strategy in place? 1) The country has set up and regularly maintained a strategy on health data governance, ideally complemented by an e-health strategy 2) Is there national/regional progress reporting on health data governance / e-health in place? 2) The framework on data governance and e-health is supported by regular progress reporting that is shared with all relevant stakeholders, including NGOs and civil society 3) Which ministry or other public body is responsible for conceptualizing and implementing health data governance? 3) The country or region has nominated an institution, either at the ministerial level or at a national/regional institute of public health to lead on the health governance framework. It might be equally acceptable if a specific body or committee for health data governance has been tasked with leading the initiative 4) Is there a national/regional institution in place that provides guidance on data protection and international telecommunication (IT) security regarding health data? 4) The minimum expectation would be to have in place a national or regional data protection authority providing guidance on the protection of health data and on the necessary IT security measures. A more advanced system should include a specific body that advises on IT security, cybersecurity and critical infrastructure aspects 5) Is there a structured process in place for the identification of the national health data governance priorities? 5) Based on a multistakeholder process, the priorities of the health data governance framework should be identified and re- evaluated in a structured process with regular assessment intervals. The process may, for example, follow the traditional public health cycle of assessment, policy development and assurance 6) Is there a multistakeholder coordination mechanism for the development and implementation of a health data governance strategy in place, or is the topic assigned to a specific institution or body? 6) An interministerial coordination mechanism or multistakeholder platform is in place, including all relevant ministries and public bodies (e.g. a data protection authority). Alternatively, the topic is assigned to one specific body, such as the national institute of public health 7) If there is a strategy / framework in place, is it inclusive and does it represent the societal priorities in a balanced and non-discriminatory way? 7) Depending on the set-up in the country, the health data governance framework must duly represent the priorities in the society, including priorities of ethnic or religious minorities. Therefore, the health data governance framework ensures equal representation and is non-discriminatory ADD-ON HEALTH DATA GOVERNANCE 149 Item Question Probing question Expectations HDG-2 Are the building blocks of a good health data governance implemented? 1) Is there a central body responsible for health data standards, namely, are the precise criteria, specifications and rules for the definition, creation, storage and usage of health data defined? 1) A central body has been identified and given the mandate to define data metrics, standard and quality criteria. 2) Have standard policies and procedures for all aspects of data governance and the data management life-course, including collection, maintenance, usage and dissemination, retention and destruction, been clearly defined and documented? 2) In addition to the overall framework and strategy, operational documentation is provided, including but not limited to policies and procedures, for the implementation of health data governance. The documentation provides hands-on guidance to all stakeholders that have a role in implementing data governance in the health and public health sector. Such documentation would ideally provide meta-level, concrete guidance, and as such it would help all stakeholders (mostly health and public health professionals) to solve real-life issues. 3) Did your country/region implement high-level and/or detailed rules that all health institutions utilize to manage their data assets, including enforcing authentication and access rights to data and compliance with laws and regulations? 3) Building on the documentation referred to under HDG-2 (2), a set of compliance rules has been established that guides the organizational and technical management of health data assets. The rules specifically steer the access to data, ensuring that the right people have access to the right data, and that any access beyond a need to know is avoided. 4) Is there a standardized definition of the roles and responsibilities of all key stakeholders in the health data governance process, for example, data managers, IT managers and senior management staff? 4) Health data governance is implemented via different roles. The country has defined these roles and has allocated responsibilities accordingly. 5) Has your country/region implemented the concept of data stewardship? Specifically, have you implemented a framework for the accountability and responsibility for data and processes that ensures effective control and use of data assets? Note: the concept of data stewardship implies that data stewards do not own the data, but instead are the caretakers of the data assets, ensuring the quality, accuracy and security of health data. 5) The concept of data stewardship has been implemented. Data stewards have been appointed (though not necessarily this specific term is in use) and they are responsible for defining and implementing policies and procedures for the day-to-day operational and administrative management of systems and data, including, among other things, the collection, intake, storage, processing, and transmission of data to internal and external stakeholders. Specifically regarding health data, data stewards act as guardians of the data, both in terms of the compliance and quality/ accuracy of the data. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT150 Item Question Probing question Expectations 6) Is there a national inventory / health data inventory with respect to the storage, accessibility, risks and data values of health data in the public and private sector? 6) The country has a well-defined established inventory of health data assets, or, in other words, the public health system is aware of its data assets and has created a catalogue of all relevant data sets that can be utilized for public policy-making and public health systems management. 7) Is there a national/regional training and/or awareness programme in place for health data governance? If so, does it target solely staff members in the health-care sector (public/private) or does the programme also reach out to other stakeholders (for example, public health bodies, NGOs, health insurance system) and the wider public? 7) The country has set up and maintains a training and awareness programme that, at a minimum, reaches all relevant stakeholders in the public health sphere. Ideally, the training will reach out to further stakeholders in the public and private sector, while the awareness programme also reaches out to the public, civil society and NGOs. 8) Are health data regularly provided to multinational or international organizations such as the United Nations/custodian agencies? If so, is there a legislative/regulatory framework in place for the transfer of health data? 8) In line with multinational and international obligations, the country provides health data to such institutions. The transfer of data is based on a solid regulatory framework. Such a framework may consist of different building blocks, for example, laws, regulations or international agreements. HDG-3 Is there appropriate risk management in relation to the health data governance framework? 1) Is there a risk assessment and reporting structure in place with respect to the governance risks of health data? If so, have standardized metrics for the assessment and reporting of risks been implemented? 1) An important element of data governance is the assessment and mitigation of governance risks. Based on standard risk management methodologies, the country has implemented a risk monitoring and risk management process regarding risks associated with the governance of health data. The process is formalized and rolled out through all public institutions involved in the governance of health data. 2) Is there a policy or framework in place for the evaluation and the acceptance criteria regarding the governance risks identified in the process? 2) As part of the risk management framework, the country has developed criteria and guidance for the acceptance of risks. It is practically impossible to completely avoid risk in the field of health data governance. Therefore, stakeholders that have to “accept” risk in a management process, must have clear guidance and instructions on what risks are acceptable and how these risks are procedurally accepted and documented. HDG-4 Is there a technical framework in place for a good health data governance? 1) Are there risk-appropriate technical standards for the protection of health data in place? Examples could be encryption of data at rest and in transit, and well-defined identity and access management. If so, are these standards provided by a national body on IT security / cybersecurity? 1) Technical standards regarding the handling, processing, storage and destruction of health data exist. The guidance is provided by a specialized institution, such as a national data protection authority or a specific body for IT security. The guidance builds on international best practices, for example, ISO standards or guidance provided by the United States’ National Institute of Standards and Technology or the European Union’s European Union Agency for Cybersecurity. ADD-ON HEALTH DATA GOVERNANCE 151 Item Question Probing question Expectations 2) Is there a national/regional IT infrastructure in place for the storage and processing of health data, for example, a secure cloud for health data? 2) A sophisticated health system includes a national or regional infrastructure for the storage of health data. This infrastructure follows the highest security standards. If data are stored in a cloud, there are specific security perimeters implemented for the cloud, for example, secure encryption with encryption keys that are fully managed by the data controller. 3) Is there a framework in place for the assessment of IT security compliance, for example, and an IT audit programme? Is there a specific national/regional institution that conducts independent audits on the governance of health data security? 3) The country has established a framework for the assessment and auditing of the compliance with IT security standards and policies. The country has required larger health institutions to establish and maintain an IT audit programme that independently assesses compliance. A national or regional institution for IT security or cybersecurity is established, equipped with the right and the means to conduct audits at institutions involved in the governance of health data. Compliance with data protection as an element of good health data governance HDG-5 Is the national/regional health data governance fully compliant with applicable data protection laws and standards? 1) Is there a specific legislative framework in place for the processing of special categories of personal data, such as health data or data on ethnicity and race? 1) The country has established a data protection law, and subsequent regulations and guidance documents, regarding the protection of health data and other particularly sensitive personal data processed in the health-care and public health domain. Such laws may be based on or aligned with the European Union’s GDPR or comparable laws and regulations. 2) Are there specific guardrails for the legal basis for the processing of personal data for (public) health purposes? Is such processing primarily based on the public task of health institutions, the consent of the data subject or a preponderant legitimate interest? 2) The country has developed a conceptual understanding of the legal basis for the processing of data in the health data governance domain. The data processing may be based on the specific legislation, or it may be based on the public task of health and public health institutions. Ideally, a country would avoid processing data based on the informed consent of data subjects, as such consent may be revoked or denied, putting data consistency and quality at risk. 3) Are there standards in place for general transparency regarding data processing (e.g. information on websites, leaflets, posters), information to be provided to data subjects (e.g. as part of a consent process) and regarding the handling of data subject requests? 3) Data protection laws stipulate clear requirements regarding the information to be provided to data subjects, or to the wider public if an individualized information is not feasible or would require disproportionate efforts. Health institutions have implemented mechanisms and standards for informing data subjects. For example, health institutions provide relevant information on their websites, either specifically or as part of their privacy terms. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT152 Item Question Probing question Expectations 4) Is there a legal framework in place for secondary use of personal health data, for example, use of health data for research purposes or statistics? This is particularly relevant for personal health data from the medical setting (hospitals, polyclinics, general practice, etc.) for public health purposes and research. 4) The processing of health data for research or public health reporting often requires secondary use of such data, as data may originally stem from health-care provision or other sources. The data protection law of the country provides clear and concise guidance on such secondary use of personal data, and spells out the documentation requirements for such use. 5) Is there a national policy in place for the storage and retention of personal health data? If so, does this policy provide specifically for the technical and organizational measures to be taken regarding the destruction/ deletion of health data? 5) The data protection law provides clear guidance on the storage, retention and deletion of personal data. Specific guidance is available for the health-care and public health sector on the requirements regarding the primary and secondary use of such data. Technical and organizational measures are in place regarding the deletion of data and the physical destruction of drives and other storage media. The guidance also covers data on paper to the extent such data on paper are part of a structured filing system. HDG-6 Is data privacy institutionalized with health institutions and across competent authorities? 1) Is there a national/regional data protection authority exercising oversight over all institutions that process personal data? 1) There is a national and/or regional data protection authority that has full oversight over the processing of personal data in health care and public health. 2) Are there institutional data protection officers or data protection committees in place that provide independent oversight over an institution? 2) Institutional data protection officers are established in all relevant institutions (for example, the institution has more than 10 employees processing personal data), and in particular, all public institutions have a data protection officer in place. Alternatively, the function of the data protection officer is allocated to a specific committee that acts as a data protection officer and that is sufficiently independent to perform this task. 3) Are there enforcement mechanisms in place, either enforcement through independent data protection authorities, and / or a direct enforcement / legal redress through NGOs or data subjects? 3) Compliance with the applicable data protection laws and the subsequent regulatory framework is enforced by the data protection authority. In addition, data subjects can directly enforce their rights, both via complaints to the institution (as a data controller) or by seeking legal redress via the data protection authority or a court. Ideally, NGOs have the right to launch complaints and can enforce the rights of data subjects by seeking legal redress in the court system. ADD-ON HEALTH DATA GOVERNANCE 153 Item Question Probing question Expectations HDG-7 Are there initiatives in place regarding the accuracy and quality of health data? 1) Is there a competent body or multistakeholder platform steering initiatives on the quality and accuracy of health data? 1) A body or multistakeholder platform has been assigned the task to lead enhancement processes regarding the quality and accuracy of data and the standardization of the collection and processing of health data. Such institution may be part of the public health system, like a national institute of public health, but, depending on the organization of the system, this task may also be allocated to the public health insurance sector or a mix of these institutions. 2) Is the country applying metrics/ standards for health data that promote the interoperability of health data and the exchange of such data? Do the metrics or performance indicators and standards effectively support the interoperability of health data in the country? 2) The country has implemented standards / metrics for the processing of health data and has standardized the nomenclature and systemization of data across data silos in the health-care and public health domain. 3) Does the country follow multinational / international initiatives for the standardization of health data? If so, are the outcomes and the guidance provided by such initiatives implemented in the national/regional framework, or are these initiatives used to inform national/regional processes that deviate from the multinational/ international guidance? In other words, are the international standards adopted or adapted? 3) The country follows, and ideally is part of, international initiatives, for example, initiatives led by WHO or the European Union, to standardize health data and to increase the value proposition of health data. The international best practices are then transposed into the national/regional context and form the basis for national/regional improvement actions. 4) Have policies and procedures been established in the health and public health sector to ensure the continuity of data services in the event of a data breach, loss or other disaster (this includes a disaster recovery plan or business continuity plan)? 4) The country has defined and implemented a national/regional disaster recovery plan or business continuity plan. The plan adequately addresses the main risks to health data and stipulates measures and procedures to restore data processing capabilities as quickly as possible. The plan looks specifically at the integrity and availability of data. HDG-8 Are there any initiatives in the field of data ethics? 1) Is there any national/regional multistakeholder discourse on the acceptable use of health data for the public or private good? 1) The country has set up a communication between relevant stakeholders and civil society in order to define and set the national priorities for the use of health data. The discourse is value-based and reflects the societal needs and ethical constraints. 2) Are there governmental initiatives in place to strengthen the exchange between relevant public institutions regarding the acceptable use of health data? 2) The country has set up and maintains platforms for the communication with civil society and involves all stakeholders, including NGOs and civil society, in an inclusive manner. Ideally, the communication also involves laypeople or patient representatives. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT154 Item Question Probing question Expectations Health data as an asset HDG-9 Is there a conceptual framework for the assessment of the value of data? 1) Is there a medium-to-long-term strategy regarding the value of health data and ways to maximize the value of health data? 1) As part of the data governance framework and activities, the government institutions leading the process also look at the value of data and how the use of health data can be maximized in a compliant manner. The public institutions have defined a medium-to-long- term strategy for unlocking the potential of health data within the health data governance framework. 2) Are health data shared, even in a restricted manner, with commercial entities such as private insurance companies, pharmaceutical companies, medical device companies, start- ups in the health sector or start-up incubators? 2) The country has developed a policy and framework for sharing data with private commercial entities. Such sharing is restricted – for example, only aggregated or fully deidentified information is shared. The government controls and audits the use of such data by private entities, for example, by exercising control via data protection authorities or audit units. 3) Is there a national/regional framework for the sharing of health data with public research institutions? 3) The country has developed and implemented a framework for the sharing of health data in the public sector, in particular, across the different data silos. Data are shared on a need-to-know basis, supported via a secure data sharing infrastructure. HDG-10 Are health data used in a structured manner for public policy-making? 1) Are available health data analyzed in a structured manner for health policy-making? 1) The available analyses provide the evidence base that policy-makers need. 2) Do policy-makers and other relevant stakeholders know which health reports / health data analyses are available? 2) The reports and their publication schedules are well-known by policy-makers and other stakeholders, and these users know how to access the reports. 3) Are policy-makers able to steer the health data analytics, or, in other words, is there a clear process of looping policy-making requirements into health data analytics? 3) Policy-makers and other stakeholders know how to interpret and use the reports. The evidence is directly feeding into policy- making processes. 4) Is there a long-term vision for the use of health data in public policy- making, for example, forecasts for resource planning in health care, foresight studies for strategic policy development, or benchmarking to compare and improve the quality of health service delivery? If so, is such a long-term vision embedded in multinational / international initiatives? 4) The country has a defined long-term vision for the use of health data in public policy-making. The vision is embedded in a multinational or international framework and aligned with all relevant stakeholders. ADD-ON HEALTH DATA GOVERNANCE 155 Item Question Probing question Expectations Resources HDG-11 Are enough resources available for health data governance and data ethics? 1) Do all stakeholders have adequate financial resources for data governance, for example, in terms of data security? 1–4) All stakeholders have adequate financial, human and technical resources for data governance, and have access to relevant training and capacity building offerings. 2) Do all stakeholders have access to adequate tools for data governance? 3) Do all stakeholders have adequate manpower for data governance, IT security and data protection? 4) Do all stakeholders have adequate access to training and capacity building for data governance? ENISTA: Agency for Cybersecurity; GDPR: General Data Protection Regulation; ISO: International Organization for Standardization; NIST: National Institute of Standards and Technology. SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT156 Template for the HIS strategy roadmap including budget estimates, with some pre-filled exam- ples. Adapted from: Health Metrics Network. Guidance for the Health Information Systems (HIS) Strategic Planning Process. Steps, Tools and Templates for HIS Systems Design and Strategic Planning (2009). HIS improvement intervention activitiesa Output Time frame Responsible agency Types of resources needed D = one-off developmental costs R = recurrent costs Estimated budget needed Start (m/y) Complete (m/y) HIS improvement priority objective: 1. Increase availability of service records from private health-care providers HIS improvement intervention: 1a. Organization of a meeting with private health-care providers to understand the underlying reasons for the current lack of data delivery 1a1. Organize meeting with a representative delegation of the private sector Meeting 9/2020 11/2020 Agency for health services research and ministry of health • Staff for organizing and attending meeting (D) • Funding for meeting venue (D) • Funding for travel costs for participants (D) • 40 hours x hourly rate(s) = XX amount • Quotation from meeting venue = XX amount • 10 people x estimate of average travel costs = XX amount 1a2. Write meeting report with conclusions and recommendations for the ministry of health on how to improve data availability Meeting report 11/2020 12/2020 Agency for health services research • Staff for writing the report (D) • 40 hours x hourly rate(s) = XX amount Total = XX amount Annex 3. Template for HIS improvement roadmap ANNEx 3. TEMPLATE FOR HIS IMPROVEMENT ROADMAP 157 HIS improvement intervention activitiesa Output Time frame Responsible agency Types of resources needed D = one-off developmental costs R = recurrent costs Estimated budget needed Start (m/y) Complete (m/y) HIS improvement intervention: 1b. Investigate models for private sector involvement implemented in other countries to identify international good practices 1b1. Perform literature search and conduct interviews with international health systems experts Literature overview and interview transcripts 10/2020 03/2021 • Staff for conducting the research (D) • 120 hours x hourly rate(s) = XX amount 1b2. Write meet- ing report with conclusions and recommenda- tions for the min- istry of health on how to improve data availability Scientific report 03/2021 05/2021 • Staff for writing the report (D) • Funding for editing and design (D) • 60 hours x hourly rate(s) = XX amount • Quote from editor/graphic designer Total = XX amount HIS improvement priority objective: 2. Facilitate both formal and informal HIS stakeholder coordination and collaboration HIS improvement intervention: 2a. Establishing a multistakeholder HIS coordination group with a formal mandate 2a1. Establish coordination group and define its terms of reference (ToR) Coordination group with ToR 9/2020 Continuous Ministry of health • Staff for establishing the coordination group and writing ToR (D) • 30 hours x hourly rate(s) 2a2. Organize regular meetings Meetings 10/2020 Continuous Ministry of health Staff for organizing and attending meeting (R) Funding for meeting venue (R) Funding for travel costs for participants (R) 15 hours x hourly rate(s) x number of meetings per year = XX amount Quotation from meeting venue x number of meetings per year = XX amount 15 people x estimate of average travel costs x number of meetings per year = XX amount SUPPORT TOOL TO STRENGTHEN HEALTH INFORMATION SYSTEMS GUIDANCE FOR HEALTH INFORMATION SYSTEM ASSESSMENT AND STRATEGY DEVELOPMENT158 HIS improvement intervention activitiesa Output Time frame Responsible agency Types of resources needed D = one-off developmental costs R = recurrent costs Estimated budget needed Start (m/y) Complete (m/y) 2a3. Write meeting reports Reports 11/2020 Continuous Ministry of health Staff for writing the report (R) 20 hours x hourly rate(s) x number of meetings per year = XX amount Total = XX amount (partly developmental costs and partly recurrent costs (costs per year)) HIS improvement intervention: 2b. Organizing a series of workshops on specific topics where HIS stakeholders can learn more about each other’s expertise, roles, and tasks, and personal relationships can be established 2b1. Set up working group to organize the workshop and select the topics that will be addressed in the workshops Workgroup and agendas for the workshop 9/2020 12/2020 National public health institute and ministry of health • Staff for setting up the working group and developing the workshop agendas (D) • 40 hours x hourly rate(s) = XX amount 2b2. Organize series of workshops 4 workshops 1/2021 12/2022 National public health institute and ministry of health • Staff for organizing the workshops (D) • Funding for workshop venues (D) • 30 hours x hourly rate(s) x number of workshops = XX amount • Quotation from meeting venue x number of workshops = XX amount Total = XX amount ANNEx 3. TEMPLATE FOR HIS IMPROVEMENT ROADMAP 159 HIS improvement intervention activitiesa Output Time frame Responsible agency Types of resources needed D = one-off developmental costs R = recurrent costs Estimated budget needed Start (m/y) Complete (m/y) HIS improvement priority objective: 3. XXXXX HIS improvement intervention: 3a. XXXXX 3a1. XXXXX 3a2. XXXXX Etc. a Cross-reference the HIS improvement intervention activities with: y the outcomes of the mapping of planned and ongoing HIS strengthening activities to create synergies and avoid overlaps where possible; y the guiding principles for HIS development to maximize the impact of the HIS improvement interventions and activities; and y the PRISM conceptual model to identify relevant activities for addressing the technical, organizational, and behavioural factors that influence the outputs and outcomes of HIS improvement interventions. Defining HIS improvement objectives and interventions on page 31 carries more information.
G U ID A N C E FO R C R E A T IN G I M P A C T FU L H E A LT H R E P O R T S World Health Organization Regional Office for Europe UN City, Marmorvej 51 DK-2100, Copenhagen Ø, Denmark Tel: +45 45 33 70 00 Fax: +45 45 33 70 01 Email: eurocontact@who.int Website: www.euro.who.int The WHO Regional Office for Europe The World Health Organization (WHO) is a specialized agency of the United Nations created in 1948 with the primary responsibility for international health matters and public health. The WHO Regional Office for Europe is one of six regional offices throughout the world, each with its own programme geared to the particular health conditions of the countries it serves. Member States Albania Andorra Armenia Austria Azerbaijan Belarus Belgium Bosnia and Herzegovina Bulgaria Croatia Cyprus Czechia Denmark Estonia Finland France Georgia Germany Greece Hungary Iceland Ireland Israel Italy Kazakhstan Kyrgyzstan Latvia Lithuania Luxembourg Malta Monaco Montenegro Netherlands North Macedonia Norway Poland Portugal Republic of Moldova Romania Russian Federation San Marino Serbia Slovakia Slovenia Spain Sweden Switzerland Tajikistan Turkey Turkmenistan Ukraine United Kingdom Uzbekistan