Monitoring, evaluation and review of national health strategies: a country-led platform for information and accountability

MONITORING, EVALUATION AND REVIEW OF NATIONAL HEALTH STRATEGIES

A COUNTRY-LED PLATFORM FOR INFORMATION AND ACCOUNTABILITY

+ Related Initiatives

A country-led platform for information and accountability | A

Monitoring, evaluation and review of national health strategies A country-led platform for information and accountability

10 November 2011

+ Related Initiatives

WHO Library Cataloguing-in-Publication Data Monitoring, evaluation and review of national health strategies: a country-led platform for information and accountability. 1.National health programs. 2.Program evaluation - methods. 3.Data collection. 4.Health policy. 5.Information systems. I.World Health Organization. II.International Health Partnership. ISBN 978 92 4 150227 6 (NLM classification: WA 540)

© World Health Organization 2011 All rights reserved. Publications of the World Health Organization are available on the WHO web site (www.who.int) or can be purchased from WHO Press, World Health Organization, 20 Avenue Appia, 1211 Geneva 27, Switzerland (tel.: +41 22 791 3264; fax: +41 22 791 4857; e-mail: bookorders@who.int). Requests for permission to reproduce or translate WHO publications – whether for sale or for noncommercial distribution – should be addressed to WHO Press through the WHO web site (http://www.who.int/about/licensing/ copyright_form/en/index.html). The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by the World Health Organization to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall the World Health Organization be liable for damages arising from its use. Printed by the WHO Document Production Services, Geneva, Switzerland. Design by Inís Communication – www.iniscommunication.com Photo credits: Chapter I - WHO/Christopher Black; Chapter II - WHO/Tom Pietrasik; Chapter III - WHO/Jim Holmes and WHO/Lisa Schwarb; Chapter IV - WHO/Olivier Asselin.

Acknowledgements This document was developed through a collaborative process within the context of the International Health Partnership+ (IHP+), with inputs from monitoring and evaluation experts from the World Health Organization (WHO), the World Bank, the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM), the Global Alliance for Vaccines and Immunization (GAVI), the United States Agency for International Development (USAID), the Joint United Nations Programme on HIV/AIDS (UNAIDS), the United Nations Children’s Fund (UNICEF), MEASURE Evaluation, the UK Department for International Development (DFID), the Norwegian Agency for Development Cooperation (NORAD), the Centers for Disease Control and Prevention (CDC), and the Health Metrics Network (HMN), among others. Particular thanks are extended to all those who contributed to the meeting on Strengthening Monitoring and Evaluation of National Health Strategies, held in Glion-sur-Montreux, Switzerland, 14–15 July 2010. WHO is particularly grateful to country monitoring and evaluation experts in Benin, Ethiopia, Ghana, Kenya, Mali, Mozambique, Nepal, Rwanda, Sierra Leone and Uganda, who facilitated the testing and review of the document at country level. This document was produced by the World Health Organization.

A country-led platform for information and accountability | III

Contents Glossary of terms, acronyms and abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V Aim of the document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VI Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VIII I. The national health strategy as the basis for information and accountability . . . . . . . . . . . . . . . . 1 Attribute 1. The national health strategy specifies a sound monitoring, evaluation and review component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 II. Institutional capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Attribute 2. Roles, responsibilities and coordination mechanisms for monitoring, evaluation and review are clearly defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Attribute 3. Capacity strengthening in monitoring, evaluation and review is addressed . . . . . . . 6 III. Monitoring and evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Attribute 4. There is a comprehensive framework that guides the monitoring, evaluation and review work, including core indicators and targets . . . . . . . . . . . . . . . . . . . . . . . . . 9 Attribute 5. The monitoring, evaluation and review component specifies data sources, identifies and addresses data gaps, and defines responsibilities for data collection and information flow . . . . . 13 Attribute 6. Data analysis and synthesis work is specified, and data quality issues are anticipated and addressed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Attribute 7. Data dissemination and communication are effective and regular . . . . . . . . . . . . 18 Attribute 8. Prospective evaluation is planned and implemented . . . . . . . . . . . . . . . . . . . 21 IV. Country mechanisms for review and action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Attribute 9. There is a system of joint periodic progress and performance reviews . . . . . . . . . 23 Attribute 10. There are processes by which related corrective measures can be taken and translated into action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Annex 1. Monitoring, evaluation and review platform: sample outline of a monitoring, evaluation and review plan for the national health strategy . . . . . . . . . . . . . . . . . . . . 29 Annex 2. Annual health sector progress and performance report: sample outline . . . . . . . . . . . . . . 30 Annex 3. Potential basket of indicators for monitoring health sector progress and performance . . . . . 31 Annex 4. How to conduct a rapid assessment of country monitoring, evaluation and review practices and mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

IV | Monitoring, evaluation and review of national health strategies

Glossary of terms, acronyms and abbreviations AIDS CSO DQAA Evaluation acquired immunodeficiency syndrome civil society organization data quality assessment and adjustment Rigorous, science-based analysis of information about programme activities, character­ istics, outcomes and impact that determines the merit or worth of a specific programme or intervention. human immunodeficiency virus International Health Partnership and related initiatives. IHP+ is a group of partners who share a common interest in improving health services and health outcomes by putting the principles of the Paris Declaration on Aid Effectiveness into practice. IHP+ was launched in September 2007. joint assessment of national strategies Health Metrics Network Millennium Development Goal Monitoring, evaluation and review of activities of the national health strategy. ministry of health The routine tracking and reporting of priority information about a programme and its intended outputs and outcomes. A national health strategy or a national health plan. A national health strategy is a document or set of documents that lays out the context, vision, priorities, objectives and key interventions of the health sector, multisectoral or disease programme, as well as providing guidance to inform more detailed planning documents. A strategy provides the “big picture” and the road map for how goals and objectives are to be achieved. A national health plan is a document or set of documents that provides details of how objectives are to be achieved, the time frame for work, who is responsible and how much it will cost. This may come in the form of a multi-year plan, supported by annual operational plans that allow for adjustment as a programme. national statistics office Country processes for assessing health system progress and performance. sector-wide approach. This is an approach to international development that brings together governments and donors within any sector. It is characterized by a set of operating principles rather than a specific package of policies or activities.

HIV IHP+

JANS HMN MDG M&E MoH Monitoring NHS

NSO Review SWAp

A country-led platform for information and accountability | V

Aim of the document This document provides guidance to countries and partners for strengthening monitoring, evaluation and review (M&E) of national health plans and strategies (NHS).1 It outlines the key attributes and characteristics of a sound country-led platform for monitoring, evaluation and review of health sector progress and performance, as the basis for information and accountability. It also aims to show how development partners can contribute to the strengthening of such a platform. Sound M&E systems are built on inclusive policy dialogue and regular evidence-based assessments that inform progress and performance reviews, and that result in remedial action and mutual accountability among all stakeholders. This should form the basis for resource allocation, policy-making and effective management of programmes. In the context of NHS, most countries already have some sort of monitoring, evaluation and review mechanism in place. The aim of this document is to build upon and strengthen existing country mechanisms. The document can be used in conjunction with the International Health Partnership and related initiatives (IHP+) work on joint assessment of national strategies (JANS)2 and can be considered as an effort to strengthen the fifth component of the JANS, i.e. monitoring, evaluation and review mechanisms. The way in which M&E is done is unique to each country, therefore this document is deliberately generic, setting out the essential attributes and characteristics for four areas considered to be the foundation of a sound monitoring, evaluation and review platform for a national health strategy. 1. The national health strategy as the basis for information and accountability: M&E is an integral component of a comprehensive national health strategy, established through inclusive policy dialogue, and with support and alignment of country and development partners. II. Institutional capacity: this specifies the roles and responsibilities of in-country institutions to support regular monitoring, review and (remedial) action, and also addresses capacity strengthening. III. Monitoring and evaluation: monitoring and evaluation includes a comprehensive framework that addresses indicator selection, related data sources, and analysis and synthesis practices, including quality assessment, performance review, communication and use. IV. Country mechanisms for review and action: this implies the existence of well-established, transparent processes at country level, involving multiple stakeholders to ensure quality of data and independence. The structure of this document is based on the key attributes and characteristics of these four areas. This document can be used to assess, improve or develop the M&E component of the NHS, or of a specific programme (such as HIV/AIDS, maternal health or immunization), or to assess health system strengthening actions. It can be used to develop the M&E component of a new plan, to strengthen reviews of progress and performance during implementation of an existing plan, and at the end of a planning cycle to evaluate what has been achieved and to lay the foundation for a new cycle. 1 Different countries and partners use the terms “national health plan”, “national health strategy” and “national health sector plan” in slightly different ways. In line with the IHP+, throughout this document, the term “national health strategy” (or “national strategy”) will be used to include the higher level strategy documents and more operational plans. 2 Joint assessment of national strategies (JANS): http://www.internationalhealthpartnership.net/en/about/j_1253621551. The JANS has five key attributes: (1) situation analysis and coherence of strategies and plans with this analysis; (2) the process through which national plans and strategies have been developed; (3) financing and auditing arrangements; (4) implementation and management arrangements; and (5) monitoring, evaluation and review mechanisms.

This document is also intended to provide guidance to partners working with countries, who are aiming to better align their support to monitoring and evaluation activities (such as data collection, data analysis, and reporting), with the monitoring and review processes and mechanisms of the NHS. This includes streamlining global reporting with country planning and review cycles, in terms of timing, methods and contents. Throughout this document, the term M&E is used to describe the full process of monitoring and review of results, as well as evaluation and learning.

A country-led platform for information and accountability | VII

Introduction The scale-up of resources and initiatives for better health is unprecedented both in terms of the potential resources available as well as in terms of the number of initiatives involved. There is growing recognition that harmonized monitoring, evaluation and review is required to demonstrate results, secure future funding, and enhance the evidence base for interventions. Strategic planning and programme implementation should be based on strong monitoring, evaluation and review of progress and performance as the basis for information, results and accountability. The United Nations Commission on Information and Accountability for Women’s and Children’s Health defines accountability as a cyclical process of monitoring, review, and (remedial) action.3 Accountability implies the existence of well-established transparent processes for monitoring progress and performance at the country level, supported by a well-functioning health information system, regular multi-stakeholder review processes, and systematic actions in-country, providing the basis for mutual accountability between country citizens, country decision-makers and the international community. Global partners and countries have been working towards better harmonization and alignment in support of a strong national health strategy. IHP+ has been focusing on achieving better health results by mobilizing donor countries and other development partners around a single country-led national health strategy.4 M&E is one of the key attributes of a sound national health strategy, and a common M&E framework for health progress and performance reviews and health systems strengthening has been developed,5 which builds upon principles derived from the Paris Declaration on Aid Effectiveness.6 Core to the operationalization of this framework is the strengthening of a common platform for monitoring, evaluation and review of the NHS. Building upon existing country efforts and mechanisms, the platform brings together all the elements relating to M&E of the NHS including the processes by which M&E is developed; the country coordination mechanisms and institutional capacity; the technical elements addressing M&E (such as indicator selection, data sources, analysis and synthesis practices, data quality assessment and data dissemination); as well as the country review processes for planning and decision-making. Table 1 describes these key attributes and their characteristics. The primary aim of the platform is to serve as a strong and harmonized M&E component of the NHS, covering all major disease programmes as well as health system actions. The platform serves as the mechanism for subnational, national and global reporting, aligning partners at country and global levels around a common approach to country support and reporting requirements. The platform aims to be relevant for countries and for global health partnerships, donors and agencies, and to result in better alignment of country and global monitoring systems. The platform should reduce duplication of efforts, focus on results monitoring, and result in better accountability and harmonization of M&E systems.

3 Keeping promises, measuring results. Report of the Commission on Information and Accountability for Women’s and Children’s Health, Geneva, World Health Organization, 2011 (http://www.who.int/pmnch/media/membernews/2011/20110519_pr_health_account_report/en/). 4 International Health Partnership and related initiatives (IHP+): http://www.internationalhealthpartnership.net/en/home. 5 WHO, GAVI, the Global Fund and the World Bank. Monitoring and evaluation of health systems strengthening: an operational framework. Geneva, World Health Organization, 2010 (http://www.who.int/healthinfo/HSS_MandE_framework_Oct_2010.pdf). 6 The Paris Declaration on Aid Effectiveness and the Accra Agenda for Action. OECD, 2005 (http://www.oecd.org/dataoecd/30/63/43911948.pdf).

VIII | Monitoring, evaluation and review of national health strategies

Table 1. A monitoring, evaluation and review platform for national health strategies KEY ATTRIBUTES 1. The national health strategy specifies a sound monitoring, evaluation and review component.

CHARACTERISTICS 1.1 1.2 1.3 1.4 Monitoring, evaluation and review addresses the goals and objectives of the national health strategy and is based on a sound situation analysis. Disease- and programme-specific monitoring, evaluation and review are aligned with that of the national health strategy. The monitoring, evaluation and review plan is costed and funded with full partner alignment and support. Monitoring, evaluation and review is regularly assessed.

I. The national health strategy as the basis for information and accountability

II. Institutional capacity 2. Roles, responsibilities and coordination mechanisms for monitoring, evaluation and review are clearly defined. 3. Capacity strengthening in monitoring, evaluation and review is addressed. III. Monitoring and evaluation 4. There is a comprehensive framework that guides the monitoring, evaluation and review work, including core indicators and targets. 5. The monitoring, evaluation and review component specifies data sources, identifies and addresses data gaps, and defines responsibilities for data collection and information flow. 6. Data analysis and synthesis work is specified, and data quality issues are anticipated and addressed. 4.1 4.2 4.3 5.1 5.2 5.3 6.1 6.2 6.3 7. Data dissemination and communication 7.1 are effective and regular. 7.2 7.3 8. Prospective evaluation is planned and 8.1 implemented. There is a balanced and parsimonious set of core indicators with welldefined baselines and targets. Disease- and programme-specific indicators are aligned. Integrated with the national health information system strategy. Data sources are specified in a comprehensive and integrated manner. Critical data gaps are identified and addressed. Responsibilities for data collection and management are specified. Data analysis and synthesis work is specified. There are regular assessments of progress and performance, including systematic analyses of contextual and qualitative information. Specific processes for data quality assessment and adjustment are in place and are transparent. Analytical outputs as the basis for national and global reporting are defined and produced. Appropriate decision-support tools and approaches are used. Data, methods and analyses are publicly available. Prospective evaluation is planned and linked to monitoring, evaluation and review of national health strategies. 2.1 2.2 3.1 There is an effective country-led coordination mechanism for monitoring, evaluation and review. Key institutions and stakeholders have clear roles and responsibilities. Capacity strengthening requirements are identified and addressed.

IV. Country mechanisms for review and action 9. There is a system of joint periodic progress and performance reviews. 9.1 A regular and transparent system of reviews with broad involvement of key stakeholders is in place. 9.2 There are systematic linkages between health sector reviews, diseaseand programme-specific reviews, and global reporting. 10.1 Results from reviews are incorporated into decision-making, including resource allocation and financial disbursement. 10.2 Multi-stakeholder mechanisms are specified to provide routine feedback to subnational stakeholders.

10. There are processes by which related corrective measures can be taken and translated into action.

A country-led platform for information and accountability | IX

I. The national health strategy as the basis for information and accountability The prerequisite for a sound monitoring, evaluation and review platform is a comprehensive and robust national health strategy (NHS). Essential attributes and criteria of a comprehensive and sound NHS are described in the joint assessment of national strategies (JANS) and its associated guidelines. The JANS is a shared approach to assess and guide the development of a national health strategy. It is a generic tool that sets out the essential ingredients or criteria of a “good “national strategy, along five sets of attributes: • • • • • the situation analysis, and coherence of strategies and plans with this analysis; the process through which national plans and strategies have been developed; financing and auditing arrangements; implementation and management arrangements; monitoring, evaluation and review mechanisms.

A national health strategy that meets the JANS criteria is one that has been based on sound analysis and response to the context; has been developed in a transparent and participatory process with multistakeholder endorsement; is accompanied by a sound financial and auditing framework and plan; specifies arrangements and systems for implementing and managing the programmes in the national strategy; and relies on strong country-led monitoring, evaluation and review mechanisms. This document focuses and builds upon the fifth set of attributes and criteria for monitoring, evaluation and review mechanisms, and aims to provide countries with detailed guidance on how to build a strong country-led monitoring, evaluation and review platform for national health strategies.

A country-led platform for information and accountability | 1

Attribute 1. The national health strategy specifies a sound monitoring, evaluation and review component A sound NHS should specify mechanisms for monitoring, evaluation and review that lead to corrective action. Monitoring means bringing together data from all relevant sources to analyse what is happening, where and to whom. Monitoring uses a set of core indicators and targets to provide timely and accurate information to governments and partners in order to inform progress and performance reviews, and policy dialogue. Evaluation builds upon the monitoring data but the analysis goes much deeper, taking into account contextual changes, addressing questions of attribution, and looking at counterfactual situations. Reviews are based on the evidence gathered through monitoring processes and require national institutional mechanisms involving multiple stakeholders. Existing country health-sector review processes are a key entry point for assessing progress and performance, and can influence priority-setting and resource allocation. Such reviews need to be systematically linked to actions in countries and provide the basis for mutual accountability.

1.1 Monitoring, evaluation and review addresses the goals and objectives of the national health strategy and is based on a sound situation analysis Monitoring, evaluation and review (M&E) should be an integral component of the national health strategy (NHS), and accordingly monitor progress towards the goals and objectives of the NHS. The NHS objectives should address all major public health problems. The M&E activities should accordingly cover all the relevant disease areas and activities for strengthening health systems. The NHS must be based on a thorough understanding of the health needs, priorities and risks within a country. The development of a new NHS should build upon the final review of progress and performance of the previous strategy. The final evaluation could serve as the initial situation analysis and evidence base for the subsequent health strategy development process. The situation analysis should include a detailed analysis of the context covering political, social, cultural, gender, epidemiological, legal and institutional factors. The core characteristics of a situation analysis are defined in the JANS tool.7 The analysis should be based on a comprehensive and participatory analysis of health determinants and health trends, taking into account the epidemiological, political, socioeconomic and organizational context prevailing in the country, and pay adequate attention to equity issues. The analysis and subsequent policy dialogue should include information on quantitative data as well as qualitative information. The results should be useful to both planners and decision-makers. The M&E component of the NHS should be developed in transparent and systematic consultation with all the key stakeholders, including different levels of government, civil society organizations (CSOs), international partners, and academia, among others.

7 Joint assessment of national strategies: http://www.internationalhealthpartnership.net/en/about/j_1253621551.

2 | Monitoring, evaluation and review of national health strategies

1.2 Disease- and programme-specific monitoring, evaluation and review are aligned with that of the national health strategy The M&E component of the NHS should specify the coordination and alignment of M&E processes and mechanisms across specific programmes. In many countries, there is a disconnect between the national health strategy and the strategies and plans of disease-specific programmes, often fuelled by separate funding channels related to global initiatives. This disconnect often leads to a lack of coherence between the planning and monitoring efforts, with different operational planning cycles and stakeholders, poorly linked review processes, and fragmented investments in data collection and analysis. The alignment and coherence of disease- and programme-specific plans with the national health sector plan can be improved by ensuring that there is one comprehensive national M&E plan that specifies how it is linked to more detailed, disease-specific plans in a logical and cascading manner. Conversely, disease programmes should be clear how they are linked to M&E of the NHS. The cross-cutting M&E of health system strengthening should be included within the overall M&E component of the NHS. All disease- and programme-specific M&E should use the same technical framework and M&E platform as that of the NHS. This implies that data collection, transfer and analysis are well coordinated, including a common plan for e.g. household survey data collection and facility assessments, as well as cross-cutting efforts to strengthen the health facility reporting system. This also implies aligning the review processes so that the results of programme-specific review feed into the overall health sector review.

1.3 The monitoring, evaluation and review plan is costed and funded with full partner alignment and support The M&E component of the NHS specifies the data sources and the frequency of data collection. These should be costed and linked to the overall costed plan of the national health information system. Global partners should align their investments and activities with the NHS and M&E plan. The M&E investments should also be closely linked to the national strategy for the development of statistics, which for example, includes household survey plans.8

1.4 Monitoring, evaluation and review is regularly assessed Regular planned assessments of the M&E system are required in order to ensure that indicators are measuring what they are meant to, that data are generated according to standards, that data analysis and communication of results give the information needed by decision-makers, and that data management includes an assessment of overall data quality. It is essential that the indicators used for M&E are ‘fit for purpose’; that is, relevant to the needs of different users and sensitive to change. If health priorities, strategies or activities have changed, indicators should be reviewed to see if they are still relevant and revisions should be made as appropriate. The underlying data need to be accurate, complete and timely. Quality is essential, both in terms of validity and reliability. Transparency is critical. Such assessments should be carried out every 2–3 years and the reports need to be made public and discussed at the annual review process.

8 The International Household Survey Network: http://www.ihsn.org.

A country-led platform for information and accountability | 3

II. Institutional capacity A well-functioning M&E system requires a supportive institutional environment, with defined roles and responsibilities for the different stakeholders. There needs to be sufficient human resources with adequate technical capacity to manage the various components of an effective M&E system in support of progress and performance reviews.

Attribute 2. Roles, responsibilities and coordination mechanisms for monitoring, evaluation and review are clearly defined 2.1 There is an effective country-led coordination mechanism for monitoring, evaluation and review The implementation of the NHS is usually overseen by a coordinating committee led by the ministry of health (MoH), with involvement of subnational stakeholders, other ministries, CSOs, the private sector, academic institutions and development partners. Such committees often have technical working groups or subcommittees, with one focused on M&E. The M&E subcommittee should have the technical expertise to guide all M&E work, including the analytical review process, and involve all key stakeholders. Regular, well-attended meetings with minutes, formalized reporting to the overall coordinating committee, and key document outputs are essential features of a well-functioning M&E subcommittee.

2.2 Key institutions and stakeholders have clear roles and responsibilities The M&E component of the NHS should identify country institutions and stakeholders that will be involved in M&E and in the country health sector reviews. Roles and responsibilities are defined at both national and subnational levels and cover data collection, analysis, synthesis and use. Transparency – a prerequisite for effective planning and managing accountability – is essential in all steps. In terms of the institutional arrangements to oversee M&E, various options can be envisaged. For example, oversight could be the responsibility of a group within the MoH or could be the responsibility of a separate private or non-profit organization. Governance and financing structures may also differ. However, monitoring, evaluation and capacity strengthening efforts should involve institutions that are independent of programme implementation so as to maximize objectivity. In some countries, national statistics offices that have aligned themselves with the Fundamental Principles of Official Statistics9 can provide this degree of objectivity and transparency. Elsewhere, academic, research and public health institutes may be well placed to provide this function. A landscaping exercise of the institutional context can be used as the foundation for decision-making regarding capacity strengthening activities. 9 Fundamental Principles of Official Statistics: http://unstats.un.org/unsd/dnss/gp/fundprinciples.aspx.

A country-led platform for information and accountability | 5

Attribute 3. Capacity strengthening in monitoring, evaluation and review is addressed There is a need for a clear process for capacity building at country level in all aspects of M&E, including collection, analysis, synthesis, quality assessment, dissemination and use of data for progress and performance reviews. Leveraging the expertise and capacity of in-country institutions such as academic, public health and research institutions can contribute to improving the quality of the health-related statistics.

3.1 Capacity strengthening requirements are identified and addressed The specific areas in which capacity is required include the following. • Data collection. In general, the national statistics office (NSO) is responsible for household health surveys and vital statistics from birth and death registrations, however, the MoH often plays a major role as well. The MoH often leads on the compilation of administrative and clinical data, and may work with specific institutions to assess data quality. In addition, facility assessments are often conducted by the MoH, in which case, some degree of independence is needed for data collection, e.g. by employing staff from training schools for the field work. • Data compilation and storage. This involves bringing together data generated by the NSO, MoH, researchers, donors, development partners, nongovernmental organizations and others. This is usually the responsibility of the MoH or the NSO, but sometimes a semi-independent institution plays a major role. Providing public access to the health data is a critical element of transparency in a sound M&E system. • Data quality assessment, validation and adjustment. This should include independent assessments of the quality of data generated from clinical and administrative sources, ad hoc surveys, and other data sources. This is ideally done by independent country institutions such as research and academic centres, working in collaboration with the MoH and the NSO. • Data analysis and performance reviews. This involves synthesizing data from multiple sources for the purpose of reviews, planning, policy analysis, regional and global reporting, and evaluation. This work is ideally carried out by country institutions in collaboration with the MoH and NSO. Global partners may also provide technical assistance. • Estimation and statistical modelling. Focusing on key health statistics, this includes the application of global standards, tools and methods to correct for bias and missing values; the generation of estimates; and forecasting for planning purposes. Academic institutions as well as analytical staff in the MoH or NSO have the main responsibility for estimation and statistical modelling.

6 | Monitoring, evaluation and review of national health strategies

• Data presentation and dissemination to different target audiences. The focus of data presentation and dissemination is on major decision-making processes, where effective communication of results may lead to an adjustment of implementation and revisions of plans. Global reporting should be aligned as much as possible with national reporting. Communicating to the general public and media is also critical and usually requires special skills. The responsibility for data presentation and dissemination often lies with analysts in government and academic institutions, but special communication skills are required.

A country-led platform for information and accountability | 7

III. Monitoring and evaluation The M&E component of the NHS should be based on a logical framework for M&E that is comprehensive and addresses the selection of a core set of indicators; identifies the data sources for each indicator; and specifies plans for filling data gaps, conducting analysis, data quality assessment, communication, and dissemination of the results. M&E of the NHS should not be implemented in parallel to a country’s health information system, but should be derived from it. The components and standards for a country health information system are articulated in the Health Metrics Network (HMN) framework document.10 A key element of a strong health information system is a comprehensive approach to determining which data should be collected and used, at which levels of the system, and by whom. The summary indicators required for the M&E component of the NHS should draw from multiple data sources and levels of the health system to serve the needs of different users.

Attribute 4. There is a comprehensive framework that guides the monitoring, evaluation and review work, including core indicators and targets The IHP+ common M&E Framework11 provides a logical and results-chain representation of the key components of the NHS M&E (see Figure 1). It comprises four major indicator domains: system inputs and processes, outputs, outcomes, and impact. The framework shows how inputs to the system (e.g. financing, infrastructure) and processes (e.g. supply chain) are reflected in outputs (such as availability of services and interventions) and eventual outcomes (e.g. intervention coverage) and impact (e.g. improved health outcomes). This results-chain framework can be used to demonstrate performance of both disease-specific and health systems interventions. The framework not only facilitates the identification of core indicators along each link in the results chain, but also links indicators to data collection methods; highlights the need for analysis and synthesis of data from multiple sources, including data quality assessment; and demonstrates how the data need to be communicated and used to inform decision-making at different levels.

10 Health Metrics Network.Framework and standards for country health information systems. Geneva, World Health Organization, 2008 (http://www.who.int/healthmetrics/documents/hmn_framework200803.pdf). 11 See: http://www.internationalhealthpartnership.net/en/working_groups/monitoring_and_evaluation.

A country-led platform for information and accountability | 9

Figure 1. IHP+ common M&E framework

Inputs and processes

Outputs

Outcomes

Impact

Governance

Indicator domains

Financing

Infrastructure; Information and communication technologies Health workforce Supply chain Information

Intervention access and services readiness Intervention quality, safety

Coverage of interventions Prevalence risk behaviours and factors

Improved health outcomes and equity risk protection Responsiveness

Data collection

Administrative sources Financial tracking system; National Health Accounts Databases and records: HR, infrastructure, medicines etc. Policy data

Service readiness, quality, coverage, health status

Facility assessments

Population-based surveys Coverage, health status, equity, risk protection, responsiveness

Clinical reporting systems

Civil registration

Analysis and synthesis Communication and use

Data quality assessment; Estimates and projections; In-depth studies; Use of research results;

Targeted and comprehensive reporting; Regular review processes; Global reporting

4.1 There is a balanced and parsimonious set of core indicators with well-defined baselines and targets Core indicators There are thousands of health indicators and dozens of global indicator guides for specific health programmes. Data for many indicators are difficult and costly to collect, hard to interpret, and often do not meet basic quality criteria of relevance, reliability and validity. In the context of the NHS it is a major challenge to select a core set of indicators that can objectively and effectively monitor progress towards the most important objectives. Selection of indicators should be informed by considerations of scientific soundness, relevance, usefulness for decision-making, responsiveness to change, and data availability. The core indicator set should be responsive to the information needs for monitoring progress and performance towards the main objectives of the NHS and should cover inputs, outputs, outcomes and impact. The challenge is to ensure an appropriate balance across the results chain and across major programme areas. 10 | Monitoring, evaluation and review of national health strategies

There is no optimal number of core indicators but an approximate indication based on country experiences would suggest that for national, high-level strategic decision-making the total number of indicators should not exceed 25. It is important to keep in mind that quantitative indicators are intended to be indicative of reality, i.e. they are tracer indicators and they are not intended to describe the totality of what is happening. Depending on the priorities and objectives of the NHS, some of the high-level core indicators of diseasespecific plans could also be reflected in the core set of indicators for M&E. An illustrative basket of indicators is provided in Annex 3. Additional indicators will be required for lower-level programme management but very few of these will need to be included in the national core set.

Baseline and targets The core indicators should have clear baselines and targets that are well documented, relevant and measurable. A schedule for updating and reporting should be specified. Different types of targets can be defined. • An absolute target reports a simple change in the level of an indicator (e.g. an increase of vaccination coverage from 70% to 85% in five years). • A relative target reports a relative change that is independent of the initial value of the starting point (e.g. a reduction of the under-five mortality rate by one third). Relative target-setting is often used when baselines are uncertain. • An annual rate of change is a third option for setting targets. For example, the target could require that the annual rate of change increases from 2% per year to 4% per year. However, this requires data on the baseline trend rather than just its level and is often hard to measure. Target-setting should be based on criteria related to the level of aspiration and feasibility desired. Aspirational targets are often chosen to be in line with international targets such as the Millennium Development Goals (MDGs), e.g. a three quarters reduction in maternal mortality ratio by 2015. In any individual country, it is more appropriate for planning purposes to set realistic targets that can be achieved within a given time frame and resource envelope. A rational selection of targets is based on computations that include the likely availability of funding, on how this can be translated into intervention access and coverage, and ultimately on health impact.

Use of global standards Indicator definitions should be aligned with global standards, which are available from WHO for virtually all disease areas and for health system monitoring. Indicators should include all necessary metadata descriptors: a clear description of its definition, the method of data collection and analysis, the frequency of measurement, and the level of disaggregation.12 Most countries have signed up to the Millennium Development Goals (MDGs) and incorporated the MDG targets into their own health strategies. Other global initiatives, such as Stop TB, Roll Back Malaria, and the push towards universal access to HIV/AIDS services, also influence national indicator selection. Through commitments to these initiatives, countries have indicated that they will work towards meeting globally agreed targets. For these reasons, both national targets and international health targets should be reflected in national health and disease-specific strategies.

12 Eventually, every country should maintain an indicator and metadata registry, linked to the country observatory of health statistics, within which core and supplemented indicators would be identified and defined along with data sources, analytic methods and the statistical values for the indicators (see: http:// www.who.int/gho/indicator_registry/en/).

A country-led platform for information and accountability | 11

Measurement and reporting frequency A certain degree of flexibility can be introduced in the periodicity of data collection for the core indicators. Some will not be expected to change rapidly so will require relatively infrequent data collection. Indicators that are particularly sensitive to change will require more detailed data collection programmes. The frequency of measurement and reporting needs to be specified. • Input and output indicators can change rapidly and should be measured frequently (at least annually), in conjunction with monitoring of annual operational plans. • Outcome indicators – intervention coverage and selected risk behaviours – should be reported every two years, though they may be reported annually if rapid changes are expected and appropriate measurement systems are available. Some coverage indicators can be obtained on an annual basis from the health facility reports. • Impact indicators should be reported once or twice every five years, which is the average duration of a national health strategy. This longer interval reflects the fact that changes in impact do not occur rapidly, and measurement is more complex and often based on recall of events.

4.2 Disease- and programme-specific indicators are aligned Disease-specific programmes have their own strategies and operational plans that are more detailed than the overall NHS. Disease-specific M&E will generally include more indicators to measure process, outputs and immediate outcomes. Most of these indicators are used for programme management at the local level such as training and logistics. The main disease programme indicators should however be aligned with the outcome and impact indicators of the M&E component of the NHS to demonstrate the effects of health system strengthening to health outcomes.

4.3 Integrated with the national health information system strategy The monitoring and evaluation work in the context of the national health strategy should be linked with the national health information system (HIS) strategy and plan. A HIS strategy is broader than a monitoring and evaluation strategy, as it should cover all details of the institutional requirements and procedures required of the different producers and users of health information system. It should also include the role of information and communication technology. A HIS plan provides specific goals and milestones, as well as the costs. An overview of the information needs and use at different levels of data collection is described in the HMN Framework and standards for country health information systems.1 Identification of the core set of indicators should be through an inclusive process, bringing together key decision-makers across diseasefocused and system-specific programmes. Such a process should focus on the lead public health priorities that are addressed in the NHS.

12 | Monitoring, evaluation and review of national health strategies

Attribute 5. The monitoring, evaluation and review component specifies data sources, identifies and addresses data gaps, and defines responsibilities for data collection and information flow For each indicator, the preferred data source should be identified along with the best alternatives. Data collection periodicity, budgets and responsibilities should be clearly identified and reflected in the M&E plan. Sources of health data can be divided into two broad groups: (i) those that generate data relative to populations as a whole; and (ii) those that generate data as an outcome of health-related administrative and operational activities. Other sources of information such as health systems research and longitudinal community studies may also feed into the M&E system. The goal is that all countries have in place the range of data sources needed to generate a critical data set.

5.1 Data sources are specified in a comprehensive and integrated manner The use of existing data from all data sources is required to assess progress and performance. It is therefore essential to specify how data on all components of health systems will be generated, including inputs, processes, outputs, outcomes and impact. The main data sources include the following. • Census of population and housing. This is the primary information source for determining the size of a population, its geographical distribution, and its social, demographic and economic characteristics. The M&E component should include reference and use of censuses that should ideally be held every 10 years and can provide vital statistical data. • Civil registration and vital statistics systems. The M&E component should include a statement on the use of vital statistics and on planned ways to improve the availability and quality of data on births, deaths and causes of death. • Population-based health surveys with a focus on service coverage, equity and population health outcomes. The M&E plan should include a health survey plan,13 integrated with the national statistics plans, with specific details on contents, funding and execution. • Facility generated data, including routine facility information systems and health facility assessments and surveys. The M&E plan should be specific about how the facility provides regular quality information for the core indicators, what complementary data will be collected through facility assessments, and other ways of data validation. • Administrative data sources including financial resource flows and expenditures to subnational levels. The M&E plan should include specific plans for annual reporting of expenditures by major programme.

5.2 Critical data gaps are identified and addressed For a comprehensive examination of health progress and systems performance, it is necessary to deal with substantial gaps in the availability of quality data that cannot be addressed through analytical and statistical procedures alone. Data deficiencies are likely to be multiple and varied from country to country, but a common feature observed in many countries is that routine reports from health facilities and districts are often incomplete, tardy, of poor quality, and subject to bias. The need to systematically address such problems is particularly acute in light of the increasing use of such data for annual health sector reviews and by donors as the basis for performance-based disbursement mechanisms.

13 The International Household Survey Network: http://www.ihsn.org.

A country-led platform for information and accountability | 13

Currently, performance is often assessed on the basis of routine reporting from health facilities, yet it is clear that there are multiple problems in clinic- and programme-based reporting systems. The application of tools to assess the readiness of health facilities and district performance can fill important data gaps and provide a mechanism for validating routine facility and district reports. It is essential to fill data gaps on all components of health systems functioning along the causal chain from inputs, processes and outputs, to outcomes and impact (see Figure 1). A comprehensive plan to improve the availability of information on health progress and systems performance should include identifying and addressing critical data gaps. • Strengthening vital events monitoring with causes of death, through existing civil registration systems, demographic surveillance sites or hospital statistics. • Harmonizing health surveys through a country-led national plan for population-based health surveys. • Improving the timeliness, completeness and quality of facility-generated data through a wellfunctioning routine facility information system. This should be complemented by a systematic annual facility assessment of service readiness, and record reviews for selected indicators to fill data gaps and to verify (and adjust for biases) the quality of routinely reported data that goes into progress and performance reports. • Improving administrative data sources including systems of tracking financial resource flows and expenditures to subnational levels, human resources systems and logistics management systems.

5.3 Responsibilities for data collection and management are specified The M&E component of the NHS should specify roles and responsibilities at each administrative level (national, subnational, health facility) for data collection as well as for data management and analysis. Gaps in human resource capacities should be clearly addressed. The M&E plan should also detail administrative and facility data analysis procedures at each level (health facility, subnational or national) and should identify mechanisms for feedback on performance at each level. For household surveys and civil registration, mechanisms for data collection and analysis, as well as for data management, should generally be the responsibility of the NSO and be in line with international standards. Information technologies can play an important role in innovation, for example, for efficient data generation (patient and facility records, field-based data collection), and data sharing and exchange through interoperable databases, which may be located at facility, district, regional and national levels.

Attribute 6. Data analysis and synthesis work is specified, and data quality issues are anticipated and addressed The M&E component should include analysis and synthesis of data from multiple sources including the use of qualitative data. This helps to ensure that contextual developments are taken into account. Engagement of national academic, public health and research institutions will foster a broader understanding of the potential of analysis to improve the health-related statistics.

6.1 Data analysis and synthesis work is specified M&E involves data analysis and synthesis, summarizing the results into a consistent assessment of the health situation and trends, and using core indicators and targets to assess progress and performance. This can be complemented by more complex analyses that generate best estimates, for example, of the burden

14 | Monitoring, evaluation and review of national health strategies

of disease, patterns of risk behaviour, health service coverage, and health system performance. There is also a need to make use of health systems research as well as qualitative data gathered through systematic processes of analysing health system characteristics and changes.

6.2 There are regular assessments of progress and performance, including systematic analyses of contextual and qualitative information The overall assessment of progress and performance is based on the analysis of progress with equity and efficiency analyses. The results of these analyses are interpreted in light of national strategies and policies and take into account international developments as well as contextual changes. A sample outline for a health sector progress and performance report is included in Annex 2. Health progress and performance assessment brings together the different dimensions of quantitative and qualitative analyses and should include analyses on: (i) progress towards the NHS goals; (ii) equity; (iii) efficiency; (iv) qualitative analyses of contextual changes; and (v) benchmarking. • Progress towards NHS goals. The M&E should measure the extent to which the objectives and goals of the NHS (core indicators and their targets) have been attained. This can be complemented by a stepwise analysis to assess which policies and programmes were successful, from inputs such as finances and policies; to service access, quality and utilization; to coverage of interventions; to health outcomes, financial risk protection and responsiveness (see Figure 2). • Equity. M&E should measure progress in terms of distribution of the health system goals. Equity involves analyses of differences within and between groups, among peer countries, etc. using a series of stratifiers and summary measures. In terms of equity, subnational analyses are of particular interest and are conducted by most countries. The utility of subnational analyses not only lies at the national level but is of particular interest to provincial and district health decision-makers. • Efficiency. M&E should measure the extent to which the resources used by the health system have produced the maximum possible benefit to society. This measure relates the level of attainment of goals to the inputs used to achieve them. Efficiency measures the extent to which the resources used by the health system achieve the goals that people value. It is a way of ensuring that the resources available to the system are combined to produce the maximum possible benefit to society. Efficiency analyses should be part of the final reviews of a NHS. • Qualitative assessment and analyses of contextual changes. M&E should take into account nonhealth system changes, such as socioeconomic development that affect both implementation and the outcomes and impact observed. The quantitative analyses cannot capture all elements that are required to review progress and performance. Qualitative information on the leadership, policy environment and context is crucial to understand how well and by whom government policies are translated into practice. A systematic and participatory effort is required to gather, analyse and communicate qualitative information. This needs to be brought together with quantitative data as the basis for a policy dialogue, which then becomes a solid basis to inform planning cycles, regular reviews, and monitoring and evaluation. Such analyses should include an assessment of: –– primary health care reforms and policies stating the importance of assuring service coverage for all communities; –– the health sector or disease programme response and adherence to policy direction to date; –– how cultural and political factors impact on health or multisectoral programmes and outcomes; –– organizational context, leadership and accountability mechanisms; –– the regulatory environment and how it enables or hinders improvements in health systems and programme delivery.

A country-led platform for information and accountability | 15

• Benchmarking. Benchmarking refers to comparisons between and within countries to assess performance. There are different types of benchmarking which may vary according to the level of comparison (international or national), level of assessment (individual service provider, facility, care organization, district/province, national), measurement focus (process, outcomes, quality, performance), and uses of data (public reporting, accountability, internal reporting only, self-learning and improvement). There are multiple ways of benchmarking, involving (i) systematic comparisons between for instance the median or mean of “peers” with or without percentiles; (ii) the best performers among the “peers”; (iii) an international or national target and (iv) past performance, based on a time series. Peer countries or subnational units may include geographically close units and units with the same level of economic and social development.

6.3 Specific processes for data quality assessment and adjustment are in place and are transparent All data sources have their strengths and weaknesses that need to be taken into consideration when analysing the data. For example, data from household surveys are subject to uncertainty due to sampling and non-sampling errors; they may not be up-to-date; and they can rarely generate usable data for district level analysis due to sample size and cost limitations. In terms of routine health information systems, reports from health facilities and districts are often incomplete, tardy, of poor quality, and subject to bias.

Figure 2. Stepwise approach to review health progress and performance Inputs and processes Infrastructure; Information and communications technology Health workforce Supply chain Information

Outputs

Outcomes

Impact Improved health outcomes and equity Social and nancial risk protection Responsiveness

Governance

Financing

Intervention access and services readiness Intervention quality, safety and e ciency

Coverage of interventions Prevalence risk behaviours and factors

Have nances been disbursed? Have policies been changed? Is the process of implementation happening as planned? Has access to services improved? Has the quality of services improved? Has utilization improved? Has intervention coverage improved? Have risk behaviours improved?

Contextual changes Non-health system determinants

Contextual changes Non-health system determinants

Have health outcomes and equity improved? Are services responsive to the needs? Are people protected against nancial risks?

16 | Monitoring, evaluation and review of national health strategies

Annual progress and performance are often assessed on the basis of routine reporting from health facilities, yet it is clear that there are multiple problems in clinic- and programme-based reporting systems. Financial incentives may aggravate the problem by leading to data manipulation. Priority attention should be given to developing the capacity and motivation of sector and programme implementers to collect, analyse and use data to improve services and interventions. The M&E component should include regular and systematic data quality assurance processes that are transparent and in line with international standards. These processes include the following. • Data quality assessment and adjustment (DQAA). Identifying and accounting for biases due to incomplete reporting, inaccuracies and non-representativeness is essential and will greatly enhance the credibility of the results. This involves a multi-step process including: (i) assessment of the completeness of reporting by facilities and districts; (ii) assessment of the accuracy of subnational population denominators (often obtained from Bureau of the Census projections); (iii) accuracy of coverage estimates from reported data; (iv) systematic analysis of facility-based and household survey-based indicator values; and (v) adjustments of the indicator values, using transparent and well-documented methods. The DQAA should be done on a regular basis and the results should be made public. The M&E component should also specify the institutions responsible for the process. • An annual system of health-facility assessments, including an assessment of service readiness combined with a record review. This will serve to fill critical data gaps on service delivery as well as serve to verify the quality of routine facility data. Every year, prior to the annual health sector review, a facility assessment can be conducted in a sample of facilities to independently review the quality of health data and the status of service delivery. The assessment can be combined with a review of the records to ascertain the completeness and quality of reporting by comparing the results with aggregated data at district, provincial and national level. A facility assessment provides essential information on service delivery (such as the availability of key human and infrastructure resources), on the availability of essential medicines, and on the readiness of health facilities to provide basic health-care interventions relating to family planning, child health services, basic and comprehensive emergency obstetric care, HIV, tuberculosis, malaria and noncommunicable diseases.14 • Regular training of staff and provision of routine feedback to staff at all levels on the completeness, reliability and validity of data. • A functional national-level M&E committee that meets on a regular basis and supports data quality assurance checks at facility level can help to raise the credibility of the data and reporting system. • Establishing a data and information repository as a shared resource at national, subnational and district levels is an important step in improving information practices and enabling high-quality data analyses.

14 Measuring service availability and readiness. Geneva, World Health Organization, 2011.

A country-led platform for information and accountability | 17

Attribute 7. Data dissemination and communication are effective and regular Data have little value in themselves but need to be translated into information that is relevant for decisionmaking. This requires packaging, communication and dissemination of statistics in a format and language accessible to higher-level policy- and decision-makers. Information is used at various levels of the health system for health service delivery, health system management, resource allocation, planning, advocacy and policy development. A broad range of users are involved, each from different technical perspectives, and each with associated vocabularies and methods of communication. Dissemination should address the needs of different users and identify the most effective packaging and channels of communication. The timing of information dissemination should be planned carefully to fit in with the planning cycles and needs of users. Communications experts can assist with the packaging of information for different audiences. Information technology provides new ways of effectively communicating data to specific audiences. The dynamic links between demand, supply and quality of information should be addressed by encouraging an information culture where information is demanded and the use of information promoted. In practical terms, this depends on the establishment of institutional mechanisms and incentives for information use. Some of the most effective mechanisms involve linking data and information to actual resource allocation (budgets) and developing indicator-driven planning. The key is to build these mechanisms around existing data processes in a country, and to strengthen the availability, quality and use of data within those processes rather than propose new ones.

7.1 Analytical outputs as the basis for national and global reporting are defined and produced The M&E plan should specifically define the analytical outputs that will be produced as the basis for decision-making, programme management, financial disbursements and global reporting. In general, there are two critical outputs of the health information systems that serve the basis for national and global reporting.

Health sector progress and performance report A health sector progress and performance report is the key input for NHS annual reviews and is based on the analysis and synthesis work described in the previous section (Attribute 6). This includes a systematic assessment of progress against specific objectives and goals in the national health strategy, as well as in disease-specific plans. The health sector progress and performance report brings together all data from the different sources, including the facility reporting system, household surveys, administrative data and research studies, to answer the key questions on progress and performance using the country’s core indicators and health goals.

Annual health statistical report This report presents a comprehensive analysis of all health data derived from administrative and health facility reporting, including the most relevant data by district. Detailed data should also be available on the web. The annual health statistical report provides ample attention to data quality issues, including timeliness, completeness and accuracy of reporting, as well as to data adjustments and methods used, if applicable.

18 | Monitoring, evaluation and review of national health strategies

7.2 Appropriate decision-support tools and approaches are used Policy-makers need to make decisions on many topics, often in tight time frames, and therefore require information that is synthesized and packaged to inform their priority decisions. However, this is not how data are always presented to them. Frequently, researchers and M&E specialists present data by data source or health indicator, rather than by programmatic decision area or programmatic target. When data are presented around decision areas, the policy-maker can immediately see the link between what is being presented and upcoming decision needs. It allows them to ask follow-on questions and seek additional information that will inform action. Moreover, policy-makers are often unfamiliar with research and M&E terminology and methods. They may be ill-equipped to assess the appropriateness of a specific data point or research finding to a programmatic action. They may also often find themselves overwhelmed with data. It may be unclear to them which result or specific finding is the most relevant to the decisions that they are required to make. They may struggle to find meaning in a sea of numbers, tables, charts and graphs that are presented to them. Greater emphasis needs to be placed on linking the tables, charts and graphs to the specific programmatic decisions facing the policy-maker. Strengthening this linkage requires a focused presentation limited to the priority data needed to inform the decisions at hand. This will in turn reduce the perception of data overload that may be felt by the policy-maker. In this streamlined context, the policy-makers’ understanding of how the data presented to them can support their upcoming decisionmaking processes is improved. The use of decision-support tools and approaches such as data dashboards, health summary bulletins, health status report cards, and colour-coded data presentation techniques have proven effective in improving evidence-based decision-making, especially when tailored to the needs of specific policy audiences. A brief description of the benefits of each decision support tool/approach is provided below. • Health summary bulletins usually contain information on key health indicators in a specific programme area. The information is usually presented in tables and graphics with some explanatory text. Summary bulletins often contain large amounts of information compiled from different data sources. This information is usually not interpreted in the context of specific decision-making, and recommendations for programmatic changes are usually not provided. However, they are an important way to display synthesized data that provides an overall picture of the health status in a given country. Typically, they are often best targeted to programme managers and other decision-makers with in-depth knowledge of the specific programme area. • Health status report cards report on key health indicators in a specific country or programme area. A report card is different from a health summary in that it reports on fewer health indicators and compares current progress to a target or to past report card trends. A grade is developed to convey the programme’s success in meeting the specific target or in improving progression in each health indicator over a period of time, to allow for direct comparison between reports. The grade is usually depicted to match the common grading system for the specific country. The grading provides decision-makers with an at-a-glance indication of whether or not a specific service or health indicator needs attention.15 • Policy briefs highlight actionable recommendations for decision-making in a 2–6 page format. The typical format identifies a problem, proposes a solution and presents a compelling and feasible recommendation. Non-academic language is used and images, quotes, photographs, and bullets are recommended. The supporting evidence is also highlighted. This format is ideal for conveying specific evidence-based policy recommendations.16

15 North Carolina women’s health report card 2009. Center for Women’s Health Research, University of North Carolina at Chapel Hill, 2009 (http://cwhr. unc.edu/files/report-card-pdfs/rc-2009/2009_RC_Eng.pdf). 16 Preparing a policy issue brief (https://www.courses.psu.edu/hpa/hpa301_fre1/IBInstructions_fa02.PDF).

A country-led platform for information and accountability | 19

• Data dashboards visually present critical data in summary form so that decisions can be made quickly. Dashboards give an at-a-glance perspective on the current status of a project in the context of predetermined metrics for that project. Dashboards are linked to a database so that users can change key inputs to see how they affect what is displayed on the dashboard, and so that they can drill down to source data to understand the relationships they see on the dashboard. Dashboards assist in the management of the large amounts of data that are being collected by health programmes by tracking key programme metrics and displaying trends. This allows users to identify problems and target specific follow-up activities to improve services.17 • Colour coding is a strategy used to group data and suggest action. Most commonly the colours red, green and yellow are used to depict a traffic stop light. Specific numerical ranges are pre-determined for each colour and indicator, based on progress towards a programmatic target. This technique allows decision-makers to see at a glance if action is required around a specific indicator. There are similarities with this strategy and with the grading found in health report cards.

7.3 Data, methods and analyses are publicly available Public access to data and reports is an essential element of accountability. This should include primary data, metadata, and the methods used to adjust and compute the statistics and the final reports, e.g. data on the completeness of reporting. Better access to data and statistics in the public domain will generate important benefits at country and global levels by fostering collaboration and innovation in statistical and analytical methods, both for new data collection and for better use of existing data. Data sharing requires collaboration between primary data producers, and primary and secondary users, as well as measures to protect confidentiality and security. At country level, there is a need to enhance individual and institutional capacities for data management, including data archiving, supported by development partners and donors as an integral part of programmes and projects. Quantitative and qualitative data should be brought together and made publicly accessible through country health observatories or intelligence portals, and synthesized to monitor the country health progress and performance, and to support the policy dialogue. This country repository of information should be built on a web-based platform and aim to improve access to all available data on key health indicators and on systems performance. Type of documents that can be included in the observatory are country health data, statistics documents, country health statistics, estimation tools and results, communication tools and results, and international standards. This web platform is not intended to replace existing or planned country websites which often cover multiple purposes. Ministries of Health and National Statistical Offices maintained websites however should be able to draw freely and easily from the health observatory country pages.

Attribute 8. Prospective evaluation is planned and implemented 8.1 Prospective evaluation is planned and linked to monitoring, evaluation and review of national health strategies Evaluation is a neglected aspect of M&E. Evaluation is defined as the rigorous, science-based analysis of information about programme activities, characteristics, outcomes and impact that determines the merits

17 Using dashboards to facilitate data informed decision making in health programs. Data Use Net discussion summary (http://www.cpc.unc.edu/ measure/networks/datausenet/dashboards-and-data-use-forum-may-2010/summary.pdf).

20 | Monitoring, evaluation and review of national health strategies

of a specific programme or intervention. This differs from monitoring which is defined as the regular tracking and reporting of priority information about a programme and its intended outputs and outcomes. Evaluations are often brought in as an afterthought, when multiple data gaps preclude a useful assessment of what has worked and what has made the greatest impact. Ideally, evaluations need to be planned well in advance and be conducted in a prospective manner. Evaluations should be planned at the same time as the development of the monitoring and evaluation plan for the national health strategy and should be a key component of the plan. Prospective evaluation should be based on a solid monitoring system with data on baseline trends for key indicators, provided by the country M&E platform. District-level information is required for large-scale effectiveness evaluations, including continuous monitoring of different levels of indicators.18 Such data need to be complemented by in-depth studies – both quantitative (preferably longitudinal) and qualitative – and analyses that bring together all data and aim to draw conclusions about the attribution of changes to specific interventions and to carefully assess the role of contextual changes. Furthermore, if effectiveness of the interventions can be established, this is where cost-effectiveness analysis is essential to draw the ultimate conclusions. Analytical techniques are required to deal with gaps and biases. Evaluation should constitute an integral part of progress and performance reviews. The interim and summative evaluations are complementary to mid-term and final reviews of the NHS. The main difference is that evaluations aim to assess the extent to which specific interventions have contributed to observed outcomes and impact. Evaluations therefore ideally include a determination of the counterfactual – i.e. a determination of what would have happened in the absence of the interventions. Where possible, evaluations should use data from, and strengthen, health sector reviews. They should build upon existing country systems and include an explicit capacity building and system strengthening objective, where appropriate.

18 Victora CG et al. Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness evaluations. The Lancet, 2010, 377(9759):85–95.

A country-led platform for information and accountability | 21

IV. Country mechanisms for review and action Attribute 9. There is a system of joint periodic progress and performance reviews The experience from sector-wide approaches (SWAps)19 and multisectoral AIDS strategies, among others, has shown that periodic progress and performance reviews are critical for updating all stakeholders on programme progress, discussing problems and challenges, and developing a consensus on corrective measures or actions needed. Progress and performance reviews are part of the governance mechanisms that help ensure transparency and allow for debate between partners. The M&E component of the NHS should describe processes by which M&E results can be reviewed and influence decision-making, including financial disbursement. In general, based on the analysis, synthesis, and inclusive policy dialogue, periodic revisions are made to national strategic and operational plans. Figure 3 illustrates how progress and performance are assessed by annual reviews, which result in adjustments to annual operational plans by a mid-term review, and by a final evaluation. The development of any new NHS should build upon the final review of the mid-term evaluation results, and of the prospective evaluation results of the previous strategy. To close the loop, the results of the prospective evaluation should serve as the initial situation analysis and evidence base for the subsequent annual health strategy development process.

9.1 A regular and transparent system of reviews with broad involvement of key stakeholders is in place The stakeholders that should be included in any joint review mechanism include key MoH staff and departments, representatives of subnational teams, other key ministries (e.g. finance, planning), global partners, non-state implementing partners such as CSOs, and the private sector. It is up to each government to determine, with its partners, how best to include its many stakeholders in the review of sector, programme or multisectoral programme performance, and what the periodicity of the reviews should be. Experience from many SWAp countries indicates that annual reviews are particularly helpful when they can be used to feed into the next annual operational plan. An annual review then becomes an opportunity to take stock of progress made, to analyse what is working well and what is not, and to assess whether a reprioritization, change of direction or reallocation of funding is required. The NHS generally spans a five-year period. Regular progress and performance reviews should evaluate performance using service or programme output and outcome indicators. Mid-term and end-of-plan reviews should be more extensive and also cover impact indicators.

19 Sector-wide approaches: http://www.who.int/trade/glossary/story081/en/.

A country-led platform for information and accountability | 23

Figure 3. Progress and performance review of the national health strategy Year 1 Year 2 Year 3 Operational plans Annual review Annual review Mid-term review Annual review Final and situation analysis

Year 4

Year 5

Progress and performance reviews and policy dialogue

Data analysis and synthesis Systematic qualitative assessment Surveys Facility data Civil registration Administrative data Research Leadership Policies Context

Reporting on progress and performance of the NHS involves three types of reviews. • Annual review. The annual review is focused on the indicators and targets specified in annual operational plans. These are mainly input, process and output indicators. If available, coverage indicators are also used. The annual reviews should help inform evaluation on a regular basis. • Mid-term review. This is normally conducted half way through implementation of the NHS. It covers all the targets mentioned in the strategy, including targets for outcome and impact indicators, and also takes contextual changes into account. The mid-term review should coincide with the annual review (e.g. the third year in a five year plan). The results are used to adjust national priorities and objectives. • Final review. This involves a comprehensive analysis of progress and performance for the whole period of the NHS. The final review builds upon the annual and mid-term reviews, but also brings in results of specific research and of prospective evaluation that should be built in from the beginning. Different issues may be addressed with varying levels of depth or rigour during reviews, depending on the needs of the country. For example, a country may want to review a specific issue, e.g. human resources for health or health financing, during each of its annual reviews, but it may choose to examine the issue in greater depth in some years than in others, depending on its specific information needs.

9.2 There are systematic linkages between health sector reviews, disease- and programmespecific reviews, and global reporting Detailed programme-specific reviews should not be conducted as separate, parallel activities – rather they should be linked to the overall health sector review and contribute to it. Ideally a programmespecific review should be conducted prior to the overall health sector review, and help inform the content of the health sector review in relation to that specific programme area. It is important that the specific programme reviews involve staff and researchers not involved in the programme itself to obtain an objective view of progress.

24 | Monitoring, evaluation and review of national health strategies

Global reporting requirements should primarily be based on ongoing country processes of data generation, compilation, analysis, synthesis, communication and use for decision-making. This requires harmonization and minimization of global reporting requirements and increased coordination between “vertical” disease programmes and “horizontal” health systems actions. The M&E component of the NHS should serve as the basis for all M&E that is related to processes such as the health sector component of the SWAps and IHP+. It should also serve as the basis for programme activities supported by GAVI, the Global Fund and other donors, and for disease- and programme-specific needs. The aim is to minimize transaction costs for countries and global partners, reduce fragmentation and duplication, and jointly strengthen national health information systems, while meeting global standards. Getting it right is an iterative process that combines the strengthening of country reviews with alignment of global reporting and reviews. Ultimately, this will lead to much greater efficiency and better quality that benefits all. Figure 4 shows the common M&E platform for national health strategies with country data generation and use processes in the centre. The following principles provide guidance to enhance the alignment of global performance reviews with M&E of the NHS. • Use of country-led reporting and health reviews. These include explicit indicators and time-bound targets and provide results on health systems strengthening as well as specific health and disease programmes. The focus is on annual cycles to which all partners are committed to use as a basis of continued partner funding. This also implies that external performance reviews are used in country review processes. • Explicit, transparent performance ratings. These should be based on an initial country review of performance as part of the health review process. These include explanations of deviations between results and targets, including catch up plans, supported by guidance on the ratings and how they are adjusted by contextual factors. • Use of clear performance incentives. These should be used to accelerate funding to grants related to activities that are supported, to invest in systems strengthening for those which do not fully perform but show potential, to reduce funding where it is not used, and exceptionally to stop funding. • Joint M&E system strengthening for performance reviews. These should include investments to build the capacity of countries in reporting, data quality and analysis, as well as to improve performance reviews. Partners could commit as much as 5 to 10% of health funding to these activities.

Attribute 10. There are processes by which related corrective measures can be taken and translated into action 10.1 Results from reviews are incorporated into decision-making, including resource allocation and financial disbursement The M&E component should describe processes by which monitoring results can influence decisionmaking, including resource allocation and financial disbursement. The M&E plan should detail how results from progress and performance analyses will be formally incorporated into future decision-making, including mechanisms used by government and funding partners to make resource allocation decisions and financial disbursements to programmes and subnational levels.

A country-led platform for information and accountability | 25

Figure 4. Country-led platform for monitoring and review of the national health health strategy External validation and estimates Monitoring reports GAVI reporting

Common standards and tools

Analysis and synthesis

MDG/UN reporting

Global Fund reporting

Da as ta q ses ua sm lity en t

Country data information generation and compilation

Programme reports

Reviews

Evaluation

Statistical reports

Reviews

t en nd s pe de ew In revi

Programme Reporting (Tuberculosis, Maternal and child health, HIV, etc.)

PEPFAR reporting

Minimization of | reporting requirements

Harmonization of reporting requirements

In the context of their national health strategies, most countries use annual operational plans to prioritize activities and allocate resources. The national reviews should form the basis for the decision-making process. This implies that there needs to be a time lag between the review of the monitoring data and the development of the next annual operational plan. An important element is the presence of formal mechanisms, such as the presence of multi-stakeholder M&E or a financial management committee that meets regularly to review progress, identify constraints and bottlenecks, and advise on ways to reduce them.

26 | Monitoring, evaluation and review of national health strategies

10.2 Multi-stakeholder mechanisms are specified to provide routine feedback to subnational stakeholders The M&E component should describe multi-stakeholder mechanisms to provide routine feedback on performance to subnational and non-state providers. Just as it is important to take stock and review performance at a national level, so it is important to provide feedback on performance to subnational levels and implementers. Feedback loops, where information flows to central levels and back to those providing the information in the first place, have been shown to give a number of benefits. First, performance feedback can help local managers, supervisors and implementers to consider what their own strengths and weaknesses are, and where they need to be making more of an effort. Second, for those collecting the information, seeing how that data are used, and how it can assist their own work and the work of their colleagues, helps motivate them to improve the quality of the information they provide.

A country-led platform for information and accountability | 27

Annexes

Annex 1. Monitoring, evaluation and review platform: sample outline of a monitoring, evaluation and review plan for the national health strategy Chapter 1: National health strategy as basis for results and accountability 1.1 1.2 1.3 1.4 Goals and objectives of the national health strategy Current status of the health information system Process for development of the monitoring, evaluation and review component Disease- and programme-specific monitoring, evaluation and review alignment

Chapter 2: Institutional capacity 2.1 Key country-led coordination mechanisms 2.2 Roles and responsibilities of key country institutions and stakeholders 2.3 Country capacity-building strategy

Chapter 3: Monitoring and evaluation framework 3.1 Monitoring and evaluation framework 3.2 Indicators • Core indicators, baselines and targets, and reporting frequencies • Disease- and programme-specific indicators: links 3.3 Data sources • Data collection needs for all core indicators • Critical data gaps and weaknesses and how to address these • Data management 3.4 Data analysis, synthesis and quality • Data analysis and synthesis work • Regular assessments of progress and performance • Processes for data quality assurance 3.5 Evaluation component 3.6 Data dissemination and use • Analytical outputs and responsibilities

Chapter 4: Country mechanisms for review and action 4.1 System of joint periodic progress and performance reviews for use in decision-making 4.2 Links between programme-specific reviews and the general health sector review 4.3 Decision-making processes for remedial action and financial disbursement

Budget

A country-led platform for information and accountability | 29

Annex 2. Annual health sector progress and performance report: sample outline Executive summary Chapter 1: Introduction • Goals and objectives of the national health strategy • Methodology for the development of the progress and performance report, including data quality issues

Chapter 2: Measuring health sector performance • Analysis of progress and trends (based on measurement of baseline and targets of core indicators of the national health strategy) for core indicators • Has money been disbursed as planned? Have policies been changed? • Is the process of implementation happening as planned? • Did access to and utilization of services improve? Did the quality of services improve? • Did intervention coverage or risk behaviours improve? • Have health outcomes and equity improved? Are services responsive to the user’s needs? Are people protected against financial risks? What is the efficiency of service delivery (value for resources)?

Chapter 3: Summary findings, policy recommendations and proposed actions

30 | Monitoring, evaluation and review of national health strategies

Annex 3. Potential basket of indicators for monitoring health sector progress and performance WHO illustrative list of core indicators for health sector reviews Inputs and processes Health financing • Total health expenditure per capita • General government expenditure on health as % of general government expenditure

Outputs Service access and readiness • General service readiness (service specific readiness)

Outcomes Coverage of interventions • Antenatal care (4+ visits) • Skilled birth attendance • DPT3 immunization (measles, HiB, HepB, pentavalent) (institutional delivery; postnatal care)

Impact Health status • Life expectancy at birth • Child mortality rates (under 5) (perinatal, neonatal, infant)

• Average availability of 14 selected medicines • Median price ratio for tracer medicines • Number and mean outpatient visits per person per year

• Maternal mortality ratio • Mortality rates by major cause of death, by age and sex (mortality between 30 and 70 due to CVD, cancer, chronic respiratory diseases and DM)

Health workforce • Health workers per 10 000 population (doctors, nurses, midwives; urban–rural)

• Family planning need satisfied (contraceptive prevalence)

• Annual number of graduates per 100 000 population

Service quality and safety • TB treatment success rate (DOTS) • 30 day hospital case fatality rate acute myocardial infarct (stroke) • Waiting time to elective surgeries (cataract, PTCA, hip replacement)

• Children with ARI taken to facility (received antibiotics)

• TB prevalence • HIV prevalence among young people (15-24) • Notifiable disease incidence (measles, neonatal tetanus)

• Children with diarrhoea receiving ORT (with continued feeding)

Infrastructure • Health facilities per 10 000 population (hospital beds per 10 000 population)

• ITN use among children (ITN among pregnant women; household ITN possession)

• Adolescent fertility rate

• ARV therapy among those in need • ARV prophylaxis among HIV+ pregnant women • TB case detection • Cervical cancer screening, 20-64 years (breast cancer screening)

Information • Percent of deaths that are registered (births registered)

• Surgical wound infection rate (% of all surgical interventions)

Financial risk protection • Out of pocket expenditure as % of total health expenditure (catastrophic health expenses)

Responsiveness • User satisfaction

Governance • National health strategy having the main attributes (IHP+) (presence of key policies: essential medicines, and pharmaceutical; TB; Malaria; HIV/AIDS; maternal health; child health/ immunization)

Risk factors and behaviours • Tobacco use among adults (youth)

• Hypertension prevalence • Alcohol per capita consumption (per drinker) • Obesity among adults • Access to safe water • Access to improved sanitation • Children under 5 who are stunted (underweight, overweight, wasted)

• Low birth weight among new born • Exclusive breastfeeding (initiation on first day)

• Condom use at last highrisk sex (15-24)

A country-led platform for information and accountability | 31

Example: Illustrative core indicators for maternal, new born and child health programme reviews Inputs and processes Health financing • MNCH expenditure per target population (children, women)

Outputs Service access and readiness • Facilities that offer and meet tracer criteria for basic and comprehensive obstetric care per 10 000 pregnant women • Caesarean section rate in rural population • Facilities that offer and meet tracer criteria for child health services per 1000 children

Outcomes Coverage of interventions • Antenatal care (4+, with TT) • Skilled birth attendance • DPT3 Immunization coverage • % Need of family planning satisfied • Children with ARI taken to health facility • Children with diarrhoea receiving ORT • Children with fever receiving anti-malarials • ITN use among children • ARV prophylaxis among HIV+ pregnant women • Vitamin A supplementation among children • Postnatal care

Impact Health status • Child mortality (under-5) • Maternal mortality ratio • Child mortality by major cause of death by sex and age

General government • Expenditure on health as % of general government expenditure

Financial risk protection • Out of pocket as % of total health expenditure

Health workforce • Midwives per 10 000 population

Governance • Presence of key policies to promote MNCH

Information • Birth registered • Deaths registered (with cause)

Risk factors and behaviours • Condom use at last higher risk sex • Access to safe water • Access to improved sanitation • Low birth weight among newborns • Breastfeeding exclusively for 6 months • Children under 5 who are stunted/underweight

32 | Monitoring, evaluation and review of national health strategies

Metadata and data sources for potential indicators for monitoring health system progress and performance INDICATOR (additional dimension) Inputs and processes DEFINITION DATA SOURCES (preferred alternatives) TOPIC

Health financing 1. Total health expenditure per capita3,5,6,7,8 (MNCH expenditure per target population) General government expenditure on health as % of general government expenditure3,5,6,8 Per capita total expenditure on health (THE) (MNCH expenditure per target population) expressed in PPP international dollar Level of general government expenditure on health (GGHE) expressed as a percentage of total government expenditure National health accounts; Expenditure review Financing

2.

National health accounts; Expenditure review

Financing

Health workforce

3.

Health workers per 10 000 population3,5,7,8 (Doctor, nurse/ midwife; urban–rural) Annual number of graduates per 100 000 population5,7 (By occupation, specialization and sex)

The number of health workers (doctor, nurse/midwife) available in a country relative to the total (urban–rural) population Number of graduates (male–female) from health profession educational institutions (including schools of medicine, dentistry, pharmacy, nursing, midwifery and other health services) during the last academic year, divided by the total population

Routine administrative records; census; facility assessments Routine administrative records from individual training institutions

Human resources

4.

Human resources

Infrastructure 5. Health facilities per 10 000 population5,7 (Hospital beds per 10 000 population) The number of health facilities (hospital beds per 10 000 population) available relative to the total population for the same geographical area District and national databases of health facilities Access

Information 6. Percent of deaths that are registered5,7 (Births registered) Number of deaths registered (births registered) as reported by civil or sample registration systems, hospitals and community-based reporting systems Administrative records Information

Governance 7. National health strategy having the main attributes (IHP+)5 (Policies on: essential medicines and pharmaceutical; TB; malaria; HIV/AIDS; maternal health; child health/ immunization) National health policies (policies on: essential medicines and pharmaceutical; TB; malaria; HIV/AIDS; maternal health; child health/immunization) outline priorities and the expected roles of different actors, inform and build consensus, and estimate the resources required to achieve goals and priorities Review of national health strategy Governance

A country-led platform for information and accountability | 33

INDICATOR (additional dimension) Outputs

DEFINITION

DATA SOURCES (preferred alternatives)

TOPIC

Service access and readiness 8. General service readiness5 (Service-specific readiness) Facilities that offer and meet tracer criteria to provide general services (service specific readiness with tracer criteria for MNCH, TB, HIV, Malaria, etc.) The average percentage of medicines outlets (public–private outlets), where a selection of essential medicines are found on the day of the survey Consumer price ratios are calculated as the ratio between median unit prices (e.g. price per tablet or therapeutic unit) and Management Sciences for Health (MSH) median international reference prices for that exact product for the year preceding the survey (per public–private medicine dispensing points) The number of outpatient visits to health facilities (hospital admissions; % of caesarean sections) relative to the total population of the same geographical area (percentage of live births delivered by caesarean section) Facility assessment HSS service delivery; readiness

9.

Average availability of 14 selected essential medicines5,7,8 (Public–private outlets) Median price ratio for tracer medicines5,7,8 (Public–private)

Facility assessment

Access

10.

Facility assessment

Access

11.

Number (and mean) outpatient visits per person per year5 (Hospital admission rate; caesarean section rate3,8)

Facility reports; facility assessment, populationbased surveys

Utilization

Service quality and safety 12. TB treatment success rate (DOTS)10 The proportion of new smear-positive TB cases registered under DOTS in a given year that successfully completed treatment, whether with or without bacteriological evidence of success (“cured” or “treatment completed” respectively) Proportion of hospital in-patients with primary diagnosis acute myocardial infarction (AMI) (stroke) who died within 30 days after the admission Average inpatient waiting time for elective (i.e. non-urgent) surgeries (cataract, coronary angioplasty (PTCA), hip replacement), measured in number of days Surgical wound infection rate, as % of all surgical operations Facility reports Quality; TB

13.

30-day hospital case fatality rate acute myocardial infarction4 (stroke) Waiting time to elective surgeries4 (cataract, coronary angioplasty (PTCA), hip replacement) Surgical wound infection rate (% of all surgical interventions)4

Hospital records

Quality; NCD

14.

Hospital records

Access; NCD

15.

Hospital records

Quality

34 | Monitoring, evaluation and review of national health strategies

INDICATOR (additional dimension) Outcomes

DEFINITION

DATA SOURCES (preferred alternatives)

TOPIC

Coverage of interventions 16. Antenatal care (4+ visits)2,3,6,7,8 (ANC coverage (1+ visits)) The percentage of women aged 15–49 with a live birth in a given time period who received antenatal care by a skilled health provider at least four (or more) times (1+ visits) during pregnancy (Percentage of women attended at least once during pregnancy by skilled health personnel for reasons related to the pregnancy) The proportion of live births attended by skilled health personnel (percentage of mothers who gave birth in a health institution; percentage of mothers who receivedpostnatal care visit within two days of childbirth) The percentage of infants (aged 12–23 months) who received three doses of the combined diphtheria, tetanus toxoid and pertussis (measles2, HiB, HepB, pentavalent) vaccine in a given year The proportion of women of reproductive age (15–49 years) who are married or in union and who have met their need for family planning, i.e. who do not want any more children or want to wait at least two years before having a baby, and yet are not using contraception (the contraceptive prevalence rate is the percentage of women who are practising, or whose sexual partners are practising, any form of contraception. It is usually reported for women ages 15–49 in marital or consensual unions) Percentage of children ages 0–59 months with suspected pneumonia (with suspected pneumonia receiving antibiotics) taken to an appropriate health provider Proportion of children aged 0–59 months who had diarrhoea in the last 2 weeks and were treated with oral rehydration salts or an appropriate household solution (ORT) (receiving oral rehydratation therapy and continued feeding) Percentage of children ages 6–59 months who received two doses of vitamin A during the calendar year Survey; facility reports MNCH

17.

Skilled birth attendance2,3,6,7,8 (institutional delivery rate; postnatal care6)

Survey; facility reports

MNCH

18.

DPT3 immunization coverage3,6,7,8 (measles2, HiB, HepB, pentavalent) Percentage of need for family planning satisfied2,3,6,8 (contraceptive prevalence)

Survey; facility reports

MNCH

19.

Survey; facility reports

MNCH; RH

20.

Children with ARI taken to health facility3,8 (received antibiotics6)

Survey

MNCH; pneumonia

21.

Children with diarrhoea receiving ORT3,8 (with continued feeding)

Survey

MNCH; diarrhoea

22.

Vitamin A supplementation among children3,8

Survey

MNCH

A country-led platform for information and accountability | 35

INDICATOR (additional dimension) 23. ITN use among children2,3,8 (ITN use among pregnant women; household ITN possession) ARV therapy among those in need1,2,8

DEFINITION Percentage of children under five years of age (pregnant women; household) in malaria endemic areas who slept under an insecticide-treated nets (ITN) the previous night The percentage of adults and children with advanced HIV infection currently receiving antiretroviral combination therapy in accordance with the nationally approved treatment protocols (or WHO/ UNAIDS standards) among the estimated number of adults and children with advanced HIV infection The percentage of HIV-infected pregnant women who received antiretroviral medicines to reduce the risk of motherto-child transmission, among the estimated number of HIV-infected pregnant women The term “case detection”, as used here, means that TB is diagnosed in a patient and is reported within the national surveillance system, and then to WHO. The case detection rate is calculated as the number of cases notified divided by the number of cases estimated for that year, expressed as a percentage Proportion of women (aged 20–64) reporting to have undergone a cervical cancer screening test within the past three years (percentage of women (aged 50–69) who have undergone a breast cancer screening test, measured as the coverage rate of mammography testing)

DATA SOURCES (preferred alternatives) Survey

TOPIC MNCH; malaria

24.

Facility reports

MNCH; HIV

25.

ARV prophylaxis among HIV+ women1,2,3,6,8

Facility reports

MNCH; HIV; PMTCT

26.

TB case detection8,10

Facility reports

Quality; TB

27.

Cervical cancer screening (20–64 years)4,9 (breast cancer screening (50–69 years))

Survey; facility reports

NCD; MNCH

Risk factors and behaviours 28. Tobacco use among adults4,9 (Youth (13–15); male, female) Current smoking of any tobacco product prevalence estimates, resulting from the latest adult (youth; male, female) tobacco use survey (or survey which asks tobacco use questions), which have been adjusted according to the WHO regression method for standardizing (cf. WHO Method of Estimation). Percent of defined population with raised blood pressure (systolic blood pressure ≥ 140 mmHg and/or diastolic blood pressure ≥ 90 mmHg or on medication for raised blood pressure) Survey NCD

29.

Hypertension prevalence4,9

Population-based surveys; surveillance systems

NCD

36 | Monitoring, evaluation and review of national health strategies

INDICATOR (additional dimension) 30. Alcohol per capita consumption (APC)9 (per drinker8)

DEFINITION Adults (15+ years) per capita amount of alcohol consumed in litres of pure alcohol in a given population (heavy episodic drinkers is defined as the proportion of adults (15+ years) who have had at least 60 grams or more of pure alcohol on at least one occasion weekly. A consumption of 60 grams of pure alcohol corresponds approximately to 6 standard alcoholic drinks) Percentage of defined population with a body mass index (BMI) of 30 kg/m2 or higher (overweight = (BMI) of 25 kg/m2 or higher ) The percentage of population (urban– rural population) using an improved drinking water source The percentage of population (urban– rural population) using an improved sanitation facility Percentage of children under five years of age whose height-for-age is below minus two standard deviations from the median of the WHO Child Growth Standards11 (prevalence of (moderately or severely) underweight children is the percentage of children under five years old whose weight for age is less than minus 2 standard deviations from the median for the international reference11 population ages 0–59 months; Percentage of overweight (weight-for-height above +2 standard deviations of the WHO Child Growth Standards median11) among children aged 0-5 years; Wasting : Weight-for-height less than -2 standard deviations of the WHO Child Growth Standards median11) The percentage of live births that weigh less than 2500 g out of the total of live births during the same time period Percentage of infants ages 0–5 months who are exclusively breastfed (initiated on first hour; first day) Percentage of young people ages 15–24 (adults (15–49 years old)) reporting the use of a condom during sexual intercourse with a non-regular sexual partner in the last 12 months

DATA SOURCES (preferred alternatives) Administrative records, survey

TOPIC NCD

31.

Obesity among adults (over 15)4,8,9 (overweight)

Survey

NCD

32.

Proportion of population using improved water drinking source2,3,8 (urban–rural) Proportion of population using improved sanitation facilities2,3,8 (urban–rural) Children under 5 who are stunted3,6,8 (underweight2; overweight8; wasted8)

Survey

Envivonmental health

33.

Survey

Envivonmental health

34.

Survey

MNCH; NCD

35.

Low birth weight among newborn3,4,8,9 Exclusive breastfeeding3,4,6,8 (initiation on first hour; first day) Condom use at last high-risk sex (15–24)1,2,8 (adults (15–49 years old))

Survey, facility reports

MNCH

36.

Survey

MNCH

37.

Survey

HIV/STI

A country-led platform for information and accountability | 37

INDICATOR (additional dimension) Impact

DEFINITION

DATA SOURCES (preferred alternatives)

TOPIC

Health status 38. Life expectancy at birth4,8 (life expectancy at age 65; male–female) The average number of years that a newborn (person at age 65; male–female) could expect to live, if he or she were to pass through life exposed to the sexand age-specific death rates prevailing at the time of his or her birth, for a specific year, in a given country, territory, or geographic area The probability of a child born in a specific year or period dying before reaching the age of five, if subject to agespecific mortality rates of that period (perinatal mortality: number of stillbirths and deaths in the first week of life per 1000 live births; Neonatal mortality: number of deaths during the first 28 completed days of life per 1000 live births in a given year or other period; Infant mortality: probability of dying between birth and exactly one year of age expressed per 1000 live births) The maternal mortality ratio (MMR) is the annual number of female deaths from any cause related to or aggravated by pregnancy or its management (excluding accidental or incidental causes) during pregnancy and childbirth or within 42 days of termination of pregnancy, irrespective of the duration and site of the pregnancy, per 100 000 live births, for a specified year Probability that a 15 year old person will die before reaching his/her 60th birthday (per 1000 population) Death registration; survey; census All

39.

Child mortality rates (under 5)2,3,6,7,8 (perinatal4; neonatal; infant2,4)

Death registration; survey; census

MNCH

40.

Maternal mortality ratio2,3,6,7,8

Death registration; survey; census; facility reports

MNCH

41.

Mortality rates by major cause of death by age and sex2,7,8 (mortality between 30 and 70 due to CVD, cancer, chronic respiratory diseases and DM; top 20 major causes of death, IDC based) TB prevalence2,8 (TB notification rate; TB incidence10)

Death registration; facility reports, survey

All

42.

The number of cases of tuberculosis (all forms) in a population at a given point in time (the middle of the calendar year). It is sometimes referred to as "point prevalence". Estimates include cases of TB in people with HIV (TB notification rate: annual number of newly notified TB cases in a population; TB incidence: number of new and relapse case of TB (all forms) occurring in a given year)

Survey, facility reports

TB

38 | Monitoring, evaluation and review of national health strategies

INDICATOR (additional dimension) 43. HIV prevalence among young people (15–24)1,2,8 (HIV incidence among adults 15–49 years old)

DEFINITION The estimated number of young people aged 15–49 years with HIV infection, whether or not they have developed symptoms of AIDS, expressed as per cent of total population in that age group (HIV incidence: number of new HIV infections among the 15–49 years old during a certain time period) A disease that, by statutory/legal requirements, must be reported to the public health or other authority in the pertinent jurisdiction when the diagnosis is made The annual number of births to women aged 15–19 years per 1000 women in that age group. It is also referred to as the age-specific fertility rate for women aged 15–19

DATA SOURCES (preferred alternatives) Sentinel facilities; survey

TOPIC HIV

44.

Notifiable diseases incidence (IHR)8 (measles; neonatal tetanus; meningitis; cholera12) Adolescent fertility rate2,3,8

Disease surveillance systems

All

45.

Survey

MNCH; RH

Financial risk protection 46. Out of pocket expenditure as % of total health expenditure3,5,7,8 (catastrophic health expenses;% of households impoverished annually by out-of-pocket payments) The number of households in each region where direct out-of-pocket payments (catastrophic health expenses; % of households impoverished annually by out-of-pocket payments) to providers for health during the past 12 months was more than 40% of their household income net of subsistence, or 10% of their total income National health accounts; survey Protection; financing

Responsiveness 47. User satisfaction

References 1. UNAIDS. Monitoring the Declaration of the Commitment on HIV/AIDS: Guidelines on the Construction of Core Indicators, 2010 Reporting. Geneva, UNAIDS, 2009 (http://www.unaids.org/en/ dataanalysis/monitoringcountryprogress/). 2. UN. Indicators for monitoring the Millennium Development Goals: definitions, rationale, concepts and sources. New York, United Nations, 2003 (http://unstats.un.org/unsd/mdg/Host. aspx?Content=IAEG.htm). 3. WHO/UNICEF. Countdown to 2015 decade report (2000-2010): taking stock of maternal, newborn and child survival. Washington, WHO/UNICEF, 2010 (http://www.countdown2015mnch.org/). 4. ECHIM. European Health Indicators: Development and Initial Implementation. Helsinki, National Public Health Institute, 2008 (http://www.echim.org/).

A country-led platform for information and accountability | 39

5. WHO. Monitoring the Building Blocks of Health Systems: a handbook of indicators and their measurement strategies. Geneva, World Health Organization, 2010 (http://www.who.int/healthinfo/ systems/WHO_MBHSS_2010_full_web.pdf). 6. Keeping promises, measuring results. United Nations Commission on Accountability for Women’s and Children’s Health. Geneva, World Health Assembly, Sixty-fourth session, 2011 (http://www.who.int/ topics/millennium_development_goals/accountability_commission/en/). 7. The Global Fund. Monitoring and evaluation toolkit: HIV, Tuberculosis and Malaria and Health Systems strengthening. The Global Fund, 2009. 8. WHO. World Health Statistics. Geneva, World Health Organization, 2011 (http://www.who.int/whosis/ whostat/2011/en/index.html). 9. WHO. Global Status Report on noncommunicable diseases. Geneva, World Health Organization, 2010 (http://www.who.int/nmh/publications/ncd_report2010/en/). 10. WHO. Global tuberculosis control. Geneva, World Health Organization, 2010 (http://www.who.int/tb/ publications/global_report/2010/en/index.html). 11. WHO. WHO child growth standards and the identification of severe acute malnutrition in infants and children. Geneva, World Health Organization, 2009 (http://whqlibdoc.who.int/hq/2010/WHO_HSE_ IHR_2010.1_eng.pdf). 12. WHO/UNAIDS. WHO recommended Surveillance Standards. Second edition. Geneva, World Health Organization, 2000 (http://www.who.int/csr/resources/publications/surveillance/ WHO_CDS_CSR_ISR_99_2_EN/en/).

40 | Monitoring, evaluation and review of national health strategies

Annex 4. How to conduct a rapid assessment of country monitoring, evaluation and review practices and mechanisms Approach It is recommended that efforts to assess and strengthen monitoring, review and remedy/action should as much as possible be based on ongoing work, aiming to strengthen country and global processes and catalyse further action where needed. At country level, this includes the general work carried out within the context of IHP+ and related initiatives, focusing on strengthening a comprehensive, integrated and inclusive policy dialogue and robust national health strategies, as well as ongoing efforts made by governments committed to increasing accountability. A suggested approach involves a structured situational analysis/assessment of the strengths and weaknesses in the country health information system, including the identification of the major actions required to address gaps and needs. The assessment is based on an analysis of the country situation with respect to the key attributes and characteristics of the monitoring, evaluation and review platform for a national health strategy. The situational analysis/assessment can be used during the development of the M&E plan for the national health strategy, to help ensure that the process and content of the strategy are appropriate to the country needs and that the resulting strategy will have wide ownership and commitment. It can also be used when the strategy is near completion, to review the strategy as a basis for decisions on how to support and fund the strategy. The situational analysis/assessment can be used to assess and monitor progress towards greater accountability in the health sector as a whole, as well as in specific programmes, such as women’s and children’s health, HIV, TB, malaria etc. Next steps or “suggested action” are part of the situational analysis/assessment. The purpose of the next steps are to take action to improve the weaknesses/gaps identified in the M&E component of the national health strategy. A natural progression from the listing of next steps is developing a plan, a “Roadmap”, to implement these remedial actions to improve M&E. A key step towards enhancing and monitoring accountability at country level is the development of a country accountability framework or “Roadmap”. Such a framework should be based on the structured assessment/situation analyses of the country’s monitoring, evaluation and review practices and mechanisms. The process for developing a “Roadmap” is provided in point 4. hereafter.

1. Preparation for conducting the situation analysis 1.1 Planning the assessment The initial steps in planning a situation assessment are typically discussed and agreed upon in a countrylevel sector coordination group or sub-sector partner forum. In many cases, the initial assessment is planned as a first step in developing the monitoring and evaluation plan for the national health sector strategy as part of the IHP+ JANS process. Other entry points have included requests from governments for support in the assessment of country health information systems, in the development of country compacts, analytical reviews and capacity building.

1.2 Engaging political leaders Engagement of decision makers and political leaders is of critical important if the assessment is to lead to significant improvements in accountability. This starts with political leaders making specific commitments

A country-led platform for information and accountability | 41

towards health-related goals, such as spending on health, training of health workers, but also coverage of interventions or bringing down disease or mortality. Some make specific commitments towards reducing equity gaps in health indicators. Sometimes, special events are an important way to ensure greater commitment to health of government, members of parliament and other political or societal leaders. Linking the country accountability framework with such events is an important way to enhance its potential for success. Civil society can play an important role in following up.

2. Situational analysis/assessment tools The situational analysis/assessment is based on an analysis of the country situation with respect to the key attributes and characteristics of the monitoring, evaluation and review platform for a national health strategy, as detailed in previous sections. There are two component parts of the situational analysis: 1) the checklist for assessing the key elements of a country’s M&E (Table 1), and 2) the accountability assessment tool (Table 2). These two documents should be used together when conducting the situational analysis and are described below.

2.1 The M&E checklist Table 1 below operationalises the information presented in each of the attribute chapter into a checklist form. The checklist provides a list of “what to look for” when evaluating the key attributes. The list of items includes critical elements required for a strong country M&E. Included with the checklist are also a list of items that can signify weakness or gap in the country M&E. This checklist is a companion to the accountability assessment tool and should be used together.

Table 1. Checklist for assessing key elements of a country M&E and review platform for accountability I. The national health strategy as the basis for information and accountability

1. The national health strategy specifies a sound monitoring, evaluation and review component Characteristics 1.1 Monitoring, evaluation and review addresses the goals and objectives of the national health strategy and is based on a sound situation analysis. 1.2 Disease- and programme-specific monitoring, evaluation and review are aligned with that of the national health strategy. 1.3 The monitoring, evaluation and review plan is costed and funded with full partner alignment and support. 1.4 Monitoring, evaluation and review is regularly assessed.

What to look for • The NHS includes a comprehensive and robust M&E component or plan that refers to the goals, objectives of the NHS. • A situation analysis section of the NHS summarizing the country’s health determinants and disease burden based on the most recent data available. • There is a clear description of how the M&E plan of the NHS links to other national M&E plans (e.g. MNCH, HIV; TB; malaria, EPI, others). • Description of a transparent, participatory process involving the key stakeholders. • Partners agree to use and contribute to one M/E plan and system.

Warning signs/gaps • Little or no reference is made to the goals and objectives of the NHS. • Information on the key health priorities or disease burden data is unavailable or more than five years old. • Multiple M&E plans exist with little or no evidence of evidence of efforts made to align them. • Lack of clarity about the process involved in development of the plan and there is no specification of costs. • Development partners continue to invest in separate parallel M&E and reporting systems.

42 | Monitoring, evaluation and review of national health strategies

II. Institutional capacity

2. Roles, responsibilities and coordination mechanisms for monitoring, evaluation and review are clearly defined Characteristics 2.1 There is an effective countryled coordination mechanism for monitoring, evaluation and review. 2.2 Key institutions and stakeholders have clear roles and responsibilities.

What to look for • Description of the institutional framework and governance mechanisms for M&E and review. • Terms of reference/clear roles and responsibilities specified of multiple stakeholders.

Warning signs/gaps • No formal mechanisms in place to ensure the coordination and participation of all the major stakeholders. • No participation of key national stakeholders from outside the Ministry of Health.

3. Capacity strengthening in monitoring, evaluation and review is addressed Characteristics 3.1 Capacity strengthening requirements are identified and addressed.

What to look for • Analysis of human resource capacity and constraints throughout the health information system (from data collection, management, to analysis and dissemination & report writing). • Roles and responsibilities in data collection, analysis and dissemination are clearly defined, involving country partners, national institutions.

Warning signs/gaps • Little or no effort has been made to analyse capacity needs. • No detailed costing of human resource improvements or no budget for addressing resource constraints. • No supervision system is in place to monitor the capacity of staff and quality of data.

III. Monitoring and evaluation

4. There is a comprehensive framework that guides the monitoring, evaluation and review work, including core indicators and targets Characteristics 4.1 There is a balanced and parsimonious set of core indicators with well-defined baselines and targets. 4.2 Disease- and programme-specific indicators are aligned.

What to look for • There is an M&E framework agreed with all the partners that includes a core set of indicators that reflect the main elements of the national strategy. • The strategy sets out realistic targets, interim annual milestones or targets and uses benchmarks to assess progress. • The M&E plan provides details on source of information and the method of data collection for each indicator. • There is accurate and up-to-date data on the core set of indicators with disagregation to look at equity, including main stratifiers (gender, socio-economic position, subnational data). • Global partners have streamlined reporting around core indicators.

Warning signs/gaps • The indicators are poorly defined and do not align with the overall strategy objectives and targets. • There are no baseline s and targets. • Too many indicators in any specific area to monitor. • There is no reference or links to disease specific plans. • Development partners continue on collection of separate data for own indicators.

• There is an alignment with the core indicators with those of disease specific M&E plans. A country-led platform for information and accountability | 43

5. The monitoring, evaluation and review component specifies data sources, identifies and addresses data gaps, and defines responsibilities for data collection and information flow Characteristics 5.1 Data sources are specified in a comprehensive and integrated manner. 5.2 Critical data gaps are identified and addressed. 5.3 Responsibilities for data collection and management are specified.

What to look for • Monitoring draws on wellfunctioning health information systems, including facility routine systems, facility assessments, surveys, administrative data sources. • There is an analysis of information gaps, including problems with data quality, incomplete reporting and a plan to how to address. • There is a plan that describes coordinated plan for population based surveys as well as facility assessments including service readiness and quality assessments: -- Periodic health facility record reviews and service delivery assessments are implemented to verify data quality. -- There is a plan to conduct at least 2 large-scale population based surveys over next 5 years. -- There is a plan and strong country commitment to strengthening Civil registration and vital statistics. -- A systematic assessment of the current status of the system to monitor death and causes of death has been carried out using a standardized tool. • There is a clear description of the roles and responsibilities for data collection, management at each level of the health system.

Warning signs/gaps • The M&E plan gives little or no detail on the data source for each indicator. • There is no analysis of data gaps. • There is no survey plan.

6. Data analysis and synthesis work is specified, and data quality issues are anticipated and addressed Characteristics 6.1 Data analysis and synthesis work is specified. 6.2 There are regular assessments of progress and performance, including systematic analyses of contextual and qualitative information. 6.3 Specific processes for data quality assessment and adjustment are in place and are transparent.

What to look for • There is a clearly thought out plan for data analysis and synthesis, with clear roles and responsibilities. • There is a regular (annual) report of progress and performance that covers progress against the objectives and targets, equity and efficiency. • Regular data for sub-national levels are available with a mechanism for assessing data quality, through independent verification such as including a facility record review and service delivery assessment.

Warning signs/gaps • Plans are not clear on who will analyse the data and when. • There is no plan or description of mechanism for data quality assessment and adjustment. • Independent verification of data quality has not been carried out in the past one/year. • No mechanism for stakeholders to review and discuss data quality.

• There is use of transparent analytical methods and adjustment plan . 44 | Monitoring, evaluation and review of national health strategies

7. Data dissemination and communication are effective and regular Characteristics 7.1 Analytical outputs as the basis for national and global reporting are defined and produced. 7.2 Appropriate decision-support tools and approaches are used. 7.3 Data, methods and analyses are publicly available.

What to look for • There is a plan for the production of annual progress and performance reports and statistical publications that will meet needs at all levels (global, national, district). • There is effective data sharing mechanism including public access to data and reports (for accountability), via for example a national data repository /observatory. • There is a description of feedback mechanisms at all levels.

Warning signs/gaps • No regular publication of health data or programme results available to stakeholders and the public. • Unclear from M&E plan what reporting will be produced. • There is no feedback mechanisms.

8. Prospective evaluation is planned and implemented Characteristics 8.1 Prospective evaluation is planned and implemented in a forwardlooking manner and linked to monitoring, evaluation and review of national health strategies.

What to look for • Plans for evaluation incorporated in the national health strategy. • The life of the evaluation is for the length of the national strategy. • Clear linkages are made to the review cycle (including annual health sector review). • There are clear links of the evaluation of key interventions in specific program areas to the overall evaluation of the national health strategy. • To meet learning needs, evaluation should provide information as close to real-time as possible. • Continuous availability of disaggregated data (by administrative units) is available for key national health strategy indicators for the duration of the national health strategy. • The evaluation focuses not only on quantitative measures of success, but also addresses context, complexity and positive and negative unintended consequences. • Independent individuals/institutions responsible for evaluation clearly identified • Evaluators have access to data.

Warning signs/gaps • Plans for evaluation are not mentioned in the national health strategy. • There is no relationship between the evaluation in the national health strategy and the evaluation in specific program areas. • Evaluators are not independent. • No data availability for key indicators. • Barriers of accessibility to data for evaluators.

A country-led platform for information and accountability | 45

IV. Reviews

9. There is a system of joint periodic progress and performance reviews Characteristics 9.1 A regular and transparent system of reviews with broad involvement of key stakeholders is in place. 9.2 There are systematic linkages between health sector reviews, disease- and programme-specific reviews, and global reporting.

What to look for • National strategy describes how performance will be monitored over time, including specifying the stakeholders to be involved. • Sub-national performance reviews are also described where appropriate. • Comprehensiveness of review mechanisms (central , sub-national) and in terms of health sector vs. specific programmes.

Warning signs/gaps • There is no indication that regular performance reviews should take place. • Strategy suggests that the review is only an internal function and does not include all relevant stakeholders. • Programme reviews are held separately and with no reference or linkages to health sector review.

10. There are processes by which related corrective measures can be taken and translated into action Characteristics 10.1 Results from reviews are incorporated into decision-making, including resource allocation and financial disbursement. 10.2 Multi-stakeholder mechanisms are specified to provide routine feedback to sub-national stakeholders.

What to look for • Description of mechanism in place to feed programmatic and financial information and performance review results into decision-making at senior management level. • Mechanisms used by government and funding partners to make resource allocation decisions based on reports of performance. • Feedback mechanism are described with roles and responsibilities for monitoring performance and providing feedback assigned.

Warning signs/gaps • No formal mechanisms are in place to link M&E results and findings into decision-making process. • Mechanisms are in place, but little or no action is taken. • No feedback loop is described. • There is no link between measures proposed in performance reviews and allocation of financial and other necessary resources.

46 | Monitoring, evaluation and review of national health strategies

2.2 The accountability assessment tool The accountability assessment tool allows for a systematic review of attributes and characteristics of the national health sector in general, as well as for a particular programme area such as reproductive, maternal, newborn and child health. The assessment tool is complementary to the M&E checklist, which provides a basis for the accountability assessment tool. A template for the assessment tool is shown in Table 2 below. While the checklist only provides a list of items to look for, the assessment tool allows countries to qualitatively evaluate the attribute and the respective characteristics for strengths and weaknesses. For each attribute, the assessment profile includes descriptive and qualitative feedback and recommendations on each of the following broad areas: • The strengths and weaknesses of current country practices in monitoring and evaluation in relation to each Attribute, including acknowledgement of any plans in place to move towards achievement of each characteristic where weaknesses exist. • The identification of critical gaps and priority areas to be addressed in order to identify how to improve the strategy and how to support its implementation • General suggestions/next steps for how country stakeholders can further improve /enhance accountability. The user would use the assessment tool in conjunction with the checklist. Each of the item listed in the checklist can be used as a springboard for the analyses of the strengths and weaknesses of each of the characteristics. While the checklist is a guide for the assessment tool, it should not constrain the user from exploring other issues that they feel are relevant to the review of the monitoring and evaluation component of their national health strategy. After a thorough evaluation of each attribute and their characteristics, remedial actions should be suggested on how to address gaps and weaknesses and where further work is needed. These actions will form the basis of a country accountability framework or “Roadmap” for enhancing accountability. A template for summarizing the results of the assessment is shown below.

Table 2. Accountability Assessment tool Key elements and characteristics National Health Sector Plan Suggested actions I. The national health strategy as the basis for information and accountability Characteristic 1.1

Strengths • • • Gaps/weaknesses • • • Strengths • • • Gaps/weaknesses • • •

Characteristic 1.2

A country-led platform for information and accountability | 47

3. Conducting the situational analysis/assessment The situational analysis/assessment can be (and should be) completed using different available instruments. Given below are some of the methods used to do the situational analysis/assessment. Convening a national stakeholder workshop: For many countries a multi-partner scoping mission is considered as a useful entry point for the situation analysis/assessment and development of a country roadmap or framework for strengthening M&E and review. Usually, an initial workshop will be organized with key stakeholders to conduct a participatory analysis of the current situation, using the checklist and the accountability assessment tool . The stakeholders typically include representatives from Ministry of health (planning , monitoring and evaluation, specific health and disease programmes), Ministry of financing, H4 agencies, key bilaterals, academic and public health institutions and civil society groups working on monitoring, evaluation & accountability. Interview with key informants: While it is very useful and convenient to do a situational analysis/ assessment of the M&E component of the national health strategy in a stakeholder workshop, an analysis should be done even if it is not possible to hold a stakeholder workshop. Inputs from key informants is critical to a situation analysis/assessment, with or without a stakeholder workshop. Some examples of key informants are key people in the monitoring and evaluation, HMIS or policy and performance departments at the ministry of health, people involved in the development of the national health strategy (and the M&E component), other stakeholders involved in the surveys and other essential data collection that are not in the ministry of health (i.e. key bureau of statistics personnel and key personnel in institutions that provide independent support to program evaluation). Reviewing relevant documents: In addition to meetings with key informants and stakeholders, a review of relevant documents is a core input into the situation analysis . Key documents required include: • Recent needs assessments or situation analysis in terms of country health information systems, monitoring, evaluation and review practices; • Health sector planning and strategy documents; • The M&E plan for the national health strategy, and operational plans for information systems; • The M&E plans of national strategies on HIV, TB, malaria, reproductive, maternal newborn and child and adolescent health, etc; • Health sector performance reports, joint annual review reports, mid term reviews, consultant reports on progress; • Annual statistical reports, information bulletins; • Declarations of commitments to international agreements and targets such as the MDGS, the Global Strategy on Women’s and Children’s Health etc.

4. Developing the “Roadmap” for the review of M&E of the national health strategy There is no blueprint for developing a roadmap for the review of M&E of the national health strategy, as each one needs to be specific to the country specific context1. Typically however the country roadmaps tend to be organized around the core elements of the M&E platform: • Strengthening policy environment and institutional capacity • Strengthening monitoring of results and tracking of resources 1 WHO, Accountability for Women’s and Children’s Health in Countries: Current practices and challenges in Ghana, Rwanda and Tanzania. A case study prepared for the Commission on Information and Accountability for Women’s and Children’s Health. 20 April 2011, Geneva. http://www.who.int/healthinfo/ country_monitoring_evaluation/situation/en/index.html

48 | Monitoring, evaluation and review of national health strategies

• Strengthening progress and performance reviews, e.g. annual health sector reviews • Strengthening mechanisms for taking remedial action. Under each core element, priority activities required to address the identified gaps and weaknesses are described and costed . The roadmap should provide a description of each activity for a 5 year period (or for the period of the National Strategy). Roles and responsibilities of country and partner stakeholders should be agreed upon and assigned.

A country-led platform for information and accountability | 49

ISBN 978 92 4 150227 6

50 | Monitoring, evaluation and review of national health strategies