National nutrition information systems: technical note: assessing a national nutrition information system

ANNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM Assessing a National Nutrition Information System TECHNICAL NOTE Acknowledgements The Technical Note on Assessing a National Nutrition Information System is a component of the full guide on National Nutrition Information Systems (NNIS), a product of the WHO-UNICEF Technical Expert Advisory group on nutrition Monitoring (TEAM), which is supported by the Bill & Melinda Gates Foundation. The TEAM Working Group on Nutrition Information Systems was responsible for the development of this Technical Note. TEAM acknowledges the contributions of David Hales (Consultant), Chika Hayashi (UNICEF Headquarters), Rebecca Heidkamp (TEAM Working Group Lead), Louise Mwirigi (UNICEF Headquarters) and Kuntal Saha (WHO Headquarters), who conceptualized and led the production of this document, including its writing and revision. The Working Group acknowledges the contributions of Julia D’Aloisio (Editor) and Nona Reuter (Designer, UNICEF). The Working Group is also grateful to nutrition colleagues who reviewed the draft and shared specific experiences and insights. Suggested Citation: National Nutrition Information Systems; Technical Note: Assessing a National Nutrition Information System, Geneva: World Health Organization and the United Nations Children’s Fund (UNICEF), 2022. - - - - - The NNIS Fundamentals Series includes five modules: Module 1. What is a national nutrition information system? Module 2. How does a national nutrition information system support a country’s nutrition programmes? Module 3. What is needed to build a useful national nutrition information system? Module 4. What are the main attributes of a national nutrition information system? Module 5. What are the main types of data used in a national nutrition information system? PDF versions of the five modules and different Technical Notes on specific topics can be downloaded from the following website: https://data.unicef.org/resources/nutrition-nnis-guides/ A four-module e-course on national nutrition information systems is available on the same website. © United Nations Children’s Fund (UNICEF) and World Health Organization (WHO) ISBN: (UNICEF) 978-92-806-5374-8 ISBN: (WHO) 978-92-4-007189-6 (electronic version) ISBN: (WHO) 978-92-4-007190-2 (print version) November 2022 Permission is required to reproduce any part of this publication. Permissions will be freely granted to educational or non-profit organizations. Please contact: UNICEF Data Analytics and Innovation Division of Data, Analytics, Planning and Monitoring 3 United Nations Plaza New York, NY 10017, USA 1NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM INTRODUCTION Full operational assessments can help determine the extent to which the overall NNIS is meeting the needs of its users and make practical recommendations on activities that will strengthen its ability to meet those needs. Targeted operational assessments can focus on specific issues that should be addressed to maintain or improve the integrity of the system. Full assessments are likely to be most beneficial if they are done on a multi- year cycle; for example, every two years. Targeted assessments can and should be done as needed; for example, if there is a recurrent problem that has been difficult to solve or an emerging opportunity that should be capitalized on. Operational assessments are developmental activities designed to support ongoing quality improvement of the NNIS; they are not formal evaluations. Consequently, it is important to implement these assessments in an inclusive and transparent fashion to ensure open and honest engagement with the relevant stakeholders throughout the process. It is equally important to focus these operational assessments on the NNIS, not on issues that are outside its mandate or remit. This technical note provides guidance on a series of key questions related to operational assessments of an NNIS: 1. Who is responsible for the management and oversight of an assessment? 2. What is the purpose and objective of the assessment? 3. What is the scope of the assessment? 4. What is the best approach to the assessment? 5. What issues should be prioritized in an assessment? 6. Who should be on the assessment team? 7. What is a reasonable timeframe for completing the assessment? 8. What is a reasonable budget for the assessment? 9. How often should operational assessments be done? 10. What are the next steps after an assessment? The technical note also includes two checklists related to an assessment of an existing NNIS. Checklist #1 includes a set of possible investigative questions that could be used in an assessment (see page 7). Checklist #2 includes a series of key actions that could be followed to plan and conduct an assessment (see page 11). WHO IS RESPONSIBLE FOR THE MANAGEMENT AND OVERSIGHT OF AN ASSESSMENT? The established governing body for the NNIS should have general responsibility for any assessments. While there are different ways to structure the governing body, Module 3 of the NNIS Fundamentals TECHNICAL NOTE Assessing a national nutrition information system Periodic operational assessments can be a valuable tool for understanding the functionality, utility and quality of a national nutrition information system (NNIS). These assessments can yield critical insights on how well the system is working and how it can be improved. They also have the flexibility to look at a full NNIS or specific aspects of it, depending on the purpose, objective and scope of the assessment. 2NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM series noted that the multisectoral nature of nutrition lends itself to having a representative steering committee as this body. While the full steering committee should be involved in deciding to conduct an assessment and determining its purpose and scope, there is no need for the full steering committee to be directly involved in the management and oversight of an assessment. A small working group — or even one or two key people — is a more efficient way to manage an assessment. The working group could include members of the steering committee or other qualified individuals selected by the steering committee. The working group will be directly involved in critical decisions related to an assessment, including drafting the terms of reference and selecting the assessment team. It will also be the primary point of contact for the team while it is conducting the assessment and when it is presenting its findings. However, the responsibility of ensuring the overall quality of the assessment, including its inclusiveness and its transparency, should be shared by the steering committee and the working group. Questions to consider… • Does a functioning governing body exist for the NNIS? If not, what individuals or organization(s) are responsible for the operation of the NNIS and would commission an assessment? • Are there qualified people, including individuals on the governing body, who could form a small working group to provide the necessary management and oversight to an assessment team? WHAT IS THE PURPOSE AND OBJECTIVE OF THE ASSESSMENT? The purpose of an assessment refers to why it is being done; the objective refers to what the assessment is trying to achieve. For every assessment, whether full or targeted, it is essential to have a clear idea of both the purpose and the objective. For example, the purpose could be to determine the extent to which the NNIS is meeting the needs of its users; the corresponding objective would be to identify practical actions that will improve its ability to meet those needs. A more targeted purpose would be to assess the value for money of the funds allocated to the NNIS, including whether the system is used in critical decision-making; the objective would be to determine what can be done to ensure and/or improve value for money. An even more targeted purpose would be to determine if the data included in the NNIS on a specific issue are providing an accurate and actionable picture of the situation, with a corresponding objective of confirming the value of the data and possibly identifying other types or sources of data. When thinking about the purpose and objective, it can be useful to develop a set of framing questions highlighting the underlying issues. For example: • How useful and relevant are the data in the NNIS on the country’s priority nutrition issues? • What are the gaps in the data and/or the analysis? • How available, accessible and understandable are the data to users? • How are stakeholders using the NNIS (e.g., in planning, decision-making, course correction, monitoring)? • How efficiently and effectively is the NNIS operating? Questions to consider… • If a full assessment is planned, what are the main issues or concerns about the NNIS that should be captured in the purpose or objective? • If a targeted assessment is planned, what are the specific issues or concerns that should be captured in the purpose or objective? • How do the framing questions link to the more focused ‘investigative questions’ that will guide the work of the assessment team? (See ‘What issues should be prioritized in an assessment?’) WHAT IS THE SCOPE OF THE ASSESSMENT? In general, the scope of an operational assessment will be either full or targeted. A full assessment encompasses all of the main components of the information system and their interactions with each other. A targeted assessment focuses on specific components or issues within a component, particularly those with concerns or known problems. Full assessments can be an effective way of understanding the overall functioning of the NNIS and its value to people and organizations working on nutrition in the country. As mentioned above, a full assessment must look at all of the components in the information system and the connections between and across these components. These components are reflected in the nutrition data value chain (see Figure 1). The value chain reflects the core function of an NNIS: to provide data to stakeholders on the status of nutrition priorities and the performance of 3NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM nutrition-related programmes and projects linked to those priorities at national and subnational levels. The value chain is a series of linked processes that provide accurate and timely information to support informed, evidence-based decision-making that will help countries improve nutrition outcomes. A targeted assessment looks at one or more individual components in the value chain or the connection between them. For example, a targeted assessment could look solely at the ‘generation and collection of data’ or the relationship between ‘analysis of the data’ and ‘dissemination of the data’ or the quality and reliability of individual data sources. As mentioned above, a targeted assessment is more likely to be done because of concerns or known problems with certain parts of the information system. Depending on the NNIS or the country context, it is also possible to do a targeted assessment of a specific issue or sub-component of the system. For example, a targeted assessment could look at the quality, reliability and availability of data from different sources or it could consider potential variations in the quality, reliability and availability of data on different nutrition issues (i.e., the reasons that data are better and more accessible for some issues than others). Questions to consider… • Are there existing concerns or known problems with the NNIS overall or related to specific components that should be assessed? • When was the last full assessment of the NNIS conducted? If it has been more than two years, would there be a benefit in doing a full assessment to understand the current strengths and weaknesses of the overall system? • Are there any specific components or sub- components of the NNIS that have been underperforming in the past 12–18 months? If so, have targeted assessments of these components been conducted? WHAT IS THE BEST APPROACH TO THE ASSESSMENT? Operational assessments of an NNIS are done to understand how well the system is working and how it can be improved. The operational focus places a premium on timely assessments that produce actionable findings and recommendations. This focus makes a strong case for using a rapid assessment approach for both full and targeted assessments of an NNIS. A rapid assessment combines a mixed methods approach with rapid assessment techniques. A mixed methods approach uses different kinds of data (e.g., quantitative and qualitative) collected in various ways (e.g., key informant interviews, focus groups, observations, document review, data review, surveys and mapping), which are then triangulated to make conclusions (e.g., findings and recommendations). The relevant and available mixed methods approaches are integrated with rapid assessment techniques, including an accelerated timetable, active engagement with key stakeholders (e.g., data sources, data analysts, data users), high levels of collaboration among assessment team members and an iterative cycle of data collection Figure 1. Main components of an NNIS: Nutrition data value chain See the NNIS Technical Note on nutrition data value chains for additional information: <https://data.unicef.org/resources/nutrition-nnis-guides/> Identification and prioritization of nutrition issues and indicators Generation and collection of data Aggregation and processing of the data Analysis of the data Dissemination of the data Use of the data 4NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM and analysis (e.g., initial analysis of data determines if/when additional data should be collected and more analysis should be done). The overall approach encourages the assessment team to make thoughtful decisions about priorities in each area of their work to make the most effective use of their time. The rapid approach is appropriate for NNIS assessments for several reasons, including the above- mentioned value of having timely and actionable findings and recommendations; the diversity of stakeholders who will have useful insights for the assessment; the range of data in the NNIS and the many ways it is handled (e.g., collection, aggregation, processing, analysis); and the various issues involved in building and maintaining the system (e.g., technical, human resources, financial). In addition, the flexibility of the approach means it can be organized and implemented more quickly than other types of assessments. When planning an assessment using a rapid, mixed methods approach, it is important to carefully consider the specific methods that will yield the most useful insights for the assessment; for example, desk reviews of relevant documents, formal and informal interviews, focus group sessions, site visits, observation sessions, surveys and data quality assessments. The discussion about the specific methods should carefully consider the scope and objective of the assessment, the sources of information and the framing questions. The discussion should also identify and address constraints that may limit the ability to conduct a proper assessment. Participants in this discussion should include the working group (see above) and the assessment team members. Questions to consider… • Is there an up-to-date inventory of data, documents and stakeholders related to the NNIS that can be used to plan a rapid assessment? • What are the potential challenges of conducting a rapid assessment? How could they be addressed? WHAT ISSUES SHOULD BE PRIORITIZED IN AN ASSESSMENT? There are many different issues that can be explored during an operational assessment, ranging from the match between national nutrition priorities and the indicators and data included in the NNIS, to the approaches to data analysis, to the integration of NNIS data and analysis into decision-making. Consequently, when planning an assessment, it is essential to first create relevant framing questions (see “What is the purpose and objective of the assessment?”) and then complement these questions with more specific investigative questions to focus the work of the assessment team, specifically around known concerns or problems with an NNIS. Given the core function of an NNIS — i.e., to provide data to monitor the status of nutrition priorities and the performance of nutrition-related programmes and projects linked to those priorities — it is logical for operational assessments to prioritize issues related to the data. These types of issues include: • The quality of data generated by the provider (e.g., are they accurate, current, representative, complete?) • The flow of data (e.g., is the flow of data smooth, timely and secure within the provider’s system and from the provider to the NNIS?) • Data aggregation and disaggregation in the NNIS (e.g., are they accurate, representative and useful?) • Data analysis (e.g., is it reasonable, logical, credible and useful?) There is also a strong argument for including the user perspective in the investigative questions (i.e., the individuals and organizations that will be using the NNIS to strengthen and shape their own nutrition- related activities). The value of an NNIS is to provide these users with better data for decision-making; this includes users at all levels, ranging from implementers in facilities and communities to policymakers at national level. Questions to consider… • Are mechanisms in place to capture areas of concern or specific problems with an NNIS identified by different stakeholders, including users of the information system? • How can these concerns or problems be addressed in the assessment? • Is there a map of how data flow through the NNIS (i.e., from where they are generated or captured through to how they are presented to users)? Does the map identify who is responsible for the data at each point to ensure that relevant stakeholders are engaged in the assessment? 5NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM WHO SHOULD BE ON THE ASSESSMENT TEAM? The scope of the assessment (i.e., full or targeted) will influence who participates as well as how many people are on the team. For example, a full assessment may benefit from having a larger team, whereas targeted assessments typically need a smaller team. Every participant should be able to make substantive contributions to the assessment. It is important that the team have a mix of internal and external members for any assessment. A mixed team is an effective way to combine strong ‘insider’ knowledge of the NNIS with an impartial outside perspective. Internal team members should have direct knowledge of the NNIS, including specific knowledge related to any targeted assessment. External members should also have relevant knowledge about one or more aspects of an NNIS; for example, experience as an end user of the system or technical knowledge about the operation of the system (e.g., data analysis tools). Each member of an assessment team should be as objective as possible, focusing on the effectiveness of the information system and how it can be improved. It is important to ensure individuals bring relevant and applicable knowledge and skills to the assessment team. It is equally important to build a balanced team of qualified members who collectively have the ability and the mindset to conduct a rapid, mixed methods assessment that prioritizes actionable recommendations. Assessment teams should be kept to a manageable size. A reasonable guideline is a maximum of five people for a full assessment and a maximum of three people for a targeted assessment. In principle, there should be more external members than internal members on an assessment to help ensure broad perspectives. Questions to consider… • Is there a roster of potential internal and external team members with the necessary qualifications? • Who is responsible for selecting participants for an assessment (e.g., the steering committee or working group)? • What steps are in place to ensure participants can freely express their views in the assessment? WHAT IS A REASONABLE TIMEFRAME FOR COMPLETING THE ASSESSMENT? A primary benefit of a rapid approach to assessments is the ability to generate constructive findings and recommendations in a matter of weeks rather than months. This benefit is particularly relevant for operational assessments, where there is a high value attached to an efficient process that can generate actionable results on an accelerated timetable. The timeframe for an NNIS assessment will vary based on different factors, including the purpose, scope and objective of the assessment, as well as the number of inputs (e.g., documents, interviews, surveys), the availability and skills of the assessment team and the urgency of the situation. It is important to be realistic about the amount of time needed for an assessment, being careful not to underestimate or overestimate the timeframe. While the point of rapid assessments is to generate useful findings and recommendations on an accelerated timetable, the quality of those findings and recommendations should not be compromised in the process. Questions to consider… • What are the benefits of implementing an assessment on an accelerated timetable? • What are the barriers? How can those barriers be addressed? WHAT IS A REASONABLE BUDGET FOR THE ASSESSMENT? The flexibility and adaptability of the rapid, mixed methods approach can make assessments affordable and cost-effective. Although the flexibility and adaptability of the approach enables planners to adjust the parameters of the assessment to make the best use of the available budget, it is important to provide sufficient funds for the work. Under-resourced assessments can easily undermine the value of the exercise. As is the case with the timeframe for an assessment, the budget will vary based on different factors (e.g., scope, objective, number and range of inputs, size of the team). Common cost categories in an assessment budget are staffing, travel, equipment and/or services, communications and materials and supplies. • Staffing – The main expense is likely to be compensation for external members of the assessment team. This expense will vary based on the number of external members, their agreed 6NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM pay rate and the amount of time allocated for their involvement. • Travel – Expenses can include airfares, local transport, per diems and off-site meeting venues. • Equipment and/or services – Equipment needs (e.g., computers, software, printers) are likely to be minimal for an NNIS assessment. Services (e.g., development of an online survey, translation services, meeting organization) will vary based on the purpose, objective and scope of the assessment. • Communications – Expenses can include mobile phone, Internet, wifi and video conferencing. • Materials and supplies – Expenses can include printouts, photocopying, food/beverages for meetings. Questions to consider… • Are funds for assessments included in the overall budget for operating an NNIS? If not, are there potential sources of funds for this activity? • Can funds be used to compensate members of the assessment team, particularly external people? • What can be done to ensure the quality and value of the assessment are balanced with cost effectiveness? HOW OFTEN SHOULD OPERATIONAL ASSESSMENTS BE DONE? In general, it can be valuable to conduct a full operational assessment of an NNIS at least once every two years. In settings where the NNIS is continually evolving, it can be beneficial to do a full assessment every year. Regular assessments provide stakeholders with a consistent way of monitoring the evolution of the system, including the effectiveness of steps taken to strengthen it. Targeted assessments can be done when there is a clear need; for example, when a specific component of the NNIS is not performing as designed. An important benefit of the rapid, mixed methods approach to assessment of an NNIS is the ability to organize and complete it quickly and cost-effectively, making it possible to conduct such assessments on a more regular basis. This is in direct contrast to formal evaluations, which have a more structured timeline (e.g., baseline, mid-term and final evaluations) and a more expensive cost structure. WHAT ARE THE NEXT STEPS AFTER AN ASSESSMENT TEAM HAS COMPLETED ITS WORK? The conclusion of an assessment is essentially the starting point for the vital process of actually addressing the findings and recommendations. Rather than being conducted solely to understand what is and is not happening with the NNIS, operational assessments should provide the impetus for making ongoing improvements to the system. After the assessment team has completed its work, key steps for the working group and/or steering committee include: (1) selecting a lead person or small team that is a stakeholder in the NNIS to manage the next steps; (2) identifying who is best positioned to address the findings and recommendations in the assessment and establishing a constructive dialogue with these stakeholders to fully engage them in the process; 3) establishing and implementing a process and timetable for addressing the findings and recommendations, including identifying and managing any financial implications; 4) reviewing how the findings and recommendations have been addressed, in collaboration with the stakeholders involved in addressing them; 5) monitoring how the response to the findings and recommendations has affected the performance of the NNIS, including by consulting widely with stakeholders to understand the outcomes of any changes to the information system. Questions to consider… • Is there a commitment to using the findings and recommendations from an assessment to strengthen the NNIS? • Are there champions who will drive the strengthening efforts? • Are mechanisms in place to support strengthening activities, including technical assistance, human resources and financial resources? 7NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM CHECKLIST #1 INVESTIGATIVE QUESTIONS FOR AN NNIS ASSESSMENT This checklist includes a set of investigative questions that could be used when conducting an NNIS assessment. The questions are organized by the different components in the NNIS data value chain. The various questions are designed to be broadly useful, with applications ranging from framing questions to guide the assessment to specific questions for key informants. IDENTIFICATION AND PRIORITIZATION OF NUTRITION ISSUES, INCLUDING CORRESPONDING MEASURES/INDICATORS  Are priority issues in the national nutrition policies, strategies and plans reflected in the design and structure of the NNIS? P If yes, how many of the priorities are part of the NNIS? Are they the most important priorities? If there are priorities not included in the NNIS, what are the reasons? P If no, why are priority issues not part of the NNIS? • Are credible measures/indicators identified for each of the priorities? P If yes, do they capture relevant data on key aspects of the activities, outcomes and/or targets associated with the nutrition priorities? Do the measures/indicators include useful disaggregations of the data (e.g., age, gender)? P If no, can appropriate measures/indicators be identified? Note: These measures/indicators may be included in national policies, strategies and plans or within a related monitoring and evaluation document.  Are representative data for these measures/ indicators available from existing sources in the country? P If no, is it feasible to generate/collect these data from existing and/or new sources?  Is there agreement on which measures/indicators should be included in the NNIS? P If no, what steps need to be taken to reach this agreement? Note: The focus should be on including a manageable number of priority measures/indicators in the NNIS that provide practical data on nutrition status and/or the performance of nutrition programmes and projects.  Are policies and procedures in place to periodically review the priority issues included in the NNIS?  Are policies and procedures in place to periodically review the quality and value of measures/ indicators used in the NNIS? Can other measures/ indicators be explored to collect additional and/or better data?  How can the process of prioritizing nutrition issues and selecting measures/indicators be improved? Why should it be improved? GENERATION AND COLLECTION OF DATA ON THE PRIORITY ISSUES  What are the existing sources of data for each of the measures/indicators? Note: There may be multiple sources of data for individual measures/indicators.  What types of data are available for each of the measures/indicators? P Routine data, survey data, surveillance data, other? P Quantitative, qualitative?  What is the timeframe or time cycle for each data set? P Monthly, quarterly, annually, biennially, three-year cycle, five-year cycle, other? Note: The timeframe or time cycle will vary based on the type of data. For example, routine data may be available on monthly basis; survey data may be available annually, biennially (or less frequently, such as the three- to five-year cycles for conducting larger household surveys such as Demographic and Health Survey and the Multiple Indicator Cluster Surveys). 8NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM  How reliable are the sources of data? P What principles and/or procedures are in place to ensure the data generation/collection mechanisms are functioning efficiently and effectively? P What principles and/or procedures are in place to ensure the quality of the data? P What principles and/or procedures are in place to ensure the timely availability of the data? Note: It may be useful to create a ‘map’ of the data sources to understand and assess the capacity to generate and/or collect data on the priority issues included in the NNIS.  Who is responsible for management/oversight of the relationships with data sources?  What is the budget for generating and collecting data? How is the budget allocated? Is it sufficient?  How can the process of generating and collecting NNIS data be improved? Why should it be improved? AGGREGATION AND PROCESSING OF THE DATA  Is the basic organizational and operational infrastructure in place to handle the aggregation and processing of the collected data? P If yes, are there any weaknesses or areas of concern that should be addressed? P If no, what infrastructure is not in place? How does its absence affect the aggregation and processing of the data? What steps should be taken to build the infrastructure? P Note: Organization and operational infrastructure can include an institutional home for the NNIS, a functioning governing body, clear roles and responsibilities attached to tasks, required human resources (e.g., qualified staff and consultants), necessary equipment and essential financial resources. Note: Key tasks involved in aggregation and processing include data entry, data cleaning, data classification, data coordination/curation/collation, data quality, data storage and data privacy and security.  Are the individuals working on the aggregation and processing of the data suited to the work? P Do the members of the data team (staff and consultants) have the requisite knowledge and skills? P Do they have the necessary support (e.g., equipment, training, management/oversight)? P Do their managers have the requisite knowledge and skills?  Are there well-defined procedures for conducting the aggregation and processing of the data? P If yes, do the procedures include quality controls? P If no, what guides the process of aggregating and processing the data?  Are there aspects of the aggregation and processing of data that are underperforming? For example, are there concerns about the time it takes to do the work, the integrity of the process or the accuracy and/or utility of the outputs?  Do the aggregated and processed data make sense? Do they present an accurate and representative picture of the situation? Are they useful inputs for the data analysis process?  How can the aggregation and processing of NNIS data be improved? Why should it be improved? ANALYSIS OF THE DATA  Is the basic organizational and operational infrastructure in place to handle the analysis of the data? P If yes, are there any weaknesses or areas of concern that should be addressed? P If no, what infrastructure is not in place? How does its absence affect the analysis of the data? What steps should be taken to build the infrastructure? Note: See also ‘Aggregation and processing of the data’.  Is there a data analysis plan with well-defined procedures for conducting the analysis? P If yes: X Does the data analysis plan align with the purpose and priorities of the NNIS? Does it address the questions about nutrition status and/ 9NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM or the performance of nutrition programmes and projects that are important to decision-makers? X Does it account for the different sources and types of data included in the analysis? X Does it include triangulation of data from different sources and of different types as part of the analysis? X Do the procedures include quality controls? X Is the analysis done as it is laid out in the plan? P If no: X What guides the analysis process? X Are there concerns about the quality of the analysis?  Are mechanisms and/or procedures in place to develop data visualizations from the completed analysis to help explain the findings and conclusions? Note: Visualizations can include charts, graphs, maps, dashboards and scorecards.  Are the individuals working on data analysis and visualizations suited to the work? P Do the members of the team (staff and consultants) have the requisite knowledge and skills? P Do they have the necessary support (e.g., equipment, training, management/oversight)? P Do their managers have the requisite knowledge and skills?  Are there aspects of the process that are underperforming? For example, are there concerns about the time it takes to do the analysis, the integrity of the process or the accuracy and/or utility of the outputs?  Do the analysed data make sense, including the visualizations? Do they present an accurate and representative picture of the situation? How can they be improved?  How can the analysis and visualization of NNIS be improved? Why should it be improved? DISSEMINATION OF THE DATA  Who is responsible for the dissemination of NNIS data? Are they qualified and equipped to do the work?  Is there a dissemination plan? P If yes: X What are the different products? (For example, can users access the ‘raw’ data after they have been aggregated and processed?) X What are the different dissemination routes (e.g., website, database, social media, publications, workshops)? X Who is the target audience for the NNIS data? X How is the plan implemented? P If no: X How are data made available to users?  What is the budget for dissemination? How is it allocated? Is it sufficient?  How aware are users of the NNIS data and visualizations that are available? Are data and data visualizations actually getting to users?  What procedures are in place to ensure data are readily available to stakeholders who can and should use them? Is there a ‘push’ capacity to proactively deliver the data to them?  Which dissemination products and routes are most popular and why?  How can the dissemination of NNIS data be improved? Why should it be improved? 10NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM USE OF THE DATA  Who has responsibility for monitoring if and how users are working with NNIS data? Are they qualified and equipped to do the work (e.g., have they done the work previously; have they been trained to do the work)?  Are procedures in place to monitor the use of these data (e.g., surveys, interviews)? P If yes: X How are the data being used? X Why are the data valuable to the users? X How are the data affecting decisions about nutrition priorities and/or programmes? P If no: X What procedures can and should be put in place?  How can the use of NNIS data be improved and expanded? Why should it be improved?  What other nutrition data would be valuable for users? 11NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM CHECKLIST #2 KEY ACTIONS WHEN CONDUCTING AN ASSESSMENT OF AN EXISTING NNIS This checklist includes a series of key actions that should be followed when conducting an assessment of an NNIS. The actions are listed in sequential order and include, where useful, supporting issues and/ or questions to consider as well as the individuals or groups primarily responsible for the action.  Appoint a small working group to manage and oversee the assessment P Responsible party: NNIS Steering Committee  Determine the purpose and objective of the assessment P Why is the assessment being done (purpose)? P What is the assessment trying to achieve (objective)? P Responsible party: NNIS Steering Committee  Determine the scope of the assessment P Is it a full or targeted assessment? If it is targeted, which components are being assessed? P Responsible party: working group in collaboration with the steering committee and key NNIS stakeholders  Determine the size of the assessment team P What is the breakdown by internal and external members? P Are there any specific roles and responsibilities that affect the size of the team? P Responsible party: working group (approved by the steering committee)  Develop terms of reference for the assessment team P Includes scope, objective, methodology, timeframe, team member responsibilities, deliverables, preliminary list of relevant documents and preliminary list of key informants. P Responsible party: working group (approved by the steering committee)  Recruit members for the assessment team P Responsible party: working group  Finalize the membership of the assessment team P Includes selection of a team leader P Provide clear roles and responsibilities to each team member P Responsible party: working group  Finalize the schedule for the assessment P Includes start date and timeline P Responsible party: working group and team leader  Provide team members with available background information P Includes documents, NNIS data, contact information for key informants P Responsible parties: working group and NNIS stakeholders  Plan the assessment activities P Determine the mix of activities (e.g., desk reviews of relevant documents, formal and informal interviews, focus group sessions, site visits, observation sessions, surveys and data quality assessments) P Identify and address constraints that may limit the effectiveness of the assessment P Set a timetable for the activities P Responsible party: assessment team  Conduct assessment activities P Review documents; review data; develop interview questions, conduct individual and group interviews; develop surveys, conduct surveys; make site visits, etc. P Hold team meetings/discussions as needed P Responsible party: assessment team  Hold midpoint meeting with the working group P Includes update on the assessment, preliminary presentation of any findings and discussion of next steps P Responsible party: assessment team 12NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM  Prepare and submit a draft report P Includes key findings and prioritized recommendations P Responsible party: assessment team  Provide feedback on draft report P Includes selecting a manageable number of qualified and impartial individuals who can provide meaningful feedback and ensuring the process is completed efficiently P Responsible party: working group  Prepare and submit the final report P Responsible party: assessment team  Hold a dissemination workshop for steering group members and NNIS stakeholders to discuss the findings and recommendations in the report and options for moving forward P Responsible parties: assessment team and working group NEXT STEPS (overseen by the steering group)  Select a lead person or small team to manage the process of addressing the findings and recommendations from the assessment  Identify NNIS stakeholders who are best positioned to address the findings and recommendations  Establish and implement a process and timetable for addressing the findings and recommendations, including any financial implications  Review how the findings and recommendations have been handled  Monitor how the response to the findings and recommendations has affected the performance of the NNIS 13NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM KEY TERMINOLOGY Data Facts and/or figures; pieces of quantitative or qualitative information Database An organized collection of data stored electronically for rapid search and retrieval Data provider An organization that produces data; may be referred to as a data generator; see also data source Data source Type of data and/or modality of data collection (e.g., routine data, survey data); can also be synonymous with data provider Data value chain A framework used to guide the transformation of raw data into a valuable resource to better understand situations and improve decision-making Disaggregated data Data that have been broken down into detailed subcategories (e.g., by age, gender) Indicator Indicators make collected data understandable and useful for monitoring performance, assessing achievement and determining accountability. They can be used to determine a proportion (e.g., prevalence) and are often designed to track inputs, outputs, outcomes and impact. National data Data that are common to or characteristic of a whole nation; see also subnational data Qualitative data Data collected using qualitative methods, such as interviews, focus groups, observation and key informant interviews; generally expressed in narrative form, pictures or objects (i.e., not numerically) Quantitative data Data that are measured on a numerical scale, can be analysed using statistical methods and can be displayed using tables, charts, histograms and graphs Routine data Data continuously collected as part of a regular activity/procedure Sentinel site A dedicated location (e.g., facility, community) where surveillance data are collected Subnational data Data disaggregated by administrative units below the national level (e.g., provinces, districts, counties); may also include other breakdowns below the national level (e.g., urban, peri-urban, rural) Surveillance data Data collected on a recurring basis from designated locations (see sentinel sites) to provide insights on trends into a broader area and/or larger population 14NNIS TECHNICAL NOTE | ASSESSING A NATIONAL NUTRITION INFORMATION SYSTEM NATIONAL NUTRITION INFORMATION SYSTEM © United Nations Children’s Fund (UNICEF) and World Health Organization (WHO) November 2022 Permission is required to reproduce any part of this publication. Permissions will be freely granted to educational or non-profit organizations. Published by: UNICEF Data Analytics and Innovation Division of Data, Analytics, Planning and Monitoring 3 United Nations Plaza New York, NY 10017, USA