Implementation guidance for assessments of frontline service readiness: strengthening real-time monitoring of health services in the context of the COVID-19 pandemic, 1 July 2021

IMPLEMENTATION GUIDANCE FOR ASSESSMENTS OF FRONTLINE SERVICE READINESS Strengthening real-time monitoring of health services in the context of the COVID-19 pandemic 1 July 2021 WHO continues to monitor the situation closely for any changes that may affect this implementation guidance. Should any factors change, WHO will issue a further update. Otherwise, this implementation guidance document will expire 2 years after the date of publication. © World Health Organization 2021. Some rights reserved. This work is available under the CC BY-NC-SA 3.0 IGO licence. WHO reference number: WHO/2019-nCoV/HCF_assessment/Frontline_services/2021.1 iii Contents 1. Introduction ....................................................................................................................................................... 1 1.1 Context ...................................................................................................................................................... 1 1.2 Tools .......................................................................................................................................................... 1 1.3 Objectives and recommended approach .................................................................................................. 2 2. Core modules ..................................................................................................................................................... 3 2.1 COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other health products – facility assessment tool ....................................................................................................................... 3 2.2 Continuity of essential health services: facility assessment tool .............................................................. 5 2.3 Community needs, perceptions and demand: community assessment tool ............................................ 8 3. Preparation for implementation ..................................................................................................................... 11 3.1 Governance and coordination ................................................................................................................. 12 3.2 Sampling .................................................................................................................................................. 16 3.3 Survey planning and preparation ............................................................................................................ 18 4. Guide for interviewers ..................................................................................................................................... 26 4.1 Interview skills ......................................................................................................................................... 26 4.2 Obtaining consent ................................................................................................................................... 27 4.3 Completing the questionnaire ................................................................................................................. 27 4.4 Using and updating the interviewer call log ............................................................................................ 31 4.5 Troubleshooting ...................................................................................................................................... 31 5. Guide for data managers ................................................................................................................................. 33 5.1 Data download, cleaning, management and analysis ............................................................................. 34 5.2 Field-check tables .................................................................................................................................... 36 5.3 Chartbooks .............................................................................................................................................. 36 5.4 Dashboard ............................................................................................................................................... 40 6. Data use and action plan for system strengthening........................................................................................ 41 6.1 Purpose .................................................................................................................................................... 41 6.2 Action plan for system strengthening ..................................................................................................... 41 7. References ....................................................................................................................................................... 49 8. Annex 1. Suite of health service capacity assessment modules ..................................................................... 50 9. Annex 2. Interpreting results from a stratified sample ................................................................................... 51 iv Abbreviations COVID-19 novel coronavirus 2019 disease CSV comma-separated values IMST incident management support team IPC infection prevention and control MoH Ministry of Health PPE personal protective equipment WHO World Health Organization 1 1. Introduction 1.1 Context Countries face a multitude of questions and decisions that must be addressed to prepare for and respond to the COVID-19 (novel coronavirus 2019 disease) pandemic while simultaneously maintaining the delivery of other health services. Key decisions made and actions taken to mitigate the risk of potential health system collapse must be informed by accurate and real-time data collected through ongoing monitoring during all phases of the COVID-19 pandemic. Against this rapidly evolving situation, many countries are facing challenges in ensuring the availability of accurate and timely data on the capacities of frontline health providers and facilities to deliver essential COVID-19 tools. At the same time, routine data systems are falling short in their ability to detect and track the extent of disruptions across essential health services that could inform mitigation strategies and responses to evolving community needs and barriers to accessing care. In such contexts, countries should consider implementing regular and rapid assessments in facilities and communities to prioritize needs for service availability; workforce capacities, training and protection; the availability of essential health products and supplies; vaccine readiness; infection prevention and control (IPC) capacities; and safety measures. 1.2 Tools To address the dual-track challenge of responding to COVID-19 while maintaining the delivery of essential health services, the World Health Organization (WHO) has developed the Suite of health service capacity assessments in the context of the COVID-19 pandemic, a new collection of tools for health facilities and communities to support rapid and accurate monitoring of current, surge, and future frontline service capacities throughout the different phases of the COVID-19 pandemic. The suite consists of modules that can be used to prioritize actions and decision-making at the health facility, subnational and national levels. The suite includes three core modules. • COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other health products for COVID-19 – facility assessment tool. This tool aims to assess health facilities’ capacities for COVID-19 case management, including the availability of diagnostics, therapeutics and other essential health products, such as oxygen and personal protective equipment (PPE), as well as cold chain capacities. • Continuity of essential health services: facility assessment tool. This tool aims to assess the capacities of primary care and hospital facilities to deliver essential health services (including the availability of health workers and their infection rate, isolation and triage capacities, adherence to IPC standards, and the availability of essential medicines and supplies) and helps to track changes in service utilization and modifications to service delivery. • Community needs, perceptions and demand: community assessment tool. This tool aims to collect information on unmet health needs, changes in care-seeking behaviours, and barriers to care affecting service demand using information from interviews with key community informants. When implemented regularly, the tools can help alert authorities and other stakeholders to modifications needed in service delivery or investment, or both, to guide the response of health systems. The suite also includes additional modules that can be used for one-time or recurrent in-depth assessments, which countries may select and customize according to their context and needs. The modules have been developed based on WHO’s guidance on COVID-19 preparedness and response, facility readiness for COVID-19 case management and on ensuring the continuity of essential health services during the COVID-19 outbreak, including Maintaining 2 essential health services: operational guidance for the COVID-19 context (1). The complete set of modules is listed in Annex 1. 1.3 Objectives and recommended approach The primary objective of the assessment (or survey) is to rapidly detect and monitor bottlenecks in health systems and health service capacity as well as gaps in readiness to ensure that essential health services continue to be provided while health systems respond to the rapidly changing context of the COVID-19 pandemic. As such, the surveys must respond to urgent needs for regular, real-time data, and they must be implemented rapidly, efficiently and safely in the emergency setting of the pandemic (e.g. surveys must be implemented in a manner that respects social distancing and travel restrictions and makes use of limited resources that have a high return, thus reducing the burden on facilities to the greatest extent possible). The recommended method of implementation is to use an electronic questionnaire delivered rapidly and regularly through telephone interviews to a sample of sentinel facilities. (These types of surveys are also referred to as computer-assisted telephone interviews.) While such a survey will not necessarily be fully representative of the national context, sentinel facilities can nonetheless provide early evidence of changes in the provision and utilization of health services, and findings can be used to inform strategies for modifications to service delivery and to guide investments of resources. A single round of data collection should be completed in a short time frame (approximately 1 week) to enable the use of real-time data. Telephone interviews also have the potential to save costs, as travel is not required and a greater number of people in a greater number of facilities can be interviewed during a given day. Nevertheless, compared with in- person interviews, the method carries its own unique complexities, and so clear governance structures, coordination and planning, operational protocols, and trainings to build skills to address the challenges of telephone interviews (see Section 4) are needed. 3 2. Core modules 2.1 COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other health products – facility assessment tool This tool was developed to assess capacities for COVID-19 case management, and it focuses on ensuring the provision of health products for COVID-19 patients to facilities designated to serve these patients. The tool allows regional or national governments, or both, and health facilities (if it is used for self-assessment) to evaluate the availability of and status of stock-outs of critical COVID-19 medicines, equipment and supplies at each site and to identify areas that need further attention to enable the facility to respond effectively to the pandemic. It is intended for use in health facilities treating moderate, severe and critical cases of COVID-19, including hospitals and designated COVID-19 treatment facilities. See Tables 1 and 2 for further details on the module’s objectives, use, content areas and key performance indicators. Table 1. COVID-19 case management capacities module: objectives, use and content areas Assessment Description Objective Assess current and surge capacities of health facilities for COVID-19 management (i.e. clinical tools and essential supplies) Use Guide rapid deployment and scale up of essential COVID-19 clinical tools and supplies Target audience • Incident management and emergency operations officers • Facility managers • Pharmacists • Biomedical engineers • Infection prevention and control officers • Planning officers • Procurement officers • Laboratory staff Respondents Facility managers or facility management team members, or both, in hospitals and COVID-19 treatment centres Content Section 1: Health facility identification and description Section 2: Hospital incident management support team Section 3: Case management and bed capacities for COVID-19 Section 4: Selected medicines and supplies for COVID-19 case management Section 5: Personal protective equipment and infection prevention and control Section 6: COVID-19 laboratory diagnostics Section 7: Medical equipment for diagnosis, patient monitoring and case management Section 8: COVID-19 vaccine readiness (optional) Section 9: Interview results When to use From the early stages of the emergency to early recovery 4 Table 2. COVID-19 case management capacities module: questions this module helps to answer and key performance indicators Sections Key questions Key performance Indicators 1 Health facility type and description • What are the facility’s characteristics? • All key performance indicators can be disaggregated by facility type, residence area (rural/urban) and managing authority (public/private) 2 Hospital IMST • Have facilities adopted and activated IMST protocols? • Percentage of facilities with IMST protocols adopted and activated 3 Case management and bed capacity • Do facilities have sufficient beds and space to manage COVID-19 patients? • Total no. of beds for COVID-19 patients for care for moderate, severe or critical patients • No. of beds currently occupied by COVID- 19 patients • Total no. of beds available for surge (intensive care unit, respiratory isolation) 4 Medicines and supplies • Do facilities have the necessary medicines and medical supplies to manage COVID-19 patients? • Percentage of facilities with tracer medicines available • Percentage of facilities participating in the Solidarity clinical trial (and availability of trial medications) 5 PPE and IPC • Do facilities have the necessary PPE for health workers? • Do facilities have the necessary IPC supplies? • Percentage of facilities with PPE available for staff (masks, gowns, goggles) • Percentage of facilities with IPC supplies available (soap, biohazard bags, sanitizer stations) 6 COVID-19 laboratory diagnostics • Do facilities have the necessary diagnostic supplies for COVID- 19 testing? • Percentage of facilities with laboratory diagnostic capacities with tracer items (specimen collection supplies, on-site PCR or rapid diagnostic tests, system for off-site testing) • Percentage of facilities receiving timely test results 7 Medical equipment • Do facilities have the medical equipment necessary for COVID-19 diagnosis, monitoring and case management? • Percentage of facilities with available and functional medical equipment on site for COVID-19 diagnosis, monitoring and case management (X-ray capacity, pulse oximeters, ventilators, oxygen) • Percentage of facilities with malfunctioning equipment (and reasons for these) 5 8 COVID-19 vaccine readiness • Do facilities have a functioning cold chain ready to support the introduction of COVID-19 vaccines? • Percentage of facilities with functional cold chain capacity to deliver COVID-19 vaccines (vaccine refrigerator with continuous temperature recorder, vaccine carriers and cold boxes, ice packs) IMST: incident management support team; IPC: infection prevention and control; PPE: personal protective equipment. 2.2 Continuity of essential health services: facility assessment tool The continuity of essential health services module is a comprehensive tool that can be used to rapidly assess a health facility’s and workforce’s capacities to continue to provide essential health services during the COVID-19 pandemic. The tool collects information about the health workforce’s capacities, financial management of the facility, changes in the delivery and utilization of health services and reasons for the changes, IPC capacities, and capacities to manage mild to moderate COVID-19 cases in primary care settings. It also includes optional modules that can be used to assess the availability of therapeutics and diagnostics, vaccine readiness and infrastructure. See Tables 3 and 4 for further details on the module’s objectives, use, content areas and key performance indicators. Table 3. Continuity of essential health services module: objectives, use and content areas Assessment Description Objective Assess a health facility’s and health workforce’s capacities to continue to safely provide essential health services throughout the pandemic Use Detect and track changes in service utilization, modifications to service delivery, staff capacities and protection for staff to guide strategies and plans to mitigate disruptions and maintain essential health services Target audience • National and subnational health authorities • National and subnational COVID-19 incident management support teams • Facility managers • WHO and other partners Respondents Facility managers or facility management team members, or both, in hospitals and primary care facilities 6 Content Section 1: Health facility identification and description Section 2: Staffing Section 3: Financial management Section 4: Service delivery and utilization Section 5: COVID-19 infection prevention and control and personal protective equipment Section 6: Management of suspected and confirmed COVID-19 cases in primary care facilities Section 7: Availability of selected tracer therapeutics (optional) Section 8: Availability of diagnostics (optional) Section 9: Vaccine readiness (optional) Section 10: Facility infrastructure (optional) Section 11: Interview results Annex: Routine data record review When to use From the early stages of an emergency to recovery and continuity after recovery 7 Table 4. Continuity of essential health services module: questions this module helps to answer and key performance indicators Sections Key questions Performance Indicators 1 Health facility type and description • What are the facility’s characteristics? • All key performance indicators can be disaggregated by facility type, residence area (rural/urban) and managing authority (public/private) 2 Staffing • How many staff are available? • How many staff have been diagnosed with COVID-19? • Are additional training and support being provided to staff? • Percentage of staff diagnosed with COVID-19, by occupation • Percentage of facilities with staff leave or absences and reasons for absences or changes in staff management • Percentage of facilities providing staff training, supportior supervision 3 Financial management • Are facilities continuing to charge user fees during the pandemic? • Are facilities receiving additional funding for essential health services? • Are staff salaries and overtime being paid on time? • Percentage of facilities that waived or increased user fees • Percentage of facilities receiving additional funding for essential health services and sources of funding • Percentage of facilities maintaining on-time salary and overtime payments 4 Service delivery and utilization • Has the delivery of services that are unrelated to COVID-19 changed? • Has service utilization increased or decreased, and, if so, what are the reasons for the change? • Has the facility implemented community communication campaigns? • Has the facility made catch-up plans for missed routine appointments? • Percentage of facilities with modifications to service delivery • Percentage of facilities with observed increases or decreases in tracer services (outpatient, inpatient, emergency) and reasons for the changes • Percentage of facilities with plans for restoring services • Percentage change in service utilization (record review) 5 IPC and PPE • Are processes and protocols in place to ensure the safe delivery of health services? • Do facilities have triage and isolation capacities? • Do staff have sufficient PPE to safely deliver essential services? • Percentage of facilities with measures to ensure a safe environment (triage capacity, isolation capacity) • Percentage of facilities with IPC guidelines in place • Percentage of facilities with adequate PPE for staff 8 6 Management of COVID-19 in primary care • Which COVID-19 primary care services are being delivered in the facility? • What support is being provided to deliver these services? • Percentage of primary care facilities with measures to manage mild cases of COVID-19 • Percentage of facilities with capacity to provide COVID-19 services in primary care (collecting specimens for diagnosis, on-site testing, referrals) 7 Therapeutics • Do facilities have therapeutics available to provide essential health services? • Percentage of facilities with tracer therapeutics, supplies and vaccines 8 Diagnostics • Do facilities have diagnostic tests and supplies available to provide essential health services? • Percentage of facilities with tracer diagnostics 9 Vaccine readiness • Do facilities have functioning cold chain capacity? • Percentage of facilities with cold chain capacity 10 Facility infrastructure • Have facilities experienced unplanned closures? • Have facilities experienced infrastructure-related issues? • Percentage of facilities that have experiences unplanned closures • Percentage of facilities with infrastructure-related issues IPC: infection prevention and control; PPE: personal protective equipment. 2.3 Community needs, perceptions and demand: community assessment tool This tool can be used by countries to conduct a pulse survey to assess a community’s health needs, perceptions around access to essential health services during the COVID-19 outbreak, attitudes towards COVID-19 vaccination, and assets and vulnerabilities. See Tables 5 and 6 for further details on the module’s objectives, use, content areas and key performance indicators. Table 5. Community needs, perceptions and demand module: objectives, use and content areas Assessment Description Objective Assess community health needs, changes in care-seeking behaviours and barriers to care affecting the demand for services, and disruptions to community-based care Use Guide strategies and plans to address unmet health needs and eliminate barriers to care Target audience • National and subnational health authorities • National and subnational COVID-19 incident management support teams • Facility managers Respondents Community leaders, community health workers, leaders of civil society organizations Content Section 1: Identification and informed consent Section 2: Community needs and use of essential health services Section 3: Barriers to seeking essential health services in the community Section 4: Attitudes towards COVID-19 vaccination 9 Section 5: Barriers to the delivery of community-based services Section 6: Interview results When to use From the early stages of an emergency to recovery and continuity after recovery Table 6. Community needs, perceptions and demand module: questions this module helps to answer and key performance indicators Section Key questions Key performance indicators 1 Identification and informed consent • Who is the key informant providing the responses? • What is the residence setting of the community? • All key performance indicators can be disaggregated by type of key informant (community health worker, community leader, representative of a civil society organization) and residence area (rural/urban) 2 Community needs and use of essential health services • How has the COVID-19 pandemic affected the utilization of essential health services? • What are the current unmet needs for health services in the community? • Percentage of key informants who believe that community has unmet health needs 3 Barriers to seeking care • What are the main barriers faced by people seeking to use essential health services during the COVID-19 pandemic? • Are there marginalized groups that have been more affected during the COVID- 19 pandemic? • Where or what is the first point of contact for health care during the COVID-19 pandemic? • Percentage of key informants who believe that the community faced barriers to seeking care before the COVID-19 pandemic and percentage who believe the barriers have become worse • Percentage of key informants who believe there are disadvantaged groups in the community • Percentage of key informants who report specific type of service delivery points as the first point contact during the COVID- 19 pandemic in the community 4 Attitudes towards COVID-19 vaccination • What is the community’s attitude towards a COVID-19 vaccine? • Percentage of key informants who believe there is community demand for the COVID-19 vaccine (adults, children) • Reasons for low demand 5 Barriers to the delivery of community-based services • Have community health workers been able to continue their work during the COVID- 19 pandemic? • Have community health workers experienced stigma • Percentage of community health workers with perceived risks and reasons for these perceptions • Percentage of key informants lacking support to perform their work 10 while pursuing their responsibilities? • Percentage of communities with changes in service volume, by type of service (malaria prevention, social support for tuberculosis patients) 11 3. Preparation for implementation Each assessment (or survey)has been designed so that it can be implemented rapidly, frequently and at low cost. However, it is critical to ensure good governance, coordination and planning to prepare for successful implementation. Table 7 outlines the generic steps and activities necessary for conducting an effective survey, from planning and preparation to data analysis, visualization and the use of the collected information. A concept note and roadmap should document the specific timeline, responsible staff and any required resources for each activity. Each assessment prioritizes the rapid collection and use of data in the pandemic context: rapid turnaround is possible only with sufficient planning and preparation prior to data collection. It is recommended that the survey manager leads and coordinates detailed activities during the planning and preparation step. In addition, although the analysis and dissemination of results, and data use is listed separately as Step 5 (see Table 7), preparation for data analysis starts as soon as the modules are adapted (Table 7, Step 3), and planning for data use must start when the key stakeholders are identified (Table 7, Step 1). The steps are described in further detail in the following sections. Table 7. Outline of survey steps and activities for the workplan for implementing assessments of frontline service readiness for the COVID-19 pandemic Step Activities 1 Governance and coordination  Ministry of Health submits formal request for support to WHO and partners  Establish a dedicated coordination entity led by the Ministry of Health and comprising key stakeholders in the country and partners  Determine the scope of the assessment (i.e. the modules to be used) and frequency of the survey (e.g. monthly or quarterly)  Identify an in-country team to support the overall management and implementation of the assessment, including a survey manager, a supervisor, data collectors or interviewers, a data manager or analyst  Prepare an overall plan  Identify resource requirements and prepare a budget 2 Sampling  Obtain a master list of facilities (i.e. a list of all health facility sites, both public and private) and develop a list of facilities eligible for inclusion  Determine the sentinel facilities to be sampled based on the sampling strategy being used  If conducting a community survey, consider sampling the key informants for communities within the catchment areas of the sentinel sites 3 Assessment planning and preparation  Review and adapt modules and training materials to meet country-specific needs  Update online interview tools according to the county-specific adaptations made to the modules  Test the tools, evaluate the results and make amendments if necessary  Review and update the standard analysis plan and automated outputs according to the county-specific adaptations made to the modules  Procure the necessary equipment and coordinate logistics  Plan and conduct training for interviewers  Develop a contact list based on the facilities being sampled 12  Do outreach to facilities before the survey to make them aware of it, confirm who will be the respondent and schedule an appointment  Develop a schedule for overall data collection  Develop a daily call schedule for interviewers  Prepopulate the background questions about facilities in the modules with details from the master facility list 4 Data collection  Conduct telephone interviews  Monitor progress and update the data collection schedule, as needed  Update the daily call schedule for interviewers, as needed  Validate and check the completion of questionnaires daily  Download survey data from the server daily 5 Analysis, results dissemination and data use  Update and produce automated database and outputs daily  Develop a roadmap for decision-making that is based on the findings  Incorporate feedback on information that needs to be included into the next round of data collection, as needed  Integrate findings into or scale up for inclusion in the national preparedness and performance monitoring dashboard (e.g. long-term focus) 3.1 Governance and coordination 3.1.1 Define roles and responsibilities Once the steps for the survey have been identified, it is essential to clearly assign roles and responsibilities. The survey should usually be undertaken under the overall leadership of the Ministry of Health (MoH). Given the context of the COVID-19 pandemic, it is recommended that the operational organization should be substantially simplified compared with typical health facility assessments. Summaries of the roles and responsibilities of the key parties involved are outlined below. • Ministry of Health: The MoH is responsible for the overall coordination and steering of the implementation of the assessment. The MoH should designate a focal point (or points) who will have overall responsibility for coordinating the process, including relevant national institutes and other key stakeholders. The MoH is responsible for identifying the implementation team, defining the scope and frequency of the surveys, guiding the adaptation of the questionnaire (see Section 3.3.1), coordinating the survey, ensuring access to the master list of facilities and permission from the health facilities, and presenting and ensuring that the data are used to define and update policies and planning to address continuity in delivering essential health services and improving responses to the COVID-19 pandemic. If relevant, the MoH may also establish a survey coordinating group with partners. • Coordinating group: Depending on the presence and interest of stakeholders and partners, a coordinating group may be established under the leadership of the MoH and involve key partners (e.g. WHO, the United Nations Children’s Fund, the World Bank, the Global Financing Facility, the Global Fund). The coordinating group provides oversight of implementation, provides technical input about tools and protocols, helps to define the scope and frequency of the surveys, and guides the adaptation of the relevant questionnaire. When there is not a coordinating group, all of the above responsibilities will be coordinated by the MoH. Even if a formal group is not established, WHO and other partners can provide input into the review processes and technical or financial support, or both, and support the interpretation and use of results and the policy dialogue and planning processes. 13 • Implementation team: The implementation team leads all operational and implementation aspects of the survey and of data collection. The team comprises a survey manager, a supervisor for data collection (if different from the survey manager), a team of interviewers, and at least one data manager or analyst. Consultants can be recruited as interviewers or they may be part of the MoH workforce. In cases in which an existing national public health or statistical agency is available to collaborate in a timely manner, data collection and analysis responsibilities can be assigned to that agency. 3.1.2 Define the scope and frequency of the surveys and the mode of data collection Given that the objective is not to conduct a comprehensive assessment of a health system’s performance, the recommended approach is to conduct regular telephone interviews, facilitated by online data entry, with respondents in a sample of sentinel facilities to enable rapid analysis and use of data. This approach enables near real-time data collection, analysis and use, rapid and safe implementation in an emergency context and a high return with the use of only limited resources. The modules have been harmonized across the suite to maximize efficiency during the pandemic, and it is critical that the surveys have a targeted scope. The MoH and survey coordinating group must select appropriate modules according to the country’s most urgent needs and priorities. The optimal frequency of data collection will depend on the course of the pandemic, the anticipated magnitude of changes, and the resources and capacity available to implement the tools. It is recommended to plan for at least quarterly assessments and consider more frequent implementation if needed and feasible. The way in which data will be collected should be confirmed early in the planning process. The assessment tools are available in both a paper and electronic version. For the purpose of quality control and rapid turnaround of results during the pandemic, it is suggested that the electronic version is used to facilitate data entry and management. WHO has a standard electronic form that is available for immediate implementation (currently in LimeSurvey, a web-based survey platform). The standard form must be customized based on any country- specific adaptations made to the tool (see Section 3.3.1). Programming skills are not required to modify the LimeSurvey-based forms. Currently, the LimeSurvey online tool can be used in two different ways: in a web-based format that is accessible via any browser or as an application (or app) installed on an Android mobile phone. Internet connectivity is an important consideration when determining the best option for use in a country. Accessing the web-based format on a computer makes it easy for interviewers to conduct interviews and enter data, with potentially fewer errors in data entry. All data are saved as the interview proceeds, but an internet connection is required to conduct the interview. Thus, in places without a stable internet connection, the web-based form can be used on a mobile phone that has a data plan. Alternatively, the app on Android telephones can be used, since it does not require an internet connection during an interview. Interviewers can upload the data saved to their phone when a connection is available. However, data entry on a mobile phone may require more practice to minimize data entry errors. Countries may consider using platforms other than LimeSurvey, depending on which systems are available at the MoH or a national statistical agency, if involved, and their capacity. In such cases, at a minimum the application should have a built-in check for internal data consistency that pays particular attention to skip patterns and the entry of numeric data. 14 3.1.3 Identify the implementation team Even when data output is as fully automated as possible, it is essential to allocate time and human resources to interpret and use the data, especially during the first round of the survey. The implementation team should include the following positions:. • survey manager to plan and coordinate overall implementation at the country level; • supervisors (if different from the survey manager) to be responsible for overseeing data collection teams and to troubleshoot during the data collection phase. Note that small implementation teams may not require supervisors if all responsibilities can be completed by the survey manager; • data collectors or interviewers to complete questionnaires during telephone interviews with staff at facilities or key community informants; • data manager or analyst (at least one) to manage the national database and real-time analyses of the data. 3.1.4 Develop an overall plan for the survey An overall operational plan or Gantt chart (a bar chart used for project management) should be developed by the coordinating group; the plan or chart should clearly outline the key steps and activities described in Table 7. 3.1.5 Identify resource requirements and prepare a budget A budget should be prepared based on the identified requirements for resources. Multiple, context- specific factors will determine the budget. If possible, it is recommended that existing resources at the MOH be leveraged to avoid purchasing new equipment for individual interviewers. The key resource requirements are summarized below. The technical documents needed to plan the assessment include the: • master facility list; • standard modules (to be adapted according to the country’s context); • standard supporting and training materials (to be adapted as necessary); • standard data analysis code and code for the automated output of figures and tables in Excel (also referred to as chartbooks, and to be adapted as necessary). The human resources required include: • an assessment manager; • a supervisor; • a data manager or analyst(s). The equipment and infrastructure required for the interviewers includes: • a telephone and headset or earphone; • a laptop or tablet computer (note that the form can be completed on a smart phone, but data entry is easier on a tablet or computer); • a stable internet connection at a central location or in each interviewer’s home; • mobile phone top-up credits, as necessary. 15 If calls are made from a central location, additional equipment may be needed to strengthen internet and Wi-Fi connections (e.g. a router, Wi-Fi signal enhancer). If the internet connection used by laptops or tablets is unstable, then they can be used with a mobile phone connection. Human resources are an important budget line item. The number of interviewers will depend on the sample size and the survey plan. The data collection period for each module should not exceed 7 days, and it is expected that an interviewer can complete 4–5 interviews in a day. Therefore, it is recommended that 1 interviewer is recruited for about every 20 facilities. For example, for a sample size of 100 facilities, 5 interviewers will be able to complete data collection in 5 business days. If multiple modules are implemented, countries may use the same data collection team to implement all modules (lengthening the timeline) or train different data collection teams for each module (for more rapid deployment). The budget for interviewers should cover training (2 days for the first round and 1 day for subsequent rounds), 2–3 days to conduct pre-interview outreach (if interviewers are mobilized for this activity) and the data collection days. Table 8 provides illustrates how to calculate the human resource requirements. Table 8. Example of how to calculate the number of interviewers and the budget for implementing assessments of frontline service readiness for the COVID-19 pandemic Example No. of interviewers Budget calculation 100 health facilities • Assume that 1 interviewer can cover 4 facilities per day • 1 interviewer can cover 20 facilities per week • 5 interviewers can cover 100 facilities per week Training: 5 interviewers × [daily rate] × [training days per year] Pre-interview outreach: 5 interviewers × [daily rate] × [outreach days per year] Data collection: 5 interviewers × [daily rate] × [5 working days] × [frequency per year] 50 community interviews • Assume that 1 interviewer can cover 5 communities per day • 1 interviewer can complete 25 community interviews per week • 2 interviewers can complete 50 community interviews per week Training: 2 interviewers × [daily rate] × [training days per year] Pre-interview outreach: 2 interviewers × [daily rate] × [outreach days per year] Data collection: 2 interviewers × [daily rate] × [5 working days] × [frequency per year] 16 3.2 Sampling This section describes the recommended sampling approach for the surveys. 3.2.1 Sampling approach Depending on the resources available and the context, strategies to select the sentinel facilities will vary across counties. Generally, a stratified sampling approach is recommended for selecting sentinel sites, if feasible and a master facility list is available. Within each of the select sampling strata, facilities will be selected randomly with equal selection probability. Since data are collected via telephone, there are no extra cost implications to surveying sentinel sites that are geographically dispersed, which is a potential result of random sampling within a stratum. Stratified sampling will provide representative sentinel facilities for each analysis domain as well as for the country, adjusted for sampling weight. Observed trends from the sentinel sites may be interpreted to be representative for all facilities in the country, depending on the acceptable level of precision.1 If a stratified sample is planned, sampling weights should be provided in the chartbooks (see Section 5.3) in order to calculate nationally representative estimates. Further information on calculating sampling weights is available elsewhere (2). In settings where the master facility list is incomplete or outdated, alternative approaches may be considered, for example, selecting facilities known to have high-quality data reporting, targeting districts with poor service provision prior to the pandemic or utilizing existing sentinel facilities that have been selected for other purposes. If sentinel sites are selected for convenience, observed trends may not be representative of all facilities in the domain. Therefore, it is important to document the reasons why sites were selected and communicate to data users any implications for interpreting the results. 3.2.2 Eligible facilities and sampling frame For the module on COVID-19 case management capacities, eligible facilities include all those that provide COVID-19 case management, and for the module on the continuity of essential health services, eligible facilities are those that provide essential health services. Eligible facilities must meet any predefined criteria that are specific to the country. Ideally, an existing master facility list will serve as a sampling frame. For example, if the aim is to include facilities across all managing authorities and all types in the country, all facilities in the master list are eligible for the assessment. However, if the assessment focuses only on public facilities in selected regions – which may be a strategic option in settings in which a majority of essential health services are provided predominantly in the public sector – then the eligible facilities are all public facilities in the master list that are in the selected regions. Depending on the country’s context, any special facilities that typically do not provide a common package of essential health services (e.g. convalescent hospitals) should be excluded from the sampling frame. Further, since all sentinel facilities must be accessible via telephone, facilities without valid telephone numbers should also be excluded from the sampling frame. WHO does not recommend updating or constructing a master facility list for this assessment during the pandemic (3). Updating or constructing a master facility list is an important step when conducting conventional health facility assessments. But it can be a labour- and time-intensive undertaking, and it is unlikely to be feasible during the pandemic, given the many competing priorities. These assessments rely on the existence of a master list that is fairly up to date in terms of both the number of facilities, by their characteristics, and their contact information. 1 Unless the magnitude of changes is substantial, the total sample size is most likely too small to make statistical inferences about the estimates over time, even at the national level (see Annex 2). 17 3.2.3 Sample size These assessments aim to monitor signals of disruption to the provision of services among sentinel surveillance facilities during the COVID-19 pandemic. The assessments emphasize monitoring trends among sentinel sites rather than making statistical inferences about point estimates, as in typical sample surveys. Thus, although it is unconventional to do so, sample size can be rather simply determined to provide stable estimates among the sentinel facilities. The key domains for analysis should be identified. These may include the facility type (e.g. hospital, primary care facility, long-term care facility), the managing authority (i.e. public or private), administrative units selected by COVID-19 transmission status (i.e. regions that are more or less affected by COVID-19) and geographical distribution (i.e. rural or urban). The number of analysis domains and the number of categories within each domain will need to be determined strategically based on the implementation resources and timeline to ensure rapid data collection, analysis and use during the pandemic Since some indicators are calculated among only a subset of facilities (e.g. reasons for staff absenteeism among facilities that had staff absence during the preceding 3 months), it is recommended that 35 facilities are included per analysis domain to ensure that at least 25 facilities are in the denominator for most of the key indicators. The highest number of strata in a domain determines the total sample size. Table 9 illustrates how sample sizes are calculated based on the type and number of domains to be included. If a selected facility cannot be reached after three attempts, a replacement facility should be selected to maintain the sample size (see Section 3.3.5). Table 9. Illustration of the total number of sentinel sites needed based on different sets of analysis domains. Note that the total sample size is determined by the domain that has the highest number of strata (in these scenarios, they are 3, 4 and 2, from top to bottom). The domains that determined the sample sizes are in bold. Sampling or analysis domain Highest number of strata Calculation Sample size • Facility location (urban versus rural) • Managing authority (public versus private) • Facility type (hospital, health centre and health post) Three strata (by facility type) 35 × 3 105 • Facility location (urban versus rural) • Managing authority (public versus private) • Facility type (hospital, health centre and health post) • Four administrative units purposively selected Four strata (by administrative unit) 35 × 4 140 • Facility location (urban versus rural) • Managing authority (public versus private) • Facility type (only primary care facilities) Two strata (by facility location and managing authority) 35 × 2 70 18 3.2.4 Sentinel surveillance facilities Target facilities within each domain should be selected randomly from the master facility list. The assessment is designed to be repeated over time among the sentinel facilities, thus it is critical to avoid targeting interventions to the sentinel facilities that are included in the assessment. Thus, it is important that only members of the implementation team and interviewers are aware of the sites that are included in the assessment and during follow up. 3.3 Survey planning and preparation This section outlines the key activities necessary for planning and preparation (Table 7, Step 3). Survey managers should plan and lead all activities described in this section, and they must be included in discussions about the survey as early as possible, ideally at the outset of implementation (Box 1). Survey managers should receive and review all technical assistance documents and tools. Box 1. Roles and responsibilities of the survey manager Survey managers are critical for successful implementation. They lead and coordinate all activities during planning, preparation and data collection (Table 7, Steps 3 and 4). They also participates in analysing the assessment, disseminating the results and ensuring that the data are used appropriately (Step 5), in close collaboration with the data manager or analyst. The primary responsibilities of the survey manager are described below; however, some of the responsibilities can be shared with other team members, depending on the structure of the implementation team. Assessment managers:  coordinate the finalization of country-specific adaptations of interview tools and supporting documents;  coordinate the finalization of sampling;  secure appropriate, functioning equipment and manage it and other resources;  recruit, train and supervise interviewers;  conduct or manage outreach to sites before data are collected;  develop and update the daily call schedule for all interviewers;  manage day-to-day operations during data collection, including troubleshooting;  with the data manager, monitor the progress of the assessment and quality of the data;  participate in interpreting the results;  participate in disseminating the results and in ensuring the data are used appropriately. Table 10 illustrates a fairly rapid timeline for key preparation activities. Certain activities can be carried out earlier than shown. For example, the facility contact list can be developed as soon as sampling is completed. However, certain activities have prerequisites. When the timeline for the prerequisites changes, the timelines for subsequent activities and data collection must be adjusted accordingly. 19 Table 10. Illustrative timeline of detailed activities necessary for survey planning and preparation before data collection for assessing frontline service readiness for the COVID-19 pandemic. Note that the number of days needed may vary depending on the context. Also, a day does not necessarily refer to a full business day. Shorter timelines may be possible during follow-up rounds Activity Day and order of activities 1 2 3 4 5 6 7 8 9 10 Develop tools Review and adapt modules and training materials to meet country-specific needs X X X X X X Update and pilot test online toolsa X X X X X Recruit and train interviewers Recruitmenta,b X X X X X X Training and pilot testing in the field X X Prepare for interviews Procure necessary equipment and coordinate logisticsa,b X X X X X X Develop a facility contact listc X Conduct outreach to facilities to make them aware of the assessment, confirm respondents and schedule appointmentsd X X Develop an interview schedule, including web links to online tools specific to each module and facility X Prepopulate the data for facility background questions in the modules with details from the master facility list X Prepare for data analysis and of data productse Review and update the standard analysis plan and automated outputs according to changes made for county-specific adaptation of modules X X X X X X X X Review and update the standard template for the data dashboard X X X X a This step must be completed prior to training interviewers. b Recruitment and procurement may take substantially longer depending on the local regulations and context. c This step can start as soon as sampling is completed; however, it must be completed prior to pre-interview outreach. d This step can start as soon as the facility contact list is completed if staff are available. e These activities are completed by data analysts or managers, with inputs from the survey manager. They are described in detail in Section 5. 3.3.1 Review and adapt modules and materials Based on the objectives and scope of the assessment in each country, the modules are selected by the MoH and survey coordinating group (see Section 3.1.2). The modules include standardized questions and response options that can be used in different settings. But some questions and response options require country- specific adaptation to reflect the specific needs of national health systems. It is recommended that the survey manager makes basic changes to the relevant options (e.g. facility type in the country, name of the administrative units) prior to engaging a technical group. Technical group members should be requested to review the standard modules and share questions or suggestions prior to a group meeting. Technical members should be briefed about (i) the goal and specific aims of the assessment and (ii) the sample 20 design. It is recommend to plan for 1–2 hours of discussion and decision-making for each of the facility modules and less time for the community module. Standard annotated modules can be used during the group meeting to provide answers to common questions. There are four types of adaptations: optional sections can be chosen within each module; words or phrases in the questions and responses can be customized; optional questions and responses can be chosen; and country- specific questions can be added. 3.3.1.1 Choosing optional sections within each module Within each selected module, countries should choose to include or exclude the optional sections. The decision about these sections should be based on whether the information is available through other data sources, the urgency of the need for the data and potential programmatic responses that may be taken based on the data. Countries may consider including all or only certain sections in every round of data collection or only during certain rounds. For example, if a country has data from strong logistics management information systems, the availability of selected tracer therapeutics may not be prioritized for the module assessing the continuity of essential health services. Or if a recent health facility assessment covered, for example, diagnostics readiness or infrastructure, the availability of diagnostics or infrastructure may not need to be included in the assessment. The module assessing COVID-19 case management capacities includes the following optional section – • Section 8: COVID-19 vaccine readiness. The module assessing the continuity of essential health services includes the following optional sections – • Section 7: Availability of selected tracer therapeutics • Section 8: Availability of diagnostics • Section 9: Vaccine readiness • Section 10: Facility infrastructure. In addition, in settings for which repeated assessments are planned, certain mandatory (non-optional) sections may not need to be repeated in every round, based on the expected rate of change during a short period. For example, in the module assessing COVID-19 case management capacities, Section 7 (Medical equipment for diagnosis, patient monitoring and case management) may not need to be repeated quarterly. 3.3.1.2 Customizing phrases or words in the questions and in response options Each module includes options for country-specific questions and responses that will need adaptation, noted as “county-specific question adaptation” or “country-specific response adaptation” in the paper questionnaires. Examples of such tailoring include using the country’s names for administrative units and classifications for different types of facilities. 3.3.1.3 Choosing optional questions and responses The modules have what are known as country-specific optional questions. These questions are included in the standard tools because they may be relevant for multiple countries but not necessarily for all countries. Exclude the optional questions unless (i) the subject is relevant to the country context and (ii) the sample design already allows for adequate analysis of the particular question. To facilitate decision-making, modules include footnotes that indicate which country-specific questions are optional. 21 3.3.1.4 Adding country-specific questions In addition, questions specific to the country can be added. To protect data quality and given the time frame and intended scope of the survey, it is recommended that new questions be added only when information is urgently needed and it will have direct and immediate programmatic implications. Also, adding country-specific questions requires more preparation by data managers who will need to update and test the analysis code for the country-specific modules, which is a critical part of finalizing the modules. 3.3.2 Test country-specific tools and update training and supporting materials Adaptation should first be completed using the paper tool, then the online tool should be updated and tested prior to interviewer training. This preparation is critical to ensure smooth implementation and good data quality, and it is recommend that at least 1 full day per module should be allotted for updating the online tool prior to training. If the country-specific adaptation is extensive, more staff time may be needed. The survey manager coordinates the pilot testing of the online tool. If feasible, this should include online data entry and interviews with a few facilities that will not be sampled as surveillance sites. If this is not feasible, then mock interviews can be organized involving supervisors or those who participated in adapting the questionnaires, or both. In either case, it is recommended that the tool is tested with different types of facilities and respondents (e.g. urban versus rural facilities, facilities providing high-level care versus those providing low-level care), so that different skip patterns can be tested and the range of interview times can be estimated. The survey manager should consolidate all feedback and make amendments as needed to the tools, processes and logistics. Final adjustments must be reflected in both the paper and online tools. Once the modules are finalized, other supporting materials must be updated to reflect the country-specific changes, including: • the interviewer training slides. Template slides are provided, including example questions that are likely common across settings. However, details of logistics and processes must be updated according to the country’s context; • annotated questionnaires. The questionnaire has been carefully developed with technical experts, and all questions have been specifically written to provide as much clarity as possible. Still, there may be questions about the meaning of certain words or phrases during interviewers’ training, field practice and even the actual interviews, and it is important to have a consistent approach to addressing these questions and answers. Annotated questionnaires include an additional column describing frequently asked questions and the answers to them, as well as images of certain items asked about in the tool. Survey managers should update these annotations so they can be used as a reference during training and data collection; • data management and analysis codes, chartbooks and dashboards. Suggestions for updating data analysis and data products are discussed in Section 5. 3.3.3 Secure equipment and logistics support All necessary equipment that was identified during the planning and coordination phases must be procured prior to training for the interviewers (see Section 3.1.5). How the interviewers will be trained and the venue for training must be determined and prepared for according to the country’s context. 22 3.3.4 Train the implementation team The number of interviewers needed will be determined by the sample size, timeline and budget for the assessment (see Section 3.1.5). The survey manager is responsible for recruiting the interviewers. Once the implementation team is complete, training sessions must be planned and conducted. 3.3.4.1 Interviewer training The survey manager organizes and leads training. It is recommended that training is as interactive and engaging as possible, while the experience of the interviewers and dynamics of the group are also considered. Template training slides are provided. See Sections 4 for further information. The recommended contents and required materials for training are shown in Table 11. A structured training schedule should be developed that includes the recommended contents, which can be rearranged as necessary. Table 11. Recommended contents and required materials for interviewer training for conducting assessments of frontline service readiness for the COVID-19 pandemic Category Description Contents • Introduction of team and survey, including a brief description of the facilities to be contacted and respondents to be interviewed • Conducting interviews ­ Skills for successful interviews ­ Tips on completing the questionnaire ­ Module review, using the paper tool ­ Classroom practice, using the online tool ­ Field practice, using the online tool • Overview of the data collection schedule ­ Daily schedule, including daily debrief and troubleshooting ­ Using and updating the interviewer call log Materials • Updated country-specific paper tool (one copy per interviewer) • Updated country-specific annotated modules • Updated country-specific online tool • Web link for accessing the online tool • Training slides • List of facilities and respondents recruited for field practice and their telephone numbers It is important to review the modules using the paper version, but the review can be tedious for the interviewers. Try to provide as engaging and creative a training as possible. For example, consider reviewing the paper questionnaire and practicing using the online tool section by section, rather than reviewing all of the paper questionnaire at one time. Briefly explain the purpose of each section or break down longer sections into shorter groups of questions. It is recommended that the facilitator reads the first section to the interviewers; however, for later sections, to keep the interviewers engaged, the facilitator can invite each interviewer to read a group of questions. 23 For classroom practice, interviewers should be paired up, choose a hypothetical facility and respondent, and conduct a mock interview by alternating roles (i.e. respondent and interviewer). For field practice, facilities and respondents who are not included in the sample should be recruited. For this part of the training, it is recommended that interviewers are grouped into teams with each team interviewing at least two facilities per module of two different types, if feasible. Interviewers can take turns to cover different sections while the rest of team listens to the practice interview. The field practice will highlight aspects of interviewer training that need to be re-emphasized or addressed. It may also suggest minor changes that can be made in the tool. Any such adjustments must be reflected in both the paper and online tools, as well as in the data analysis processes. 3.3.4.2 Data manager training The survey manager should coordinate or conduct training for the data analysts and managers. Template training slides are provided. For further information, including standard analysis code and information about chartbooks, see Section 5. Experienced data managers may study the training materials on their own. 3.3.5 Prepare for data collection There are a number of additional activities that are essential and must be completed prior to data collection. 3.3.5.1 Make adjustments to and confirm availability of equipment and ensure logistics support Based on feedback from the training for interviewers, make adjustments to equipment and ensure logistics support for data collection. For example, if data collection is conducted in a central location, potential feedback might include the need to improve internet connectivity and to provide quiet space for each interviewer. Confirm any logistics support that will be provided by groups other than the implementation team (e.g. information technology, the venue, catering for meals) 3.3.5.2 Develop the facility contact list The survey manager should develop a facility contact list. The list will be used for pre-interview outreach. The contact list will include the sentinel facilities to be sampled and selected background information (Fig. 1). At a minimum, it should include three pieces of information: a unique identification number for the facility from the master list (or sampling frame), the facility’s name and the telephone number. It is recommended that additional background information is included, such as the first-level administrative unit, facility type and address. This additional information can be used to verify the sampled facilities during the pre-interview outreach, and it can be prepopulated into the online tool, saving the interviewer time. 24 Fig. 1. Example of a facility contact list with the results of pre-interview outreach Facility contact information Pre-interview outreach Facility identifica tion number Name Region Facility type Address Telephone number Outreach call date Main respondent Main respondent telephone number Date and time of appointment Note 135 AAA XX Health centre … … 6/12/2020 J Mwangi … Not applicable Facility could not be reached after three calls. Replacement facility needed. 890 BB XX Hospital … … 9/12/2020 P Baressi … 17/12/2020 10 AM 159 CC YY Health centre … … 7/12/2020 C Park … 18/12/2020 12 PM 456 DD YY Health centre … … 7/12/2020 J Smith … 18/12/2020 2 PM 567 EE ZZ Health centre … … 7/12/2020 T Gichangi … 18/12/2020 4 PM 3.3.5.3 Pre-interview outreach (only for facility assessments) Reaching out prior to data collection to facilities that will be sampled is critical for four reasons. First, operationally the pre-interview call will identify a main respondent to be interviewed, obtain a telephone number for the respondent (if different from the facility’s telephone number in the master list) and determine a preferred date and time for the interview (Fig. 1). It is recommended that interview appointments are made largely during the first two thirds of the data collection period, since some appointments may need to be rescheduled. Also, sufficient time should be allowed for each interview based on the results of pilot testing and classroom and field practices. Second, pre-interview outreach is also an opportunity for identifying any facilities that will be difficult to include in the assessment (Fig. 1). It is recommended that a facility is called three times (on different days and at different times). If the facility cannot be reached, it should be replaced with another eligible facility. Third, if the module on the continuity of essential health services is being used, another important purpose of the outreach is to alert respondents that data about service volume and trends will be collected during the interview. This is the only part of the tool that requires respondents to answer based on information in documents – that is, patient registries in the facility or data provided for health management information system reporting. Before the interview, respondents should be sent the appropriate form either via mail or electronically, and they should be encouraged to complete it prior to the interview. Finally, in large facilities, different sections of the tool may require information from different persons. Consider sending a copy of the tool to the main respondents so that they can consult the relevant staff as needed. 3.3.5.4 Develop a data collection schedule Based on the results of outreach, a data collection schedule should be developed and monitored. Then facilities and appointments should be systematically assigned to interviewers based on the appointment dates and times 25 arranged during outreach and the expected length of interviews. Again, the schedule should allow sufficient time for each interview, depending on the module and facility type. It is recommended that the data collection period for each module does not exceed 7 days, and it is expected that an interviewer can complete 4–5 interviews in a day. The schedule may need to be updated regularly as a result of rescheduled interviews, dropped calls or interviews that last longer than their allocated time. It is recommended that the same interviewer is assigned to complete any interviews on their schedule that were missed. 3.3.5.5 Develop a daily call schedule for interviewers (the interviewer call log) Each day, interviewers will receive a list of facilities to contact and respondents to interview at prearranged appointment times, and this is referred to as the daily interviewer call log (Fig. 2). The log provides interviewers with all of the necessary contact information. Interviewers will also complete the final column of the log, where they record the results of each call (see Section 4.4). If the community module is also used, the call log must include a list of key community informants and their telephone numbers, although there will be not be a prearranged interview date or time. If LimeSurvey is used, the log will also include a web link that is specific to each sentinel facility. The same web link will be used when an interviewer calls a facility for follow up, if needed, to complete the interview. Fig. 2. Example of daily interviewer call log for facility assessment Facility identific ation number Facility name Facility type Facility address Facility telephone number Outreach call date Main respondent Main respondent telephone number Date and time of appointment Note Interviewer Call results 890 BB Hospital … … 9/12/2020 P Baressi … 18/12/2020 11 AM Partially completed on 17/12/2020. Rescheduled. Start again from Section 7. A 159 CC Health centre … … 7/12/2020 C Park … 18/12/2020 12 PM … A 456 DD Health centre … … 7/12/2020 J Smith … 18/12/2020 2 PM … A 567 EE Health centre … … 7/12/2020 T Gichangi … 18/12/2020 4 PM … A 3.3.5.6 Prepopulate the background information about the facility If feasible, background information about the facility can be prepopulated into the online tool to save time during the interview. Information that can be prepopulated includes the facility’s name, identification number and the administrative unit level, all of which are available from the master facility list. Prepopulating these responses can improve the efficiency of the interviews if such information is up to date and can be extracted readily from the master facility list. 26 4. Guide for interviewers The findings of the assessment are only as good as the data from which they are calculated, and data quality relies to a large extent on the interviewers. There are core interview skills that are helpful during any kind of interview, such as interacting with respect and neutrality. Telephone interviews, however, require additional skills as well as practice to address the specific challenges posed by this type of interview. Below are some basic instructions for the training and practice. 4.1 Interview skills Interviewers should be encouraged to develop the following skills. The instructions in the list below may be helpful. • Build rapport over the telephone. Greet the respondent in a confident, sincere and friendly manner. A respondent's first impression of the interviewer will strongly affect their willingness to fully participate in the interview. All respondents should be treated respectfully and politely. The respondents should know that their cooperation and the time they are taking to help make the survey successful are appreciated. • Read the questions verbatim and with the appropriate emphasis. Respondents only hear the questions and possible responses during the interview (they do not see them), so reading the questionnaire exactly as written is critical and requires concentration. The wording of each question and the options for responses has been carefully chosen and, for that reason, it is essential that the interviewer reads each question to the respondent exactly as it is written and in the same manner to all respondents. At times, the interviewer may need to repeat a question to be sure that the respondent understands it. In these cases, the interviewer should not paraphrase the question but repeat it exactly as it is written. During practice sessions, if interviewers find that they have to repeat certain questions, supervisors and assessment managers should be made aware so that the wording can be changed, if necessary. • Speak clearly and slowly. Pay attention to how fast you are speaking, especially when reading an introduction and throughout the interview. Nervousness may cause interviewers to read too fast or inflect up at the end of sentences that are not questions, thus making them sound like questions. Also, pay attention to the position of the mouthpiece. If it is too close to the interviewer’s mouth, respondents may become distracted by the sound of breathing. • Control the tempo and tenor of the engagement, subtly but firmly. This is especially important if the respondent is confused, frustrated or rushed, which may be expressed verbally or nonverbally. For example, if a respondent starts answering quickly, it can be a sign that they are growing impatient, bored or annoyed. • Maintain neutrality. Refrain from making conversational noises that may be interpreted by the respondent as approval or disapproval of their answers. Such misinterpretation can lead to unintended biases, as respondents may seek an interviewer’s approval. Also, interviewers should not read the response options in such a way that may convey their own opinions or thoughts. Interviewers must not express surprise or astonishment at any answer. 27 • Listen closely. Listening carefully is as important as asking the questions. Some questions require the interviewer to listen to what the respondent says and then record the answers by clicking the corresponding boxes for all applicable categories. Interviewers should not rush into assigning categories before the respondent has finished replying completely, otherwise interviewers are at risk of attributing their own biases to the respondent’s replies. Occasionally, a respondent may answer a question incompletely or seem to have misunderstood the question. The first step is simply to repeat the question as written. If this does not help, the interviewer needs to ask for further information without influencing the response – that is, probe for a response. For example, the interviewer might ask, “Could you explain that a little more?” or “Could you be more specific about that?” The interviewer must never interpret a respondent's answer and then ask the respondent if the interpretation is correct. Finally, the interviewer must listen closely to determine when a respondent is not paying attention, which may be signalled when respondents rush, say “I don’t know,” give inconsistent or irrelevant answers, or ask for questions to be repeated. This happens most often when respondents are concerned about their other responsibilities and want to return to work. In these cases, the interviewer must try to re-engage the respondent’s interest in the conversation. For example, if the interviewer senses that the respondent is growing restless, the respondent could be reassured that there are not many more questions and that their responses are valuable. • Enunciate clearly. Ensure that all words in the questionnaire, including medical terminology, can be read clearly and confidently. • Record information accurately. Never leave a response blank: record responses as they are given whether negative or positive, including “no”, “not applicable” and “I don’t know.” Always verify the typed answers on the screen. Sometimes the respondent may appear to understand the question, but the response may be deemed inconsistent: in this case, the interviewer must record the response as given by the respondent. • Thank respondents. Ensure that respondents know that their efforts and time are appreciated. This should be acknowledged at least at the beginning and end of the survey, and possibly during the survey as well, if appropriate. 4.2 Obtaining consent Interviewers are required to obtain consent to participate from each respondent before starting to ask questions. The consent form, which includes background information for the survey, is located at the start of the tool. 4.3 Completing the questionnaire Each module has multiple sections. Each section has instructions as well as questions. 4.3.1 Instructions The tool has instructions for respondents and for interviewers. • There is information that introduces a section or a set of questions. These are not questions but interviewers need to read them to respondents. 28 The following types of instructions are aimed only at interviewers and should not be read to respondents. • In a small number of open-ended questions, there is a specific instruction for interviewers not to read the response options aloud. • In a small number of cases, interviewers are required to record information without asking a question, such as the date of the interview and the interview results. • Footnotes that spell out acronyms are provided for the interviewer’s information. • Skip patterns refers to certain questions that are asked only to selected respondents based on their previous answers. In the paper form, these are presented in a grey background. In the electronic form, the skip pattern is built in, and interviewers (or respondents in the case of a self-administered assessment) do not see the skip instructions. However, it is critical for interviewers to understand the skip pattern to ensure the correct flow of the survey. 4.3.2 Questions There are three major types of questions in the tool. This section describes them and provides examples, using questions and response options in a web-based online form. However, please note that the examples and the question numbers may not align with those of a specific module used in a particular country. 4.3.2.1 Precoded responses: single response The most common type of question asks the respondent to select only one of the precoded responses. In the online interview, the response options will have small circles (radio buttons), and only one can be selected for each question (Fig. 3). Fig. 3. Example of a single response question that uses radio buttons (option buttons) Sometimes responses must be entered in the response grid (or table). When recording a response in one of these grids, the interviewer must be sure that the answer is entered in the correct row and column (Fig. 4). 29 Fig. 4. Example of a response grid in which the response must be entered in a particular column In a few cases, a precoded question will include an “Other” category. “Other” should be selected when the answer provided is not included in the precoded responses. Answers that fall into this category are uncommon responses and, in most cases, respondents are not asked to explain their answer further, as shown in Fig. 3. For a small number of questions, however, interviewers are asked to specify to what Other refers (Fig. 5). Fig. 5. Example of an answer that requires a respondent and interviewer to provide information about a response coded as Other Only a small number of questions have a response option for “Do not know”. If “Do not know” is not one of the options, then interviewers must probe further. 4.3.2.2 Precoded responses: multiple responses For a small number of questions, interviewers need to listen to the respondent and select all applicable answers from among the precoded responses. Do not read any of the precoded responses to the respondent. After respondents complete their answer, ask “Any other reasons?” or “Anything else?”, as written in the questionnaire. Again, interviewers should not rush into selecting categories before they have finished listening to the respondent to avoid injecting their own biases into the responses. 30 In the electronic form, these questions have small boxes that allow multiple responses to be selected instead of radio buttons (Fig. 6). Fig. 6. Example of a question that allows multiple responses to be selected and has instructions for interviewers 4.3.2.3 Numeric responses Several questions require a numeric response. These should be recorded in the appropriate spaces in the right column of the table (Fig. 7). There are preset ranges of allowed numbers (e.g. a positive integer for the number of beds); interviewers will get an error message if the numeric entry is outside the range. Fig. 7. Example of a question requiring a number to be entered into the form In addition, at the end of each call, the result of the interview must be recorded by selecting one of the predetermined categories (Fig. 8). This information is critical both for monitoring survey progress and for analyses. 31 Fig. 8. Example of categories for recording the result of the interview 4.4 Using and updating the interviewer call log Data collection is conducted during a short period of time, and the survey team prepares a call schedule prior to the start of the data collection process. Each day, interviewers will receive a list of respondents with their appointment times, known as the interviewer call log. The log will include basic information about the facilities, the date and time of interview appointments and any relevant notes about the facility. The notes may include information about previous attempts to contact the facility or partially completed interviews. See Fig. 2 for an example. After each call, interviewers must record the result both in the tool (Fig. 8) and the call log (Fig. 2). This applies to all kinds of calls, including completed interviews, rescheduled interviews, dropped calls and unsuccessful attempts to reach respondents. If calls are rescheduled, new dates and times must be provided in the log. In addition, for rescheduled or dropped calls interviewers need to record the specific section or question from where the new call should start. This information will be used when the respondent is called again. At the end of each day, the survey manager will review the logs from all interviews and then create new call logs for the next day, based on the updated interview schedule. 4.5 Troubleshooting Occasionally, interviewers may need to skip a section of the survey for logistical reasons. In a large facility, the main respondent may need to consult other staff for information about a particular section, and it may be more efficient for interviewers to skip the section and continue the interview with questions that the main respondent can answer. In such cases, interviewers must ensure that the skipped section is completed either with other staff at the facility or with the main respondent after consultation with the relevant staff. Dropped calls are a common and critical challenge in telephone surveys, particularly in low-resource settings. Interviewers must call back immediately after a call is dropped. If the respondent cannot be reached on the same day, the interviewer should call back on the next business day. In addition, while interviewers are strongly encouraged to complete the survey during one call, respondents may request to 32 be called back at another time or day for reasons that are beyond their control. The interviewer should acknowledge the time they have spent and their efforts, and respectfully accommodate the request. Interviewers should try to schedule a call back on the same day or the next day. To resume a rescheduled or dropped call, the interviewer should first open the questionnaire for the specific facility, go to the section or page where the call will start from (determined by reviewing notes in the call log) and then call the respondent to resume the interview. 33 5. Guide for data managers Data managers are an integral part of the implementation team. An important aspect of the assessment is ensuring rapid data analysis and creation of data products, such as chartbooks and dashboards, to facilitate the timely use of data during the pandemic. Data managers supervise and adjust the data systems used in the assessment, ensuring that analyses are accurate and timely and that chartbooks (the automated figures and tables that are output in Excel) and the assessment dashboard are updated daily, thus providing results in near real time (Fig. 9). At the end of the data collection period, the assessment results should be available for immediate review by the survey manager and technical staff. Data managers should support the review and update the analyses and results, as needed (Box 2). Figure 9. Data and analysis flow for assessments of frontline service readiness during the COVID-19 pandemic A data manager must join the implementation team during the planning and preparation processes for the assessment (Table 7, Step 3). Specifically, during the process of adapting modules to the country’s context, the data manager will revise the standard analysis code and code for the chartbooks to reflect the adaptations. At the end of each day of the data collection period, the data manager will update the chartbooks and run the quality check. The technical assistance package for data managers includes: • standard modules; • data management and analysis codes for two statistical programmes (Stata and RStudio); • mock data for practice testing; • standard automated output of figures and tables in Excel (the chartbooks). The following sections describe the standard tools that will be used by data managers to ensure the timely flow of data for the assessment (Fig. 9). 34 Box 2. Roles and responsibilities of the data manager for assessments of frontline service readiness during the COVID-19 pandemic  Update and test the analysis code and update and test the chartbooks (automated outputs of figures and tables in Excel)  Daily: download and clean data, and produce field-check tables  Create an analysis data set  Calculate estimates for key indicators  Update chartbooks and compile results from the assessment  With the survey manager, interpret the results  Assist the survey manager in disseminating the results and using the data appropriately to achieve necessary changes 5.1 Data download, cleaning, management and analysis Code for downloading, cleaning and analysing the data is provided for two statistical software packages: Stata and RStudio. Both produce the same outputs, although there are minor differences in the order of some steps, and data managers should choose one package, depending on their experience and preferences. Table 12 presents the table of contents for the standard analysis code and notes to facilitate adaptation by data managers. In summary, the analysis code will: • import and clean data sets downloaded from LimeSurvey; • create field-check tables to monitor survey progress and data quality; • create indicator variables in the data set; and • calculate estimates for indicators In addition, to facilitate further data analysis and transparency, the analysis code exports three different types of data to Excel chartbooks (Fig. 9; see also Section 5.3): • a de-identified raw data set at the facility or respondent level, downloaded from LimeSurvey (raw data in Fig. 9); • a de-identified, cleaned data set with additional analytical variables at the facility or respondent level (clean data in Fig. 9); • estimates of indicators (i.e. results at the overall and analysis domain levels) (summary indicator data in Fig. 9). This summary data set (also referred to as the purple tab in the chartbook, Section 5.3) is also used for the dashboard (Section 5.4). 35 Table 12. Contents of the analysis code and possible country-specific adaptations for assessments of frontline service readiness during the COVID-19 pandemic Section Adaptation A. Setting Required actions • Set working directories in accordance with the data manager’s local computer settings • Set local macros for the country’s name, and survey round, year and month, and the LimeSurvey identification number B. Import and drop duplicate cases B.1. Import raw data from LimeSurvey None B.2. Export and save the data daily in CSV format with the date None B.3. Export the data to the chartbook None B.4. Assess and drop duplicate cases None C. Data cleaning C.1. Change variable names to lowercase None C.2. Change variable names to make them coding friendly None C.3. Assess nonnumeric variables and, as relevant, change the format from string or character to numeric None required unless new country-specific questions are added C.4. Recode yes/no and yes/no/NA answers None required unless new country-specific questions are added C.5. Label values None required unless new country-specific questions are added D. Create field-check tables None required, but more field-check tables can be added as needed E. Create analytical variables E.1. Country-specific code: local Required actions • Set local macro for survey implementation and Section 1 of all modules E.2. Construct analysis variables Required actions The data manager must review this section carefully and revise it based on any country-specific changes made to the modules, if needed COVID-19 case management capacities module • Standard questions: review and update to reflect any changes made to response options • Include new country-specific questions, if any Continuity of essential health services module • Update the number of staff and COVID infection variables 36 • Other standard questions: review and update to reflect any changes made to response options • Include new country-specific questions, if any Community needs, perceptions and demand module • Standard questions: review and update to reflect any changes made to response options • Include new country-specific questions, if any E.3. Merge with sampling weighta Required actions • Choose one of the two options based on the sample design used in the country E.4. Export clean respondent-level data to the chartbook None F. Create and export indicator estimate data F.1. Calculate estimates None F.2. Export indicator estimate data to the chartbook None CSV: comma-separated values. a This step is performed only if relevant based on the sample design for facility assessment. It is not relevant for interviews with key community informants. 5.2 Field-check tables The analysis code also creates an Excel file with cumulative summary information about the interviews, known as field-check tables (Table 12). The standard field-check tables include the: • total number of facilities and respondents, by interview result; • length of each interview; • percentage of interviews that have missing responses for selected sets of variables. The field-check tables allow data managers to monitor the progress of the survey and interviewers’ performance as a group over time. For example, for the group of questions assessing the availability of PPE, the percentage of missing responses would ideally be zero. The data manager and survey manager should review the results together, and the survey manager should use the information to provide feedback to the interviewers. Although data collection is designed to be completed during a short period of time, it is important to monitor and support interviewers as much as possible and help them improve their performance if necessary. 5.3 Chartbooks During each day of data collection, the data should be analysed, with updated results presented in Excel chartbooks. Chartbooks provide the most comprehensive presentation of data generated from the assessment (Fig. 10). Their Excel format allows most users to review and explore the results without needing to use other programmes. Chartbooks also include the de-identified data set for those who want to pursue further analysis beyond estimating predefined indicators (facility-level cleaned data in Fig. 11). 37 Fig. 10. Example of a chartbook cover page 38 Fig. 11. Different worksheets in a chartbook: required updates and automatically generated figures and tables Fig. 11 shows that there are three areas that require user modifications: the instructions (red text); weighing, if applicable (orange tab); and country-specific information pertaining to the cover, acknowledgements and methodology (dark grey tabs). The indicator estimates (purple tab) are automatically updated by the Stata or RStudio code based on the comma-separated values (CSV) file that is downloaded daily. Finally, all standard tables and figures in the light grey tabs are automatically updated based on the indicator estimates (purple tab, exported from the data analysis code). Importantly, tables and figures in the light grey tabs can be modified to meet the specific needs of readers in different contexts. Fig. 12 displays a sample summary results tab, which is automatically updated. Each section of the survey maps to a separate summary tab, and each summary has five parts: (1) prepopulated information with details about the module and section, including the key tracer indicators; (2) figures for each of the key tracer indicators. These figures are generated from the data tables in part 4 (Fig. 12) and are automatically updated daily based on the data that are downloaded; (3) key results. These need to be updated manually by the data manager or survey manager based on the results seen in the figures; (4) tables that summarize the data and populate the figures. These are automatically updated daily based on the data that is downloaded; (5) detailed results. These results are automatically updated daily based on the data that is downloaded. Users can modify or add new figures as needed. 39 Fig. 12. An example of the five parts of a summary results tab in a chartbook 40 5.4 Dashboard The dashboard presents information about select key indicators from the assessment (Fig. 13). The underlying database for the dashboard is the summary estimate of indicator data (a CSV file, as described in Fig. 9). Thus, the results for key indicators are identical between the chartbooks and the dashboard. However, customization of the dashboard is limited, in contrast to the chartbooks. When users visit the dashboard, they see the most up to date summary of indicator estimates that have been generated and uploaded to a designated server and folder by the data manager. Footnotes in the dashboard indicate when the database was most recently updated. The dashboard is currently hosted on a WHO server, but it can be hosted on a country’s server, after installation of the bundles required for visualization. Fig. 13. Example of a dashboard generated for assessments of frontline service readiness during the COVID-19 pandemic 41 6. Data use and action plan for system strengthening After analysing the data from the frontline services readiness assessment, the findings should be put to use to address weaknesses in the health system. An action plan for system strengthening should be developed to prioritize and guide appropriate interventions. This section discusses general guidance on developing an action plan, 2 and it can be adapted in the context of the pandemic and repeated, rapid telephone surveys. 6.1 Purpose The purpose of the action plan for system strengthening is to outline the steps and inputs required to address the causes of gaps in health care found during the readiness assessment. Needs should be identified and prioritized, and interventions should be developed and costed to address those needs. Mechanisms for monitoring and coordination should be identified or created to ensure that the interventions are implemented in good time and within the allotted budget. The goal of the plan is to improve health service delivery and the performance of the health system. To meet these goals, the plan should provide specific and practical actions that when implemented will improve the delivery of health services. 6.1.1 Best practices for developing an action plan for system strengthening The best practices for developing an action plan are summarized below. • The development and implementation of the action plan for system strengthening should be led by the MoH or other government ministry responsible for managing the health system. • The improvement plan should be developed in collaboration with important stakeholders, such as donors, partners and nongovernmental organizations, to ensure that consensus and stakeholder buy-in are achieved. • The activities and interventions in the improvement plan should be relevant to the country’s context and address the priorities of the country or organization, including its subunits. • The interventions should build on what already exists and be feasible and appropriate for the context of the health system and its workforce. • The activities and interventions should promote and facilitate the sustainability of the health system, so that the system can satisfy the needs of the present and evolve as those needs change. 6.2 Action plan for system strengthening 6.2.1 Engage stakeholders To ensure optimum development and implementation of the action plan for system strengthening, important stakeholders should be encouraged to participate. Being part of the development process will ensure that stakeholders are invested in the success of the plan and help ensure their continued support and buy-in. The interests, requirements and priorities of stakeholders should be understood, as well as their capacity to commit resources to ensure success. There are likely many stakeholders, and not all will need to be involved. Sometimes having too many stakeholders can inhibit the development of a responsive plan. Know the stakeholders and choose them strategically – that is, choose those who give the action plan the best chances for success. Stakeholders can help advocate for necessary changes and mobilize resources to assist with implementation. A stakeholder engagement matrix can help identify organizations and individuals who have an interest in improving the information system. The matrix (Table 13) can help identify the organizations, people and groups 2 This section is adapted from reference 4. 42 who are the stakeholders in the process of health system strengthening, as contributors, influencers or beneficiaries. The matrix is a structured way to define the roles that stakeholders will play in the activity and assess the resources they can bring to it. It also provides a framework for assessing stakeholders’ interests, knowledge, positions, alliances, resources, power and importance. For example, who will resist the initiative? Who will support it? What are their reasons? The matrix helps assess which stakeholders to include in the process by determining their relative importance. Identifying and engaging with relevant stakeholders contributes to developing an improvement plan that meets everybody’s expectations and needs. 43 Table 13. Example of a stakeholder engagement matrix that can be used to evaluate which stakeholders should be included in the process of developing a plan for health system strengthening Name of stakeholder organization, group or individual (Are they national, regional or local?) Stakeholder description (What is their primary purpose, affiliation, funding?) Potential role in the issue or activity (What is their interest in the activity?) Level of knowledge of the issue (What are their specific areas of expertise?) Level of commitment (Will they support or oppose the activity, to what extent, and why?) Resources available (Could they provide staff, volunteers, money, technology, information, influence?) Constraints (What are their limitations? Are funds needed to participate? Is there a lack of personnel or political or other barriers?) Engagement strategy (How will this stakeholder be engaged in the activity?) Follow-up strategy (What are the plans for feedback or continued involvement?) Government sector Political sector Commercial sector Nongovernmental sector Other civil society organization 44 Donors and partners 45 6.2.2 Review the results of the assessment of service readiness Conducting a formal review of the results of the assessment is a good way to understand and prioritize how to address any problems with data quality that were identified, discuss potential solutions, prioritize recommendations and prepare a realistic action plan. To encourage and promote ownership of the assessment results, it is recommended to begin by conducting an internal review with the MoH, followed by a review in a workshop setting with a broader group of participants. In these phases of reviewing and analysing the assessment’s results, ensure that the designated participants from the MoH and other stakeholder organizations have the ability to analyse the the findings of the assessment and are knowledgeable about the country’s context and health system and, therefore, have the ability to recommend appropriate actions for system strengthening. To ensure a productive review workshop, the assessment report, charts, graphs and other reading materials (the information products from the assessment) should be given to the participants in advance, so they can prepare. To achieve the workshop’s expected outcomes, facilitators who have the skills to keep participants focused will also need to be identified. The review workshop can be combined with the action-planning phase. If action planning is part of the workshop, be sure that the participants have the authority to make decisions. Alternatively, in the first part of the workshop, health system or health programme experts can review and validate the quality and relevance of the assessment results and prepare summaries and presentations for the decision-makers. In the second part of the workshop, the relevant decision-makers can join the health system experts, be briefed on the results and the recommendations, and contribute to identifying actions and interventions that can be taken to address the findings. They can also help define the timelines, responsible persons and organizations, and required resources. For both the review and the formulation of relevant recommendations to be effective, it will be important to conduct discussions in small groups. The groups should have equal representation from the following categories of participants: • health policymakers; • health programme managers; • data managers and specialists in monitoring and evaluation; • health providers. Each group may be composed of and have tasks assigned according to: • health programme area (e.g. maternal health, child health, HIV/AIDS, tuberculosis, malaria); • level of the health system (e.g. national, regional, district, health facility or community health care); • type of health facility (e.g. hospital, health centre). A plenary session should follow the group discussions to enable all participants to provide feedback and input on all groups’ ideas and proposals and to allow participants to learn from one another. If the results review and the formulation of recommendations are conducted separately from the action- planning session, the results and recommendations should be disseminated to the relevant decision-makers to inform them and guide them in identifying the appropriate strategies and actions for strengthening the health system. 46 6.2.3 Action planning The process of developing an action plan for system strengthening follows the review and discussion of the assessment results and recommendations, as well as the identification and prioritization of strategies to achieve improved service delivery in the health system. Similar to the earlier part of the process, the planning process requires effective facilitation to ensure the development of an action plan that describes specific, measurable, achievable, relevant and time-bound (known as SMART) objectives and activities; the responsibility for the implementation of each activity should be assigned to a specific person or organization. 6.2.3.1 Prioritizing interventions for system strengthening When formulating recommendations and developing the action plan, it is important to prioritize those activities that will lead to the greatest improvement in service delivery with the resources available or for which resources can be mobilized. The sustainability of the interventions should also be considered when prioritizing activities for improving service delivery. Participants in the action planning session can use a prioritization matrix (Table 14) to score the proposed activities based on their expected impact on health service delivery and on the ability of the organization and stakeholders to implement the activities. The scores help prioritize the interventions that are the most feasible and likely to yield the greatest results. The prioritization exercise is conducted through a consensus process. Participants agree on the level of impact that each recommended intervention will have and the ability of the stakeholders to implement it while also considering the available resources (e.g. human, financial, information and communication technology). Participants can work in small groups to discuss and complete the matrix and then come together in a plenary session to produce one completed and mutually agreed matrix. The prioritization matrix is arranged with a scale for impact on the vertical axis and, on the horizontal axis, a scale for the ability to implement an intervention and the required level of investment (e.g. human and financial resources, effort and time). Each axis is divided into four scores: 1 represents the lowest score for the attribute and 4 represents the highest. The interventions with the most impact, that are the easiest to implement and that require minimal investment are put in cells in the the top right of the matrix, and the interventions with the least impact and that are least feasible (i.e. require a high level of human or financial resources or efforts) are put in the cells in the bottom left of the matrix. 47 Table 14. Example of an intervention prioritization matrix that can be used to rate activities proposed for health system strengthening High Impact Low Impact 4 3 2 1 1 2 3 4 HARD to implement (low ability) EASY (high ability) Depending on the context, using this matrix helps distinguish the relevant interventions that are easy or relatively easy to implement and that produce moderate to high impact from those that are less feasible or yield only a low impact. Once the interventions have been identified, they should be broken down into well-defined subactivities so that a person or organization can be assigned to be responsible for implementation and funding. Table 15 provides an example of how a main intervention can be split into subactivities that result in improvements in health service delivery. 6.2.3.2 Scheduling and budgeting activities The purpose of scheduling and budgeting is to elaborate the overall action plan for system strengthening, thereby providing a roadmap for the activities supporting each recommended intervention. Understanding the effort required to implement each recommended intervention allows participants in the action-planning workshop to split activities into steps and accurately estimate the resources and time required for implementation. Aligning the activities with the resources they require makes it possible to estimate the costs of efforts to improve health service delivery, determine how much time is required and the timetable for implementation. 48 Table 15. Example of a table showing how a primary intervention can be split into subactivities, for example, for scheduling and budgeting Priority actions Objective Activities Short term Medium term Long term Responsible entity Supporting partner Budget Intervention 1 Action 1 1 Activity 1 X Activity 2 X Activity 3 X Action 2 2 X X X 6.2.4 Monitoring and follow up Monitoring and evaluation helps measure performance and assess the impact of different strategies, interventions and inputs on the efforts to improve health service delivery. The results of monitoring and evaluation contribute to the learning experience and help decision-makers improve the interventions. The action plan itself serves as a monitoring tool that can be used to follow up the implementation of the interventions and activities it stipulates. Moreover, the use of supervisory checklists can help track the progress made in improving health service delivery at the facility level. For evaluation purposes, the next scheduled implementation of the service readiness assessment can be used to measure the success of interventions. Regularly reviewing the implementation of the action plan and monitoring findings helps stakeholders identify when mid-course corrections are needed. 49 7. References 1. Maintaining essential health services: operational guidance for the COVID-19 context. Geneva: World Health Organization; 2020 (interim guidance, 1 June 2020; https://apps.who.int/iris/handle/10665/332240, accessed 15 March 2021). 2. Service availability and readiness assessment (SARA): an annual monitoring system for service delivery. Reference manual, version 2.2, revised July 2015. Geneva:World Health Organization; 2015 (https://apps.who.int/iris/handle/10665/149025, accessed 15 March 2021). 3. Master facility list resource package: guidance for countries wanting to strengthen their master facility list. Geneva: World Health Organization; 2018 (https://apps.who.int/iris/handle/10665/326848, accessed 15 March 2021). 4. Performance of Routine Information System Management (PRISM) user’s kit: moving from assessment to action. Chapel Hill (NC): MEASURE Evaluation; 2018 (https://www.measureevaluation.org/resources/tools/health-information-systems/prism/performance-of- routine-information-system-management-prism-users-kit-moving-from-assessment-to-action/view, accessed 15 March 2021). 50 8. Annex 1. Suite of health service capacity assessment modules On 30 January 2020, the Director-General of the World Health Organization (WHO), declared the COVID-19 outbreak to be a global public health emergency of international concern under the International Health Regulations (2005). Following the spread of COVID-19 cases in many countries across continents, COVID-19 was characterized as a pandemic on 11 March 2020 by the Director-General, upon the advice of the International Health Regulations (2005) Emergency Committee. In response to this situation, the Suite of health service capacity assessments in the context of the COVID-19 pandemic has been developed to support rapid and accurate assessments of the current, surge and future capacities of health facilities throughout the different phases of the pandemic (Table A1.1). Table A1.1. The WHO suite of tools for assessing health service capacity in the context of the COVID-19 pandemic Module or tool Use Health facility and community assessment tools (core modules) 1 COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other health products -facility assessment tool To assess present and surge capacities for treating patients with COVID-19 in health facilities 2 Continuity of essential health services: facility assessment tool To assess health facility and health workforce capacities to continue to safely provide essential health services 3 Community needs, perceptions and demand: community assessment tool To assess community needs, changes in care-seeking behaviours and barriers to accessing care In-depth assessment tools and modules 4 Rapid hospital readiness checklist To assess overall hospital readiness in order to identify priority actions to prepare for, be ready for and respond to COVID-19 5 Biomedical equipment for COVID-19 case management – inventory tool To conduct an in-depth facility inventory of biomedical equipment reallocation and procurement, and planning measures for COVID-19 6 Ensuring a safe environment for patients and staff in COVID-19 health care facilities To assess the structural capacities of health facilities to allow for safe service delivery and enable planning for surge capacity 7 Infection prevention and control health care facility response for COVID-19 To assess infection prevention and control capacities to respond to COVID-19 51 9. Annex 2. Interpreting results from a stratified sample Using a stratified sampling approach will ensure that sentinel facilities are representative for each domain as well as for the country at baseline, after adjustment for sampling weight. Observed trends from the sentinel sites may be interpreted to be representative of all facilities in the country, depending on the acceptable level of precision. It should be noted, however, that even at the national level the total sample size can be too small and the margin of error too large to make statistical inferences about trends unless the change is substantial. Table A2.1 shows examples of margins of error by estimates for key indicators, desired precision and sample size. • With a sample size of 100, if the key indicator is estimated to be 20% and if a 95% confidence interval is applied, the true value is between 12.4% (20 ­ 7.6) and 27.6% (20 + 7.6). • With a sample size of 100, if the key indicator is estimated to be 50% and if a 95% confidence interval is applied, the true value is between 40.4% (50 ­ 9.6) and 59.6% (50 + 9.6). Table A2.1. Margin of error to calculate 95% and 90% confidence intervals, by estimates for key indicators and sample size, assuming a 95% response rate Estimate for key indicator (%) Margin of error, by confidence interval and sample size For 95% confidence interval For 90% confidence interval Sample size 50 Sample size 100 Sample size 150 Sample size 50 Sample size 100 Sample size 150 10 8.1 5.7 4.7 6.6 4.7 3.8 20 10.8 7.6 6.2 8.8 6.2 5.1 30 12.4 8.8 7.1 10.1 7.1 5.8 40 13.2 9.4 7.6 10.8 7.6 6.2 50 13.5 9.6 7.8 11.0 7.8 6.4 60 13.2 9.4 7.6 10.8 7.6 6.2 70 12.4 8.8 7.1 10.1 7.1 5.8 80 10.8 7.6 6.2 8.8 6.2 5.1 90 8.1 5.7 4.7 6.6 4.7 3.8