WEB ANNEX G SAMPLING, ABSTRACTION AND REPORTING GUIDANCE FOR EARLY WARNING INDICATORS OF HIV DRUG RESISTANCE CONSOLIDATED GUIDELINES ON PERSON-CENTRED HIV STRATEGIC INFORMATION STRENGTHENING ROUTINE DATA FOR IMPACT WHO/UCN/HHS/SIA/2022.11 © World Health Organization 2022 Some rights reserved. This work is available under the CC BY-NC-SA 3.0 IGO licence. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. EWI sampling guidance 3 1.1 Primary sampling 3 1.2 Secondary sampling 4 ART clinic-level EWI reporting 7 2.1 National aggregate prevalence estimated when all clinics report EWI or a random sample of clinics reports EWI 10 2.2 National aggregate prevalence estimated when a combination of randomly sampled clinics and conveniently sampled clinics report EWIs 10 Data abstraction from ART clinics 11 Data quality assessment 12 CONTENTS
31 EWI SAMPLING GUIDANCE This annex presents a method for expanding the monitoring of early warning indicators (EWIs) of HIV drug resistance or health facility-based EWI indicator monitoring through random sampling of ART clinics. This approach can provide representative data while monitoring is being progressively scaled up to all clinics in a country. Use of random sampling allows countries to calculate an aggregated national prevalence estimate for each EWI. In addition, this method can incorporate information from clinics with conveniently available data without sacrificing representativeness. In this annex primary sampling refers to the sampling of clinics, and secondary sampling refers to the sampling of patients within a clinic. 1.1 Primary sampling Ideally, quality of care indicators including EWIs of HIV drug resistance should be reported from all ART clinics within a country. In countries where it is not possible to report EWIs from all clinics, it is recommended that EWI reporting be progressively expanded, using a multiyear scale-up approach. Countries may start by sampling a fixed percentage of clinics and then annually expand EWI monitoring until all clinics report. For example, in Year 1 a country samples 20% of clinics. In Year 2, all clinics reporting in Year 1 plus an additional sample of another 20% of clinics report, for a total coverage of 40%. Each year, the country expands EWI coverage by an additional 20% of clinics (60% in Year 3; 80% in Year 4) until 100% coverage is achieved in Year 5. Countries can decide the rate of scale-up, in terms of the percentage of clinics added per year and total number of years. In expanding the inclusion of clinics reporting EWIs, clinics may be selected using a combination of convenience sampling and random sampling (primary sampling), and their results can be summarized in a nationally representative manner through the use of weighting. Convenience sampling may be used for clinics already reporting EWI data or for clinics with readily exploitable data. Clinics historically reporting EWIs should continue to report and, thus, should always be included in the sample. Similarly, countries may decide to include clinics with readily exploitable data, such as clinics with electronic medical records (EMR), because of the relatively low cost of data abstraction. While convenience sampling can be used for clinics such as those described above, it is recommended that the primary mode of clinic selection be random sampling because this promotes national representativeness. Representative sampling of clinics can be achieved by simple random sampling or stratified random sampling. If stratified random sampling is used, it is recommended not to use more than one stratum, for example, district or province. An example of Country Z follows. In this example in Year 1 Country Z will monitor EWIs at five clinics with readily exploitable data, Clinics U, V, W, X and Y, as well as Clinics A, I, P and R (Fig. G1). In Year 2 the country continues to monitor EWIs at Clinics A, K, P, R (randomly sampled in Year 1) and U, V, W, X and Y (conveniently sampled). In addition, the country randomly samples four more clinics to expand EWI uptake by an additional 20%, or roughly half of all clinics. To identify these four additional clinics, Country Z must create a new systematic sampling table. Country Z must repeat the same process described above, excluding clinics that are already reporting EWIs either because they were conveniently sampled (Clinics U through Y) or because they were randomly sampled in the previous year (Clinics A, K, P and R). In this example, in the second year Country Z randomly samples Clinics G, J, M and T. Each year, the country expands EWI monitoring to additional clinics until, at Year 5, all clinics are reporting EWIs. After year 5 all clinics in the country continue to report EWIs annually. 4Fig G1 Example of representative scale-up of EWIs using random clinic sampling Sampling of ART clinics A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y Year 1 – Clinics with readily exploitable data Year 3 – sample four additional clinics (identified in blue) Year 1 – sample four clinics (identified in blue) Year 4 – sample four additional clinics (identified in blue) Year 2 – sample four additional clinics (identified in blue) Year 5 – sample four additional clinics (identified in blue) Source: Jacob et al., 2020 (44) 1.2 Secondary sampling All EWIs except ARV medicine stock-out (ART.12) rely on the collection of patient-level data from clinics. Abstracting data from a census of patient-level data (that is, data for all eligible patients) is strongly preferred. Where a census is not possible, sampling of patient records (for example, systematic sampling or simple random sampling) at an individual clinic (secondary sampling) achieves a result generalizable to the entire eligible population of interest at the clinic. To determine the necessary sample size for secondary sampling, the clinic must determine the sizes of the eligible patient populations. Note that the eligible patient population is not the same for all EWIs. For example, the eligible patient population for “ART adherence proxy” (on- time ARV drug pick-up) (paediatric) is the annual total number of eligible paediatric patients in care or receiving ART at the clinic. For “total attrition from ART” (adult), the eligible patient population is the number of people living with HIV reported to be receiving ART at the end of the last reporting period plus those who newly initiating ART during the current reporting period, with those known to have transferred out to another facility removed from the total number eligible. For “VL suppression” (adult), the eligible patient population is the total number of adult patients with HIV receiving ART (for at least six months) who have a VL test with result available. 5Clinics can use Table G1 to determine the appropriate sample size based on the eligible patient population for each EWI. Sample size calculations are presented in Box G1. Note that within the same clinic, different indicators may require different sample sizes. The sample sizes shown in Table G1 are calculated to achieve 95% confidence intervals of ±7% for clinic-specific results. Once the sample size is determined for a given EWI at a particular clinic, patient records are randomly sampled (either systematic sampling or simple random sampling) until the required sample size is reached. Patient records known to have missing or incomplete data should not be excluded. Table G1 Sample size required to estimate EWI and achieve a 95% confidence interval of ±7% at a reporting clinic Annual number of “eligible patients” at the clinic Number to be sampled at the clinic 1–75 All 76–110 75 111–199 100 200–250 110 251–299 120 300–350 130 351–400 135 401–450 140 451–550 145 551–700 155 701–850 160 851–1600 175 1601–2150 180 2151–4340 200 4341–5670 210 5671–1000 215 >1000 220 6Box G1 Sample size calculations for monitoring EWIs The formula used to calculate the sample size for monitoring WHO HIVDR EWI has two parts. The first equation calculates a sample size for large populations. The second equation applies a finite population correction factor. This formula produces samples that will allow a 95% confidence interval of ±7% if the true proportion of patients meeting the target for the indicator is 50%. Equation 1 (large population sample size): n0 = Z2*p*(1−p) / e2 where: Z = 1.96 p = 0.5 (that is, 50% is assumed as the “true prevalence” of the proportion of patients meeting the target, because this gives the most conservative estimate of the sample size required) e = precision = 0.07 (based on the confidence interval of ±7%) Equation 2 (finite population correction factor): n = n0 / (1 + ((n0-1)/N)) where: N = population size of the eligible individuals at the clinic WHO provides an EWI data abstraction tool in MS Excel format to facilitate data abstraction and automatically assign the appropriate classification. Note, however, that, with the WHO tool, a census of all patients on ART is used to calculate all indicators. The tool keeps track of complete entries and reports a grey score if ≥30% of information is missing. The Excel tool is available at the WHO HIVDR website: https://www.who.int/teams/global- hiv-hepatitis-and-stis-programmes/hiv/treatment/hiv-drug-resistance/prevention. 72 ART CLINIC-LEVEL EWI REPORTING After data abstraction is complete, each clinic is responsible for calculating the EWIs and assigning the appropriate classification. A point prevalence (numerator/denominator) is estimated for each EWI, and this point prevalence is compared with the EWI-specific performance strata to determine the appropriate classification (red, amber or green). A grey classification is used if 30% or more of the data required for a calculation is missing. It is not necessary to calculate a 95% confidence interval for the EWI in order to make a classification. The classifications are illustrated in the example clinic score card (Fig. G2). Fig G2 Example of a clinic EWI score card Clinic: National clinic # 1 Scorea ART.2 Total attrition from ART 10% ART.3 VL suppression 95%b ART.6 VL testing coverage 85% ART.8 Appropriate second VL test 92% ART.12 ARV medicine stock-outs >0% ART.13 ART adherence proxy (ARV drug refills or on time-pill pick up) 82%b ART.14 Appropriate switch to second-line ART 100% a In this example the respective point prevalence estimate corresponding to the colour assigned to each EWI is presented in the score box. b While results for ART.2 are representative of the clinic sampled, the VL suppression indicator reflects only those who had a VL test result. In this example only 50% of eligible patients had a VL test result; thus, the prevalence of VL suppression cannot be generalized to the entire eligible population and this indicator (ART.3) is reported in grey. In the case of the VL suppression indicator, the proportion of available data during the reporting period is captured and reported as VL testing coverage (ART.6). For all other EWIs, if ≥30% of data are unavailable, a grey score is assigned, and no prevalence estimate is reported. At the national level, it is recommended that countries report the fractions of clinics monitored that achieve green, amber, red and grey classifications for each indicator and for both adult and paediatric patient populations. Fig. G3 presents an example of a table with results reported as the percentage of clinics monitored achieving a specific colour score. 8Fig G3 Fraction of clinics monitored in a given year achieving a specific performance stratum – example based on 100 clinics Indicator number Indicator Green Amber Red Grey ART.2 Total attrition from ART Target: <15% 15–25% >25% Example: 50/100 20/100 10/100 20/100 ART.3 VL suppression Target: ≥90% 80–<90% <80% Example: 60/100 20/100 10/100 10/100 ART.6 VL testing coverage Target: >95% 85–95% <85% Example: 80/100 10/100 10/100 NA ART.8 Appropriate second VL test Target: ≥90% NA <90% Example: 63/100 30/100 7/100 ART.12 ARV medicine stock-outs Target: 0% NA >0% Example: 92/100 8/100 ART.13 ART adherence proxy (ARV drug refills) Target: >90 80–90% <80% Example: 74/100 14/100 10/100 2/100 ART.14 Appropriate switch to second- line ART Target: 100% NA <100% Example: 80/100 20/100 NA = not applicable. For the EWI VL testing coverage (ART.6), a grey score is not possible as this indicator classifies missing information. Classifications of Appropriate second VL test (ART.8), ARV medicines stock-outs (ART.12), and appropriate switch to second-line ART (ART.14) are binary, and no amber classification exists. Providing strata of performance on a score card helps programme managers to identify areas of greatest need and also, over time, to grossly monitor for improvement or decline across these indicators. The score card presents the results to health ministries and stakeholders in a manner that is clear and easily interpreted. Additionally, the card will reflect whether any of the indicators cannot be measured at a specific ART clinic. Fig. G4 presents an example of an at-a-glance assessment of clinic performance at the national level. Fig G4 National at-a-glance assessment of ART clinic performance by EWI Clinic Indicator Green (good performance) Amber (fair performance) Red (poor performance) No classification due to missing data ART.2 Total attrition from ART <15% 50/100 15–25% 20/100 >25% 10/100 20/100 ART.3 VL suppression ≥90% 70/100 80–<90% 20/100 <80% 10/100 10/100 ART.6 VL testing coverage >95% 80/100 85–95% 10/100 <85% 10/100 NA ART.8 Appropriate second VL test ≥90% 63/100 NA <90% 30/100 7/100 ART.12 ARV medicine stock-outs 0% 92/100 NA >0% 8/100 ART.13 ART adherence proxy (ARV drug refills) >90 74/100 80–90% 14/100 <80% 10/100 2/100 ART.14 Appropriate switch to second-line ART 100% 80/100 NA <100% 20/100 NA = not applicable. For the EWI VL testing coverage (ART.6), a grey score is not possible as this indicator classifies missing information. Classifications of Appropriate second VL test (ART.8), ARV medicines stock-outs (ART.12), and appropriate switch to second-line ART (ART.14) are binary, and no amber classification exists. 9In addition to providing individual clinic-level classifications, EWI monitoring can also provide information on ART programmatic function at the national level. ART programme managers may wish to estimate the average prevalence of each indicator across the country as a summary measure of overall programme performance. Nationally representative prevalence estimates for each EWI can be calculated by data aggregation and weighting. To aggregate results, all sampled clinics must report a measure of relative size for each EWI (except “pharmacy stock-outs”). Table G2 defines population sizes, which are additional information required for national aggregate weighting. Data analysis must be performed in Stata or another statistical programme that can accurately manage two-stage clustered survey data. The drug stock-out indicator is not weighted. The interpretation of the aggregated values for all EWIs except “ARV medicine stock-out” is the proportion of the patient population in the country with the relevant outcome (for example, adults receiving ART with VL suppression among those with an available VL test result during the reporting period). The interpretation of the aggregated value for the “ARV medicine stock- out” EWI is the average proportion of months with stock-outs of routinely dispensed ARV drugs among clinics in the country during the reporting period. Table G2 provides information required for calculation of national weighted estimates for each indicator. Weighting is performed unless a census of all patients from all clinics is used to calculate the corresponding prevalence. Table G2 Clinic-level information required for national aggregate weighting Indicator Denominator definition Total attrition from ART (ART.2) The total number of patients (adults or paediatic) at the clinic reported to be receiving ART at the end of the last reporting period and/or newly initiating ART during the current reporting period. This number is a count of the total number of records (adult or paediatric) that are eligible to be in the denominator and excludes those known to have transferred out on ART to a different clinic. VL suppression (ART.3) Number of patients (adult or paediatric) receiving ART for at least six months and who received a VL test with available result VL coverage (ART.6) Number of patients (adult or paediatric) who by national policy should have received a VL test during the reporting period. Generally, at least one VL per year is anticipated. Thus, this number is the number of people receiving ART at the end of the prior reporting period plus new ART initiators during the reporting period who have been on ART for at least six months at the time of indicator monitoring. Appropriate second VL test (ART.8) Number of patients with a VL test result available and whose test result showed VL >1000 copies/mL during the reporting period ARV medicine stock-outs (ART.12) None; this EWI is not aggregated but may contribute to national-level ARV stock-out indicator ART.12. ART adherence proxy (ARV drug refills, also known as on-time pill pick up) (ART.13) Total number of patients (adult or paediatric) on ART, by clinic Appropriate switch to second-line ART (ART.14) Number of patients with viral non-suppression, defined as two VL test results >1000 copies/mL during the reporting period for individuals receiving non-NNRTI-based ART and one VL >1000 copies/mL for individuals receiving NNRTI-based ART 10 2.1 National aggregate prevalence estimated when all clinics report EWI or a random sample of clinics reports EWI If all clinics report EWIs or if random sampling is used to select clinics for EWI scale-up, results can be aggregated across sites to generate nationally representative statistics. When aggregating, clinics with greater patient burdens are weighted more heavily than smaller clinics. For each EWI a country can calculate the aggregate point prevalence and 95% confidence interval. An excessive amount of missing (grey) data may complicate or prohibit the interpretation of the aggregated point prevalence. As aggregate analysis will be used for benchmarking and reporting, it is important that results are representative of the respective eligible population. If less than 70% of data are available for the nationally eligible population monitored by a specific indicator, aggregation should not be performed. 2.2 National aggregate prevalence estimated when a combination of randomly sampled clinics and conveniently sampled clinics report EWIs Estimating a nationally representative point prevalence using a combination of randomly sampled clinics and conveniently sampled clinics can be achieved through appropriate weighting. As long as random sampling is used to select a portion of the clinics, the aggregated point prevalence will be nationally representative even if some clinics are sampled by convenience. As an excessive amount of missing (grey) data may complicate or prohibit the interpretation of aggregate point prevalence, it is advised to ensure that results will be representative of the respective eligible population. As noted, if <70% of data are available for the nationally eligible population monitored by a specific indicator, aggregation should not be performed. A flowchart, Fig. G5, is designed to aid in assessing the feasibility of aggregation at the national level. 11 Fig G5 Flow diagram to assess feasibility of data aggregation at the national level 1. All clinics in the country included in EWI monitoring Aggregated EWIs will be nationally representativea Data can be readily aggregated across clinics 2. A subset of clinics included by random sampling Aggregated EWIs will be nationally representativea Data can be aggregated across clinics using weighting to account for clinic sizes 3. A subset of clinics included by random sampling + by convenience sampling Aggregated EWIs will be nationally representativea Data from randomly sampled clinics can be aggregated with data from conviently sampled clinics using weighting 4. Only conveniently sampled clinics included Aggregated EWIs will NOT be nationally representativeb Data may be biased if excluded clinics are different from included clinics a Aggregated EWIs should be reported only if data availability exceeds the recommended 70% threshold. b If only conveniently sampled clinics are included, data may be considered nationally representative if conveniently sampled clinics represent >70% of the eligible patient population for each indicator. 3 DATA ABSTRACTION FROM ART CLINICS Paper-based medical records. If paper-based records are in place, clinic staff trained in the national EWI monitoring strategy should abstract data at their respective sites. Generally, data are abstracted retrospectively, once per year. Whenever possible, countries should combine EWI data abstraction with other indicators and patient monitoring activities taking place in country. EWI monitoring may also be used as, or combined with, a quality assurance assessment of record-keeping at ART clinics. For the purpose of manual data abstraction, an Excel-based data abstraction tool is available on the HIVDR webpage: https://www.who.int/ teams/global-hiv-hepatitis-and-stis-programmes/hiv/treatment/hiv-drug-resistance/prevention. Electronic medical records. If EMRs are in place, experts from the national programme should guide a programme to abstract data for EWI monitoring. Generally, it is not feasible to obtain EWI information from summary reports already produced by electronic record-keeping systems; feasibility may be limited by varying definitions of an indicator or varying methods of applying a definition. If EMRs are used to produce EWIs, validation procedures that use abstraction from paper records should be set up. When electronic query programmes are written, they must also keep track of available data and classify clinics with >30% missing data for a particular indicator as “grey”. Electronic query programmes must also keep track of the overall proportion of missing data for all clinics for the purposes of assessing the feasibility of national aggregation of a specific EWI. 12 4 DATA QUALITY ASSESSMENT Data quality should be assessed throughout the EWI monitoring process. During the data abstraction process, data quality assessments provide critical information for ensuring that the correct data are abstracted in the appropriate way. After the data are analysed and reported, data quality assessment provides programme and clinic managers with the level of confidence that may be placed in the results, and how fit the data are for use in operations, planning and decision-making. Three elements of data quality should be considered: data reliability, data completeness and data consistency. Data reliability. This element assesses the reliability of data abstracted for each indicator. Assessing the quality early in the monitoring process will identify problems that can be addressed through additional support or training. Data completeness. Missing some data is expected. However, a large percentage of missing information in patients’ records at any clinic, for any EWI, presents challenges for achieving the required sample size and for interpretation of results. Monitoring of data completeness should occur during the data abstraction process, and clinics with <70% available data for any indicator should report a “grey” score for that indicator. A “grey” score is not punitive but rather signals that the clinic requires support in record-keeping before it can derive full benefit from indicator monitoring. Aggregation should not be performed unless ≥70% data are available from all sites sampled. Data consistency. Data consistency refers to consistency of patient information across different record systems within the same clinic. Clinic and pharmacy records are the primary sources of information used for EWI monitoring. Some clinics use both paper-based and electronic systems for clinic and pharmacy records. A records assessment process should be performed prior to data abstraction for EWI monitoring to evaluate the consistency of information across these different sources. This is a crucial step in assessing which sources provide the most accurate information. While not specifically designed to address the data used for EWI monitoring, data quality assessments identify strengths and weaknesses of existing record systems in participating clinics. Incompleteness and inconsistency of data generally indicate more systemic problems in record-keeping that should be addressed. Thus, the results of data quality assessments can inform changes that will improve patient monitoring systems and clinic practices. For more information, contact: World Health Organization Department of Global HIV, Hepatitis and STIs Programmes 20, avenue Appia 1211 Geneva 27 Switzerland Email: hiv-aids@who.int www.who.int/hiv
WEB ANNEX G SAMPLING, ABSTRACTION AND REPORTING GUIDANCE FOR EARLY WARNING INDICATORS OF HIV DRUG RESISTANCE CONSOLIDATED GUIDELINES ON PERSON-CENTRED HIV STRATEGIC INFORMATION STRENGTHENING ROUTINE DATA FOR IMPACT WHO/UCN/HHS/SIA/2022.11 © World Health Organization 2022 Some rights reserved. This work is available under the CC BY-NC-SA 3.0 IGO licence. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. EWI sampling guidance 3 1.1 Primary sampling 3 1.2 Secondary sampling 4 ART clinic-level EWI reporting 7 2.1 National aggregate prevalence estimated when all clinics report EWI or a random sample of clinics reports EWI 10 2.2 National aggregate prevalence estimated when a combination of randomly sampled clinics and conveniently sampled clinics report EWIs 10 Data abstraction from ART clinics 11 Data quality assessment 12 CONTENTS
31 EWI SAMPLING GUIDANCE This annex presents a method for expanding the monitoring of early warning indicators (EWIs) of HIV drug resistance or health facility-based EWI indicator monitoring through random sampling of ART clinics. This approach can provide representative data while monitoring is being progressively scaled up to all clinics in a country. Use of random sampling allows countries to calculate an aggregated national prevalence estimate for each EWI. In addition, this method can incorporate information from clinics with conveniently available data without sacrificing representativeness. In this annex primary sampling refers to the sampling of clinics, and secondary sampling refers to the sampling of patients within a clinic. 1.1 Primary sampling Ideally, quality of care indicators including EWIs of HIV drug resistance should be reported from all ART clinics within a country. In countries where it is not possible to report EWIs from all clinics, it is recommended that EWI reporting be progressively expanded, using a multiyear scale-up approach. Countries may start by sampling a fixed percentage of clinics and then annually expand EWI monitoring until all clinics report. For example, in Year 1 a country samples 20% of clinics. In Year 2, all clinics reporting in Year 1 plus an additional sample of another 20% of clinics report, for a total coverage of 40%. Each year, the country expands EWI coverage by an additional 20% of clinics (60% in Year 3; 80% in Year 4) until 100% coverage is achieved in Year 5. Countries can decide the rate of scale-up, in terms of the percentage of clinics added per year and total number of years. In expanding the inclusion of clinics reporting EWIs, clinics may be selected using a combination of convenience sampling and random sampling (primary sampling), and their results can be summarized in a nationally representative manner through the use of weighting. Convenience sampling may be used for clinics already reporting EWI data or for clinics with readily exploitable data. Clinics historically reporting EWIs should continue to report and, thus, should always be included in the sample. Similarly, countries may decide to include clinics with readily exploitable data, such as clinics with electronic medical records (EMR), because of the relatively low cost of data abstraction. While convenience sampling can be used for clinics such as those described above, it is recommended that the primary mode of clinic selection be random sampling because this promotes national representativeness. Representative sampling of clinics can be achieved by simple random sampling or stratified random sampling. If stratified random sampling is used, it is recommended not to use more than one stratum, for example, district or province. An example of Country Z follows. In this example in Year 1 Country Z will monitor EWIs at five clinics with readily exploitable data, Clinics U, V, W, X and Y, as well as Clinics A, I, P and R (Fig. G1). In Year 2 the country continues to monitor EWIs at Clinics A, K, P, R (randomly sampled in Year 1) and U, V, W, X and Y (conveniently sampled). In addition, the country randomly samples four more clinics to expand EWI uptake by an additional 20%, or roughly half of all clinics. To identify these four additional clinics, Country Z must create a new systematic sampling table. Country Z must repeat the same process described above, excluding clinics that are already reporting EWIs either because they were conveniently sampled (Clinics U through Y) or because they were randomly sampled in the previous year (Clinics A, K, P and R). In this example, in the second year Country Z randomly samples Clinics G, J, M and T. Each year, the country expands EWI monitoring to additional clinics until, at Year 5, all clinics are reporting EWIs. After year 5 all clinics in the country continue to report EWIs annually. 4Fig G1 Example of representative scale-up of EWIs using random clinic sampling Sampling of ART clinics A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y A B C D E F G H I J K L M N O P Q R S T U V W X Y Year 1 – Clinics with readily exploitable data Year 3 – sample four additional clinics (identified in blue) Year 1 – sample four clinics (identified in blue) Year 4 – sample four additional clinics (identified in blue) Year 2 – sample four additional clinics (identified in blue) Year 5 – sample four additional clinics (identified in blue) Source: Jacob et al., 2020 (44) 1.2 Secondary sampling All EWIs except ARV medicine stock-out (ART.12) rely on the collection of patient-level data from clinics. Abstracting data from a census of patient-level data (that is, data for all eligible patients) is strongly preferred. Where a census is not possible, sampling of patient records (for example, systematic sampling or simple random sampling) at an individual clinic (secondary sampling) achieves a result generalizable to the entire eligible population of interest at the clinic. To determine the necessary sample size for secondary sampling, the clinic must determine the sizes of the eligible patient populations. Note that the eligible patient population is not the same for all EWIs. For example, the eligible patient population for “ART adherence proxy” (on- time ARV drug pick-up) (paediatric) is the annual total number of eligible paediatric patients in care or receiving ART at the clinic. For “total attrition from ART” (adult), the eligible patient population is the number of people living with HIV reported to be receiving ART at the end of the last reporting period plus those who newly initiating ART during the current reporting period, with those known to have transferred out to another facility removed from the total number eligible. For “VL suppression” (adult), the eligible patient population is the total number of adult patients with HIV receiving ART (for at least six months) who have a VL test with result available. 5Clinics can use Table G1 to determine the appropriate sample size based on the eligible patient population for each EWI. Sample size calculations are presented in Box G1. Note that within the same clinic, different indicators may require different sample sizes. The sample sizes shown in Table G1 are calculated to achieve 95% confidence intervals of ±7% for clinic-specific results. Once the sample size is determined for a given EWI at a particular clinic, patient records are randomly sampled (either systematic sampling or simple random sampling) until the required sample size is reached. Patient records known to have missing or incomplete data should not be excluded. Table G1 Sample size required to estimate EWI and achieve a 95% confidence interval of ±7% at a reporting clinic Annual number of “eligible patients” at the clinic Number to be sampled at the clinic 1–75 All 76–110 75 111–199 100 200–250 110 251–299 120 300–350 130 351–400 135 401–450 140 451–550 145 551–700 155 701–850 160 851–1600 175 1601–2150 180 2151–4340 200 4341–5670 210 5671–1000 215 >1000 220 6Box G1 Sample size calculations for monitoring EWIs The formula used to calculate the sample size for monitoring WHO HIVDR EWI has two parts. The first equation calculates a sample size for large populations. The second equation applies a finite population correction factor. This formula produces samples that will allow a 95% confidence interval of ±7% if the true proportion of patients meeting the target for the indicator is 50%. Equation 1 (large population sample size): n0 = Z2*p*(1−p) / e2 where: Z = 1.96 p = 0.5 (that is, 50% is assumed as the “true prevalence” of the proportion of patients meeting the target, because this gives the most conservative estimate of the sample size required) e = precision = 0.07 (based on the confidence interval of ±7%) Equation 2 (finite population correction factor): n = n0 / (1 + ((n0-1)/N)) where: N = population size of the eligible individuals at the clinic WHO provides an EWI data abstraction tool in MS Excel format to facilitate data abstraction and automatically assign the appropriate classification. Note, however, that, with the WHO tool, a census of all patients on ART is used to calculate all indicators. The tool keeps track of complete entries and reports a grey score if ≥30% of information is missing. The Excel tool is available at the WHO HIVDR website: https://www.who.int/teams/global- hiv-hepatitis-and-stis-programmes/hiv/treatment/hiv-drug-resistance/prevention. 72 ART CLINIC-LEVEL EWI REPORTING After data abstraction is complete, each clinic is responsible for calculating the EWIs and assigning the appropriate classification. A point prevalence (numerator/denominator) is estimated for each EWI, and this point prevalence is compared with the EWI-specific performance strata to determine the appropriate classification (red, amber or green). A grey classification is used if 30% or more of the data required for a calculation is missing. It is not necessary to calculate a 95% confidence interval for the EWI in order to make a classification. The classifications are illustrated in the example clinic score card (Fig. G2). Fig G2 Example of a clinic EWI score card Clinic: National clinic # 1 Scorea ART.2 Total attrition from ART 10% ART.3 VL suppression 95%b ART.6 VL testing coverage 85% ART.8 Appropriate second VL test 92% ART.12 ARV medicine stock-outs >0% ART.13 ART adherence proxy (ARV drug refills or on time-pill pick up) 82%b ART.14 Appropriate switch to second-line ART 100% a In this example the respective point prevalence estimate corresponding to the colour assigned to each EWI is presented in the score box. b While results for ART.2 are representative of the clinic sampled, the VL suppression indicator reflects only those who had a VL test result. In this example only 50% of eligible patients had a VL test result; thus, the prevalence of VL suppression cannot be generalized to the entire eligible population and this indicator (ART.3) is reported in grey. In the case of the VL suppression indicator, the proportion of available data during the reporting period is captured and reported as VL testing coverage (ART.6). For all other EWIs, if ≥30% of data are unavailable, a grey score is assigned, and no prevalence estimate is reported. At the national level, it is recommended that countries report the fractions of clinics monitored that achieve green, amber, red and grey classifications for each indicator and for both adult and paediatric patient populations. Fig. G3 presents an example of a table with results reported as the percentage of clinics monitored achieving a specific colour score. 8Fig G3 Fraction of clinics monitored in a given year achieving a specific performance stratum – example based on 100 clinics Indicator number Indicator Green Amber Red Grey ART.2 Total attrition from ART Target: <15% 15–25% >25% Example: 50/100 20/100 10/100 20/100 ART.3 VL suppression Target: ≥90% 80–<90% <80% Example: 60/100 20/100 10/100 10/100 ART.6 VL testing coverage Target: >95% 85–95% <85% Example: 80/100 10/100 10/100 NA ART.8 Appropriate second VL test Target: ≥90% NA <90% Example: 63/100 30/100 7/100 ART.12 ARV medicine stock-outs Target: 0% NA >0% Example: 92/100 8/100 ART.13 ART adherence proxy (ARV drug refills) Target: >90 80–90% <80% Example: 74/100 14/100 10/100 2/100 ART.14 Appropriate switch to second- line ART Target: 100% NA <100% Example: 80/100 20/100 NA = not applicable. For the EWI VL testing coverage (ART.6), a grey score is not possible as this indicator classifies missing information. Classifications of Appropriate second VL test (ART.8), ARV medicines stock-outs (ART.12), and appropriate switch to second-line ART (ART.14) are binary, and no amber classification exists. Providing strata of performance on a score card helps programme managers to identify areas of greatest need and also, over time, to grossly monitor for improvement or decline across these indicators. The score card presents the results to health ministries and stakeholders in a manner that is clear and easily interpreted. Additionally, the card will reflect whether any of the indicators cannot be measured at a specific ART clinic. Fig. G4 presents an example of an at-a-glance assessment of clinic performance at the national level. Fig G4 National at-a-glance assessment of ART clinic performance by EWI Clinic Indicator Green (good performance) Amber (fair performance) Red (poor performance) No classification due to missing data ART.2 Total attrition from ART <15% 50/100 15–25% 20/100 >25% 10/100 20/100 ART.3 VL suppression ≥90% 70/100 80–<90% 20/100 <80% 10/100 10/100 ART.6 VL testing coverage >95% 80/100 85–95% 10/100 <85% 10/100 NA ART.8 Appropriate second VL test ≥90% 63/100 NA <90% 30/100 7/100 ART.12 ARV medicine stock-outs 0% 92/100 NA >0% 8/100 ART.13 ART adherence proxy (ARV drug refills) >90 74/100 80–90% 14/100 <80% 10/100 2/100 ART.14 Appropriate switch to second-line ART 100% 80/100 NA <100% 20/100 NA = not applicable. For the EWI VL testing coverage (ART.6), a grey score is not possible as this indicator classifies missing information. Classifications of Appropriate second VL test (ART.8), ARV medicines stock-outs (ART.12), and appropriate switch to second-line ART (ART.14) are binary, and no amber classification exists. 9In addition to providing individual clinic-level classifications, EWI monitoring can also provide information on ART programmatic function at the national level. ART programme managers may wish to estimate the average prevalence of each indicator across the country as a summary measure of overall programme performance. Nationally representative prevalence estimates for each EWI can be calculated by data aggregation and weighting. To aggregate results, all sampled clinics must report a measure of relative size for each EWI (except “pharmacy stock-outs”). Table G2 defines population sizes, which are additional information required for national aggregate weighting. Data analysis must be performed in Stata or another statistical programme that can accurately manage two-stage clustered survey data. The drug stock-out indicator is not weighted. The interpretation of the aggregated values for all EWIs except “ARV medicine stock-out” is the proportion of the patient population in the country with the relevant outcome (for example, adults receiving ART with VL suppression among those with an available VL test result during the reporting period). The interpretation of the aggregated value for the “ARV medicine stock- out” EWI is the average proportion of months with stock-outs of routinely dispensed ARV drugs among clinics in the country during the reporting period. Table G2 provides information required for calculation of national weighted estimates for each indicator. Weighting is performed unless a census of all patients from all clinics is used to calculate the corresponding prevalence. Table G2 Clinic-level information required for national aggregate weighting Indicator Denominator definition Total attrition from ART (ART.2) The total number of patients (adults or paediatic) at the clinic reported to be receiving ART at the end of the last reporting period and/or newly initiating ART during the current reporting period. This number is a count of the total number of records (adult or paediatric) that are eligible to be in the denominator and excludes those known to have transferred out on ART to a different clinic. VL suppression (ART.3) Number of patients (adult or paediatric) receiving ART for at least six months and who received a VL test with available result VL coverage (ART.6) Number of patients (adult or paediatric) who by national policy should have received a VL test during the reporting period. Generally, at least one VL per year is anticipated. Thus, this number is the number of people receiving ART at the end of the prior reporting period plus new ART initiators during the reporting period who have been on ART for at least six months at the time of indicator monitoring. Appropriate second VL test (ART.8) Number of patients with a VL test result available and whose test result showed VL >1000 copies/mL during the reporting period ARV medicine stock-outs (ART.12) None; this EWI is not aggregated but may contribute to national-level ARV stock-out indicator ART.12. ART adherence proxy (ARV drug refills, also known as on-time pill pick up) (ART.13) Total number of patients (adult or paediatric) on ART, by clinic Appropriate switch to second-line ART (ART.14) Number of patients with viral non-suppression, defined as two VL test results >1000 copies/mL during the reporting period for individuals receiving non-NNRTI-based ART and one VL >1000 copies/mL for individuals receiving NNRTI-based ART 10 2.1 National aggregate prevalence estimated when all clinics report EWI or a random sample of clinics reports EWI If all clinics report EWIs or if random sampling is used to select clinics for EWI scale-up, results can be aggregated across sites to generate nationally representative statistics. When aggregating, clinics with greater patient burdens are weighted more heavily than smaller clinics. For each EWI a country can calculate the aggregate point prevalence and 95% confidence interval. An excessive amount of missing (grey) data may complicate or prohibit the interpretation of the aggregated point prevalence. As aggregate analysis will be used for benchmarking and reporting, it is important that results are representative of the respective eligible population. If less than 70% of data are available for the nationally eligible population monitored by a specific indicator, aggregation should not be performed. 2.2 National aggregate prevalence estimated when a combination of randomly sampled clinics and conveniently sampled clinics report EWIs Estimating a nationally representative point prevalence using a combination of randomly sampled clinics and conveniently sampled clinics can be achieved through appropriate weighting. As long as random sampling is used to select a portion of the clinics, the aggregated point prevalence will be nationally representative even if some clinics are sampled by convenience. As an excessive amount of missing (grey) data may complicate or prohibit the interpretation of aggregate point prevalence, it is advised to ensure that results will be representative of the respective eligible population. As noted, if <70% of data are available for the nationally eligible population monitored by a specific indicator, aggregation should not be performed. A flowchart, Fig. G5, is designed to aid in assessing the feasibility of aggregation at the national level. 11 Fig G5 Flow diagram to assess feasibility of data aggregation at the national level 1. All clinics in the country included in EWI monitoring Aggregated EWIs will be nationally representativea Data can be readily aggregated across clinics 2. A subset of clinics included by random sampling Aggregated EWIs will be nationally representativea Data can be aggregated across clinics using weighting to account for clinic sizes 3. A subset of clinics included by random sampling + by convenience sampling Aggregated EWIs will be nationally representativea Data from randomly sampled clinics can be aggregated with data from conviently sampled clinics using weighting 4. Only conveniently sampled clinics included Aggregated EWIs will NOT be nationally representativeb Data may be biased if excluded clinics are different from included clinics a Aggregated EWIs should be reported only if data availability exceeds the recommended 70% threshold. b If only conveniently sampled clinics are included, data may be considered nationally representative if conveniently sampled clinics represent >70% of the eligible patient population for each indicator. 3 DATA ABSTRACTION FROM ART CLINICS Paper-based medical records. If paper-based records are in place, clinic staff trained in the national EWI monitoring strategy should abstract data at their respective sites. Generally, data are abstracted retrospectively, once per year. Whenever possible, countries should combine EWI data abstraction with other indicators and patient monitoring activities taking place in country. EWI monitoring may also be used as, or combined with, a quality assurance assessment of record-keeping at ART clinics. For the purpose of manual data abstraction, an Excel-based data abstraction tool is available on the HIVDR webpage: https://www.who.int/ teams/global-hiv-hepatitis-and-stis-programmes/hiv/treatment/hiv-drug-resistance/prevention. Electronic medical records. If EMRs are in place, experts from the national programme should guide a programme to abstract data for EWI monitoring. Generally, it is not feasible to obtain EWI information from summary reports already produced by electronic record-keeping systems; feasibility may be limited by varying definitions of an indicator or varying methods of applying a definition. If EMRs are used to produce EWIs, validation procedures that use abstraction from paper records should be set up. When electronic query programmes are written, they must also keep track of available data and classify clinics with >30% missing data for a particular indicator as “grey”. Electronic query programmes must also keep track of the overall proportion of missing data for all clinics for the purposes of assessing the feasibility of national aggregation of a specific EWI. 12 4 DATA QUALITY ASSESSMENT Data quality should be assessed throughout the EWI monitoring process. During the data abstraction process, data quality assessments provide critical information for ensuring that the correct data are abstracted in the appropriate way. After the data are analysed and reported, data quality assessment provides programme and clinic managers with the level of confidence that may be placed in the results, and how fit the data are for use in operations, planning and decision-making. Three elements of data quality should be considered: data reliability, data completeness and data consistency. Data reliability. This element assesses the reliability of data abstracted for each indicator. Assessing the quality early in the monitoring process will identify problems that can be addressed through additional support or training. Data completeness. Missing some data is expected. However, a large percentage of missing information in patients’ records at any clinic, for any EWI, presents challenges for achieving the required sample size and for interpretation of results. Monitoring of data completeness should occur during the data abstraction process, and clinics with <70% available data for any indicator should report a “grey” score for that indicator. A “grey” score is not punitive but rather signals that the clinic requires support in record-keeping before it can derive full benefit from indicator monitoring. Aggregation should not be performed unless ≥70% data are available from all sites sampled. Data consistency. Data consistency refers to consistency of patient information across different record systems within the same clinic. Clinic and pharmacy records are the primary sources of information used for EWI monitoring. Some clinics use both paper-based and electronic systems for clinic and pharmacy records. A records assessment process should be performed prior to data abstraction for EWI monitoring to evaluate the consistency of information across these different sources. This is a crucial step in assessing which sources provide the most accurate information. While not specifically designed to address the data used for EWI monitoring, data quality assessments identify strengths and weaknesses of existing record systems in participating clinics. Incompleteness and inconsistency of data generally indicate more systemic problems in record-keeping that should be addressed. Thus, the results of data quality assessments can inform changes that will improve patient monitoring systems and clinic practices. For more information, contact: World Health Organization Department of Global HIV, Hepatitis and STIs Programmes 20, avenue Appia 1211 Geneva 27 Switzerland Email: hiv-aids@who.int www.who.int/hiv