The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems.

503Bulletin of the World Health Organization | July 2005, 83 (7) Objective To evaluate the consistency and quality of immunization monitoring systems in 27 countries during 2002–03 using standardized data quality audits (DQAs) that had been launched within the framework of the Global Alliance for Vaccines and Immunization. Methods The consistency of reporting systems was estimated by determining the proportion of third doses of diphtheria–tetanus– pertussis (DTP-3) vaccine reported as being administered that could be verified by written documentation at health facilities and districts. The quality of monitoring systems was measured using quality indices for different components of the monitoring systems. These indices were applied to each level of the health service (health unit, district and national). Findings The proportion of verified DTP-3 doses was lower than 85% in 16 countries. Difficulties in verifying the doses administered often arose at the peripheral level of the health service, usually as the result of discrepancies in information between health units and their corresponding districts or because completed recording forms were not available from health units. All countries had weaknesses in their monitoring systems; these included the inconsistent use of monitoring charts; inadequate monitoring of vaccine stocks, injection supplies and adverse events; unsafe computer practices; and poor monitoring of completeness and timeliness of reporting. Conclusion Inconsistencies in immunization data occur in many countries, hampering their ability to manage their immunization programmes. Countries should use these findings to strengthen monitoring systems so that data can reliably guide programme activities. The DQA is an innovative tool that provides a way to independently assess the quality of immunization monitoring systems at all levels of a health service and serves as a point of entry to make improvements. It provides a useful example for other global health initiatives. Keywords Diphtheria-tetanus-pertussis vaccine/administration and dosage; Immunization programs/statistics; Data collection/ standards; Quality control (source: MeSH, NLM). Mots clés Vaccin diphtérie-tétanos-coqueluche/administration et posologie; Programmes de vaccination/statistique; Collecte données/ normes; Contrôle qualité (source: MeSH, INSERM). Palabras clave Vacuna difteria-tétano-pertussis/administración y dosificación; Programas de inmunización/estadística; Recolección de datos/normas; Control de calidad (fuente: DeCS, BIREME). Bulletin of the World Health Organization 2005;83:503-510. Voir page 509 le résumé en français. En la página 509 figura un resumen en español. The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems O. Ronveaux,1 D. Rickert,2 S. Hadler,3 H. Groom,2 J. Lloyd,4 A. Bchir,5 & M. Birmingham1 1 Vaccine Assessment & Monitoring, Department of Immunization, Vaccines and Biologicals, World Health Organization, 20 Avenue Appia, CH1211 Geneva 27 Switzerland. Correspondence should be sent to Dr Ronveaux at this address (email: ronveauxo@who.int). 2 Immunization Services Division, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA, USA. 3 Global Immunization Division, National Immunization Program, Centers for Disease Control and Prevention, Atlanta, GA, USA. 4 Children’s Vaccine Program, Program for Appropriate Technology in Health, Ferney Voltaire, France. 5 Secretariat for the Global Alliance for Vaccines and Immunization, Geneva, Switzerland. Ref. No. 04-014860 (Submitted: 28 May 2004 – Final revised version received: 16 December 2004 – Accepted: 14 January 2005) .510 Introduction The Global Alliance for Vaccines and Immunization (GAVI) was launched in 2000, and since then it has provided annual financial support to improve childhood immunization services in 52 developing countries through a performance-based grant programme (via the Vaccine Fund). GAVI allocates invest- ment funds to all participating countries and then provides financial rewards based on a single indicator: the reported and independently verified number of children younger than 12 months of age who have been vaccinated with all three doses of diphtheria–tetanus–pertussis vaccine (DTP-3) (1, 2). Thus, an audit was needed to verify the quality of countries’ reports of the number of children immunized with DTP-3. Most countries track the performance of their immuniza- tion programmes through hierarchical administrative moni- toring systems. In a typical system, staff at local health facilities compile vaccination data from daily immunization logs or tally sheets and report these to a district health officer monthly. Ide- ally, staff at both the health facility and at the district level use these reports to evaluate progress in achieving immunization coverage goals. The district officer compiles the coverage data 504 Bulletin of the World Health Organization | July 2005, 83 (7) Research Auditing the quality of immunization data O. Ronveaux et al. of reporting consistency. For each district, the VF is calculated for the previous year’s reported activity: first the DTP-3 vac- cinations are re-counted from paper records in the selected health units; these are then divided by the number of DTP-3 doses found in district file records as reported by these health units. Second, the quotient found above is multiplied by the ratio (reported DTP-3 found at district level/reported DTP-3 found at national level) to account for any reporting differences between these latter two levels as follows: with i = district indicator (i = 1, 2, 3, 4) and j = health unit in- dicator (j = 1, 2, …, 6) and where xij = the number of re-counted DTP-3 vaccinations found in the records of the j th health unit of the i th district yij = the number of reported DTP-3 vaccinations from the j th health unit of the i th district Rdi = at the district level, the number of all DTP-3 vaccina- tions reported from all health units from the i th district to the national level Rni = at the national level, the number of reported DTP-3 vaccinations reported by the i th district. The national VF is calculated as the weighted average of district VFs. A VF of < 1 indicates an inability to verify all of the doses of DTP-3 reported to have been administered (overreporting). Conversely, a VF > 1 indicates that a higher number of doses were recorded as being administered at peripheral health-service levels than are reflected in the number sent to more central levels (underreporting). To characterize reporting consistency at the level of vaccine delivery results from health units were classified into three categories: consistent if the ratio (re-counted DTP-3/reported DTP-3) was  85% and < 115%; under- reported if the ratio was  115%; and overreported if it was < 85%. Health units who had overreported were classified fur- ther depending on whether the inconsistency was primarily due to missing health unit tally sheets or logs, discrepant tally from all facilities and reports them either monthly or quarterly to the national level. National level staff use these data to as- sess national and district performance and to compile annual reports that are submitted to WHO and UNICEF. When ef- ficient, accurate and timely reporting occurs at each level of the hierarchy, administrative monitoring systems provide a strong basis from which to guide planning, review progress and deter- mine which areas need additional efforts in order to cope with low-coverage or high drop-out rates (3). However, over the past 20 years, community-based surveys have sometimes reported coverage levels that were inconsistent with those reported by the administrative monitoring system. Evaluations of these systems identified problems with data quality and validity (4–6). To verify the consistency of national reports based on administrative monitoring systems, WHO developed an evalu- ation protocol, known as the immunization data quality audit (DQA), using administration of three doses of the DTP vac- cine before the age of 12 months as the sentinel indicator (7). The DQA also assesses the quality, efficiency, security and use- fulness of the system at each reporting level to enable practical recommendations to be made for improving the system. In 2001, an independent consortium field-tested the DQA in 8 countries (8). A revised protocol was subsequently administered in all GAVI-supported countries receiving an initial investment greater than US$ 100 000 (7). This paper presents the findings of the 2002–03 DQA effort. Methods The DQA process GAVI hired two independent companies to conduct DQAs according to a recommended protocol (7), which called for on-site evaluations at each level of the system, starting at the national level. The companies were trained by WHO. A mul- tistage sampling procedure was used that included assessments of four districts selected by a probability proportional to the reported doses of DTP-3 administered as well as assessments in six health units in each of the four sampled districts. The chosen sample size, and hence the precision of the results, was dictated by logistical and financial considerations; the sample allowed for the maximum number of structures that could be visited by two evaluation teams within a 2-week period. The DQA measures The DQA reviews two key performance measures. One mea- sure is the verification factor (VF), a district-based indicator Table 1. National and district verification factor values from 25 audits of immunization data quality, 2002–03a National No. District No. (%) districts Mean percentage (range) of Mean percentage verification countries verification with verification unverified data attributable (range) reported factor factors factors > 0.85 to differences between (administrative) national and < 1.15 health units and districts DTP-3 coverage  0.85b 9 0.48–1.31 26 (72) 81 (32–100) 69 (22–97) 0.70–0.84c 7 0.31–1.43 12 (43) 98 (83–100) 69 (51–95) < 0.70d 9 0.04–1.06 4 (11) 89 (55–100) 60 (43–82) Total 25 0.04–1.43 42 (42) 89 66 a Verification factors could not be calculated for Nigeria and Sudan. b Includes Afghanistan, Bangladesh, Cambodia, Ghana, Niger, Pakistan, Rwanda, Tajikistan and the United Republic of Tanzania. c Includes Ethiopia, Mali, Nepal, Senegal, Uganda, Yemen, and Zambia. d Includes Burkina Faso, Cameroon, Côte d’Ivoire, Guinea, Haiti, Kenya, Lao People’s Democratic Republic, Madagascar and Mozambique. 505Bulletin of the World Health Organization | July 2005, 83 (7) Research O. Ronveaux et al. Auditing the quality of immunization data sheets or logs (e.g., health unit data on DTP-3 doses ad- ministered were available but did not match what had been reported more centrally), or a mixed problem of missing and discrepant data. The second key measure is the Quality Index (QI), a quantitative measure of the quality of each component at each level of the monitoring system. QIs are based on questions and observations at national level (53 questions), district level (38 questions) and health-unit level (31 questions) (7). The ques- tions and observations in the QIs are grouped into five com- ponents: recording practices, storage and reporting practices, monitoring and evaluation, denominators used at district and national levels, and system design at the national level. To assess the proficiency of recording practices, workers at the health- unit level were asked to complete national immunization cards for 20 hypothetical children. In calculating the QI scores, 1 point is given for each question answered correctly or task observed to have been performed correctly. Scores are calculated for each level of the health service and for each of the five components, with the number of correct answers and correctly performed tasks as the numerator and the number of answers and observations as the denominator. Data analysis VFs for the countries and districts were calculated with 95% confidence intervals based on a t distribution with m-1 degrees of freedom (where m is the number of clusters selected). To provide composite information on the specific strengths and weaknesses of the immunization reporting systems, we aggre- gated responses for each QI-component question or observa- tion for all countries, and we present overall responses for key activities for health units, districts and countries. To identify factors associated with high consistency within a system, zero-order Pearson correlation coefficients were calculated between national-level and district-level VFs and national-level, district-level, and health-unit-level QIs. To account for the fact that VF is not a linear concept, VFs > 1 were transformed by subtracting the verification excess (greater than 1) from 1. Descriptive statistics and bivariate tests for association between classes at each level (Fisher’s exact and χ² tests for trend) were calculated using SAS for Windows, release 8.02 (SAS Institute, Cary, NC, USA). Findings During 2002–03, DQAs were completed in 27 countries: 18 in Africa, 1 in the Americas, and 8 in Asia (Table 1). Four districts were assessed in each country, but in 11 countries the number of facilities visited was less than the 24 called for in the protocol (range = 14–23), either because one or more of the four selected districts had fewer than six health units or because security or logistical constraints precluded visiting some of them. Although QIs are provided for all 27 countries, VFs could not be calculated in two countries (Nigeria and Sudan) because of inadequately reported data at the national or district level. Reporting consistency National verification factors ranged from 0.40 to 1.06. Nine countries had VFs suggesting consistent data ( 0.85 and  1.15); 7 had VFs suggesting moderate overreporting (0.70–0.84) and 9 had results that suggested considerable overreporting (< 0.70) (Table 1). Most national VFs had wide confidence intervals, although confidence intervals were nar- rower in countries with VFs that showed they had consistent data. District VFs ranged from 0.04 to 1.43, with 42% within the high range of verification (0.85–1.15). Inaccuracies pri- marily occurred at peripheral levels of the health service (e.g., discrepancies between data from health units and data from districts), except in five countries where discrepancies between district and national DTP-3 data also substantially reduced the national VF. In all VF categories we observed a wide range of reported national DTP-3 coverage levels; countries with low VFs tended to have lower DTP-3 coverage than countries with high or moderate VFs (Table 1). In countries with high national VFs, 73% of health units were classified as consistent; this proportion was only 31% (range = 21–53%) in countries with low national VFs. Among the 557 health units assessed, 53% (296) had consistent data; 7% (38) had underreported DTP-3 doses; and 40% (223) had overreported (Fig. 1). Overreporting was most often attributed to data discrepancies in 22% (120) of health units and to missing data in 14% (80). Data were most often missing as a result of failure to retain daily tally sheets for the previous year, and in two countries (Cameroon and the Lao People’s Democratic Republic), DTP-3 data were miss- ing for more than 40% of health units. Overreporting and underreporting were found in some health units in almost all countries. These discrepancies may have occurred because data from outreach sessions went missing or there may have been errors in arithmetic or transcribing. Quality of monitoring systems Quality Index scores varied widely at all levels, and the overall analysis showed substantial deficiencies in all countries. The mean national QI for the 27 countries was 63% (Table 2). The mean district QI was 63% (range = 15–97%), and the mean Fig. 1. Distribution of health units by reporting consistency of vaccination with three doses of diphtheria–tetanus–pertussis vaccine and national verification factor category from data quality audits in 25 countries, 2002–03. (A more complete explanation of overreporting and underreporting is given in the text) WHO 05.33 100 % o f h ea lt h un it s All 0 Overreporting (mixed problem) 80 60 40 20 VFa >0.84 VF 0.70–0.84 VF <0.70 Overreporting (discrepancy) Overreporting (due to missing data) Accurate Underreporting a VF = Verification factor. 506 Bulletin of the World Health Organization | July 2005, 83 (7) Research Auditing the quality of immunization data O. Ronveaux et al. health unit QI was 58%. However, these means masked wide variation in district and health-unit performance across and within countries; individual district QI values ranged from 15% to 97%, and individual health unit ranged from 4% to 100%. In most countries, we were able to identify districts and health facilities that performed well. Both district and health-unit QIs were significantly higher in countries with high national verification factors, and values decreased systemati- cally with decreasing VFs (Table 2). In contrast, national QIs had no significant association with national VFs. There was no correlation between QI scores at the national level and those at the district level and health-unit level. Mean QI scores for each component and level of the health system varied from 50% to 80% (Fig. 2). Of the five component groups of questions and observations in the QIs (recording, storage and reporting, monitoring and evaluation, denominators, and system design), the weakest one at all levels was “monitoring and evaluation”. The strongest one (at national and district levels) was the “appropriate use of denominators” (used to calculate coverage rates). However, aggregating the results of each component for each level of the health service hides variations in results for each question and observation within each component (Table 3). Among specific question and observation within the “re- cording practices” component, when health workers were asked to complete a series of vaccination cards for 20 hypothetical children, only 59% (292/495) were observed to complete all of them correctly. We also found substantial deficiencies in the monitoring of vaccine and injection supplies; proper monitor- ing is required to manage vaccine supplies and correctly calcu- late vaccine wastage. At the health-unit level, vaccine ledger books used to monitor vaccine and injection supplies were usually not up-to-date, and wastage could be calculated for only 32% (175/538) of facilities. Vaccine stock ledgers were better maintained at district level (68% (72/105) of district facilities had well maintained ledgers), and best at the national level (74% (20/27) of national facilities had well maintained ledgers). The monitoring of injection supplies also was best at the national level (77% (20/26) of national facilities compared with 56% (51/91) of district facilities and 24% (110/456) of health units), but it was not implemented at national or district level at all in three countries (Ethiopia, Guinea, Mali). The greatest weaknesses among questions and observa- tions in the “storage and reporting practices” component were found in the failure to retain forms, to develop procedures for Table 2. Percentage value of quality indices at national level, district level and health-unit level by verification factor, from 27 audits of immunization data quality, 2002–03 National No. Mean percentage (range) Mean percentage (range) Mean percentage (range) verification countries national Quality Index score district Quality Index score health-unit Quality Index score factor  0.85 9 63 (47–86) 71 (32–94) 69 (26–100) 0.70–0.84 7 65 (46–87) 65 (22–97) 61 (12–100) < 0.70 9 66 (49–76) 57 (15–81) 49 (4–86) Totala 27 63 (46–87)b 63 (15–97)c 58 (4–100)d a The total includes two countries with no verification factors (Nigeria and Sudan). b National Quality Index scores do not differ significantly by verification factor stratum (P = 0.74). c District Quality Index scores differ by verification factor stratum (P < 0.007). d Health unit Quality Index scores differ by verification factor stratum (P < 0.0001). handling late reports, to have procedures for computer back- up and data processing, and to have procedures for reporting adverse events. Recording forms (registers and tally sheets from the previous year) were fully available in only 60% (348/581) of health units. Procedures for timely reporting were well es- tablished at each level, but procedures for handling late reports were available in only 45% (47/105) of districts and 11% (3/27) of national offices. Computers were utilized for data processing in all countries at the national level and in one-third of districts, but procedures for back-up and data transfer were adequate in only a minority of cases. Procedures for reporting adverse events had been established in only 19% (5/27) of countries and 31% (21/67) of districts. In contrast, procedures for filing, dating and signing reports were adequate in 60–80% of health units and districts, and in more than 80% of national offices. At all levels, the “monitoring and evaluation” component received the lowest scores. Among the questions and observa- tions in this component, monitoring charts, which graphically display progress in reaching coverage targets throughout the year, were displayed in only 43% (248/572) of health units, Fig. 2. Mean scores on quality indices by health-service level and immunization reporting system component for data quality audits in 27 countries, 2002–03. (More complete explanations of the components measured are given in the text) WHO 05.34 Q ua lit y In de x sc or e (% ) National 0 Recording 80 60 40 20 Storage and reporting Denominators System design Monitoring and evaluation District Health unit 100 Health-service level 507Bulletin of the World Health Organization | July 2005, 83 (7) Research O. Ronveaux et al. Auditing the quality of immunization data Table 3. Aggregated correct responses to selected component questions and observations, by monitoring-system component and health-service level, from audits of immunization data quality conducted in 27 countries, 2002–03a Component Item Health- Selected quality question % with a service level correct response Recording Vaccine stock management Health unit DTP ledger completeb 32 and vaccine wastage District DTP vaccine ledger up to date 68 calculation National National wastage calculated correctly 27 Stock system for injection Health unit Implement stock system for injection supplies 24 supplies District Ledger system for syringes 56 National Reporting of injection supplies 77 Recording of vaccination Health unit Vaccine card test for DTP-3 59 on child’s health card Storage and Availability of form Health unit Individual recording forms available for 60 reporting whole yearc Reporting procedure Health unit Method exists for sending reports on timed 89 District Procedure for following up on late reports 45 National Written procedure for following up 11 on late reports Computer procedures District Computer back-up last done within 1 week 8 National Recent computer back-up 31 Adverse events monitoring District System for reporting adverse events works 31 National Written system for reporting adverse events 19 Monitoring Monitoring of coverage Health unit Up-to-date monitoring chart for children on display 43 and evaluation District Up-to-date monitoring chart for children on display 48 National Up-to-date monitoring chart for children on display 26 Target setting Health unit System to collect information on new birthsd 63 District Target number of children who need vaccination 93 set for the district Dropout monitoring Health unit System for monitoring drop-outc 31 District Drop out monitoring chart displayedc 41 Completeness/timeliness District Chart on health unit completeness displayedd 41 Monitoring National Timeliness of district reporting chart displayedd 50 Feedback and supervision Health unit Dated feedback received from district within 19 4 monthsd District Routine feedback to health unit with analysis 23 and discussiond National Last feedback given to district within 4 months; 46 feedback contains analysis and discussion 61 Denominator Denominators District District denominator same as national 35 National All district DTP coverage < 100% 31 a For a more complete explanation of terms, see full text of paper. b DTP = diphtheria–tetanus–pertussis. c Question asked only in 2003 and only in 11 countries. d Question asked only in 2002 and only in 16 countries. although 63% (208/330) of units could provide the estimated number of children who needed to be vaccinated in the target population. Similarly, only 48% (52/108) of districts and 26% (7/27) of national offices displayed monitoring charts. Fewer than 50% of health units (81/258) and districts (18/44) monitored drop-out rates (i.e., children who did not receive the full series of vaccinations). Monitoring of completeness and timeliness of health-unit and district reporting was variable (oc- curring in only 41–50% of sites). Supervisory visits had been conducted within the past 4 months in fewer than 50% of districts (27/64) and health units (66/341); standard feedback formats were used in only 50% (55/108) of districts; and there was limited analysis and discussion of information: only 61% (14/23) of countries and 23% (15/64) of districts routinely gave feedback that contained analysis and discussion. The main issues within the “appropriate use of denomi- nators” component was the discrepancy between district num- bers and national numbers of target populations in districts (observed in 65% of the countries). All countries were found to update their denominators annually. However, 18 countries (69%) reported some district-coverage values that were higher than 100%, an indicator that data sources used for denomina- tors may be inaccurate, that children outside of the district(s) in question were being vaccinated, or that doses administered to children who were 12 months of age or older may have been recorded as doses administered to infants. 508 Bulletin of the World Health Organization | July 2005, 83 (7) Research Auditing the quality of immunization data O. Ronveaux et al. Within the final component — “system design” — im- munization reporting was integrated within comprehensive health information systems in 13 (50%) countries. In addition, the design of monthly reporting forms was usually appropriate (according to WHO recommendations), with only one country not reporting vaccinations of infants (children younger than 12 months) separately from those of older children. All countries reported tetanus toxoid immunizations for pregnant women separately. Correlation analyses The national verification factor showed a strong correlation with district and health-unit QIs (Pearson’s correlation coef- ficients of 0.41 and 0.50, respectively; P < 0.001), but showed no correlation with national QIs. District VFs also correlated strongly with district and health-unit QIs (Pearson’s correla- tion coefficients of 0.40 and 0.48, respectively; P < 0.0001) (Fig. 3). This suggests that the quality of reporting systems in districts and health units are important predictors of consistency in reporting systems in countries. Discussion The data quality audits conducted in 27 countries in 2002–03 provide a comprehensive evaluation of immunization monitor- ing systems. This is the first time a standard method has been applied to assess monitoring systems for a public health interven- tion in developing countries. The DQA provides a quantitative indication of reporting consistency and quality, and this can facilitate comparisons of results over time or place and motivate those who are monitoring the system to make improvements in monitoring activities (9). The DQA also provides information about which level of the health service contributes most to inac- curacies, and it diagnoses specific weaknesses in the monitoring system that, if addressed, could improve its precision, efficiency, security and usefulness. This analysis identifies important challenges that must be faced in order to improve immunization monitoring systems. It indicates that corrective efforts must be focused at peripheral levels (districts and health units) if quality and consistency are to be improved. The DQAs reveal a particular need to improve the skills and practices of those involved in the analysis and use of data at all levels of the health service to help guide strategies to increase coverage, manage vaccine supply and monitor vac- cine safety. Previous studies have shown that immunization staff in provincial and local areas have weak skills in using quantita- tive immunization data (10). Efforts to improve data analysis and use at the local level could in turn stimulate improvements in the accuracy of the data collected because staff may take an interest in their own data and value the opportunity to dem- onstrate local achievements and guide local planning. The wide variation of QIs at all levels reinforces the need to examine the information from each level independently. We did not observe a significant association between im- munization coverage levels and VFs, although data verifiability tended to be lower in countries with lower coverage. This sug- gests that all countries, regardless of coverage levels, may have substantive inconsistencies in their reporting systems and would benefit from systematic assessments such as the DQA. Community-based surveys have traditionally been used to verify reported vaccine coverage and obtain point estimates of immunization coverage levels; they have also revealed Fig. 3. Correlation between verification factor at the district level and Quality Index score at the health-unit level, from data quality audits, 2002–03 WHO 05.35 Ve ri fic at io n fa ct or d is tr ic t le ve l ( % ) 0 0 120 Quality Index health unit (%) 20 40 60 80 100 100 80 60 40 20 substantive inaccuracies in administrative monitoring systems (11). Coverage surveys are not, however, a substitute for admin- istrative reporting and the timely monitoring of programme effectiveness at the community level. Coverage surveys typi- cally provide information on birth cohorts from previous years and therefore do not provide the continual flow of information needed for local programme management. Coverage surveys also vary in precision, and their data may be subject to recall bias and quality problems (12). Furthermore, they do not provide information on the quality of the monitoring system or identify the cause of inaccuracies. Therefore, administrative monitoring systems are needed to provide critical information on an ongo- ing basis to local staff in order to determine whether coverage targets are being met. WHO has developed a data quality self- assessment tool (DQS) for use by district-level staff to help them identify problems in their monitoring system. An abbreviated version of the DQS can be integrated into rapid assessments or supportive supervision visits at the district level and health- unit level. In addition, WHO and its partners are promoting the Reaching Every District approach (13), one component of which aims to strengthen district monitoring capacity and to promote the use of local data in planning. The DQA has two important limitations. The first is the imprecision of the verification factor in validating the reported number of children receiving three doses of DTP. Due to the small sample size and a large variation in VFs among districts in the same country, average confidence intervals of national VFs were wide (averaging ± 30%). Greater precision in VFs could be achieved by increasing the size of the sample, but there would be a trade-off in terms of increases in cost and the time necessary to conduct the audit. A second weakness is that verification of immunization at the point of recording a dose as being administered is not addressed in the current DQA; the audit tool therefore may miss doses that were recorded but were not administered or vice versa. Community-based verification is necessary if this type of inaccuracy is suspected. Through the DQA process, GAVI has invested in strength- ening existing monitoring systems by providing both diagnostic information and incentives to countries to improve their sys- tems. That countries have accepted this process is evidenced by the many countries that plan to improve their systems and in the substantial improvements in DQA results seen in several countries since the initial audit in 2001. 509Bulletin of the World Health Organization | July 2005, 83 (7) Research O. Ronveaux et al. Auditing the quality of immunization data Résumé Contrôle de la qualité des données de vaccination : vérification de la qualité et de la cohérence des systèmes de vaccinovigilance Objectif Évaluer la cohérence et la qualité des systèmes de vaccinovigilance dans 27 pays pendant la période 2002-2003 à travers des contrôles de la qualité des données (DQA) normalisés, lancés dans le cadre de l’Alliance mondiale pour les vaccins et la vaccination. Méthodes On a estimé la cohérence des systèmes de notification en déterminant la proportion de troisièmes doses de vaccin diphtérie-tétanos-coqueluche (DTC3) rapportées comme ayant été administrées, dont l’administration peut être confirmée par des documents écrits tenus par les installations de santé et les districts. On a mesuré la qualité des systèmes de vaccinovigilance à l’aide d’indices de qualité s’appliquant aux différents composants de ces systèmes. Ces indices ont été appliqués à chaque niveau de service de santé (niveau de la formation sanitaire ou du district ou encore niveau national). Résultats La proportion de doses de DTC3 vérifiées était inférieure à 85 % dans 16 pays. Les difficultés pour vérifier les doses administrées se rencontrent souvent au niveau périphérique du service de santé et résultent habituellement de divergences dans les informations détenues par les formations sanitaires et les districts correspondants ou de l’impossibilité d’obtenir, auprès de ces formations sanitaires, des formulaires d’enregistrement remplis. Les systèmes de vaccinovigilance de tous les pays présentent des points faibles. : incohérences dans l’utilisation des tableaux de contrôle, surveillance insuffisante des stocks de vaccins, matériel d’injection et manifestations postvaccinales indésirables, pratiques informatiques peu sûres et suivi insuffisant de l’exhaustivité et de la promptitude des notifications. Conclusion Les données de vaccination présentent des incohérences dans de nombreux pays, ce qui compromet leurs possibilités d’utilisation pour gérer les programmes de vaccination. Il faudrait que les pays utilisent ces résultats pour renforcer leurs systèmes de vaccinovigilance, de manière à ce que les données qu’ils fournissent puissent guider de façon fiable les activités des programmes. Le DQA est un outil novateur permettant d’évaluer de manière indépendante la qualité des réseaux de vaccinovigilance à tous les niveaux de service de santé et servant de point d’entrée pour introduire des améliorations. Il fournit un exemple utile pour d’autres initiatives mondiales dans le domaine sanitaire. Conclusion The DQA is thus an innovative tool that meets its two main objectives: it serves as a diagnostic tool for a country’s im- munization monitoring system by providing practical recom- mendations as to how it could be improved. It also provides critical, objective and independently collected information to the donor community on the quality and consistency of the measure that is used to determine rewards. Because of the in- tensive international efforts, resources and funds that are be- ing directed towards supporting the Millennium Development Goals, there is an urgent need for accurate and timely health data to be collected on an ongoing basis (14). It is, therefore, important to improve the quality and usefulness of relatively low-cost, pre-existing monitoring systems within developing countries (15). The challenge now is to use the findings from the DQA to strengthen monitoring systems so that their data can reliably guide activities to reduce morbidity and mortality from diseases that vaccines can prevent. O Acknowledgements We would like to thank Linda Archer who initially designed and field-tested the DQA tool, Elizabeth Zell for technical as- sistance in the development of the DQA, and Stacy Woodard for statistical expertise. Funding: The Data Quality Audits were financed by the Global Alliance for Vaccines and Immunization. Competing interests: none declared. Resumen Auditoría de la calidad de los datos de inmunización: comprobación de la calidad y coherencia de los sistemas de monitoreo de la inmunización Objetivo Evaluar la coherencia y calidad de los sistemas de monitoreo de la inmunización en 27 países durante 2002–2003, usando para ello auditorías normalizadas de la calidad de los datos emprendidas en el marco de la Alianza Global para las Vacunas y la Inmunización. Métodos La coherencia de los sistemas de notificación se estimó determinando la proporción de terceras dosis de la vacuna contra difteria, tétanos y tos ferina (DTP-3) cuya administración previamente notificada pudo verificarse mediante documentación escrita en establecimientos y distritos de salud. La calidad de los sistemas de monitoreo se midió usando índices de calidad para distintos componentes de los sistemas de monitoreo. Estos índices se aplicaron a cada nivel del servicio de salud (puestos de salud, distrito y ámbito nacional). Resultados La proporción de dosis de DTP-3 verificadas fue inferior al 85% en 16 países. La comprobación de las dosis administradas tropezó a menudo con dificultades en la periferia del servicio de salud, generalmente como resultado de discrepancias entre la información de los puestos de salud y la de sus correspondientes distritos, o porque los puestos de salud no aportaban formularios de registro completos. Todos los países presentaban puntos débiles en sus sistemas de monitoreo, como por ejemplo el uso incoherente de gráficos de monitoreo; una vigilancia inadecuada de las reservas de vacuna, los suministros de inyección y los eventos supuestamente atribuibles a la vacunación o la inmunización; la baja seguridad de las prácticas de computación; y un escaso monitoreo de la integralidad y puntualidad de los informes. Conclusión Los datos de inmunización adolecen de incoherencia en muchos países, lo cual limita la capacidad de éstos para gestionar sus programas de inmunización. Los países deberían usar 510 Bulletin of the World Health Organization | July 2005, 83 (7) Research Auditing the quality of immunization data O. Ronveaux et al. References 1. Wittet S. Introducing GAVI and the Global Fund for Children’s Vaccines. Vaccine 2000;91:385-6. 2. Brugha R, Starling M, Walt G. GAVI, the first steps: lessons for the Global Fund. Lancet 2002;359:435-8. 3. World Health Organization. Immunization in practice: a practical resource guide for health workers. 2004 update. Geneva: WHO; 2004. WHO document WHO/IVB/04.06. 4. Onta SR, Sabroe S, Hansen EH. The quality of immunization data from routine primary health care reports: a case from Nepal. Health Policy and Planning 1998;13:131-9. 5. World Health Organization. WHO vaccine-preventable diseases: monitoring system. Geneva: WHO; 2003. WHO document WHO/V&B/03.20. 6. Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet 2003;362:1022-7. 7. World Health Organization. The immunization data quality audit (DQA) procedure. Geneva: WHO; 2003. WHO document WHO/V&B/03.19. 8. LATH Consortium. Immunization data quality audit evaluation report: final report from Deloitte Touche Tohmatsu Emerging Markets, Euro Health Group, and Liverpool Associates in Tropical Health. Geneva: WHO; 2001. 9. Weeks RM, Svetlana F, Noorgoul S, Valentina G. Improving the monitoring of immunization services in Kyrgyzstan. Health Policy and Planning 2000;15:279-86. 10. Loevinsohn BP. Data utilization and analytical skills among mid-level health programme managers in a developing country. International Journal of Epidemiology 1993;23:194-200. 11. Henderson RH, Sundaresan T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bulletin of the World Health Organization 1982;60:253-60. 12. Boerma JT, Sommerfelt AE. Demographic and health surveys (DHS): contributions and limitations. World Health Statistics Quarterly 1993;46:222-6. 13. WHO Regional Office for Africa. Building capacity to reach every district with immunization services in the African region: status and perspectives. Afro EPI-Newsletter, Vaccine Preventable Diseases Bulletin 2003;33:1-3. Also available from: http://www.afro.who.int/ddc/vpd/bulletins/2003/ mar2003.pdf 14. Lee JW. Global health improvement and WHO: shaping the future. Lancet 2003;362:2083-8. 15. United Nations Development Programme. Millennium Development Goals. Available from: http://www.undp.org/mdg/ estos resultados para fortalecer los sistemas de monitoreo de modo que los datos puedan orientar de manera fiable las actividades programáticas. La auditoría de la calidad de los datos es un instrumento innovador que permite evaluar independientemente la calidad de los sistemas de monitoreo de las inmunizaciones a todos los niveles de un servicio de salud y sirve de punto de entrada para introducir mejoras. Además, constituye un valioso ejemplo para otras iniciativas de salud mundiales.