Evaluation of immunization coverage by lot quality assurance sampling compared with 30-cluster sampling in a primary health centre in India.

Evaluation of immunization coverage by lot quality assurance sampling compared with 30-cluster sampling in a primary health centre in India J. Singh,1 D.C. Jain,2 R.S. Sharma,3 & T. Verghese4 The immunization coverage of infants, children and women residing in a primary health centre (PHC) area in Rajasthan was evaluated both by lot quality assurance sampling (LQAS) and by the 30-cluster sampling method recommended by WHO's Expanded Programme on Immunization (EPI). The LQAS survey was used to classify 27 mutually exclusive subunits of the population, defined as residents in health subcentre areas, on the basis of acceptable or unacceptable levels of immunization coverage among infants and their mothers. The LQAS results from the 27 subcentres were also combined to obtain an overall estimate of coverage for the entire population of the primary health centre, and these results were compared with the EPI cluster survey results. The LQAS survey did not identify any subcentre with a level of immunization among infants high enough to be classified as acceptable; only three subcentres were classified as having acceptable levels of tetanus toxoid (TT) coverage among women. The estimated overall coverage in the PHC population from the combined LQAS results showed that a quarter of the infants were immunized appropriately for their ages and that 46% of their mothers had been adequately immunized with TT. Although the age groups and the periods of time during which the children were immunized differed for the LQAS and EPI survey populations, the characteristics of the mothers were largely similar. About 57% (95% Cl, 46-67) of them were found to be fully immunized with TT by 30-cluster sampling, compared with 46% (95% Cl, 41-51) by stratified random sampling. The difference was not statistically significant. The field work to collect LQAS data took about three times longer, and cost 60% more than the EPI survey. The apparently homogeneous and low level of immunization coverage in the 27 subcentres makes this an impractical situation in which to apply LQAS, and the results obtained were therefore not particularly useful. However, if LQAS had been applied by local staff in an area with overall high coverage and population subunits with heterogeneous coverage, the method would have been less costly and should have produced useful results. Introduction The 30-cluster survey method of WHO's Expanded Programme on Immunization (EPI) is currently used in most countries to evaluate immunization cover- age; it is simple, easy to implement under field condi- tions, and provides useful measures of immunization activity. But the method does not allow identification of smaller, operationally important, population subunits where the coverage is at an unacceptable level (1-5). Lot quality assurance sampling (LQAS) Assistant Director, National Institute of Communicable Diseases (NICD), 22 Shamnath Marg, Delhi-110054, India. Correspondence should be sent to Dr Jagvir Singh at this address. 2 Joint Director, NICD. 3 Director, National Malaria Eradication Programme, NICD Campus. 4 Former Director, NICD. Reprint No. 5699 is designed to classify subunits (lots) of a population, and has been field-tested in Peru (6) and Costa Rica (3) to evaluate immunization coverage in small population units. In a previous, small study in India, we also found LQAS to be operationally feasible (5). However, the time required, costs and results of the LQAS and EPI methods were not compared. In the present study, we evaluated immunization coverage in a primary health centre (PHC) area with LQAS and EPI surveys using paramedical personnel as field staff. LQAS is a stratified random sampling method where small samples, randomly selected from each lot, are used to classify the lot as acceptable or unac- ceptable on the basis of a particular characteristic(s). A sample size (n) and an allowable number (d) of units in the sample with a characteristic (e.g., not immunized with a particular vaccine dose) are calcu- lated with a predetermined probability to identify any lot with more than a specified proportion of the Bulletin of the World Health Organization 1996, 74 (3): 269-274 © World Health Organization 1996 269 J. Singh.et al. characteristic (e.g., unimmunized children) as unac- ceptable (by producing more than d with the charac- teristic in the sample n). Because LQAS is stratified random sampling, the results from the lot samples can be combined to obtain a point estimate for the entire population. Poisson, binomial or hyper- geometric distributions may be applicable for determination of n and d depending upon the size of the population (N), the proportion with the characteristic (P), and the sampling fraction (n/N). More information about LQAS can be found elsewhere (1-5, 7). Materials and methods LQAS survey The two surveys (LQAS and EPI) were carried out in Malakhera Primary Health Centre (PHC) area, Alwar district, Rajasthan, during the period 22 Sep- tember to 13 November 1992. This PHC served a population of 168000 distributed in 27 subcentre areas (average of 6200 for each subcentre). It was estimated, based on a crude birth rate (CBR) of 34 live births per 1000 population and an infant mortality rate (IMR) of approximately 90 infant deaths per 1000 live births, that at least 120 infants >3 months of age would be residing in each sub- centre area at any,point in time. The Alwar district health authorities thought that immunization coverage in the PHC was ap- proximately 50% for infants. Based on this informa- tion, an LQAS sampling plan of n = 12:d = 3 was chosen. This plan has an a value of 0.05 when the proportion with immunization is 48% (i.e., the threshold for unacceptable coverage was 48%, and 95% of lots with this level of coverage would be identified as unacceptable with this LQAS plan). This sampling plan was used in 14 of the 27 subcentres. During the survey, it was realized that more time was required to visit the more remote subcentres. Therefore, an LQAS sampling plan of n = 11:d = 3 (which has an a value of 0.05 at a coverage level of 44%) was used in the remaining 13, more distant, subcentres. The lots were classified as acceptable or unacceptable on the basis of these sampling plans. Using stratified sampling theory and weights proportional to the population in each lot, the results of the samples from the lots were com- bined to obtain estimates of immunization coverage for all the lots combined (2-4, 8). Lists of all the households in the subcentre areas were available in "family survey registers" and "eligible couple registers" kept in the subcentres, in registers of surveys carried out by anganwadi (para- medical) workers, or in "economic registers" and "voter lists" kept at Block Development Offices. Prior to our study, the Chief Medical and Health Officer of Alwar District instructed all staff of the PHC and of its subcentres to be available with their eligible couple and child/mother immunization regis- ters. The registers kept at the subcentres are updated every year in April, and in a pilot study it was found that the eligible couple registers in Alwar district were relatively accurate and complete; however, in four subcentres, these registers were not up-to-date (three for 1990 and one for 1987). Child/mother im- munization registers were found to be grossly incom- plete. Therefore, the eligible couple registers were chosen to be used as the sampling frame. The required numbers of households needed in each subcentre were selected at random from the subcentres' registers of eligible couples. The survey- ors visited the first selected couple's household (HH) and enquired whether an eligible child in the survey age group lived there. If there had been more than one eligible child in a household, all would have been selected but there were no such households. Most subcentres served populations living in three to five villages; to reduce the amount of travel time between villages it was decided that if there was no eligible child in a selected household, a replacement would be randomly selected from the HH list for the same village. A small number of houses were found to be locked, or the mothers were not present during the survey. These households were discarded. All children aged 3-11 months, residing in a subcentre area, constituted the population (N) of that lot. The ages of infants were recorded by months of age groupings which were selected to permit the identification of the particular vaccines and doses the infants should have received according to the re- quirements and limitations of the Universal Immu- nization Programme (UIP) immunization schedule (Table 1). Birth certificates were not generally avail- able in the study PHC and age could not be verified from records for all the infants and children. How- ever, the mothers could usually provide the birth month of their child. Age classification was more of a problem with the EPI survey, as the children were older (12-23 months) than in the LQAS survey. In the LQAS survey, children were considered appro- priately immunized for their ages, if they had re- ceived the following immunizations at the time of the survey: Dose DPT1, OPV1, BCG DPT3, OPV3, BCG DPT3, OPV3, BCG, measles Age 3-5 months 6-10 months 11 months DPT = diphtheria-pertussis-tetanus; OPV = oral poliovirus vaccine WHO Bulletin OMS. Vol 74 1996270 Lot quality assurance sampling and EPI cluster sampling Table 1: Immunization schedule under the Universal Immunization Programme, India Beneficiary Age Vaccine No. of doses Infants 6 weeks OPV, DPT, BCGa 1 10 weeks OPV, DPT 1 14 weeks OPV,DPT 1 9 months Measles 1 Pregnant 16-36 weeks Tetanus toxoid 2b,c women of pregnancy a For institutional delivery, BCG should be given at birth. b Interval between 2 doses should not be less than 4 weeks. c Only one dose (booster) in subsequent pregnancy. For particular doses of vaccines to be valid, it was stipulated that DPT1 and OPV1 should not have been given before 6 weeks of age, and the interval between subsequent doses of DPT and OPV could not be less than 28 days. Measles vaccine should not have been given before a child was 270 days of age to be a valid immunization. Mothers of sampled children were assessed for tetanus immunization status prior to the birth of the index child. A record of two doses of tetanus toxoid (IT) with an interval of 4 weeks between doses (or a booster dose during the pregnancy) was accepted as evidence that the mother had been appropriately immunized. The second Ti dose in an initial series, or a booster dose, had to be given 4 weeks before the delivery of the child to be considered an effective, and thus valid, dose. EPI cluster survey. The 30-cluster sample survey was carried out in accordance with training materials provided by EPI. The technique has been described previously (2, 4).a Proof of immunization. Immunization cards were available for only 20% of the children and women surveyed. For those without cards, the surveyors ac- cepted health workers' records or convincing histo- ries of immunizations given by the mothers. Health workers and anganwadis accompanied the surveyors and were encouraged to bring their immunization records with them to the field. The lack of cards was common to both the LQAS and the EPI surveys, but the shorter period of recall for mothers' histo- ries of immunizations for infants in the LQAS survey reduced the problem with that survey. The presence of a typical scar was accepted as proof of BCG immunization. a Expanded Programme on Immunization. Training for mid- level managers. The EPI coverage survey. Unpublished document WHO/EPI.MLM, 91.10, 1991. Resources required. A detailed record was kept of the money and time required for the two surveys. All persons conducting the surveys were government officials who drew their salaries as well as travel and daily allowances to cover expenses during the survey period. Salaries were not considered in the calcula- tions of expenses. Results None of the 27 subcentre populations (lots) could be classified as having an acceptable level of immuniza- tion coverage among infants, and only three of them were classified as having an acceptable level of TT coverage of the mothers (Table 2). The LQAS data for the 27 lots were combined using standard statisti- cal methods (8) to obtain estimates of the overall immunization coverage among infants above 3 months of age and for 1T coverage of their mothers, as shown in Table 3. About a quarter (24%) of the infants were immunized appropriately for their age by the time they had reached 11 months; this propor- tion was similar (21%) for children in the EPI sur- vey. Full TI protection among the mothers of the infants was found to be 46% by LQAS and 57% in the EPI survey (Table 4). The coverage levels for individual doses of vaccines among 12-23-month-old children (but given before 12 months of age) sampled in the EPI survey are compared, in Table 4, with the coverage levels for the same vaccines among 11-month-old infants in the LQAS survey. Also shown for comparison are the estimates of TT coverage among the mothers. The time required and the expenses incurred in the two surveys are shown in Table 5. On average, 2 hours and 6 hours, respectively, were needed to com- plete the survey in a cluster and in a lot. The travel time averaged 60 minutes for a cluster and 90 min- utes for a lot. The estimated costs of the surveys were Rs 12000 (US$ 375) for the cluster survey, and Rs 19000 (US$ 595) for the LQAS survey, a ratio of 1:1.6. Table 2: Immunization coverage by LQAS in the Malakhera primary health centre. See text for explana- tion of n and d Threshold No. of No. of coverage subcentres subcentres n d (%) surveyed accepted Child 12 3 48 14 0 11 3 44 13 0 Mother 12 3 48 14 1 11 3 44 13 2 WHO Bulletin OMS. Vol 74 1996 271 J. Singh et al. Table 3: Combined immunization coverage for all the lots by LOAS Age group Immunizations (months) No. surveyed up-to-date % coverage Child 3-5 121 29 26.6 (20.4-32.9)a 6-10 132 39 27 (19.8-34.2) 11 58 14 24.1 (16.8-31.3)b All age groups 311 82 27.7 (23.2-32.1) Mother 311 139 46.1 (41.0-51.3) a Figures in parentheses are 95% confidence intervals. b No child in this age group was selected from 3 subcentres. The analysis pertains to 24 subcentres comprising a population of 146599. Table 4: Comparison of immunization coverage by 30- cluster sampling and LQAS in the Malakhera primary health centre EPI methodology LQAS 1 2-32-month-olds 1 1 -month-olds (n = 210): (n = 58): Vaccine and dose % coveragea % coveragea DPT1/OPV1 63.3 (55-71.6)b 71.4 (64.7-78.1)b DPT2/OPV2 51.4 (42-60.8) 62.8 (55.4-70.1) DPT3/OPV3 42.4 (33.1-51.7) 41.5 (33.4-49.6) BCG 40.0 (30.5-49.5) 36.4 (28.9-44.0) Measles 25.4 (15.5-34.9) 25.7 (18.2-33.1) Fully immunized 21.4 (11.8-31) 24.1 (16.8-31.3) Mothers: Age of children 0-11 months 3-11 months TT2/booster 56.7 (46.3-67.1) 46.1 (41.0-51.3) a The differences between coverage levels by the two sampling techniques were not statistically significant. b Figures in parentheses are 95% confidence intervals. Discussion The classification of the 27 subcentre populations of infants and their mothers using the LQAS plan was of little value in this particular application. The apparent homogeneity and low level of coverage among the subcentres would have required much larger n and d values to permit discrimination of the apparently small differences in coverage between the subcentres. In addition, the level of coverage in the mid-range (i.e., 48% and 44%), selected as the thresholds for unacceptable coverage, produce the poorest precision. Because all the subcentres were classified as unacceptable, the results did not allow interventions to be focused on subcentres that need assistance to reach higher levels of coverage. Even the finding that three subcentres were accepted for TT coverage is not very useful when the threshold for unacceptability is below 50%! In this study the combined results of the LQAS were perhaps the most useful. Table 5: Time and money spent on the surveys in the Malakhera primary health centre 30-cluster LQAS sampling Unit used in the survey Subcentre Cluster No. of subcentres or clusters 27 30 No. of villages visited 128 30 No. of villages visited per subcentre 4.7 1 or cluster Total households visited 1 795 700 Total eligible children visited 311 210 No. of households visited per eligible 5.8 3.3 child No. of households visited per 66 23.3 subcentre or cluster Total man-days required for the survey 150 90 Officer days 30 30 Driver days 30 30 Surveyor days 90 30 Time spent (hours) on the actual 162 60 survey (excluding travelling and training time) Average time spent on the survey per 6 2 subcentre or cluster (hours) Average time spent on travelling per 90 60 subcentre or cluster (minutes) Money spent (rupees)a Travelling and daily allowances: Officers 1 600 1 600 Driver 750 750 Surveyors 4500 1 200 Petrol/oil/lubricant 10000 7000 Contingency/minor vehicle repairs 2000 1 500 Total 18850 12050 a Rupees 32 = US$ 1 The estimated overall immunization coverage appropriate for age among infants was 28% (95% CI, 23-32%) from the LQAS results. It is notable that immunization coverage across the age subdivi- sions of the infants indicated no significant decline in that measure with the increase in the infants' age. In other words, coverage levels did not seem to increase in this area over a period of one year (see below). 272 WHO Bulletin OMS. Vol 74 1996 Lot quality assurance sampling and EPI cluster sampling A clear gap was evident between the re- ported coverage and the survey results, especially for measles vaccine. It has been shown that consistency between the reported coverage and survey results is good in areas with high immun- ization coverage, but poor in low coverage areas (9), perhaps due to overreporting of perform- ances. Another area of concern is a low retention of immunization cards (20%), perhaps because of poverty and low literacy rates, especially in females; these cards help in administering an ad- equate dose of vaccine at the proper time to eligible persons. Like the cause of immunization failure in an individual, these issues will need to be addressed if the goal of universal immunization is to become a reality. The children included in the LQAS survey and the 30-cluster sampling survey were not exact con- temporaries, being separated by a time interval. However, the coverage in 11-month-old children surveyed by LQAS can be compared with the results obtained by the EPI method, because the child coverage was not supposed to be affected during the intervening period (see above); addi- tional inputs were not provided, nor were efforts made during this period to increase the coverage in the primary health centre. As shown in Table 4, the comparable antigen-dose coverages for the children are remarkably similar in distribution and levels of coverage. Nevertheless, the characteristics of the mothers were largely similar. About 57% (95% CI, 46-67) of mothers were found to be fully immunized by 30-cluster sampling, compared with 46% (95% CI, 41-51) by stratified random sampling. The difference was not statistically significant. The results vindicate the ability of the EPI methodology to estimate the immunization coverage. LQAS was found to be more time-consuming than the EPI survey. While it took an average of about 3 hours to complete an EPI survey of one cluster (including travel time), more than 7 hours were needed to complete the LQAS survey in a subcentre population. This comes as no surprise since cluster surveys are known to be less costly than random sample surveys like the LQAS. However, the cost in time and travel associated with the use of LQAS could be reduced with a few simple modifications of the field procedures we used. For example, when a household selected for the LQAS was found not to contain a child, or the mother was not at home, a replacement household was randomly selected from the same village. This procedure re- quired selecting, locating and moving to the new household to check on the availability of an eligible infant and mother. It would be easier, less time- consuming, and there should be no bias intro- duced, if the surveyors were instructed to visit the next-nearest household (as in the EPI surveys) until they found one with an eligible infant and mother. This study shows that an LQAS survey con- ducted by external personnel travelling into the area is impractical for routine monitoring of immun- ization coverage. A monitoring plan using local staff to collect this information in the course of performing other duties in the subcentres would be much less costly. Routine monitoring using local staff to collect data over a period of time that corres- ponded with immunization schedule intervals could be developed, the senior staff being involved only in designing the sampling plan(s) and combining the results. As it is more time-consuming and costly, LQAS is not a good alternative to the current EPI methodology for independent evaluation of over- all immunization coverage in a large administrative area (e.g., in India, at district level and higher). Nevertheless, LQAS may be a useful method for routine monitoring of immunization programmes in small areas. The decision of the government to issue identity cards to all the voters in India may soon provide the basis for more accurate sampling frames in all the subcentres throughout the country. Medical officers and health supervisors regularly visit their areas to supervise field workers; these routine visits can also be used to evaluate the immunization programme with the LQAS method (4, 5), for which minimal training is required (4). The problem of the time needed to complete a survey in a subcentre could be solved by collecting the data over several visits. Simple tables for selecting an optimal sampling plan and for inter- preting the data can be prepared easily. LQAS has potential as a method to monitor and evaluate health programmes in small population subunits, especially when the characteristics to be measured are heterogeneously and disparately distributed among the subunits. Acknowledgements We are extremely thankful to Dr George Stroh Jr (Centers for Disease Control and Prevention, Atlanta, GA, USA) for technical guidance in developing the LQAS technique with special reference to India and for extensive revision of the earlier version of the manuscript. We also thank the Chief Medical and Health Officer and staff of Alwar district for their cooperation and support to undertake the study. The assistance of NICD staff, Alwar branch, in the collection of data is also acknowledged. WHO Bulletin OMS. Vol 74 1996 273 J. Singh et al. Resume Evaluation comparative de la couverture vaccinale par la methode d'echantillonnage pour l'assurance de la qualite des lots et par la methode d'echantillonnage par grappes dans un centre de soins de sante primaires en Inde La couverture vaccinale des nourrissons, des en- fants et des femmes de la r6gion desservie par un centre de soins de sant6 primaires (CSP) du Rajasthan a 6t1 6valuee par la m6thode d'6chantil- lonnage pour I'assurance de la qualite des lots (lot quality assurance sampling: LQAS) et par la m6thode d'6chantillonnage par grappes recom- mand6e par le programme 6largi de vaccination (PEV) de l'OMS (m6thode des 30 grappes). La m6thode LQAS a servi a classer 27 sous-unit6s de population mutuellement exclusives, constitu6es par les r6sidents des diff6rentes subdivisions sanitaires de la r6gion, en fonction du niveau de couverture vaccinale (acceptable ou inacceptable) des nourrissons et de leurs meres. Les r6sultats obtenus par la m6thode LQAS dans les 27 sub- divisions ont 6galement 616 combines pour ob- tenir une 6valuation globale de la couverture de 1'ensemble de la population desservie par le CSP et ces r6sultats ont 6t6 compar6s a ceux de la m6thode d'echantillonnage par grappes du PEV. La couverture vaccinale des nourrissons, telle qu'6valu6e par la methode LQAS, n'a 6t6 con- sid6r6e comme acceptable dans aucune des subdivisions; seules trois d'entre elles pr6sentaient un niveau de couverture acceptable pour l'ana- toxine t6tanique (AT) chez les femmes. Pour 1'ensemble de la population desservie par le CSP, les r6sultats combin6s des diff6rents sondages LQAS montrent qu'un quart des nourrissons 6taient vaccin6s correctement pour leur age et que 46% des meres l'etaient contre le t6tanos. En ce qui concerne les enfants, les groupes d'age et les periodes de vaccination n'6taient pas les m6mes pour la methode LQAS et la methode de sondage par grappes; par contre, les caract6ristiques des meres 6taient dans 1'ensemble comparables. Le taux de couverture vaccinale de ces dernieres contre le t6tanos a ete estim6 a environ 57% (IC 95%, 46-67) par la m6thode des 30 grappes, contre 46% (IC 95%. 41-51), par la m6thode d'6chantillonnage al6atoire stratifi6e. La diff6rence n'6tait pas statistiquement significative. La collecte des donnees sur le terrain par la methode LQAS a pris environ trois fois plus de temps que par la m6thode du PEV et elle est revenue 60% plus cher. Le taux de couverture vaccinale etant uniformement faible dans toutes les subdivisions, la situation ne se pretait pas a I'application de la m6thode LQAS, de sorte que les resultats obtenus n'ont pas ete particulierement utiles. Toutefois, si cette methode avait ete ap- pliquee par le personnel local dans une region ou la couverture aurait 6t6 globalement elev6e, mais relativement variable d'une subdivision a I'autre, sa mise en oeuvre aurait ete moins coOteuse et elle aurait fourni des informations utiles. References 1. Henderson RH, Sundaresan T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bulletin of the World Health Organization, 1982, 60: 253-260. 2. Lemeshow S, Robinson D. Surveys to measure pro- gramme coverage and impact: a review of the meth- odology used by the Expanded Programme of Immunization. World health statistics quarterly, 1985, 38: 65-75. 3. Lemeshow S, Stroh G Jr. Quality assurance sampling for evaluating health parameters in developing coun- tries. Survey methodology, 1989, 15: 71-81. 4. Lemeshow S, Stroh G Jr. Sampling techniques for evaluating health parameters in developing countries. A working paper. Washington DC, National Academy Press, 1988. 5. Singh J et al. Concurrent evaluation of immunization programme by lot quality assurance sampling. Joumal of tropical pediatrics, 1995, 41: 215-220. 6. Lanata CF et al. Lot quality assurance sampling in health monitoring. Lancet, 1988, 1: 122-123. 7. Dodge HF, Romig HG. Sampling inspection tables. Single and double sampling, 2nd ed. New York, John Wiley, 1959. 8. Karmel PH, Polasek MP. Applied statistics for econo- mists, 3rd ed. New York, Pitman, 1970: 170-182. 9. Biellik R et al. Review of the universal immunization programme. Country overview. A joint report by the Government of India, WHO & UNICEF. New Delhi, Ministry of Health and Family Welfare, 1992. 274 WHO Bulletin OMS. Vol 74 1996