Immunization coverage surveys: methodological studies in Indonesia.

Bulletin of the World Health Organization, 65 (6): 847-853 (1987) © World Health Organization 1987 Immunization coverage surveys: methodological studies in Indonesia S. K. LWANGA1 & N. ABIPROJO2 Methodological studies of immunization coverage in rural and urban areas of Indonesia were carried out to quantify the relative costs ofsurveys using the standard EPI (WHO's Expanded Programme on Immunization) methodology or using seven randomly selected households within each cluster. Although the latter method gave more precise estimates, these surveys were 1.4-2.5 times more expensive. Furthermore, although in some areas immunization coverage had reached a very high level, children were not being immunized according to established national schedules. The results also indicate the potential of using "lot quality assurance" survey techniques as a management tool in immunization programmes. INTRODUCTION Managers of expanded immunization programmes are periodically interested in estimating the number of children who have been immunized. The method- ology most often used is that recommended by the Expanded Programme on Immunization (EPI) of the World Health Organization (1, 2).a Briefly, im- munization coverage surveys by the EPI method are carried out by selecting 30 communities, or "clusters", with probability proportional to the population size, and subsequently choosing one household at random within each cluster as a starting point to collect immunization data on seven children within a target age group in that cluster. Data are then collected by moving from one household to the next until information on the required number of children has been obtained. This method, which is easy to use in the field, nevertheless deviates from standard statistical procedures in that the seven children in each cluster are selected from "neighbourhood" households. The error thus introduced was investi- gated by Lemeshow et al. (3), who demonstrated, using computer simulations, that with the EPI methodology the true level of immunization can be estimated to ±10 percentage points. They also showed that if within the survey clusters there are I Statistician, Epidemiological and Statistical Methodology, World Health Organization, 1211 Geneva 27, Switzerland. Requests for reprints should be sent to this author. 2 Directorate General of Communicable Disease Control and Environmental Health, Ministry of Health, Jakarta, Indonesia. 0 WHO EXPANDED PROGRAMME ON IMMUNIZATION. Training manual for mid-level managers: evaluate immunization coverage. Unpublished document. "pockets" of either a very low or a very high pro- portion of immunized children, the estimates of immunization coverage may not lie in the expected range of the true values. While the standard immunization coverage surveys aim at giving an overall estimate of the children who have been covered by immunization programmes, individual health workers in charge of small areas have had no straightforward way of monitoring activities in their areas. In industry, lot quality assurance (LQA) sampling has been employed as an aid to making decisions about the quality of "batches" or "lots" of products. The procedure in- volves random selection of a small number of items from production batches, which are then examined for quality. A decision about the quality of a par- ticular batch is based on the probability that the number of defective items in the batch is less than or equal to a predetermined level. In this respect, an analogy can easily be drawn between an im- munization programme and an industrial production process. Objectives of the study As higher and higher immunization coverage levels are reported, attention needs to focus on the "quality" and impact of immunization programmes. It thus becomes important to ensure that immun- izations are administered according to established schedules. The computer simulation studies reported by Lemeshow et al. (3) did not quantify the differences in "costs" between the standard EPI surveys and 4839 -847- S. K. LWANGA & N. ABIPROJO those based on seven randomly selected starting households. Survey "cost" is an important factor in deciding whether a survey should be carried out, let alone carried out properly. The theoretical basis of LQA methods is well known but their application to health care delivery programmes has not been widely reported. We there- fore carried out a field study that had the following objectives: - to assess whether immunization schedules were being adhered to in the study areas; -to quantify the differences in survey time and cost for a rural and an urban setting between the standard EPI survey method and the statistically rigorous approach, whereby seven starting households are selected randomly within each cluster; and -to assess the applicability of LQA techniques as a managerial tool in immunization programmes. The results are reported here. MATERIALS AND METHODS Selection of study area, organization, and methodology Two areas in Indonesia were selected for the study: the Gianyar kabupaten (district) of Bali and the timur (east) municipality of Jakarta. In both places reliable population data were available from the 1980 census and, in Bali, these had been continually updated through the regular reporting of births, deaths, and migrations. Gianyar district was selected because it is rural, has a fairly good immunization reporting system, up-to-date household lists, and is logistically convenient. Correspondingly, the timur municipality of Jakarta was chosen because, unlike the other four municipalities in the city, no immunization coverage survey had been carried out there in the 2 years before the study commenced. Both types of surveys were carried out in each of the two study areas during September and October 1986. One survey examined the standard EPI methodology (referred to subsequently as the "EPI" method). However, in the second survey (referred to subsequently as the "SRS" method) seven house- holds were selected at random within each cluster as starting points for identification of seven children in the target age group whose status on immunization was recorded. Movement from household to house- hold in search of eligible children follows the same procedure as for the EPI method; however, while in the latter method the search for children among neighbourhood households continues till seven children are identified, in the SRS method a new randomly selected starting household is used for each child. Table 1. Stages followed in selecting households for the surveys Level Bali Jakarta 1 Kabupaten (district) Municipality 2 Desa (village) Kelurahan (village) 3 Banjar (village subdivision) Rukan warata (RW)' 4 Starting house(s) Rukan tetangga (RT)b 5 - Starting house(s) Made up of groups of Rukan tetangga. b Group of 20-40 households. Gianyar district is adjacent, and lies north-east of Bandung district where the Bali provincial capital of Denpasar is located. Thirty villages (desa) were selected out of the 56 in Gianyar, with probability proportional to their population sizes. In contrast, in Jakarta the 30 villages (kelurahan) were selected out of 59 in the timur municipality, also with probability proportional to population size. For each survey area, one day was devoted to train- ing the interviewers and supervisors in data collection procedures. Since the same interviewers collected data for both methods, half of them were randomly selected to start with the EPI method and the other half with the SRS. The procedure followed in selecting starting house- holds is shown schematically in Table 1. In Gianyar, levels 1 and 2 were chosen centrally, with the inter- viewers using the household lists for each desa (village) to select the starting house(s) with prob- ability proportional to the banjar (village sub- division) population size. However, in Jakarta, levels 1-3 were chosen centrally, with the interviewers completing the selection in the field using rukan tetangga (residential community) household records. Data collection The standard EPI protocol for immunization cover- age surveys was used with suitable adaptation to the immunization programme in Indonesia. In addition, the amount of time spent collecting data for each child and on the total activity of each day was recorded. Dates of birth and of immunization were also reg- istered by day, month, and year (if the exact day of the month was not known for either of these dates, the fifteenth of the month was assumed for the purposes of the analysis). If a child was known to have received a particular immunization but the date had not been recorded on the immunization record card, a " +" was used instead of the immunization date. Data collection took 2 days in Gianyar and 4 days in 848 IMMUNIZATION COVERAGE STUDIES IN INDONESIA Jakarta. Normally children are included in a survey if their dates of birth fall within a specified range, based on the survey date and target age group. For surveys lasting more than 1 day, the range of acceptable dates therefore shifts daily. However, in the present surveys the range of acceptable dates was fixed as the date of the first day of the survey in each area in order to simplify the procedure for the interviewers. The total costs included in the analysis are those covering the daily allowances for the supervisors, interviewers, guides, and drivers, the organizational expenses during the field exercise, and the cost of supplies (petrol, clipboards, pencils, erasers, etc.). Excluded from the analysis are pre-survey expenses such as air travel costs to Bali and all costs for the WHO staff member who participated in the surveys. Data organization and analysis Children's ages were calculated, in months, with respect to the first day of the survey (20 September 1986 in Gianyar, and 30 September 1986 in Jakarta). The age and the date of immunization of each child were then used to compute the age at immunization in months. Data were then analysed for: -the immunization status as recorded on the im- munization cards and according to the established schedule; -the time spent in each cluster and the cost for each survey method; and -the possibility of using LQA surveys (based on SRS survey results) as a managerial tool in immunization programmes. RESULTS Overallfindings In Gianyar, 207 children aged from 15 months to less than 24 months (on the first day of the survey) had their immunization status properly recorded using the EPI survey method, while 209 children were successfully recorded by the SRS method. Using the EPI method, we estimated that 75.4% of the children had received eight immunizations (standard error, 4.3%). In contrast, use of the SRS method gave an estimate of 86.6% (standard error, 2.7%). The corresponding results for the survey in Jakarta were 207 children properly surveyed by the EPI method, with an overall complete immunization rate of 25.1% (standard error, 4.4%) The SRS method, which properly surveyed 209 children, gave a rate of 24.4% (standard error, 3.0%). The results for the individual vaccines are shown in Table 2. Immunization results according to established schedule The established immunization schedule for Indo- nesia is as follows: BCG from birth to 14 months; Table 2. Immunization coverage results for four vaccines for children aged 15-23 months, Gianyar and Jakarta, Indonesia 1986 Gianyar Jakarta BCG DPT-3° OPV-3b Measles BCG DPT-3' OPV-3 b Measles EPI method No. immunized 197 (0.95)c 168 (0.81) 170 (0.82) 165 (0.80) 165 (0.80) 94 (0.45) 94 (0.45) 56 (0.27) Standard error 0.019 0.038 0.038 0.037 0.034 0.051 0.050 0.046 No. immunized according to schedule 176 (0.89) 105 (0.63) 113 (0.66) 98 (0.59) 99 (0.60) 46 (0.49) 49 (0.52) 29 (0.52) Standard error 0.026 0.038 0.036 0.045 0.067 0.067 0.064 0.075 SRS method No. immunized 197 (0.94) 181 (0.87) 181 (0.87) 187 (0.89) 173 (0.83) 99 (0.47) 99 (0.47) 57 (0.27) Standard error 0.021 0.027 0.027 0.024 0.027 0.037 0.038 0.033 No. immunized according to schedule 197 (1.00) 118 (0.65) 134 (0.74) 127 (0.68) 173 (1.00) 58 (0.59) 59 (0.60) 38 (0.67) Standard error 0 0.049 0.031 0.042 0 0.061 0.058 0.070 ' Three doses of diphtheria-pertussis-tetanus vaccine. b Three doses of oral polio vaccine. ' Figures in parentheses are proportions. 849 S. K. LWANGA & N. ABIPROJO Table 3. Survey times in Gianyar (30 clusters each for the EPI and SRS methods) and in Jakarta (30 and 29 clusters, respectively, for the EPI and SRS methods)a Time (minutes) EPI method SRS method Total Non-interview Interview Total Non-interview Interview Gianyar Mean 208.5 139.2 69.3 287.9 204.4 83.8 Variance 7107.65 5631.14 675.61 4485.98 5292.82 640.85 Minimum 87 27 28 150 77 25 Maximum 450 360 127 430 355 135 Median 189 119.5 68.5 295 197.5 80 Jakarta Mean 102.3 46.4 55.8 266.7 207 59.8 Variance 2040.73 902.78 913.74 9621.23 9724.03 819.49 Minimum 41 5 22 115 72 26 Maximum 210 127 165 522 461 150 Median 92.5 42 44.5 255 194 54 In one cluster the recorded times were so erroneous that they were omitted from the analysis. diphtheria-pertussis-tetanus (DPT) and polio from 2 to 14 months, with at.least 4 weeks' interval be- tween two immunizations; and measles from 9 to 14 months. Analysis of the data collected from the surveyed children for compliance with the established immunization schedule indicated that, based on the EPI results, in Gianyar 45% of the 156 children who received eight immunizations were properly im- munized (standard error, 4.3%). The SRS survey also indicated that 45% of the 181 children were properly immunized (standard error, 3.8%). In Jakarta, use of the EPI method indicated that of the 52 children with eight immunizations, 19(37%; standard error, 6.4%) were fully and properly immunized. The SRS method showed that 26 of the 51 children with eight immun- izations (51%; standard error, 7.0%) were properly immunized. The results for the individual antigens are shown in Table 2. Survey duration The mean times spent on all survey activities in each cluster are summarized in Table 3. Since the recording of the time spent on preparing for field work before the start of the day's activities was un- reliable, it was omitted from the analysis. The total time was therefore measured from the contact with the first child in a given cluster to the end of the data collection in that cluster. It should be noted that time spent in locating the first eligible child is therefore not included in the analysis. In both areas the SRS surveys had significantly greater non-interview time distributions than the EPI surveys (P<0.01 using the Kolmogorov-Smimov two-sample test). Although interview times were longer in the SRS method, the time distributions were not significantly different from the EPI method. Survey cost It was not possible to keep track of the survey financial costs separately for each method; costs are Table 4. Relative costs of the Indonesia, 1986 EPI and SRS methods, Proportionate cost Survey cost EPI SRS Survey time per hour Survey area method method (hours) (US $)b Gianyar 42.0 58.0 248.2 11.4 Jakarta 28.4 71.6 180.1 9.6 a Based on total survey time for each survey. b US$1 -Rp 1150. 850 IMMUNIZATION COVERAGE STUDIES IN INDONESIA therefore related to the amount of time spent on either method in each survey. These amounted to roughly US$ 11.4 per survey man-hour for Gianyar and US$ 9.6 per man-hour for Jakarta. Table 4 sum- marizes the relative costs for the two methods and the costs per unit time in the two survey areas. Lot quality assurance The distribution of the results for the 60 clusters surveyed with the seven random starting points within each cluster in Gianyar and Jakarta are shown in Table 5. From the Table it can be seen, for example, that if in a group of seven randomly selected children only three are immunized, the group can be classified with 95 % confidence as belonging to a population of children with an immunization coverage between 13% and 66%. Also shown are the likely immuniz- ation coverage levels, with 95% confidence, of the 60 clusters. Table 5. Distribution of clusters by coverage results (SRS method), Indonesia, 1986 Expected coverage No. immunized (%) per cluster of seven children Gianyar Jakarta Minimum Maximum 0 0 4 - - 1 0 10 <1 35 2 0 10 5 53 3 1 3 13 66 4 1 3 23 77 5 6 0 34 87 6 11 0 48 95 7 11 0 65 100 Total 30 30 - - I Maximum and minimum refer to the 95% confidence limits. DISCUSSION Immunization results and perfornance The areas studied did not correspond to the "worst case" category of the computer simulation described by Lemeshow et al. (3). The most recently available data for the two areas surveyed (Table 6) show consistently lower immun- ization coverages for all antigens than those found in the surveys reported here. In Gianyar, where almost all the immunization is carried out through EPI, and reports are believed to be fairly complete, the lower values may be due to an overestimate of the target population, which is based on the assumption that 2.6% of the 323 080 people living in the area are children in the target age group. The results of the SRS method gave a full immunization rate of 86.6% (95% confidence range: 81-92%). This corresponds to an estimated proportion of the popu- lation of children in the target age group of approx- imately 1.9% (if the lower confidence limit of 81 % is taken to be closer to the true coverage value and the 4897 reported measles immunizations are used (4897/(0.81 x323 080)). The results of the surveys show that the majority of the children are not vaccinated according to the national immunization schedule. Failure to follow the established schedule is also independent of coverage levels. Of the 337 children with eight immunizations in Gianyar (combining the results of the SRS and EPI surveys, since there were no overlapping clusters), only 152 (45.1 %) were immunized according to schedule. In Jakarta, 45 out of 103 (42.9%) children were correctly immunized. Table 6. Comparison of reported immunization data and EPI method survey results, Indonesia 1986 Children immunized (%) Target Period population BCG DPT-3a OPV-3b Measles Gianyar 1985 8400 59.4 65.6 64.9 58.3 1986 (survey) 95.2 81.1 82.1 79.7 Jakarta April 1985-March 1986 52 079 57.8 34.1 35.2 18.7 1986 (survey) 80.0 45.4 45.4 27.1 Three doses of diphtheria-pertussis-tetanus vaccine. b Three doses or oral polio vaccine. S. K. LWANGA & N. ABIPROJO Survey duration and cost One of the main objectives of the studies was to quantify the differences in inputs for surveys carried out according to the standard EPI methodology and those using the more rigorous SRS approach. A major difference was expected in the time spent in non- interview activities, particularly in moving between starting points, and this was clearly shown by the results. Interview times were consistently lower for the EPI than the SRS surveys in both areas, although this was not statistically significant. Since EPI surveys are conducted among neighbouring households, inter- viewers are likely to spend less time in self- introductions and elaborate welcomes. In contrast, for SRS surveys the search for each child means a fresh start in a new neighbourhood, and elaborate introductions and welcomes have, therefore, to be repeated for every child. Although the estimates obtained with the SRS surveys were more precise than those with the EPI surveys, they were more costly. When the costs were apportioned to each type of survey according to the time spent, they were 1.4 and 2.5 times more costly in Gianyar and Jakarta, respectively, for the SRS versus the EPI methodology. Nevertheless, the relatively higher precisions of the estimates do not justify the indiscriminate use of the SRS method, especially in areas similar to those studied in Indonesia. Lot quality assurance Use of the lot quality assurance (LQA) sampling methodology can assist in making decisions on the likelihood that a given area has a predetermined immunization rate. Application of this method to our results shows that in Gianyar 37% of the clusters have an immunization coverage value of not less than 65 %, while a simnilar proportion has a minimum value of 45 %. The remaining clusters have coverage values of not less than 20%. The survey results for Gianyar based on the SRS method give an overall coverage rate of 87% (95% confidence interval, 81-92%). In Jakarta the overall coverage was 24% (95% con- fidence interval, 19-30%). No cluster in the Jakarta survey would have been classified by LQA as having a minimum coverage value of more than 25 %. If, therefore, an immunization programme manager had wanted to make "spot" checks of levels of immun- ization in selected areas in the municipality, the decision would have been that the coverage values were less than 25 %, in agreement with the conclusion that would have been ascertained through an elaborate survey. While interview time is the same for both SRS and EPI methods, the time spent on travelling is greater for the SRS method. Both this aspect and the need to establish sampling frames for the random selection of the starting households are the additional "costs" for undertaking LQA surveys. Although in the full immunization coverage survey the establishment of sampling frames may seem difficult, usually because the interviewers involved are not the regular health workers in the areas to be surveyed, the task need not be too difficult for health workers in their own areas. Lot quality assurance surveys do not need to be completed in one day, as immunization surveys usually are, but can be incorporated into the regular activities of health workers. For this purpose, house- holds for inclusion in the survey would be pre- determined (randomly) and visited in the course of regular activities within a reasonable period of time. As Lemeshow & Stroh (4) have pointed out, while the individual results are mere benchmarks on the level that could have been achieved, the cumulative results collected over a service area could eventually give an estimate of the rate in that area. Before immunization programme managers can be expected to use LQA methods as a monitoring tool, simple guidelines have to be prepared for them. These should explain how the surveys should be carried out, how the results should be interpreted and used, and the extent of their limitations. CONCLUSIONS The results we have reported established the following: -A number of children in the study areas are not being vaccinated according to established and recom- mended schedules. The EPI coverage surveys cur- rently concentrate on estimating coverage rates for the individual doses of vaccines. As national expanded immunization programmes become well established, however, it will be important to ascertain the proportion of doses of vaccines administered between 1 week before and 6 weeks after the dates of immunization prescribed by the national schedules. All immunization coverage surveys, not only in areas with established high coverage figures, should, therefore, include an examination of the adherence to recommended schedules. -Immunization coverage surveys carried out using the standard EPI method are cheaper than those done "correctly" and which give more precise estimates. In order to justify the use of seven randomly selected households as starting points in the search of the target children, the survey concerned would have to be conducted in a well-organized community of high population density where a good communication network exists. 852 IMMUNIZATION COVERAGE STUDIES IN INDONESIA 853 - Individual clusters can be correctly classified according to their immunization level using LQA methods. Possible use of these methods in immuniz- ation programmes as a programme implementation management tool should be further investigated. ACKNOWLEDGEMENTS This work was supported financially by the Expanded Programme on Immunization of the World Health Organization, Geneva. Generous assistance was also given by the Bali Provincial and Gianyar kabupaten health officials, Jakarta timur municipality officials, the Field Epidemiology Training Programme, and the Ministry of Health, Jakarta. RESUME INDONESIE: ETUDE DE LA METHODOLOGIE APPLIQUEE AUX ENQUETES DE COUVERTURE VACCINALE Des dtudes sur les methodes d'enquete relatives a la couverture vaccinale ont e effectuees dans une zone urbaine et une zone rurale d'Indondsie avec, pour objectifs: v6rifier si les vaccinations sont effectuees conform6ment aux calendriers etablis; quantifier dans ces deux zones les differences quant a la duree de l'enquete et a son cofit, suivant que l'on applique la methode d'enquete classique du PEV ou une methode statistiquement "correcte", dans laquelle on fait appel au depart a la selection aleatoire de 7 foyers dans chaque grappe (methode dite "SRS"); estimer quelle est l'applicabiite des techniques d'assurance de la qualit6 des lots (AQL) en tant qu'outil de gestion dans les programmes de vaccination. Les r6sultats ont montre que la majorite des enfants n'etaient pas vaccin6s conformement au calendrier national de vaccination, et que cette impossibilite a suivre le calendrier etabli 6tait inddpendante du degre de couverture. A l'ayenir, lors des enquetes du PEV sur la couverture vaccinale, il faudra evaluer quelle est la proportion de doses de vaccin administr6es selon les calendriers de vaccination prescrits au niveau national. La duree des entretiens 6tait uniformement plus courte dans 1'enquete du PEV que dans 1'enquete SRS, mais sans que cette diff6rence soit statistiquement significative. Alors que les enquetes SRS ont donn6 des estimations plus precises que les enquetes du PEV, elles ont ete plus co&iteuses: lorsqu'on a rapporte le co&t au temps passe a enqueter, il s'est avere que les enquetes SRS etaient 1,4 fois plus co&teuses que les enquetes du PEV en region rurale et 2,5 fois plus en region urbaine. Compte tenu de ce coOit plus 6leve, la pr6cision legerement meilleure des estimations obtenues nejustifie pas que l'on emploie sans discernement la methode SRS, surtout dans des endroits du type de ceux 6tudies. L'exploitation des rdsultats de l'enquete SRS pour 6valuer l'applicabilite de la m6thode d'echantillonnage en vue de l'assurance de la qualite des lots, a montre que le degre de couverture vaccinale dans toutes les grappes avait 6t6 correctement classifie. Avant que les directeurs des pro- grammes de vaccination puissent utiliser les m6thodes d'AQL comme outil de surveillance, il faudra preparer des directives simples expliquant la maniere de mener les enquetes et d'interprdter et utiliser leurs resultats, et quelles sont leurs limites. REFERENCES 1. HENDERSON, R. H. & SUNDARESAN, T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bulletin ofthe World Health Organization, 60: 253-260 (1982). 2. LEMESHOW, S. & ROBINSON, D. Survey to measure programme coverage and impact: a review of the methodology used by the Expanded Programme on Immunization. World health statistics quarterly, 38: 65-75 (1985). 3. LEMESHOW, S. ET AL. A computer simulation of the EPI survey strategy. International journal of epi- demiology, 14: 473-481 (1985). 4. LEMESHOW, S. & STROH, G. Sampling techniques for evaluating health parameters in developing countries. Washington, DC, National Academy of Sciences (In press).