Field evaluation of vaccine efficacy.

Bulletin of the World Health Organization, 63 (6): 1055-1068 (1985) © World Health Organization 1985 Field evaluation of vaccine efficacy* WALTER A. ORENSTEIN,' ROGER H. BERNIER,' TIMOTHY J. DONDERO, ALAN R. HINMAN,' JAMES S. MARKS,3 KENNETH J. BART,' & BARRY SIROTKIN' This paper describes the epidemiological techniques available for measuring vaccine efficacy and recommends a practical approach to their use. Many of the examples relate to measles vaccine, the efficacy of which was tested by the techniques described, although the methods are applicable to other vaccines as well. The main advantages anddisadvantages of the techniques are indicated. FIELD EVALUATION OF VACCINE EFFICACY The effectiveness of a vaccine to prevent disease depends on the vaccine being potent and on its proper administration to individuals capable of responding. Useful techniques are available to test the potency of vaccines and the response of the host. Potency testing is important in monitoring the production of vaccines (to maintain stipulated standards), and their transport through the "cold chain". In the latter instance, vaccines from the field are retrieved and tested to ensure that they have not lost their potency. Serological studies can also be used to determine a vaccine's efficacy (1). Seroconversion is useful to measure the induction of an immune response in the host and, in the absence of disease, indicates the persistence of antibody and immunity. Before begin- ning an immunization programme, these studies can help in identifying appropriate target groups for vaccination. Seroprevalence studies monitor the prevalence of antibodies due to disease in the popu- lation and indicate the pattern of occurrence of disease. Seroconversion studies are particularly useful in choosing the appropriate age for vaccination (2). By knowing the age distribution of measles cases and age-specific seroconversion rates, for example, estimates of the number of preventable cases using different vaccine policies can be derived (3). These two techniques (vaccine potency testing and * Requests for reprints should be addressed to Dr W. A. Orenstein, Technical Information Services, Center for Prevention Services, Centers for Disease Control (CDC), Atlanta, GA 30333, USA. ' Division of Immunization, Center for Prevention Services, CDC. 2 International Health Program Office, CDC. 3 Division of Nutrition, Center for Health Promotion and Education, CDC. serological testing) can play useful roles in the establishment and execution of immunization pro- grammes. However, since these procedures depend on laboratory support and may be expensive, it is not feasible to carry them out under all circumstances. Moreover, the use of seroconversion as an indicator of vaccine efficacy only measures the efficacy under relatively controlled conditions during a short period, since pre- and post-immunization sera must be col- lected and the vaccinator is aware that these tests are being done. This may not be possible under field conditions, such as in an integrated immunization programme where many different immunization centres and vaccinators are involved. The success of vaccinations performed under field conditions can be realistically assessed by measuring the protection against the disease by epidemiological means. This can be done without laboratory support. Because of the ease of this technique it can be very useful, particularly when the disease occurs in vaccinated individuals and there is doubt about the effectiveness of the vaccination programme. This problem becomes more prominent as vaccine coverage increases, because there will be more cases of illness occurring in vaccinated persons, even when the vaccine efficacy is high (4). When the vaccine efficacy is found to be lower than expected, detailed investigations should be carried out to determine the causes and take corrective action. The techniques described below have been used in the field evaluation of measles vaccine efficacy, but they are applicable to other vaccines as well. For those interested in detailed aspects of the methodology, particularly regarding potential biases and how they may be anticipated and corrected, a more detailed report has been published by the WHO Expanded Programme on Immunization.' a The field evaluation of vaccine efficacy (unpublished WHO document EPI/GEN/84/10). 4617 1055- 1056 W. A. ORENSTEIN ET AL. CALCULATION OF VACCINE EFFICACY: GENERAL PRINCIPLES Vaccine efficacy is measured by calculating the incidence rates (attack rates) of disease among vaccinated and unvaccinated persons and determin- ing the percentage reduction in the incidence rate of disease among vaccinated persons compared to un- vaccinated persons. The basic formula is: VE=ARU-ARV x 100 ARU where VE = vaccine efficacy, ARU = attack rate in the unvaccinated population, and ARV = attack rate in the vaccinated population. For example, if the vaccine were totally effective, there would be no disease in the vaccinated popu- lation and the calculation would simplify to: ARU-0x 100= 1000°ARUxlOl% By contrast, if the vaccine had no effect at all, ARU would equal ARV and the calculation would simplify to: ARU xlO0% In practice, vaccines are neither perfectly effective nor totally ineffective. Measles vaccine, for example, is 80-95% effective when appropriately administered (1, 5-8). The ideal vaccine efficacy study is a clinical trial starting with persons susceptible to disease. In a double-blind randomized placebo controlled trial, half the subjects receive vaccine and half receive placebo. To calculate the vaccine efficacy, both groups are followed prospectively to determine the attack rates for disease in vaccinees and non- vaccinees. This type of study is generally not possible after a vaccine has been licensed because, when the vaccine is of proven benefit, the use of a placebo is unethical. In most countries today, measles vaccine has been used in a proportion of the population. These vaccinees are not a randomly selected group and their susceptibility prior to vaccination is often unknown. None the less, vaccine efficacy studies are still possible by reducing biases to a minimum and recreating as closely as possible the "ideal" conditions of the prospective clinical trial. Four major factors affect most epidemiological studies of vaccine efficacy. (1) Case definition. It is important that a uniform definition of cases should be developed and applied to all individuals in the study. This definition should be as specific as possible. Laboratory confirmation of at least some cases can help demonstrate the accuracy of the case definition. A useful clinical case definition for measles is: an illness with a generalized rash of 3 or more days duration, fever () 38.3 °C), and any one of the following -cough, coryza, or conjunctivitis.b When all three criteria are met, the illness is likely to be measles. (2) Case ascertainment (case detection). It is important to ensure that efforts to detect cases among vaccinated and unvaccinated populations are equal. Surveillance surveys of the total population, in which investigators go from door to door using a clinical case definition to find cases, give the least biased estimate of vaccine efficacy. (3) Vaccination status determination. Vaccination status must be determined accurately and, whenever possible, based on a recorded date of vaccination. If the records of many vaccinees are lacking, the vaccine efficacy calculations may be biased. Definitions of vaccination status will depend on the actual type of investigation used. In general, persons can be considered as vaccinated against measles if they received vaccine on or after the minimum recom- mended age for vaccination and at least 14 days prior to the onset of disease or of an outbreak. Persons who received vaccine prior to the recommended age should not be classified as unvaccinated but should be classified in a separate category. Persons vaccinated in control clinics during an outbreak should be classified according to their vaccination status prior to the outbreak. Persons of unknown vaccination status or with an incomplete series of vaccinations should be excluded from the calculations. (4) Comparability of exposure. Vaccine efficacy should be measured under conditions where both vaccinees and non-vaccinees have an equal chance of exposure to measles. This is most likely to be the case when the incidence rate of the disease is relatively high. SPECIFIC METHODS Screening A preliminary estimate can easily indicate whether vaccine efficacy is within expected limits. For measles, an attack rate of greater than 10% in vaccinated individuals immediately suggests the need for further evaluation since maximum efficacy will be less than 90%. (Under conditions of 90% efficacy, 10% of the vaccinated population is susceptible; b Provisional guidelinesfor the diagnosis and classification ofthe EPI target diseasesfor primary health care, surveillance and special studies (unpublished WHO document EPI/GEN/83/4). FIELD EVALUATION OF VACCINE EFFICACY therefore less than 100o of persons would be expected to become ill). However, an attack rate of < 10% among vaccinated persons by itself does not mean the vaccine is effective; comparison with attack rates in the unvaccinated group is necessary. In many situations, attack rates among the vac- cinated and unvaccinated will not be known with precision. However, vaccine efficacy can be estimated from other available information. The vaccine efficacy equation has been formulated as follows: PCV = PPV - (PPV x VE) 1- (PPV x VE) where PCV = the proportion of cases occurring in vaccinated individuals; PPV = the proportion of the population vaccinated; and VE = vaccine efficacy (J.M. Kobayoshi & J.P. Brennan, personal com- munication, 1980). When any two of the three variables are known, the third can be calculated (4). Fig. 1 shows curves generated from this equation. These curves indicate the theoretical proportion of cases that will have a vaccination history in a given setting for specified levels of vaccine efficacy. The curves do not predict the occurrence of an outbreak in any given set of circumstances, but do show the expected proportional distribution of cases by vaccination status if an outbreak should occur. By knowing or estimating the proportion of cases occurring in vaccinated persons and the proportion of the population at risk that is vaccinated, an estimate 100 PPV-(PPV X 100 1-(PPV X VE) ___A l -- <~~~~~~~IZAII 80 4~~/ y/ 70 - I-I- I -I-20 - -.~~~ -. / "I--- 30 inI L L _ 2b 3b 4b sb 6b 7b Ob 9b J O00 PPV Fig. 1. The relationship between the percentage of cases vaccinated (PCV) and the percentage of the population vaccinated (PPV) for seven different percentage values of vaccine efficacy (VE). of vaccine efficacy can be made. This screening technique will indicate whether there is need for more careful evaluation. It should not be relied upon for precise estimates of vaccine efficacy. There is a small danger that vaccine efficacy may be overestimated if the vaccination levels in the community are overestimated (particularly when the vaccine coverage is > 80%o), or if the proportion of cases with a vaccination history is underestimated. However, in most circumstances with reasonably accurate estimates, an overestimation of vaccine efficacy should be rare and this screening will provide a rough guide as to whether further evaluation is necessary. Outbreak investigations: community-wide or total population assessment Criteriaforselection. The best situation in which to measure vaccine efficacy is probably in outbreaks in defined settings such as villages, towns, cities or schools (5, 7, 8). Although any outbreak can be investigated, biases will be minimized if the following criteria are kept in mind: (1) absence of substantial prior disease activity in the studied age group, (2) both vaccinees and nonvaccinees included in the study population, (3) adequate numbers in the population in the age group to be studied, (4) high overall attack rate (for measles, generally in excess of 5% in the chosen age group), and (5) good vaccination records available to differentiate non-vaccinees from vaccinees. Efficacy is probably best measured in settings where measles has been intermittent rather than endemic. Generally the selected study group should be old enough to be susceptible to measles and to have been vaccinated, yet young enough not to have had substantial exposure to measles prior to the outbreak. In villages, the appropriate age group to be studied may be determined by questioning village elders about the time of the last outbreak and by deter- mining the proportion of each age group with a history of disease before the outbreak. Age groups in- which substantial proportions of persons have had the disease should not be included in the study. The lower age limit is determined by the age recommended for vaccination (2, 3). This is usually after the transplacentally derived immunity has waned and before the attack rates for the disease become high. For measles, in most developing areas, this age group probably includes children between 9 months and 3 years old. If the overall attack rates in the selected age groups are in excess of 5%, the risk of exposure may be considered to be somewhat comparable. Ideally, good vaccination records should be available. 80 70 60 u 50 40 30 20 10 0 1057 II 1058 W. A. ORENSTEIN ET AL. Methods. The following should be taken into consideration. (1) Case definition: a sensitive and specific clinical case definition should be used. The definition given above may be used to classify cases of measles. (2) Case ascertainment: a total population based surveillance survey should be conducted. In villages, workers should go from door to door taking a census of all persons in the target age group and determining whether any have had an illness clinically compatible with measles. In the case of persons in the target age group who have died (from whatever cause) during the outbreak period, details of the clinical illness and vaccination status should be collected; these persons should be included as cases or non-cases, as appro- priate. (3) Vaccination status determination: as the health workers go from door to door, they should obtain histories of vaccination for all persons in the target age group. Dates of birth and of vaccination should be recorded, if available. If the latter are not avail- able, the age at vaccination should be estimated. Equal effort should be made to determine the vacci- nation status of cases and non-cases. For measles, persons should be considered as immunized if they received the vaccine at or after the minimum recom- mended age, and if vaccinated 14 or more days before the onset of the outbreak. Persons vaccinated before the recommended age should be classified separately. Persons vaccinated during the outbreak should be classified based on vaccination status prior to the outbreak. (4) Prior disease: disease that occurred before the outbreak will have minimal effect on the vaccine efficacy calculation if the incidence rate of the disease in the area under study and in the chosen age group was low.' Hence, age groups that probably will have been heavily exposed to measles prior to the outbreak (e.g., persons > 3 years of age) should be excluded from the investigations. However, once the appro- priate age group is selected, persons with prior disease should be included in the denominators of the appro- priate attack rate calculation. The numerator, however, should include only cases that occurred during the outbreak. Analyses (see Table 1). These must take into account the following: (1) ARU = b+e c The effects of disease prior to the outbreak will also be minimized if persons with prior disease were as likely to obtain immunization as persons without disease and the risk of exposure to measles of vaccinees and non-vaccinees prior to the outbreak was similar. where b is the number of unvaccinated cases during the outbreak and e is the number of unvaccinated who did not develop measles during the outbreak. (2) ARV = a a+d where a is the number of vaccinated cases during the outbreak and d is the number of vaccinated who did not develop measles during the outbreak. (3) Persons with unknown vaccination histories should be excluded from the calculation, whether or not they had illness (excluding c +f, see Table 1). (4) Vaccine efficacy can be calculated according to the standard formula. Special analyses to evaluate efficacy by the age at vaccination, the duration of vaccine-induced immunity, and the effects of two doses can also be calculated using the ARV for the specific group being studied. For example, to calculate vaccine efficacy for persons vaccinated at exactly 12 months of age, the ARV would equal the number of cases who had been vaccinated at this age divided by the total population vaccinated at the same age. The overall ARU is usually used for each of the special analyses; more refined estimates may be obtained by controlling for factors such as age. In the above example, the ARU might be calculated only among children aged > 12 months at the time of the outbreak. Measurement of vaccine efficacy in outbreaks can be complicated if an extensive vaccination pro- gramme was carried out during the outbreak because the vaccination status of individuals may change during the middle of the outbreak. Correction for the effects of control programmes is necessary if a substantial proportion of the cases occur after the programme and a substantial proportion of the popu- lation was vaccinated during the programme (7, 9).d When either of these proportions is small, no cor- rection is necessary. Outbreak investigations: estimating vaccine efficacy in large populations (cluster samples) When outbreaks occur in large populations, deter- mination of the vaccination status of all the individuals involved may be unmanageable. In these situations, a coverage survey of children in the at-risk population can be used to estimate the pre-outbreak immunization levels. For example, thirty neighbour- hood clusters may be chosen, as already described and seven or more children in the age group for the study d MARKS, J. S. ET AL. A new stochastic method for the epidemiologic evaluation of vaccine efficacy. Paper presented at the Robert Wood Johnson Clinical Scholars Program Meeting, Scottsdale, Arizona, 12-15 November 1980. FIELD EVALUATION OF VACCINE EFFICACY Table 1. Data to be collected in an outbreak investi- gation for the calculation of vaccine efficacy (VE) a Vaccination status Clinical status Vaccinated Unvaccinated Unknown Unwell (ill) Well a d b e ~~b aVE ARU-ARV (b+e) (a d)x 100ARU bxOO (b + e) a See text for details. (e.g., 9-35 months) may be selected from each cluster (10). The vaccination status and any history of an illness clinically compatible with measles are deter- mined for each participant; vaccination status is assessed preferably from vaccination records. If the attack rates are high so that the number of cases in the sample is large, vaccine efficacy can be calculated directly from the coverage survey as shown in Table 1. Vaccine efficacy (VE) can also be expressed in the form of relative risk (RR), which is the ratio of ARV to ARU, as shown below: VE ARU - ARV x 100 ( ARV x 100V= ARU ARU, = (I - RR) x 100 If the number of cases is low, they can be sup- plemented by other cases found through disease sur- veillance systems. This will increase the precision of the estimate (i.e., decrease the width of the confidence interval). On the assumption that the additional cases obtained from the surveillance system are representa- tive of the cases from the population surveyed, then the relative risk may be calculated directly (see Table 2) without calculating ARV and ARU. This is the equivalent of a case exposure study in chronic disease epidemiology (11). Methods. The following should be taken into consideration. (1) Case definition: same as in previous section (see above). (2) Case ascertainment: if the attack rates are high, the number of cases in the sample may be sufficient to accurately estimate vaccine efficacy. If the attack rates are low, other surveillance data can be used to gain additional cases and increase precision. (3) Vaccination status determination: preferably from written records of persons in the sample and all cases. (4) Prior disease: same as in previous section (see above). Analyses. Attack rates and vaccine efficacy can be calculated as in an outbreak investigation (see Table 1) when the coverage survey above is used. If sup- plemental cases are added, the calculation shown in Table 2 is used. Secondary attack rates in families The possibility that the exposure of vaccinees and non-vaccinees to the disease during outbreaks may be different can result in biased estimates of vaccine efficacy and is a potential problem with such investigations. An alternative approach to reduce this bias is to measure the secondary attack rates of the disease in family members of index cases. Studies on measles have demonstrated that secondary attack rates in non-vaccinees are generally consistent from family to family, implying that within the household there is generally uniform exposure (12, 13). Secondary attack rate determinations have not been used as frequently as outbreak investigations in measuring the efficacy of measles vaccine (6, 14). Nevertheless, the technique has been thoroughly evaluated and has proved useful not only for measles, but also for other vaccine-controlled diseases such as pertussis (15, 16). An additional advantage of the Table 2. Calculation of vaccine efficacy (VE) using coverage survey data and supplemental information on casesa Vaccination status Clinical status Vaccinated Unvaccinated Unknown Unwell (ill) from coverage survey a b c Total in coverage survey d e f Other ill from population surveillance g h i Total ill a+g b+h c+i VE=(1-RR)x100= 1 - (b+h Xl00 a See text for details. 1059 1060 W. A. ORENSTEIN ET AL. secondary attack rate method is that vaccinees and non-vaccinees from several families can be added to determine the overall attack rates in the vaccinated and unvaccinated populations, provided the same definitions for cases and immunization status are used. To minimize the effect of previous exposure to measles disease, the age group studied should be restricted to between 9 and 35 months. Methods. The following should be taken into consideration. (1) Case definition: same as described above. (2) Case ascertainment: as described above under outbreak investigations, a good population based surveillance survey should be carried out. Otherwise, families with single cases may be less likely to be reported than families with multiple cases. For measles, all cases in a given family should be listed by the date of onset of the rash. Families should be followed up for at least 18 days after the onset of rash in the first case in the family, the maximum interval for secondary cases being determined from earlier studies of measles (12). Persons should be classified as cases or non-cases by their status 18 days after onset of the first case in the family. For other diseases, the appropriate maximum incubation period can be substituted for the 18-day interval in the case of measles. (3) Vaccination status determination: same as described above under outbreaks. For measles, persons should be classified as vaccinated or unvaccinated by their status on the day of onset of the rash in the first case in the family. For other diseases, the cut-off should be determined after considering the average incubation period for the disease in question and the average time required for the vaccination to become effective. (4) Prior disease: same as described above. Analyses. These must be based on the following: (1) For measles, persons in the family who develop a rash within the first six days following onset of rash in the first (index) case are considered as co-primary cases. The first case in a family and all co-primary cases should be excluded from the analysis. A similar approach should be taken with other diseases. (2) Length of the follow-up for all families should be at least 18 days (for measles) after the onset of rash in the first case. (3) All secondary cases in all families in the target age group (other than the index case and co-primaries) are added to give the total number of cases, and all the non-cases are added to give the total number without illness. (4) Vaccine efficacy is calculated using the data in Table 1. Secondary attack rate in clusters A modified form of household investigation has been used in urban and semi-urban settings. This technique has been less well studied than outbreak investigations and secondary attack rates in families.e It is less rigorous (because comparability of exposure of vaccinees and non-vaccinees is less definite) but is logistically easier to carry out than intra-household studies. In the course of an outbreak, or towards the end of the measles transmission season, the study must be conducted in a group of neighbourhood clusters in each of which at least one known case of measles occurred during the most recent transmission period or some other specified period. The study subjects (e.g., children between 9 and 35 months) are those who live in close proximity to a known case, i.e., no more than one house away from the open area in front of the doorway of the house with a case. Clusters are defined operationally by the investi- gators, who start at the household of an identified case,f then proceed to the neighbouring households listing all children in the target age group. If another case which had occurred during the outbreak period is found in one of the visited households, the house- holds adjacent to it are visited, and so forth, until no further cases are found. Thus all the children investi- gated will have an equally close neighbourhood relationship to a case. This approach requires a second visit to the neighbourhood clusters at least 18 days later for confirmation of cases seen too early to determine whether they met the case definition and to detect any secondary cases among their contacts. Methods. The following should be taken into consideration. (1) Case definition: same as described above. (2) Case ascertainment: all cases in the target age group found in the surveyed households during the predetermined time period are included, as well as the case which led to studying the cluster. (3) Vaccination status determination: same as in secondary attack rates in families. (4) Prior disease: same as described above. e DONDERO, T. J. ET AL. Efficacy of measles vaccination in Cameroon (manuscript in preparation). f The index case may be reported from any source including hospitals, clinics, schools or a population based surveillance survey. FIELD EVALUATION OF VACCINE EFFICACY Analyses. See above, as described under outbreak investigations. Coverage survey methods in endemic areas Vaccine efficacy can be assessed in the absence of a definable outbreak in urban populations with highly endemic measles by using coverage survey methods. Conceptually this approach is similar to that used in outbreak investigations except that vaccination status is ascertained as of a given age (e.g., 12 months) rather than as of the beginning of the outbreak, and disease history is ascertained up to the current age of the children in the survey rather than over a shorter outbreak period of time. No actual outbreak is required. However, because the interval from disease to the time of the survey may be long (up to 2 years), a history taken from parents rather than specific clinical information is used to identify and define the cases. This coverage survey approach is attractive because the situation it seeks to deal with is one frequently encountered in urban measles control programmes, and because the sampling techniques used are well recognized and easily applied. However, there has been only limited experience in the use of this approach. Modifications have been introduced to minimize the biases, but more changes may be required as additional experience accumulates. The mnethod presented below is an alteration of the cluster sample design frequently employed to estimate coverage (10). However, other random sample designs can also be used. Thirty neighbourhood clusters are randomly selected using the sampling methods described for vaccination coverage surveys. Fourteen or more 2- year-old children are sampled from each neighbour- hood cluster, and two new questions on previous measles disease and the age when this happened (if applicable) are added to the usual coverage survey questions on current age and date of vaccination. Age at vaccination must be calculated from these data. The 2-year-old children sampled from each cluster must include at least 7 never-vaccinated children and at least 7 children vaccinated between 9 and 11 months of age (or within three months of the recommended age at vaccination, if older than 9 months).9 This approach will yield an estimate of the efficacy 8 If an estimate of vaccination coverage is desired for 2-year-old children, the first seven such children encountered in each cluster should be used, regardless of their vaccination status or age at vaccination. Some of these children can be included in the group of fourteen children being sought for a vaccine efficacy if their vaccination status or age at vaccination qualify them for inclusion. In this manner, estimates of vaccine coverage and vaccine efficacy can be obtained simultaneously using the familiar coverage survey sampling methods. of vaccine when administered to children between 9 and 11 months of age. To obtain an estimate of measles vaccine efficacy when any interfering influence of maternal antibodies is low, 7 children vaccinated between 12 and 14 months of age could also be included in each cluster. This technique requires an expansion of the usual age group employed in coverage surveys from 12-23 months to 24-35 months. If the usual coverage assessments are desired in the 12-23-month age group, each cluster should include seven children in that age group as well. With the information on the age when the disease occurred among vaccinated and unvaccinated children, it is possible to calculate the measles attack rates experienced by both groups of children from 12 months of ageh up to their current age at the time of the survey, and the resulting vaccine efficacy. The attack rates are calculated as follows: No. of cases after age 11 months' ARU = among never vaccinated children Total number of never vaccinated children No. of cases after age 11 months ARV = among vaccinated children Total number of vaccinated children (ARU-ARV) VE (%) = ARU x 100 Methods. The following should be taken into consideration. (1) Case definition. This approach assumes that measles is sufficiently distinctive to be recognized as measles in areas of high endemicity by the mothers of the children surveyed. Since the recall period is two years or less, this is considered sufficiently recent for the mother to recall the disease accurately. Any illness with a rash that was diagnosed as measles by the mother is accepted as a case of measles. In Abidjan; Ivory Coast, where this method was developed, such histories have been shown to be reliable (17). (2) Case ascertainment. All cases of measles reported by questioning the mother are of interest, the deter- mination of the age (in months) at the time of the illness in those with positive histories requiring careful attention. If the exact age cannot be elicited, a determination of whether or not the disease occurred h If children vaccinated between 12 and 14 months are included, their disease experience would be calculated starting as of 15 rather than 12 months. i If the age of the vaccinated group changes, this should be ad-justed (see text). 1062 W. A. ORENSTEIN ET AL. before 12 months of age (or before 15 months if 12-14-month-old vaccinees are also sampled) is the minimum information required. All cases which occurred among these 2-year-old children when they were between 12 months and 2 years are included in the study. (3) Vaccination status determination. Two-year-old children vaccinated between 9 and 11 months of age constitute the principal vaccinated group included in the survey. Children vaccinated prior to this age are excluded from the analysis. Children vaccinated after this age should also be excluded from the analysis unless they are a specific target group included in the survey for vaccine efficacy purposes. (4) Prior disease. Some children vaccinated at 9-11 months will have a history of measles prior to 12 months of age either before or after their vaccination. Some unvaccinated children will also have a history of measles prior to 12 months. All vaccinated and unvaccinated children with such histories should be excluded from the numerators, but not from the denominators of the appropriate rates calculated. Such a procedure will ensure that a minimally biased estimate is obtained. It assumes only that children with a history of measles disease are as likely to obtain vaccination as children without such a history. Analyses. These must be based on the following: (1) Persons with uncertain vaccination or disease histories are excluded. (2) Data obtained by using the criteria described in the above (Methods) section are used to calculate vaccine efficacy. Case-control studies Case-control studies can be most useful when personal immunization records are not generally available but some other source such as records from one or more clinics can be obtained. Intensive effort can be made to determine the vaccination status of a limited number of cases and non-cases (controls) instead of concentrating on the whole population at risk. The traditional vaccine efficacy equation cannot be used in such studies. Cases in a case-control study represent one sampling fraction of all cases and the controls represent a different sampling fraction of the population that is not ill (18). In general, these sampling fractions are unknown so that the total populations of vaccinated and unvaccinated persons cannot be calculated, thus preventing calculation of the attack rates. The vaccine efficacy equation can be expressed in the form of relative risk (RR). In case-control studies the RR can be approximated by the odds ratioi and vaccine efficacy can be calculated. By knowing the vaccination histories of cases and of non-cases (controls), the odds ratio and vaccine efficacy can be estimated. Case-control studies have not been thoroughly evaluated in the measurement of vaccine efficacy. The results of some studies of measles and rubella outbreaks suggest that they reflect the vaccine efficacy accurately; with further use, refinements may be made (19).k In addition, case-control sets can be added together from several outbreaks to increase the numbers and the power of the calculations. Methods. The following should be taken into consideration. (1) Case definition: same as described in outbreak investigations. (2) Case ascertainment: same as described in outbreak investigations, although all cases need not be detected. (3) Vaccination status determination: this is only necessary for selected cases and selected controls (non-cases); otherwise, the same as described in outbreak investigations. (4) Control selection. Matched pair analysis: one control for every case should be selected and matched with the case for age, sex and residence. The controls should be well at the time of the investigation and should preferably be selected at random from surrounding houses in the village. This offers the convenience of choosing the controls on the same occasion as the cases are interviewed. Potential controls should continue to be identified until one is found which meets the matching criteria. For measles, the cases should be aged 9-35 months and the controls in the same range as well as within 2 months of the corresponding cases. Close matching in ages is most important for cases .18 months old. Records of each case-control pair should be kept. Unmatched analysis: controls are selected ran- domly from the affected village or villages in the age range 9-35 months. The total number of controls should be one to two times the number of cases. The unmatched approach may be more difficult than matched sampling since it requires the planning and i The odds ratio is the ratio of the odds that a case is vaccinated divided by the odds that a control is vaccinated. See Table 3 for calculation of the odds ratio; this ratio may not approximate the relative risk when attack rates are high. When the attack rates in the vaccinated are greater than 100o, the vaccine efficacy will be erroneously high. In most instances the attack rates in the vaccinated will be < 1007o so that this error will not be important. k ORENSTEIN, W. A. ET AL. Vaccine efficacy: a new application of case-control and case exposure methodology. Paper presented at the Society for Epidemiologic Research meeting, Cincinnati, Ohio, 16-18 June 1982. 1063FIELD EVALUATION OF VACCINE EFFICACY Table 3. Determination of vaccine efficacy (VE) in a case-control study by (A) matched pair analysis and (B) unmatched analysis Controls Vaccinated Unvaccinated A. Matched pair analysis Vaccinated cases j p Unvaccinated cases k q RR = odds ratio = k VE (%)=(1 -RR)x 100= (1 - x 100 Cases Controls B. Unmatched analysis Vaccinated a b Unvaccinated c d RR _odds ratio = adbc VE (%)=(1 -RR)x 100= (bad) 100 execution of a separate programme to randomly pick and interview the controls. (5) Prior disease: cases and controls with histories of prior disease should not be excluded from the calculation (20). Analyses. Matched pairs: (i) Case and control pairs, in which the vaccination status of either one is unknown, should be analysed separately. (ii) Table 3A shows an analysis of the matched case-control pairs. Instead of individuals, each cell contains pairs. For example, j represents the number of pairs in which both the case and control were vaccinated while p represents the number of pairs where the control was unvaccinated and the case was vaccinated. The sum of j+k+p+q is equal to half the number of participants. The odds ratio equals p divided by k. Unmatched design: Table 3B shows how to calculate vaccine efficacy from unmatched data. Confidence intervals for vaccine efficacy Confidence intervals for vaccine efficacy determinations are shown in Table 4 (21). The formulas in Table 4A apply to outbreak investi- gations and secondary attack rates in households or clusters. Approximate confidence intervals can also be obtained for efficacy using cluster samples. Formulas for approximate confidence intervals using supplementary cases and cluster samples in outbreaks can be obtained from Hogue et al.(1J) (see under "case exposure studies"). The same reference can be used for confidence intervals for case-control studies supplemented by Bayes' theorem. Confidence intervals for case-control studies using matched pair analysis are shown in Table 4B (22). DISCUSSION The efficacy of vaccines in clinical use can be deter- mined by a variety of means including screening, out- break investigations, secondary attack rates in families or clusters, vaccine coverage assessments, and case-control studies. They all offer a means of monitoring vaccine programmes under conditions of day-to-day vaccine use. The different techniques for measuring efficacy are summarized in Table 5. The screening technique is the most useful and rapid means of determining whether there is a problem with a vaccine. All that is needed is a reliable estimate of the proportion of cases occurring in vaccinated individuals and an estimate of the vaccine coverage in the population at risk. If the estimated efficacy is within expected limits, more detailed studies are not warranted. However, if the results suggest low efficacy, more rigorous methods are needed to assess the efficacy more accurately. Of the more accurate methods available, outbreak investigation offers the simplest means of measuring vaccine efficacy and is the preferred technique if the situation permits. The biases inherent in the method can be minimized, particularly if the disease incidence rate is high during the outbreak and accurate records exist. A low measles incidence rate prior to the out- break is important, so the age group chosen should be narrow (e.g., 9-35 months) and rural areas where measles is less likely to be endemic are best used. In large populations, the underlying immunization status prior to the outbreak can be estimated using the same cluster sampling method used in coverage assessments. Calculation of secondary attack rates in families is also an excellent and accurate means of measuring vaccine efficacy and is an acceptable alternative to the outbreak investigation. Secondary attack rates in clusters are also probably useful although further evaluation is needed. Vaccine coverage methods in endemic areas are best suited to urban areas where the measles incidence rate is high after age 11 months and low before 12 months, and maternal histories of disease are thought 1064 W. A. ORENSTEIN ET AL. Table 4. Determination of 95% confidence intervals for vaccine efficacy estimates A. All studies except case-control and outbreak investigations using cluster samples and supplementary cases. Vaccinated Unvaccinated Cases Non-cases Total a b a+b c d c+d Total a+c b+d a+b+c+d RR- a / c(R + b) /(c + d) VE (%) = (1 -RR)x 100 (where RR = relative risk) To get 95% confidence intervals the following formulas are used: (1) for lower limit of VE: ( 1- a ) +1 (cd) upper limit of relative risk=RRu=(RR) exp +1.96i a+b + __ +d ) lower limit of VE = (1- RR u) x 100 \ (2) for upper limit of VE: ( 1- ( a ) 1 c \ lowerlimitofrelativerisk=RRL=(RR)exp -1.96 a+b ) + c+d / j upper limit of VE =(1 - RR L) X 100 B. Case-control studies: matched-pair analysis. Controls Vaccinated Unvaccinated Vaccinated cases j p Unvaccinated cases k q Relative risk (RR)_ odds ratio =p/k The upper limit of RR = RRu= 1-Pu and the lower limit of VE = (1 - RR u) x 100 where Pu b= 42-a and2a a = (p + k) x (p + k + 3.84) b= (p+k) x (2 (p+ 1 )+3.84) C = (p + 1)2 The lower limit of RR = RR L= PLl-_PL and the upper limit of VE = (1 - RR L) x 100 d- J/d2-~4ae where PL = 2a and2a a = (p + k) x (p + k + 3.84) d= (p+k) x (2 (p- 1 )+ 3.84) e=(p- 1)2 1065FIELD EVALUATION OF VACCINE EFFICACY Table 5. Summary of techniques for measuring vaccine efficacy with their main advantages and disadvantages Technique Comments Advantages Disadvantages 1. For measles, if the estimate is 1. Rapid within expected levels, no further investigations are needed. 2. If estimate < expected levels, 2. Requires few resources more accurate techniques are needed 1. Estimates may be inaccurate if the proportions of population vaccinated and cases vaccinated are inaccurate 2. Outbreak investigations (a) Total census (b) Cluster samples 1. The preferred technique, situation permitting 2. Indicated during outbreaks in small populations where immunization and disease status of all individuals can be assessed 3. Biases can be kept to a minimum using a population-based surveillance survey in an area with high attack rates and a low incidence of disease before the outbreak 1. Mostly indicated during large outbreaks in large populations when determination of vaccination and disease status on all individuals is not feasible. Otherwise same as 2 (a) 1. Most frequently evaluated technique 2. Allows collection of clinical information on cases for more accurate diagnosis 1. Requires substantially more resources than screening 2. Exposure of vaccinees and non- vaccinees to disease may not be absolutely equivalent 3. With high attack rates, the exposure of vaccinees and non-vaccinees to disease becomes more comparable 4. One of the easiest of the more accurate methods to perform 1. Same as for 2 (a) 1. Same as for 2 (a) 2. Because samples are taken rather than a census, there may be some loss in precision of the estimate 1. Next to outbreak investigations, 1. Corrects for potential this technique has been evaluated differences in exposure most and is an acceptable between vaccinees and alternative non-vaccinees 2. Potentially one can add results of many different family investigations in different areas together, allowing more accurate estimates 1. Probably requires more resources than for outbreak investigations, since each family must be followed for at least 18 days, e.g., often >2 visits 2. Only a small number of children will be in the right age group in a given family so that many families must be visited 3. Case definition will be less predictive for measles in the absence of an outbreak 4. Secondary attack 1. Modification of the secondary rates in clusters attack rates in families using localized neighbourhoods 2. Most indicated when resources for family investigations are limited. Because the numbers of persons exposed in a cluster are greater than in a family, fewer visits are needed 1. May correct for potential differences in exposure between vaccinees and non- vaccinees in outbreak investigations. 2. Assuming the exposure within clusters is comparable from one cluster to another, the results of multiple clusters can be added together 1. Same as for technique 3 2. Needs further evaluation of uniformity of exposure within clusters Table 5: continued on next page 1. Screening 3. Secondary attack rates in families 1066 W. A. ORENSTEIN ET AL. Table 5: continued Technique Comments Advantages Disadvantages 5. Vaccine efficacy 1. Modification of routine coverage 1. Requires only minor changes 1. Relies on the parent's diagnosis using coverage assessment using older age of a technique with which and recall of the disease rather methods in groups, 24-35 months, and most EPI personnel are familiar than clinical information endemic areas adding questions on disease history 2. Most indicated in non-outbreak 2. Requires similar resources as 2. If coverage assessments in settings when the disease is in coverage survey 1 2-23-month-olds are desired, highly endemic, such as in large the number of children per cluster urban areas might have to be increased 3. With minor changes in design it will provide coverage data simultaneously 6. Case-control 1. The vaccination histories of 1. Allows maximal resources to 1. Will give a falsely high vaccine studies cases and matched non-cases be utilized in finding the efficacy if the attack rates in the are compared. Vaccine efficacy vaccination status of cases vaccinated are high is calculated by using the and a few matched controls, odds ratio to approximate the instead of the entire relative risk population 2. Most indicated when vaccination status is difficult to obtain, as when individual immunization records are poor, but another source such as one or more clinics have records to be accurate. This technique has not been used widely and refinements may be made as a result of greater experience. Case-control studies are best suited to areas where reliable personal immunization records may be diffi- cult to find but other information, such as clinic records, may be available. In this way, intensive efforts can be applied to determining the vaccination status of the cases and a few selected controls instead of the entire population at risk. It may be noted that no epidemiological method is perfect because it cannot exactly duplicate the experi- mental conditions of a prospective randomized clinical trial. The most accurate results will be obtained when biases are anticipated and corrective measures are taken whenever possible. Clinical vaccine efficacy determinations are carried out in order to assess whether the observed pattern of illness is consistent with the proper use of a highly effective vaccine. The results can also be used to make changes in the programme if necessary. A lower than expected efficacy should lead to a careful evaluation of the vaccine management and vaccine adminis- tration technique. If these are unsatisfactory, correc- tive measures should be taken. If satisfactory, other explanations should be sought, e.g., a transient problem due to a fault in a single lot of vaccine or a single shipment. The components of a vaccine efficacy evaluation- case definition, case ascertainment, and vaccination status determination -apply to studies on all vaccines. Case definitions will vary depending on the disease and some will require more laboratory support than others (23).' Similarly, case ascertainment should generally be population based rather than clinic or hospital based. Diseases with high proportions of infections that are subclinical, such as poliomyelitis, can be evaluated solely on the clinical illness rather than total infections. If the proportion of subclinical infections is assumed to be the same in the unvaccinated group and among vaccine failures, vaccine efficacy will be accurately reflected by the number of clinical illnesses alone. The general methodology can be applied to vaccines requiring multiple as well as single doses. The efficacy of each dose can be calculated using the attack rate in the unvaccinated (ARU) with no prior doses, and compared in successive calculations to the attack rates in recipients of 1 prior dose, 2 prior doses, 3 prior doses, etc. Care must be taken to ensure that both numerator and denominator reflect the same group. Each component of the vaccine efficacy evaluation is potentially associated with problems that can lead to substantial biases in the estimate of vaccine efficacy. Awareness of these potential biases can lead ' See footnote b on p. 1056. FIELD EVALUATION OF VACCINE EFFICACY to corrective measures to keep them to a minimum. In general, if a method cannot be totally corrected, the techniques recommended will tend to slightly under- estimate the vaccine efficacy. On occasion, this may lead to intensive investigations of vaccine handling practices and other aspects of the immunization program which may not really be necessary. However, it is better occasionally to investigate unnecessarily than to fail to investigate when an intensive examination may be important. Clinical vaccine efficacy studies provide useful information to health care providers concerning the effectiveness of vaccines and can help in the evaluation of policy decisions and the determination of trouble spots in vaccine programmes. In addition, the coordinators of routine immunization pro- grammes will be able to increase the public's confidence in immunization. RtSUMt EVALUATION SUR LE TERRAIN DE L'EFFICACITE DES VACCINS II est possible d'evaluer l'efficacite des vaccinations pratiqu6es sur le terrain en mesurant par des moyens epi- demiologiques le degre de protection contre la maladie. Ces techniques sont particulierement utiles lorsque l'efficacit6 de la vaccination est mise en doute. Ce probleme gagne en acuite a mesure que la couverture vaccinale s'etend, car un nombre de plus en plus grand de cas s'observera chez des personnes vaccin6es, meme lorsque l'efficacite du vaccin est bonne. On peut obtenir une estimation rapide de l'efficacite en utilisant les donn&es existantes sur la proportion de cas survenant chez des sujets vaccines et l'estimation actuelle de la couverture vaccinale dans la population a risque. Si cette premiere approche montre une faible efficacite, des methodes utilisant la collecte de nouvelles donn&es et donnant une estimation plus precise sont indiqu&es. Ces methodes s'appuient sur les enquetes sur les flamb6es, even- tuellement avec echantillonnage en grappe, les taux d'atteinte secondaire dans les familles, les taux d'atteinte secondaire dans les grappes, les enquetes sur la couverture vaccinale dans les zones d'end6mie, et les etudes r6trospec- tives. Quand les circonstances le permettent, les enquetes sur les flamb6es constituent la m6thode de choix, notamment pour 1'6valuation de l'efficacit6 du vaccin antirougeoleux. Les taux d'atteinte secondaire dans les familles constituent aussi une excellente methode. Les autres methodes pre- sentent des avantages dans certains cas. Quelle que soit la m6thode choisie, les estimations les plus precises de l'effica- cite des vaccins s'obtiennent lorsque les biais potentiels sont pris en compte et reduits au minimum lors de la conception de l'enquete ou de l'6tude. Les r6sultats qui se situent dans les limites prevues doivent susciter une conflance accrue dans le programme de vaccination. Une efficacite plus faible que prevu au contraire doit amener a entreprendre des enquetes pouss6es sur les pratiques en matiere de manipulation des vaccins et sur d'autres aspects du pro- gramme de vaccination pouvant etre A l'origine de la perte d'efficacite. REFERENCES 1. MARKS, J. S. ET AL. Methodologic issues in the evalu- ation of vaccine effectiveness. Measles vaccine at 12 versus 15 months. Am. j. epidemiol., 166: 510-523 (1982). 2. HALSEY, N. A. The optimal age for administering measles vaccine in developing countries. In: Halsey, N. A. & de Quadros, C., ed., Recent advances in immu- nization, a bibliographic review (Scientific Publication No. 451), Washington, Pan American Health Organ- ization, 1983, pp. 4-17. 3. EXPANDED PROGRAMME ON IMMUNIZATION. The optimal age for measles immunization. Wkly epid. rec., 57: 89-91 (1982). 4. CENTERS FOR DISEASE CONTROL. Measles vaccine efficacy-United States. Morb. Mortal. Wkly Rep., 29: 470-472 (1980). 5. LINNEMANN, C. C. JR. Measles vaccine: immunity, reinfection and revaccination. Am. j. epidemiol., 97: 365-371 (1973). 6. MCCORMICK, J. B. ET AL. Measles vaccine efficacy determined from secondary attack rates during a severe epidemic. J. pediatr., 90: 13-16 (1977). 7. MARKS, J. S. ET AL. Measles vaccine efficacy in children previously vaccinated at 12 months of age. Pediatrics, 62: 955-960 (1978). 8. HULL, H. F. ET AL. Measles mortality and vaccine efficacy in rural West Africa. Lancet, 1: 972-975 (1983). 9. PETO, R. ET AL. Design and analysis of randomized clinical trials requiring prolonged observation of each patient. II. Analysis and examples. Br. j. cancer, 35: 1-39 (1977). 10. HENDERSON, R. H. & SUNDARESAN, T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method. Bull. Wld Hlth Org., 60: 253-260 (1982). 1068 W. A. ORENSTEIN ET AL. 11. HOGUE, C. J. R. ET AL. Estimators of relative risk for case-control studies. Am. j. epidemiol., 118: 396-407 (1983). 12. HOPE-SIMPSON, R. E. Infectiousness of communicable diseases in the household (measles, chickenpox, and mumps). Lancet, 2: 549-554 (1952). 13. Top, F. H. Measles in Detroit, 1935. 1. Factors influencing the secondary attack rate among susceptibles at risk. Am. j. public hfth, 28: 935-943 (1938). 14. MCINTYRE, R. C. ET AL. Measles and measles vaccine efficacy in a remote island population. Bull. Wld Hlth Org., 60: 767-775 (1982). 15. MEDICAL RESEARCH COUNCIL. The prevention of whooping cough by vaccination. Br. med. j., 1: 1463- 1471 (1951). 16. BROOME, C. V. ET AL. Epidemiology of pertussis, Atlanta, 1977. J. pediatr., 98: 362-367 (1981). 17. BERNIER, R. H. ET AL. A new practical approach to determining measles vaccine efficacy. Am. j. epidemiol., 118: 410-411 (1983). 18. SCHLESSELMAN, J. J. Case-control studies. Design, conduct, analysis. New York, Oxford University Press, 1982, pp. 36-38. 19. GREAVES, W. L. ET AL. Clinical efficacy of rubella vaccine. Pediatr. infectious disease, 2: 284-286 (1983). 20. SMITH, P. G. ET AL. Assessment of the protective efficacy of vaccines against common diseases using case-control and cohort studies. Int. j. epidemiol., 13: 87-93 (1984). 21. KLEINBAUM, D. G. ET AL. Epidemiologic research. Principles and quantitative methods. Belmont, CA, Lifetime Learning Publications, 1982. 22. SCHLESSELMAN, J. J. Case-control studies. Design, conduct, analysis. New York, Oxford University Press, 1982, pp. 209-211. 23. GEMERT, W. ET AL. Agents affecting health of mother and child in a rural area of Kenya. The diagnosis of measles under field conditions. Trop. geogr. med., 29: 303-313 (1977).