Research steps in the development and evaluation of public health interventions.

Update/Le Point Research steps in the development and evaluation of public health interventions 1. de Zoysa,1 J.-P. Habicht,2 G. Pelto,3 & J. Martines3 Presented is a conceptual framework forplanning intervention-related research. Altogether, nine steps in the process of developing and evaluating public health interventions are specified. This process is dynamic and iterative, and all steps are not always required, or need follow in sequence. The framework can be used to set research priorities by verifying where there is sufficient knowledge to move forward and by identifying critical information gaps. It can also help select appropriate research designs, as each step is characterized by certain types of studies. Greater effort is required to move beyond descriptive epidemiological and behavioural studies, to intervention studies. Field trials of public health interventions require particular attention as they are often neglected, despite their significance for public health policy and practice. This century research has paved the way for major advances to be made in public health. More research is, however, required to develop new or improved public health interventions to deal with the common causes of morbidity, mortality or disability against a background of dwindling resources, especially in de- veloping countries. In this article, we describe an approach to planning intervention-related research based on observations of research managed by a number of agencies and on personal experience in conducting or facilitating research in various coun- tries. Considerable wastage and confusion occur be- cause of poor understanding of the process of intervention development and evaluation. Neverthe- less, goals can be clarified, arguments resolved, and outstanding research accomplished when reference is made to an explicit conceptual framework (Fig. 1). Progress is made from left to right in Fig. 1, from a description of the problem towards the testing and improvement of public health interventions. Below we describe the different steps that we have identified. I Department of International Health, The Johns Hopkins Univer- sity School of Hygiene and Public Health, Baltimore, MD, USA. 2 Division of Nutritional Sciences, Cornell University, Ithaca, NY, USA. I Division of Child Health and Development, World Health Organi- zation, 1211 Geneva 27, Switzerland. Requests for reprints should be sent to Dr Martines at this address. Reprint No. 5832 The framework 1. Describe the problem First, a given problem has to be confirmed to be a public health issue by carrying out basic epidemio- logical research to describe its nature and magni- tude. Usually, the focus is on measuring the frequency of biological outcomes, such as death or disease or infections with a specific agent, in terms of population groups or geographical areas affected. An example of research of this kind is provided by the ongoing effort to document the burden of repro- ductive tract infections among women, following recent reports of very high rates of associated mor- bidity in certain areas, and a new commitment of the international health community to the concept of reproductive health. Sometimes, there is also in- terest in assessing the social or economic dimensions of the problem; in this way, studies have described the devastating impact of the acquired immuno- deficiency syndrome (AIDS) pandemic on commu- nities and their infrastructure in various parts of the world. 2. Identify risk factors The next step is to carry out research to identify factors that are associated with the outcome of inter- est. Such risk factor studies can provide important clues to causal mechanisms and serve to specify the groups at greatest risk. This step may, if knowledge is Bulletin of the World Health Organization, 1998, 76 (2): 127-133 © World Health Organization 1998 127 I. de Zoysa et al. Fig. 1. Flow diagram showing the conceptual framework for research steps in the development and evaluation of public health interventions. sufficient, be combined with the first step. However, to be useful, risk factor studies need to be carefully designed, and guided by epidemiological theory. The unplanned observation of epidemiological associa- tions (so-called "fishing expeditions") should be dis- couraged, since they often produce spurious results (1). Most risk factor studies are observational. Oc- casionally, however, intervention studies may be necessary to establish that an identified risk factor is associated causally with the health outcome of inter- est, and not simply as a result of confounding or misclassification (2). This can be done through a con- trolled experiment which manipulates exposure to the risk factor. Such studies can be costly and diffi- cult to conduct, and may, unless coupled to field trials of interventions, pose ethical problems. How- ever, definitive evidence in controversial areas has been marshalled in this way. For example, the role of vitamin A deficiency in childhood mortality was only shown persuasively by randomized controlled trials that redressed the deficiency through vitamin A sup- plementation (3). 3. Explore the context and identify the determinants This step focuses on describing the behavioural or social processes that lead to the risk factors. This step is critical because the majority of public health prob- lems arise one way or another as a result of human behaviour, e.g. diarrhoeal diseases, cardiovascular diseases, lung cancer, acquired immunodeficiency syndrome (AIDS), and other sexually transmitted diseases (STDs). Research is needed to understand how people behave and why they behave the way they do. The basic premise here is that, to be success- ful, interventions need to build upon people's own perceptions and motivations, and take into account the cultural, social and economic factors that may facilitate or constrain behaviour change. Much of this research is qualitative, seeking to understand rather than to explain. However, there is also a role for quantitative analyses; for example, to assess the prevalence and distribution of risky (or protective) behaviours and examine associations with possible determinants. These analyses can lead to identification of those factors that most strongly influence the performance of the behaviours of inter- est in the study population. This research step should refer to theoretical models of behaviour, to structure the enquiry, allow generalizability to broader populations, and to give insights into approaches to the intervention. Exam- ples of such theoretical models are those provided by Stanton et al. (4) for the control of diarrhoeal diseases. At the same time, this step needs to be focused and practical; it should not only inform about patterns and predictors of current behaviour but also suggest promising routes to inducing changes in behaviour. 4. Select or formulate possible interventions This step involves the development of a solution to the identified problem. Most of the research effort centres on the search for biological approaches to eliminate a specific risk factor or agent, or to modify its effect. The development of a biological solution usually starts in the laboratory or clinical research ward. It builds upon existing understanding of a process, drawn from, say, molecular biology and immunology (for vaccine development), physiology (for the formulation of medical or surgical thera- WHO Bulletin OMS. Vol 76 1998128 Development and evaluation of public health interventions pies), or engineering (for the development of physi- cal aids such as condoms). An example of research of this kind is the work conducted to investigate the physiological and clini- cal features of acute, secretory diarrhoeas, which led to the discovery of the mechanism of active glucose- mediated transport of water and electrolytes across the gut and the formulation of oral rehydration therapy for the management of dehydration. Other examples relate to the development of vaccines against a number of communicable diseases such as AIDS, malaria, cholera, etc. Considerable resources are expended worldwide on basic research of this kind. Biological solutions may, however, not be avail- able and are always, in any case, insufficient. Behav- ioural solutions are also required, as all public health interventions have a behavioural component, even though they do not necessarily have a biological one. It is prudent to be wary of mechanistic and reduc- tionist intervention approaches. For example, con- doms are of little use in preventing the spread of human immunodeficiency virus (HIV) unless they are correctly and consistently used. The develop- ment of approaches to change behaviour should build upon previous knowledge about patterns and determinants of behaviour, focusing first on factors that trigger behaviour change, and subsequently on the processes that reinforce and sustain the desired behaviours. The outcome should be an explicit model of behaviour change, for subsequent testing and refinement. Hornik describes how models of behaviour change can be used to design and plan interventions for the prevention of sexually transmit- ted diseases (5). 5. Test interventions In this step, studies are conducted to examine the safety, feasibility, acceptability, and efficacy of the intervention at the level of the recipient. If a biologi- cal solution to the problem is proposed, it usually undergoes formal evaluation according to approved rules. First, its safety and its effect on intermediary outcomes (such as antibody responses to immuniza- tion or changes in micronutrient levels in response to dietary supplementation) are assessed. If the results are promising, experimental studies are conducted to assess the efficacy of the intervention on health outcomes. For example, clinical trials are conducted to measure the impact of oral rehydration solutions in correcting diarrhoea-associated dehydration. An example of an efficacy trial of a biological solution is provided by the large field trial conducted by Rahmathullah et al. in southern India which demon- strated that periodic small doses of vitamin A could prevent a substantial amount of childhood mortality (6). It is important to note that in this study the means of delivering the vitamin A - home visits once a week did not represent a feasible public health intervention. This study provided information on the efficacy of small, regular doses of vitamin A, permitting extrapolation to what could be expected from interventions to improve dietary intake of vita- min A precursors. Behaviour change interventions may also lend themselves to trials of this kind, par- ticularly if they can be applied at the level of the individual, such as a drug or a vaccine. This can be exemplified by a randomized controlled trial of HIV counselling and testing which examines the influence of a standardized counselling and testing encounter on sexual behaviours (7). Unfortunately, formal tests of approaches to behaviour change are few and far between (8). When confirmation of the efficacy of an inter- vention is crucial, double-blind randomized control- led trials provide the best evidence needed to move public health policy forward. Such studies are diffi- cult and expensive to conduct, however, they are the "gold standard" by which the intervention is judged. If done properly, they require few replications, once efficacy is established, especially if the presence of impact-enhancing and impact-depressing factors are documented, so that it is possible to extrapolate the results to other settings. Nevertheless, trials follow- ing such stringent rules may not always be possible or appropriate (9), particularly for behavioural inter- ventions, and greater use needs to be made of alter- native, more creative methodologies for intervention evaluation (10, 11). 6. Formulate public health interventions The preceding step concerned testing an interven- tion in its pure, reduced form. Subsequently, the task is to formulate approaches to make this intervention available to those who need it within the context of a public health programme. For example, let us assume that a safe, acceptable and efficacious vaginal microbicide has been developed. It is now necessary to consider how this microbicide might best be introduced into a public health programme for the prevention of STDs, including HIV. Clearly, its programme effects will be different if it is delivered to clients attending clinic-based STD treatment services or if it is marketed at community level. This step is developmental or formative, involv- ing decisions about alternative programme designs. To facilitate this decision-making, it is often useful to conduct small-scale, formative, studies of possible delivery systems. For example, studies of the fea- WHO Bulletin OMS. Vol 76 1998 129 1. de Zoysa et al. tures of existing family planning programmes in a particular setting can indicate the feasibility of intro- ducing protocols for the assessment and manage- ment of STDs into these programmes. 7. Assess the efficacy of public health interventions Public health efficacy trials measure the impact of an intervention that is feasible in public health settings, but which is delivered under ideal conditions for the purposes of the study. Such trials provide informa- tion about the changes in behavioural or health outcomes that could be achieved as a result of the intervention programme and are conducted by scien- tists who seek definitive statements about the as- sumptions that underlie a proposed public health intervention. They may also be required by policy- makers and programme planners to justify moving forward with extensive or costly programme imple- mentation. These trials can resemble the studies discussed above that determine the causality of purported risk factors or that test the efficacy of a biological intervention; however, they differ in that they assess the impact of a public health pro- gramme. Thus the double-masked randomized controlled trial of periodic vitamin A supplemen- tation conducted by West et al. in Nepal (12) used an experimental design that was similar to earlier studies; however, it differed in the way that the intervention was delivered. This efficacy trial was intended to examine the impact of a vitamin A supplementation programme of the kind that is ongoing in many countries for the prevention of xerophthalmia (four-monthly distribution of vita- min A capsules at the community level). The study showed that this public health intervention could reduce childhood mortality in that setting. Hall & Aaby argued for increased attention to "practical trials' of interventions before they are introduced into the routine health services (13) and explained the importance of such trials in plan- ning measles immunization programmes, for exam- ple, where it is unlikely that all infants will be vaccinated at exactly 9 months of age. The effects of variations in vaccine uptake at different ages may not readily be predicted from a standard vaccine efficacy trial in which the study team controls the delivery of the vaccine to a selected cohort of children. Randomized controlled trial designs are usually prescribed for public health efficacy trials; however, such trials often need to be carried out on a large scale (on occasion involving numerous sites or communities) and raise many difficult design issues. They are not to be embarked upon lightly and may need to be abandoned altogether in favour of other approaches such as observational studies, which, as stated by Susser (14) "take second place in a hierarchy of rigour, but not in practicability and generalizability". 8. Assess the effectiveness of public health interventions Public health effectiveness trials measure the im- pact of an intervention delivered under normal programme conditions; they take into account the vagaries of public health practice, and provide a more realistic statement of potential impact in the "real world". Public health effectiveness trials are helpful to policy-makers and programme planners who wish to know whether a programme that is already under way and whose continuation or ex- pansion is being considered will have its intended effect. Such individuals are less concerned about the exact mechanism of a trial's action; the fact that the careful implementation of a programme leads to certain desired outcomes is generally adequate for their needs. If the programme's introduction coincides with the desired outcome, and if other nonprogramme factors that might have caused a similar result are absent or taken into account, it is reasonable to assume that the programme was responsible. Such "plausibility designs" (15) still require comparison groups and control of con- founding and other influences, but they can adopt more pragmatic evaluation designs (16) than the randomized, controlled trial. Sometimes these are the only feasible design approaches: for example, for community-based interventions (10, 11). Public health effectiveness trials, however, do need to meet strict standards of process evaluation. Thus, information should be collected on how the inter- vention was delivered, the participation of pro- gramme staff and recipients, and the problems that were encountered, in order to understand the role of factors that may blunt or obscure impact. It is also useful to include an assessment of the cost of introducing the new or improved intervention, to enable cost-effectiveness analyses, which are help- ful in selecting among alternative approaches to addressing a particular health problem. Public health effectiveness and cost- effectiveness studies yield information about the likely effect of the intervention in the routine health services, the resources required for its implementa- tion, and the efficiency of the intervention relative to others aimed at the same health problem. This infor- mation enables policy-makers to make an informed decision about the value of implementing the inter- vention on a large scale. WHO Bulletin OMS. Vol 76 1998130 Development and evaluation of public health interventions 9. Monitor the adequacy and impact of large-scale public health interventions As cost-effective public health interventions are scaled up, research continues to be necessary. Moni- toring and evaluation of the new or enhanced pro- gramme is required to assess its adequacy with respect to progress in implementing programme ac- tivities and to achieve predetermined targets in out- puts and coverage. The primary purpose of such process evaluation activities is to help a programme function more effectively. On rare occasions, it may also be useful to pose the question "Did it work?" and to assess the impact of the programme on behav- ioural or health outcomes. Outcome evaluations are more demanding and are rarely necessary if effec- tiveness data are available (17). Controlled studies may no longer be possible; thus ascribing any change in outcomes to the implementation of programme becomes less persuasive. However, at this juncture it is more important to ensure that the programme goals are being attained than to determine whether any observed change in outcomes is due to the pro- gramme and not to other factors. If the impact of the programme falls short of expectations, despite ad- equate implementation, further, more intensive re- search is needed to find the reason. Using the framework The framework lays out a logical process for the acquisition of knowledge about public health inter- ventions. This does not imply that intervention- related research can, or must, always proceed in an orderly, mechanistic sequence; rather, the process should be dynamic and iterative. There may be op- portunities to omit some steps if there is great ur- gency and accumulated evidence provides sufficient confidence to proceed. Some steps may be combined or conducted concurrently with others. At each step, judgement needs to be exercised, and there may be surprises. Some outcomes may call into question conclusions made in earlier steps, requiring a return to more basic research considerations. There will always be gaps, some of which were inevitable, oth- ers which cannot now be addressed. For example, no efficacy trial has been conducted to demonstrate that increased fluid intake in the early stages of diarrhoeal illness can prevent dehydration. The con- viction that underpins worldwide efforts to promote home-based oral rehydration therapy is based on two decades of clinical experience and information extrapolated from clinical research on the treatment of dehydration. Testing this intervention would, of course, now no longer be feasible, mainly for ethical reasons. The main purpose of the framework we have described here is to determine research priorities. Its use can help to decide what questions need to be answered next - arguably the most difficult, but most important task in any research endeavour. There is no place for studies that are of outstanding technical quality but which pursue a redundant or irrelevant issue. Our observation is that there are many redundancies on the left-hand side of the framework shown in Fig. 1; in particular, observa- tional epidemiological studies and descriptive behav- ioural studies are often repeated beyond the time when they are most productive. On the other hand, gaps are common in the right-hand side of the framework; intervention studies are far less com- mon, probably because they are difficult and more time-and resource-intensive. The framework also helps to select appro- priate research designs and avoid common pitfalls. The first of these consists of mistaking a biological or a behavioral intervention for a public health intervention. There is a widespread misconception that the development of a "quick fix", such as a drug or a vaccine, or a client-health-provider interaction, is all that is required to solve most public health problems. In our experience, it is an important, but only preliminary step. Research to define feasible, acceptable, and cost-effective approaches to de- livering the intervention is often neglected. A second, related problem is the assumption that public health efficacy can be expected in routine practice. Effectiveness trials are often skipped. A familiar situation is that in which an intervention has been demonstrated to be efficacious in a research setting and is immediately introduced into the public health system and scaled up without further assess- ment. This paves the way for disappointment when the impact falls short of expectations, and there is no guidance into appropriate approaches to improve the delivery system and to increase participation by its targeted beneficiaries. The third failure consists of glossing over the behavioural components of an intervention, or dismissing them as cultural idio- syncrasies to be dealt with by programme managers. There is no public health intervention without behaviour change, and it is wise to invest in carrying out the necessary behavioural research. However, this kind of research, as any other, has to be guided by theory, otherwise interventions will remain largely intuitive with limited opportunities for generalization to other problems or populations. Hornik has argued that the application of theory to intervention-related research is an eminently practical activity (5). Such research is less concerned with grandiose theory building, than with prosaic decision-making, and avoiding unexplained associa- WHO Bulletin OMS. Vol 76 1998 131 1. de Zoysa et al. tions between treatments and effects (or "black boxes"). Conclusions There are many health problems that continue to place an intolerable burden of suffering and death on people throughout the world, and for which we still lack the knowledge to develop effective public health programmes. Most health-related research has the objective, in some way or another, of contrib- uting to the development of new or improved inter- ventions to deal with major public health problems. Nevertheless, much research is a waste of time and resources in terms of guiding public health policy and practice (although many papers may be pub- lished in the process). Unfortunately, the global financial and human resources available for intervention-related research are scarce. The bulk of these resources serves to support the basic, descrip- tive research on the left-hand side of the framework. Few field trials of public health interventions are performed, and global public health strategies may be determined by these few trials (13). Priorities must be set and inefficiencies avoided. There is a need to encourage more intervention studies and to push the research agenda towards greater public health relevance, so that research can contribute as rapidly and efficiently as possible to the needed pub- lic health programmes. Acknowledgements We are grateful to the WHO Division of Child Health and Development for the incentive to develop the frame- work described in this paper and encouragement in its application. Resume Les etapes de la recherche dans 1'elaboration et 1'evaluation des interventions de sante publique Cet article pr6sente un cadre conceptuel pour la planification des activites de recherche li6es aux interventions de sante publique. Ce cadre comporte neuf 6tapes, selon un processus dynamique et it6ratif dans lequel elles ne sont pas toujours toutes n6cessaires ni ne doivent etre effectu6es dans l'ordre indiqu6. Ce cadre peut etre utilise pour 6tablir les priorit6s de recherche, en v6rifiant les secteurs ou les connaissances permettent de passer a l'etape suivante et en identifiant les lacunes de l'information. 11 peut aussi servir a choisir des plans de travail appropri6s puisque chaque 6tape est caract6ris6e par un certain type d'6tudes. 11 est n6cessaire de depasser le stade des 6tudes 6pid6miologiques et comportementales descrip- tives pour passer a celui des 6tudes d'interven- tion. 11 faut etre particulierement attentif aux essais pratiques d'interventions, souvent n6glig6s malgre leur int6ret du point de vue des politiques et des pratiques de sant6 publique. References 1. Davey Smith G, Phillips AN. Confounding in epide- miological studies: why "independent" effects may not be all they seem. British medical journal, 1992, 305: 757-759. 2. Mertens TE, Hayes RJ, Smith PG. Epidemiological methods to study the interaction between HIV infec- tion and other sexually transmitted diseases. AIDS, 1990, 4: 57-65. 3. Beaton GH et al. Effectiveness of vitamin A supple- mentation in the control of young child morbidity and mortality in developing countries. Toronto, Inter- national Nutrition Program, University of Toronto, 1992. 4. Stanton B et al. Theory-driven behavioral interven- tion research for the control of diarrheal diseases. Social science and medicine, 1992, 35: 1405- 1420. 5. Hornik R. Alternative models of behavior change. In: Wasserheit JN et al., eds. Research issues in human behavior and sexually transmitted diseases in the AIDS era. Washington, DC, American Society for Microbiology, 1991: 201-219. 6. Rahmathullah L et al. Reduced mortality among chil- dren in southern India receiving a small weekly dose of vitamin A. New England journal of medicine, 1990, 323: 929-935. 7. de Zoysa I et al. Role of HIV counseling and testing in changing risk behavior in developing countries. AIDS, 1995, 9 (suppl. A): S95-S101. 8. Oakley A, Fullerton D, Holland J. Behavioural inter- ventions for HIV/AIDS prevention. AIDS, 1995, 9: 479-486. 9. Black N. Why we need observational studies to evalu- ate the effectiveness of health care. British medical journal, 1996, 312: 1215-1218. 10. Smith PG. Evaluating interventions against tropical diseases. International journal of epidemiology, 1987, 16: 159-166. 11. Kirkwood BR, Morrow RH. Community-based inter- vention trials. Journal of biosocial science, 1989, 10 (suppl.): 79-86. 12. West KP et al. Efficacy of vitamin A in reducing pre- school child mortality in Nepal. Lancet, 1991, 338: 67- 71. 13. Hall AJ, Aaby P. Tropical trials and tribulations. Inter- national joumal of epidemiology, 1990, 19: 777- 781. 132 WHO Bulletin OMS. Vol 76 1998 Development and evaluation of public health interventions 14. Susser M. The tribulations of trials - intervention in communities. American journal of public health, 1995, 85:156-158. 15. Habicht J-P. Judging efficacy, effectiveness and cost-effectiveness of diarrheal control interventions. Paper presented at: Infant Feeding Practices and the Control of Diarrheal Diseases, Johns Hopkins Univer- sity, Baltimore, USA, 25-28 April 1988. 16. Cook TD, Campbell DT. Quasi-experimentation: de- sign and analysis issues for field settings. Boston, Houghton Mifflin, 1979. 17. Hornik R. An approach to evaluation for breast- feeding campaigns. Philadelphia, Center for Interna- tional, Health, and Development Communication, Annenberg School for Communication, 1986 (Work- ing Paper No. 1231). WHO Bulletin OMS. Vol 76 1998 133