The World Health Organization was established in 1948 as a spe- cialized agency of the United Nations serving as the directing and coordinating authority for international health matters and public health. One of WHO’s constitutional functions is to provide objec- tive and reliable information and advice in the field of human health, a responsibility that it fulfils in part through its publications pro- grammes. Through its publications, the Organization seeks to sup- port national health strategies and address the most pressing public health concerns.
The WHO Regional Office for Europe is one of six regional offices throughout the world, each with its own programme geared to the particular health problems of the countries it serves. The European Region embraces some 870 million people living in an area stretching from Greenland in the north and the Mediterranean in the south to the Pacific shores of the Russian Federation. The European programme of WHO therefore concentrates both on the problems associated with industrial and post-industrial society and on those faced by the emerging democracies of central and eastern Europe and the former USSR.
To ensure the widest possible availability of authoritative infor- mation and guidance on health matters, WHO secures broad inter- national distribution of its publications and encourages their transla- tion and adaptation. By helping to promote and protect health and prevent and control disease, WHO’s books contribute to achieving the Organization’s principal objective – the attainment by all people of the highest possible level of health.
Environmental health services in Europe 5
Guidelines for evaluation of environmental health services
Text editing: Jerome Rosen Cover design: Axiom Design, London, United Kingdom
WHO Library Cataloguing in Publication Data
Drew, Christina H. Environmental health services in Europe 5: guidelines for evaluation of environmental health services / Christina H. Drew, Jaco van Duivenboden, and Xavier Bonnefoy
(WHO regional publications. European series ; No. 90)
1.Environmental health 2. Health services 3. Evaluation studies 4.Guidelines 5.Europe I.Duivenboden, Jaco van II.Bonnefoy, Xavier III.Title IV.Series
ISBN 92 890 1357 5 (NLM Classification: WA 670) ISSN 0378-2255
World Health Organization Regional Office for Europe
Copenhagen
Environmental health services in
Europe 5
Guidelines for evaluation of environmental health services
Christina H. Drew, Jaco van Duivenboden
and Xavier Bonnefoy
WHO Regional Publications, European Series, No. 90
ISBN 92 890 1357 5 ISSN 0378-2255
The Regional Office for Europe of the World Health Organization wel- comes requests for permission to reproduce or translate its publications, in part or in full. Applications and enquiries should be addressed to the Publications unit, WHO Regional Office for Europe, Scherfigsvej 8, DK-2100 Copenhagen Ø, Denmark, which will be glad to provide the latest information on any changes made to the text, plans for new edi- tions, and reprints and translations already available.
© World Health Organization 2000
Publications of the World Health Organization enjoy copyright protec- tion in accordance with the provisions of Protocol 2 of the Universal Copyright Convention. All rights reserved.
The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the Secretariat of the World Health Organization concern- ing the legal status of any country, territory, city or area or of its authori- ties, or concerning the delimitation of its frontiers or boundaries. The names of countries or areas used in this publication are those that ob- tained at the time the original language edition of the book was pre- pared.
The mention of specific companies or of certain manufacturers’ prod- ucts does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.
The views expressed in this publication are those of the authors and do not necessarily represent the decisions or the stated policy of the World Health Organization.
PRINTED IN DENMARK
Contents
Page
Foreword ......................................................................................... vii
Introduction ..................................................................................... 1
1. Evaluation principles and procedures .................................... 7
2. Theoretical framework for evaluating environmental health services ........................................................................... 33
3. Data and indicators ................................................................... 53
4. Instruments of evaluation ........................................................ 83
5. Making evaluations work ......................................................... 127
6. Case studies of evaluation in practice .................................... 141
Annex. Participants in preparatory meetings, 1994–1996 ..................................................................... 169
Foreword
For over a decade, the Member States of the WHO European Region have been demonstrating their commitment to improving environ- mental health services. The First European Conference on Environ- ment and Health, held in Frankfurt-am-Main in 1989, spelled out the principles for integrating environmental health services. Five years later, the Second European Conference in Helsinki provided a clear mandate for action on environment and health in Europe through the development of national environmental health action plans (NEHAPS). The need for improved policy tools to strengthen the management of environmental health services was also emphasized in Helsinki. Subsequently, at the Third European Conference, in Lon- don in 1999, even further attention was paid to local processes for planning, implementing and evaluating environment and health ac- tivities.
NEHAPS have become a tool that provide Member States with a substantive framework for developing a path towards sound envi- ronment and health management, in partnership with all sectors. Many Member States have also asked for guidance on specific as- pects of the delivery of environmental health services. This series of publications entitled Environmental Health Services in Europe was developed to respond to these requests. The first two volumes out- line the concepts applicable to environmental health and environ- mental health services and describe the current status of environ- mental health practice in the European Region. The third and fourth
vii
volumes address staffing and professional profiles, and provide guid- ance on the education and training of environmental health profes- sionals.
This, the fifth in the series, provides practical guidelines – re- peatedly requested by the Member States – for evaluating environ- mental health services. Evaluation is an essential component of all health service delivery that is too often overlooked. It is frequently ignored because it is not always seen as directly productive. Never- theless, evaluation is the only way to learn from the successes and failures of the past and thereby improve policies and procedures for the future. To illustrate the difficult concept of evaluation, various examples of environmental health service evaluation are presented.
The production of this book involved a series of discussions and meetings with a wide range of dedicated and knowledgeable indi- viduals and representatives of Member States. In particular, I would like to thank Ms Drew and Mr van Duivenboden, without whose persistence and commitment the book could not have been com- pleted. In addition, the many consultants and advisers who have worked on this project must be thanked for their gracious donation of time and expertise. Specif ically, Mr Martin Eriksson and Dr Andreas Kappos were instrumental in the development of the case studies presented in Chapter 6, and Ms Isabelle Goi, Mr Jean Pierre Foirry and Mr Laurent Gilotte contributed significantly to the sec- tion on economic evaluation tools in Chapter 4.
It is hoped that this volume will lead to a better understanding of evaluation, to new initiatives to introduce evaluation into daily prac- tice and to more discussion of evaluation among environmental health service providers.
Marc Danzon WHO Regional Director for Europe
viii ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
The aim of this book is to provide practical guidelines that assist local environmental health practitioners in evaluating their services. Because the productive worth of evaluation is often obscure, this management step is frequently overlooked. Nevertheless, it is an essential component in developing, implementing and improving the policies designed to protect health and the environment and en- hance the quality of life. Without assessing existing services, their intended consequences will remain unknown, as will knowledge of whether the proper instruments and timing have been employed. In a climate where environmental health services are challenged by the sheer magnitude and diversity of services, limited resources and many scientific uncertainties, evaluation is especially critical. It is the only way to learn the benefits and cost to society of environmen- tal services.
The management cycle includes policy development, implemen- tation and the crucial step of evaluation. Because the broad range of environmental health services demands different evaluation strate- gies, and because the different evaluation concepts, tools and com- munication techniques can be combined in an infinite number of ways, no single or ideal evaluation model can be offered. The goal of this publication is, therefore, to introduce environmental health professionals to evaluation principles, tools and examples from which they can pick and choose elements to suit their specific needs. It is designed not to answer all the questions practitioners have, but rather
1
Introduction
2 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
to guide them towards questions that must be answered to evaluate and improve services.
This volume has evolved from several interactions with Member States of the European Region that have identified gaps in the man- agement and evaluation of environmental health ser-vices. The three European conferences on environment and health – held in Frank- furt-am-Main, Germany (1989), Helsinki, Finland (1994) and Lon- don, United Kingdom (1999) – all emphasized the need for inte- grating environmental health services, developing policy tools, and strengthening local processes for planning, implementing and evalu- ating environment and health activities (1–5). In addition, previous volumes in the series on environmental health services in Europe have identified the lack of a clear evaluation methodology (6,7) and emphasized the need for environmental health service managers to obtain evaluation skills (8,9). The need to improve evaluation meth- odology was specifically endorsed by the Second Consultation on Environmental Health Services, held in Vilnius, Lithuania, in No- vember 1994 (10).
The structure and format of the present publication are the result of a comprehensive study of evaluation methodology, a review meet- ing with several environmental health professionals and the conclu- sions (11) and recommendations of the Second Consultation of Sen- ior Government Officials on the Development of National Environ- mental Health Action Plans, held in Dubrovnik, Croatia, in October 1996 (12).
The elements discussed and proposed in this book should be used by environmental health professionals (at international, national and local administrative levels) to assist in a number of management activities, such as:
• improving the quality and quantity of information reaching managers, thereby strengthening their ability to make deci- sions about future environmental health services;
• choosing among and prioritizing the various types of envi- ronmental health service;
INTRODUCTION 3
• accounting for expenditures on services to political leaders and communities;
• assessing whether policies have been effective in achieving their objectives;
• identifying opportunities for more efficient delivery of envi- ronmental health services;
• providing a mechanism for recognizing changes in public needs and wants, thereby allowing environmental health ser- vices to be adjusted accordingly;
• identifying more specific research needs; and
• interacting with the community to ascertain its wants and needs.
This book is organized into six chapters. Chapter 1 lays the foundation for the rest of the book. It reviews evaluation princi- ples and focuses on issues pertaining to preparation, planning and design.
Chapter 2 reviews the concepts that underlie environmental health services and discusses how they relate to evaluation. It reviews the development of environmental health services and the inherent dif- ficulties in evaluating these services. The objectives of, potential subjects of and partners for evaluation are also discussed.
Chapter 3 discusses in detail data and indicators, the essential building blocks of evaluation. It describes characteristics that make indicators most useful and discusses in some detail several types of indicator (including process indicators, environmental indicators, health indicators and urban indicators).
Chapter 4 focuses on the process of conducting an evaluation. It presents methods for collecting and analysing the information re- quired for evaluation. Particular attention is given to two areas of environmental health service evaluation that have thus far been over- looked: economic evaluation tools and qualitative evaluation tech- niques.
4 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
Chapter 5 focuses on the final stages of evaluation: writing and disseminating results and conclusions in ways that foster their use in management decisions. It reviews potential uses for evaluation and discusses factors contributing to effective use.
Chapter 6 presents several case studies of evaluation.
The need for evaluation is well documented, as are many strate- gies for its execution. The purpose of this book is not to repeat exist- ing efforts, but to apply evaluation concepts specifically to environ- mental health services. With this objective in mind, references and suggestions for further reading appear at the end of each chapter, and key recommendations are presented in boxes throughout the text.
REFERENCES
1. Environment and health. The European Charter and commen- tary. Copenhagen, WHO Regional Office for Europe, 1990 (WHO Regional Publications, European Series, No. 35).
2. Environment and health. Report on the Second European Confer- ence, Helsinki, Finland, 20–22 June 1994. Copenhagen, WHO Re- gional Office for Europe, 1995 (document EUR/ICP/CEH 212 (FIN)).
3. Declaration of the Third Ministerial Conference on Environment and Health, London, United Kingdom, 16–18 June 1999. Co- penhagen, WHO Regional Office for Europe, 1999 (document EUR/ICP/EHCO 02 02 05/18).
4. Helsinki Declaration on Action for Environment and Health in Europe. Copenhagen, WHO Regional Office for Europe, 1995 (document EUR/ICP/CEH 212).
5. Environmental Health Action Plan for Europe. Copenhagen, WHO Regional Office for Europe, 1995 (document EUR/ICP/CEH 212(A)).
6. MACARTHUR, I. & BONNEFOY, X. Environmental health services in Europe 1. An overview of practice in the 1990s. Copenhagen, WHO Regional Office for Europe, 1997 (WHO Regional Publi- cations, European Series, No. 76).
7. MACARTHUR, I. & BONNEFOY, X. Environmental health services in Europe 2. Policy options. Copenhagen, WHO Regional Office for Europe, 1998 (WHO Regional Publications, European Se- ries, No. 77).
INTRODUCTION 5
8. FITZPATRICK, M. & BONNEFOY, X. Environmental health services in Europe 3. Professional profiles. Copenhagen, WHO Regional Office for Europe, 1998 (WHO Regional Publications, European Series, No. 82).
9. FITZPATRICK, M. & BONNEFOY, X. Environmental health services in Europe 4. Guidance on the development of educational and train- ing curricula. Copenhagen, WHO Regional Office for Europe, 1999 (WHO Regional Publications, European Series, No. 84).
10. Second consultation on environmental health services. Techni- cal report on a WHO consultation, Vilnius, Lithuania, 28–30 No- vember 1994. Copenhagen, WHO Regional Office for Europe, 1994 (document EUR/ICP/EPOL 94 03/MT05).
11. Evaluation of environmental health services in the European Region. Report on a WHO working group, Istanbul, Turkey, 27– 29 June 1996. Copenhagen, WHO Regional Office for Europe, 1996 (document EUR/ICP/NEAP 04 01 02).
12. Dubrovnik initiative on actions for environment and health pro- tection in Europe. Report on a WHO consultation, Dubrovnik, Croatia, 21–23 October 1996. Copenhagen, WHO Regional Of- fice for Europe, 1996 (document EUR/ICP/NEAP 01 02 01).
7
1 Evaluation Principles
and Procedures
This chapter reviews the critical components and purposes of evaluation, as well as general theories of evaluation. Ques- tions discussed here include: Why evaluate? Who evaluates? What can be evaluated? and What evaluation criteria can be used? In addition, many of the terms used throughout the book are defined. Finally, this chapter presents a general evaluation plan, which lays a foundation for the rest of the book.
Evaluation of health services is an established field. Several books and reports have been published on the subject, both inside and outside of WHO (1–3). This publication does not intend to rewrite these documents or restate their arguments. Instead, it aims to fill a gap identified by Member States: to provide a practical introduction for environmental health professionals and managers to evaluate their services. The first task is to review the most critical components and aim of an evaluation, to see how these relate to the overall frame- work of environmental health service management.
The WHO Regional Office for Europe has developed a strategy for promoting health for all in the twenty-first century. Goals and objectives of this plan are broken down into 21 targets (4). These targets not only define the health goals for the European Region, but also describe actions and initiatives that can be taken by Mem- ber States to reach the targets. Health for all monitoring reports such
8 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
as the one published in 1998 (5) present the latest health statistics for the European Region. Reporting progress towards the targets is a key component of the health for all evaluation strategy. These re- ports, however, represent an extremely broad effort to evaluate health programmes at the regional level. Individual Member States are also responsible for monitoring and evaluating their health services.
WHAT IS EVALUATION?
Over the years many definitions of evaluation have been developed (1–3,6,7). In 1981 WHO defined evaluation as, “A systematic way of learning from experience and using the lessons learned to im- prove current activities and promote better planning by careful se- lection of alternatives for future action” (1). Another definition of policy evaluation is: “A systematic assessment and appraisal of policy content, organization, process and effects” (6). These definitions provide a foundation for action, but they do not fully describe the contribution of evaluation to the overall management process.
Evaluation is one aspect of the broad management strategy sought by WHO and its Member States. Fig. 1 shows a simplified concep- tion of this management “cycle”. Policies are first developed and then implemented. Thereafter, evaluation assesses the quality of vari- ous aspects of those policies. Policy effects are not the sole focus of evaluation because other aspects of policy implementation and de- velopment may also be covered. Furthermore, even before a policy is implemented, evaluation may occasionally contribute to the man- agement process. Moreover, identifying and appraising alternative actions can be one of the contributions of evaluation. Finally, the cycle is completed by taking experience and knowledge gained through the evaluation and feeding them back into the policy devel- opment phase. This feedback is an integral part of the management process. If the data collected are ignored in formulating future poli- cies, then valuable resources are wasted. To guide future planning and decision-making, evaluation should be both a flexible and action- oriented part of the management cycle.
Comparisons are a necessary part of evaluation. They provide a framework for analysis; a given situation is often more fully understood
EVALUATION PRINCIPLES AND PROCEDURES 9
Fig. 1. Management cycle
Policy formulation
Implementation
Evaluation
if it is compared with something else. Generally, evaluators make one of three possible types of comparison: the present situation is compared with the past situation, with other situations (similar in context) or with a predetermined standard based on statistical, eco- nomic, political or social theories.
Fig. 2 illustrates the three types of comparison. The dashed line represents planned progress of a particular programme, the dotted line represents actual progress and the solid line represents the progress of another programme.
Fig. 2. Types of comparison used in evaluation
Better
WorseS ca
le fo
r th
e ob
je ct
iv e Planned progress
Actual progress
Progress of another programme
Time
C
X
B ○
○ ○
○ ○
○ ○
○ ○
○ ○
○ ○
○
A
10 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
Looking at point X in isolation presents little useful information to a manager. However, when point X is compared to points A, B or C (that is, to the past situation, to another situation or to the planned targets), it is possible to judge the speed and progress of the pro- gramme – in other words, to evaluate it. Comparing data from the present situation to historical data (point X to point A) is one of the most common comparisons, and rightly so. It is important to under- stand the trends (direction) associated with the progress of a pro- gramme. Comparing point X to another programme (point B) may be useful as well, as long as circumstances support a fair compari- son. The comparison of actual to planned progress (points X and C) is also possible. In some cases, however, there is no specific or op- erational target with which to compare point X.
Time is an extremely important element in all evaluations; ide- ally, comparisons should be made more than once. One-time analy- ses provide only limited “snapshots” of a given situation. To judge the progress of an activity over time, a whole series of snapshots is preferable. If the time scale is long enough, several evaluations can be conducted at regular intervals, and a monitoring system would provide information for the interim periods. If a project’s time scale is shorter, the frequency of such snapshots should be adjusted.
DESIGNING AND PREPARING FOR EVALUATIONS
Evaluation is complex and requires time, resources and planning. This section will address the overall evaluation design and several factors that contribute to the success of an evaluation. Additional details are also provided in Chapter 5.
Designing evaluations Before the individual components of an evaluation are addressed in detail, the overall design must be understood. First, a manager should decide on the general scale of the evaluation – large, small, or some- where in between. It may be helpful to remember that the magnitude of an evaluation should correspond to its purpose and that all the elements will generally have the same scale. Fig. 3 portrays (in an array) some of the different characteristics of an evaluation. Each characteristic is discussed further in subsequent sections of this chapter.
EVALUATION PRINCIPLES AND PROCEDURES 11
Fig. 3. Scale and characteristics of evaluation
Scale Characteristic
Intended purpose
Expected results
Intended audience
Length of evaluation
Frequency
Cost
Public participation and openness
Skill mix required of personnel
Number of staff needed
Smaller
Continuous feedback
Minor adjustments
Management
10 minutes
Daily
Less expensive
Larger
Periodic feedback
Major adjustments
Government
Expected product
Low
Narrow
Few (2–5)
Individual notes
Several days
Once every 2 years
More expensive
High
Broad
Many (15)
Written report
The characteristics vary in scale. In general the level of intensity for all the different characteristics will be relatively similar.
Examples that illustrate the extreme cases are relatively easy to find. A small-scale evaluation might be a 15-minute daily staff meet- ing intended to provide an ongoing channel of communication within a certain group of people. This is an inexpensive activity, adding little to the overall cost of a project. Only a few people are involved, with no specific skill mix, and the intended audience would be the managers themselves. Little planning is required, but a good leader might be needed to keep discussions relevant and productive. Pub- lic participation would be limited given the frequency and detailed nature of this type of evaluation. A clear advantage to this type of evaluation is its ability to quickly and inexpensively provide infor- mation about day-to-day operations. Such an evaluation is limited, however, because of the tendency to focus on small day-to-day prob- lems rather than on larger institutional or structural problems.
In contrast, a large-scale formal evaluation can be appropriate for a service funded by a national or international grant programme. Such an evaluation might take place after a programme has been in place for several years, and may focus on the efficiency of the ser- vice or on the outcome in the population. It may take several days or
12 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
weeks to complete, and might be organized during major project milestones only (for example, the midpoint or end). Personnel re- quirements are likely to be greater owing to the need for qualitative and quantitative (including statistical) analysis and owing to the broader range of issues that are likely to be evaluated. To provide evaluators with the information required, collaboration with a broad base of programme staff is necessary. Systematic planning and f inancial resources are also required in this type of evaluation. A formal report is likely to be written and disseminated to interested parties, including the granting organization. The advantage of this type of evaluation is that it is a thorough, fundamental analysis rather than an incremental improvement. But this also implies that small- scale developments within a service may go unnoticed, especially when evaluators are not from the programme under study.
Fig. 3 is a simplified presentation, and most evaluations will fall somewhere between the two extreme examples just described. Because the scales will not always be exactly proportional and because straight lines cannot always be drawn across the various characteristics displayed in Fig. 3, flexibility has to be maintained, and managers need to keep in mind how the information collected for the evaluation will be used to help them improve services.
Preparing for evaluations Preparing for an evaluation requires consideration of several factors that affect its success. These factors include support, flexibility, cost and public participation. Each factor is discussed in more detail be- low.
Support Appropriate financial and political support is essential for effective evaluation, as is the support of staff and management. For large- scale evaluations, broad support at both the national and the local level helps provide an atmosphere that is more conducive to pro- ducing meaningful results. For smaller-scale evaluations, such as the daily 15-minute staff meeting, support is manifested in other ways, such as staff punctuality, germane discussions and the intro- duction of new ideas that improve service. In both large-scale and small-scale evaluations, a sound communication strategy is essential to secure political support – before, during and after an evaluation.
EVALUATION PRINCIPLES AND PROCEDURES 13
Flexibility and flow of information Evaluations have to be responsive to unforeseen situations that arise. For time-consuming, large-scale evaluations in particular, the strat- egy should be flexible enough to deal with changes in context. One key to maintaining flexibility is information flow, the foundation of the management process. Essential to information flow is the selec- tion and delivery of important information to the right person at the right time. Because an evaluation is largely a matter of collecting data, gathering information and feeding these back into the deci- sion-making process, it is important to integrate the evaluation with the information system. Again, the exchange of information is the key issue here. Medium- to large-scale evaluations, therefore, should follow a similar process, consisting of codified steps defined before performing evaluation activities. In smaller-scale projects and on- going evaluations, some of the steps are less formal or can be com- bined. To ensure flexibility and the flow of information, every envi- ronmental health service activity should establish evaluation as a component at the outset of that activity. Evaluation plans can and should contribute significantly to effective, efficient and equitable environmental health service management.
Cost The cost of an evaluation is also an important consideration. Each step in the evaluation process (outlined below) is associated with a specific cost, either direct or indirect. Direct costs, such as actual resources needed for the evaluation team (supplies, salaries, office space, time) are easier to account for than indirect costs (which are not identifiable with a specific product, function or activity). The cost of not completing an alternative task – the opportunity cost – should also be considered. For the planning process to be effective, proper financing, time and other resources have to be set aside to perform the evaluation, as well as to produce, copy and disseminate results. Cost issues and economic tools are discussed in more detail in Chapter 4.
Public participation and openness The importance of public participation, open decision-making and access to information has steadily gained recognition throughout the European Region. Agenda 21 and the Rio Declaration empha- sized the need for public participation in achieving sustainable
14 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
development. The European Charter on Environment and Health, adopted in Frankfurt-am-Main, Germany, in 1989, highlighted the importance of the public in dealing with environmental and health challenges. In Helsinki, Finland, in 1994, the public and nongovernmental organizations (NGOs) were recognized as indis- pensable partners in the development of national environmental health action plans (NEHAPS). The most significant development in this area occurred at the Fourth Ministerial Conference “Environ- ment for Europe” held in Århus, Denmark, in 1998 (8). At this land- mark event, the participants endorsed the Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters, which provided a legal framework for environmental civic rights in the Region.
Building on the Århus Convention was a major topic of discus- sion at the Third Ministerial Conference on Environment and Health, held in London, United Kingdom, in 1999. The following are some examples of important issues that relate to access to information, public participation and access to justice that were highlighted at the Conference (9).
• Access to information is not only critical for making sound public decisions that relate to environment and health policy, but is also critical for individuals, so that they can make in- formed choices to improve their own health and environment.
• Information policy should aim to ensure that public authori- ties are accountable and operate transparently, and that the public is better informed through awareness-raising measures.
• A system that meets demands for public access to information should contain a general presumption in favour of access, de- lineate the scope of information covered and the range of bod- ies required to provide it, broadly define the terms of access (time limits, costs, form), and make provisions for a limited number of exemptions.
• Individuals should always have the right to access informa- tion about their own health. Individual privacy should be re- spected by structuring health information to prevent identifi- cation of individuals.
EVALUATION PRINCIPLES AND PROCEDURES 15
• Information held by the private sector should be addressed to ensure its adequate flow into the public domain.
• Public communication and active dissemination of informa- tion are essential for developing and implementing environ- mental health policies. A well informed public is more able to participate effectively in decision-making and more likely to support policies designed to create a healthier environment.
• Communication on environmental and health matters is not a simple one-way process but should involve debate, dialogue and feedback, especially during decision-making processes.
• Several key “actors”, including governments, doctors, envi- ronmental health professionals, educational institutions, busi- nesses, trade unions and NGOs, play important communica- tion roles.
• Communication technologies are rapidly evolving and should be incorporated into communication policies. Television, ra- dio, newspapers, the Internet, educational institutions and prod- uct labelling are just some of the important ways to provide information to the public. New technologies for expanding the dialogue with the public are also evolving. It must be rec- ognized, however, that new technologies bring new problems as well as new opportunities.
• The Århus Convention provides a legal framework for public participation in specific activities: programmes, plans and poli- cies (Article 6), and general rules and regulations (Articles 7 and 8, respectively).
• Access to the judicial system is fundamental to providing the public with meaningful involvement in environmental and health matters. Efforts should be made to overcome financial and practical barriers to such access.
In accordance with the principles laid out in Århus and London, evaluations should occur in an open and interactive environment that includes dialogue with anyone affected by or interested in the
16 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
activities under review (stakeholders).1 Public involvement will not always be feasible; certain evaluations may be too small in scope and too detailed for extensive public interaction. Open and trans- parent communication, however, is always warranted and, in most cases, evaluation results should be made publicly available.
THE EVALUATION PLAN
Before embarking on any kind of evaluation, the development of a clear plan is necessary. This plan informs all parties involved about the scope, purpose, methodology, information base and intended use of the results. The plan should clearly state what data are to be collected. Since many evaluations are based on comparisons over time of the same data, managers have to ensure that the correct data are collected from the outset. A clear plan also helps to maintain fairness in evaluations. The plan should clearly state what criteria will be used for the evaluation. This is particularly important when evaluators are drawn from outside the programme or process to be evaluated. A clear plan can also avoid resistance to an evaluation. Many people do not like to be evaluated because they feel they are being judged under circumstances they cannot control. If managers know at the outset what criteria will be used to judge them, they can act to improve these elements. In addition, criteria have to be chosen and defined very carefully, because there is a risk that service ac- tivities will only focus on the criteria that are measured specifically. Whenever possible, the plan should be developed and agreed to in writing. Finally, to remain responsive, the plan should be flexible and allow for adjustments that might become necessary during the evaluation process. The steps presented in Fig. 4 provide a broad guide that can be tailored to, and thus used in, almost any type of evaluation.
Fig. 4 reflects the most important elements of virtually all evalu- ations. The factors (or conditions) that foster effective evaluations include sufficient support, sound communication, openness and flex- ibility. These factors are discussed below.
1 Throughout this volume, the term “stakeholder” is broadly defined as parties inter- ested in or affected by environmental health services.
EVALUATION PRINCIPLES AND PROCEDURES 17
Fig. 4. Suggested steps for an evaluation plan
EVALUATION DESIGN
Purpose
Scope
Subject
Timing
Frequency
Dissemination
ACTUAL EVALUATION
Draw conclusions Methodology Assess subject
DISSEMINATION
EVALUATOR
SECURE SUPPORT
AUDIENCE
CONDITIONS
Support Communication strategy Openness Flexibility
The first major step in an evaluation plan is to secure operational, political and financial support. In some cases (as indicated by dotted arrows in Fig. 4), the later phases of an evaluation may also influ- ence support. For example, preliminary results of an evaluation may encourage managers to support a more elaborate dissemination strat- egy. The “audience”, the people for whom the evaluation takes place, can also influence the amount of support an evaluation will get along the way, though sometimes only indirectly.
The second major step is the development of an evaluation de- sign. This involves the identification of the purpose, scope, subject, frequency and timing of the evaluation. It is also recommended that before carrying out the evaluation, the design should account for the dissemination of the evaluation results. Though the evaluation design is partially the responsibility of the evaluator, it may also be the work of the initiators of the evaluation.
18 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
The third major step is to perform the evaluation itself. This step involves three basic activities. First, identify and select one or more suitable methods for gathering and analysing the necessary data. Evaluators should consider using both quantitative and qualitative tools for gathering data. A combination of these is recommended (see also Chapter 4). Second, gather data and use them to determine such components as service effectiveness, efficiency and outcome. Last, develop and document the results and conclusions.
The fourth major step is to disseminate the results to the appro- priate audience. Keep in mind that the steps and activities in Fig. 4 comprise only a very rough outline of a real evaluation. (A more complete sample outline for an evaluation plan is presented in Box 1.) In addition, it is important to remember that the steps described should be closely integrated into the management cycle. Evaluation is not a separate or isolated process, but an essential element of formulat- ing, implementing and improving environmental health services.
THE PURPOSE: WHY EVALUATE?
There are many reasons for evaluating services. In a health service context, often the broad purpose is to “improve health programmes and the health infrastructure for delivering them, and to guide allo- cation of resources in current and future programmes” (9). In individual situations, many different questions about a particular service may be answered by the evaluation. The following is a list of such questions.
• What has been achieved? • Are the objectives being fulfilled? • Could the objectives be fulfilled more cheaply? • Are the programmes and practices fair and are costs and ben-
efits equitably distributed? • Are the activities sustainable for future generations? • What difference has the service made? • Are the objectives correct and sufficient for this population? • Are the needs of the population being met? • What are the strengths and weaknesses of the service? • Could the resources used be put to better use by providing
different services?
EVALUATION PRINCIPLES AND PROCEDURES 19
Preface: Approaches to foster support, openness, good communication and flexibility
I. Support
A. Organization providing funding B. Expected support from management and appropriate political officials C. Target audience and other stakeholders
II. Design
A. Purpose B. Scope C. Subject D. Frequency E. Timing F. Cost and benefits G. Public participation H. Evaluation personnel � required skill mix for personnel � number personnel needed
I. Dissemination strategy, audience
III. Actual evaluation
A. Methodology � data to be collected � indicators � methods of analysis
B. Results C. Conclusions and recommendations � linking evaluation to future plans and policies � strategy for maximizing use of results
IV. Dissemination of results
A. Audience(s) B. Method(s) of dissemination C. Format of expected product
Box 1. Evaluation plan outline
• Is the amount of information collected appropriate? • What can be done to improve efficiency? • How can this information be used to plan future services? • What are the implications for future service levels? • How should current policies be adapted in response to the
new knowledge?
Managers have to ask themselves why questions such as those above should be answered. Many of the questions are narrowly
20 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
focused, but in combination these questions may lead to a greater understanding of the overall context. Though economic factors may often be the primary motivation for starting an evaluation, this is not always the case. For example, reviewing programme targets and the means to reach those targets on a periodic basis can provide oppor- tunities to learn what is and is not working in a particular activity or programme. This is discussed further in Chapter 2.
Often the information collected, the way it is collected and the way it is presented will vary considerably because of its intended use. Defining the purpose and scope of an evaluation is therefore an integral, preliminary step. Additionally, evaluations should distin- guish between outcomes that result from the services themselves and those that result from uncertainty or chance. Often this distinc- tion is blurred, but stating what is unknown is very important to the long-term viability of the management process.
Minimizing conflict It is important that evaluations be used constructively. An evalua- tion should go beyond simply justifying past actions and identify- ing past inadequacies. The information should be directed towards future decisions and actions. Evaluations are not meant to be a judge- ment that affects staff on a personal level. Instead they should serve as an educational tool for both those evaluating and those being evaluated, and should focus primarily on institutions and policies rather than on the actions of individuals. Since evaluation involves judgement, the character and purpose of this judgement are impor- tant. As mentioned before, the primary reason for evaluating any health service is to learn about the implementation of policies, de- termine their consequences, and feed this knowledge and informa- tion into future efforts to formulate and implement policy.
Undertaking a full-scale evaluation of services can be risky for an organization. A participatory approach from the outset is highly encouraged, particularly when the performance of personnel is part of an overall evaluation; otherwise, an adversarial relationship may evolve between evaluators and those evaluated. For example, per- sonnel being evaluated may perceive the evaluators as opponents who try to take away funding or reduce staffing, and evaluators may characterize personnel being evaluated as obstructive to the process.
EVALUATION PRINCIPLES AND PROCEDURES 21
Both these conflicting claims may be legitimate in certain situations. It is a manager’s task to recognize and prevent resistance to evalua- tion. Managers can emphasize that learning about implementation of policies, or determining the specific outcome of a service, does not necessarily mean taking funds away from existing programmes. In addition, if from the start the individuals being evaluated know, or help develop, the criteria on which they will be judged, the entire procedure is likely to be less intimidating and confrontational. Em- phasis on the educational aspect of an evaluation goes a long way towards reducing confrontation between those being evaluated and those evaluating.
There is great potential for distorting the meaning of evaluation. Many health programmes and services claim to evaluate, yet they do not carry out any systematic analysis of their efforts. Although it may be difficult to evaluate a service in a strictly scientific way, the most systematic evaluation procedures available should be utilized. Similarly, avoiding simplistic conclusions from evaluations is also necessary. It is difficult, but tempting, to classify health programmes and services as either successful or unsuccessful, good or bad. Usu- ally, such designations are of little value because they omit further explanation and discussion of the factors contributing to programme success or failure. Furthermore, uncertainties and unanswered ques- tions about an evaluation will always exist. Though completely avoid- ing these unknowns is impossible, stating them explicitly is necessary.
THE SUBJECT: WHAT CAN OR SHOULD BE EVALUATED?
The subject of an evaluation should reflect its intended purpose. For example, if the main purpose of an evaluation is to determine whether the needs of the population are being met, then the subject will be the relationship between the specified goals and the identified needs in the population. All the different components of the service, either individually or in conjunction with other components, may be evalu- ated. Fig. 5 illustrates the composition of a generic health service. All governmental services and programmes begin at a certain point in time, then they alter inputs and activities to create outputs and desired outcomes. Inputs are the resources, including the raw mate- rial, money, personnel, training and time, that are used to plan and
22 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
carry out different activities. Process refers to the transformation of inputs into activities and outputs, which are the results of the planned activities. Outcome can be thought of as the change in the popula- tion as a result of the outputs. The food inspection service is a sim- ple example of an environmental health service. Personnel and money (inputs) are budgeted for a certain number of inspections (activities) during a given period of time. As a result of these inspections, some establishments are f ined or closed because of health code viola- tions. These results are the outputs. The reduction in, or lack of, food poisoning episodes in the population are then said to be the outcome of the food inspection service.
Confusion may arise about the distinction between output and outcome. The following example may help in better understanding these concepts. The Regional Office for Europe of the World Health Organization has specified 21 health targets for the region (4). Gen- erally, these targets correspond to outputs – that is, measurable, spe- cific results. Together, the targets contribute to the overall goal of the World Health Organization, ensuring health for all in the twenty- first century. This goal, like others (such as becoming a more sus- tainable society), is related to outcome – that is, more general, popu- lation-wide results that may be difficult to measure. Targets, there- fore, generally refer to measurable, specific actions or results. On the other hand, goals refer to more general sets of actions, which may be hard to measure. When planning an activity, it is useful to set targets that are measurable: this may facilitate the production of meaningful results when evaluating programmes or services.
In the past, WHO has defined six basic subjects for an evalua- tion: relevance, adequacy, process, impact, efficiency and effective- ness (1). To these subjects we add equity and sustainability. Each subject is discussed in some detail in the following sections. Defini- tions for these terms are based on those found in the WHO publication
Fig. 5. Generic health service model
Initial situation Input Outcome
PROCESS
Activity Output
EVALUATION PRINCIPLES AND PROCEDURES 23
on guiding principles for health programme evaluation (1). The sub- jects are presented in a specific order intentionally. Ideally, relevance and adequacy should be the first items explored and evaluated. For example, the impact of a service is irrelevant if it does not respond to a specific need in the population.
Relevance The ability of programmes, activities, services and institutions to respond appropriately to human needs (reflected in social and eco- nomic priorities) should be evaluated. Relevance relates these dif- ferent components of environmental health services to the social and economic policy goals of the community. In other words, rel- evance refers to the appropriateness of the service. For example, even if a service caused a change in health status, that change may not be related to the changes envisaged by the planners – that is, it is not relevant to their pre-established needs. This concept is illustrated in Fig. 6 as the relationship between the whole community and the health service, because human needs provide the basis for the rela- tionship. Goals and objectives should also be evaluated for relevance.
Adequacy Adequacy refers to sufficient attention being paid to certain previ- ously determined courses of action, such as the various issues to be considered during broadly based programming. In other words, do the goals specified sufficiently address the needs identified? Both the goals themselves and the relationship between goals and ser- vices can be evaluated for adequacy.
Fig. 6. Evaluation subjects: relevance and adequacy
Initial situation Input Outcome
ADEQUACY RELEVANCE
Activity Output
Community Goals
PROCESS
24 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
Process Process refers to the relationship between inputs and outputs (see Fig. 7). Evaluating a process deals with comparing actual activities with those scheduled. The reasons for achievements and shortcom- ings should be noted, along with remedies for shortcomings. The purpose of a process review is often to facilitate the monitoring of day-to-day activities, the supervision and implementation of ongoing activities, milestones, personnel matters, supplies and equipment, and the monitoring of how money is spent in relation to the budgets allocated. Process evaluations also allow managers to discriminate between problems that arise from the planning of activities and those that arise from preparing for and implementing activities.
Impact Impact is an expression of the overall effect of a service on health and related socioeconomic development. The evaluation of impact is an attempt to measure the more encompassing population-based consequences of the services. The outcome is thus compared to the initial situation. In general, it is much easier to study process than to study impact.
Efficiency Efficiency refers to the direct relationship between the results ob- tained from a service (outputs) and the amount of effort expended in terms of human, financial and other resources (inputs), health processes, technology and time. Efficiency demonstrates how well the outputs have been produced from the inputs. The aim of assess- ing efficiency is to improve implementation, but it also serves to identify the strengths and weaknesses of monitoring. Financial
Fig. 7. Evaluation subjects: process and impact
PROCESS EVALUATION
IMPACT EVALUATION
Initial situation Input Outcome
PROCESS
Activity Output
EVALUATION PRINCIPLES AND PROCEDURES 25
auditing is a particular type of monitoring that often aims to improve efficiency. Placing a monetary value on inputs, outputs and outcome is one method of establishing programme efficiency. This, however, is not an easy task, especially for measurements of outcome. In fact, economic quantification of environmental impacts and environmen- tal health impacts is the subject of much debate in the field of environ- mental economics (more about efficiency can be found in Chapter 4).
Effectiveness Effectiveness expresses the degree of attainment of the predeter- mined objectives and targets of a service. Effectiveness is assessed in order to improve the formulation of services, objectives and goals. Qualitative analyses of effectiveness are often preferred to quantita- tive analyses because they provide an in-depth, specific insight into context and circumstances (more about quantitative and qualitative analyses can be found in Chapter 4). Assessment of effectiveness should also be carried out with the community in mind: their satis- faction and dissatisfaction with services are essential to a successful programme. When combined with adequacy, effectiveness repre- sents the degree of attainment by the services themselves. For exam- ple, a cursory evaluation may show that environmental health ser- vice targets are being met. More detailed research, however, may show that this success is not the result of the service’s activity, but is instead the result of a wholly different development, such as the status of the national economy. Thus, the impact of the service may not be particularly effective. An important distinction between ef- fectiveness and impact is that effectiveness focuses broadly on the contribution of a particular service in attaining the predetermined targets of a service or policy, while impact does not.
Equity Equity implies “that everyone should have a fair opportunity to at- tain his or her full health potential and, more pragmatically, that no one would be disadvantaged from achieving this potential” (1). Equi- ty, however, is a particularly difficult subject to evaluate because it can be defined in a number of different ways, even for the same situation (3). Owing to differences in socioeconomic conditions, spending the same amount of money in different areas can have different results. For example, in providing for clean drinking-water, there are at least three ways to define equity:
26 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
1. in terms of input: authorities may decide to spend equal amounts of money on water supply in both richer and poorer areas;
2. in terms of outputs: authorities may decide to spend money so that richer and poorer areas receive the same water supply services (more money may be needed in poorer areas to com- pensate for prior differences in services); and
3. in terms of outcomes: authorities may decide to spend money so that richer and poorer areas have the same level of a certain disease (again, even more resources may be required in poorer areas to compensate for prior differences in services, as well as to compensate for differences in socioeconomic conditions).
The definition chosen influences the results of the evaluation and, thus, should be adhered to throughout the process.
Sustainability At the United Nations Conference on Environment and Develop- ment, held in Rio de Janeiro, Brazil, in 1992, sustainable develop- ment was defined as “development that meets the needs of the present without compromising the ability of future generations to meet their own needs” (10). World governments recognized that economic de- velopment, health status and environmental resources are closely related, and they agreed to pursue sustainable development. This is a long-term process and may not result in significant health changes in the short term. In addition, sustainable does not necessarily mean healthy, because promoting sustainable development may remove or diminish attention to important health issues (11).
Programmes and procedures can be evaluated in terms of their sustainability. An intervention should be sustainable not only in terms of environmental resources, as suggested at the Rio conference, but should also be sustainable in terms of financing, personnel commit- ments and political will. Despite growing attention to sustainable development, the ability to conclude that a particular service is or is not sustainable remains elusive. So far, it has only been possible to argue that one activity is more or less sustainable than another.
EVALUATION PRINCIPLES AND PROCEDURES 27
In summary, the subject of an evaluation may vary widely, de- pending on the questions that managers wish to answer. In addition, the order in which the above subjects are addressed is important. For example, impact cannot be evaluated unless the evaluator is reason- ably sure that the programme has been implemented as planned (that is, unless a process evaluation has been done), and measuring effi- ciency cannot be done without first knowing the impact of a service.
WHO EVALUATES AND WHEN?
Choosing the appropriate evaluator is an important concern that is closely related to the designated purpose of any evaluation. Desig- nating a team of people from relevant sectors who have an interest in planning and managing the progress of the evaluation is an exer- cise frequently performed to determine the subject, scope and time frame of the evaluation. Evaluators should possess the necessary skills and knowledge to do an evaluation. One of the purposes of this publication is to help provide these skills.
Internal and/or external It is important to consider the position of the evaluators and their relationship to the programme being evaluated (12). Internal evalu- ators are affiliated with the organization being evaluated and exter- nal evaluators are not, and each has its strengths and weaknesses.
Internal evaluations are useful because the analysis is likely to contribute directly to the information needs of an organization. Moreover, data collection is more straightforward and less expen- sive. The credibility of internal evaluations, however, is often chal- lenged because the results may only have limited use outside the organization, the potential subjects for evaluation are narrow and the evaluators are perceived as being biased.
External evaluations are more effective when addressing broad governmental issues and programmes between different sectors of government. Results may be more widely used because external evaluators have greater access to decision-making processes and are perceived as having more authority and credibility. In compari- son with internal evaluators, professional evaluators (consulting firms,
28 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
auditing offices, academic institutions) may be more experienced, or possess a broader range of evaluation skills. Lack of familiarity with the programmes to be evaluated may, however, limit external evalua- tions, and managers may be less willing to accept an outsider’s view.
Often, especially in larger evaluations, both internal and external reviewers are employed, which helps to prevent conflicts that arise from the perceptions that outside evaluators do not fully appreciate a programme and that internal evaluators are not objective. The best combination of internal and external evaluators will depend on the programme or service to be evaluated.
Timing and frequency The frequency of evaluations is an important decision. It should be often enough to produce meaningful information, but not so often as to prevent the progress of the work at hand. It is natural for smaller- scale evaluations to occur more frequently than large-scale evalua- tions. For example, it is not feasible to conduct large-scale evalua- tions on a weekly or even a monthly basis. But it is also undesirable for them to occur only at the end of a project or at an intervention. The monitoring and information systems should be able to provide information on a regular basis, thus allowing infrequent full-scale evaluations. Midterm evaluations may be a viable solution to the issue of frequency. The optimum time frame should be derived from a consensus among advocates and designers of an evaluation. It is important to remember that evaluation should reflect the continuing nature of the management process. Sometimes, evaluations are seen as discrete events that occur only at the end of a project, service or intervention. Although it is recommended that projects always be evaluated at their conclusion, this is clearly not the only time for an evaluation; less formal information and monitoring systems should identify problems as they arise throughout a programme’s imple- mentation. The essential task is to pass on correct information in time to make the necessary decisions. Thus, the evaluation manager needs to strike a balance among such competing factors as time, cost and personnel constraints in order to determine the optimal time frame and optimal format (formal or informal) for a major evalua- tion of the information collected.
Box 2 summarizes the key recommendations of this chapter.
EVALUATION PRINCIPLES AND PROCEDURES 29
Recognize evaluation as a part of the ongoing cyclical management process
Identify interested and affected parties as soon as possible; develop political and community support by bringing them into the decision-making process
Keep evaluations open to new circumstances, new facts, new opinions, new ideas and new stakeholders
Develop and follow a written plan for evaluation; disseminate it to the stakeholders at the beginning of the project (see Box 1)
Prevent the perception that evaluation is a judgemental activity; stress its edu- cational and constructive nature
Narrow the scope of the evaluation according to its intended purpose
Strive to develop a clear monitoring and information protocol that delivers im- portant information to decision-makers but limits unimportant information
Include a mixture of internal accounting of programme operations and external reviews of them
Distinguish between outcomes that result from the services themselves and those that result from uncertainty or chance
Avoid oversimplified conclusions
Be both flexible and action-oriented in order to guide future planning and decision-making
Promote issues of equity, intersectoral collaboration and community partici- pation
Box 2. Key recommendations from Chapter 1
REFERENCES
1. Health programme evaluation. Guiding principles for its appli- cation in the managerial process for national health develop- ment. Geneva, World Health Organization, 1981 (“Health for All” Series, No. 6).
2. Health systems research in action. Case studies from Botswana, Colombia, Egypt, Indonesia, Malaysia, the Netherlands, Norway and the United States of America. Geneva, World Health Organi- zation, 1988 (document WHO/SHS/HSR/88.1).
3. REINKE, W., ED. Health planning for effective management. New York, Oxford University Press, 1988.
30 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
4. HEALTH21. The health for all policy framework for the WHO Eu- ropean Region. Copenhagen, WHO Regional Office for Europe, 1999 (European Health for All Series, No. 6).
5. Health in Europe 1997. Report on the third evaluation of progress towards health for all in the European Region of WHO (1996– 1997). Copenhagen, WHO Regional Office for Europe, 1998 (WHO Regional Publications, European Series, No. 83).
6. TIJSSEN, I. & ELSINGA, E. Evaluatie-onderzoek op het terrein van de gezondheidszorg [Evaluation research in health care]. In: Maarse, J. & Mur-Veeman, I., ed. Beleid en beheer in de gezondheidszorg [Health care policy and management]. Assen, Van Gorcum, 1990.
7. ROSSI, P. ET AL. Evaluation: a systematic approach, 6th ed. Thou- sand Oaks, Sage Publications, 1999.
8. UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE COMMITTEE ON
ENVIRONMENTAL POLICY. Convention on access to information, public participation in decision-making and access to justice in environmental matters. Report on the Fourth Ministerial Conference “Environment for Europe”, Århus, Denmark, June 23–25, 1998. New York, United Nations, 1998 (document ECE/CEP/43).
9. Managerial process for national health development. Guiding principles. Geneva, World Health Organization, 1981 (“Health for All” Series, No. 5).
10. Terminology for the European Health Policy Conference. A glos- sary with equivalents in French, German and Russian. Copen- hagen, WHO Regional Office for Europe, 1994.
11. Our planet, our health. Report of the WHO Commission on Health and Environment. Geneva, World Health Organization, 1992.
12. Improving evaluation practices: best practice guidelines for evaluation and background paper. Paris, Organisation for Eco- nomic Co-operation and Development, 1999 (report no. PUMA/ PAC(99)1).
FURTHER READING
Access to information, public participation and access to justice in environment and health matters. Report for the Third Ministerial Conference on Environment and Health, London, United Kingdom,
EVALUATION PRINCIPLES AND PROCEDURES 31
16–18 June 1999. Copenhagen, WHO Regional Office for Europe, 1999 (document EUR/ICP/EHCO 02 02 05/12).
FINK, A. Evaluation fundamentals: guiding health programs, research and policy. Thousand Oaks, Sage Publications, 1993.
Healthy decisions. Access to information, public participation in decision-making and access to justice in environment and health matters. Copenhagen, WHO Regional Office for Europe, 1999.
Manual on public participation in environmental decision making. Current practice and future possibilities in central and eastern Eu- rope. Budapest, Regional Environmental Center for Central and East- ern Europe, 1994.
Program evaluation kit. A Project of the Center for the Study of Evalu- ation at the University of California, Los Angeles. Thousand Oaks, Sage Publications, 1988.
SCRIVEN, M. Evaluation thesaurus. Thousand Oaks, Sage Publica- tions, 1991.
The previous chapter presented general principles and con- cepts of evaluation. This chapter builds on that foundation and applies the same concepts specifically to environmental health services. It begins by defining terms and briefly track- ing the historical development of environmental health ser- vices. Next, reasons for evaluating environmental health services are provided. Then, some of the particular difficulties of environmental health service evaluation are discussed. The last two sections explore the evaluation subject, authors and audience.
CONCEPTS AND DEVELOPMENT OF ENVIRONMENTAL HEALTH
SERVICES
Environmental health is a relatively new term that has been used to define both a condition affecting human health and a profes- sional discipline. The term has evolved and has come to mean different things to different people. Recent international meet- ings (1–3) have attempted to develop consensus def initions of environmental health and environmental health services. The terms described here have been generated and accepted at these inter- national meetings and have been used in previous volumes in this series (4–7).
Theoretical Framework for Evaluating
Environmental Health Services
2
33
34 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
Environmental health Many countries have difficulties in working with the definitions of public health, sanitation, hygiene and environmental health, and of- ten freely interchange the terms. Questions also arise about the rela- tionship between healthy lifestyles and environmental health. These difficulties are further magnified when terms are translated into other languages, which naturally reflect different cultural and historical developments. In many countries, the term environmental health arouses interest because, historically, environmental and health mat- ters have been dealt with in a generally unrelated manner. The WHO Regional Office for Europe has been actively pursuing and promot- ing a common definition for environmental health (4):
Environmental health comprises those aspects of human health, including quality of life, that are determined by physical, chemical, biological, social and psychosocial factors in the environment. It also refers to the theory and practice of assessing, correcting, controlling and preventing those factors in the environment that can potentially affect adversely the health of present and future generations.
Environmental health comprises diverse interest areas, including but not limited to: air pollution, food safety, the built environment, injury prevention, land use planning, noise pollution, occupational health, chemical safety, radiation, tourism, waste management and water management. As with other disciplines, including infectious disease control, health education, epidemiology, health research and mental health, the discipline of environmental health serves as one component of public health.
One of the main goals of integrating environment and health is to look at many different sectors with a holistic view. The exact mix- ture of sectors to be included in the discipline of environmental health will inevitably be defined differently by different countries, and debating the inclusion or exclusion of particular sectors would not serve the purpose of this publication. The point to be emphasized is the need to measure and explore more broadly the interactions and interrelationships of the different sectors.
Environmental health services The term environmental health services is also a relatively new con- cept and is defined as follows (5):
35THEORETICAL FRAMEWORK
Environmental health services can be defined as those services implement- ing environmental health policies through monitoring and control activi- ties. They carry out that role by promoting the improvement of environ- mental parameters and by encouraging the use of environmentally friendly and healthy technologies and behaviour. They also have a leading role in developing and suggesting new areas of policy.
This definition implies a very specific meaning for the word ser- vice – that is, it refers to the institutions that implement, create or otherwise address environmental health issues. But the term envi- ronmental health services is used equally to describe the work and activities that are carried out by these environmental health institu- tions and agencies. Environmental health services act as the direct interface between the policy-makers and the subjects of that policy. They also relate directly to the general public, in dealing with their complaints and concerns about environmental health issues. There is a need for such services to be appropriately targeted and sympa- thetically responsive to public needs, while at the same time being representative of the views of the authorities (local, regional or na- tional government). One of the more difficult tasks for any service delivery is to ensure that services are appropriately targeted and re- sponsive, and this is particularly relevant for environmental health services. Environmental health service managers require accurate assessments upon which decisions for the future can be based; how- ever, the methodology for performing such assessments in the envi- ronmental health arena is not very strong.
An improved methodology to gauge the process, impact, relevance and adequacy of services is required. Such a methodology would help to raise the quality and quantity of information reaching man- agers, thereby strengthening their ability to make decisions about the delivery of future services. In addition, methods for evaluating health services can also help in developing more clearly the role of a service and its formal strategy. This volume aims to fill some of these gaps.
Historical development of environmental health The issues addressed by environmental health services must be de- termined for a specific situation and are particularly dependent on socioeconomic circumstances and traditions. It is impossible to ar- gue that present expenditures for preventive health measures will
36 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
abolish the need for future expenditures. Clearly, the nature of hu- man ailments has changed in the past and will continue to change in the future, constantly requiring new resources. It is the job of public health professionals to track that change as it occurs and to try to predict future changes in order to provide the best health services for the population. The diagram shown as Fig. 8 has been used to model the long-term development of two aspects of public health: changes in disease patterns and subsequent professional responses.
Health transition theory suggests that in most countries the char- acter of disease has changed with industrial development. Sanitary diseases (that is, acute health responses associated with poor living conditions, improper sanitation and overcrowding) have given way in many areas to “environment and lifestyle” diseases, which gener- ally develop over a longer period of time and have less direct links to the environment. Similarly, as these diseases are better controlled, psychosocial diseases, such as mental depression, will need to be addressed. The focus of professional development is changing from a substantially medical and engineering discipline to social, behav- ioural and management disciplines.
The model appears applicable to most countries in the European Region, although certainly each country might fall on a different place along the time axis. The location depends primarily on the
Source: MacArthur & Bonnefoy (5).
Fig. 8. Professional responses to changing disease patterns
D is
ea se
le ve
l
Unknown Social and management basis
Engineering basis
Medical basis
Time
Psychosocial diseases
Environment and lifestyle diseases Sanitary
diseases
37THEORETICAL FRAMEWORK
level of economic wealth within a country and its individuals. Simi- larly, different regions or localities within a country (and even within cities) might appear in substantially different places along the dis- ease axis. This can be clearly demonstrated by the rise of tuberculo- sis and other “diseases of the poor” in many declining areas of large cities. It is possible to move along the time axis, but as progress occurs the requirements of the environmental health services will change. Once the basic needs for survival of a population are ad- dressed, the possibilities for environmental health activities seem only to be limited by resources. Nevertheless, the problem of scarce re- sources is a serious one throughout the Region; thus, in all coun- tries, environmental health services should be carefully chosen and planned to have a specific impact. Again, this publication does not aim to prescribe which environmental health goals national, regional or local authorities should choose; they have to complete this task on their own. It aims instead to provide a methodology for evaluat- ing the services provided at a given time, and after the decisions have been made. Once the tools for evaluation are used, and a well organized evaluation plan is established, prioritizing services should become easier.
Some of the disciplines addressed here will not be applicable immediately to all countries in the Region, but will take on a grow- ing importance as their development progresses according to this model. Applying the theory in this way will give countries along the first part of the time axis opportunities to share experiences with others, thereby possibly moving them further along the time axis. Integrated information sharing may allow countries to better prioritize services and to utilize or apply research completed elsewhere. This volume is especially targeted at those countries currently in the mid- dle of this process – that is, countries with enough economic strength or sufficient political will to devote substantial resources to the pre- vention of diseases of environment and lifestyle.
Fig. 8 also serves to illustrate the changing qualities needed by health-related professionals to address problems associated with changes in disease patterns. How the progression and rise of one type of disease occurs is open to speculation. Perhaps as one disease begins to be cured the others become more apparent, or perhaps the public begins to demand progressive improvements once one disease
38 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
has become manageable. The progression and rise of diseases is also closely connected to broader developments, such as those in economy, culture and science.
For diseases of environment and lifestyle, the skills required have changed from purely medical to those more focused on engineering and management. For example, it is already apparent that reports of diseases related to psychosocial factors are increasing and, in the future, will need to be addressed by professionals with different skills. This emphasizes the need for building competency within environ- mental health services. Two previous volumes in this series address these aspects of training and curricula development (6,7).
In general, the diagram depicted in Fig. 8 is relevant for most European countries. Where sanitary or hygienic problems have been predominant, the response has been to utilize the skills of doctors and engineers. This is the present situation in many of the countries of central and eastern Europe (CCEE), although the need for envi- ronmental health professionals trained in management and social science is now recognized. Facing the new environmental health chal- lenges is possible only if service staff have the appropriate and var- ied skills necessary. The breakdown of traditional roles does, how- ever, take time and a lag phase is inevitable. It is very difficult to predict and prepare for future service needs, but this is a basic func- tion of all managers and politicians. Evaluation of current services is the foundation upon which decisions about the future are based.
WHY EVALUATE ENVIRONMENTAL HEALTH SERVICES?
There are numerous reasons for carrying out an evaluation of an environmental health service. Primarily, it is important to address the general management goals of evaluating the effectiveness, effi- ciency, impact, process, relevance and adequacy of policies, activi- ties, objectives, services and goals. Evaluations can be used to de- termine what has been achieved by the service, how the service works, and how to improve the efficiency and process of services. Without evaluating environmental health projects, it is difficult to ascertain whether and how well objectives have been achieved. Thus, an examination of the capabilities and effectiveness of services can
39THEORETICAL FRAMEWORK
be used to assess whether policies have been effective in achieving their objectives. In addition, evaluations provide valuable informa- tion about the process of delivering services. Furthermore, learning more about how the services operate may demonstrate problems in the implementation process. Knowing what the problems have been helps define ways to improve the efficiency and effectiveness of the service.
As a tool, improved evaluation also assists managers in choosing between the various types of environmental health service that an authority can provide and, perhaps, in prioritizing them. In the past, the mixture of environmental health services that authorities have chosen to provide has rarely been the result of scientifically calcu- lated or strategic planning. A careful evaluation of the benefits of current services could identify unnecessary redundancies, or gaps that require filling. Unfortunately, while it is relatively easy to find material that provides advice on how to design and implement spe- cific environmental health services (for example, waste disposal or food inspection), there is little guidance on how to choose between different services, or between specific actions. One can ask: Given limited resources, which service has the better impact on the health of the population? One can also ask: Which has a smaller, negative impact on the environment? Both questions are important, but not easily answered, given the current methodology and practices of service delivery.
Evaluations also serve to improve the knowledge of programme impact. This knowledge in turn helps justify expenditures on
Draw upon experiences to improve overall service delivery Improve efficiency and process of services Determine effectiveness and impact of goals, objectives and services Ascertain relevance and adequacy of goals, objectives and services Describe expenditures on environmental health services to political leaders and communities Recognize and respond to public needs and wants Prioritize research and plan for the future Prioritize actions and decisions Find allies in other agencies, services or sectors
Box. 3. Primary reasons for evaluating environmental health services
40 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
environmental health services to political leaders and communities. An evaluation of services may also act as a mechanism through which changes in public needs and wants can be recognized. In order to evaluate any of the service goals, objectives, outputs and outcomes, managers f irst need to know public expectations and wishes. A proper method of evaluation helps to identify inadequate and irrel- evant services and helps to ensure that such services will be ad- justed accordingly.
Additionally, an evaluation may be used as a mechanism for prioritizing research. “Further research required” is a familiar phrase. Often, public and environmental health officials do not have the luxury of waiting for additional research; instead, they make deci- sions based on the best available information. In such cases, the precautionary principle2 is applied and the need for more research remains. It is therefore important that the most relevant questions be addressed first, and evaluation is a tool for determining where to begin. Research is also tied to the need for applying the knowledge gained from an evaluation to making decisions about the future, because once the gaps have been defined, research can be directed to address these gaps. A balance should be struck between the need to act immediately and the need to wait for further information.
Evaluation may also foster interactions between agencies and or- ganizations that ordinarily do not cross paths. Through these inter- actions, personal relationships and alliances between individuals with similar objectives and perspectives may be developed.
As stated in Chapter 1, managers should ask themselves why they are evaluating services (Box 3). Sometimes an evaluation is the result of economic or financial considerations; in other cases man- agers may simply wish to know how much progress has been made. The need to inform and convince the public about the value of the service is also an important underlying reason to evaluate. Evalua- tion may, in some cases, open the door to two-way communication with the public (see Chapter 5 for more information on communication
2 The precautionary principle instructs managers to take action in the interest of protecting public health, even when fully conclusive evidence is unavailable. In other words, when faced with large uncertainties, managers are encouraged to err on the side of overprotection.
41THEORETICAL FRAMEWORK
issues). Finally, the value of having an integrated programme of serv- ices is greater than the simple sum of its components. A thorough evaluation of programmes may provide examples of this to service managers.
BARRIERS TO EVALUATING ENVIRONMENTAL HEALTH SERVICES
Environmental health services are particularly difficult to evaluate through traditional mechanisms. Many years ago a WHO working group (8) identified several of these difficulties, but most of them have not yet been adequately addressed. There are six main barriers to effective evaluation.
1. Environmental health services try to produce a non-event. How can it be proved that a non-event is the result of a service?
2. Services are administered by several agencies, such as health, environment, agriculture, industry and transport.
3. Environmental health services address three different time pe- riods: the past, present and future.
4. There are many weak and indirect links between environment and health effects; long latencies are usually associated with known environmentally caused diseases.
5. It is difficult to define the community and, thus, to determine the relevance and adequacy of the services.
6. Unreliable information limits long-term comparisons.
An important way to overcome these barriers is to recognize and discuss them with the evaluation team at the beginning of and throughout the process. The remainder of this section discusses each of these barriers and suggests ways in which they can be addressed.
Proving a negative Many environmental health services strive to produce non-events; in other words, the purpose of many environmental health services
42 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
is to prevent an adverse effect from occurring. Thus, if the services are well targeted and managed properly, their effects will be to pro- duce the absence of an event. If there are no events to measure or discuss, how can a manager prove that the services are working prop- erly? Similarly, it is a problem that many environmental health ser- vices work to maintain the status quo. Measurement is especially difficult when health effects fall below what is detectable individu- ally, or what is unacceptable socially. So, while it may be easy to identify services that are not working properly (complaints from citi- zens generally provide a good warning system), it may be difficult to prove that services are working properly. Typically, these prob- lems have been overcome by identifying changes in health status, in environmental parameters or in indicators of them that, given a proper design, might be related to service activities.
Multi-agency responsibility for environmental health services Environmental health services have traditionally been administered through more than one governmental agency, thus making coordi- nation of evaluation efforts difficult. Typically, it is not just minis- tries of health or of environmental protection, or both, that deliver environmental health services. Many other sectors and their minis- terial authorities, such as agriculture, industry, energy, transport, tour- ism and labour, are involved. Generally, none of these authorities knows what services the other sectors and authorities are providing, and so a comprehensive evaluation of the effects of the services is not possible. There have been recent efforts to promote inter- sectorality. The Helsinki Declaration (9), for example, places par- ticular emphasis on improving coordination among different gov- ernmental authorities dealing with subjects that relate to environ- mental health. Additional efforts, however, are required to increase the probability of a sufficient evaluation; for instance, a coordinated evaluation of all environmental health services in the relevant sec- tors could be incorporated into NEHAPs.
Past, present or future? Another barrier to effective evaluation methods is that environmen- tal health services must address and act in three different time peri- ods: the past, present and future. These services must repair past damage, protect the present environment and provide good practices
43THEORETICAL FRAMEWORK
in the future. Policies cannot simply address only one of these peri- ods of time; a balance should be made among them. For instance, it may be difficult to quantify the value a particular resource will have in the future.
Inadequate links to health outcomes Despite the advances of the past 20 years in understanding the links between exposure to environmental agents and its impact on health, there are still many questions and unproved theories. This lack of a clear scientific link has hindered the development of comprehen- sive indicators that measure such health effects. Furthermore, the long latency periods associated with many environmentally caused health effects create difficulties for comprehensive evaluations. Many types of environmental health service are trying to prevent health effects that occur over a long period of time, and that only rarely manifest themselves in the general population. Programmes that are implemented today may not produce measurable impacts on health for months or sometimes years. Thus, the use of traditional epide- miological methods to measure and evaluate the impact of pro- grammes is often inadequate.
The entire community is the client Evaluating environmental health service activity can be done within the broader context of its clients or target population. Generally, it is easier to evaluate a service directed at a well defined, limited popu- lation; however, unlike some health services that have more or less defined populations (for example, mothers and children of low so- cioeconomic status in many maternal and child health services), the community as a whole is usually the “client” of environmental health services. In many cases, it is therefore very difficult to narrow or further define and measure the “community” that benefits from par- ticular services. In such cases it may also be problematic to promote or evaluate equity – that is, it is hard to identify which individuals receive attention, protection or care, and which do not. When the entire community is the client, it is sometimes difficult to strengthen a manager’s abilities to improve services that are based on the needs and wants of the population. This, however, is not always a major problem. For example, environmental health services may be focus- ing only on specific risk groups, such as children or the elderly, that are in specific need of help. This focus on a well defined, limited
44 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
population facilitates population-based measurements in evaluation research.
Data of unknown quality One of the biggest barriers to effective evaluation is unreliable in- formation. Because of their nature, most environmentally caused health effects require analyses of long-term health trends. Though there is a great deal of long-term health and environmental quality monitoring data available for the CCEE and the newly independent states (NIS), the validity and reliability of much of these data have been questioned (for example, see the Estonian case study in Chap- ter 6). A decade after the fall of the Berlin wall, countries, and some- times local authorities, are developing their own monitoring sys- tems that are occasionally incompatible with each other. This makes comparisons between and within countries unreliable.
Problems of data quality and compatibility are not limited to the CCEE and may even cause difficulties in western Europe. For exam- ple, significant advances in technology in the past few decades have improved water sampling techniques, making them much more sen- sitive to toxic substances. In some areas, current raw data, when compared with data from as recent as 5 years ago, appear to indicate that drinking-water is becoming more toxic. It is much more likely, however, that the change is due to more accurate sampling tech- niques than to truly higher levels of toxicity. But it is still a problem for managers to distinguish between these two scenarios, especially if they are trying to use the level of toxicity in drinking-water as an indicator of success of their water protection service. It is a dilemma: on the one hand, it is desirable to utilize new technologies as they become acceptable and unrealistic to expect managers to ignore such technologies; on the other hand, managers clearly have to act to ensure that their water sampling techniques are consistent and that they produce accurate and reliable results.
Tools from other disciplines The barriers to evaluating environmental health services have not been adequately addressed or resolved since a WHO working group (8) met almost 30 years ago. Furthermore, barriers such as poor data quality are unlikely to be fully resolved in the near future. Admit- tedly, there is much to be done in certain areas, but this should not
45THEORETICAL FRAMEWORK
be used as an excuse for not conducting evaluations. Evaluation is absolutely necessary for the delivery of quality environmental health services. Much can be done immediately to improve evaluations based on tools used by other disciplines. The remainder of this book describes some of these tools in some detail and suggests how they might be adapted and applied in the context of environmental health services.
WHAT ASPECTS OF THE SERVICES ARE TO BE EVALUATED?
As stated in Chapter 1, what is evaluated (the subject) is closely related to the goal or purpose of the evaluation. It is strongly recom- mended that a team of interested parties first clearly define the pur- pose of an evaluation and then decide the subject and scope of the evaluation. A balance should be reached between extremely spe- cific evaluations, which provide more detailed information on a lim- ited range of topics, and general evaluations, which provide conclu- sions that are less specific but may be more widely applicable.
It is impossible for authorities to evaluate environmental health services in their entirety at one time. The tasks and duties of envi- ronmental health services are simply too vast and diverse. Further- more, resources are not likely to cover evaluation of each subject (such as efficiency or relevance) for each of the different aspects of environmental health services. In this context, “aspects” refers to the various programmes that can be evaluated (such as accidents, waste management and laboratory practices) or to management ac- tivities (such as enforcement, financial management and training). Individual countries and districts have to prioritize evaluations ac- cording to local circumstances. Fig. 9 presents three questions re- lated to the subject of an environmental health service. This figure is intended to serve as a guide and to initiate discussion, but is not intended to relate all potential services that fall within the realm of environmental health.
It is clear that both specific programmes (interventions) and dif- ferent elements of management should be evaluated, so these two areas have been separated in Fig. 9. As the figure suggests, all types of management and intervention can be evaluated, yet some will
46 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5
inevitably be more useful than others. Some examples have been singled out in Box 4, with the intention of clarifying what each type of evaluation achieves for the evaluator. Examples of questions that might be answered by each type of analysis are included in the brief descriptions that follow. In addition, these examples are meant to provide a more concrete understanding of a manageable subject for an evaluation.
WHO AND FOR WHOM?
As stated in Chapter 1, it is recommended that a team of interested people be selected to manage large-scale evaluations. Creating a team that includes a mixture of people directly involved with the programme (or decisions to be evaluated) and impartial reviewers with evalua- tion experience is often the preferred strategy. Depending on the
Fig. 9. The subject of environmental health service evaluations
(1) What should be evaluated?
(2) What part of service delivery should be evaluated?
(3) What subject or criteria should be evaluated?
Process Output Outcome Input Impact
Programmes Management elements
Costs Reinforcement
Information system Financing
Regulations Staffing Training
Accidents Air pollution
Built environment Chemical safety
Food safety Global environmental issues
Laboratory practices and results Noise pollution
Occupational health Soil pollution
Waste management Water pollution Water supply
Adequacy Effectiveness Relevance Efficiency Sustainability Equity
47THEORETICAL FRAMEWORK
Inputs and outputs of training personnel Analysing inputs and outputs determines the utilization of resources and the delivery of planned activities. For training personnel, this might include an analysis of the cost, supplies, personnel and person-hours (the inputs) and an analysis of the number of people trained and the number of training sessions completed (the outputs). Such an analysis might include the information provided in the training courses, but it would not give any indication of the quality of the ser- vices.
Outcome of a car accident prevention programme Outcome is studied in order to measure changes that occur at the population level as a result of programme activities (outputs). For example, investigating the outcome of a car accident prevention programme would mean looking for changes in the number and types of car accident. If alcohol were an important consideration, the number of accidents related to alcohol might be measured. Also, a change in behaviour (such as the practice of wearing a seat belt) may be a key consideration.
Relevance of a particular regulation Relevance pertains to whether the right thing is being addressed. Regulations, for example, can be studied for relevance. To do this, one might ask whether a risk being addressed is important enough to warrant regulation. For the case of anti-smoking regulations, one might want to determine whether they would be warranted in public restaurants.
Adequacy of an air pollution monitoring service Adequacy attempts to measure whether enough of the right thing is being done. Air pollution monitoring services may be studied to determine whether the right substances are being monitored in the right place, at the right time, with the right techniques and for a long enough period of time.
Process of waste management Process is the means by which inputs are transformed into outputs – that is, the activities that occur. Analysing the process of waste management services can identify important strengths and weaknesses in the procedure. By tracking the process of waste generation, collection and end-use/disposal, one can gain valuable insights into ways to improve processes and create incentives for recycling, reducing and reusing waste.
Impact of food safety programmes Impact analysis is concerned with the broad, population-based consequences of services (outcomes). For example, fluctuations in food-related morbidity and mortality might be a measure of the impact of food safety programmes.
Efficiency of the enforc