HEAL THY CITIES - HEALTHY ISLANDS PROGRAMME The WHO programme "Healthy Cities - Healthy Islands" for the Western Pacific Region has been developed in response to the need of Member States to integrate efforts of various stakeholders in improving the health of people living in urban areas and islands. The objectives of the programme are: (1) to minimize health hazards in urban areas/islands through the integration of health and environmental protection measures in the physical and economic planning process; (2) to enhance the quality of the physical and social environment supportive of health in urban and island settings; (3) to increase public awareness towards healthier behaviour, lifestyle and habits; (4) to improve the provision of health services through developing appropriate health care systems in urban areas and islands; and (5) to upgrade country capabilities and develop policies to improve health in urban areas and islands through better intersectoral coordination and public participation.
****************************************************************************** This document was prepared by an editorial team consisting of Dr Fran Baum, Head, Department of Public Health, Flinders University of South Australia, and Director, South Australian Community Health Research Unit; Ms Gwyn Jolley, Senior Research Officer, South Australian Community Health Research Unit; and Ms Danielle Bament, Research Officer, Department of Public Health, Flinders University of South Australia, in collaboration with the Healthy Settings and Environment Focus of the Building Healthy Communities and Population Division of the WHO Regional Office for the Western Pacific. ******************************************************************************
The documents in the Healthy Cities - Healthy Islands Series are published informally by the Healthy Settings and Environment Focus of the Building Healthy Communities and Population Division of the WHO Regional Office for the Western Pacific. The findings, interpretation and conclusions are entirely those ofthe authors.
Printed and distributed by: World Health Organization Regional Office for the Western Pacific P.O. Box 2932, 1000 Manila Philippines Telephone No.: Facsimile: E-mail: (632) 5288001 (632) 521 1036 postrnaster@wpro.who.int
WHO{wpRO LIBRARY MANILA. PHILIPPINES
Evaluation Team Fran Baum Gwyn Jolley Danielle Bament Head, Department of Public Health, Flinders University Director, South Australian Community Health Research Unit Senior Research Officer, South Australian Community Health Research Unit Research Officer, Department of Public Health, Flinders University
Contents Page
Acknowledgments Acronyms Executive Summary 1 Introduction 1.1 Background and Aims 1.2 Noarlunga Community Action on Drugs 1.2.1 Background 1.2.2 Noarlunga Community Action on Drugs Forum 1.2.3 Noarlunga Community Action on Drugs Activities 1.2.4 Relationship to Healthy Cities
iii
iv v
1 I I I 2 2 3
2 Literature Review of Healthy Cities and Evaluation 2.1 2.2 2.3 2.4 2.5 2.6 Introduction Evaluation that is relevant for Healthy Cities Qualitative and Quantitative Evaluation Establishing Causality Indicators Community Participation
7 7
7 8 9 10 11
3 The Healthy Cities Evaluation Framework 4 Methodology 4.1 Background: Forming an Evaluation Partnership 4.2 Evaluation Stages
13 15 15 IS
5 Implementing the Evaluation of Noarlunga Community Action on Drugs 5.1 Stage One of NCAOD Evaluation 5.1.1 Evaluation Workshop 5.1.2 Workshop Feedback 5.1.3 Results 5.2 Stage Two 5.2.1 Implementation of Project 5.2.2 Short Term Impact Analysis
17 17 17 18 19
21 21 21
6 Assessment of Framework 6.1 6.2 6.3 6.4 6.5 Development of Framework through Stages Evaluation workshop Evaluation Advisory Group Short and intermediate-term measures Building the case for change based on prediction and theory
31 31 32 33 33 34
7 Conclusion and Recommendations 7.1 7.2 7.3 7.4 7.5 7.6
37 37 37 37 38 38 39
Endorse the value of the Healthy Cities Evaluation Framework Rigorous Project Planning Causality in Healthy Cities Evaluation Evaluation Teams Ensuring Evaluation Findings are implemented Further Testing and Development of Healthy Cities Evaluation Framework
References Appendices AppendIX A: Noarlunga Community Action on Drugs Forum Membership Appendix B: Evaluation Workshop Appendix C: Survey of Noarlunga Community Action on Drugs Forum Members
41 43 43 45 61
List of Boxes, Figures and Tables Box 3.1 Box 6.1 Figure 3.1 Figure 5.1 Figure 5.2 Figure 5.3 Figure 6.1 Figure 6.2 Table 5.1 Table 5.2 Key Features of the Healthy Cities Evaluation Framework Stages for a Healthy Cities Evaluation Healthy Cities Evaluation Framework Example of Worksheet for NCAOD Results ofNCAOD Evaluation Plan into HCEF Audit of Attendance at NCAOD Forum Meetings The Revised Healthy Cities Evaluation Framework NCAOD Suggested Theory of Change Model Average Total Attendance at each NCAOD Forum Meeting by Year Total Attendance at NCAOD Forum Meeting for Each Sector Annually 13 32
13 18 20 22 31 34 23 23
ii
Acknowledgments Richard Hicks, Director, Community and Allied Health, Noarlunga Health Services, and Chairperson, Healthy Cities Noarlunga, provided background information and description of project, advised on evaluation process as member of the Evaluation Advisory Group. David Watts, Senior Project Officer, Special Projects - Drug and Alcohol Services Council, provided background information and description ofproject, advised on evaluation process as member of the Evaluation Advisory Group. Gill Faulkner, Manager, Second Story, Division of Child and Youth Health, Noarlunga, advised on evaluation process as member ofthe Evaluation Advisory Group. Members of Healthy Cities Noarlunga and Noarlunga Community Action on Drugs. Members of Healthy Cities Noarlunga, Noarlunga Community Action on Drugs and Child and Youth Health representatives, who attended the Evaluation Workshop. Thanks to Stacey Masters, Research Officer, South Australian Community Health Research Unit, for assistance and advice on data analysis. Thanks to Maryanne Martin, Information Officer, South Australian Community Health Research Unit, for her desktop publishing work.
iii
Acronyms DASC HCEF NCAOD NHS PSG Drug and Alcohol Services Council Healthy Cities Evaluation Framework Noarlunga Community Action on Drugs Noarlunga Health Services Parent Support Group Youth Drug Peer Action Project
YDPAP
iv
Executive Summary This report concerns a project designed to test a framework for the evaluation of Healthy Cities and other complex human service and development projects. The aims of the research were: t t t To test the early stages of the evaluation framework of Healthy Cities projects. To assess the usefulness of the evaluation framework to Healthy Cities participants. To recommend revisions to the framework.
The evaluation was conducted by a small team from the Department of Public Health, Flinders University, Adelaide and the South Australian Community Health Research Unit between November 2000 and April 200 I. Evaluation of an initiative of Healthy Cities Noarlunga was used to operationalise the testing of the Healthy Cities Evaluation Framework (HCEF). This report provides a description of the initiative, a review ofliterature on Healthy Cities and evaluation, and a report on the early stages of the evaluation completed as part of this report. The field-testing of the Healthy Cities Evaluation Framework has enabled lessons to be drawn from the application of the evaluation framework and suggestions for its development to be made in the final two sections of this report. Noarlunga is the centre of the local government area of the City of Onkaparinga situated in Adelaide. South Australia. Healthy Cities Noarlunga began in 1987 as one of the pilot cities to test the European Healthy Cities model in Australia. Noarlunga Community Action on Drugs is community partnership under the auspices of Healthy Cities Noarlunga. The mitiative aims to prevent and reduce harm caused by drug use to the community.
a
Healthy Cities offers a mechanism for improving the health and well being of individuals and communities. However, the evidence base to support this claim is fairly limited and there is growmg recognition of the need to make evaluation an integral part of all Healthy Cities projects. The majority of literature on Healthy Cities argues for evaluations that are process based and include participation by stakeholders. Although evaluations that emphasise predominantly qualitative measures appear the most effective and appropriate, the problem of assigning causality is recognised. As Healthy Cities projects may take many years before a health status outcome can be seen. and the nature of communities is such that there are many influences on health outSIde of the project. it is difficult to assign improvements in a community's health status to the Healthy Cities initiative. The Healthy Cities Evaluation Framework attempts to overcome the causality problem by predicting short-term impacts, intermediate health and wellbeing outcomes. and long-term health and development outcomes. The credibility of the strategies and the causal link is increased if the predicted outcome occurs. The evaluation framework was conceptualised into four stages: Preparing for Evaluation: Describing the Project and Evaluating Short Term Objectives; Evaluating IntermedIate Health and Health Promotion Outcomes: and Individual, Community Health and EnvIronmental Outcomes. In the initial stage of the evaluation, an evaluation workshop was held with project stakeholders. The goals of the workshop were to: 1. Gam a community perspective on Healthy CitJes evaluation. and 2. Develop an evaluation framework for Noarlunga Community Action on Drugs. The evaluation plan and other information arising from the workshop were presented to participants at the following Forum meeting and feedback sought, before the results were finalised
and transferred into the framework for the purpose of undertaking the short-term impact evaluation in Stage Two. Data collection for the evaluation consisted of: audit of attendance at Forum meetings, document analysis, interviews and a postal survey. Overall, the findings show that the Forum has been successful in achieving to some extent all but one short-term impact. The findings have further revealed preliminary information to suggest they have also met some of the predicted intermediate-term impacts/outcomes. We conclude that the framework proved to be a robust means of evaluating Healthy Cities type projects. From the experiences gained in the testing process we have made some changes to its conceptualisation. Figure 6.1 on page 35 shows the revised framework. Breaking the application of the framework in to four stages proved useful as it provides a clear idea of what the evaluation is attempting to achieve at each stage. This process also allows for collaborative planning of the evaluation and for engagement of key players in the process. It provides the means to combine qualitative and quantitative methods and contributes to establishing causal relationships between Healthy Cities interventions and eventual health and environment outcomes. It also encourages participation from the key stakeholders in the project. We recommend that the Healthy Cities Evaluation Framework using the four stages is tested further in a variety of Healthy Cities settings in the Western Pacific Region. The experiences from the various test sites should be collated and general lessons drawn from these.
vi
1 1.1
INTRODUCTION Background and Aims
This report concerns a project designed to test a framework for the evaluation of Healthy Cities and other complex human service and development projects. The aims of the research were: t To test the early stages of the evaluation framework of Healthy Cities projects t To assess the usefulness of the evaluation framework to Healthy Cities participants t To recommend revisions to the framework The evaluation of Healthy Cities initiatives has been discussed since the program was initiated by the World Health Organisation in the mid 1980s. An increasing literature about Healthy Cities evaluation has developed (see details in the literature review in Section 2) and this is slowly leading to an evidence base for the advantages oflong-term development initiatives such as Healthy Cities. This report contributes to both the methodological debate and the evidence base about the operation of Healthy Cities style initiatives. The evaluation was conducted by a small team from the Department of Public Health, Flinders University and the South Australian Community Health Research Unit between November 2000 and April 2001. Both these organisations have maintained links with the Healthy Cities Noarlunga project since the late 1980s and consequently have long established working relationships that assisted the implementation of this project. The team held discussions with Healthy Cities Noarlunga and determined that the most appropriate aspects to evaluate was the Noarlunga Community Action On Drugs (NCAOD) initiative. The coordinating mechanism for the NCAOD has been an intersectoralForum. This report provides details of the application of the initial stages of the Healthy Cities Evaluation Framework (HCEF) to the workings of the Forum and to a Youth Drug Peer Action Project (YDPAP) which has been one of the initiatives of the Forum. A more detailed evaluation of these has attracted funding from the South Australian Department of Human Services and will occur in the second half of 200 1. Evaluation of these initiatives of Healthy Cities Noarlunga was used to operationalise the testing of the Healthy Cities Evaluation Framework. This report provides a description of the initiatives, a review of literature on Healthy Cities and evaluation, and a report on the evaluation completed as part of this report. The field-testing of the Healthy Cities Evaluation Framework has enabled lessons to be drawn from the application of the evaluation framework and suggestions for its development to be made in the final two sections ofthis report.
1.2
NoarJunga Community Action on Drugs: A Healthy Cities Initiative Background
1.2.1
Noarlunga is the centre of the local government area of the City of Onkaparinga situated in Adelaide, South Australia. The region encompasses the outer southern metropolitan suburbs of Adelaide and has a population of approximately 140,000 people. In comparison with the rest of the Adelaide metropolitan area, the area has a younger population, with 75% of residents under the age of 45 years. The city also has a higher than state average unemployment rate and lower household incomes.
Healthy Cities Noarlunga was established in 1987 as one of the Australian pilot cities to test the European Healthy Cities model in Australia. The pilot ended in 1990 with the end of national funding. Healthy Cities Noariunga continued though, and has done so strongly for over a decade, with sponsorship and support from Noarlunga Health Services. Through the collaboration of government and non-government agencies, local government representatives and community members, Healthy Cities Noarlunga addresses health issues pertaining to the City of Onkaparinga. Healthy Cities Noarlunga's approach is based on the frameworks of the World Health Organisation's Ottawa Charter and the strategies of the World Health Organisation's Healthy Cities Project. Healthy Cities Noarlunga aims to stimulate community action about issues that affect health and to develop possible solutions for these. Noariunga Community Action on Drugs is a community partnership under the auspices of Healthy Cities Noarlunga. The initiative aims to prevent and reduce harm caused by drug use to the community within the City of Onkaparinga. The need for an initiative such as the NCAOD became evident after a public forum with community members in August 1997. During the forum, concern was expressed by parents and other members of the community regarding drug related issues in the southern metropolitan area of Adelaide.
1.2.2
Noarlunga Community Action on Drugs Forum
Following this, exploratory meetings were held with local agencies, interested groups and individuals, in late 1997 and early 1998, to determine the local community's interest in implementing a five-year strategy. From these meetings, a Forum was established and has met on a regular basis, to develop strategies to tackle drug related issues in the City of Onkaparinga. Membership of the Forum consists of government and non-government service providers, including Health Services, Family and Youth Services, Police, Education Department, religious groups, local and state members of parliament, and community representatives (for the full list see Appendix A). . Once the Forum had been established, an audit was conducted of programs and services addressing drugs and related issues in existence in the community. As a result of the audit and subsequent discussions, five broad areas in need of action were identified. These were: 1. Support for parents of children with identified drug issues: 2. Support for schools to develop effective drug policy and programs; 3. Intervention and support for an identified group of young women at risk of drug use and other harm; 4. The provision of accurate drug information for the community; 5. Professional development on drug issues for local service providers and workers.
1.2.3
Noarlunga Community Action on Drugs Activities
From these identified needs, NCAOD has been involved with a number of programs and support services: • Parent Support Group - established a peer support program for parents of children with identified drug problems. • Needle and Syringe Program Outreach Trial - run by an outreach worker who provided advice and support to local drug users.
2
t
t
Youth Drug Peer Action Project - to be run by The Second Story Noarlunga (a partner of NCAOD), consulting with young people regarding drug issues important to them. The information gained from this project will inform prevention and harm minimisation programs. The project has received funding from the 'National lllicit Drug Strategy Community Partnerships Initiative' and is expected to begin in early 2001. Community Grant Scheme - This is a grant of AU$5000 provided to the Forum annually by the Drug and Alcohol Services Council. Small grants are then given by the Forum to community initiatives focusing on drugs and related problems in the Onkaparinga area. Projects that have received funding and support from NCAOD to date from this scheme include: Hunting in Packs: a theatre production developed and performed by young people, focussing on drug use and other issues relevant to young people; Reynella Enterprise Skate Comp: a series of skateboard competitions organised by young people and facilitated by a counsellor at Reynella Enterprise Youth Centre, held to provide drug and alcohol free recreation for young people in the area; Wesley Uniting Mission Music Group: a group of recovering drug users who hold music workshops for young people to learn an instrument, and who are forming a band to play at drug and alcohol free entertainment events.
1.2.4
Relationship to Healthy Cities
The underlying principle ofthe Healthy Cities movement is an approach to health as a positive, holistic concept, that focuses on inequalities in health status (Baum, 1998, Hancock, 1993, WHO, 1995). Health as a positive concept is important, as it goes beyond the belief that health is merely the absence of disease, and aims to promote good physical, mental, social and emotional health as a prerequisite for well being. The approach recognises that there are many influences on health, such as cultural, political, social, economic and environmental factors. This holistic notion also highlights the importance of focusing on the health of communities and not just individuals. Reducing health inequalities is also a central concern of Healthy Cities. NCAOD reflects these basic principles underlying a Healthy Cities framework for a number of reasons. Firstly, their approach is not treatment based, but a positive, preventative and health promoting one. It focuses on reducing the harm caused by drugs to the community and not simply individuals. Secondly, the initiative has recognised that drug related problems are unequally distributed in the community. Socially disadvantaged people experience drug problems far more often than non-disadvantaged members; NCAOD is focussing its strategies accordingly. Lastly, NCAOD has recognised that there are external social and cultural influences on drug related behaviour, through support of the theatre production "Hunting in Packs". This project encouraged young people to explore any issues important to them, not only drugs. The Ottawa Charter draws on the principles of health underlying Healthy Cities and offers a strategic approach that informs Healthy Cities frameworks, describing health promotion as a "process of enabling people to take control over and to improve their health" (WHO, 1995). WHO (1995) recognises six key factors informed by the Ottawa Charter that are common to Healthy Cities projects, as a guide to good practice. The following describes these key factors, and NCAOD's relationship to them.
Commitment to Health This refers to a commitment to health as a holistic and positive concept, focusing on the promotion of health and prevention of illness for individuals and communities.
3
NCAOD is committed to health as a holistic and positive concept as described above, and is active in the prevention of illness and promotion of health for groups in society. The 'Needle and Syringe Outreach Trial' is an excellent example of this, preventing illness by providing easy access to clean needles and promoting health by offering advice and support to drug users in the community. Another good example is the 'Parent Support Group', recognising the need for education about drug issues for parents and for social and emotional support to those facing drug related problems within their families.
Political Decision Making Effective Healthy Cities projects are believed to be those that can influence the political decisions of local councils (WHO, 1995). To date, NCAOD has not directly led to the change of policy in local government regarding drug issues. However, NCAOD hopes to have a role in the future in informing school drugs policy, through its partnership with local primary and secondary schools. Also, NCAOD hopes to have some influence in the future on local government policies regarding drugs, through collaboration with the local government representatives who are part of the Forum.
Inter-sectoral Action This refers to the adoption of health related concerns by organisations outside of the health sector, and a change in their activities, enabling them to contribute more to the community's health (WHO, 1995). The composition of the NCAOD Forum is a reflection of the commitment to intersectoral action, with collaboration not only from health providers, but also from the police, education department, local council, family and welfare services, religious institutions and community members. A specific example of NCAOD's commitment to intersectoral action is the plan to work with the Education Department and add a module on drug use to the school curriculum.
Community Participation Community participation is an essential element of any Healthy Cities project (WHO, 1995). It is important that community members have an influence on projects and strategies, to ensure that the issues being addressed are those that the local community identifies as important to their health. The program also enables the community to have decision-making power and influence in the project (Baum, 1998, Bracht and Tsourus, 1998, WHO, 1995). NCAOD attempts to ensure community needs are met and that the community has influence upon the initiative and projects run within it, by having community members as members of the Forum. The number of community members currently on the Forum is relatively small though, and NCAOD hopes to increase this in the future. One planned to achieve this is the YDPAP. This initiative will involve training a group of young people to gather information from other young people in the community, on issues that are important to them, and how they feel these should be addressed by service providers in the area. The information gathered from this project will be used to inform future health promotion and prevention programs on drugs in the area.
Innovation To be innovative, a Healthy Cities project needs to strive to come up with new ideas and ways of implementing them (WHO, 1995). NCAOD can certainly claim to be innovative, with programs such as the theatre production 'Hunting in Packs', which gave young people an innovative way to express themselves and their concerns.
4
Healthy Public Policy Healthy public policy is an outcome of the above five factors coming together to achieve policies that promote good health for the community, in all aspects of their lives, ego home, work and school (WHO, 1995). NCAOD has not yet achieved changes to drug related policy to promote health in the community. This may be due to the fact that the project is at a relatively early stage. Overall therefore, NCAOD can be seen to be operating according to the principles of Healthy Cities.
5
2 2.1
Literature Review of Healthy Cities and Evaluation Introduction
Healthy Cities offers a mechanism for improving the health and well being of individuals and communities. However, the evidence base to support this claim is fairly limited and there is growing recognition of the need to make evaluation an integral part of all Healthy Cities projects (Baum, 2000, Boonekamp et ai, 1999, Burton, 1999, de Leeuw, 1999, Dooris, 1999, Milio, 1990, Rootman et ai, 1997, WHO, 2000). As well as building up an evidence base (Baum, 2000, Dooris, 1999, Rootman et ai, 1997), evaluation in Healthy Cities is deemed important in order to: define and gain a better understanding ofthe principles of Healthy Cities; be accountable to key stakeholders i.e. policy makers, funding bodies, community members (de Leeuw, 1999); build a resource of common knowledge to inform other Healthy Cities projects (WHO, 1999) and to monitor and develop the progress of projects (Baum, 2000, Burton, 1999). The relative lack of data surrounding Healthy Cities evaluation is hardly surprising, as such projects do not lend themselves to traditional scientific based evaluation methods that are designed to measure effects achieved in a relatively short time scale. Healthy Cities initiatives are complex as they are involved with social, political and cultural contexts. As such they do not produce results in the clear linear fashion that is required for purely scientific based, epidemiological studies (Boonekamp, 1999, Burton, 1999, Curtice and McQueen, 1990, Kelly, 1992, Rootman et ai, 1997, WHO, 2000a). Further, Healthy Cities projects can take decades to see the results of their implementation (Baum, 2000).
2.2
Evaluation that is Relevant for Healthy Cities
The majority ofliterature on Healthy Cities argues for evaluations that are process based (Baum, 1998, Baum, 2000, Co stings and Springett, 1997, Curtice and McQueen, 1990, Milio, 1990, Rootman et ai, 1997, Trogan et ai, 1992, Werna and Harpham, 1995, WHO 2000b). Two main reasons are put forward in support of process evaluation. Firstly, there is a strong belief that process evaluation can capture the true nature of Healthy Cities projects (Boonekamp et ai, 1999, Burton, 1999, Costings and Springett, 1997, Curtice and McQueen, 1990, de Leeuw, 1999, Dooris, 1999, Gillies, 1997, Milio, 1990, Rootman et ai, 1997.). Healthy Cities is not solely concerned with what changes take place, but rather why these changes take place, it is interested in the social, cultural and political influences on health. Process evaluation focuses on the why, and allows us to examine the relationships, activities, resources etc, that are likely to produce certain outcomes throughout a project. This is favoured over evaluation based solely on outcome measures, that examines the end result and cannot determine all of the factors that have influenced a community's health during the implementation ofa long-term project. Secondly, process evaluation is important because of its ability to inform a project while it is operating (Baum, 2000, Costings and Springett, 1997). As Healthy Cities projects are complex and long term, it would be easy for a project to lose sight of what it set out to achieve and take a "wrong tum". Process evaluation is proposed as a mechanism to ensure the project is in line to achieve its goals, and acting in accordance with Healthy Cities and Ottawa Charter principles. Costings and Springett (1995) provides us with an example of the benefits of process evaluation. This paper documents an evaluation of a draft 'City Health Plan' in Liverpool, UK, where key informants from a wide range of sectors, organisations and the community were interviewed about the draft City Health Plan regarding its relevance and usefulness for the community. The
7
evaluation found that there were a number of issues that needed to be addressed prior to the implementation of the plan, in order for the plan to achieve its goals and promote better health and well being in the City. Burton (1999) on the other hand, gives an example of two healthy city projects in Bangladesh, where evaluations were conducted four and five years into the projects. The evaluations highlighted problems of power relations between various stakeholders, resulting in the needs of community members in one city in particular not being adequately met. Burton notes the usefulness of looking back and examining such relationships after the projects have been in place for several years. However, she also recognises the importance of earlier and continuous evaluations in Healthy Cities projects if such funds were available. Had process evaluation been an integral part of the projects, and the relationships between the stakeholders examined from the beginning, perhaps problems could have been highlighted in the initial stages and the needs of the community more closely met.
2.3
Qualitative and Quantitative Evaluation
Most of the literature on evaluation in Healthy Cities argues for a strong focus on qualitative evaluation. The use of traditional epidemiological methods is generally rejected for reasons including their inappropriateness for Healthy Cities projects and the inflexible nature of such methods (Baum, 1998, Costings and Springett, 1995, Curtice and McQueen, 1990, Milio, 1990, Rootman et ai, 1997). Firstly, with regards to appropriate evaluation, many authors argue that it would rarely be possible in Healthy Cities research to use traditional experimental quantitative methods such as Random Control Trials (RCTs) (Baum, 1998, Baum, 2000, Curtice and McQueen 1990, Gillies, 1997.). Experimental methods such as RCTs are believed inappropriate as they require clear linear results, achieved from the use of an experimental and control group. Unlike research in a laboratory setting, communities have many factors affecting health that cannot be controlled for (e.g. varying employment levels) and a linear relationship between program implementation and result is not possible. A WHO Working Group on Health Promotion Evaluation (1998) comments that: "The use of randomised control trials to evaluate health promotion initiatives is, in most cases, inappropriate, misleading and unnecessarily expensive. " Further, it is almost impossible to find two communities that are identical except for the presence or absence of a health promotion program and even if this was found, there is no guarantees the control community will not follow suit and begin a similar program themselves (Baum, 2000, Rootrnan et ai, 1997». An example of this can be seen in a program in Wales called "Heart Beat Wales" where the control community began a "Healthy Heart" program of its own (Baum, 2000). Secondly, purely quantitative tools are out of step with the nature of Healthy Cities projects (Gillies, 1997, Kelly, 1992). Healthy Cities projects are interested in people's experiences and the social, cultural and political influences upon why certain changes occur, and not simply if the program is successful or not. Quantitative methods do not lend themselves to this kind of analysis, only giving the end result. As Gillies (1997) states: "(RCTs) simply are not sufficiently sophisticated to deal with the complexity and diversity of the process and outcome of health promotion at the community level" (p.21).
8
Qualitative research, on the other hand, allows us to determine people's subjective experiences of the Healthy Cities initiative which many authors argue is of most importance (Baum, 1998, Baum, 2000, Co stings and Springett, 1995, Costings and Springett, 1997, Curtice and McQueen, 1990, Smithies and Adams, 1993). Qualitative data informs us what the project means to the key stakeholders, (funding bodies, community members, politicians, etc), how they feel about it and why. Many authors argue that information such as this is important in Healthy Cities, as it not only determines whether the project is successful or not, but why it is or is not successful for various groups in the community (Costings and Spririgett, 1997, Costings and Springett, 1995, Curtice and McQueen 1990). As Costings and Springett (1997) summarise:
"It is important to consider the values, aspirations and motivations of the people involved, which cannot be quantified" (p.348). Lastly, Curtice and McQueen (1990) argue that purely scientific based research is unsuitable for Healthy Cities evaluations due to its inflexible nature. Healthy Cities projects are concerned with being responsive to the changing needs of the key stakeholders. Unlike qualitative studies, Curtice and McQueen (1990) argue that quantitative based research is unable to respond to such changes. Although most authors discussing evaluation methods for Healthy Cities programs do favour qualitative over quantitative methods, some literature recommends a combination of the two, along with interpretive techniques, encouraging the use of the most appropriate tools for the evaluation question (Baum, 1998, Baum, 2000, Costings and Springett, 1997, Gillies, 1997, Milio, 1990, Rada, 1999, Rootman et ai, 1997). Baum and Cooke (1992) provide such an example in an evaluation of Healthy Cities Noarlunga, using primarily interview and survey techniques to document people's subjective experiences of the various programs, and media analysis as an interpretive measure. Quantitative methods were also employed to asses how many people responded to the surveys for one particular project, and to determine the percentage of community members who were aware of Noarlunga Healthy Cities compared to the previous year. Trogan et al (1992) offer a similar methodological approach in a research proposal consisting of document analysis, participant observation, open ended and semi structured interviews as well as quantitative analysis in the form of a street survey.
2.4
Establishing Causality
Although evaluations that emphasise predominantly qualitative measures appear the most effective and appropriate mechanism for evaluating Healthy Cities initiatives, the problem of assigning causality is recognised (Baum, 1998, Baum, 2000, Co stings and Springett, 1997). As Healthy Cities projects take many years before an outcome can be seen, and the nature of communities is such that there are many influences on health outside of the project, it is difficult to assign improvements in a community's health status to the Healthy Cities initiative. Costings and Springett (1997), recognise this issue and offer the use of triangulation as a possible solution to increase the validity ofthe results. They also propose ongoing feedback to all stakeholders during the evaluation as a mechanism to increase the validity ofthe results and help to assign causality (Co stings and Springett, 1997). They advise that such feedback would allow the findings of the evaluators to be verified or rejected with the stakeholders, as to whether they believe certain outcomes are a direct result of the Healthy Cities initiative or due to other external factors in the community.
9
The issue of attributing causality in complex community-based initiatives has recently received attention in a number of contexts in addition to Healthy Cities. In the USA, Connell and Kubisch (1998) argue for the use of the theory of change approach to evaluation. They defme this as a "systematic and cumulative study of the links between activities, outcomes and contexts of the initiative". It involves a process of making explicit links between the original problem or context with which the initiative began and the activities planned to address the problem, and the intermediate and longer-term outcomes planned. In the UK, Judge and Bauld (2001, p. 25) have suggested this approach is an appropriate framework with which to evaluate the Health Action Zones, which share much in common with the Healthy Cities approach. They comment that if a theory of change is articulated early in the life of an initiative and stakeholders can agree to it, that this "helps to reduce problems associated with causal attribution of impact".
2.5
Indicators
The importance of indicators in Healthy Cities evaluations to determine the success of a project is recognised and not questioned in the literature. What it up for debate is whether indicators should be designed and used solely for the use of each Healthy City project (local indicators), or whether there should be a set of relatively generic indicators (global indicators) for use in many projects (Boonekamp et aI, 1999, Costings and Springett, 1997, Doyle et aI, 1999, Werna and Harpham, 1995). In WHO's (1999) guidelines for evaluation in Healthy Cities "improvement of practice for future use and reference" is listed as one reason for the importance of evaluation (p.23). Werna and Harpham (1995) similarly note that: "WHO publications recommend indicators which should be used in HCP (Healthy City Projects) evaluations throughout the world" (p.633). If results from Healthy Cities projects are to be compared with others and used as a source of reference, the indicators that inform us whether a project has been successful or not need to be similar between the communities. Some authors suggest though, that since an important principle of Healthy Cities is the recognition that each community is unique and that improvements in health should be identified by the community for the community, it is not possible to come up with a set of indicators that can be transferred to other projects for comparison (Costings and Springett, 1997, Dooris, 1999, Doyle et aI, 1999, Hayes and Williams, 1990, Werna and Harpham, 1995). As Hayes and Williams (1990) state: "If one concedes that circumstances giving rise to conditions in one community are not the same as those operating in another, then the validity of employing indicators that would allow for comparison with other communities may be questioned on the grounds that neither the baselines, nor the processes are truly comparable" (p.165). Dooris (1999) similarly notes the importance of indicators that hold meaning for each community, arguing that they should be not be imposed, but identified by community members themselves; "As well as being valid and reliable, it is important that indicators are meaningful to people - that they are demystified and developed not by epidemiologists but by the communities whose health and sustainability is being measured" (p.372). Doyle et al (1999) also discuss the difficulty involved in obtaining a set of global indicators. They conducted a survey of forty-seven countries involved in the WHO European Healthy Cities project and sent each country a questionnaire regarding various aspects of health in their country, as an attempt to compare health levels. However, problems arose with varying cultural
10
interpretations and political and religious structures that meant very diverse interpretations of each question were apparent in the answers. Doyle et al (1999) give an example in the reporting of suicide rates between countries in which Hungary reported significantly lower levels of suicide than Poland. They note: "".in reality it is unlikely to be true, but has more to do with religiOUS and cultural values and taboos regarding suicide and reporting it" (p.295). It seems there is some confusion among those involved in evaluating Healthy Cities projects, regarding how indicators can be created and used for anything other than the local level. However, as WHO advise (1999), the use of evaluation material to compare health levels between communities and as a source of future reference and guide for Healthy Cities projects is invaluable and important for the future evolution of Healthy Cities globally.
2.6
Community Participation
Community participation in the evaluation of Healthy Cities is an idea discussed and favoured by many authors writing about Healthy Cities research (Baum, 1998, Curtice and McQueen, 1990, Costings and Springett, 1997, de Leeuw, 1999, Kennedy, 1995, Milio, 1990, Smithies and Adams, 1993, Whitehead, 1993). It is considered an important element of such evaluations for a number of reasons. Firstly, it is in line with Ottawa Charter and Healthy Cities principles, breaking down power relations between professionals and community members, allowing community members to be empowered and "own" the results (Costings and Springett, 1997, Kennedy, 1995, Smithies and Adams, 1993, Whitehead, 1993). Costings and Springett (1997) believe that by "owning" and understanding the results through participation, people are more likely to believe in them. Milio (1990) puts forward a similar argument, stating that the research process must be collaborative, " ... so that findings will be deemed usefol and credible to potential users" (p.293). Secondly, community participation in research is believed to lead to research methods that are more appropriate for the community under evaluation (Baum, 1998, Whitehead, 1993). Thirdly, it is thought to provide a more accurate interpretation of the data, as it is interpreted by those who are most affected by the project implementation (Curtice and McQueen, 1990, Kennedy, 1995, Smithies and Adams, 1993). The authors that promote community participation in evaluations of Healthy Cities also highlight problems and difficulties associated with doing so. Milio (1990) for example, warns against the risk to scientific integrity of having the "subjects" of the evaluation involved in the evaluation process. Further, it is noted that community members are often unwilling to participate, due to lack of time, confidence or interest, or are only willing to participate on issues that specifically concern them (Baum, 2000, Costings and Springett, 1997, Smithies and Adams, 1993). Kennedy (1995) conducted before and after questionnaires with a group of community members participating in an evaluation and found that these problems of lack of participation subsided after community members had attempted joining in on evaluations. They recorded that people felt much more confident about being involved in research and placed significantly higher importance on being involved once they had had some level of experience. Ouellet et al (1992) provide another example of a participatory Healthy Cities evaluation, conducting preliminary studies on two Healthy Cities projects in the Montreal area in Canada. They note that involving community members in evaluation meant that unusual demands were placed on the researchers, such as the need to have special skills in communication, and the ability to motivate and negotiate (Ouellet et ai, 1992). Contrary to their usual practice, researchers were
II
forced to simplify concepts and use straight-forward language for the benefit of the community members. Thus, theoretically, community participation in Healthy Cities evaluations improves the quality of research and provides many benefits for the community members themselves. However, there are numerous challenges in taking a participatory approach for both researchers and community members.
12
3
The Healthy Cities Evaluation Framework
The Healthy Cities Evaluation Framework (HCEF) (shown in Figure 3.1.) considers all of the issues raised in the literature review and promotes evaluation that is process based, utilising both qualitative and quantitative methodologies, and incorporates community participation where possible. In regards to issues of causality, the framework aims to overcome this problem by predicting short term impacts, intermediate health and well being outcomes, and health and development outcomes. The outcomes from the strategies are predicted in the initial stages of the project. The credibility of the strategies and the causal link is increased if the predicted outcome o·ccurs. This approach is similar to the theory of change approach described in section 2.4. above. Further evidence of causality can be obtained by reviewing and assessing the evaluations of similar programs and relevant literature which link the intervention with predicted outcomes.
Figure 3.1 Healthy Cities Evaluation Framework
Project Objectives
Short-term Predictions impacts implementation
Intermediate
Health and Development outcomes
-+
health outcomes
Box 3.1 Key Features of the Healthy Cities Evaluation Framework t t t t t t Promotes process based evaluation Enhances causal links through the predictions of Short Term Impacts, Intermediate Health and Well Being Outcomes and Health and Development Outcomes Supports claims for causal links with past research and literature Promotes the use of the most appropriate qualitative and quantitative methods Encourages community participation in evaluation Identifies environmental influences on health and wellbeing that are external to the project
~--~~----------------------------------------------------------~~
I
A further aspect of the framework is the examination of environmental factors. By looking at factors occurring in the environment outside of the initiative, we can gain a better picture of the impact of the strategies. For example, ifit was found that the health of a group in the community was actually reduced following the implementation of health promoting strategies, and employment has greatly declined during the same period, the framework suggests it would be unreasonable to expect the strategies to improve health, given the environmental conditions. The Healthy Cities Evaluation Framework therefore builds upon previous work in Healthy Cities evaluation and proposes a tool that is appropriate for the evaluation of such projects. This framework also offers solutions to problems of causality that have been identified in previous evaluations of Healthy Cities. Key features of the framework are shown in Box 3.1.
13
4 4.1
Methodology Background: Forming an Evaluation Partnership
The HCEF was used to guide the evaluation ofNCAOD. A four stage model was implemented that incorporates short, intermediate and long term impacts and outcomes. The stages are intended to promote evaluation of a cyclical nature and are not a rigid linear process from stage 1 to 4. For example, stages I to 3 of the evaluation process may have been conducted on a project, when it becomes obvious that the predicted outcomes will not be achieved, thus changes would need to be implemented and evaluated from stage 1 again. Prior to the implementation of the stages, steps were taken to form a partnership between the evaluators and project participants. Firstly, two evaluators from the South Australian Community Health Research Unit went to a NCAOD Forum meeting to negotiate the evaluation of the Healthy Cities initiative. The evaluators described the aim of the evaluation and how it would involve and benefit the Forum, who then agreed to take part. Following the agreement by Forum members for NCAOD to undergo evaluation, an Evaluation Advisory Group was set up, consisting of three evaluators and key members of the Forum. The role of the group is to monitor the evaluation process. The group meets regularly in order for the evaluators to keep project participants informed on steps in the evaluation process and findings. Project participants provide feedback, raise any concerns they have about the evaluation process and problem solve issues along with the evaluators.
4.2
Evaluation Stages
Stage 1: Preparing for Evaluation In the first stage, the evaluation method was designed involving key players of the project. This process involved setting clear, evaluable, short, intermediate and long term objectives, and strategies to achieve these objectives. In this stage, indicators were also established, along with the identification of appropriate measurement tools and possible influences on health outside of the project.
Stage 2: Describing the Project and Evaluating Short Term Objectives The second stage involved describing how the Healthy Cities project has been implemented according to Healthy Cities Principles and Ottawa Charter guidelines, and whether it is moving in the right direction to improve health and well being, according to the criteria. Short term impacts are also measured in this stage. These include information such as the number and type of agencies involved, number of community people involved, representation ofthe community and funding that has been granted.
Stage 3: Evaluating Intermediate Health and Health Promotion Outcomes Stage three is concerned with measuring intermediate impacts that can be linked to the desired health and environmental outcomes. Links are made with the use of past research and experience. For the purpose of this project, preliminary intermediate-term impacts have been measured and will be analysed more extensively in the next phase of the evaluation. Health promotion outcomes are those outcomes that it is predicted will ultimately be linked to a health or environmental outcome. An example is that increased knowledge of potential harm from drug use would eventually led to a reduction in drug use, or the introduction of drug free entertainment options might led to a reduction in drug use.
15
Stage 4: Individual, Community Health and Environmental Outcomes This stage measures individual, community and environmental health and development outcomes. For example, community members report higher levels of good health, injury rates have decreased, or air quality has improved. Outcomes of this nature, are not expected to be achieved in less than three years and often take five years or more.
16
5
Implementing the Evaluation of Noarlunga Community Action on Drugs Stage One of NCAOD Evaluation Evaluation Workshop
5.1 5.1.1
In the initial stage of the evaluation, an evaluation workshop was held with fifteen participants, including stakeholders from the YDPAP and NCAOD and community members from Healthy Cities Noarlunga.
The goals of the workshop were as follows: 1. To gain a community perspective on Healthy Cities Evaluation, and 2. To develop evaluation frameworks for the Noarlunga Community Action on Drugs and Youth Drug Peer Action Project by: • confirming objectives and strategies; • categorising objectives into short, intermediate and long term outcomes, with appropriate time scales; • determining indicators and ways to measure the achievement of the objectives based on the predicted (hoped for) effects. A brief introduction to evaluation in Healthy Cities and an explanation of the principles behind the HCEF was given to participants, followed by a short explanation of the terminology used in evaluation (e.g. objective, strategies, indicators etc.) and the importance of evaluation for Healthy Cities projects. Next participants were broken up into three groups to undertake exercises designed to meet the goals of the workshop.
NCAOD and YDPAP Groups Forum members were separated into the two groups, with representatives from Child and Youth Health joining the YDPAP group. With assistance from evaluators, participants were required to examine objectives and strategies that had been gathered previously from project documents for each initiative, suggest changes to these and/or come up with new ones. Along with this they were asked to identify outcome statements for each objective and categorise them into a time-scale of short, intermediate or long term outcomes. The groups were then asked to consider each objective and determine external/environmental factors that could enhance or inhibit the potential for achieving the outcome, identify indicators of achievement of the objective and ways of measuring this. To complete the activities, participants were given work sheets, an example of which is shown in Figure 5.1 (Fully completed worksheets can be found in Appendix B).
Healthy Cities Noarlunga Group This group was made up of community members from Healthy Cities Noarlunga along with the Director of Allied and Community Health Services, Noarlunga Health Services, who has been intensively involved with Health Cities Noarlunga and NCAOD from their beginnings. The participants were asked to reflect on Healthy Cities Noarlunga and record their views on the important pre-conditions for success of a Healthy Cities project. They were then asked to reflect on what they consider appropriate and effective evaluation techniques that encourage the involvement of community members. (Results from this group can be found in Appendix B)
17
Figure 5.1 Example Worksheet for NCAOD ~art
)Jecti ves and Sho rt, Intermed· 0 ne: Identfi I IcatlOn 0 fOb· late, Long-Term o ut comes Strategy Foster an intersectoral approach by establishing an ongoing community forum Outcome statement Communication in the South between community members, non-government and government organisations regarding drug related issues is increased leading to effective strategies to address local drug issues. Time-scale* Short - Long
Objective To increase communication and collaboration in the community to address local drug issues
Part Two· Identification of Environmental Factors , Indicators and Measurement Environmental Factors Indicators Measurement Support from the Drug and Attendance and representation Audit of Forum meetings, Alcohol Services Council and IS ongomg numbers and diversity ofthose Healthy Cities Noarlunga Funding attracted to drug and who attend Funding alcohol programs Document analysis of Work demands outside of the Increased networking and partnerships that have been Forum on Forum members. communication in the formed from the Forum community surrounding drug Survey of members motivation related issues for and perception of the Forum
*Key: T" Ime ScaIe Short Term Intermediate Long Term I month - I year 1-2 years 2 years +
5.1.2
Workshop Feedback
At the end of the workshop, feedback sheets were given to all participants to complete. Fourteen out of fifteen participants returned them (n=14). They were asked how useful the workshop was in helping them understand more about evaluation. On a scale of I (not at all useful) to 5 (very useful), seven participants recorded 5, four recorded 4, two recorded 3 and one recorded 2. Those who found it less useful were likely to already have a good understanding of evaluation. Participants were also asked how useful the workshop was in helping them evaluate their group or project. Thirteen participants responded to this question, with six people recording a score of 5 and six people 4. These results indicate that the majority of participants felt the workshop was useful to very useful in its ability to help them gain a better understanding of evaluation and to assist in the evaluation of their projects. When asked what was the most useful part of the workshop, participants noted the working in groups with other Forum members, the inclusion of community members and the discussions and clarifications surrounding objectives, strategies, outcomes, measurement etc.
18
For example:
"Interaction and bouncing of ideas - group discussion i.e. development of ideas and thoughts .. "Opening up of ideas and conversations which were constructive and objective" "Understanding the evaluation approach beingfollowed and discussion/clarification of objectives". When asked if anything could have been done differently or included in the workshop, only a small number of participants responded, with suggestions such as involving young people in the workshop and making more time so that the exercise could be completed for each project. Full results from the feedback sheet can be found in Appendix B.
5.1.3
Results
The evaluation plan and other information arising from the workshop were presented to participants at the following NCAOD Forum meeting and feedback sought, before the results were fmalised and transferred into the HCEF (as shown in Figure 5.2) for the purpose of the short-term impact evaluation in Stage Two of the evaluation process.
19
Figure 5.2. Results of NCAOD Evaluation Plan into HCEF Pro.ject Ob.jectives Increase communication and collaboration in the community to address local drug issues Short-term Impacts Intersectoral community forum established, and attendance and representation from various sectors is ongoing Resources for prevention and treatment programs are identified and attracted Intermediate-term health and well bein2 outcomes Increased networking and communication between government and non-government agencies and community members in the south concerned with drug related issues Funding and resources utilised for more effective prevention and treatment programs More people who use drugs are accessing the Needle and Syringe Exchange Program Health & Development Outcomes
, Reduction in level of harm caused by drugs to the community
o
'"
Improve intervention and support strategies for people at risk of harm from drugs to prevent further harm
Peer Worker established and sustained at Clients of Needle and Syringe Exchange program are the Needle and Syringe Exchange -Program accessing more information about treatment and support services Drug free recreation for young people created and promoted Young people are using drug free recreation Community members report positive attitudes towards drug free recreation Decrease in individual drug related health problems
Increase awareness and knowledge of drug issues and available support services for community members Increase effective information and support services in the community for family and friends of people with drug related problems
Strategies to gain funding for media campaign to advertise local information and support services in place Advisory committee for media campaign set up Strategies in place for Parent Help Line to act as a service capable of offering support and information to parents of young people with drug related problems Establishment of ongoing Parent Support Group
Media campaign implemented
Increased use of Parent Help Line by parents of children with drug related problems
Community members are better able) to resolve drug related issues
Increase in number of parents who are seeking information and support
5.2 5.2.1
Stage Two Implementation of Project
An important part of this stage is detennining the extent to which the initiative is implemented in line with Healthy Cities principles. Section 1 of this report describes NCAOD in terms of its compatibility to Healthy Cities principles and Ottawa Charter guidelines.
5~2.2
Short Term Impact Analysis
This section reports on the data collection, analysis and discussion of a short-term impact analysis. In the course of the data collection, some preliminary findings on intermediate-term impacts also became apparent and are reported here.
Data Collection Audit of Attendance at NCAOD Forum Meetings An audit was conducted on the attendance of members to NCAOD Forum meetings, which are held every six weeks. The audit covered the period from late 1997 when the Forum first convened, to the end of 2000. The audit collected data on the number of people attending each meeting and the sector of the community each person represented. This information was taken from Forum meeting minutes which were produced after each meeting. Document Analysis A document analysis was conducted on all documentation relating to NCAOD from the beginning in 1997 to February 2001. Documents analysed included minutes, reports, letters and funding applications. Face to Face Interviews In-depth face to face interviews were conducted with three key members of NCAOD. Two were concerning the Parent Support Group, and involved a facilitator of the group and a parent who had been through the Parent Support Group program and has remained a member and taken on a role as co-facilitator. The third interview was concerned with fmding out background information ofthe Forum, when it was started, criteria for membership, projects it has been involved in etc. Survey A questionnaire was sent to fifty-six forum members (n=56), consisting of open and close ended questions. The questions were based around objectives and short-term impacts identified in the evaluation workshop. Seventeen people who were sent surveys were unable to respond to the survey for various reasons including being away on long term leave and having left their occupational position as stated on the membership list. Of the thirty-eight people who were available to reply, 27 surveys were returned, giving a response rate of71 %. (A copy of the survey is shown in Appendix C).
21
Findings Objective: Increase communication and collaboration in the community to address local drug issues. Short-Term Impact 1: Intersectoral community forum established, and attendance and representation from various sectors is ongoing. An inter-sectoral community forum has been established since late 1997 and has continued to meet and discuss drug related issues in the community. Meetings are held every six weeks and the membership consists of a wide range of sectors with representation from DASC, Noarlunga Health Services, Child and Youth Health, Police, Education Department, government and nongovernment welfare and community services, local and state government members and community members (see Appendix A for detailed list). The room the meetings are held in is usually full, with twenty or more people attending. Figure 5.3 (detailing results of the audit) shows that a forum has been established and Table 5.1 that attendance is ongoing. In fact Table 5.1 indicates that attendance rates have increased steadily each year from 1998 to 2000. Survey results similarly reflect the ongoing nature of the Forum, as 30% of those surveyed have been members for over three years.
Figure 5.3 Audit of Attendence at NCAOD Forum Meetings: Percentage of Total Attendences by Sector Over 3 Years COmmunity Members 5%
local and State Government 7%
OASC
Welfare & Community Services Gov 8%
Welfare & Community Services Non Gov 10%
Noarlunga Health Services Child and Youth Health 15%
19%
In terms of the representation of the forum being intersectoral and attendance and representation from the various sectors ongoing, results are again positive. Figure 5.3 shows representation from a wide range of sectors and Table 5.2 shows that attendance by each sector although in some cases has varied, has been ongoing. Similarly survey results reveal respondents believe the community of the Onkaparinga area is well represented, giving a median response of 4 on a scale of 1 (poorly represented) to 5 (well represented).
22
However, concerns were raised in the survey and are apparent in the audit, about the possible under-representation of some groups in the community. Figure 5.3 clearly demonstrates that, in terms of numbers, the traditional service providers dominate Forum meetings, with DASC, Noarlunga Health Services and Child and Youth Health, Police and Education Department recording the highest proportions of representation over the three years that the Forum has been operating. Representation from local and state government members and community members is comparatively much smaller, with community members for example only making up 5% of the total attendance rate.
Table 5.1 Average Total Attendance at Each NCAOD Forum Meeting by Year 1998 1999 2000 15.3 16.3 18.1
Table 5.2 Total Attendance at NCAOD Forum Meetings for Each Sector Annually DASC NHS & Child & Youth Health Education Police Welfare and Community Services Non Government Welfare and Community Services Government Local and State Government Community Members
1998 21 14 18 20 0 12 4 1
1999 18 13 16 25 13 5 12
2000 21 23 18 17 20 11 9 4
12
Concerns for under representation were expressed in the survey in response to the question "Is there any section of the community not represented by the Forum that you believe should be?" Twenty-two out of thirty-eight respondents (57%) identified sections of the community not represented in the forum that they believed should be. Of these twenty-two, seventeen (62%) identified community members as a group that needed greater representation on the Forum, with particular emphasis on young people and those who use drugs. Examples of responses include:
'Young people affected by the drug issue' 'Young at risk community members - presenting issues in person' 'Possible representation from an ex user would be good' 'Young people, perhaps invite representatives from student councils in local schools' 'More community members - non-professionals. More users' Some respondents also identified Aboriginal people and people from non-English speaking backgrounds as in need of greater representation.
'Aboriginal community members. People with drug dependency. Local people in the community, e.g. from community centres'
23
'Aboriginal, Non English Speaking Background' Along with indicating community members were under represented on the Forum, survey respondents also reported that community members have less influence over the decision making processes. On a scale of 1 (no influence) to 5 (high influence), respondents recorded a median of 4.5 reflecting they felt service providers had a high influence over decision making processes of the Forum. This is in comparison to a perceived moderate influence for community members, with a median score of 3.
Written comments in response to the questions on the level of influence community members and service providers have in the forum were mixed. Some felt that the decision making power was equally shared. 'It is a democratic decision of all members .. 'Everybody has an equal opportunity to make comment and vote
if necessary'
'Community members have a strong and equal role along with service agency reps. No one group dominates proceedings and there does not seem to be any hidden agendas ..
Others reported in support of the quantitative results, that community members did not have as much influence as service providers. "Community members are far out numbered by services - therefore may feel intimidated .. "Input and participation valued but not sure in terms of influence re final decision making. The top heavy nature of the committee (service providers) potentially might discourage taking on power by community".
Short-Term Impact 2: Resources for prevention and treatment programs are identified and attracted Results from the survey and document analysis indicate that the Forum has been successful in attracting resources for prevention and treatment programs. Survey respondents recorded a median response of 4 on a scale of I (not at all effective) to 5 (very effective), in response to the question "How effective has the Forum been in attracting resources for prevention and treatment programs?" A document analysis similarly revealed that the Forum has been active in and successful at identifying and attracting resources for prevention and treatment programs. The Forum identified a possible resource in grants being offered through the federally funded 'National Illicit Drug Strategy Community Partnerships Initiative' and was successful in gaining a grant for eighteen months to support the Youth Drug Peer Action Project discussed earlier. The Forum has also been successful in attracting AU$5000 annually from DASC for what the Forum have termed the 'Community Grant Scheme'. This is money given to the Forum to allocate to initiatives in the community that are addressing drug related issues, that the Forum members agree are likely to benefit the community. To date money from the 'Community Grant Scheme' has been helped fund: Reynella Enterprise Skateboard Comp; Hunting in Packs; Wesley Music Group (for further details refer to Section 1 of this report); and the cost of a community member from the Parent Support Group to go to a national conference on drug related issues.
24
Preliminary results on the intermediate-term impacts in relation to this objective, are also revealed through survey findings.
Intermediate-Term Impact 1: Increased networking and communication between government and non-government agencies and community members in the south concerned with drug related issues. Respondents to the survey reported that the Forum has led to an increase in communication and collaboration in the community concerning drug related issues. On a scale of 1 (not at all effective) to 5 (very effective) respondents recorded a median of 4 for both communication and collaboration in terms of the effectiveness of the Forum to increase these in regards to drug related issues in the community.
Intermediate-Term Impact 2: Funding and resources utilised for more effective prevention and treatment programs. Survey respondents reported that prevention and treatment programs supported by the Forum have been moderately effective to very effective. On a scale of 1 (not at all effective) to 5 (very effective) respondents were asked to rate the effectiveness of various strategies. The median results were as follows: Parent Support Group: Hunting in Packs: Of Crime and Substance: Community Grant Scheme: 4 5 3 4
It should be noted that in this survey respondents were only asked to rate the effectiveness of individual strategies supported by the Forum, and were not required to comment on the Forum's overall ability to utilise the funding for more effective prevention and treatment programs in the community. Such data would need to be obtained for a detailed intermediate-term analysis.
Objective: Improve intervention strategies for people at risk of harm from drug use to prevent further harm. Short-Term Impact 1: Peer worker established and sustained at the Needle and Syringe Exchange Program Documentation of the Forum suggests that a Peer Worker was placed at the Needle and Syringe Exchange Program for a trial period in 1998. However, upon seeking to undertake further analysis for the purpose of this project, confusion about the role of the person who was to act as a Peer Worker became evident. Some sources advised that there was never formally a person in this role, but that the person was only required to conduct an audit of needle and syringe use and may have taken on other duties of a peer support nature on their own. Other sources state that in fact a person was employed as a Peer Worker to offer advice and support for injecting drug users and that this was effective. Attempts have been made to locate a job description for the position, but have to date been unsuccessful. In terms of a Peer Worker being sustained at the Needle and Syringe Exchange Program, it can be confirmed that this has not occurred, and whether the position was taken on formally or not, the person employed was only contracted for a few months in 1998. Minutes of a Forum meeting held
25
recently though, revealed that money has been obtained to establish a Peer Worker at the Needle and Syringe Exchange Program on an ongoing basis. Further analysis would need to be conducted at a later date to determine whether this has eventuated.
Short-Term Impact 2: Drug Free Recreation for young people created and promoted The document analysis revealed that a number of drug free recreation events for young people have been created and promoted with support by the Forum. These include: Wesley Uniting Mission Music Group: A musical group made up of recovering drug users trying to stay "straight", who have convened the group as a form of recreation that is drug and alcohol free. The group intend to play at drug and alcohol free events for young people and develop music workshops to contribute to youth activities in the South. Reynella Enterprise Skate Comp: Three drug and alcohol free skateboard competitions were held in July last year. The events were organised by local young people for local young people and supervised by a worker from Reynella Enterprise Youth Centre. Blue Light Discos: Blue Light Discos are drug and alcohol free events for young people and have been run throughout the state off and on for many years. The discos were re-started in the Onkaparinga area with support of NCAOD in late 1998 and have continued since.
Intermediate-Term Impact: Young people are using drug free recreation The document analysis has revealed some preliminary intermediate-term impacts in regards to drug free recreation. Young people do seem to be using drug free recreation. In regards to the Blue Light Discos, in late 1998, six events were held with 800 young people in total, with 300 - 400 attendees required for the events to be viable. After one year, that attendance had nearly doubled to nearly 400 people attending each event. Presently the events are still very popular, so much so that the Police, who are the organisers of the event, are selling tickets before hand and are often selling out.
Also, the Reynella Enterprise Skateboard Comp revealed good results with written reports confirming the events were very successful, attracting 300 young people in total and remained drug and alcohol and trouble free. The organisers plan to run the competition again this year.
Objective: Increase awareness of drug issues and available support services for community members. Short-Term Impact: Strategies to gain funding for media campaign to advertise local information and support services in place. Short-Term Impact: Advisory Committee for media campaign set up. The document analysis revealed that both of these short-term impacts did occur in mid - 1998. During this time a Media Advisory Committee was formed and a AU$2200 grant given to them by DASC to undertake a Media Campaign. The aim of the campaign was to increase access for the community to drug information and counselling support services. It is again possible to determine preliminary results for an intermediate-term impact, for the above objective.
26
Intermediate-Term Impact: Media Campaign Implemented The committee met regularly for a period of four months and devised plans to implement the Media Campaign. The committee was in the process of producing pamphlets and advertisements for the local paper, informing the community on services available for drug related issues, along with organising a series of 'talk back' radio sessions on the local station between key NCAOD members and the public. Besides details of these plans, the document analysis did not find any information regarding the implementation ofthe campaign. Enquires with a key NCAOD member revealed that the campaign was not actually implemented due to what was referred to as "bureaucratical issues". The plans were dropped and the Advisory Committee disbanded.
Objective: Increase effective information and support services in the community for family and friends ofpeople with drug related problems Short-Term Impact: Strategies in place for Parent Help Line to act as a service capable of offering support and information to parents of young people with drug related problems. The document analysis revealed no evidence of the Forum putting in place any strategies for the Parent Help Line (which is already a service offered state-wide by Child and Youth Health) to take on the role of offering support and information to parents of young people with drug related problems.
Short-Term Impact: Establishment of ongoing Parent Support Group Interviews with members of the Parent Support Group (PSG) revealed that a such a group has been run "on and off' for a number years in Noarlunga. The most recent group began with support of the Forum and involves members of NCAOD. Interview respondents revealed that the goal of the PSG is to provide support to parents of children with drug related problems.
'Our primary goal is to get parents to look after themselves, it's not just about looking after your children, it's about looking after yourself' 'The aim of the group was to support each other, to reduce isolation, to equip parents with coping skills and support and education' The interviewees were very positive about the group and felt it was useful for those who accessed it.
"So we are saying you have to learn to take these things in a sensible, rational, reasonable way where you don't blame yourself and meanwhile you retain the things that are important to you. So you survive it because the most important thing is the family is still a supportive component when that person finishes their drug use because that is when they are going to need you very much, at the end. I think that message was clear and we provide a lot of support around that. " "I noticed that it is predominantly the mothers that come to those groups, a lot of single mothers and they are really struggling with drug related issues and that support group taught them I guess how to take care of themselves. That's
27
certainly what it taught me, you know, how to look after myself and that's what you have to do so that when the journey has ended and everybody is well and safe then we can pick up the pieces and be well enough to do so. " In regards to the short tenn impact being met and the PSG being ongoing, the group has not yet achieved this. The latest group convened in March 1999 and ended in late 2000, although a working group has remained since the end of formal meetings and is in the processes of beginning the next PSG with plans to make it of an ongoing nature. Interviewees revealed the difficulty of attracting the numbers to enable the PSG to be ongoing. "... it's always been an issue to get people along to these things, I mean people when they reach a crisis will ring and seek out some assistance, but to get them to follow up on that once the crisis is over or to try and prevent it again is a very difficult thing". To address the problems, interviewees advised that they have been given a small grant by DASC and will be using this to promote the PSG in the local shopping mall and are hopeful that this will recruit a larger number of parents than has been possible in the past. Along with this new recruitment tactic, respondents advised that they will take a similar approach to last time which they believed worked the best, and that is to advertise the group from a parent's perspective. Talking of an advertisement she placed in the local newspaper, one respondent noted: " .. .I think people just read that feature ifyou like and thought, well that's a mum, it's not authority, it's not government, it's not police, it's a mum. So that's what I think worked well and I think that will work well again. I would be confident that would work well again ".
General Comments on Forum When asked if there are any improvements that need to be made regarding the Forum, many respondents reported a need for the Forum to have greater direction and become more focussed on specific issues. Of these, many suggested having sub-committees or smaller working groups to do so. "Infonnation sharing is good, but maybe more specific direction towards local issues - sub committees should be developed"
"If action plans can be put into action and acted upon then the Forum would have more credibility " "Setting defined goals and tasks. May need sub-groups formed for short tennl specific tasks" As well as identifying areas in need of improvement, many respondents concluded the survey with very positive feedback, highlighting the value they hold for the Forum. 'A well worthwhile Forum - the fact that it exists as strongly as it does after a few years speaks volumes" "Forum must continue"
28
"The Forum is a positive approach to dealing with complex issues" "/ believe this is afabulous Forum and have really appreciated wide range of mechanisms used to monitor, review and evaluate the Forum for continued improvement"
Discussion Overall, the findings show that NCAOD has been successful in achieving to some extent all but cine short-term impact. The findings have further revealed preliminary information to suggest they have also met some of the predicted intermediate-term impacts/outcomes. This section will focus on the extent to which these impacts have been met, in terms of the likelihood of the Forum achieving their objectives. Increase communication and coUaboration in the community to address local drug issues. The Forum can definitely be seen to be effective at increasing communication and collaboration in the local community regarding drugs and related issues. The Forum can also be seen as effective in attracting resources to address drug related issues. An interesting question is whether this increased communication and collaboration is addressing drug issues that are relevant for the local community. Community members have very low attendance rates at Forum meetings, Forum members indicated the need for greater community member representation at meetings, and that service providers have greater influence over the decision making processes of the Forum than community members. This suggests then that many of the issues being addressed are predominantly those nominated by service providers. Certainly service providers would be acting on behalf of community members in making decisions, but the question remains if they are the issues community members themselves would concentrate on if given the chance. Also, we need to question whether the prevention and treatment programs resourced by the Forum are those community members believe are the most effective for the Onkaparinga community. Such questions raise the issue of whether the needs of the community can be adequately addressed through service providers. Healthy Cities advocates for the identification of needs to come from community people (Bauro, 1998, Bracht and Tsourus, 1990, Smithies and Adams, 1993), with professionals acting as 'facilitators' rather than 'directors' (Baum, 1998). Community identified need is thought to lead to issues being addressed that are most relevant for the community, in ways that are most effective for the community. Having power in decision making processes that affect them is also believed to lead to empowerment of individuals and communities, an essential component, according to the Ottawa Charter, in improving health and well being (Baum, 1998, Bracht and Tsourus, 1990, Rissel, 1994, Smithies and Adams, 1993). It seems therefore that the Forum needs to continue its success with increasing communication and collaboration and attracting resources for prevention and treatment programs that address drug related issues. However, there is room for improvement with the inclusion of more community members, and the referring of power so that community members hold sway over decision making processes.
Improve intervention and support strategies for people at risk of harm from drugs to prevent further harm. Figure 5.2 shows that the NCAOD Forum members are predicting that the establishment of a peer support worker at the Needle and Syringe Exchange Program and the creation of drug free recreation for young people, will lead to increased use of the services and eventually achieve the hoped for health and development outcomes. A short-term analysis would normally need to
29
examine the literature surrounding the topic to determine whether there is support for these predictions. However, due to time constraints, we have not done so for the purpose of this framework testing exercise. It can be seen that the Forum has been successful in creating and promoting drug free recreation and that a number of young people in the community are using it. Further analysis into the intermediate-term impacts though would need to determine who is using the drug free recreation and if it is involving young people from all sub-groups of the community. Further analysis would also need to determine whether the drug free recreation is preventing further harm from drugs, to those who are at risk.
In terms of a peer worker being established and sustained at the Needle and Syringe Exchange Program, confusion surrounding this position is such that a future analysis would need to examine the program again and see if the position has been fulfilled in an ongoing manner as planned. The confusion that exists highlights an interesting point about the difficulties that can arise in complex community partnerships, particularly in terms of administrative tasks such as record keeping and documentation, when the task is not "owned" by one particular organisation.
Increase awareness of drug issues and available support services for community members. NCAOD do not appear to be actively involved in increasing the awareness of drug issues and available support services for community members. They nominated a media campaign as a useful strategy for doing so, and this would seem a logical step. However, although a Media Advisory Committee was set up in the past, reasons (undiscovered) prevented the implementation of the campaign. Again, this highlights some of the challenges faced when working in intersectoral community partnerships where (for whatever reasons) the inability of one partner to fulfil its role, can have strong implications for the workings of the project. It is important to note here though that financially it is very difficult for NCAOD to be active in raising community awareness of drug issues and support services. NCAOD are operating under very limited funding, with the only guaranteed funds AU$5000 per year from DASC.
Increase effective information and support services in the community for family and friends of people with drug related problems. In terms of increasing information and support services in the community for family and friends of people with drug related problems, the forum appears to be involved in creating effective strategies for parents of people with drug related problems, with the Parent Support Group. The only improvement required in terms of the PSG is the need to involve greater numbers, but as the findings above indicate, the PSG coordinators have strategies in place to address this. Further analysis would determine whether these have been successful. The Forum also planned to enhance Child and Youth Health's 'Parent Help Line' to be capable of assisting parents of young people with drug related problems, however no action has been taken at this point. Again though this strategy would only target parents.
In regards to other family members and friends of people with drug related problems, there does not appear to be any plans in place to address this. The inclusion of other family members and friends when looking at improving information and support services is obviously required to fulfil the above objective and achieve the predicted health and development outcomes. It is important to note here, as with all of the objectives, that the Forum is predicting these strategies will lead to the desired outcomes. An evaluation would normally need to turn to relevant literature to find further support for the strategies.
30
6 6.1
Assessment of Framework Development of Framework through Stages
The framework provided a generally appropriate model for conducting an evaluation of complex, community-based initiatives such as Healthy Cities projects. Our experience suggests that a systematic and planned approach to evaluation enables a realistic and useful evaluation to be conducted. From the experiences gained in the testing of the Healthy Cities Evaluation Framework we have made some changes to its conceptualisation. These are shown in Figure 6.1.
Figure 6.1 The Revised Healthy Cities Evaluation Framework
Stage 1 Project Objectives Evaluation Plan
At An Stages Consider environmental influences on Healthy Cities initiative
Stage 2 Short-term Impacts Implementation
At All Stages Use a mix of qualitative and quantitative methods and ensure all stake holders are
Stage 3 Intermediate health promotion outcomes
Stage 4 Health & Development outcomes
In the course of planning the evaluation, we developed four stages to the evaluation research and these proved to be robust in Stages 1 & 2 (the two stages that were tested in this research). These stages are a useful to conceptualise the evaluation and have been added to the HCEF. Box 6.1 spells out in more detail the activities that would be typically associated with each stage.
31
Box 6.1. Stages for a Healthy Cities Evaluation Stage 1 • Engage with the key players in the project, typically by presentation at the forum or advisory and/or management committee; • Hold workshop with key players (including community representatives) in order: to clarify objectives, strategies, predictions in terms of short term impacts, intermediate term health and well-being outcomes and long term health and development outcomes, and ways to measure their achievement; to determine likely social, political, economic and physical environmental impacts on the topic that is the subject of the Healthy Cities initiative; and to agree the outline of an evaluation plan; • Confirm work of workshop with advisory/management committee; • Apply literature review to support predictions made which link short and intermediate term health promotion outcomes to the long term ones; • Establish an Evaluation Advisory Group with membership from evaluation team and project committee. Stage 2 • Implement evaluation plan in regard to measuring short term impacts; • Collect and analyse data; • Report back and encourage action based on findings. Stage 3 • Check whether objectives need to be modified; • Implement evaluation plan in regard to measuring intermediate term health promotion outcomes; Collect and analyse data; Report back and encourage action based on findings. i I
!. I.
I.
i
Stage 4 Check whether objectives need to be modified; Implement evaluation plan in regard to measuring long term health and development I outcomes; Collect and analyse data; ~ Report back and encourage action based on findings.
We found that, in the first stages of the evaluation, the data collected was primarily subjective reports from the key players. These data are helpful in testing a shared perception of the project among key players and are particularly useful for clarifying the rationale and objectives, and for refining the ways in ~hich the initiative is being run. Specific aspects of the framework are discussed in detail below.
6.2
Evaluation Worksbop
The evaluation workshop held with key stakeholders including agency representatives and community members, proved to be a highly effective mechanism for involving these players and increasing their understanding of evaluation. The workshop provided an opportunity for the evaluation team to explain in detail the particular challenges of evaluating a Healthy Cities initiative. Presentations were made by two members of the evaluation team and covered much of the content presented in the literature review. The evaluation of the workshop demonstrated that
32
for most of the participants this was new information and provided them with a more detailed understanding of the complexities of evaluation. The process of clarifying objectives was also helpful. While the Noarlunga Community Action On Drugs did have some broad goals prior to the workshop, these were not really articulated in a way that made them easily evaluable. The workshop provided participants with the opportunity to take time out to develop a shared understanding of the initiative, to articulate objectives that are more evaluable and to consider what indicators of these might be. The evaluation team was somewhat concerned that not all Forum members were present at the workshop and that the feasibility of one or two ofthe objectives may not have been thoroughly debated. Nonetheless, the broader Forum endorsed the set of objectives and the newly established objectives were much clearer and evaluable than the previous rather broad goals. It also proved useful to encourage people to think about the varied social, economic, political and physical environmental factors that might impinge (either positively or negatively) on the achievements of the Healthy Cities objectives. This process brings to the fore the fact that Healthy Cities initiatives are embedded in community settings and so are, inevitably, subject to influence by the many factors that affect health. In traditional research frameworks these factors are seen as "confounders". Yet this seems to ignore the reality of everyday life. It seems preferable to incorporate consideration of these factors directly into the evaluation and acknowledge them as a central part of the assessment of the success or otherwise ofthe initiative. The workshop also meant that the evaluation process and methodology was developed in partnership with the project partners rather than imposed from outside researchers. The benefits of this are that the evaluation is more likely to have an impact on practice as people feel satisfied with the process and so will be more open to listening to the findings. The particular research team working on this evaluation had an additional advantage in that the Department of Public Health and South Australian Community Health Research Unit have had an on-going link with the Healthy Cities Noarlunga initiative sInce 1987, so the relationship is well-established.
6.3
Evaluation advisory group
The establishment of an Evaluation Advisory Group has proved to be important in order to maintain communication between the evaluation team and the project. This group was able to assist with survey design and advise on details of methodology. This group is also the main mechanism by which the emerging findings from the evaluation feedback to the project. So they have a very crucial role in encouraging use ofthe.evaluation results.
6.4
Short and intermediate-term measures
The clear delineation of objectives into short term impacts and intermediate term outcomes worked well. This made it very clear that only certain indicators could be measured early on in the initiative and that others could only be expected to be achieved over the longer term. The use of predictions as to the effect of certain process achievements also contributes to the strength of the case that the initiative has led to a particular outcome. An example in the case ofNCAOD would be that increasing the options for drug-free entertainment would lead to a reduction in youth drug use. We also recommend asking the question recommended in WHO (2000) "What else happened in these projects?" in addition to measuring progress against initial objectives. Unintended consequences of the project are important and should be documented.
33
6.5
Building the case for change based on prediction and theory
There was little evidence that the objectives set for NCAOD were based on a clear statement of existing theory and predictions regarding the change that might happen as a result of interventions by the NCAOD. To some extent this was because the assumptions underlying the project were implicit rather than explicitly stated. We would suggest that more attention to building theories of change is given in Healthy Cities initiatives. We understand that project stakeholders are usually keen to advance with the action stage of their project. However, spending more time on conceptualising the model underlying the project would make evaluation easier to design and make it more possible to make claims about the impact of the Healthy Cities initiative. Judge and Bauld (2001, p. 35) note that "an understanding of cause and effect is remarkably difficult to establish in complex open systems". Consequently it is crucial to justify the rationale behind an intervention in detail and suggest doing this through the application of a logic and theory of change model. An example of how the notion of logic models and theory of change might apply to the NCAOD is provided in Figure 6.2.
We found that the rationale and strategy for NCAOD was well articulated in terms of the general need for an intersectoral approach to drug issues in the local community but less well articulated in terms of the selection of the particular activities.
34
The issue of building the case for change will become more crucial in Stages 3 & 4 of the evaluation. We suggest that five questions will be crucial in these stages in relation to measuring outcomes. They are taken from the work of Pelikan et al (1999): 1. What changes to what variables/dimensions can be expected as a consequence of he intervention? 2. How can these changes be measured? 3. Is there evidence of change in these variables/dimensions? 4. Are measured changes meaningful? 5. Can changes be attributed to the intervention? To what extent? The final question is particularly challenging and will be further addressed in the following conclusion section.
35
7
Conclusion and Recommendations
This report has described the application of the Healthy Cities Evaluation Framework to an initiative that is part of Healthy Cities Noarlunga. The value of the framework and the lessons learnt from the testing have been described in the previous section
7.1
Endorse the Value of the Healthy Cities Evaluation Framework
We conclude that the framework proved to be a robust means of evaluating Healthy Cities type projects. Breaking the application of the framework in to four stages proved useful as it provides a clear idea of what the evaluation is attempting to achieve at each stage. This process also allows for collaborative planning of the evaluation and for engagement of key players in the process. We recommend that Healthy Cities Evaluation Framework as shown in Figure 6.1. is a suitable means of evaluating Healthy Cities and like initiatives. It provides the means to combine qualitative and quantitative methods and contributes to establishing causal relationships between Healthy Cities interventions and eventual health and environment outcomes. It also encourages participation from the key stakeholders in the project.
7.2
Rigorous Project Planning
This evaluation and the testing of the Healthy Cities Evaluation Framework highlighted the importance of rigorous project planning from the onset of the project. Good evaluation is dependent on agreed, clear, evaluable objectives, backed up by a rationale linking planned strategies to expected outcomes. Often in community projects, the significance of such planning is overlooked. Rigorous planning is most likely to occur when initiatives are planned in a reasonable time-scale and where those undertaking the planning have a good understanding of the requirements ·of sound evaluation. This is because they will appreciate the need for clear objectives and a statement of why particular objectives and strategies have been selected. Where there is little understanding of evaluation among project participants, it may be necessary for them to consult with individuals or institutions who specialise in evaluation, to develop a project plan that is well thought out and thus conducive to good evaluation.
7.3
Causality in Healthy Cities Evaluation
While the project did not implement Stages 3 and 4 of the evaluation framework, the evaluation team has given thought to the requirements of these stages. Stage 3 would be relatively simple to implement, being, in most respects an extension of Stage 2. Stage 4, however, will be far more challenging. This requires measurement of relevant health and development outcomes (which is possible) and then linking these to the Healthy Cities initiative in such a way that a causal link can be established. As explained in the literature review, this is difficult. Our recommendation is that the likelihood of causality is established through a technique of triangulation and building up the case in favour of the impact of the Healthy Cities initiative. This form of evidence is that most commonly used outside medical and scientific research. We would recommend that the evaluators build up this case by collecting a variety of evidence and then examining the case for the impact the initiative has had. Paying careful attention to the theory of change and logic underlying the initiative can contribute to this. Thus we recommend that in Stage 1 of the evaluation, policyrnakers and practitioners pay careful attention to the development of the rationale and strategy underlying the selected interventions. Once these are established then they can be used in
37
building the case for a causal relationship between the Healthy Cities intervention and any observed change. In the process of establishing the theory for change we recommend that policy makers and practitioners should be able to fulfil the following factors that were identified by Judge and Bauld (2001, p. 36) in relation to complex community-based projects such as Healthy Cities. They suggest they should be able:
t
t t t t
To explain their starting assumptions and how they are related to critical aspects of the economic, social and political environments in which they work To specify in a plausible and preferably evidence-based way, why their chosen investments in interventions and process will take them in the direction of the long term outcomes they are seeking to achieve; To identify in advance the expected consequences of their actions in ways that lend themselves to be monitored and evaluated To commit themselves to a continuous process oflearning from the feedback that they obtain To be willing to modify their theories of change and the associated investments in the light of what is observed during the life of an initiative
Evaluators should be prepared to work with Healthy Cities stakeholders in order to fulfil the above criteria. The data they collect to support the case for change will be typically qualitative and quantitative and selected to cross-collaborate (triangulate) each other. This case should also consider the many environmental factors that may have impinged on the issue that is the focus of the initiative and determine how this consideration affects the evaluation conclusions. The evaluation team is firmly of the opinion that the evidence for Healthy Cities initiative can not and should not be expected to be built through the application of research methods designed for use in laboratories. The forms of evidence most suited to Healthy Cities as those that can be applied robustly in community settings. The evaluation framework allows for such an approach to be used.
7.4
Evaluation Teams
The approach taken in the pilot was for the evaluation to be conducted by a team of specialist evaluators with backgrounds in social science and public health. The evaluation team was advised by an Evaluation Working Group that reported back to the Healthy Cities committee (in this case the Noarlunga Community Action on Drugs Forum). We recommend that this approach is effective and should be used in other evaluations. It combines the advantages of internal and external evaluation. The evaluation team was independent of the initiative being evaluated but had an on-going relationship with the Healthy Cities program. They were able to bring an outsiders perspective but were well-placed to work closely with the initiative stakeholders. We also recommend that community participants are given an opportunity to contribute to the evaluation design.
7.S
Ensuring evaluation findings are implemented
An on-going issue for evaluation is ensuring that its findings are used. Pawson and Tilley (1997) note that they know of decision makers 'whose daily work involves removing from their life's agenda the uncertainties so typically raised by research'. They also note 'We know well the symptoms of 'myopia', 'selective vision' and indeed 'blind eye' that afflicts many policy makers when they do actually confront evaluation documents'. In the early stages of the evaluation we did not encounter this problem and suspect this was because of the involvement of the key stakeholders in the project. Also the evaluation was concerned with process issues and most of these were not controversial or even very surprising. We recommend that the issue of implementation of evaluation findings is discussed with the Healthy Cities initiative at the end of
38
each stage so that concerns from the project and from the evaluators can be addressed. This should help ensure the findings led to organisational learning and change in relevant policies and practices.
7.6
Further Testing and Development of Healthy Cities Evaluation Framework
The funding from WHO permitted testing of the very early stages of the evaluation framework. The testing work will continue through 200 I as funding from the South Australian Department of Human Services has enabled the extension of the project. We recommend that the Healthy Cities Evaluation Framework using the four stages described above is tested in a variety of Healthy Cities settings in the Western Pacific Region. The experiences from the various test sites should be collated and general lessons drawn from these.
39
References Baum F. & Cooke R. (1992) Healthy Cities Australia: the evaluation of the pilot project in Noarlunga, South Australia. Health Promotion International, 7(3), 181-193. Baum F (1998) The New Public Health: An Australian Perspective. Melboume, Oxford University Press. Baum F. (2000) The Effectiveness of Community-based health promotion in Healthy Cities Programmes: Issues of Measurement and Evaluation. Plenary paper presented at the International Conference on Healthy Cities and Urban policy Research, 12-16 March 2000, Tokyo, Japan. Boonekamp G., Colomer c., Tomas A. and Nunez A. (1999) Healthy Cities Evaluation: the co-ordinators perspective. Health Promotion International, 14(2), 103-187. Bracht J .H. and Tsourus A. (1990) Principles and Strategies of effective community participation. Health Promotion International, 5(2), 199 - 208. Burton S. (1999) Evaluation of healthy city projects: stakeholder analysis of two projects in Bangladesh. Environment and Urbanisation, 11(1),41-52. Connell J.P. and Kubisch A.C. (1998) Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects and problems. In Fulbright - Anderson et al (eds). New Approaches to Evaluation Community Initiatives, Vol. 2 Theory, Measurement and Analysis, Washington, DC: Aspen Institute Costings C. and Springett J. (1995) City Health Plan: The Effectiveness of Joint Working. Institute for Health, Liverpool, John Moores University, UK. Costings C. & Springett J. (1997) Towards a Framework for the Evaluation of Health-related Policies in Cities. Evaluation, 3(3), 345-362. Curtice L. and McQueen D. (1990) The WHO Healthy Cities Project: An Analysis of Progress. Presented at WHO Healthy Cities Business Meeting, Belfast, Ireland. de Leeuw E. (1999) Monitoring, accountability, reporting, impact assessment: A framework for Phase III Healthy Cities. Centre for Urban Health, WHO, Regional Office for Europe. Dooris M. (1999) Healthy Cities and Local Agenda 21: the UK experience - challenges for the new millennium. Health Promotion International, 14(4),365-375. Doyle Y., Tsouros A., Cryer C., Hedley S. and Russell-Hodgson C. {I 999) Practical lessons in using indicators of determinants of health across 47 European Cities. Health Promotion International, 14(4), 289299. Gillies P. (1997) Review and Evaluation of Health promotion. The Effectiveness of Alliances or Partnerships For Health promotion. Presented at the Fourth International Conference on Health Promotion. Jakarta, Indonesia, 21-25 July 1997. Hancock T. (1993) The Healthy City from Concept to Application. In JK Davies & MP Kelly. (Eds) Healthy Cities: Research and Practice. London, Routledge. Hayes M. and Williams S. (1990) Healthy community indicators: the perils of the search and paucity of the find. Health Promotion International, 5(2), 161-166. Judge K. and Bauld L. (2001) Strong theory, flexible methods: evaluating complex community -based initiative, Critical Public Health, 11(1), 19-38.
41
Kelly M. (1992) Theoretical Problems and Practical Applications: Developing an Appropriate Science for Healthy Cities. Paper presented at the Research for Healthy Cities Seminar, Maastricht. Kennedy A. (1995) Measuring Health For All- a feasibility study in a Glasgow community. In N Bruce et al. (Eds) Research and Change in Urban Community Health. Ashgate Publishing Company. Milio N. (1990) Healthy Cities: the new public health and supportive research. Health Promotion International, 5(4), 291-298. Ouellet F., Durand D. and Forget G. (1992) Case Study and Interactive Approach: Preliminary results of an evaluation of three Healthy Cities Communities initiatives in the Montreal Area. Paper presented at the International Experts Panel 'Healthy Cities Research Agenda for the 1990s', Maastrich, November 15 th _ th 18 , 1992. Pawson R. & TilIeyN. (1997) Realistic Evaluation London, Sage. Pelikan J., Krajic K. and Dietscher C. (1999) The Evaluation of Health Promotion Interventions in Systems and Settings. WHO Draft Paper, 2001. Rada 1., Ratima M. and Howden-Chapman P. (1999) Evidence-based purchasing of health promotion: methodology for reviewing evidence. Health Promotion International, 14(2), 177-187. Rissel C. (1994) 'Empowerment: the Holy Grail of Health Promotion?' Health Promotion International, volume 9(1), 39 - 47. Rootrnan I., Goodstadt M., Potvin L. and Springett J. (1997) Toward a Framework for Health promotion Evaluation. WHO Regional Office for Europe, Copenhagen, Denmark. Smithies J. & Adams L. (1993) Walking the Tightrope: issues in evaluation and community participation for Health for All. In JK Davies & MP Kelly. (Eds) Healthy Cities: Research and Practice. London, Routledge. Trogan A., Stumm B. and Sub W. (1992) Health promotion, Citizen Participation and Urban Redevelopment: Description of a Research Project. Institut fiir Medizin-Sociologie, Martinistrasse, 52, D 2000, Hamburg, Germany. Werna E. and Harpham T. (1995) The evaluation of Healthy Cities projects in developing countries. Habitat International, 19(4), 629-641. Whitehead M. (1993) The Ownership of Research. In JK Davies & MP Kelly. (Eds) Healthy Cities: Research and Practice. London, Routledge. WHO (1995) Twenty Steps for developing a Healthy Cities Project. Copenhagen, Regional Office for Europe. WHO (1998) Health Promotion Evaluation: Recommendations to Policy-makers. Report of the WHO European Working Group in Health Promotion Evaluation, Copenhagen, WHO. WHO (1999) Workshop on Healthy Cities: Preparing for the 21" Century. WHO Regional Office for the Western Pacific, Malacca, Malaysia, 1999. WHO (2000) Healthy Cities in Action: 5 Case studies from Africa, Asia, Middle East and Latin America, Department of Protection of the Human Environment, WHO: Geneva.
42
Appendix Appendix A NCAOD Membership List Drug and Alcohol Services Council Other Health Christies Beach Second Story division • Child and Youth Health: Southern Women's Community Health Centre • Noarlunga Health Services: Noarlunga Health Village Noarlunga Hospital Community and Allied Health South Australian Police Education Sector • Christies Beach High School • Hackham South Primary School • Department of Education Training and Employment • Life Education Community and Welfare Services (government) • Family and Youth Services Community and Welfare Services (non-government) • Offenders Aid Rehabilitation Service • Wesley Uniting Mission • Aboriginal Sobrietry Group • Mission Australia • Southern Youth Ensemble • Adelaide Central Mission Local and State Government • Crime Prevention Unit • Member for Kuarna • Member for Reynella • Labour candidate for Kingston
t t
t t
•
43
AppendixB Evaluation Workshop Workshop Invitations an invitation to participate
The South Australian Community Health Research Unit is trialing an evaluation process on behalf of the World Health Organisation. The Noarlunga Community Action on Drugs has agreed to participate, and the trial evaluation will look at the initiative as a whole, with specific focus on the Youth Peer Drug Action Project. I hope that you will participate in the evaluation process by attending a workshop with other members of the Action Group, along with delegates and project participants from Second Story. When Where Thursday, February 8, 9:00 AM St Basil's Aegean Village Complex Morton Rd, Christie Downs
You are important to the evaluation process As part of the Noarlunga Healthy Cities initiative, we are hoping to operate in line with Healthy Cities ideologies, involving all key stakeholders in the decision making processes of the project, including the evaluation The aim of the workshop therefore is to identify as a group; the objectives of the project, the short to long-term outcomes we are expecting to see, the factors that will indicate the project's success, and how we should measure these. The workshop will run until I :00 PM, with a break for morning tea and lunch afterwards. Please feel free to contact SACHRU if you have any questions or concerns regarding the th workshop. I will be overseas until the 15 of January, but will be more than happy to answer any queries after this date on (08) 8204 3926 or e-mail danielle.bament@flinders.edu.au. In the mean time Gwyn Jolley a Senior Research Officer in the unit will be available on (08) 8204 5978 or e-mail Gwyn.lo11ey@frnc.sa.gov.au. RSVP Can you please RSVP to myself or Gwyn on the contact details above re your attendance to the workshop no later than Thursday, January 25, 2001.
Yours Sincerely,
Danielle Bament Research Assistant
45
Workshop Exercise for NCAOD
Group: Noarlunga Community Action on Drugs Decide who will facilitate and record discussions.
Part One (4Sm) Look at the proposed objectives and strategies. These have been derived from project documents. Reflect and discuss why the Forum was set up, what do you want to change, what do you hope (predict) to achieve (objectives) and how you plan to do it (strategies). Remember objectives are about change in individuals, communities, organisations, legislation, social and environmental factors. On the work-sheets, write down the objectives and strategies you have agreed on. (25m) Rewrite the agreed objectives as outcome statements. Decide whether they are immediate short term impacts, intermediate health and well being outcomes, and long term health and development outcomes. Decide at what time-scale each objective could be assessed using the key. Record these on the work-sheets. (20m) Consider what external factors might influence the achievements of your predicted outcomes and record these on the work-sheet. (10m).
Part Two Consider what environmental factors might influence the achievements of your predicted outcomes and record these on the work-sheets (10m). Taking each objective in turn, discuss how you wiJI know if you have been successful, if your objective has been achieved? What are the signs or indicators of success? Write them down in the indicators column. (25)m Using indicators, think about how you will assess and measure your achievements. What information will you need to collect? Write this in the measurement column. Now repeat with the other objectives. (25m).
46
Completed worksheets for Draft Evaluation Plan: Youth Drug Peer Action Project Goal: To work in partnership with young people in the Onkaparinga area to prevent and reduce harm arising/rom drug use. Objective 1 • To increase the effectiveness of current strategies in place to prevent and reduce harm arising from drug use by young people in Onkaparinga Strategy • review literature and other drug strategies Outcome statement Strategies to prevent and reduce harm arising from drug use by young people in Onkaparinga are more effective Time-scale long term 2 years +
•
... ...., Environmental factors • changes in employment levels changes in supply and availability of drugs • extent of inter-generation use • political and funding changes • impact of new' Drug Diversion Strategy' • large amount of support for this project in the community
use information gained to inform production of new Youth Drug Strategy Measurement • pre- and post-project implementation survey of young people
•
Indicators • young people's change in perception of effectiveness of strategies
•
strategies are more effective in Onkaparinga than similar community
•
compare statistics on harm from drug use in Onkaparinga and another similar community
Objective 2 • To increase young people's involvement in raising awareness and knowledge of drug issues in a range of Gov, non Gov and community organisations
Stratein' • provide support and training for young people
Outcome statement Young people are actively involved in consultations with organisations Organisations have increased awareness and knowledge about drug issues
Time-scale short < I year
• •
young people consult with a range of organisations set up a web site?
intermediate \-2 years, continuing
""
....
Environmental factors • responsiveness of organisations • rhetoric of participation • new methods of communication eg Internet
Indicators number of peer educators • proportion of peer educators completing training • proportion of peer educators retained for \ year • number of organisations involved • quality of training and support • organisations change policy on youth participation • organisations change to a youth participation culture
•
Measurement • project records
• •
feedback from peer educators and project officer audit of organisations
t
Ob_iective 3 • To increase awareness and knowledge about drug issues in young people and the general community, by young people
Strategy
• • •
provide support and training for young people young people consult with a range of organisations set up a web site?
Outcome statement Young people and the general community have increased awareness and knowledge about drug issues
Time-scale short to intermediate 1-2 years, could be measured more than once
'"
....
Environmental factors • media coverage: tool to increase awareness and knowledge, but reinforces stereotypes • DETE policy changes • individual school responses • different cultural backgrounds and ideas about drug use
Indicators number of peer educators • number of young people involved in NCAOD • level and content of media coverage • number of invitations to community groups number of people reached • pre and post 'test' of awareness and knowledge • number of schools that adopt changes in drug policy • number of hits on website
•
Measurement • project records
•
•
survey survey count
•
•
Objective 4 • To assist young people to be partners in developing youth drug strategies
Strategy • provide support and training for young people
Outcome statement Young people's input is included in the development of the YDS
Time-scale Intermediate to long, throughout
• • o v.
young people consult with a range of organisations the result of consultations informs development of YDS Measurement project records
Environmental factors • problems in reaching and engaging with young people other agencies may block partnerships • • power differentials
•
Indicators number of peer educators • proportion of peer educators completing training • proportion of peer educators retained for I year • number of organisations involved • quality of training and support • young people are engaged as active partners with agencies
•
• •
feedback from peer educators and project officer
feedback from young people I
Objective 5 • To increase coordination of services for young people Environmental factors • DHS initiatives and policies: youth development coordinators, youth funding focus, child and youth policy • political
'"
Stratein' • liaison and consultation with agencies establishment of steering group • Indicators • numbers of services involved in consultations number of services represented • on steering group • number of meetings of steering group and level of attendance and support for project • policy changes to reflect coordination • number of new collaborative initiatives - - -
Outcome statement Services for young people are more coordinated Measurement • project records
Time-scale intermediate to long
• •
audit of agencies audit of new initiatives -------
---
--
Completed Worksheets for Draft Evaluation Plan: Noarlunga Community Action on Drugs Evaluation Plan Goal: To prevent and reduce harm caused by drug use to the community within the city of Onkaparinga Objective 1 • To increase communication and collaboration in the community to address local drug issues Strate~
• •
Foster an intersectoral approach by establishing an ongoing community forum
N
U>
Environmental factors • Support from the Drug and Alcohol Services Council and Healthy Cities Noarlunga
Identify and attract resources for prevention and treatment programs Indicators • Attendance and representation is ongoing
Outcome statement Communication in the South between community members, non-government and government organisations regarding drug related issues is increased leading to effective strategies to address local drug issues.
Time-scale Short- Long Term
Measurement • Audit of Forum meetings, numbers and diversity of those who attend
• •
Funding Work demands outside of the Forum on Forum members.
•
Funding attracted to drug and alcohol programs Increased networking and communication in the community surrounding drug related issues
•
Document analysis of partnerships that have been formed from the Forum Survey of members motivation for and perception of the Forum I
•
•
---
Objective 2 • To improve intervention and support strategies for people at risk of harm from drugs to prevent further harm
... v.
Environmental factors • Funding • Diversion programs from other organisations (eg SA Police) • Availability of drug free recreation sites • Availability of housing
Strategy • Establish Peer Worker at the Needle and Syringe Program • Increase/improve training for community workers who deal with workers who deal with people facing drug related problems • Provide appropriate accommodation for recovering drug users (eg shelter) • Establish a 'street work team' • Create drug free recreation for young people Indicators • Number of people who attended counselling and treatment for a regular period • Number and type of services available • Change in attitudes for drug users towards local treatment and support services • Change in attitudes for community workers towards people who are facing drug related problems
Outcome statement Strategies to intervene, prevent and support people at risk of harm from drug use are more prevalent and effective
Time-scale Short - long term
Measurement • Audit of number and type of services available
• •
Focus groups with community health workers and people who are recovering from drug use Statistics from accident and emergency admissions at Noarlunga Hospital
Objective 3 • To increase awareness and knowledge about drug issues and support services for family and community members
u.
... Environmental factors • Transport for those living outside of the city Competition and overload of information in the media context • School drug policies
•
Strategy Media campaign to advertise local information and support services • Run Adolescent Health Information Evenings for parents • Re-establish links with Family and Youth Services Establish ongoing Parent Support • Group • Contact Parent Help Line to negotiate promotion of service for parents of young people with drug related problems Indicators • Greater number of family and significant others of people with drug related services access information and support services • More information and support services for family and members and significant others are available
•
Outcome statement Family and community members feel more informed, confident and capable about resolving drug issues
Time-scale Intermediate to Long
Measurement • Audit of support services before and after strategies are implemented
• •
Audit of usage of services Phone survey of community members regarding knowledge of local services before and after media campaign
-
Workshop Results/rom Community Members Small group with community members on evaluation of Healthy Cities Questions t What has helped the project to stay alive and be successful? t What are the pre-conditions for success for Healthy Cities Initiatives? t What supportive environments need to be present? t How do you know if they have been successful? t What information would need to be collected? t What environmental factors might influence the achievement of your predicted outcomes? t What are the best ways to consult with community people? t What are the best ways to involve communities in evaluation of Healthy Cities initiatives? Agreement that it is necessary to consider the pre-conditions for Healthy Cities projects. This is now very important in Noarlunga where the success of one project builds on that of the next. Would be good if we could come up with a checklist for rating the preconditions as part of this work. Perhaps something that adds up to six areas of measurement which could then be plotted to measure progress. The group discussed the features of successful projects that should be monitored in any evaluation and this produced the information in the following table Features of Successful Projects Effective Community Involvement Indicators Representatives on Management Committee of Healthy Cities Number of community involved in activities Resources provided eg conference attendance AGM attendance Do community people stay involved? Community reps - represent Healthy Cities on other activities Knowledge of Healthy Cities in community Number of projects that are active Report back and perceptions from community members of Healthy Cities spin-off projects There is a strategic planning process Reflect things that have been identified in the strategic planning and are actioned Number of visits from overseas and interstate people Visits by Noarlunga people to events/conferences/other Healthy Cities Short Course and site visit - on-going Links with Universities Link with WHO bodies Number of requests for information on Healthy Cities Noarlunga Website - hits on
Real and Relevant Initiative
NationallInternational InvolvementlNetworks
55
Features of Successful Projects Leadership (diffuse model ofleadership) Nurturing!Empowerment
Effective collaboration between sectors
Supportive policies Decision to adopt Healthy Cities project has local support/was initiated locally Committed host agency
Indicators How many people report Healthy Cities to other bodies/projects Effective team work People have access to full information Perception of community members that Healthy Cities is nurturing/empowering fuformal mentoring between community members Evidence of skills development (community memberships, evaluation etc) Mix of long term and new members Range and number of sectors involved in management Number of projects more than one sector/community involved Range and number of sectors involved in projects Do sector representatives have mechanism to report back to own agency Number of events involving mix of sectors Number of sectors at AGM Number of sectors at Planning Forums Activity reported in Annual Report Demonstrated, describable outcome Perceptions of players from sectors of effectiveness Description of model of collaboration Visit to project in action to assess "vitality" - getting a "feel" for project Evidence of sustainability over time Extent of trust between the sectors Agency have policies which support Healthy Cities approach Practical evidence of Local Government and local agency support e.g. resources, Mayoral receptions Higher understanding of Healthy Cities approach in local agencies Long-term agency support in terms of administrative support Budget allocation
Answers in response to the question "What environmental factors might influence the achievements of your predicted outcomes?" t Traditional role of other agencies might directly affect Healthy Cities projects (e.g., DASC and CYH in terms of Youth Drug Project t StatelFederal priorities e.g. Strengthening/ Building Communities project of the Federal Government t State of the economy/unemployment t Community cynicism t Other integrated planning initiatives such as Southern Social Planning fuitiative
56
Suggestions for methods to consult communities t Cut out slip in Messenger Press to ask basic question about Healthy Cities and community's knowledge of it t Could also be distributed through networks t Group discussion to bounce idea of each other but also recognised that you are more likely to get critical information from individual interviews t Noted that you need to structure feedback to ensure people can be critical and questioning. t They felt these types of workshops are good for involvement of community in evaluation.
57
Healthy Cities Evaluation Framework Testing Project: Feedback Results from Community Workshop l. How useful was the workshop in helping you understand more about evaluation? (scale 1 = not at all useful- 5 = very useful)
1 person indicated 2 2 people indicated 3 4 people indicated 4 7 people indicated 5
2. How useful was the workshop in helping you evaluate your group or project? 6 people indicated 4 7 people indicated 5
3. What was the most useful part of the workshop? Participation of members and diversity. Development of a process for evaluation and initiating a range of strategies to support project officer role. Small group reflection of evaluation assisted further refining ofproject objectives and measurement. ClarifYing of objectives re: issues Working with other group members sharing ideas about the forum and evaluation Understanding the evaluation approach being followed and discussion/clarification of objectives. Being really structured, precise and a range of different service's options and input Focussing on Healthy Cities evaluation with community members Interaction and "bouncing" of ideas - group discussion i.e. development of ideas and thoughts The other group evaluations Opening up of ideas and conversations which were constructive and objective Group activity in defining the objectives, strategies, outcomes, environmental factors, indicators, measurement. The liming as the peer education project begins. Involving clinicians with planners/evaluators
58
4. What could have been done differently? Accepting limited time, many issues have to be crammed into a short time. Broad range of topics discussed. This exercise could have been utilised for each project. Nothing - fine as it was. Available overheads - make handouts available to take home -refer back to
5. Is there anything else that should have been included? Overview on NCAOD and YDPAP for those of us unfamiliar with these projects. Share info? Data from other districts in SA Could young people have been included Gathering some idea of the participants current knowledge
6. Any other comments Very helpfol process for C & YH to work collaboratively internally and externally Very usefol exercise Found it very usefol Well done, I have enjoyed, learnt and felt energised A good and important involvement of community members and representatives from other agencies involved with drugs Time to mix with other agency members, who where who etc. participation Assisted me in professional development and understanding of research evolution and leadership values Response to question one - already has a good understanding of evaluation but was still needed for me to set the scene for the tasks we undertook. Has helped me to clarifY what we are doing and why and what we want to achieve and the strategies to achieve An excellent day A useful morning
59
AppendixC Survey Cover Letter Dear MrlMrs,
Re: Evaluation of Noarlunga Community Action on Drugs The South Australian Community Health Research Unit is currently in the process of evaluating Noarlunga Community Action on Drugs as part ofa Healthy Cities Evaluation Testing Framework. We are seeking feedback from all members on the success of the Forum in regards to reducing drug related harm to the community, in particular through new strategies, communication and collaboration. I would greatly appreciate if you could take a few minutes to complete the enclosed survey. It can be returned in the envelope provided or faxed back. All responses will be treated confidentially, and you will not be identifiable in any reports. The identification number at the bottom of the survey is to allow us to determine if you have returned your surveyor not and to send out reminders. We need your comments as soon as possible, and by the 21 51 March. Your feedback is .important in the evaluation of the project and future improvements. If you have any queries, please phone Danielle Bament (SACHRU) on 8204 3926, or Gwyn Jolley (SACHRU) on 8204 5978. We thank you in advance for your participation in the evaluation of this important initiative. Yours sincerely, DanieUe Bament Research Assistant
61
Survey of Noarlunga Community Action On Drugs Forum imate 1. The Forum was set up in Janua ry 1998, please indicate the approx you have been a memb er. From the beginn ing
length of time
o o o
More than 3 years
1-2 years
o
6 months - I year
o o
2 - 3 years Less than 6 months
o
o
memb ers you 2. Has your memb ership of the Forum been useful for the comm unity repres ent? Yes No
Please explain the reason s for your answe r:
is repres 3. How well do you think the comm unity of the Qnkap aringa area Forum ?
ented in the
2 poorly represe nted
3
4
5 well represe nted
that you believe 4. Is there any section of the comm unity not repres ented by the Forum should be?
ce over the decision 5. To what extent do you believe comm unity memb ers have an influen makin g processes of the Forum ?
2 no influence Any comme nts?
3
4
5 high influen ce
62
6. To what extent do you believe service provider members have an influence over the decision making processes of the Forum? 1 no influence Any comments?
2
3
4
5 high influence
7. How effective has the Forum been at increasing communication in the community about drug related issues? 1 not at all effective Any comments?
2
3
4
5 very effective
8. How effective has the Forum been at increasing coUaboration in the community about drug related issues? 2
3
4
not at all effective Any comments?
5 very effective
9. How effective has the Forum been in attracting resources for prevention and treatment programs? 1 not at all effective Any comments?
2
3
4
5 very effective
63
10. Below is a fist of strategies that the Forum has supported, please indicate how effective you befieve these have been. Parent Support Group 1 2 not at all effective
3
4
5 very effective
Hunting In Packs 1 not at all effective
2
3
4
5 very effective
Of Crime and Substance Project 1 2 not at all effective
3
4
5 very effective
Community Grants Scheme 1 not at all effective 2
3
4
5 very effective
11. Please fist any other strategies that have been supported by the Forum not fisted above, and rate their effectiveness.
2 not at all effective
3
4
5 very effective
2 not at all effective
3
4
5 very effective
1 not at all effective Any comments?
2
3
4
5 very effective
12. What is it about the Forum that you befieve has worked well?
64
13. What is it about the Forum that you believe could be improved?
14. Are there any other comments you wish to make?
Thankyou for your time. Please return this in the envelope provided or by faxing it to SACHRU on (08) 8374 0230.
65