'ANH/PSF/96.2. :.ev 1 English onlv :istr. G~neral LIFE SKILLS EDUCATION PLANNING FOR RESEARCH DIVISION OF MENTAL HEALTH AND PREVENTION OF SUBSTANCE ABUSE WORLD HEALTH ORGANIZATION GENEVA LIFE SKILLS EDUCATION '.1NH/PSF/96 2 R8V I =ngl1sh on1v ::istr. General PLANNING FOR RESEARCH AS AN INTEGRAL PART OF LIFE SKILLS EDUCATION DEVELOPMENT, IMPLEMENTATION AND MAINTENANCE This document describes research related to the development, implementation and maintenance of life skills education in schools. The different aspects of research related to life skills education are described within a logical framework which can be used to develop more detailed research plans at country level. The document is a guide for project managers responsible for overseeing the development and evaluation of life skills education, as well as for those who will undertake the research tasks. DIVISION OF MENTAL HEALTH AND PREVENTION OF SUBSTANCE ABUSE PROGRAMME ON MENTAL HEALTH WORLD HEALTH ORGANIZATION GENEVA 1996 Acknowledgemen;s This document was developed with guidance from pamc1pants of an informal consultation held at WHO Geneva ( l 995). co-sponsored by UNICEF. and with financial -;uppon from the Johann Jacobs Foundation. The document was prepared by Mrs Rhona Birrell Weisen of the WHO Programme on Mental Health. with the suppon of Dr John Orley. Progranune Manager of the WHO Programme on Mental Health. The WHO Progranune on Mental Health would like to thank the following people who attended the informal consultation for their contributions: Dr Janet Collins. Centre for Disease Control, USA Dr Jacob Adetunji, Harvard Centre for Population and Development Studies, USA Dr Anhur Richardson, University of the West Indies, Caribbean Ms Amy Lee University of the West Indies, Caribbean Dr Reiner Hanewinkel, Institute for Therapy and Health Research. Germany Dr Martha Givaudan. Institute Mexicano de Investigacion de Familia y Poblacion. Mexico Mr Jeff Lee, TA CADE. UK Mrs Vivienne Evans. TA CADE. UK Mr Daniel Pellaux. Mandat Plus Prevention, Switzerland Dr Joseph Foumbi. Health Promotion Unit, UNICEF (New York) Dr Mary-Liz Callaway, Programme on Substance Abuse, WHO (Geneva) Mr Jack Jones, Division of Health Promotion and Education, WHO (Geneva) Mrs Jane Ferguson. Division of Adolescent Health, WHO (Geneva) Dr Mariella Baldo, Global Programme on AIDS, WHO (Geneva) Thank you also for the comments from all those who reviewed the draft version of this document and for contributions from: Prof Roger Weissberg, The University of Illinois at Chicago, USA Dr Maurice Elias, Rutgers University, USA Dr Stefano Geuna. E. Lugaro Research Institute, Italy This document is not a formal publication of the World Health Organization (WHO), and all rights are reserved by the Organization. The document may, however. be freely reviewed, abstracted, reproduced or translated. in part or in whole, but not for sale or use in conjunction with commercial purposes. The views expressed in documents by named authors are solely the responsibility of those authors. Contents CHAPTER 1: INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Life skills education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Review of research literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 CHAPTER 2: THE RESEARCH PROCESS . . . . . . . . . . . • . . . . . . . . . . . . . . . . . . . . . . . . 17 Situation analysis -1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Formative research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Pilot test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Monitoring dissemination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Evaluation during implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Situation analysis -2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Long term evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Summary of the research process ................................. 45 CHAPTER 3: USING THE RESEARCH FRAMEWORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Suggested time-lines for the research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Who would be doing the research? ............................... 49 Costs of research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Communicating research findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 APPENDIX I: EXAMPLES OF RESEARCH RELATED TO LIFE SKILLS EDUCATION ..•....... 57 Example 1: Use of focus groups during a situation analysis in Zimbabwe .... 57 Example 2: Research to guide the development of a life skills programme in the UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Example 3: Pilot test evaluation of a life skills programme in Mexico ....... 61 APPENDIX II : DESCRIPTIONS OF SELECTED INSTRUMENTS USED IN LIFE SKILLS PROGRAMME EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Example 1: Instrument used to evaluate a programme for suicide prevention . . 63 Example 2: Instrument used to evaluate a programme for drug abuse prevention 65 Example 3: Instrument used to evaluate a programme to promote social adjustment and prevent alcohol abuse . . . . . . . . . . . . . . . . . . . . 67 Example 4: Instrument used to evaluate a programme for the promotion of social competence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Example 5: Instrument used to evaluate a programme for violence prevention . 71 Example 6: Instrument used to evaluation implementation of a life skills programme in Switzerland ................................ 72 Table 1: Table 2: Figure 1: Figure 2: Figure 3: Figure 4: Figure 5: Figure 6: Figure 7: List of Tables Life skills research overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Estimating personnel costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 List of Figures Model of the effects of life skills education . . . . . . . . . . . . . . . . . . . . . 4 Overview of the research process . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Analysis of situation analysis research findings . . . . . . . . . . . . . . . . . . 22 Model of the variable effects of life skills education over time . . . . . . . 30 Diagram of an experimental study ........................... 33 Model of a pilot test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Time-lines for research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 CHAPTER 1: INTRODUCTION :HAPTER , ·NiROCUCTICN :::age i Where life skills education initiatives are being set up, much time and resources will be put into writing materials. training teachers. promoting, disseminating and implementing life skills education. These are described in other documents in the life skills series. referred to below. This document describes the place of research in the process of life skills programme development, implementation and maintenance. It reflects the view that, for life skills education to be appropriate, to be effective and to continue to be relevant, research needs to be undertaken as an integral part of the whole process. Research can contribute to the effectiveness of life skills education by ensuring that its contents respond to the needs of the target groups. When a programme is implemented, research can serve as a gauge of the quality of the work being done. Research is sometimes seen as a threat by programme developers. but this need not be the case. Research can be introduced in a constructive way, to provide feedback to encourage people in their efforts. as well as to provide information to ensure longer term commitment of resources for the intervention. Research findings will not always be positive, but when there is scope for revising and adapting the intervention based on research findings, research can be an empowering process, to help implementers of an intervention contribute to its further development. Research can measure the effectiveness of the intervention and provide essential feedback about whether the programme achieves its aims. To do all this, it is not enough to take on research as an afterthought, only to be considered once a life skills programme is being implemented. It is important to begin research before the life skills programme is used, and before it is even developed. The review of research literature, described later in this chapter, reveals that research and practice in the life skills field have developed separately in many cases. Because of this, there are life skills programmes that have never been adequately evaluated. and on the other hand, there are numerous research studies which provide information on the need for life skills education which seem to have gone unnoticed. Much of the research on effectiveness of life skills education has focused on outcomes, such as reductions in smoking and alcohol use. The framework for life skills education research described in this document highlights the need to consider the effectiveness of the inputs into life skills education as well as its outcomes. This involves getting answers to questions such as: Is the programme well implemented? Does it get sufficient support? Only if the responses to such questions are positive, is it appropriate to ask the question: Does it work? Furthermore, there is a need to consider life skills education in terms of its effects on the school. on families and communities. on mental well-being and child development, as well as on the young person's health-related behaviour. The aim of this document is therefore to help bridge the gap between research and practice in the life skills field. and to encourage planners of research activities to take a broad view of the place of research in a life skills education initiative. Chapter 1 provides an introduction to life skills education and a brief review of research studies related to life skills education. Chapter 2 describes a framework for planning research. In Chapter 3. there are practical suggestions for time-lines for research as an integral part of life skills programme development. implementation and maintenance, and suggestions are made for estimating the costs of research. Appendix I describes some case studies of research related to life skills _ ~c SKli..LS E:CCATICN PLANNING;::-; -:'ESEARCH :::'Jqe 2 education in different countries. Appendix II describes some of the research instruments used in life skills programme evaluation. For more information and guidelines about life <;kills education development and implementation readers are referred to the following WHO materials: life Skills Education in Schools (Parts 1 and 2): Introduction and Guidelines to Facilitate the Development and Implementation of Life Skills Programmes life Skills Education in Schools (Part 3): Training Workshops for the Development and Implementation of Life Skills Programmes The Development and Dissemination of life Skills Education: an Overview Skills for life Newsletter To obtain copies of these materials contact WHO at the address below: Dr John Orley Programme Manager Programme on Mental Health Division of Mental Health and Prevention of Substance Abuse World Health Organization 1211 Geneva 27 Switzerland Who is this document for? In the above mentioned WHO document, life Skills Education in Schools. people who have responsibility for development of life skills education in schools are described as the "project managers" of a Life Skills Development Group. This document is for those project managers, to provide them with an overview of the research process and to help with estimations of the resources required for research. as well as to facilitate planning. The document will also be useful to the researchers employed to take on th·:- tasks of research: providing them with a multi-component framework within which they can p1 n research activities. The document targets project managers and researchers who are working on the development of life skills initiatives that are planned to have wide-scale impact in a country. The aim is to provide a framework for integrating research into such initiatives. Evidently, there will be many life skills initiatives with more modest goals. The basic research process described, and the research issues discussed in this document. can be interpreted and applied to guide research of all types of life skills initiatives: be it county level. province level or nation-wide. The setting for life skills education is normally the school: this document therefore refers to teachers as the principal implementers of life skills education. For readers whose work is in settings other than schools, "teachers" can be interpreted more broadly as animators or facilitators, and "schools". as any setting in which young people learn. Life skills education ::-oAFTEP 1 NTROCUCTiCN ::age 3 Life skills education differs in its objectives and contents from country to country and from one locality to another. However, across cultures. life skills education is similar in three important ways. First, at the heart of life skills education is the learning of life skills. Life skills have been defined as "abilities for adaptive a11.d positive behaviour. that enable individuals to deal effectively with the demands and challenges of everyday life" (WHO. 1993). Life skills. from this perspective. are essentially those abilities which help to promote mental well-being and competence in young people as they face the realities of life. Secondly, to enable children to learn and practice skills, life skills education is based on a child-centred and activity oriented methodology. And finally, life skills education is based on the philosophy that young people should be empowered to take more responsibility for their actions. The learning of life skills has been facilitated through the development of curricula for schools. The study of numerous interventions based on this approach has identified important areas of life skills that are being taught for the promotion of the health and well-being of children and adolescents. They are: decision making creative thinking communication self-awareness coping with emotions problem solving critical thinking interpersonal skills empathy coping with stress Life skills education is based on the teaching of this generic type of skills for life. and it includes the practice of life skills in relation to major health and social needs. In life skills education, children and adolescents are actively involved in a dynamic teaching and learning process that is often based on the principles of social learning theory (Bandura. 1977). Methods used to assist skills acquisition therefore include modelling of the use of skills. and practice of skills, primarily in classroom-based activities. The methods used include working in groups, brainstorming, role play, games and debates. The pedagogy of life skills education is therefore based on cooperative learning, participative activities and experiential learning. Life skills, when taught as generic skills for life, are taught in the context of holistic health and social issues, such as relationships, learning about social influences on behaviour and learning about rights and responsibilities, as well as being taught in the context of health problems. This is frequently the case, even when programmes have one principal objective. like drug abuse or violence prevention.. Other objectives for which life skills programmes have been developed include: suicide prevention. child abuse prevention. prevention of teenage pregnancy, AIDS/HIV prevention, peace education and the promotion of mental well-being. The life skills areas described above are taught to a greater or lesser extent in programmes with these different objectives. The skills are relevant to such wide-ranging objectives because they address the psychosocial factors that affect behaviours, which in turn contribute to health and social problems. Psychosocial factors that contribute to health and social problems include stress. poor coping skills, low self-esteem and low self-efficacy. . =: Si<!LLS :::UCATlrnJ ::1_<\NNING ~-:::R ~ES::A;.CH =·~qe .i Life skills educauon is. however. more than a set of classroom-based activities. ft is an empowering approach to health promouon and primary prevention. helping young people to take posmve actions to protect themselves and to promote health and posnive social relationships. To do so. life skills educanon neeas to be designed to create a supponive environment. where young people are exposed to positive roie models and where they are rewarded by peers. families and teachers for health promoting and pro-social actions. The contribution of life skills education to the prevention of health and social problems Health damaging behaviour and social problems are the result of a complex interaction between personal. social and environmental factors. However. even though environmental and social influences have an important effect on how a person behaves. behaviour stems essentially from individuals. The individuals needs therefore represent a focal point for life skills education approaches. Life skills approaches facilitate skills learning and promote social and environmental suppons that aim to influence health and social behaviour in a positive way. Figure 1 shows a hypothetical model for how life skills education equips individuals to behave in health giving and pro-social ways. through the promotion of mental well-being and by promoting "behavioural preparedness''. The latter construct builds primarily on two things: i) the individuals psychosocial competence (based on the acquisition and practice of life skills): and ii) behavioural intentions. The hypothesis is that to reach positive behaviour for health. promotion and prevention, the outer layers of the model. we must first have an effect on the inner and middle layers. To penetrate the layers. life skills education needs to be developed as a long term intervention. A shon term intervention. spanning several weeks. might be expected to have a shon term impact on mental well-being. A medium length intervention. spanning several months. might be expected to have an effect on mental well-being, as well as having some effect on skills and behavioural intentions. As suggested by the model, it is expected that only when the intervention is maintained for a longer term. spanning several years. to create longer term improvements in mental well-being and longer term impact on behavioural preparedness. can we expect to have a significant and lasting impact on health and social behaviour. Figure 1 : Model of the effects of life skills education i)1re.:1h)O ,,t Review of the research literature .::HAPTER 1 :NTROCUCT!ON ;:ages A review of research literature related to life skills education revealed the variety of research studies which have been carried out to provide information about life skills education. The research studies referred to here have been grouped to describe the different aspects of research related to life skills education. However, this is not intended to be a meta-analysis of matched research studies. The need for life skills education There is much research literature that indicates that life skills education is needed by young people and should therefore be developed. The literature points to deficits in life skills, and the need to teach life skills for abused children, for delinquent children, shy children, pregnant adolescents. for anger control. and for the prevention of anorexia and bulimia nervosa. Indeed, there is a great wealth of research that identifies the need for life skills in many areas of young people lives. A nationwide survey of nearly 47000 students in grades 6-12 in the USA identified assets and deficits in students· lives which influenced their ability to make positive choices. The results suggested that students had deficits in life skills. and the recommendations were that educators should work to enhance the social competencies of young people. including the teaching of friendship making skills. caring skills, assertiveness skills and resistance skills (Benson, unpublished). Numerous research papers cite the need for life skills to fulfil important needs for socialising young people in modem societies, as well as to fulfil the role of schools, one of the main domains of socialization of children and adolescents. According to Tippelt's studies of socialization, the process has become more complicated due to the inherent tensions and controversies of modern life (Tippelt. 1988). While in the past. children's development was modeled and monitored largely within families. the current influences of socialization. including the role of the media, requires that children and adolescents respond as individuals to contrasting values and competing pressures. One common conclusion from studies of socialization is that the didactic foundation of education. relying on lectures. is no longer suitable for rapidly changing societies. and that life skills education. using more active teaching methods. should be introduced in schools. The need for life skills is frequently emphasized in the findings of research studies set up to investigate what qualities and competencies employers look for in young recruits. Interpersonal and communication skills. two important life skills areas. are frequently reported in such studies, along with creativeness and negotiation skills (Pedersen, 1993; Rossouw. 1990; Hudecek, 1988; Karmel 1984). The need for life skills has also been identified in studies set up to investigate what promotes resilience in the face of adversity. Recent findings of an on-going international research project on resilience (coordinated by the Civitan International Research Center in the USA). found that life skills are an important component of resilience in older children, and suggests that there is a need to equip younger children with skills to promote resilience (Civitan. 1995). Two of the most frequently used skills for resilience were found to be communication and problem solving skills. Numerous other studies also point to life skills as important factors in resilience. including empathy .. ~E Si<ILLS cDUCATlON :::_.:.NNING ~.:;~· r.·ESEARC!-i >::geo ( Parker et al. 1990), seif-understanding ( Beardslee. 1989), communication ( Werner and Smith 1982, Luthar 1991) and problem solving (Shure 1991). The need for life skills education approaches is also frequently concluded from research which shows the psychosocial origins of behaviour-related health problems. ln one study. three important factors found to correlate with child substance abuse. were low self-esteem. inability to discuss feelings and lack of communication skills (MacDonald et al. 1991 ). Another study, based on questionnaire data obtained from 1.684 students in grades 7-12, showed that substance abuse is related to levels of stress ( Mitic et al, 1987). It was found that for both male and female subjects, regular inhalant users showed greater levels of stress than abstainers. In a study of the antecedents of cigarette smoking for 810 students. social conformity and peer influence were found to be mediators between stress factors and the (maladaptive) coping response factor of cigarette smoking (Castro et. al. 1987). Finally, Magoudi and Nastasi (1988) found that drug addiction in its early stages was associated with a failure to relate to peers and parents. Life skills approaches typically work to promote self-esteem and self-efficacy. The need for approaches of this nature. which emphasize psychosocial factors in health promotion and primary prevention, is highlighted by the large number of research studies which show the place of psychosocial factors as mediating or causal factors linked to behaviour-related health problems. The place of self-esteem and self-efficacy as mediating factors in behaviour-related health and social problems are frequently cited. Low self-esteem has been found to be associated with alcohol and drug use (Kumfer and Turner. 1990: Singh and Mustapha 1994), delinquency (Dukes and Lorch, 1989) teenage pregnancy ( Keddie, 1992: Plotnick, 1992) and suicidal thoughts ( Choquet et al. 1993). Research studies point to links between low self-efficacy and smoking, alcohol and drug use and abuse, anorexia and bulimia nervosa. risk taking behaviour and cognitive performance. In studies by Bandura ( 1990), the adoption of AIDS-preventive health behaviour has been found to be related to perceived self-efficacy. This finding was also found in another study of 423 Dutch adolescents. The results of the study showed that previous condom use was related to feelings of self-efficacy, and that self-efficacy had a strong effect on whether precautionary measures were taken. especially for girls (Richard and Van-der-Pligt, 1991). Self-efficacy has also been found to be associated with improvements in academic and cognitive performance. One study in India found that high perceived self-efficacy results in higher cognitive performance ( Singh. 1985). In Sri Lanka, a study of 832 tenth graders in 37 schools further demonstrated this relationship between a sense of competence and school achievement (Niles. 1986). It has also been suggested that student's perceived efficacy for self-regulated learning, influences their expectations for academic achievement (Zimmerman et al. 1992). The effectiveness of life skills programmes As described above, there is much research which suggests that life skills education should help to promote the health and well-being of children and adolescent<;. There is also research which points more directly to the effectiveness of life skills education approaches, described below. Effects on health and social behaviour ::-JAPTER 1 NTRODUCTICN Page i In experimental studies by Botvin et al. at Cornell University. the Life Skills Training programme ( teaching skills including communication, decision making, stress and anxiety management), was found to be effective in reducing smoking, alcohol and drug use <Botvin et al. 1980. 1984). The reduction of substance abuse following implementation of a programme based on life skills lessons was also found in a large scale national evaluation of the Teenage Health Teaching Modules (which includes teaching of communication skills, decision making, problems solving). Using a rigorous experimental design, 2530 subjects and 2530 controls were studied. The results demonstrated changes in students knowledge and attitudes related to substance abuse. Among senior high school student's, there was a decline in reported use of tobacco. alcohol and other drugs (Errecart et al. 1991). In a large study in Finland of the North Karelia Youth Project (involving 4, 523 young people) it was found that schools that implemented programmes based on life skills training, designed to prevent cigarette smoking and alcohol use. had an overall trend toward less smoking and alcohol use ( V artiainen et al, 1986). Violence and delinquency prevention is another major public health issue to which life skills approaches have been applied and studied. In one violence prevention programme 135 students were taught problem solving skills. It was found that, in comparison to control students, the programme participants were much less likely to define social problems in adversarial ways, were less likely to provide violent solutions in hypothetical conflict situations, listed more negative consequences to using violence and were less inclined to legitimise violence (Gainer et al. 1993). A six-year longitudinal study of the outcome of a primary prevention and life skills programme in elementary schools found that overall levels of pro-social behaviour were higher and negative, self- destructive behaviour decreased (Elias et al. 1991 ). A similar. positive effect on pro-social behaviour was found in a research study of the conflict resolution programme Working Towards Peace. Cases of aggressive misconduct were half as frequent and pro-social interactions five times higher for students of Working Towards Peace when compared to students who did not take part in the programme. It was also found that violent activities decreased by almost 70% for students who took part in the programme (Laird and Syropoulos. 1996). The Yale-New Haven Social Competence Promotion programme includes the teaching of core life skills such as stress-management, problem-solving, decision making and communication skills. Studies of this programme found that, relative to controls. students in the experimental group demonstrated improvements in their ability to plan ahead and choose effective solutions to problem situations that they face. Involvement with peers. as measured by self report and teacher ratings. increased as a function of the programme. Teacher ratings also indicated gains in self- control and sociability. Later studies of the programme. with an added domain specific focus on substance abuse prevention. suggest some preventive impact on self-reported substance use intentions and excessive alcohol use. The findings also report positive training effects on subjects' skills in handling interpersonal problems and coping with anxiety. Teacher ratings revealed improvements in subjects' constructive conflict resolution with peers. impulse control and popularity (Caplan et al. 1992). This research provides some evidence for the effectiveness of life skills programmes in achieving multiple promotion and prevention goals. ·_H SKILLS EDUCATION: PLA.NNING rCR RESEARCH ::age a Preliminary research findings also point to the effectiveness of a life skills programme for suicide prevention < Lafromboise. 1994). The Zuni Life Skills Development Curriculum for suicide prevention includes the teaching of communication skills. coping with oppression. anger and stress management and goal setting. The programme was found to contribute positively to the reduction of suicide potential. Assessment of suicide potential included questions about suicidal behaviour, suicidal risk factors and personal and social skills. Comparative studies Research studies have not only looked at the effectiveness of life skills approaches per se, but have compared life skills approaches to more traditional. typically information based approaches. This type of research has provided even more encouraging feedback about the effectiveness of the life skills approach. In Mexico. research findings suggest that a sex education programme based on life skills (including decision making, communication and self-awareness) was more effective in bringing about changes in adolescent contraceptive use than traditional sex education. that did not include a life skills component. The effectiveness of the programme was found for adolescents, especially boys, when they took part in the programme before their sexual debuts (Pick de Weiss et al, unpublished). In AIDS prevention, it was found that strategies promoting sexual abstinence or "just say no" appear less effective than those that involve the teaching of decision-making skills (Gold and Kelly, 1991). AIDS education, when coupled with life skills education for self-esteem and for gain ng social support, was successful in changing the knowledge, attitudes and behaviours necessary for preventing the spread of HIV (McLean. 1994). Reviews of studies of prevention education programme effectiveness, have indicated that prevention programmes based on the teaching of life skills are more effective than traditional approaches. For instance. Perry and Kelder ( 1992) found that comprehensive approaches to substance abuse prevention, incorporating life skills training for the promotion of social competencies, have had greater success in delaying the onset of alcohol and marijuana use than information and peer leader approaches. Similarly, following a review of effective approaches to smoking prevention, the learning of skills was considered to be a necessary component of effective programmes (Glynn, 1989). In a review of programmes to prevent adolescent pregnancy, the importance of life skills education as a component of adolescent pregnancy prevention was identified (Dryfoos. 1990). Similarly, Kirby et al, ( 1994) found that, in the USA, all effective programmes to reduce sexual risk behaviours devoted some time to the development of communication. negotiation and refusal skills. Skills acquisition and competence enhancement :;-;APTER 1 :NTROCUCTlC>N ;:age 9 The measurement of effectiveness of life skills programmes in terms of the acquisition of skills seems to be a logical strategy. It is however, somewhat surprisingly, not a frequently reported aspect of life skills programme evaluation. The reason for this is related to the relative complexity in measuring changes in life skills. However. there are some studies where this has been measured. directly by measuring skills performance or indirectly by assessing self perceptions of skills efficacy. In one large controlled study of the drug education programme DARE (Drug Abuse Resistance Education), the effectiveness of improving coping skills was investigated using data obtained from 678 fifth grade students. The results showed that there was a significant pre-/post-test improvement in coping skills for the experimental group as compared to a control group of students that did not take part in the programme (Aniskiewicz and Wysong, 1990). Similar positive results have been established following the teaching of problem solving skills (Tellado 1984; Caplan et al, 1992), coping with anxiety (Caplan et al. 1992) and communication skills (Pick de Weiss et al. unpublished). This latter study was an opinion survey of a representative sample of 1,795 students that took part in a life skills-based programme for the prevention of adolescent pregnancy. The results suggested that students had improved communication with their parents (87.9%). with their teachers (87.2%). and with their friends (90.1%) (Pick de Weiss et al. unpublished). When skills acquisition is studied. this is usually measured as one aspect of programme effectiveness. However. there are some interesting studies which have looked more closely at life skills, in order to understand better their role. For example. a study investigating the effects of greater self-awareness. found that it was correlated with being more honest. less aggressive and more altruistic (Wicklund. 1979). Tellado and Patton (1984) found that improved problem solving ability correlated with increased internal locus of control and self esteem. Interpersonal problem solving skills have been positively correlated with positive social interaction with peers. helpfulness and empathy (Shure, 1991 ). Effects on Mental Health Changes in mental health have been observed in studies where aspects of psychosocial well- being have been assessed. Following implementation of life skills based programmes in schools, research findings have shown significant improvements in self-image (Kreuter et al, 1991); self- esteem (Ennett et al, 1994); self-efficacy (Elias et al, 1991); social and emotional adjustment (Caplan et al, 1992) and a significant reduction in social anxiety (Botvin and Eng, 1982). However, many life skills programmes for which similar changes could be hypothesised have made no attempt to measure psychosocial factors. The impact on school performance Many studies in the life skills field have been set up to determine the impact of life skills programmes. although non-experimental in design. For example. small-scale and non-experimental studies of the use of the Lions-Quest Skills for Adolescence programme in 12 countries points to several areas of potential impact of life skills programmes, including; improvements in teacher and _.FE SKILLS EDUCATION ~·.M~NING ~::R RESEARCH =·'.lge n) student relationships. decreased absenteeism and improvements in student self-confidence. These studies also report teacher and student satisfaction with the programme ( Quest. unpublished). In an evaluation of Skills for Adolescence in the UK, the programme was associated with improvements in teacher-student relationships and teachers reports suggested that students showed better behaviour in the classroom l Parsons et al. 1988). Another study of the Skills for Adolescence programme reported increased attendance and increases in grade point average, as well as fewer detentions for misbehaviour among students that took part in the programme (Navarrette, 1987). One major study in this area was the assessment of the teaching of thinking in schools in Venezuela (based on the CoRT Thinking Programme). The study compared 900 students that took part in the thinking skills lessons to a matched control group of a similar size. Both groups received intelligence tests and tests of academic achievement at the beginning and end of the experiment. The results show that the experimental group had satisfactory significant increases in intelligence. Teachers changed in their attitudes towards the teaching process; displaying increased interest in their students and gaining in assertiveness. A subsequent larger trial of the Thinking Programme was carried out with :24000 students. A 10% sub-sample of this group took part in another study based on the original study protocol. Similar significant results were obtained for the experimental group (Gonzalez, 1990). Factors associated with effective life skills programmes Reviews of research on numerous school-based life skills programmes in the US have revealed factors associated with effective programmes (Consortium on the School Based Promotion of Social Competence, 1994). Key factors are summarised below. Effective programmes: • teach both generic life skills and specific skills for prevention of specific behaviours; • teach specific skills at appropriate developmental ooints; • have a peer leadership component for work with 1der adolescents; • have clear links with existing subject areas in tht 'ichool curriculum: • are based on active student involvement; • use clear, up-to-date and "user-friendly" materials; • are implemented long term; • are delivered by trained teachers. Effectiveness of dissemination and implementation of life skills education One major study of the dissemination of a life skills programme was the national evaluation of the dissemination and practice of the life skills programme Skills for the Primary School Child in the UK. The purpose of the evaluation was to provide an analysis of the materials, to determine the perceptions of the programme by teachers, parents and students. and to assess the impact of the programme on the classroom and on the culture and the school. Analysis of the materials assessed: ease of use: the appropriateness of context. language and concepts; the programme's potential to :;-;APTER 1 NTRODUCT:ON ?age 11 meet the needs of individual children: the links with the national curriculum: and the avoidance of racial and gender stereotyping. All of these areas were assessed on the basis of a questionnaire sent to a random sample of 230 schools across England and Wales. and on the basis of case studies in 15 schools. Apan from establishing the appropriateness of the content and successfulness of the dissemination. the main findings were that effective implementation of life skills education in schools depends upon ·whole school' personal and social health education policies. and relies heavily on the teacher responsible for coordinating the programme throughout the school. Another main finding of the study was that many teachers had changed their teaching style as a result of using the materials (Lloyd. 1994). Conclusions Research findings point to life skills education as a much needed part of education. Where life skills programmes are being implemented. research findings indicate that life skills education is an effective approach to health promotion and prevention education. The studies reported here reveal the variety of different indicators that have been used in investigations to determine life skills programme effectiveness. Process and impact evaluation studies have identified indicators of life skills programme effectiveness which are not normally associated with life skills programmes designed for prevention education objectives, like improvements in classroom behaviour. To determine the potential indicators of life skills programme effectiveness, there is a need for adequate conceptualisation of the potential of life skills education based on research findings, but also based on clear theoretical or conceptual foundations. Studies of life skills programme effectiveness typically concentrate on one aspect of programme effectiveness. Future research studies should be more comprehensive. Research of the effectiveness of a life skills programme should investigate dissemination and implementation processes, as well as impact and outcome variables. A framework for planning comprehensive approaches to research is described in Chapter 2 of this document. Finally, there is a need for more systematic evaluation of life skills programmes. There is a need for more research of life skills education in different countries and in different cultures around the world. This will be possible if research is planned as an integral part of the development and implementation of all major life skills initiatives. 5E SKILLS EDUCATION: PLANNING FOR RESEARCH ;:>age 12 References Aniskiewicz. R. and Wysong, E. ( 1990). Evaluating DARE: drug education and the multiple meanings of success. Policy Studies Review. 9(4) Summer. 727-747. Bandura, A. (1977). Social learning theory. Englewood Cliffs. NJ: Prentice Hall. Bandura, A. ( 1990). Perceived Self Efficacy in the Exercise of Control over AIDS Infection. Evaluation and Program Planning, 13( 1), 9-17. Beardslee, W.R. ( 1989). The role of self understanding in resilient individuals: the development of a perspective. American Journal of Orthopsychiatry, 59 (2), 266-278. Benson. P.L. (unpublished). The troubled journey: a profile of American youth. Search Institute, USA. Botvin. G.J .• Baker, E., Botvin, E.M .. Filazzola. A.O. and Millman. R.B. (1984). Alcohol abuse prevention through the development of personal and social competence: A pilot study. Journal of Studies on Alcohol. 45. 550-552. Botvin. G.J. and Eng, A. (1982). The efficacy of a multicomponent approach to the prevention of cigarette smoking. Preventive Medicine. 11, 199-211. Botvin, G.J., Eng, A. and Williams. C.L. ( 1980). Preventing the onset of cigarette smoking through life skills training. Preventive Medicine, 9, 135-143. Castro. F.G., Maddahian. E., Newcomb. M.D. and Bentler. P.M. ( 1987). A multivariate model of the determinants of cigarette smoking among adolescents. Journal of Health and Social Behavior, 28 (3), 273-289. Caplan: M., Weissberg, R.P., Grober •. S. and Jacoby, C. (1992). Social competence promotion with inner city and suburban young adolescents: effects on social adjustment and alcohol use. Journal of Consulting and Clinical Psychology, 60 (1), 56-63. Civitan International Research Centre (1995). Fifth action update. January, University of Alabama at Birmingham. USA. Choquet, M .• Kovess, V. and Poutignat. N. (1993). Suicidal thoughts among adolescents: an intercultural approach. Adolescence. 28(111) Fall. 649-659. Consortium on the school-based promotion of social competence ( 1994 ). The school-based promotion of social competence: theory, research, practice. and policy. In: Haggerty, R.J.. Sherrod, L.R .. Garmezy, N. and Rutter. M. (Eds.) Stress, risk, and resilience in children and adolescents: processes. mechanisms. and interventions. Cambridge University Press. ·:HAPTER 1 !NTRODUCT:CN Pcge 13 Dryfoos, J.G. (1990). Adolescents at risk: prevalence and prevention. New York: Oxford University Press. Dukes, R.L. and Lorch. B.D. ( 1989). Concept of self, mediating factors and adolescent deviance. Sociological Spectrum. 9(3) Fall. 301-319. Ennett. S.T .• Tobler, N.S .. Ringwalt, C.L. and Flewelling, R.L. (1994). How effective is drug abuse resistance education? A meta-analysis of Project DARE outcome evaluations. American Journal of Public Health, 84, (9), 1394-1401. Elias. M.J. (1991). An action research approach to evaluating the impact of a social decision-making and problem-solving curriculum for preventing behavior and academic dysfunction in children. Evaluation and Program Planning, 14(4), 397-401. Elias, M.J, Gara. M.A .• Schuyler. T.F., Branden-Muller. L.R. and Sayette. M.A. (1991). The promotion of social competence: longitudinal study of a preventive school-based program. American Journal of Orthopsychiatry, 61(3), 409-417. Errecart, M.T .• Walberg, H.J., Ross, J.G., Gold, R.S .• Fielder, J.F. and Kolbe, L.J. (1991). Effectiveness of Teenage Health Teaching Modules. Journal of School Health. 61(1), 19-42. Gainer, P.S., Webster, D.W. and Champion, H.R. ( 1993). A youth violence prevention program. Archives of Surgery, 128(3), 303-308. Glynn, T.J. (1989). Essential elements of school-based smoking prevention programs. Journal of School Health, 59(5), 181-189. Gold. R.S. and Kelly, M.A. (1991 ). Cultural sensitivity in AIDS education: a misunderstood concept. In Kelly, M.A. Evaluation and Program Planning, 14(4), 221-231. Gonzalez. R. ( 1990). Ministering intelligence: A Venezuelan experience in the promotion of cognitive abilities. International Journal of Mental Health. 18(3), 5-19. Hudecek, J. ( 1988). Youth: changing labor perspectives; Mladez-prace a Jeji Promeny Sociologicky Casopis, 24(6), 590-596. Karmel, P. (1984). Education and training for work and living. Australian Journal of Education, 28(2), 190-201. Keddie. A.M. (1992). Psychosocial factors associated with teenage pregnancy in Jamaica. Adolescence, 27(108) Winter, 873-890. Kirby, D .• Short. L., Collins. J.. Rugg, D .. Kolbe, L., Howard, M .• Miller, B .• Sonenstein, F. and Zabin, L.S. (1994). School-based programs to reduce sexual risk behaviours: a review of effectiveness. Journal of Public Health Reports, 19. 339-360. . rE Si<ILLS EDIXATION J::~ANNING r·)R RESi:ARCH =r::ge 1..i Kreuter. K.J .. Gewirtz. H .. Davenny. J.E. and Love. C. (1991 ). Drug and alcohol prevention project for sixth graders: first year findings. Adolescence, Summer 26(102), 287-293. Kumfer. K.L. and Turner. C.W. (1990-1991). The social ecological model of adolescent substance abuse: implications for prevention. International Journal of the Addictions. 25(4), 435-463. Lafromboise, T.D. and Howard-Pitney, B. (1994). The Zuni life skills development curriculum: a collaborative approach to curriculum development. American Indian Alaska Native Mental Health Research Monograph Series. 4. 98-121. Laird. M. and Syropoulos, M. ( 1996). Aggression and violence: the challenge for Detroit schools. Unpublished report. Lloyd. J. (1994) Skills for the Primary School Child: a national evaluation of dissemination and practice. Department of Education. London. Lortie. L., Fellers. G.L. and Kleinplatz. P.J. (1986). Value orientations of English, French, and Italian Canadian children: continuity of the ethnic mosaic? Journal of Cross Cultural Psychology, 17(3), 283-299. Luthar, S. ( 1991 ). Vulnerability and Resilience: A study of high-risk adolescence. Child Development, 62, 600-616. MacDonald L., Bradish, D.C., Billingham, S., Dibble, N. and Rice, C. ( 1991 ). Families and schools together: an innovative substance abuse prevention program. Social Work in Education, (13)2, 118- 128. Magoudi. A. and Nastasi. A. ( 1988). Identity failure: heroin addiction in pre-adolescents: (Echec de l' identite: toxicomanie a !'heroine chez les preadolescents.) Neuropsychiatrie de /' enfance et de r adolescence, 36(2-3), 69-73. McLean. D.A. (1994). A model for HIV risk reduction and prevention among African-American college students. Journal of American College Health, 42(5), 220-223. Mitic, W.R., McGuire, D.P. and Neumann, B. ( 1987). Adolescent inhalant use and perceived stress. Journal of Drug Education, 17(2), 113-121. Navarrette. C. ( 1987). Interim evaluation of Somos El Futuro project. Portland. OR: Interface Consultants. Niles. F. S. ( 1986). Socioeconomic Status, attitudes and academic achievement in a developing society. Journal of Developing Areas. 20(4) July, 491-499. Parker, G.R., Cowan, E.L.. Work, W.C. and Wyman, P.A. (1990). Test correlates of stress resilience among urban school children. Journal of Primary Prevention, 11(1), 19-35. -:::-:APTER 1 !WRODUCT!ON ?age 15 Parsons. C .• Hunter. D. and Warne. Y. (1988). Skills for Adolescence: An analysis of project material. training and implementation. Christ Church College, Evaluation Unit, Canterbury, UK. Pedersen. A. ( 1993). Qualities of the excellent health nurse. Nursing Admi11istration Quarterly. 18( l ). 40-50. Perry, C.L. and Kelder, S.H. (1992). Models of effective prevention. Journal of Adolescent Health. 13(5), 355-363. Pick de Weiss, S., Givaudan, M. and Cohen, S. (unpublished paper). Planeando Tu Vida: sex and family life education: fundamentals of development. implementation and evaluation. Plotnick, R.D. ( 1992). The effects of attitudes on teenage premarital pregnancy and its resolution. American Sociological Review, 57(6) Dec., 800-811. Poole, M.E. and Evans, G.T. (1989). Adolescents' self perceptions of competence in life skill areas. Journal of Youth and Adolescence. 18(2), 147-173. Quest (unpublished). Quest International, International Division, Summaries of evaluations of Lions-Quest Skills for Adolescence. Unpublished repon. Richard. R. and Van-der-Pligt, J. (1991). Factors affecting condom use among adolescents. Journal of Community and Applied Psychology, l (2), 105-116. Rosenstock. I.M .• Victor. J. and Becker, M.H. (1988). Social learning theory and the health belief model. Health Education Quarterly, 15 Summer, 175-183. Rossouw, DJ. (1990) What industry demands of science educators: A survey. Onderwys Education Bulletin, 34(1 ), 3-11. Schinke, S. P., Shilling, R.F .• Banh, R.P .. Gilchrist. L.D. and Maxwell. J.S. (1986). Stress management intervention to prevent family violence. Journal of Family Violence. 1(1), 13-26. Singh, S. (1985). Perceived self-efficacy and intellectual performance of socially disadvantaged students. Journal of Social Psychology, 125(2), 267-268. Singh, H. and Mustapha. N. (1994). Some factors associated with substance abuse among secondary school students in T~inidad and Tobago. Journal of Drug Education, 24(1 ), 83-93. Shure. M.B. (1991). Resilience as a problem solving skill. Paper presented at the Fostering Resilience Conference. Washington, DC. IMHI. Tellado, G.S. (1984). An Evaluation Case: The implementation and evaluation of a problem solving training program for adolescents. Evaluatio11 and Program Plan11i11g, 7(2),179-188. JFE SKILLS i:DUCATION: PLANNING FOR RESEARCH ~age 16 Tippelt, R. ( 1988). Children and adolescents in the conflict between family and other socialization domains: (Kinder und Jugendliche im Spannungsfeld zwischen der Familie und anderen Sozialisationsinstanzen) Zeitschrift fi'ir PiidaRogik, 34(5), 621-640. Vartiainen. E .• Puska. P .. Tossavainen. K .. Viri, L.. Niskanen. E .• Moisio. S .• McAlister, A. and Pallonen, U. ( 1986). Prevention of non-communicable diseases: risk factors in youth: the North Karelia youth project (1984-88). Health Promotion, 1(3), 269-283. Werner. E.E. and Smith, R.S. (1992). Overcoming the odds: high risk children from birth to adulthood. Ithaca: Cornell University Press. Wicklund, R.A. (1979). The influence of self-awareness on human behavior. American Scientist, 67(2), 187-193 World Health Organization (1993). Life skills education in schools (WHO/MNH/PSF/93.A Rev. 1), WHO, Geneva. Zimmerman, BJ., Bandura, A. and Martinez, P.M. ( 1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29(3), 663-676. CHAPTER 2: THE RESEARCH PROCESS A framework for planning research ::HAPTER 2· Tf-:E RESEARCH PROCES.S ;::age 17 There is a logical framework for research as an integral part of life skills education development. implementation and maintenance. In this framework the different aspects of research are viewed as part of an overall process. This is outlined in Figure 2. below. From this perspective. each phase of research is informed by the research which precedes it. In this way, by the time a life skills programme is implemented and is to be evaluated. there will already be much information available to help determine how success is to be measured. Figure 2: Overview of the research process Development Phase i) Situation analysis - including a needs assessment and review of the resources - to guide programme objectives and strategies ii) Formative research - to guide the design of programme materials and training iii) Pilot testing - of draft life skills programme materials and training _J, Implementation Phase iv) Monitoring dissemination of life skills programme materials and training v) Short term evaluation - including wide-scale feedback about the life skills programme implementation. and process/outcome evaluation studies in a representative sample of schools _J, Maintenance Phase vi) 2nd Situation analysis - including a review of the dissemination and effectiveness of life skills education and second needs assessment vii) Long term eyaluation - including follow up of the process/outcome evaluation studies These elements of the research process are described in more detail in the remainder of this chapter. Although each area of research addresses quite different research questions. there are a set of guiding principles which are common to them all. as suggested below. .,FE SKILLS EJUCATION· c:;!_A.NNING ;:,:·R RESEARCH ='Jge ,a Guiding principles for research1 There are some basic principles to bear in mind when planning research: • It should be action-oriented. ie. aimed at developing solutions. • An integrated multidisciplinary approach is required. • The research should be participatory in nature, involving all parties concerned (from policy makers to community members) in all stages of the project. • Studies should be scheduled in such a way that results will be available when needed for key decisions. Otherwise, the research loses its purpose, ie. research must be timely. • Emphasis should be placed on relatively simple research designs that are likely to yield practical results. • The research should be cost-effective. What is the cheapest way to answer the research questions? Adapted from Varkev1sser. C.M. Pathmanathan. I. and Brownlee. A. (1991) Desigmng and Conducting Health System Research ProJects. Health System Research Traming Series. Vol 2 (l). Intemauonal Development Research Centre. Canada. Situation analysis -1 .::HAPTER 2 :HE RESEARCH PROCES.S Page 19 A situation analysis can be designed to determine whether there is a need for life skills education. and. if it is to be developed. what should its objectives and contents be. and who should be the target group? The situation analysis is therefore not envisaged as a complete situation analysis of the needs and problems of a country. It is primarily to guide objectives and strategies for life skills education development and implementation. And, in the first instance, the situation analysis is to determine whether the life skills approach is an appropriate and timely one in the country concerned. In a situation analysis, questions can be asked about the priority problems and needs of young people, the best way to respond to these. and the potential contribution of life skills education. Such research questions are described here in the context of a needs assessment. The next area of investigation during a situation analysis is based on a review of resources (described later in this section), to determine what is already available and what should be done to respond to the findings of the needs assessment. Needs Assessment Key questions asked during a needs assessment include: • What are the priority problems and what causes them? • What are the priority needs and how can they be addressed? Life skills education can be applied to health and social problems that are related to behaviour and mental well-being. Review of epidemiological data and other statistics of health and social problems which are considered to be related to behaviour and mental health would therefore be useful. Relevant data is likely to include the prevalence of: • AIDS/HIV and other sexually transmitted diseases; • teenage pregnancy rates; • tobacco. alcohol and drug use/abuse; • suicide and suicide attempts; • school violence; • school drop-out and absenteeism. Review of available and relevant epidemiological data will help to identify priority problem areas to be addressed by life skills education. Where this data is not available, basic research may be set up to gather such data. This is, however, unlikely to be manageable in the short term, and is not an essential part of a situation analysis. The needs assessment should ask probing questions: eg. What causes these problems and needs, and why? This type of question can partly be investigated by a review of research findings. Relevant research would include studies which test out theories of the link between behaviour and mental well-being and health. For example, the importance of psychosocial factors. like stress, in alcohol and drug use and abuse has been confirmed in research studies. Where this type of _.FE SKILLS EDUCATION. FLANNING FC-R RESEARCH '<1ge 20 research. concerning the underlying factors which contribute to health and social problems. is not available in the country in question. there may be a need to set up such studies as another area of basic research. Again. this is unlikely to contribute in the short term to the situation analysis and. in this case. research findings from other countries may help to clarify the causes of health and social problems ( see Chapter l ). To answer questions about needs, problems. causes and possible solutions. it is essential to ask those who are most closely involved, ie. curriculum developers. teachers, parents and pupils. Various methods can be used. including questionnaires, individual interviews and group interviews. A combination of these methods may be used to gain information from different sources. Questionnaires can be distributed by post and so have scope for reaching large numbers of people in different parts of the country (ie. a survey). Surveys can provide quantifiable data. which is useful when comparison across populations is required. If relevant questions can be incorporated into opinion polls/sample surveys (eg. Gallup and Harris polls of public opinion) this can provide an excellent. highly respected source of data on which to construct a life skills education initiative. The very scale of the survey lends credibility to the research findings. However. like the collection of epidemiological data. setting up nation-wide surveys is unlikely to be an essential part of a needs assessment and it may be more appropriate to describe this as another suggestion for basic research that could be set up at this stage. Questionnaires are targeted at individuals and are most useful if the subject matter is clearly defined and familiar. However. this is not always the case with issues related to life skills education, since there may be a need to raise questions related to unfamiliar concepts and approaches. For this the use of interviews may be more appropriate. In particular, the group interview is recommended. It provides an opportunity to pose the same types of questions to different groups, eg. parents. pupils. or teachers, and to compare and analyze the results. Characteristics of group interviews. known as focus groups, are described below. Characteristics of focus groups2 • Focus groups allow for in-depth discussion of people's perceptions about health and social problems and needs, and about any other defined subject area. • An optimal number to include in focus groups is approximately 6-8 people. • Focus group participants are selected so that the participants are all part of a similar group, eg. they are all primary school children. or they are all secondary school teachers etc., and, to encourage group cohesion, it can help to bring together groups of a similar age, social class and culture. • The participants do not know each other prior to the focus group interview. • The focus group facilitator uses a list of open-ended questions to structure the focus group discussion. The research questions for which we hope focus groups will provide some answers include: 2 Adapted from: Hudelson. P.M. (1994). Qualitative Research for Health Programmes. World Health Organization. unpublished document WHO/MNH/PSF/94.3. Geneva. What are youn~ peopie's main concerns? :::1.A.PTER 2 ::-:E RESEARCH PROCESS ;::1Jge 21 What are youn~ people's perceptions of the problems they face and their most important areas of need? What problems do young peoplelparentslleachers perceive as most threatening? Are there any differences m perceprions of problems for boys and girls? Whar are the youn~ people' s1parents'/1eachers· l'iews of the causes and solutions to the problems that young people experience? Analysis of focus group discussion should help to identify risk factors and concerns which could be addressed in life skills education. Focus groups help to identify the main needs and problems of young people of different age groups and from different perspectives, ie. the perceptions of parents, teachers and pupils. To assist interpretation of the findings, focus group discussions can be compared across the different groups involved, eg. teachers' perceptions of the needs and priorities can be compared with those of parents. Review of resources The needs assessment identifies the priority problems and clarifies what causes the problems; it identifies the priority needs of young people and how these can be addressed. The next step is to review the resources available for responding to the identified needs and problems. Resources to be reviewed at this stage include: • existing materials used in schools, and in other settings, to promote life skills development; • personnel to support the life skills education initiative; • opportunities for teacher training; • the potential resources available within schools to support the implementation of life skills education; • the potential for political and financial support for life skills education. Research questions at this stage include: • What life skills education programmes are available (in this country and elsewhere)? • What interventions are effective? • What lessons have been learnt from experience in this approach in this country and elsewhere? • What would be the contribution of life skills education to addressing young peoples needs? What types of interventions would be required to complement life skills education? A review of $e life skills programmes currently used in schools helps to identify the gaps in provision, but it also helps to identify where there is duplication of efforts. For example, life skills education approaches. based on similar content. have been used for various objectives, including: suicide prevention, AIDS prevention. drug abuse prevention and for mental health promotion. A review of resources therefore is an opportunity to monitor efficiency in prevention education and health promotion programmes in schools. There would also be various questions to ask regarding the potential for political commitment and support for life skills education at the local and national level. including: .. : ::qe L2 Are their ~overnment poiiczes ro support primary prevenrzon and heaith promotion in schools? Are there scnooi level ooiicies {or the promotion oj the psychosocwi development of children? What are the i1J..:eiy ohstacies ar the national and local lel'el? How 1s life sJ..:11/s educatzon w be junded? Getting answers to research questions at this stage will depend upon access to infonnation on the availability and effectiveness of life skills related materials within a country and from elsewhere. It will also depend upon having access to information about government and local level school policies and resources. Interpreting and using situation analysis findings Raw research findings provide so much information. sometimes contradictory, that it can be confusing and misleading. Analysis of research findings helps to distil a large array of raw infonnation down to what is needed to answer research questions. Analysis of data may involve statistics and people working on life skills education research would be expected to have. or have access to. expertise in statistical analysis of research data. The situation analysis can generate a wealth of information with. for example, responses from policy makers. teachers. parents and pupils on the same issues. Interpretation of the research findings will be required if we are to use the findings to guide future developments in life skills education. This is illustrated in Figure 3. Figure 3: Interpretation of situation analysis research findings X=re!<Cal'Ch •Jata x eg ep1dem1olo~1cal ,1.ita (~ c111ldren's r~rceplton, 0fne..-d< x cg. resources available Multiple data sources analyzed and results combined to make conclusaons x ~!! pohc1es re. school cumculum ~ ,, bhfoskoll,...,rn,~/ \ appropnate7 / \ ( \ foke cond11Sions/dcdsions aboul h(c skill< educ:i11on Formative Research :::HAPTER 2 ,HE RESEARCH PROCESS Page 23 If the situation analysis phase points clearly to the need for a life skills education approach. the needs assessment should have helped identify the main problems and needs to be addressed in life skills education. and the review of resources will have identified the most appropriate strategy for its development and implementation. With this information to hand. it is now possible to begin a phase of formative research, which is the more detailed study required to determine the contents of life skills education and the teacher training, as well as strategies for their implementation. Formative research is therefore an integral part of the strategy for developing the life skills programme and training. It is a strategy which ensures, as far as possible. that the programme and training developed address the needs and problems in the most appropriate way. Below are examples of the type of questions that can be asked at this stage. Questions to consider during the formative research phase • What are the main objectives for the life skills programme? • What are the overall expectations? • What is the target age group for life skills education in schools? • Over how many years is life skills education to be made available? • What life skills are to be taught, in relation to which issues, and applied to which needs and problems? • Is the programme to be taught as a new subject in the school curriculum? Or. if integrated. in what subjects will it be taught? • Is training of teachers/animators necessary? If so. what is the content of teacher training? • What kind of training is required eg training ·workshops, training rideos or written materials? • How is this training to be done and by whom? • What are the objectives of training? • What is the current situation in schools? • What are the barriers to this intervention, how can the programme and training be designed to overcome the barriers? • What theories/models of health and social behaviour help to define the conceptual basis for the design of the life skills programme? More specific research questions • For teache~ • What pedagogic methods do they use in the classroom? • Are teachers familiar with active learning methods? • What resources are currently available to assist teachers to promote the positive socialisation of young people? How much time is currently spent on life skills education activities? • How much time could they envisage devoting to a life skills curriculum? • What kinds of conflicts arise in the classroom - between pupils and between pupils and teachers? • What do they perceive as the student's main problems and needs? _'FE SKILLS EDUCATION· PLA.NNING FOR RESEARCH ?age 24 • What skills do their students need to deal with these problems? What is their role/the school's role in addressing these needs? • Should the school curriculum address such issues as.for example. sexuality, sexual abuse. drugs. violence. suicide and AIDS? • For parents • What are their perceptions of children· sf adolescent's needs/ problems? • What skills do their children need to deal with these needs/problems? • How can the school help young people to cope bener with life? • What are the areas of conflict between them and their children? • What are their needs in support of their child's healthy development? • Do they think that the school curriculum should address such issues as.for example. sexuality, sexual abuse. drugs. violence. suicide and AIDS? • For young people • Do they feel that they are well-integrated at school? • Are their conflicts between them and their parents/teachers and peers? • How do they describe their main needs and problems? • What skills do they need to deal with these needs/problems? • Do they think that the school curriculum should cover issues of sexuality/AIDS/substance abuse/relationships among students/sexual abuse/violence and suicide? Research methods The most common research methods used to obtain answers to research questions are surveys/questionnaires and individual and group interviews. There are research questions which cannot easily be answered using questionnaires and interviews. such as: What is the current situation or atmosphere in schools? This kind of question may deserve observation. To gain a realistic impression of the situation in schools, it may be useful to set up observation in a random sample of schools in the country. Observation can be structured or unstructured. To gain a descriptive account of the situation in schools, without lengthy periods of research and without lengthy training of observers, unstructured observation is likely to be the best approach. The observer would simply be instructed to observe broad characteristics of the school life and should be looking for characteristics of the schools which are associated with the health and development of young people. Like the research done during a situation analysis. the objective of formative research is to gain in-depth knowledge about key issues, and to draw this knowledge from different perspectives. The questions being investigated should therefore be phrased as open questions, as far as possible. Such questions require more than a yes/no answer. Open responses to questions provide qualitative information. that can be a very rich source of information to guide programme development. However. interpretation of qualitative information can be ambiguous and it may be difficult to make definite conclusions based on qualitative responses alone. Questionnaires and interviews are likely ·.::HAPTER 2 :HE RESEARCH PROCESS ;::age 25 therefore to include a mix of open questions and closed. forced-choice questions, which are quantifiable. Formative research for programme and training development is unlikely to involve large scale studies. However. to be sure of meaningful results. attempts should be made to ensure that the groups studied are representative of the larger population for which the life skills programme is intended. For a sample to be representative it should have almost all of the characteristics of the whole population. For example. a country will have a certain percentage of rural and urban schools and may also have military schools. religious schools and schools with all female or all male students. A study based on a representative sample of schools would aim to include an equivalent proportion of the different types of schools. Formative research is followed by a period of programme development and, where training is to be set up, the design of training courses. Before this work is concluded. a final element of formative research may be carried out. that is the pre-testing of draft materials, before an official pilot test version of the materials are produced. Pre-testing materials and training Even before pilot testing a whole programme, a pre-test of programme materials with a small number of future "users" ie. teachers, children and/or adolescents, could help to test out age- appropriateness and the style and language of the lessons. For example. children and adolescents could be asked to choose words in the text which they do not understand and suggest alternatives. Similarly, they could be asked to look at the scenarios or activities described in the lessons, to choose those which they do not understand and suggest alternatives. Other simpler issues. like the size and style of the text and pictures could also be assessed at this stage. To pre-test the teacher training, a short training course could be carried out with a small group of teachers. even before the contents of the training are officially written up. The teachers in the pre-test would be asked to provide feedback on the style of training and the content of the course. The teachers would be asked to suggest improvements to the training, to make the activities more appropriate and easier to understand. The pre-test of life skills programme materials and training should also include review of a draft version of the materials by the ad visors to the life skills initiative, for instance the Life Skills Advisory Panel, as described in the WHO document Life Skills Education in Schools (see Introduction). Questions to ask of this group include: • Are the materials age and culturally appropriate? • Are the materials balanced in terms of gender. ethnic and social groups? • Are the materials suitable/adaptable for different ethniclreligiouslsocial groups in the target population? • Does the programme cover the key issues? _ r:E SKILLS EDUCATION· PLANNING FOR RESEARCH iJ()ge 26 Pilot test A pilot test is first and foremost an opportunity to examine the feasibility and acceptability of the life skills education programme being developed and the teacher training. It is also an opportunity to assess the potenual effectiveness of the intervention. The pilot test phase is an opportunity to assess the effectiveness of the life skills intervention on three levels. in terms of: i) the effectiveness of life skills programme implementation; ii) the outcomes of the life skills programme; iii) the success of efforts to gather support for the life skills initiative. The pilot test can evaluate the intervention in these ways. It also is a chance to learn how to evaluate these areas. The pilot test therefore sets the agenda for future research. The pilot test phase provides an opportunity to set up a life skills initiative on a small scale, to run training sessions and use the programme materials over a limited period of time (eg. up to one year) with a representative sample of the future users. During the pilot test phase, it is possible to gain feedback from multiple sources, which will help to improve the appropriateness and effectiveness of the intervention before it is widely implemented. The pilot test should identify elements that are missing from the initiative, the materials and training, and identify what needs to be changed. The pilot test phase is also an opportunity to clarify the way in which the effectiveness of the intervention is to be measured and clarifies the basis on which one can say that intervention has been successful, or not In the case of life skills education. this is not a straightforward issue, unlike a vaccination campaign where success can be measured by the number of people vaccinated. For example, a life skills programme set up to prevent AIDS might not show significant changes in the spread of AIDS, although there may be evidence of improvements in the relationships between pupils. If. for example, there are improvements in school attendance and pupils' academic performance, or better communication with parents about sexuality, should we say that the programme has "failed"? In this case, following a pilot test. it might be helpful to look again at the stated programme objectives, to give more emphasis to improvements in school variables. The pilot test therefore offers an opportunity to re- examine programme objectives in relation to programme outcomes. There are two other options: the programme could be revised at this stage to have more impact on HIV I AIDS related practices, and the research may need to be refined to assess in more detail positive changes in HIV I AIDS related practices. This phase would be followed by a review of the way that the life skills initiative was set up and revision of the life skills programme materials and training. It is also an opportunity for review and revision of the pilot test design and the research tools used. The pilot test phase should also clarify whether it is possible to implement the intervention wide-scale, and how much is it likely to cost. Evidently, there is a great deal to be learnt from the pilot test phase. However, it is not necessary to produce the perfect product at this stage. And there is little to be gained from lengthy pilot test projects. spanning many years. if the objective is to reach as many children as possible. ::HAPTER 2 :f-'E RESEARCH PROCESS Fage 27 If life skills education is set up in such a way that periodic review and revision of the materials and training is possible. even once it is implemented wide-scale. there will be further opportunities for improving the intervention over time. Pilot Testing the Life Skills Programme Materials There are two types of questions about the impact and effectiveness of a life skills programme to be investigated during the pilot test. There are process evaluation questions about the implementation of a life skills programme: eg. Do users understand and enjoy using the materials? On the other hand, there are outcome evaluation questions about the impact and effectiveness of the programme: eg. Does the programme have an effect on young people's self- esteem and their perceptions of competence to deal with peer pressure? The pilot study of the life skills programme should include both process and outcome evaluation. The more widespread the life skills programme materials are to be implemented, the more important it is to incorporate outcome evaluation, as well as process evaluation, in the pilot test phase. In this way, changes required to improve programme effectiveness can be incorporated before the materials are distributed wide-scale, and appropriate and well-founded claims can be made for what one can expect to achieve by implementing the programme. Frequently, pilot testing of life skills programmes has focused on one or the other. This results of such studies are of limited value. A programme may score high in terms of process evaluation, if it is well-implemented and well-received. even though the actual changes it brings about are negligible. On the other hand. a well-designed programme which should give rise to significant changes may be weakened if it is not well-implemented and subsequently, not well- received by the young people. Questions for process evaluation When a life skills programme is implemented in a pilot test, information about its use is sought from school administrators, teachers, students, parents and from community members. • Questions for teachers • Are teachers satisfied with the programme? - Are they enthusiastic about the programme? • Is the programme compatible with the values held by the teachers? • Do teachers find the lessons useful/interesting? • Are the materials age and culturally appropriate? • Do teachers find the materials easy to use? • What do teachers think of the presentation of the materials? • Does the programme cover important issues? • What other areas could have been included? • What works best in the programme? • What does not seem to work in the programme? • How often was the programme taught? • Which lessons were used? • Which lessons were not used? • Is the programme implemented in the appropriate place in the school curriculum? ~:FE SKILLS EDUCATION: PLA.NNING FCR RESEARCH ?age :28 • Does the programme ~et support from other school staff and from the school administration? • Is the programme being used with the suggested target group? • Questions for parents • Are parents satisfied with the programme? Are they enthusiastic about the programme? • Is the programme compatible with the values held by the parents? • Has the pupil talked about the programme with his/her parents? • Have the parents been involved in making decisions about life skills education? • Questions for pupils • Are pupils satisfied with the programme? - Are they enthusiastic about the programme? • Do pupils find the lessons useful/interesting? • What do pupils think of the presentation of the materials? (If a student book is used) • Do pupils understand the lessons? • Does the programme cover important issues? • What other areas could have been included? • Questions for school administrators • Is the programme compatible with the school ethos? • Is the programme implemented in the appropriate place in the school curriculum"! • Does the programme get support from other school staff and from the school administration? • Questions for community members • Is the programme compatible with the values held by the community? • To what extent have community members been involved ·with decision making for life skills education? Methods for process evaluation Most of these research questions can be put directly to the people concerned, using individual/group interviews, or indirectly using questionnaires. In each case, during a pilot test it can be useful to include open questions. and to make provision for respondents to expand upon their answers with an explanation. There are some questions for process evaluation which cannot be answered using interviews and questionnaires alone. For example: Do teachers use the materials as instructed? To assess the implementation of the programme. there may be a need for structured observation of the teacher's use of the materials with a group of children. Structured observation typically follows an obsen,ati.on schedule. which guides the observer's interpretation of the teacher's actions and reactions. The observation schedule or teacher rating form ( based on minimum criteria for teacher fidelity to the programme contents and methods) would be designed to determine if the teacher follows the lesson plans. ,-::HAPTER 2 THE RESEARCH PROCESS Page 29 It may be the case that teachers are not obliged to adhere strictly to the programme content. for instance if the materials are designed simply as a guide for the teacher to adapt. However. information should be collected during pilot testing about how the programme is used. not just as feedback about the use of programme materials per se. but also to assist interpretation of the findings of the outcome evaluation. Choosing outcome variables Outcomes frequently associated with life skills education approaches include the following: • acquisition of knowledge • attitude changes • changes in behavioural intentions in relation to health-related behaviour • improvement in self-esteem and confidence (and other indicators of mental well-being) • skills acquisition and competence enhancement • improvements in school related variables eg. school climate (eg. improved student relationships with peers and teachers) classroom behaviour academic performance • changes in health and social behaviour eg. reduced drug use less violence condom use Other educational outcomes which are not frequently studied, but which are worth investigating, are the potential benefits for teachers. such as: stress reduction, reduction of teacher "burnout" (due to nervous exhaustion), and improvement in teacher·s own life skills. Additional benefits for students also worth investigating include improvements in communication and relationships with parents. Figure 1, in Chapter 1, showed how the different outcomes of life skills education are inter- connected and to some extent depend on each other. The model suggests that the effects of life skills education in the shorter term are likely to be strongest for mental well-being. Only in the longer term can we expect changes in health and social behaviour. This is shown in more detail in Figure 4. _.FE SKILLS EDUCATION: PLANNING FOR RESEARCH ::aqe 30 Figure 4: Model of the variable effects of life skills education over time Implementation of life skills cducauon ' Short term outcomes Middle range outcomes Long term outcomes I Booster _..... (!) I sessions ........._ I (I I I ' -knowledge acquisition -attitude change -improved mental well-being -positive behavioural intensions -improved psychosocial competence/life skills -improved classroom climate -improved classroom behaviour -improved relationships between pupils. pupils and teachers -improved school attendance -improved academic achievement -significant changes in health and social behaviour -maintenance of changes over time This model of the variable effects of life skills education over time should help to determine what kind of outcomes can be expected. If a programme has only been in place for three months when we want to carry out an evaluation of its outcomes, as is likely to be the case during the pilot test phase, it would not be appropriate to look for changes in behaviour in order to judge the effectiveness of the programme. It would be more appropriate to assess outcomes at the level of knowledge, attitudes and mental well-being. Evaluation carried out after a one year period of implementation could assess changes at these levels and could also investigate changes related to school performance and behavioural preparedness. Changes in health and social behaviour, however, would the reserved as indicators of effectiveness of life skills education in the longer term, for example, following 2-3 years of implementation. Measuring outcomes The measurement of effectiveness of life skills programmes in terms of outcomes is typically done based on data obtained from pencil and paper self-report questionnaires administered before and after the period of (quasi-)experimental study of the intervention ie. as a pre-test and post-test measurement. In most cases. this is the only suitable way of obtaining data from the large numbers of subjects required to render the study findings meaningful. ::-:APTE~ 2. ·;.;E RESEARCH PROCESS ?age Ji Measurinrt improvements in student's health knowledge. What do they know about the health issues discussed in the 1:ontext of the life skills programme'? This would typically be done by asking questions. orally or written. about their health knowledge. as for any academic examination of knowledge . . \1easurmf? changes in attitudes towards health and social issues. This would relate to the attitudes explored in the life skills programme. Attitudes can be assessed by asking, using paper and pencil tests. to what extent the student agrees or disagrees with a set of statements. For example: · Smoking is a good way to look more mature·. The students selects a response. eg. agree a lot, agree a little, don't know. disagree a little. disagree a lot. Measuring changes in mental well-being. Measurement tools can be developed to assess mental well-being: including, for example. assessment of self-confidence. self-esteem. self-control and self- efficacy. Research tools can be developed to determine how young people perceive themselves. and how they attribute their successes. and failures. Measuring changes in life skills/psychosocial competence. Assessment of psychosocial competence and life skills can be done by measuring self-perceptions of efficacy in relation to the areas of skills and competencies covered in the life skills programme. Another strategy. is to determine understanding of the knowledge base of life skills. eg. by asking the student to write down the basic steps in the decision making or problem solving process. if this was covered in the programme. Measuring changes in practices related to health and social problems. Changes in health practices are frequently measured indirectly, ie. by obtaining self-reports from children and adolescents of their health and social behaviour. The questions asked correspond therefore to what the young person says they intend to do or have done in relation to a particular type of situation. There are weaknesses with this type of measurement. since what students say they will do is not. of course. always the same as what they really do in the actual situation. Self-reports of health and social behaviour are therefore sometimes validated by using other methods. For example. to determine whether young people have smoked. tests can be used to look for traces of recent cigarette smoking. Otherwise. questions can be asked to teachers and parents to find out more about the young persons behaviour and any indications of changes following the life skills programme. Measuring changes in school "climate". Assessment of improved relationships between teachers and pupils is one indicator of improved school climate. Another could be assessment of changes in teacher's and school principals' attitudes towards students. Changes in academic performa11ce. rates of school drop-out a11d absenteeism may be worth investigating, to detect any positive changes due to life skills education. It may be possible to assess these on the basis of existing school records. (However. if no changes in these areas are observed. this should not be interpreted as a failure of the life skills approach. unless these are explicitly targeted in the programme objectives). Appendix II of this document provides descriptions of selected instruments that have been used to measure effectiveness of life skills programmes. The descriptions illustrate how measurement instruments can be designed to investigate questions related to the different potential . ~E Sio:ILLS DUCATiON C:..,-ll\JNING F-:Q R:ff.:..RCH :-::ge ..i2 outcomes of life skills education. The task is to ensure that the measurement instrument covers all the areas for assessment. without becoming too long and complicated. The descriptions also illustrate how new research tools are created to suit the needs of the programme being investigate. but nevertheless are based. to a greater or lesser extent. on existing measurement tools. Measuring outcomes with young children Many life skills programmes are designed to facilitate life skills learning from a young age. With children as young as five or six years old, it will not be possible to distribute questionnaires for them to complete. Options for measuring outcomes are: i) ask the children questions orally; ii) ask teachers to complete a questionnaire about their perceptions of the children's behaviour. skills, attitudes. etc.; iii) observe the children and record findings on the basis of an observation schedule. designed to assess behaviour. skills. mental well-being etc.. (This would have to be done blind ie. by a researcher who is unaware of the nature of the intervention and whether the group being observed is a control group or implementing/experimental group). Evaluating programme outcomes If the pilot test study sets out to determine the outcomes of the life skills programme, the research design should. as far as possible. include the basic characteristics of the experimental research design. An experimental design is the only study design that can actually prove causation. In an experimental design. individuals are randomly allocated to at least two groups. One group is subject to an intervention. while the other group(s) is not. The outcome of the intervention (effect of the intervention on the dependent variable/problem) is obtained by comparing the two groups. Experimental research designs follow the format shown in Figure 5. below. Figure 5: Diagram of an experimental study3 Sludy g up ( cxpenmemal. I I First data collection (before ,11ervention1 lntervenuon I Last data collecuon Studv populauon ,,amolm~, Sample populauon ( randonnzation I :· 0 APTE~ 2 --'!: RESEARCH PROCESS ;:age JJ Control (companson group) I First dala collection (same 11me as m study group) N . I . o mtervenuon I Last data collecuon (after '"'1 .... rv-en_u_o_n, _______ _ compare ----------'s_am_e ... f lDle as m study group) Frequently, especially in school settings, it is not possible to randomly select schools for inclusion in a study, and sometimes it may be considered unethical to introduce an intervention for one group of students whilst withholding the intervention from another group (ie. the control group). Random sampling is therefore not always possible. and research which is nevertheless set up to, as far as possible include the characteristics of experimental research, is known as quasi-experimental. This kind of research design is perfectly valid. although the limitations in the research design may pose constraints when it comes to interpretation of the research findings. That is to say, the conclusions may be a less convincing proof of effectiveness. However. the study should nevertheless provide useful information about the overall outcomes of the intervention in comparison to the control group (no-intervention) condition. Following the plan for an experimental study. as shown in Figure 6. the model of the pilot test evaluation. below, describes in more detail a pilot test study. Adapted from Varkev1sser. C.M. Pathmanathan. I. and Brownlee. A.11991) Des1gmng and Conducung Health System Research ProJects. Health System Research Training Senes. Vol 2 (1). lntemauonal Development Research Centre. Canada. _ ~= Si<iLLS :~liC,~·iCN =-~t.NNING ~·'.:'{ :.ESEARCH .::-:Jge .j,4 Figure 6: Model of a pilot test i) Select a representative sample of schools in the country/region I ..... ii) Randomly select schools from the representative sample for participation in the study t iii) Randomly assign schools to the intervention and control conditions. t iv) Select the outcome indicators for evaluation of the life skills programme. Collect data on these indi .uors before implementation of the programme ie. pre-test measurement of outcome 1. iicators (dependent variables) for both the experimental groups and the control groups. t v) Implementation of the programme for three to six months t vi) Post-test measurement of outcome indicators for the experimental group and control group t vii) Ad hoe assessment of process indicators t viii) Data analysis. including comparison of process evaluation findings in relation to outcome evaluation findings Options for the pilot test The experimental study design, described above, is just one option for the pilot test. Choosing an appropriate design will depend upon many factors, including the resources available and the access to research expertise. Furthermore, in situations where there are simultaneously other major upheavals in the school system. beyond the introduc1on of a life skills programme, it might be difficult to determine which outcomes can be attributed to the life skills intervention, even if a control group is used. There may simply be too many other changes that could explain the results obtained. In such cases. ad hoe studies, based on questionnaires and/or interviews, could be carried out following a period of implementation of the programme. Ad hoe studies can be used to evaluate process indicators about the implementation of the programme. and outcomes indicators can be investigated in subj~ctive terms. ie. what have the "users" perceived about the impact of the programme. Developing a questionnaire For each new life skills programme developed. an evaluation tool will be required. The items covered in the questionnaire(s) should measure variables that are related to the content and objectives of the programme. Pre-existing questionnaires are therefore unlikely to include all the elements required to evaluate the new (or adapted) life skills programme. It is possible. however, to select and combine the most suitable items from various existing questionnaires. ~:-:.A.PTER 2 T!-'E RESEARCH PROCESS ::.age JS The creation of a questionnaire for use in an experimental or quasi-experimental study is likely to be the most challenging aspect of research tool development. The questionnaire would have to combine questions to obtain data related to the different outcome variables and may also include <for the post-test) items related to the process evaluation variables. It risks becoming rather long and complicated since there are so many questions to ask. The challenge is therefore to be as precise and concise as possible. The language of the questionnaire should be simple and pitched to the level of the respondent. Only the most essential questions should be asked. Questions should be asked in a logical order to facilitate comprehension, however, more sensitive questions, eg. penaining to sexuality or use of drugs etc., should be put towards the end of the questionnaire. The questionnaire would also have to be a reliable and valid measurement tool: to ensure, as far as possible, that the questionnaire is measuring the same thing for all those who complete it, and to ensure that the questionnaire is capable of measuring the same thing over time. To achieve this questions are as specific as possible. And questions about behaviour would specify a limited time frame for consideration ie. the last week or the last month. When developing a questionnaire, it should also be clear to the researchers how the responses will be scored and how the scores are to be interpreted. A pretest of a newly developed questionnaire is indispensable. Use of the questionnaire with a small, representative group, in an interview situation. provides an opportunity to check for problems related to misunderstanding and ambiguities. It will also help to identify questions which create defensiveness and which may lead to untruthful responses. The researcher can ask the respondents how they would rewrite the questions that are difficult to understand or to answer. Pilot testing the teacher training The training requirements for teachers and trainers of trainers will depend upon the content of the life skills programme and the methods used. The pilot test phase provides an opportunity to test out the appropriateness of the planned training course. by assessing the training course per se, and the relationship between teacher training and the life skills programme implementation processes and outcomes. As with the evaluation of programme materials, training can be assessed in terms of the process and outcome variables. as below. Process evaluation of teacher training Process evaluation, like all other aspects of research, need not be seen as an outsider judging the actions of others. Evaluation can be introduced as a part of the process itself. and this is especially important in the context of teacher training. In training courses. the research component can become a source of feedback for the trainers and the participants. Process evaluation in the training context includes such questions as: _'F~ SKILLS E:UCP.T1CN ;::~.<lNNING FG~ i<ESEO:.RCH .=,Jge 36 • How en_ioyable and useful has the traming been? • What aspects of the training were most familiar to the participants and easy to assimilate? And, which aspects of the training were least familiar to the participants and most difficult to assimilate? How comfortable were the participants with the methods used in the traintn(? course? Do they feel prepared to use those methods in the classroom? • Is the training culturaily appropriate? • In what areas do teachers feel they need more training? As in the formative research stage, the writers of training courses are still at the stage of learning about the needs and concerns of the teachers. Process questions should therefore be phrased in open and closed questions as appropriate, in order to obtain a mix of qualitative and quantitative information to be used in the review and revision of the training courses. To build process evaluation into the training course as an element of feedback for the participants. the research questions can be asked to the whole group just before the end of the training course. This offers an opportunity for reflection on the overall experience of the training. If such opportunities for verbal feedback and reflection are built into the design of training, as well as into the life skills education materials. this should encourage teachers to use process evaluation as a part of his/her teaching practice. In this way, process evaluation becomes more than a source of research information, and it serves as a source of enrichment and learning. However, to provide the qualitative and quantitative information required for research of the effectiveness of the training course design there needs also to be scope for written feedback on the course contents and methods, eg. using a questionnaire. Outcome evaluation of teacher training Process evaluation questions help to determine whether the design of the training was suitable for the participants. manageable for the trainer and appropriate in terms of content and length of sessions. Outcome evaluation, on the other hand, tells us whether training achieves its objectives and. if appropriate questions are asked. it can tell us what other impact training has had on the participants. Training for life skills educators i~ likely to include training in participatory and experiential learning methods. An outcome evaluation research question would therefore be: Have teachers learnt these methods following the training and do they feel confidenr to use them? Such questions can be asked of teachers using questionnaires and interviews. However, there is an additional level of investigation that may be necessary, that is to answer the question: Following training, do teachers use these methods in the delivery of the programme materials in the classroom? In this· case. structured obse111ation in the classroom may be required to give an accurate account of the extent to which participatory and experiential learning methods are used following training. Pilot test design (teacher training) There are two levels at which pilot test evaluation of a trammg course is possible: i) evaluation of the training course per se. and ii) evaluation of the training in relation to the implementation of the life skills programme. i l Evaluation of the training course per se -:HAPTER 2 -!-"~ RESEARCH PROCESS :i.::ge J7 In most cases. this is an ad hoe assessment. including measurement of process variables, such as enjoyment of the course and its perceived usefulness. There may also be some evaluation of outcomes. such as self-reponed use of a greater range of teaching methods and better classroom management. ii) Evaluation of the training in relation to the implementation of the life skills programme This level of investigation allows us to ask the very important question: Is training necessary for effective implementation of the life skills programme? This is not the type of research question that can be answered on the basis of qualitative or ad hoe information alone. An experimental research design or at least quasi-experimental research will be required. A pilot test evaluation of training can be carried out as a part of a study set up to investigate the effectiveness of programme materials. In this case. there would an additional experimental condition: implementation of the life skills programme with no teacher training. The experiment would therefore have the following type of format. Random assignment (if possible) of a representative sample to one of three conditions: i) Implementation of the life skills programme materials following a training course. ii) Implementation of the life skills programme materials with no training. iii) Control group with no training and no exposure to the life skills programme. All three groups would be studied using the research instruments (questionnaires, observation ratings, etc.) pre-test and post-test. Pilot test of the life skills initiative The "life skills initiative" is all the work done to promote life skills education and to gain financial, political and local support for it. Without support and commitment to it. life skills education is unlikely to be widely implemented. Evaluation of the input into the promotion of life skills education is therefore as important as evaluation of the life skills programme itself. Research questions to consider include the following: • Has the life skills initiative attracted sufficient political commitment and support? • Have there b.een relevant policy changes? • What is the level of government support and local commitment? • Has the initiative been endorsed by politicians? • Has the initiative attracted media attention? • To what extent has the intervention been institutionalized? Or, what potential is there for this? A survey could be used to ask the relevant people these questions. Otherwise individual interviews could be carried out with selected policy makers and decision makers. _ F:: SKILLS EDUCATION PLANNING F·::? RESEARCH :'Jge 38 Summary of research during the life skills education development phase The situation analysis provided much information about the priority problems and the perceptions of needs. as well as understanding about the causes of problems and possible solutions. The formative research phase took this a step further to reveal. in more detail. educauonalists', parents' and teachers· perceptions of tne role of education in enhancing the psychosocial development of young people. All this helps to define the content of a life skills programme. A pre-test of draft programme materials helps to ensure age, gender and cultural appropriateness. Following the pilot test period. the life skills programme and training would have to be revised and re-written to take into account the practical problems of implementation as well as to improve, as far as possible, the potential impact and effectiveness of the programme. Pulling together all the information gathered during the three phases of research so far. it is possible to launch a life skills programme together with an account (based on research findings) of what it is expected to contribute to the promotion of the health and education of young people in the country. Monitoring Dissemination · ::-!APTER 2 ·;-:E RESEARCH PROCESS Page 39 The research element of disseminating life skills programme materials is typically overlooked. This is perhaps because the questions about dissemination. such as - how many programmes are being used. by whom. and where'! - do not seem relevant until the programme has been disseminated. Unfortunately. especially for the most successful interventions. if these questions are asked too late. it can become impossible to give accurate responses. This becomes even more difficult if the programme materials are frequently photocopied and passed on and if the training is carried out following a "cascade" model. where a trained person trains another person and this person trains someone else. Questions relevant to the dissemination of life skills programme materials and training include: How many sets of programme materials are being distributed? or <if it is sold) how many people buy the programme? • Who is using the pro~ramme? Which teachers and in what type of schools? • What age groups are receiving the programme? • How many teachers are trained? • How many children take part in the programme? Although the answers to these questions will only be available after dissemination. strategies for collecting this information can be set up before dissemination. All trainers on completion of a course could be asked to complete a questionnaire including questions such as: how many people completed the training course?. and; how many were from eg. religious schools/private schools/non- formal education/primary schools/secondary schools? Completed questionnaires would then be sent to the life skills project manager or other such person who is responsible for the coordination of the life skills initiative. Similarly. questionnaires can accompany the distribution of the life skills programme materials. To help coordinate dissemination. it has been found to be helpful to appoint district level coordinators of life skills education. Channelling the distribution of materials via a district coordinator provides a focal point for collecting information about dissemination. Equipped with a check/ist of information to collect or questionnaire to complete. the district coordinator could collect the following information : • How many schools have the materials? • How many teachers received the materials? • What kinds of teachers are using the materials. ie. ·what subjects do they normally teach? How many pupils take part in the programme? Other interesting information to collect. which could prove useful in a later analysis of the dissemination of the programme and its use over time. would include characteristics of the schools using it. including for example. number of pupils in the school and racial mix of the schools. _ ri: SKiLLS DUCATION· PLANNING FCR RESEARC~ =·Jqe 40 The need for district coordinators is itself a research question and one which could be partly .:mswered by collecting and analysing dissemination information. For example, if we can compare the scale of dissemmation in places where there is a coordinator to places where there are no such coordinators, it becomes possible to answer the question: is dissemination more successful (widespread) when there are life skills district coordinators? Where there are no district coordinator for life skills education, dissemination information can be collected using, for example, a questionnaire attached to the programme materials, to be completed by the user and returned (freepost if possible) to the life skills project manager. There may be a need to offer incentives to encourage responding. eg offers of support, mailings etc .. Evidently, research about dissemination can provide information about the scale of dissemination and the success of dissemination (is it reaching those it was intended for), which will be useful information to have later. for the second situation analysis. It is however also a useful contribution to make to the dissemination process itself, identifying, as described above, the most effective strategies for dissemination. Analysis of distribution data The information obtained about distribution allows project managers keep track of dissemination. It also is an opportunity to learn more about life skills education and the keys to its success. For example, one study of distribution of a life skills programme in the UK showed that there were more sales of the programme in schools which had guidance teachers. This is an interesting observation, which was only noted because information was collected about the schools where the programme was implemented. It is recommended that during dissemination. information is collected about the type of schools receiving the programme, the type of teachers that are trained and their number. Evaluation during implementation ~~APTER 2 ~HE RESEARCH PROCESS ::age41 As for research during dissemination. described above. evaluative research about the implementation of life skills should begin before implementation. This will be described in the 1:ontext of the two main areas of life skills programme/training evaluation. which are: evaluation as an integral pan of life skills education. and evaluation in selected (representative) research studies. On-going process evaluation in life skills education On-going process evaluation refers to the evaluation of programme implementation which should accompany the use of every life skills programme. This is the feedback session which is built into the life skills lessons and the training workshops for life skills educators. This element of research is planned well before implementation, during programme development. At the end of life skills lessons, evaluation questions can be asked. for example: • Did you enjoy this lesson? • What did you find most interesting/ least interesting? • Is there anything that you did not understand? Similar process evaluation questions should be asked at the end of teacher training sessions. If used consistently and taken seriously, it is envisaged that this level of evaluation will ensure that problems of implementation will be identified and resolved at the local level. In this way there is scope for continuing review and revision of life skills education in light of local situations and local level needs. Life skills education and the training for teachers is based on experiential learning: the evaluation component built into training and programme design, is an extension of this, offering a further opportunity for learning from experience. Process evaluation questions are an integral part of life skills programme and training design. This is however, a form of feedback and as such the evaluation is internalised by each individual. and is therefore not in itself a source of information about life skills programme effectiveness. The challenge for life skills research will be to tap this feedback as a source of information about effectiveness by encouraging, for example, the preparation of written reports which describe this feedback. The challenge will also be to use this as a source of life skills programme evaluation without losing the positive contribution that this form of feedback provides: the evaluation of the intervention by its users and receivers for their own benefit. Written feedback reports prepared systematically where life skills programmes are implemented would provide a tremendous wealth of information about the use and perceived contribution of life skills education. Such reports can be used to prepare case studies of the use of life skills education in different parts of the country. which could be useful as a guide to the further development and implementation of life skills education. However. these methods on their own will not tell us with any certainty that life skills education achieves its aims. For this reason. wide- scale feedback should be supported by small scale research studies in selected sites. as described below. _ ~E SKILLS EDUCATiON PLANNING FOR RESEARCH =·-:iqe42 Short term studies of life skills programme implementation and outcomes Research studies would be set up to investigate the effectiveness of the implementation processes and the outcomes associated with the life skills programme. The principal research methodology used to investigate life skills programme outcomes is, as described in the context of pilot testing, the experimental or quasi- experimental research design. As described earlier, the (quasi-)experimental design requires that data relating to anticipated outcomes of the intervention have to be collected before the use of the programme as well as at some point after the programme is implemented, ie. the pre-test/post-test design. When this data is collected for a group that is taking part in the programme, as well as for a group which is not taking part in the programme (the control), changes observed after the implementation of the programme, which occur only in the group using the programme. are attributed to the intervention. Hence, the necessity to describe and quantify the effectiveness of the implementation processes, which will need to be compared with the outcomes associated with the programme. The main technical aspects of research design have been discussed earlier in the context of pilot testing. Identifying the schools in which to carry out process/outcome studies. and the collection of pre-test data. would be done before the start of life skills programme implementation in those schools. Schools included in the study should be representative. as described for the pilot studies, ie. to include a selection of urban and rural schools. primary and secondary schools (as appropriate). At this stage, if life skills education is being implemented in different regions of the country, the research should also aim to include schools in the different regions. The number of schools involved in research studies. and the number of different locations, would depend upon the size of the country and the extent of implementation of the programme, as well as the resources available for research. When identifying schools that could talce part in the study, another important factor should be taken into consideration: at least in some of the schools involved. there may be plans to follow up the research study over time. That is to say, as long;tudinal studies. The schools selected should therefore be stable educational environments that are e,: 'ected to be maintained for, at least, the next six years. In order to maximise the possibilities for following up those involved in the studies over time, it may help to collect parents' home addresses for children that talce part in the study, and to include a large number of people in the original studies, so that even if a small percentage of this group is followed up long term. the numbers in the study will still be meaningful. The studies could also focus on the youngest age groups taking part in the life skills programme. to improve the chances that these children will still be in school several years later. More will be said on longitudinal studies later. Situation Analysis -2 ·-::HAPTER 2. : -iE RESEARCH PROCES.S Page 4J The second situation analysis should be carried out during the maintenance phase, once the intervention is being used widely. It is a ume to take a step outside of the implementation process in order to gain an overview of the life skills intervention. The following questions may be asked: • Is life skills education being implemented as planned? • What evidence is there that life skills education is effective in achieving its goals? • What other impact has been observed following the implementation of life skills education? • To what extent does life skills education respond to the needs and problems identified in the first situation analysis? • What is the proportion of schools using the programme in the country (or province. county)? How many teachers have been trained? • What is the proportion of children receiving the programme in the country (or province, county)? Analysis of the quantitative data and qualitative information obtained from the research carried out during the implementation phase should help to answer these questions. This is a time to prepare reports pulling together, analysing and interpreting all the research findings. Analysis of the data to look for significant correlations between the different variables measured may help to identify the characteristics that are associated with the most successful outcomes, as well as the factors which prevented successful implementation. The data collected about dissemination of the life skills programme and teacher training should help to monitor the extent to which life skills education has been distributed around the country. However, the number of life skills programme and training materials distributed does not inform us of the number of people currently using the programme. Therefore. during the situation analysis phase. it may be necessary to verify dissemination data with investigations in randomly selected schools to determine the extent to which the life skills programme is currently in use, with what age groups and in which types of schools. Another research question to ask at the time of the second situation analysis is: to what extent are there changes in the prevalence of young peoples' problems and needs? Collecting statistics and epidemiological data at the second situation analysis is another opportunity to monitor the trends in problems and needs. Life skills education is not a static intervention, it should be a dynamic one, which can be adapted to suit changing needs and priorities. ~IFE SKILLS EDUCATION· PLANNING FC,Q RESEARCH :iage 44 Long term evaluation During the maintenance phase. there are two important aspects of long term evaluation: on- going process evaluation/feedback and longitudinal research studies. On-going process evaluation, which is built into the life skills programme design, offers teachers and pupils an opportunity to gauge the longer tenn use of. and relevance of, the life skills programme. If this feedback is written up as case studies, as suggested earlier, there will be scope for using this infonnation to guide the revision and further development of life skills education, either locally or at country level. The other aspect of long tenn evaluation is longitudinal research. Longitudinal studies of the impact and outcomes of life skills programmes is particularly important because there are key programme effects which are only observable in the longer tenn. for example, the effects on health and social behaviour. As discussed in Chapter 1, such outcomes of life skills education are only expected when the intervention itself is designed to be long term. Long tenn evaluation of a short term intervention is unlikely to produce positive results. The words "long term'' evaluation can be off-putting for some, perhaps because this implies a long term effort with continual input of time and resources. This is not necessarily true. Longitudinal research studies would be done as a follow-up to the short tenn research studies of life skills programme implementation and outcomes. The same questionnaires, interviews and/or observations would be used, with the same individuals, as far as possible, as in the studies carried out during the implementation phase. It would mean doing a repeat of the post-test measurements several years after the life skills programme was first implemented: for example. after 3 years and after 6 years of implementation. For each longitudinal study, 3 months could be allocated for data collection. Finding the participants of the original research studies, done during the implementation phase, is perhaps the most difficult aspect of long term evaluation. Measures that can be taken to improve data collection for longitudinal studies are described in the context of the short tenn studies. because the research can be design at that stage to facilitate long term follow up. In the long term, there are numerous factors affecting life skills programme implementation and effectiveness. It may be important to take note of major social and political changes experienced by young people and school systems in the country following the time at which life skills education was first introduced. This kind of information can then be taken into account when interpreting the results of the long term evaluation. :;;APTER 2 THE RESEARCH PROCESS Poge45 Summary of the Research Process In this chapter. the life skills education research process was described to give an outline of the main research questions and research methods relevant to the different stages of life skills programme development and implementation. These are summarised here. before going on in the next chapter to look at the time-lines for working through this process. Table 1 : Life Skills Research Overview PHASE TYPE OF RESEARCH OBJECTIVES R.E.5EARCH METHOD Development Phase Situation analysis To detennine objectives and Review of literature and strategies for life skills available data education Focus groups Fonnative research To guide design of life skills Survey/focus groups programme and training and small scale pre-test Pilot test Evaluate effectiveness of ( Quasi-)experimental programme and training studies implementation and outcomes Evaluate life skills initiative Survey {mterviews ( to assist revision of intervention and strategies) Implementation Monitoring To monitor distribution patterns Questionnaire/check list Phase Process evaluation On-gomg process evaluation Feedback/processing <feedback) of programme and training questions in programme implementation and training materials Shon tenn evaluation Evaluation of programme and (Quasi-)experimental training implementation process studies and outcomes Evaluation of life skills initiative Surveys/interviews (in selected sites) Maintenance Phase Situation analysis -2 Review of life skills education Review of literature and dissemination and effectiveness available data Review of current needs/problems Focus groups/surveys Long tenn evaluation On-going process evaluation/feedback Case studies Longitudinal studies oi programme ( Quasi- ,experimental implementation and outcomes studies 5E SKILLS EDUCATION: PLANNING FOR RESEARCH ::>age46 ::'-1APTER 3 IJSiNG ~:-.-: RESEARCH FRAMEWORK =age 47 CHAPTER 3: USING THE RESEARCH FRAMEWORK Suggested time-lines for the research A five-year period is considered to be a feasible length of time in which to carry out the research described in Chapter 2. This includes the time required for programme development and pilot testing, as well as the time required to disseminate the materials and carry out short-term process/outcome studies. It does not however include the long term research, which would be carried out at specified intervals following the initial five year period. Research should be incorporated into the process of programme development, implementation and maintenance. In this way, research is not an added feature and doing research does not mean that the whole process of development and implementation of life skills education takes five years longer. This may well be the length of time required to reach wide-scale implementation of a life skills programme, even if no research is carried out. The pilot test period is perhaps the part of the research process which nevertheless stands out as an additional component. This however should be considered as an important part of programme development. Having said this, the pilot test should not be allowed to carry on for years and so hold up programme implementation. Similarly, the situation analysis and formative research would have to be carried out in a defined period of time in order to move forward at a sufficiently rapid pace to achieve the goal of wide-scale implementation of life skills education within a five year period. The time-lines suggested here should help to guide the development of a time-table for research as an integral part of programme development, implementation and maintenance at the country level. The suggested time-lines are described below and summarized in Figure 7. Programme development phase. This includes three main research activities: the situation analysis, formative research and the pilot test. The situation analysis may take approximately 6 momhs to complete. This would then be followed another 6 months in which to carry out formative research. There would then follow a period of approximately 1 .vear. during which time. programme developers would concentrate on writing the programme materials and designing the training workshops for teachers and trainers. Research at this stage includes a small-scale pre-test of the programme and training. Following this. the pilot test would be set up. It could take 6 months just to plan for the pilot test, including time for the design of the study, development of research tools, and identification of research sites. A further 6 months should be allocated for carrying out the pilot test. This is not a long period for a pilot test, but it should provide sufficient time to gather important information regarding the implementation and impact of the life skills programme. The information gathered during the pilot test would be used to revise the programme and training materials. A period of 3 mo11ths is then envisaged during which the programme materials and training are revised. In all, the programme development phase may take approximately 3 - 3.5 years. . ~:: SKiLLS DUCA7iCN· ;;,_ANNING ;:::,R RESEARC!-: .:·.Jae a8 Implementation phase. There are two mam aspects of research at this stage: monitoring of dissenunation. and evaluation of the programme and training. As described in Chapter 2. research 1)n dissenunauon actually means setting up a mechanism by which to monitor dissemination in the longer term. In this case a J-6 month period is allocated <the amount of time requued for this will depena upon the planned scale of imptementation and the size of the country). This does not mean that the life skills programme will be disseminated everywhere in a 3 month period. rather this time is allocated for setting up a mechanism for life skills programme dissemination. and to determine strategies for collecting information about dissemination. Evaluation of the programme and training, including evaluation of the implementation process. the life skills initiative and the programme outcomes. would largely be based on the pilot test design and using the research tools developed for the pilot test. Process/outcome evaluation during the implementation phase is possible in a I year period. Maintenance phase. During this final phase. important elements of the programme development and implementation phase continue. Research should be carried out for on-going evaluation and revision of the life skills initiative. programme and training. This phase begins with a second situation analysis which would be carried out over a 6 month period. According to the these suggested time-lines. the second situation analysis can be carried out approximately 5 years after the start of the programme development process. and hence 5 years after the first situation analysis. By this time. life skills education should be in the process of becoming available wide-scale. There should be sufficient research data to provide a review of the scale of dissemination. the quality of implementation. the extent to which the initiative is supported. and the outcomes of the intervention. Research carried out in the longer term would build on the research described during the implementation phase. For example. the research studies could be followed up 2-3 years after the initial implementation. and again 5-6 years later. Figure 7: Time-lines for research ~tart Situauon analysis - 6 months Formauve research - 6 months Pilot testing - I year Setting up monitoring of dissemination - 3 months Shon term evaluation of initiative. programme and training - l year 5 years 2nd Situation analysis - 6 months Long term evaluauon Who would be doing the research? :;-;APTER 3 vSiNG 71-ii: RESEARCH FRAMEWORK ?age 4Y The project manager for life skills education is likely to be responsible for overseeing the research process. In addition. the project manager would collect information for the situation analysis at the beginning of the programme development period. and again at the second situation analysis during the maintenance phase. Formative research. during the programme development phase. is likely to be carried out by the programme developerts). Programme developers would investigate the formative research questions as a part of the strategy for programme development. For the pilot test. ideally an experienced researcher from a university or a research agency would design and carry out the pilot test of the draft life skills programme materials and training. To ensure appropriate high quality research, care will need to be taken when selecting research personnel. ( See section on this. below). In university settings. the research is likely to be coordinated by a senior researcher, but the work could be assisted. at minimal cost. with the involvement of students ( eg. by stimulating interest in this area of research amongst prospective PhD students). The same university department or research agency should, ideally, be able to coordinate the process/outcome studies at the implementation phase, and the longitudinal studies at the maintenance phase. Contracting out pilot testing and evaluative research to a university or a research agency, that is not involved in programme development or implementation, will help to ensure objective evaluation. For other aspects of research, such as monitoring dissemination, if the research is carried out as far as possible by those involved in the development and implementation process, it will contribute to a better understanding and "ownership" of the research process and its findings. This is particularly imponant for the remaining area of research. not yet mentioned in this section, the feedback that the teachers and students generate during training and during life skills lessons. If evaluation is purely an outside activity to judge programme effectiveness, it risks becoming alienated from those most closely involved with life skills education, and could be seen as a source of criticism. On the other hand, feedback/process evaluation, built into programme design, and carried out by and owned by the teachers and students, can become a constructive assessment of the work being done and an added strength of life skills education. Selecting research personnel Some points to consider when selecting research personnel include: • Does the proposed researcher have relevant research experience? • Does the proposed researcher understand the programme goals and operational strategies? • What does the researcher consider to be the focus of the evaluation? • What type of data does he/she plan to collect and how? • Can the researcher produce a detailed budget for evaluation and does this correspond to the funds available? .'f=:: SK!Li.S EOL.:ATION P!.ANNING F·:-;, RESEARCH :.cge 50 Has the researcher ,...,repared high quality· research reports for previous research acuviries? And. in previous research. has he/she been able to bring our key findine.s that could influence decision makers? • Does rhe proposed researcher have .e.ood references? What other professional commitments does the proposed researcher have, and how does he/she fin to conduct the evaiuation alongside other tasks? • If the proposed researcher will be carrying out the research wirh a team of other researchers. how will he/she. as team leader. delegate the research activities? Will the activities of ream members be supervised by the team leader? Costs of research :~APTER ~ :..,,3iNG -;-i-iE R::SEARCH FRAMEWORK ;:Jageol The framework outlined in this document is intended to give an overview of the research related to life skills education development. implementation and maintenance. This overview. together with suggestions for the time-lines of the research process. should help with estimating the costs of research. The principal cost of any piece of research is likely to be the cost of personnel to take charge of the research design and implementation. In many cases, carrying out research related to life skills education is unlikely to be the sole task of the research staff. or the life skills project manager. For example, it is suggested that. at most. the research tasks will take up half of a full-time position for the person responsible for each phase of the research. A schema has been prepared to provide an estimation of the research input of personnel for the different phases of programme development and implementation. Other research related expenses include the costs of: secretarial support. travel. printing, telephone. postage, and the costs of holding meetings. Schema for estimating personnel costs The life skills project manager will be engaged in setting up the life skills initiative. rallying the support of policy makers, educationalists, parents and the media. He/she will be preparing plans for life skills programme development and considering the scale and strategy for implementation. Following the time-frame outlined earlier. this would be carried out over a six-month period. The research for the situation analysis may take up to two mollth · s full time work for the project manager. For the setting up of interviews and focus groups. approximately one month's fit/I time work may be allocated to other researchers. Formative research is allocated 6 months in the time-frame described earlier. It may cost the equivalent of three month's full time work for a programme developer. The pilot test period is expected to take up to one year to complete. In all. six mollth · s full time work for one researcher (from a university or research agency) is anticipated over the one year period. Personnel input linked to research during the dissemination planning phase is expected to amount to one month of full-time work for the project manager. Evaluation during the implementation phase spans a one year period. The personal input required would be approximately six months offu/1 time work for one researcher (from a university or research agency). It is envisaged that in this time, the researcher could carry out process/outcome studies in several schools ( using the research design and tools already developed for the pilot test). If research studies are planned in distant geographical locations. and in a large number of schools. it may be necessary to engage additional researchers. 5E SKILLS cDUCATION ?Ll>.NNING FOR RESEARCH ::.oge o2 Ongoing process evaluation/feedback that is built into programme and training design does not require additional personnel input. However. in order to use this information. a researcher may be employed to collate feedback information from different sources. to analyze the findings and prepared case studies. If this work is envisaged. it may require approximately one month of full time work for one researcher. The second situation analysis is allocated a six month period in the time-frame. The research at this stage could take another two months of full time work for the project manager. Longitudinal studies are not included in this schema because few project managers will consider their budget for more than five years at a time. In total. following this schema the personnel input required would be as shown in Table 1. Table 2: Estimating personnel requirements Research Personnel Input (months) Project Manager 5 Programme Developer 3 Researcher (from University or Research 12 Agency) Other Researcher(s) 2 Communicating research findings .::-!APTER 3 USiNG ,HE f.c:SEARCH FRAMEWORK ?age 5.3 Research generates information. Like all other information, the research findings are of limited value if they are not communicated. To ensure that the information is not lost, each piece of research should be written up in a repon. Even if a report is not formally requested. it could be useful to distribute to inform other researchers and programme developers of the research design and findings. A research repon could also be the basis from which to prepare reports to send to policy makers and decision makers as well as for articles for professional journals and newsletters. Wide distribution of a carefully written research repon, assuming there are some interesting findings, could also help to stimulate media interest and coverage of research on life skills education. Suggested format for a research report' I Summary The summary ( 1-2 pages) should briefly describe the study objectives. the study design. the research site(s) and summarize the main findings. II Introduction and statement of purpose This section should describe the overall objectives of the research. as well as the specific research questions to be addressed. III Description of research site(s) This section should include the criteria used for selecting research sites. as well as background information on the group involved in the study ( eg. socio-economic and ethnic/linguistic characteristics, community and school characteristics) IV Study methodology The study design. number of subjects in the study. sampling strategies. and data collection methods would be described. The choice of research design and methods should be justified. and any problems encountered should be discussed. V Results Results from all relevant data collection activities should be drawn on to provide detailed answers to each research question. VI Conclusions and recommendations The detailed discussion of the conclusions should refer to the research objectives outlined in the introduction. What relevance do study findings have for the intervention ( eg. teacher training, content of life skills programmes etc.)? What recommendations can the researcher make concerning concrete actions to take. based on the study findings'! VII Appendices Any lengthy materials. cg. descriptions of interview schedules and questionnaires used in the study. can be placed in an appendix. 4 Adapted from: Hudelson. P.M. (1994). Qualitative Research for Health Programmes. World Health Organization. unpublished document WHO/MNH/PSF/94.3. Geneva. _;FE S•'ILLS EDUCATION: PLANNING FCR RESEARCH ::iage 54 APPENDICES ~PPENDICES ?age 55 _:FE s:.:1LLS EDUCATION· PI.ANNING rCR RESEARCH :".lge :i6 APPENDIX I: EXAMPLES OF RESEARCH RELATED TO LIFE SKIW EDUCATION Page57 APPENDIX I: EXAMPLES OF RESEARCH RELATED TO LIFE SKILLS EDUCATION Example 1: Use of focus groups during a situation analysis in Zimbabwe Answers to the following questions were provided by Bernard Gatawa. AIDS Programme, UNICEF, Harare. Why were focus groups used? To determine school children's perceptions of their needs vis-avis HIV I AIDS prevention. A study had been carried out to determine the broad themes of the courses and textbooks. There was. however. a need for qualitative information about the children's needs, in order to define in more detail what should be the content of the school AIDS education programme. How were the focus groups selected? Selection of the focus groups was guided by the need to cover the four major school types in the system - ie. urban high density, urban low density, district council and mission schools. Stratified sampling was used to select four focus groups to represent the major school types. How many children were in each focus group? On average. each focus group had 12 participants. evenly divided between boys and girls. Who carried out the focus groups? The focus groups were carried out by curriculum developers from the Ministry of Education. and UNICEF officers and consultants. What questions were asked? The questions covered the following broad areas: • Knowledge about AIDS - what they knew and what they wanted to know. • Attitudes - to AIDS. to people with AIDS, to the opposite sex. to friendships. to discussion about AIDS. • Sexual behaviour - including discussion of: their own behaviour. prescribed traditional behaviour, courtship practices, dating, what they see as problem areas and possible solutions, what they consider safe and acceptable sexual behaviour. • Methodology - how they would want to see AIDS education classes organised, teaching and learning methods. issues of assessment and examinations. role of family versus school. What were the main"jindings? Conclusions drawn from the focus group discussions were: • Knowledge of basic facts on HIV I AIDS was high although there were myths and misinformation on transmission and prevention. • Knowledge did not necessarily influence sexual behaviour. Many young people did not think the problem of AIDS could affect them. • Negative attitudes to people with AIDS were widespread. Females tended to be blamed. • Attitudes to sex tended to be casual. The young people recognised the need to discuss AIDS openly in spite of taboos. _'rE SKILLS EDUCATION· FLANNING FOR RESEARCH :::'Jge 58 • Prescribed traditional sexual behaviour was regarded as outdated. • Basic social and communication skills were not being promoted by any formal structures in society. • Children should be allowed to be children. Adults should not prescribe how they should behave. • Family structures on sex education had broken down. Adults were not giving adequate guidance and information: young people rely on their peers for this. Schools should fill the gap. • Teachers, by and large, were not considered to be good role models. • Pre-occupation with examinations and the congested time-tables were seen as constraints to effective AIDS education in schools. APPENDIX I: EXAMPLES OF RESEARCH RElATED TO LIFE SKILLS EDUCATION Page 59 Example 2: Research to guide the development of a life skills programme in the UK Situation Analysis Plans were formulated for the development of a life skills programme for primary school children. Funding was obtained to carry out a situation analysis from Re-Solv. the Society for the Prevention of Solvent and Volatile Substance Abuse in the UK. The purpose of the situation analysis was to: • confirm the need for the project and receive endorsement for the plans; • identify the needs of children. and the kinds of materials teachers need and want: • identify what has been done already in the field. and what related materials are available. The project proposal for a life skills programme for primary school children was prepared. Questionnaires (124 in total) were sent to Senior Primary Advisers in all Local Education Authorities in England. Scotland and Wales. and to four education authorities in Northern Ireland. Approximately 49% of recipients completed the questionnaire. The questions included: • Have you or your Local Education Authority produced guidelines or materials for personal and social education in the primary school? • If yes, could you outline briefly the approach and main areas covered? • In your opinion, do you consider that there is a need for the kind of materials TACADE is planning to develop? • Which skill areas do you think should be included? Analysis of the questionnaire responses showed overwhelming support for the project and the skills approach outlined in the project proposals. The respondents made a number of suggestions and recommendations for the content areas. which provided a useful checklist during the writing phase. Before beginning to write materials. programme developers reviewed the findings of recent research on the needs and problems of children. Research into children· s perceptions of the world of drugs, and research on children's understanding of relationships. helped to define the content of the programme. Pre-test A draft version of the life skills programme was developed. called Skills for the Primary School Child. It was sent to lO teachers for in-depth review. These teachers were asked to comment on the text with reference to: • appropriateness of the material for primary school children in terms of language and content; • ease of use of the materials: • any gender and cultural bias that appeared in the programme: •_'FE SKILLS EDUCATION: PLA.NNING F0R RESEARCH 0 age oO • appropriateness of the teaching methodologies; • suitability of the programme in relation to the National Curriculum and the curriculum of the primary school. The teachers in the pre-test were also asked to carry out several of the activities in their classrooms, and to comment openly on this experience. APPENDIX I: EXAMPLES OF RESEARCH RELATED TO LIFE SKILLS EDUCATION Page 61 Example 3: Pilot test evaluation of a life skills programme in Mexico To pilot test the programme Planeando tu vida (Planning your life), an experimental study was set up to investigate programme effectiveness under 4 different conditions. as below. Group l The programme was implemented with the provision of teacher training, programme materials for teachers and children, plus a parent's training course. Group 2 The programme was implemented with the provision of teacher training, programme materials for teachers and for the children. Group 3 The programme was implemented with the provision of teacher training and programme materials for teachers only. Group 4 The control group ( ie. the programme was not implemented). There were six different age groups included in the study and 35 subjects in each age group (ie. n=35 x 6, in each of the four conditions). The subjects were randomly assigned to the experimental conditions. For ethical reasons. the control group were not excluded from the intervention. Rather they were assigned to a "waiting list" and were to begin the programme at a later date. The pre-post test data collection differed for the different age groups studied, and the variables measured depended on the main objectives of the programme materials written for each age group. For example, for Grade 1, the evaluation measured knowledge acquisition. For Grade 2, the evaluation measured changes in attitudes and skills. To assess knowledge, written tests were used with older children (8 years plus) and oral tests with the younger children. Attitudes were assessed using focus groups and questionnaires. To study changes in behaviour, questionnaires were used with the older children. to obtain self-reports of behavioural intentions. For children less than 8 years old. behaviour changes relevant to the programme contents were assessed by observation of the children· s behaviour in role play situations. In this case, two researchers rated the responses of each child using an observation schedule. The study also measured self-efficacy, self-esteem and locus of control, using questionnaires. Conclusions Research findings showed that the most successful programme outcomes were achieved for Group 1, ie. where the children took part in the programme with the provision of teacher training, programme materials for teachers and children and a parent's training course. However, there was also considered to be a reasonable programme impact even when there was only provision of teacher training and programme materials for the teacher (Group 3). _ FE Si<iLLS cDUCATION PLANNING FC·~ R~SEARCH =age o2 -PDENDIX ii C:'.:SCRiPTION CF SELECTED INSTRUMENTS U~c: :!'J LIFE SKILLS PROGRAMME E\/ALUAT,ON ?age oJ APPENDIX II: DESCRIPTIONS OF SELECTED INSTRUMENTS USED IN LIFE SKILLS PROGRAMME EVALUATION Example 1: Instrument used to evaluate a programme for suicide prevention Summarized from: Lafromboise. T.D. and Howard-Pitney, B. (1994). The Zuni Life Skills Development Curriculum: a collaborative approach to curriculum development. American Indian Alaska Native Mental Health Research Monograph Series. 4, 98-121. Relevant items were selected from instruments previously employed in research with American Indians. Students supplied the following information regarding each unit: a) likes and dislikes: b) degree of interest: c) enjoyment or comfort/discomfort: d) level of personal participation in class activities: e) assessment of the unit's relevance to the student's life: f) option as to the most important thing they learned in the unit. Measurement of suicidal behaviour. Items included questions designed to ascertain: if the student had ever attempted suicide: recency of the attempt: number of times attempted: if the student had told anyone about the attempt; and. if the attempt had been accompanied by a visit to a medical clinic. Measurement of suicide risk factors. The questionnaire included several variables identified by earlier research as associated with increased risk for suicidal behaviour. including measures of: suicide potential (Suicide Probability Scale, Cull and Gill, 1982): depression (Indian Adolescent Health Survey, Geer, 1988): hopelessness (Beck Hopelessness Scale. Beck et al, 1974): psychological distress (Symptom Checklist List 90-R, Derogatis. 1977); stressful life events (adapted from the Social Readjustment Rating Scale. Holmes and Rahe. 1967); and use of drugs (Indian Adolescent Health Survey, Geer. 1988). Personal and social.skills. These items measured skills and behaviours covered in the curriculum that could help students cope better with daily life. Most items were created to measure specific curriculum components and included assessment of: anger expression: ways of coping (adapted from Folkman and Lazarus, 1980): self-efficacy for a variety of curriculum-related behaviours: problem solving; and interpersonal communication (items selected from Gay et al. 1975). Personal and family characteristics. Demographic factors. such as age and sex. were assessed as were the students' home life. including parents' marital status and drug use habits. Students were asked about their enjoyment and performance in school. sources and uses of social support. and the degree to which they identified themselves as a traditional Zuni. . ~E ~KILLS E:)UCATION· P!..~NNING ;: ... _;.R RESEARCH :i'Jge o4 Knowledge of suicide myths and facts. Several items were included to assess attitudes towards suicidal individuals and knowledge regarding suicide and community resources. References Beck. A.T., Weissman, A.. Lester, D. and Trexler. L. (1974). The measurement of pessimism: The hopelessness scale. Journal of Consulting and Clinical Psychology, 42, 861-865. Cull. J. and Gill, W. (1982). Suicide probability scale manual. Los Angeles, C.A: Western Psychological Services. Derogatis. L. ( 1977). SCL-90: Administration, scoring and procedures manual for the revised version. Baltimore: Clinical Psychometric Research. Folkman, S. and Lazarus. R. ( 1980). An analysis of coping in a middle aged community sample. Journal of Health and Social Behavior. 22. 457-459. Gay, M.L., Hollandsworth, J.G. and Galassi. J.P. (1975). An assertiveness inventory for adults. Journal of Counselling Psychology, 22. 340-344. Geer, L. (1988). Adolescent health survey (American Indian version). Minneapolis, MN: University of Minnesota Adolescent Health Program. Holmes. T. and Rahe. R. ( 1967). The social readjustment rating scale. Journal of Psychosomatic Research. 11, 213-218. .::Di:::I\JDIX ,! '.::i:SGi?Tl<::N c;: ~ELEGED INSTRUMENTS USED i~-1 LIFE SKILLS PROGRAMME EVALUATION ::age OS Example 2: Instrument used to evaluate a programme for drug abuse prevention Summarized from: Botvin. G.L Baker. E.. Dusenbury, L.. Tortu. S. and Botvin. E.M. ( 1990). Preventing adolescent drug abuse through a multi-modal cognitive-behavioural approach: results of a three year study. Journal of Consulting and Clinical Psychology, 58(4), 437-446. Measurement of cigarette smoking. A 10-point scale was used to measure smoking status. Response categories consisted of: 1 (never), 2 (once in the last 12 months). 3 (a few rimes in the last 12 months), 4 (usually once a month), 5 (a few times a month), 6 (usually once a week), 7 (a few times each week), 8 (a few rimes most days), 9 (about half a pack each day), and 10 (more than a pack a day). Measurement of alcohol use. Three items were used to assess the use of alcoholic beverages. The frequency of drinking alcoholic beverages was measured using a 9-point scale with response categories consisting of: 1 ( never), 2 ( tried them but don· t drink them now). 3 (less than once a month), 4 (about once a month). 5 (about two or three times a month), 6 (about once a week), 7 (a few times a week), 8 (about once a day>, and 9 (more than once a day). The amount of alcohol consumed per drinking occasion was measured using a 6-point scale and the frequency of getting drunk was measured on a 9-point scale. Measurement of marijuana use. Frequency of marijuana use was measured using a 9-point scale with response categories of: 1 (never), 2 (tried it, but don't use it now), 3 (less than once a month), 4 (about once a month), 5 (about two or three time a month), 6 (about once a week), 7 (a few times a week), 8 (about once a day), 9 (more than once a day). Knowledge measures. Three I 0-item true/false measures were used to assess knowledge about the immediate and short-term consequences. prevalence. and social acceptability of tobacco, alcohol and marijuana use ( Botvin et. al .. 1984 ). Measurement of attitudes and normative beliefs. Students' attitudes about tobacco. alcohol and marijuana use. the characteristics of users. and about the perceived social benefits of using these substances. were assessed using three measures. Smoking related statements were derived from the Teenager's Self Test: Cigarette Smoking (US Public Health Service, 1974). Similar statements, based on those related to smoking, were prepared to investigate attitudes to alcohol and marijuana. For items about attitudes, responses were indicated by means of a 5-point Likert scale ranging from strongly agree to strongly disagree. Responses about normative beliefs were rated on a 6-point scale from none to almost always. Measurement of skills. A measure of decision making assessed the use of sound decision making skills. e.g. "When I have a problem I get information that is needed to deal with the problem". Seven items of this nature were used to assess decision making, with responses ranging from never to almost always. Five items were used to assess the use of different skills for coping with anxiety. Each item described a relaxation technique. Frequency of use was assessed on a 5-point scale ranging from never to almost always. Both scales were derived from the Coping Inventory developed by Wills ( 1986 ). Assertiveness was assessed using an abbreviated 18-item version of the _'FE SKILLS ~i:::.UCATION· Pt.ANNING FCR RESEARCH =--:iqe 06 Gambrill and Richey ( 197 5) Assertion Inventory, with responses ranging from never to almost always. Asseniveness was assessed in terms of both resisting substance abuse offers and general asseniveness. Respondents confidence about their ability to use specific personal and social skills (skills-efficacy) was measured with 14 items rated on 5-point Likert scales. ranging from not at all confident to very confident. Sixteen true/false items were used to measure knowledge about communication skills and general social skills. Measurement of psychological characteristics. This included measurement of self-efficacy, self- esteem and social anxiety. Self-efficacy was measured by 5 items indicating the belief that goals could be achieved through personal effon and hard work (Paulhus. 1983). Responses ranged from strongly disagree to strongly agree. Self-esteem was measured by a 10-item scale developed by Rosenberg (1965). Statements were typical evocations of self-esteem (eg. "I take a positive attitude toward myself'. "I feel that I have a number of good qualities"). Responses ranged from strongly disagree to strongly agree. Social anxiety was measured by 9 items derived from the Social Anxiety Inventory (Richardson and Tasto. 1976). Statements described various social situations (eg. giving a speech in front of a group of strangers. speaking to someone in authority, and expressing a conrroversial opinion in a group). Responses assessed the amount of nervousness felt in each situation from very nervous to not at ail nervous. References Botvin. G.L., Baker, E., Renick, N., Filazzola, A.O. and Botvin, E.M. (1984). A cognitive- behavioural approach to substance abuse prevention. Addictive Behaviors, 9, 137-147. Gambrill, E.D. and Richey, C. (1975). An assertion inventory for use in assessment and research. Behavior Therapy, 6. 550-561. Paulhus. D. ( 1983). Sphere-specific measures of perceived control. Journal of Personality and Social Psychology, 44, 1253 - 1265. Richardson, F.C. and Tasto, D.L. (1976). Development of a factor analysis of a social anxiety inventory. Journal of Behavior Therapy, 7, 453-462. Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ. Princeton University Press. US Public Health Service (1974). Teenager's self-test: Cigarette smoking (DHEW Publication No. CDC 74-8723). Bethesda, MD: US Department of Health. Education and Welfare. National Clearing House for Smoking and Health. Wills, T. (1986). Stress. coping, tobacco and alcohol use in early adolescence. In Shiffman, S. and Wills, T.A. (Eds.) Coping and substance use (pp. 67-94 ). New York: Academic Press. ~;;PENDIX ii S::SCRlPTiCN CF SELECTED INSTRUMENTS USED IN LIFE SKILLS FROG,-AMME E'·,ALUATION cage o7 Example 3: Instrument used to evaluate a programme to promote social adiustment and prevention of alcohol abuse Summarized from: Caplan, M., Weissberg, R.P., Grober. J.S .. Sivo. P.J .• Grady. K. and Jacoby, C. (1992). Social competence promotion with inner-city and suburban young adolescents: effects on social adjustment and alcohol use. J ourna/ of Consulting and Clinical Psychology, 1992, 60 (I), 56- 63. Measurement of coping skills. Two skills-assessment instruments were administered. An alternative solutions test. adapted from the Decision Making Questionnaire (Gerswick et al, 1988), evaluated the quantity and effectiveness of solutions to a hypothetical vignette involving peer pressure to smoke. Students were to list the alternatives that they would try when faced with the hypothetical situation. Degree of effectiveness for each of the strategies were coded using a 4-point scale ranging from I (not very effective) to 4 (very effective). The second measure asked students to list the different things they do to calm themselves down when they feel anxious or stressed. Adaptiveness of the strategies were coded using a 4-point scale ranging from I (very ma/adaptive) to 4 (very adaptive). Measurement of social and emotional adjustment. A teacher rating scale. developed by Allen et al (1989) provided an assessment of students' school behaviour. This measure requires classroom teachers to rate how well each of four items describes the student on a 5-point scale from I (,wt at all) to 5 (very well). The four items pertain to constructive conflict resolution with peers. impulse control. popularity and assertiveness with adults. A Behavioural Conduct scale and Self-Worth scale were used to assess the children's perceptions of competence in these domains. The scales were taken from The Self-Perception Profile for Children (Harter, 1985). Students' general mood and emotional state was measured using the Rand Well-being Scale (Veit and Ware. 1983). This measure consists of 12 items that ask students to judge, on a 5-point Liken scale, how frequently they experience feelings such as loneliness. restlessness. and sadness. Students' expectations for success in relation to difficult decisions were assessed using a 4-item version of the Decision- Making Confidence Scale (Wills. 1986). This measure required students to rate on a 5-point scale how strongly they agree with four statements. Students were asked to indicate on a 4-point scale ranging from I (not al all) to 4 (very) the usefulness of each of the four problem solving steps (introduced in the programme) when facing stressful situations. Measurement of intentions, attitudes and se[f-reponed substance use. Students were asked to indicate what they would say if a friend offered different substances (cigarettes, beer, marijuana. etc.). The response choices were based on a 5-point scale from definitely 110 to definitely yes. Two 10-item scales (developed by Botvin et al, 1984) were used to assess students attitudes towards smoking and drinking. Students rated on a 5-point scale how much they agreed with 10 statements concerning the attractiveness of smoking and drinking. Substance use was assessed using a measure adapted from other surveys (Grady et al. 1986; Kandel et al. 1978). Students were asked how often they had used cigarettes. beer etc. in the past 2 months (ranging from never to two or more times a week). A fake drug was added to the list of substances to help detect over-reporting of use. as well as students who did not treat the survey seriously. Items were also included to determine excessive use of alcohol. Students indicated on a 6-point scale "how much liquor they usually drank at one time". . Fi: SKILLS EDUCATION· P'...-4-NNING i=CR RESEARCH .::".lgeo6 Measurement of attitu.des towards the programme. Students completed a consumer satisfaction survey assessing how much they liked. learned. and applied what they learned in The Positive Youth Development Program References Allen. J., Weissberg, R.P. and Hawkins, J. (l 989). The relation between values and social competence in early adolescence. Developmental Psychology, 25, 458-464. Botvin, G.L., Balcer, E .. Renick. N .. Filazzola. A.D. and Botvin, E.M. ( 1984). A cognitive- behavioural approach to substance abuse prevention. Addictive Behaviors, 9, 137-147. Gerswick, K.E., Grady, K. and Snow. D.L. ( 1988) Social-cognitive skills development with sixth graders and its initial impact on substance use. Journal of Drug Education. 18, 55-70. Grady, K .. Gerswick. K.E. and Snow, D.L. (1986). The emergence of substance use. Journal of Drug Education, 16, 203-220. Harter, S. (1985). Manual for the self-perception profile for children. Unpublished manuscript, University of Denver. USA. Kandel, D.B., Kessler, R.C. and Margulies, R.Z. ( 1978). Antecedents of adolescent initiation into stages of drug use. In D.B. Kendel (Ed.) Longitudinal research 011 drug use: Empirical findings and methodological issues. New York: Guildford Press. Veit, C.T. and Ware. J.E. (1983). The structure of psychological distress and general well-being in the general population. Journal of Consulting and Clinical Psychology, 51. 730-742. Wills, T.A. (1986). Stress and coping in early adolescence: Relationships to substance use in urban school samples. Health Psychology, 5. 503-529. ~PPENDIX II CcSCf<,PTICN CF SE:..E·:'!E) INSTRUMENTS USED l~~ LIFi: SKILLS PROG~AMME EVALUATION ?age o9 Example 4: Instrument used to evaluate a programme for promoting social competence Summarized from: Elias. M.J. Gara. M.A.. Schuyler, T.F .• Branden-Muller. M.A. and Sayette. M.A. 11991 ). The Promotion of Social Competence: Longitudinal study of a preventive school-based program. American Journal of Orthopsychiacry, 61 (3), 409-417. The children were asked to complete an anonymous questionnaire to help Rutgers University learn more about high school students. Assessment of social competence - was based on the social and activities scales of the Youth Self Repon (YSR) rating scale (Achenbach and Edelbrock, 1987). Assessment of psychopachology - was based on the clinical scales of the YSR. The YSR taps 6 factors: depression. unpopularity. aggression. delinquency, somatic complaints and thought disorders. An additional factor, involving self-destruction/identity problems was scored for boys only. The YSR also assesses social competence, with items such as: How well do you get along with your parents and peers? Assessment of self-efficacy - was based on scores on the Perceived Competence Scale for Children (see Harter. 1982; Hughes. 1984). This is a 28 item measure of students' self- efficacy, covering the domains of cognitive-academic. social, physical-attractiveness and self-worth. Assessment of antisocial and self-destructive behaviour - was based on scores on the National Youth Survey (NYS) of antisocial and delinquent behaviour (Elliot et al. 1983). The NYS consists of 35 items covering: vandalism: theft: use of various illegal substances: aggressive behaviour: school disciplinary problems: etc. Each item is scored on a 7-point scale representing frequency of occurrence over the past year. Assessment of academic performance - was based on a test of basic academic skills. Assessment of students' background. Background information was obtained by asking students about their elementary school history and about the presence, or absence, of 10 life events-stressors since they left elementary school (based on Johnson, 1986). References Achenbach. T. and Edelbrock. C. ( 1987). Manual for the Youth Self-Report and Profile. Burlington: University of Vermont Department of Psychiatry. Elliot, D .. Ageton, S .. Huizinga, D., Knowles, B. and Canter. R. (1983). The prevalence and incidence of delinquent behal'iour: The National Youth Survey (Report No. 26) Boulder, CO: Behavioural Research Institute. -'FE SKILLS EDUCATION· Pi..A.NNING FOR RESEARCH ::;age 70 Harter. S. (1982). The perceived competence scale for children. Child Development. 53, 87-97. Hughes, H. (1984). Measures of self-concept and self-esteem for children ages 3-12 years: A review and recommendations. Clinical Psychology Review. 4. 657-692. Johnson, J.H. (1986). Life events as stressors in childhood and adolescence. Beverly Hills: Sage Publications. .... ?PENDIX II DESCRIPTION OF SELECTED INSTRUMENTS USEu ;N LIFE SKILLS FROG~AMME EVALUATION ;:age 71 Example 5: Instrument used to evaluate a programme for violence prevention Summarized from: Chimenti, G. and Trilivas. S. (1994). Cognitive and behavioural effects of social skills training on Greek and Lebanese elementary school children. lmernational Journal of Mental Health, 23 (4), 53-68. Assessment of attitudes. A 22-item scale was devised to assess cognitive aspects of getting along with others. Statements relating to the content of the Conflict Control Programme (CCP) were prepared, and subjects asked to respond on a scale from agree a lot to disagree a lot. For 11 items, agree a lot was consistent with the goals of CCP; for 11 items. disagree a lot was consistent with CCP goals. Items were read aloud and children circled their responses in a booklet containing the response scales for each item. Assessment of behaviour. Three aspects of behaviour were assessed using a peer nomination procedure: aggression. prosocial-antisocial behaviours. acceptance-rejection by peers. To assess aggression. the peer assessed aggression scale. developed by Walder et al (1961 ), was used. Behavioural descriptions were used to assess prosocial and antisocial behavioural patterns, including easily/not easily angered, considers/does not consider others rights and feelings, creates/avoids trouble. etc. Assessment of status amongst peers. Children were classified as accepted. rejected or average, based on peer nominations. For peer nominations, children were asked: Whom would you like for your best friends? and Who do you wish was not in your class? References Walder, L.O .• Abelson. R .. Eron, L.D .• Banta, T.J. and Laulicht. J.H. ( 1961 ). Development of a peer rating measure of aggression. Psychological Reports. 9, 497. 5E SKILLS ~i)UCATION i=Lt..NNtNG FOR RESEARCH -=oge 72 Example 6: Instrument used to evaluate implementation of a life skills programme in Switzerland This list is summarized from a questionnaire given to teachers who had panicipated in training in the use of the life skills programme Objectif Grandir, and who had implemented the programme in schools. The questionnaire items covered the following questions: • Which activities described in the programme materials did the teacher implement? • How frequently did the teacher implement the programme? • Which activities were difficult to implement, which were easy, and why? • What help did the teacher need to implement the programme activities? • Are there parts of the programme that the teacher definitely did not want to implement, and why? • Which methods used to implement the programme did the teacher find difficult. and why? • Did the teacher notice any changes in the general functioning of the class? • Was it possible to observe changes in the students, if so, what were they? • Have other people been invited to assist in the classroom activities? • Has the teacher had contact with other teachers to discuss the implementation of Objectif Grandir? • Does the teacher think that the programme had any effects on the atmosphere in the school? • Has the teacher met with parents to discuss the objectives and methods of Objectif Grandir? If so, what did they do? • Did the programme help the teacher to have better contact with parents? • What did the teachers think of the training that they had? • What training would they like in the Jitture?
Organisation mondiale de la santé (OMS) · Technical Documents
Life skills education: planning for research as an integral part of life skills education development, implementation and maintenance
Voir le document original
Le texte intégral est hébergé par l’organisation qui le publie. lawenc.com indexe les métadonnées et renvoie vers la source officielle.
Texte intégral
Informations clés
Organisation
Organisation mondiale de la santé (OMS)
Type de document
Technical Documents
Source
Organisation mondiale de la santé