World Health Organization document is not a (WHO), and all rights are abstracted, "'.rI use in conjunction with 'I"£u....
be translated, in part or in whole, purposes. by authors are solely the 1""''''''''1'\1''1'''
views expressed in authors.
WHO/MCH-FPP/MEP/93.1
Distr.: General
CONTENTS Page I.
INTRODUCTION GENERAL OBJECTNES OF THE RAPID EVALUATION .. . . . . . . . . . . . . . .. ADV ANTAGES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. IMPLEMENTATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1. Phase 1: Planning......................................... 1.1 Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Activities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1.2
1 1 3 3 3 3 4 4 4
II. Ill.
N.
1.2.1 1.2.2 1.2.3 1.2.4 2.
Designation of a REM manager/coordinator and of a core technical group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Identification of major issues ........................ Construction of an issue-information matrix ............. REM proposal document . . . . . . . . . . . . . . . . . . . . . . . . . ..
5 5 7 7 7 7 9
Phase 2: Prepara tion ...................................... 2.1 Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. Activities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 2.2
2.2.1 2.2.2 2.2.3
2.2.4 2.2.5 2.2.6 2.2.7 2.2.8 2.2.9 3.
Review and finalization of issue-information matrix . . . . . . .. Preparation of data collection instruments . . . . . . . . . . . . . .. Sampling....................................... Preparation of training manual ...................... Preparation of data analysis and data entry formats ........ Testing of data collection instruments ................ ,. Revision of survey instruments ...................... Logistics ...................................... Report outline ............................ ,."..
9 10 10 10 11 11 12 13 13 13 13 13 14 14 15 15 15 15 15 15 16 16 16
Phase 3: Field data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 3.1 Objectives .......................... , ............ , .. 3.2 Activities· . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..
3.2.1 3.2.2 3.2.3 3.2.4 4.
Training the data collection teams ..............•..... Field data collection .............................. Data processing and analysis . . . . . . . . . . . . . . . . . . . . . . .. Preliminary report ...............................
Phase 4: Completion ...................... ,., ............ ,. 4.1 Objectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 4.2 Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..
4.2.1 4.2.2 4.2.3 4.2.4 4.2.5 5. ANNEXES 1 2 3 4
Organization of a seminar to present REM findings . . . . . . .. Preparation of a plan of action .. . . . . . . . . . . . . . . . . . . . .. Further data analyses ................ , ........... , Preparation of final report and dissemination of results , . . . ., Implementation of plan of action ............... ;.....
Institutionalization of the REM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..
An example of terms of reference for a rapid evaluation of MCH/FP services ......................,..,........,.................... An actual issue-information matrix developed in one country .............. The use of Epi Info .............................................. Focus group technique .......... , .................... , ....... , .. ,.
19 ·22 23 24
PREFACE
In health care systems of developing countries the attention of health planners and programme managers has hitherto mostly focussed, on coverage since there was a real gap in the area of prevention and health promotion in general. The health policies and programmes inspired by the Global Strategy for Health for All by the Year 2000 have succeeded in achieving real progress in disease control and in facilitating access to health care for a wider segment of the population in most developing countries, although there are still many disparities (or inequities) between and within countries. In more recent years countries have become increasingly concerned with quality of care as part of a general concern about the efficient use of health resources. Good quality care is also seen as an effective "catalyst" for social mobilization, health promotion and development. Emphasis is increasingly placed on programme monitoring and evaluation. This has enhanced the development of a new "management culture" which is emerging in many countries and in principle is being widely accepted. In the area of maternal and child health and family planning, for example, tremendous progress has been achieved in developing countries particularly in the last two decades, in terms of service coverage. While infant mortality has dramatically decreased, infant and young child morbidity, maternal mortality and morbidity, low levels of family planning acceptance remain a major concern. Many of these problems could be more appropriately addressed through better quality of care and better management of staff and services. The development of the REM began with earlier successful use of rapid techniques in evaluating the coverage and operational aspects of the national EPI programmes. These were later extended to the assessment of progress in other elements of primary health care. The format of the EPI, Primary Health Care reviews were "pre-set" with minor adaptations to country circumstance. It was soon apparent that the structure of these reviews was too rigid, and did not correspond to the varying managerial needs of the maternal and child health and family planning programmes. In response to such needs a more flexible and fully participatory method was developed, which is now referred to as the Rapid Evaluation Methodology (REM). The Rapid Evaluation Methodology (REM) by focussing on staff performance and quality of care, aims at filling such management gaps in maternal and child health and family planning programmes (MCH/FP), and in other health services as well. Its objectives and benefits are described in the guidelines. The REM has been developed through an interregional project supported by the United Nations Population Fund (UNFPA). The application of the District Team Problem Solving Appropach (DlPS) was also developed under the same project. Both are intended to strengthen management in MCH/FP programmes as well as to other health services. The REM has been developed to become a routine tool for service management to improve the quality of care, programme performance and enhance community involvement. In this respect, country participants in an interregional meeting on "strengthening management in MCH/FP programmes", held in Geneva in 1991, recommended that the REM should be integrated into the national health management systems. These guidelines have been prepared by Dr Habib Rejeb in close collaboration and in a spirit of team work with other WHO staff members from the Division of Family Health and the Division of Epidemiological Surveillance and Health Situations and Trend Assessment and who have participated in country evaluations using the REM process. Dr Steve Sapirie who initiated the development of the REM process, Dr Richard Guidotti, Mrs Martha Anker, Dr Stanislaw Orzeszyna and Dr Michel Thuriaux reviewed the draft and provided expert contributions which enriched the content and the practicability of the guidelines. The World Health Organization is grateful to a number of individuals, WHO staff members and national programme managers, who helped to develop and implement the REM. Special credit should be given to Dr Mark Belsey who advocated and supported the REM process and its philosophy from its inception. WHO also expresses thanks to Dr Anthony Radford who provided an early draft of the guidelines.
WHOIMCH-FPPIMEPI93.1
I.
INTRODUCTION
The rapid evaluation method (REM) builds on experience gained through the evaluation efforts of some WHO programmes and activities, namely the expanded programme on immunization, the diarrhoeal diseases control programme and the primary health care reviews. These used standard questionnaires and concentrated mainly on the incidence of selected conditions, the coverage of selected services, the availability of resources, and on support and policy. In contrast to these evaluation methods the REM assesses service performance. The REM gives prime consideration to the quality of care provided and to client satisfaction. It uses selected quantitative and qualitative indicators in various health facilities as well as in the community to measure the impact of services and the community's perception of them. The REM is a participatory and motivational approach to evaluation in which decision makers, programme managers, trainers and health service providers from different levels of the health care system work together on a rapid and comprehensive assessment of the health service situation in a given programme or in selected components of it. The REM has been designed to assist ministries of health and provincial or district health authorities to evaluate maternal and child health and family planning (MCH/FP) services or other components of their health services as well as to help plan a programme of activities aimed at improving the coverage and quality of those services and the participation of the community. The REM does not replace - but rather complements - other methods of collecting epidemiological information such as patterns of mortality and morbidity.
II.
GENERAL OBJECTIVES OF THE RAPID EVALUATION (a) To provide reliable information about service performance in order to strengthen the overall management of MCH/FP programmes or any other health programmes with emphasis on quality of care, staff performance and client satisfaction. To provide countries with an evaluation/management tool that can be adapted for use at all levels of the health care delivery system without additional resources and through use of local expertise. To train MCH/FP or other programme managers and service providers, through the REM, in the process of evaluation design, implementation, data analysis and use of results for action.
(b)
(c)
WHOIMCH·FPPIMEPI93.1
Overview of REM
PHASE
STEPS
PRODUCT
1.
Planning
-
-
2.
The Ministry of Health decides to conduct a REM and defines its objectives Designation of a REM manager and core technical group Identification of health problems and related major issues and service levels for the evaluation Preparation of REM proposal document Finalization of issues to be addressed· Review of existing data Identification of information to be collected· Choice of methods to be used· Finalization of issue-information matrix Preparation of survey instruments and instructions manual Testing and revision of survey instruments Selection of survey sample Composition of data collection teams Training of data collection teams Field data collection Data analysis Preliminary report Seminar to discuss REM findings Completion of data analyses and final report Completion/finalization of plan of action Distribution of final report to all concerned Provindalldistrict workshops to discuss REM findings and implications Implementation of planned activities Evaluation of progress and expected changes Steps taken for REM institutionalization
REM objectives or tenns of reference defined List of designated staff List of major issues to be evaluated Draft proposal document prepared List of final issues List of relevant existing data List of information to be collected Methods selected Issue-information matrix Draft data collection instruments Revised data collection instruments Sample selected Data collection team members identified Teams trained Raw data Preliminary results Report completed Seminar report Final report completed Final plan of action Final report distributed Workshop reports Monitoring reports Evaluation report Progress report
Preparation
-
-
3. Field data collection and analysis Completion
-
-
4.
5. Follow-up
-
•
all within the preparation of the issue-information matrix 2
WHOIUCH.fPPIMEPI93.1
III.
ADVANTAGES Benefits that may result from a REM exercise will include: • • • • • • • comparing service practices with existing policy; raising staff appreciation of strengths and weaknesses of services and motivation for a continuing improvement of the quality of care; getting the staff to appreciate the need for thorough collection of information and the importance of its reliability for sound management of activities and programmes; providing an opportunity for staff to see how information can be used in the health system to determine deficiencies in need of correction; increasing the capacity of staff to undertake the design, conduct and analysis of their own evaluations; identifying deficiencies in knowledge, technical or managerial skills or inappropriate attitude; ensuring relevance of the evaluation to local or national concerns (e.g. by encouraging the tailoring of evaluations to address the concerns of decision-makers and programme managers).
The REM provides a comprehensive view of the issues addressed and of the health care system from many perspectives (such as these of the managerial staff, the health care workers, the clients and the community at large).
IV.
IMPLEMENTATION
The implementation process described below is based on experience gained from use of the REM in several countries. Each phase in the process requires a number of steps to be taken. Usually several of these steps are carried out Simultaneously. When implementation of the REM does not require assistance from external facilitators and the methodology is institutionalized and used at the different levels of the health care system, the division of the process into phases becomes artificial. In such a situation the REM group is expected to carry out the evaluation as rapidly as possible, and usually without interruption, from initiation to completion. The follow-up would be part of the routine mOnitoring and evaluation activities. For example, a district health team or the personnel of a health care unit may wish to use parts of the REM to evaluate some aspects of a specific health care activity.
1.
Phase 1: Planning 1.1 Objectives
(1 week)
The objectives of the planning phase are: • • • to appoint a REM manager/coordinator and a core technical group; to identify the major issues to be evaluated; to produce a REM proposal document, including terms of reference.
3
WHOIMCH-FPPIMEPI93.1
1.2
Activities
1.2.1
Designation of a REM manager/coordinator and of a core technical group
Whenever the REM is carried out centrally there is particular need for a high-level policy maker or programme manager to sponsor the process. The sponsor appoints a senior officer as manager I coordinator of REM implementation who is in turn supported by a core technical group that is designated to identify the major issues. The REM managerI coordinator must be a person of considerable experience and seniority who has the necessary leadership qualities to sustain the impetus of the REM and prepare the ground for its routine application at different levels of the health care delivery system. Ideally the manager should have had prior exposure to a REM (in a nearby country, for example). The core technical group is a senior working party of about five members with experience in one or more of the issues to be studied, including administration, clinical care, data gathering and analysis. In the second phase the group is enlarged to about 10 members, including one person from each level of service to be evaluated, and can co-opt other specialists for specific tasks as necessary. In all situations, efforts should be made during the first REM exercise with external technical support to ensure a full transfer of the methodology to national staff. Specialists in qualitative methods such as focus group discussion, data processing and analysis should be identified as appropriate in each country to be part of the REM working group. This will bring the advantage of facilitating rapid implementation of the REM process and full transfer of the methodology to national staff. Any commitment less than this would not be conducive to selfreliance and to institutionalization of the methodology. The REM managerI coordinator and the core technical group are responsible for the development and conduct of the REM through to its completion. They must also prepare a plan of action based on the findings and recommendations of the seminar at which REM findings are discussed.
12.2
Identification of major issues
The initial decision to conduct a REM is usually taken by the Ministry of Health on the basis of a concern to improve, for example, the quality of services and the health status of particular population groups such as women, children or adolescents. This decision must be motivated by a felt need for action and by a commitment to use the findings as the basis for taking appropriate corrective measures. To meet specific objectives, a single REM exercise should focus on a few major issues in maternal and child health and family planning or other components of primary health care. Therefore a limited set of issues should be chosen. Issues are a combination of specific health problems, public knowledge and behaviour, and service performance. The next step is to clarify how the data that is generated will be used subsequently. It may be decided to look broadly at such issues or more specifically at representative service activities such as prenatal care, delivery care or family planning.
4
WHOIMCH·FPPIMEPI93.1
Some areas of concern may be targeted for special attention in a REM. For instance, skills and performance of specific types of staff or management procedures at a particular level might be selected for evaluation. It helps if the areas of concern are expressed as a series of objectives that are as specific as possible. This precision makes it: • • • easier to select the major representative activities for the evaluation easier to formulate the right questions easier to determine the methods to use.
1.2.3
Construction
of an issue-information matrix
Each REM group should use an issue-information matrix (see Annex 2) as the basis for planning its evaluation. The first step in this exercise is to determine the major health problem(s) and related delivery aspects that are to be addressed. Often a broad health area, such as maternal health, is selected and within this more specific problems are defined (e.g. obstructed labour, family planning, abortion, hypertension). Aspects of health service delivery to be studied are then identified, such as community involvement, training of staff and their technical and managerial performance, the quality of health facilities and the availability and use of resources. The process of developing the issue-information matrix leads to the identification of relevant issues such as community satisfaction with the services, knowledge and skills of staff in identifying risk factors, and staff adherence to defined technical and management procedures. For each issue that is identified, the group defines: • • • • the information required the data to be collected in addition to existing relevant data the service level where the data will be collected the data collection methods to be used.
The issue-information matrix helps the group to see clearly the type of information needed. Detailed elaboration of the matrix facilitates, during the next phase, the process of elaboration of the data collection instruments and helps to focus on the most relevant questions.
1.2.4
REM proposal document
Following the preparation of the issue-information matrix, the core technical group will address other aspects of the study such as sample size, budget, identification and training of the teams for field data collection, transport, and administrative support. It is imperative that a detailed budget is prepared and sources of funding secured.
5
WHO/'MCH-FPP/MEPI93.1
Overview: Phase 1 .STEPS 1.
PRODUCTS Objectives of the REM clarified by the Ministry of Health List of names
Decision to conduct a REM and definition of its objectives and expected use of findings Appointment of the REM manager/coordinator and the core technical group Identification of major issues and service levels to be addressed Construction of issue-information matrix Preparation of a detailed budget and identification of funding sources Identification of members of the REM working group and other resource persons including facilitators Preparation of REM document including a time frame for completion
2.
3. 4. 5. 6.
Draft list of issues Levels of service for the evaluation Issue-information matrix Detailed budget List of REM group members, resource persons and facilitators REM document
7.
At the completion of this phase the REM manager and the core technical group produce a document that includes terms of reference for the evaluation and gives the following details: • • • • • • • • • • reasons for conducting a REM and subsequent decisions; objectives of the evaluation and uses of the required data; selected issues and service levels of the evaluationi scale of the evaluation (national, regional, provincial, district); issue-Information matrix (if completed); a time frame for completing the evaluation; procedures for selecting and forming survey teams for field data collection (category, place, responsible officer); logistics (transport, lodging)i administrative support; schedule for each of the following phases with a target date for REM results and the preliminary report; budget and sources including fuel cost and any other costs related to transport according to local circumstances, subsistence allowances for all the phases including training costs for the field survey teams as may be arranged, costs for data collection instruments (including survey team training manual) development and printing, costs for administrative support, costs for data processing and analysis and for report preparation and printing, costs for seminar to discuss findings, preparation of plan of work, further data analysis, preparation, printing and dissemination of final report, 6
WHOIMCH-FPP.\IEP/93.1
<:osts for follow-up and review of progress. 2.
Phase 2: Preparation (2-3 weeks)
At this stage the core technical group is enlarged to include representatives from the different service levels to be evaluated. These persons should have clinical as well as managerial responsibilities. The inclusion of one or more statisticians from the Ministry of Health is also recommended. However, it is very important that the group includes someone familiar with micro-computer use and software (EPI INFO) for data analysis. If qualitative methods such as focus group discussion are to be used, the REM group must also include a professional with such expertise and the REM manager should make the necessary inquiries to identify such an expert from another sector if necessary.
2.1
Objectives
For the preparation phase the objectives are as follows: • •. • • to finalize the issue-information matrix including service levels and data collection methods; to design the questionnaires and prepare the instruction manual; to prepare software applications; to prepare the report outline.
2.2
Activities
2.2.1
Review and finalization
of issue-information matrix
Before the review the REM manager explains to the REM enlarged group the purpose of the evaluation, the expected outcomes and how these will help decide on what subsequent actions should be taken. The group, with the help of the facilitator(s), then proceeds with the review of the issues-information matrix, completes a list of the data required for each issue to be addressed and identifies the sources of that data. The review is carried out in five steps. First Second: Third: review and amendment as necessary of the major health iss'!les and areas of concern as set out in the initial issues-information matrix. review of the major service activities to be evaluated at the selected service levels. confirmation of the aspects of care to be evaluated, such as task performance, service management, availability of critical resources and client and/or staff knowledge, attitudes, skills and satisfaction. review of existing data and Its quality and relevance to selected issues and concerns selection of the most appropriate methods for each issue and type of da ta to be collected.
Fourth: Fifth:
7
The methods frequently. used for data collection are: • • • • • • • record review and data extraction, (e.g. obstetric records, family planning records, antenatal records); direct observation of tasks; checking equipment, drugs, supplies (availability, condition); health personnel interviews; exit interviews with clients (after clinic visits); focus group discussions (e.g. with staff, community target groups, community leaders); household interviews (least recommended).
Record review. The questionnaire provides a checklist of tasks the staff should carry out for each service activity being assessed. The survey team members examine a sample of records to check if the recording of tasks has been done correctly. H items are missing, this is investigated during the analysis to find out the possible causes. These may be related to training, neglect or unavailability of required equipment and supplies. Observation of tasks. This is an effective way of comparing staff performance with pre-set standards. The observer sees how staff perform in terms of communication and clinical skills. Although observation itself can influence performance, it provides additional information on quality of care. This method requires observers to be carefully selected and trained in order to produce reliable and consistent information. Checking facilities, equipment and supplies. The questionnaire checks the physical condition of health facilities, as well as the availability and condition of equipment and supplies that are important for the provision of services chosen for review. Some items of equipment may be available but are not in working condition. Essential drugs for antenatal care may be there but are beyond their expiry date. Such observations are carefully recorded. Exit interview with clients. Before the questionnaire is designed the REM group reviews technical procedures for the services addressed in the evaluation and records used during service delivery. The questionnaire is designed to find out what actually happened during the client-staff encounter. Patients are interviewed immediately following consultation in order to obtain an indication of the content and quality of care and the client's perception of the services. The client's spontaneous recall of specific procedures carried out by the staff during the visit and correct understanding of health education messages received provide qualitative information on the attitude of staff, their training and their communication skills. Focus group discussions. These semi-directive discussions with specific population groups or staff categories provide qualitative information which complements information obtained by other methods. The discussions also provide insight on the perception and attitude of concerned groups vis-a.-vis issues addressed in the evaluation (see Annex 4) or of people who do not use the service.
8
WHOIMCH-FPPIMEP!93.1
2.2.2
Preparation of data collection instruments
The REM does not use standardized instruments. In each country the REM group designs the instruments for each type of data needed and each level of service assessed. Dummy tables are also designed. Before drafting the questionnaires the group defines both the standards against which assessments are to be made and the level of skill expected. Each REM group must identify tasks that are representative of the processes it wishes to assess. Only critical tasks should be retained. The issue-information matrix provides the basis for the data requirements from each service level as well as the method of collection_ Next the REM group divides into sub-groups to prepare drafts of each data collection instrument. Members of the group who are familiar with the design of survey instruments will guide the preparation. Sample questionnaires used in other countries for similar purposes should be made available to the group at this stage. Samples of records used at the levels of service to be assessed should also be made available so that the questionnaires follow the same sequence as the records and a rapid review can be made of each. It is important, however, to ensure that the questions in a data collection instrument accurately address the issues under evaluation, are not ambiguous and can easily be understood by the survey team .. It is important to keep the number of questions to a minimum, including only those felt to be important.
2.2.3
Sampling
The REM usually consists of a number of different instruments for obtaining information from several different types of facilities, as well as from the communities they serve. The sample for study usually follows a hierarchical pattern that reflects the administrative structure of the health services. Both primary and higher level facilities are included in the sample in order to provide information on the health services as a whole. Multi-stage cluster sampling is used for this. Depending on the administrative structure of the health service, provinces are usually selected at the first stage and districts within these provinces are chosen at the second stage. Within districts, health facilities of each type are selected at random with probability proportional to the size of the population covered by each facility. For household surveys or focus group interviews, villages within the catchment area of each facility are selected at random. This random selection of villages may be stratified by distance to the health facility in order to be sure to have respondents from villages near to the facility, as well as from villages farther away. Within each health centre several different data collection instruments may be used including observation of tasks, reviews of records and interviews with women immediately after their visit. The sampling design of each REM differs though random samples of patients and patient records are generally required. In order to have an adequate sample, between 25 and 50 records of each type (antenatal records, obstetric records, etc.) should be sampled and at least 25 patients from each service being assessed should be interviewed or observed. In some clinics, especially small outposts, it may not be possible to find enough
9
WHO/MCH·FPP/MEPI93.1
subjects. In such situations it may be best to combine results from similar facilities.
2.2.4
Preparation of training manual
A manual is produced and is used both during the training sessions and as a reference field guide. The quality of the data will depend on many factors. One such factor is the enumerators' understanding of the survey instruments, and therefore the manual contains detailed descriptions of each instrument. Each item or question should be discussed together with examples of possible responses and other survey findings. Sufficient time should be allocated for preparation of the manual. The experience and education of survey workers will vary from REM to REM. Consequently the survey team's manual will have to be adapted to different situations. For example, when health workers are employed as enumerators, which is usually the case, the manual could omit definition of basic health terminology. If this is not the case, then a glossary of health terms should be included.
2.2.5
Preparation
of data analysis and data entry formats
After the questionnaires have been finalized, computer files can be prepared for data entry. A separate data entry file is prepared for each data collection instrument. In addition, some range checks and consistency checks are required in order to minimize data entry errors. Preparing files for data entry can be a time-consuming exercise. Allow 3-4 days for this process in a moderate sized REM. The files should be completed before the fieldwork starts, in order to begin processing the data as soon as it is available.
Note: The Use of Computers in REM
There are several stages of the REM for which computers are necessary, namely during development of the questionnaire (where word processing can greatly speed up the process), during preparation of data entry files and entering the data, and during data tabulation and analysis. Appropriate hardware, software and staff to carry out the various tasks should be identified early in the REM process. Software requirements include programmes for word processing, for data entry and validity checking, and for data tabulation and analysis. Epi Info has proved to be very well suited to all these tasks (Annex 3).
2.2.6
Testing of data collection instruments
The testing of the survey instruments is done by members of the REM group at service levels similar to those selected for evaluation. The objectives of the testing include: • • checking the relevance of the questions and the quality of the information obtained; checking the standardization of the clinical records used in the different facilities;
10
WHOIMCH-FPPIMEPI93.1
• • • •
checking the appropriateness and understandability (to both interviewers and interviewees) of the language used in the questions; checking the usefulness of the instructions prepared for each question; measuring the time required to complete each questionnaire; checking how team tasks and team transport are best organized.
2.2.7
Revision of survey instruments
Following the field testing of the data collection instruments the REM group meets to review and discuss the findings and to proceed with appropriate revisions and any necessary adaptation of the questionnaires. A final version of the questionnaire is prepared and, if necessary, a translation is made in the local language to ensure consistency in the way questions are asked. A date is set for printing the final questionnaires and for making various arrangements. A responsible officer is designated for this task.
2.2.8
Logistics
At this stage administrative and logistic details should be worked out within a schedule of tasks to be carried out in preparation for the field work. These tasks include: • • • • • • • • • • selection of data collection teams according to set criteria; accommodation arrangements for the teams during field work and for their operational HQ; determination of time and place for training data collection teams; arrangements for transport (number of cars required, fuel, drivers); checking distances and accessibility of the health facilities included in the sample; designation of field work supervisors; finalization of administrative details, particularly travel and expenses; confirmation of financial arrangements; printing of questionnaires and field survey training manuals; notification of health facilities.
Note:
Although members of the REM group participate in the field work as "resource persons" it is very helpful and efficient to have district/provincial medical officers take care of the logistic aspects of the field work. This requires their participation in the training of the survey teams. 1)
2) It is important to make ,sure that the field work period does not clash with public holidays or other planned activities in the survey areas and is not restricted by other local constraints. 1
11
WHO/MCH-FPP"-4EP!93.1
Overview: Phase 2 STEPS I.
PRODUCTS REM group briefed Final selected issues for the evaluation Existing relevant information listed Additional required information listed Final issue-information matrix
Briefing of the new members of the REM group Finalization of issues to be addressed Review of existing relevant information Determination of required information Choice of methods to collect additional informa tion required Designing the sample and selecting sample facilities and/ or communities Preparation of field data collection instruments and instruction manual Design of tables and formats for presentation of results of data analyses; software preparation Testing of questionnaires and their revision Revision of instruction manual and software Selection of survey team members Revision of budget as necessary Preparation of a timetable for field work Preparation of report outline
2. 3 4. 5.
6. 7.
Sample design completed, facilities and communities selected Data collection instruments and instructions manual Tables, formats prepared (dummy tables) and software Final questionnaires Final manual and software List of survey teams Revised budget Timetable for field survey Report outline
8.
9. 10.
lI.
12. 13. 14.
2.2.9
Report outline
The REM manager or a member of the core technical group should be responsible for drawing up the report outline and other members may be delegated to assume responsibility for specific sections of the report in accordance with their respective skills. There is no standard way of preparing the evaluation report. One way is to divide the report into sections related to functional activities such as management/ or training and then deal with areas of service (e.g. antenatal care, delivery care, child spacing) under each section. Alternatively, the functional activities could form the major divisions in the report. A third option is to use levels of services such as provincial, district, health centre and village as the sectional divisions.
12
WHOIMCH·FPPIMEPI93.1
In preparing the report there are some considerations to take into account: • • • • • • initial objectives and central issues; clarity and practicality of the report; readable style and format; the target users of the report; the need to produce the report quickly; the value of graphic presentations of data.
3.
Phase 3: Field data collection (3-4 weeks) 3.1 Objectives
The objectives of the field data collection phase are: • • • • to train survey teams and supervisors; to complete field data collection; to tabulate data and complete preliminary analysis; to prepare a preliminary report.
3.2
Activities
The time schedule for this phase should not exceed four weeks and actual field data collection should not take more than 10 days in most cases. Therefore appropriate preparation in phase 2 and in this phase is of paramount importance.
3.2.1
Training the data collection teams
Members of the survey teams are selected according to criteria which are determined by the type of information to be collected and the methods to be used. For example if qualitative methods such as focus group discussions are to be used it is critical (see Annex 4) to select appropriate team members and to have them trained by a person experienced in the use of this technique. Training of the survey teams is a critical activity. The quality of the information collected relies to a large extent on the appropriateness of training. The teams are briefed on the objectives of the evaluation and the methods selected for collecting information. Then they are taken step by step through each data collection instrument so that they have a good understanding of each question in the language to be used and of the instructions prepared for collecting the information. This training requires three to four days and should be as close to the real experience as possible. It should be possible, for instance, to arrange actual application of the instruments in order to learn how to use them. Team members should be made familiar with any clinical records from which information will be collected. When specific eqUipment or techniques are to be evaluated, there should be a practical demonstration during the training.
3.2.2
Field data collection
Team leaders should be familiar with local work situations in their respective areas and with clinic schedules. After training of the survey teams, the team leaders prepare the survey programme and review logistics and any possible constraints. There should be enough time to: 13
WHOIMCH-FPPIMEPI93.1
.• •
advise each health unit of the exact time of visits; identify participants (from villages) in focus group discussions (the exact time and place of each group meeting, should be known to each participant).
Before and during the data collection team supervisors should: • • • • • make sure that each team member is very clear about his/her tasks; check the questionnaires and any equipment to be used; check the logistic arrangements to make sure that work will proceed smoothly; meet with all the teams at the end of each day to assess the situation and resolve any problems related to data collection; make sure that each data set is completed before leaving each site and that questionnaires are filled in properly (checking and tallying of data should be done each night).
In the case of a large-scale evaluation, the REM manager, in order to save time, may make appropriate arrangements for team supervisors to send information collected back to the central office after careful checking and editing so that data processing can begin immediately. This will help to ensure that data analysis is done quickly and that results become available within a short period of time.
3.2.3
Data processing and analysis
Preliminary data processing and analysis starts immediately after the first results of the field survey are available. The data should be entered by a small team. The number of people engaged in entering the data depends somewhat on the number of computers and experienced staff available. There should be 2-3 computers and 4-6 people for data entry. The process should take 3-4 days of intensive work. The volume and items of data to be analyzed should correspond both to those included in the issue-information matrix and to the tabulation plans prepared in phase 2. The time frame for the analysis is tight. Preliminary analysis should be completed by the end of the first week after completion of the field work. Secondary analysis is usually undertaken later and may include multivariate analyses, such as correlations of community attitudes with service performance.
3.2.4
Preliminary report
This should include a description of the survey work, the most important data tabulations for which dummy tables had previously been prepared, a summary of major findings which correspond to the objectives and central issues, and possibly a list of recommendations. Its preparation should follow the outline prepared in phase 1. The core technical group could draft a plan of action prior to the seminar at which the findings are reported.
14
WHOIMCH·FPPIMEPI93.1
Overview: Phase 3 STEPS 1. 2.
PRODUCTS Team briefed on REM Teams trained Raw data Preliminary findings tabulated Preliminary report
Assembly and briefing of field survey teams including team supervisors Training of survey teams Field collection of data Data processing and preliminary · analyses Preparation of preliminary report
3.
4. 5.
4.
Phase 4: Completion 4.1 Objectives
The objective of phase 4 are as follows: • • • to present the REM findings to a national/regional seminar, identify main issues for detailed analysis and finalize a plan of action; to implement the recommendations and plan of action; to review progress.
4.2
Activities
4.2.1
Organization of a seminar to present REM findings
The Ministry of Health should organize a national/regional seminar, within 1-2 months of completion of phase 3, to discuss REM findings. The seminars' recommenda tions will be used to prepare or finalize the plan of action and to proceed with further analysis of issues identified as needing further attention. This effort culminates in issuing the REM final report within as a short period as possible.
4.2.2
Preparation of a plan of action
Ouring the preparation of the seminar it is recommended that the core technical group proceeds with the preparation of a draft plan of action, based on the main REM findings. This draft will then be sent, together with the preliminary report, to the participants in the seminar preferably before the seminar starts. In the process of finalizing the plan of action participants' recommendations and suggestions will be incorporated in order to improve commitment to its implementation.
4.2.3
Further data analyses
Discussion at the seminar may identify a number of issues requiring . additional analyses (cross tabulations and multivariate analyses of various types). These should be completed within one month of the seminar.
15
WHOIMCH-fPPIMEPI93.1
4.2.4
Preparation of final report and dissemination of results
Immediately after the seminar, the REM group reviews the main service and related aspects which the participants identified for focused attention, together with related data for further analysis. Then they proceed with data analysis and prepare the final report. This latter task should be carried out by the most skilled writers with support from other members of the group. The final report should highlight the most important findings which form the basis of the final plan of action. Careful attention should be given to the tables which should be easily readable. Comments about the results should be clear and conclusion& and recommendations should be realistic. The report will be distributed to participants in the seminar as well as to other managers or trainers at the different service levels. It will be used as reference by participants in local workshops to devise local strategies for implementing activities in the plan of action.
4.2.5
Implementation of plan of action
Programme managers, trainers and other directly concerned officers would be the most appropriate persons to proceed with implementation of the follow-up plan of action and monitoring of the activities. Implementation strategies may include holding provincial! district workshops to discuss the REM results. They may also include developing local plans and tactics for implementing changes and recommended activities, developing clinical procedures, clinical guidelines, supervision processes and revised consultation registers, as well as conducting district team problem solving or other activities. Overview: Phase 4 STEPS 1. 2.
PRODUCTS Seminar report Plan of action Analysis completed Report on implemented changes Progress review report
Seminar to present findings Finalize plan of action Conduct any further analysis as required Implement recommended changes in service delivery Evaluation
3. 4.
5.
5.
Institutionalization of the REM
The first few countries which used the REM recommended that it should be built into the 1.lealth care system as a complementary activity to all efforts aimed at strengthening managerial skills and improving· service performance. So far the REM has been managed at national level on a relatively large scale to provide an assessment of the MCH/FP services in the country concerned. However, if the REM is undertaken within a
, 16
WHOIMCH·FPPIMEPI93.1
decentralized health care system based on the concept of district management, it can then be used on a continuous basis by district staff to evaluate service performance and quality of care and to ensure full community participation. The use of the REM at district level would probably involve selected data capturing such as record review. The practical question is how the REM could become a familiar exercise at all service levels. The entry point, in this respect, is of crucial importance for the institutionalization of the REM and may be facilitated in the following way: Step 1:
Provide briefing to high-level decision makers in the Ministry of Health to put the REM into the prospect of health care system development. There should be an agreement of principle that the REM is not a one-time exercise, that its use should become familiar at the different service levels, that its scope can be defined according to local needs and that it can be used routinely to assess and strengthen any or all aspects of a service activity. It is obvious, however, that managers show commitment to the REM after they have tried it Identify with decision makers in the ministry the most appropriate institutions and individuals who can support the ministry in extending the use of the REM at provincial and district levels. Train a first pool of nationals through actual REM implementation. This first pool should include members with clinical, research, managerial and statistical expertise to ensure a full transfer of the REM methodology. Agree with the ministry on a programme and time frame for dissemination of the REM and facilitate discussions with funding agencies to support such dissemination at provincial level. Implement administrative measures for institutionalization of the REM at different service levels and its incorporation in the training of health personnel.
Step 2:
Step 3:
Step 4:
Step 5:
Note:
The REM database should be preserved and can be exploited from time to time. It could seroe as a baseline for future suroeys and as a model for planning health activities.
17
WHOIMCH-FPPi\lEPI93.1
18
WHOIMCH-FPPIMEPI93.1
ANNEX 1 An example of terms of reference for a rapid evaluation of MCHlFP services 1. Background
This activity was proposed as an early activity to a five-year project for strengthening MCH/FP services in country X. The project is primarily focused on. improving maternal health (reducing maternal mortality), lowering fertility through increased awareness and practice of family planning, and increasing staff knowledge of AIDS. 2. Objective of the rapid evaluation
The objective of the rapid evaluation is to establish the actual situation with respect to MCH and family planning services in terms of: • the quality of services related to maternal care (antenatal, delivery, postnatal care) family planning the provision of infonnation and service; • • the knowledge, attitude and satisfaction of clients as related to maternal care and family planning services; the knowledge, attitude, perfonnance, and job satisfaction of health service staff as related to maternal care and family planning services, and as related to training effectiveness and needs; availability, maintenance and use of critical equipment and supplies; the quality and use of record-keeping and reporting related to maternal care and family planning; the extent to which high-risk pregnandes are effectively managed, including the effectiveness of the referral system; health service staff knowledge of AIDS.
• • • •
The infonnation collected on these subjects will be used to describe the situation as the new UNFPA-supported project commences. As such it will be one basis on which the effectiveness of the project will be evaluated. 3. Subjects and levels of the service to be evaluated
The focus of this evaluation is confined to the types of services to be strengthened through the UNFPA-supported project. These services are: • • • antenatal care delivery care postnatal care
19
WHO!MCH-FPP/MEPI93.1
• • •
family planning education and promotion family planning services AIDS education.
The levels of the service to be studied are also those to be primarily addressed through the project: • • • • • • 4. community level (individual women, groups of women, TBAs and CHWs) health post health centre district hospital provincial hospital district and provincial health offices.
Uses of the rapid evaluation
In addition to providing a baseline for evaluation purposes, the information to be obtained from this rapid evaluation is to be used in a number of development activities of the project, including, but probably not limited to, the following: • • • • determining needs for procedural improvement of services, including that to be undertaken during the process of integrating MCH/FP services; determining needs for strengthening basic and post-basic training for nurses and midwives; determining requirements for critical supplies and equipment; identifying service delivery problems which warrant attention through specific problem-solving efforts, both within and outside of the UNFPAsupported project; identifying needs for strengthening recording, reporting, referral, supervision and equipment and supplies management procedures.
• 5. staff.
Participation The rapid evaluation will be designed and carried out by a core group of senior
The core group will be assisted during the actual preparation of the evaluation instruments, the field work and analysis by staff drawn from among provincial and district MCH and FP supervisors. The field work requires five or six survey teams, depending on the number of districts to be studied. 6. The evaluation area and sample
The evaluation will be carried out in randomly selected districts and facilities in the four provinces to receive priority support from this project plus one province outside the project area. As such, the evaluation will not attempt to be a national surveyor nationally representative sample, but should be representative of conditions in the selected provinces plus one control province.
WHOIMCH·FPPIMEPI93.1
One district will be randomly chosen from each of these provinces, for a total of five. Within each district the number and type of facilities and communities to be visited will be determined by the logistics and feasibility of a 5-7 day visit by teams of six staff and with two vehicles per team.
7.
Schedule of rapid evaluation preparations and conduct
It is hoped that the rapid evaluation can be designed, tested, prepared for, conducted, analyzed and the preliminary report completed within a 4-6 months period. The challenge is to complete the field work before the onset of the rains in December. If this is not possible the field work will have to be postponed until April when the weather conditions will again permit travel in rural areas.
Note: Staff support is available from WHO's, Subregional Health Development Office, Harare, and the Family Health and Health Statistics Divisions, Geneva, for activities such as design of evaluation instruments, field work, data entry and analysis and use of the evaluation results.
8.
Budget for rapid evaluation
The primary expenditure to be incurred with this activity is the cost of travel and per diem payments for the national staff who will be involved in the detailed design, field testing, training, actual conduct of the evaluation and the preliminary analysis.
21
An actual issue·information matrix developed in one country Aspeda of service HEALnf PROBLEM ANTENATAL CARE Community action Community Partidpation Attitude towards ANC Technical performance Staff KASS" Risk identification; Risk management; Administration of
Management Planning, administration Coverage with ANC care; Risk; Follow-up Information Number of supervisory visits Supervision Adequacy of training; Understanding of job description; What to add to basic and postbasic training Number of supervisory visits; VBA" supervision 50RC Number of supervisory visits; VBA supervision 50 RC FP records; Eligible couples recorded; FP coverage calculated Polio and Tl" coverage; Use of ,data to determine non-coverage; Dropout rate Number of supervisory visits; VBA supervision 50RC Number of supervisory viSits; VBA supervision
Training
Resources
Staff procedure Antenatal target set
KAPS" Attitude to referral; Knowledge of maternal deaths and high risk
tetanus toxoid; Frequency of ANC consultations RC ODS 50 EX What can be done? Deaths? Causes? Constraints; Attitude to hospital Satisfaction with performing tasks REllerrals No. of village midwives registered RC50 Immunization start; FP given; Cleek-up RC50 FP targets set Delivery records maintained; Deaths registered; Referrals recorded; Quality of records
Transport available; Budget for petrol; Availability of drugs and equipment How much done for inservice VBA training? Availability of equipment and JV>t sets, suction devices, ergometrine
DELIVERY CARE
RC50 POST-NATAL CARE Information received FP education and service; Number of PNC· RC50 No. of clinics a week; Extent to which these clinics are advertized Tetanus toxoid as part of ANC; Education; ' Administration tedmiqueof immunization ODS EX·
FAMILY PLANNING Fertility, Low coverage, Supplies lMMUNlZATION Tetanus toxoid polio
Knowledge of available FP services; Use of various methods ofFP Awareness of tetanus' and polio; Clinic schedule; Use/non-use; Knowledge of schedules Possible support; Partidpation ; Drop-out
Knowledge of FP targets; Satisfaction with FP coverage; Knowledge of eligible women Coverage; Dropout; Acceptance of policy of few contra indications
A vailability of educational materials &: contraceptive supplies Availability of working fridge,
Feedback received; Temperature of fridge recorded
50 RCODS
50RC
DATA COLLECTION MEnfOD
FG"
FG
SQ>t
various
'"
50 RCOSSO
RC50
50RC
50
ODS
·Abbreviations: ANC Antenatal care; EX Exit questionnaire; FG Focus group discussion; FP Family planning; IV Intravenous; KAPS KnOWledge, attitudes, practices, satisfaction; KASS Knowledge, attitude, skills, satisfaction; OBS Observation of health fadlity; PNC Postnatal care; RC Record check; 50 Staff interview; IT Tetanus toxoid; VBA Village birth attendant.
WHOIMCH-FPPIMEPI93.1
ANNEX 3
The use of Epi Info
Data processing and analysis is greatly facilitated by the use of Epi Info, a software developed by the Centers for Disease Contro1,l Atlanta, Georgia, and the World Health· Organization. 2 The programme and manuals are considered public domain and may be freely copied and this is in fact encouraged. Epi Info is a versatile software consisting of several modules. A word processor (EPED) can be used for designing questionnaires. The survey data are entered from questionnaires to a database (ENTER) and then analyzed using the module called ANALYSIS. The analysis module produces frequencies, tabulations, regressions, simple graphs, etc. Epi Info contains a useful feature (CHECK) which permits acceptance of specified values of variables, conditional jumps, skip patterns and linking of fields. The latest available (March 1993) version of Epi Info is Epi Info Version 5.01b which requires an mM compatible microcomputer running on MS-OOS or PC-OOS operating system, 512K of RAM and a floppy drive. A graphics board is needed to produce graphics. Results can be printed on mM or Epson compatible printers or Hewlett-Packard compatible laser printers. The number of records in data files cab be as large as the OOS and disk storage can handle. There are a few limitations on the size of the questionnaire and the variables. The number of variables is not limited but they must fit within 500 lines which is the maximum length of the questionnaire. The maximum length of a text variable is 80 characters and the total length of variables in one file cannot exceed 2048 characters. These limitations are of no practical significance for an average REM. In any case, the data can be input into two or more files which are then linked for analysis.
Dr Andrew G. Dean, The Division of Surveillance and Epidemiologic Studies, Epidemiology Program Office, Centers for Disease Control, Atlanta, Georgia 30333, USA. 2
Mr Anthony H. Burton, The Global Programme on AIDS, World Health Organization, Avenue Appia, 1211 Geneva 27, Switzerland. 23
WHOIMCH-FPPhttEPI93.1
ANNEX 4
Focus group technique1
INTRODUCfION
The Focus Group technique is an application of social psychology. For a long time the use of this technique was limited to research into social psychology, and it is only in recent years that real enthusiasm for focus groups has developed, notably in such fields as development, health, education and hygiene. With the discovery of social marketing, attention was drawn to the possibilities of qualitative research. The Focus Group is one of a set of techniques used in qualitative research, such as individual interviews, projective methods, role-playing,. simulation, etc. BRIEF OUTLINE OF THE TECHNIQUE: WHAT IS A FOCUS GROUP? It is a guided discussion that sets out to gather information and identify attitudes, opinions and beliefs within a small group. it concentrates on a specific subject and on a small popUlation sample, which has to be studied to learn about the attitudes, perceptions and behaviour of a given population. It is a qualitative research technique using a semiguided approach: the issues discussed by the group have been determined in advance by the investigator. The technique is halfway between a non-guided interview and a questionnaire. The real purpose of a series of focus groups is not to compile reliable statistics but to collect Significant data and to make an in-depth analysis o.f the feelings, perceptions, expectations and motivations of a given social group. It is a group discussion with a specific aim, its own structure, and governed by precise rules.
SPECIFIC FEATURES OF THE FOCUS GROUP
In comparison with individual interviews, the focus group technique has the advantage that it can make use of interactions between individuals, reveal the divergences, contradictions and indeed convergences within a community concerning the problems it faces, and provide more extensive information. If the subject is too complex or too sensitive to be discussed in a group, then the technique of individual interviews would be better; or if the subject could be easily exhausted in a group discussion a different research technique would be preferable. WHEN IS THE FOCUS GROUP MOST EFFECTIVE?
The focus group can be used at the beginning,. during or at the end of a research project or an activity, either on its own or to supplement other investigation methods. When used at the beginning of a research project, the focus group enables hypotheses to be established for the further conduct of the research or activity. It is an essentially exploratory process, leading to new pathways and revealing interesting opiniOns. It is particularly suitable when little is known about the subject or the study population, or if no data are available on the opinions, expectations and motivations of a community. After conducting a qualitative study it is relatively easy to compile a questionnaire for collecting quantitative data from larger samples.
1
Translated from a French version prepared by Tahar El Amouri, Director of EI Amouri Institute of Psychology, Tunis.
24
WHOIMCH-fPPIMEPI93.1
The focus group can also be applied as an evaluation technique during a programme or on evaluation mission. In this case, the results are used for direct measurement of the impact of, for instance, preventive or educational measures implemented in a community for assessing the relevance and effect of an IEe campaign (information, education, communication) and the prospects of a campaign undertaken in family planning, sanitation, etc. When used at the end of a project, the focus group provides a fuller understanding of the numerical results. Often it is not possible to interpret the quantitative data from an investigation owing to the complexity or the incomprehensible contradictions or ambiguities they contain. In this instance, the qualitative approach helps to expand the analysis and to check the assumptions made from the quantitative data. THE AIM OF THE FOCUS GROUP It is most important to place the focus group within a broader context of activity or research. In REM, the focus group is used to provide qualitative information to supplement the data obtained by other methods. It is also important to define the objectives of the focus group and to present them as dearly as possible to the various parties involved: moderators, observers, and possibly community leaders. If these objectives are not clearly defined, there is a danger of going off in wrong directions and collecting information that has no real bearing on the future course of action. It is particularly advantageous to state whether the focus group should provide working hypotheses in order to draft questions for a KAP study, to assess an awareness creation . programme, an activity already under way, or a project which has been completed.
PLANNING Once the objectives have been carefully defined, the implementation of the study can be planned, taking the resources and constraints into account. A practical way to do this is to convene the team responsible for the investigation and organize brainstOrming sessions to identify the difficulties and constraints and define the characteristics: (social, economic, cultural etc.) of the target population. By analyzing and reflecting on these data, activities can be planned and tasks appointed in a rational way. CHOOSING MODERATORS The size of the study and the availability of experts determine whether one or several moderators are appointed. What requirements must a professional moderator fulfil? Ideally, the guidance, analysis, and operation of focus groups should be entrusted to a psychologist or a psychosociologist with experience as a group moderator. A psychologist with general training would be suitable to lead focus groups after a 1-2 week training course and some practical experience. It sometimes happens that no psychologist or person of similar status is available to lead the focus groups and people without the basic qualifications for this kind of work have to be appointed. In the health sector doctors, midwives or nurses are sometimes called upon. It is important to be aware of the limitations of people who are not trained as moderators and not expect the qualitative data to be very plentiful or very reliable. If there is no other option, such personnel must be given training in order to improve their ability to listen, to get the members of a discussion group to participate in a free and easy atmosphere, to make use of the interaction of the group to find out the participants' attitudes and opinions, and to avoid exercising authority during the discussions. It is always preferable to appoint specialists external to the institution organizing the study as
25
WHOIMCH-FPPIMEP193.1
moderators of the focus groups, as identification with the institution could lead to a role conflict, both for the moderator and for the group itself. If .possible, the age of the moderator should be conducive to communication with the group. If, for instance, the research is on preventing STDs, it will be easier for adolescents to talk with a young adult, someone they consider capable of understanding their problems and unlikely to make value judgements on their sexual behaviour. The sex of the moderator is also important although perhaps less so than age. Some cultures are more tolerant than others in this respect, but, where sensitive topiCS are concerned it is advisable to choose moderators of the same sex as the participants. The ethnic group should also be taken into account when planning focus groups. It is best to appoint a moderator who is seen by the community as a neutral stranger. In certain ethnic groups which are closed societies and more mistrustful, on the other hand, it would be advisable to choose a moderator who belongs to the ethnic group in order to make them feel at ease. What has to be avoided at all costs is that the moderator is considered by the group to belong to a threatening or discredited ethnic or social group or organization.
CONSTITUTING THE SAMPLE The results of the qualitative study must enable conclusions to be drawn concerning the attitudes and behaviour of a given population. Hence the necessity to choose the participants and groups in such a way that the research findings can be generalized to a certain extent. Let us take the example of a qualitative study on family planning which has the final aim of designing and organizing a campaign to promote contraceptives. A Focus Group study will help to identify the current attitudes and behaviour of the target population. This population is not homogeneous: it includes men and women, married and single persons, women using and women not using contraceptives, people favourable to family planning and others who are against it to varying degrees, etc. This target population is surrounded by secondary targets, i.e. people with greater or lesser influence on the use of contraceptives: influential members of the family or the community (opinion leaders, legal authorities, physicians, etc.) Ideally the planned campaign should address each of these targets specifically, using the appropriate forms of language and channels. The qualitative research conducted by means of the Focus Group is, therefore, based on homogeneous groups directly concerned by the problem. There is no statistical criterion to determine the number of groups to be arranged which will depend on the number of variables that are considered relevant and have a direct influence on the studied phenomenon. It is, therefore essential to select the really relevant variables so as to keep down the cost of the study in terms of time, money and effort. Generally, a professional moderator is able to cover the different aspects of the study correctly after two or three focus groups. At a certain point the moderator comes to feel that there is nothing new to be learned from the groups, so it is pointless to continue arranging further groups and it is advisable to concentrate on thorough analysis of the data already collected. ROLE OF THE RECRUITERS The recruiters are appointed to find, contact and invite future participants in the Focus Group. They strictly apply the selection criteria by using a recruiting form. They must be thoroughly trained to apply the recruitment instructions accurately and must provide a precise estimate of the number of people who will be present on the
26
WHO/MCH·FPPIMEPI93.1
day of the group discussion. They must tell the participants who they are and who they work for and explain as clearly as possible the objective of the group meeting. Owing to a lack of personnel moderator is often asked to take over the recruiting function and any travelling incurred. This is inadvisable: the moderator should only be responsible for guiding the sessions and should avoid any social contact with the participants beforehand. The recruiters should preferably be present on the day of the group discussion to reassure the people invited who will feel more at ease when they see a familiar face. They should introduce the participants to the moderator and observer. Their role ends here because they do not participate in the discussion although they may attend the "farewell" session. SIZE OF THE GROUP
The optimum number of participants for a Focus Group is approximately eight, as this number enables the moderator to be in constant control of the situation, and to get all the participants to express themselves directly without an intermediary. GROUND RECONNAISSANCE
Whenever poSSible, it is useful for the moderator and observer to make an exploratory visit. This enables them to become familiar with the environment in which they will have to work, to contact the leaders and to visit the premises where the discussions will be held. The moderator should also take this opportunity to make preparations concerning the room, chairs, tables, and any other arrangements and facilities required. DISCUSSION MANUAL
This manual contains all the open-ended or half:-open-ended questions and states the objectives of the research, enabling free expression to take place within a predetermined framework of investigation; it is a semi-directive tool. The questions do not call for brief and precise answers, on the contrary they should stimulate quite extensive verbal reactions that convey information, hypotheses, ideas, attitudes and opinions. The moderator does not have to strictly follow the manual to the better. The manual starts with general questions and proceeds towards more and more specifiC questions, from "easy" to "sensitive". It differs from a questionnaire in that it is shorter and more general and because it does not require precise answers in a particular order. It leaves a great deal of initiative to the moderator to take up issues. Generally speaking, a separate discussion manual is prepared for each type of group. Therefore, in a research project concerning women, female adolescents, male adolescents and paramedical personnel, four discussion manuals will be compiled, some parts of which may be common to all. During the discussions the moderator may refer discreetly to the manual if necessary, but will not take notes because all the information is either recorded on tape or taken down by the observer. CHOOSING AND PREPARING THE PREMISES
The moderator and observer should arrive in good time at the place chosen for the meeting to check that the room is ready for the discussion group. The best way of arranging the room is to form a circle so that everybody can see the others easily and also to avoid giving a special status to certain seats. If this is impossible, then a semicircle or a
WHOlltCH-FPPIMEP193.1
square can be fonned; rows of seats with people sitting behind each other must be avoided. it is also recommended to find premises which are politically neutral and thus more conducive to group communication. In some cases a classroom, a room in a clinic or in a maternity centre would be suitable. PROCEEDINGS OF A FOCUS GROUP A Focus Group lasts between one and two hours, depending on the topic, on the motivation of the participants and on their ability to communicate. The success of the discussion will depend partly on how the participants are greeted by the moderator and the observer, who should introduce themselves and initiate a short welcoming conversation with the participants. When the group has assembled, everybody is invited to take a seat, in a relaxed way with no fonnalities. The first part of the meeting is taken up by expressing thanks, presenting and clarifying the objectives of the discussion. The moderator will make the group members feel at ease about the discussion being taped, and will explain that the tapes are only for use by the moderator and his or her colleagues so that they do not need to take notes during the discussion. He/she will also stress that the participants will remain anonymous, and will then suggest an activity to break the ice and produce a climate of participation from the outset. The first part of the actual discussion will be rather general, enabling everybody to participate freely. Step by step the moderator will take up the key issues of the discussion, those that will provide the first-hand infonnation that is sought. At the end of the Focus Group the moderator may briefly sum up the main issues discussed, and this sometimes sparks off an interesting further discussion. Lastly, the moderator has to end the session by thanking the group and stressing how much has been learned from it. THE MODERATOR'S BEHAVIOUR AND ATTITUDES DURING THE DISCUSSION The role of the moderator of the Focus Group could be summed up in one word: INTERACTION. In a well-led Focus Group there are exchanges between the moderator and the members of the group and also between the members themselves. As much will be learnt, if not more, from these exchanges within the group than from the dialogue between moderator and participants. A good group moderator should have above-average ability to listen. This does not mean simply the ability to register passively what is being said but the ability to listen actively to what is taking place, with a talent for decoding direct, allusive, symbolic and body language. A good listener encourages others to express themselves, because they can feel that he or she is interested in what they have to say and will not judge them but will make an effort to understand them. The moderator must also be good at recapitulating, at summing up and feeding back to the group what he or she has understood them to mean so as to make sure they are on the same wavelength. By recapitulating now and then during the discussion the moderator helps the group to advance by serial synthesis. The moderator also encourages feedback by asking individuals in the group to react to an idea or an opinion. This is an important technique that provides the moderator with a deeper understanding of the problem from the point of view of the target group. Feedback has a stimulating effect and reveals diverging opinions which are interesting to study. As the Focus Group is a semi-directive technique, the moderator will often resort to questions like:
28
WHOJMCH·FPPIMEPI93.1
"You have just raised a very interesting point, could you elaborate on it?" "You say that pills lead to complications, could you tell the group what you mean by this?" The moderator must battle continuously against the tendency of individuals to speak in general terms to state facts or opinions which are not their own. When this happens the moderator must get the participants to speak from their personal experience, constantly involving the participants so that the discussion does not turn into a mere conversation. At the end of the group discussion the moderator will thank the participants and reassure them that the discussion will remain confidential (the ethics of research oblige the moderators themselves to respect confidentiality). Furthermore, the participants must be convinced that the discussion they have taken part in will be useful. During the course of the Focus Group the moderator will often need to put out feelers in different directions to discover new pathways that may shed light on the behaviour and attitudes of a community. Therefore, he or she must get rid of any preconceptions and set out inquest of new hypotheses. One of the moderator's tasks is to assist and encourage the group members to express themselves. On the other hand, there are often people who monopolize the conversation and inhibit other group members. To curb or to neutralize them the moderator will resort to established techniques. During the discussion the moderator must give the impression of a calm and collected person who does not express his or her feelings easily, but at the same time must show human warmth. The moderator must also be very tolerant of the opinions and standpoints within the group, and must never express disapproval or approval. ROLE OF THE OBSERVER
The observer helps the moderator to welcome the participants and is responsible for recording the discussions and making sure that the tape recorder is in good working order. If it has been decided not to use the tape recorder, the observer must take the fullest possible notes of all contributions, highlighting the key points. Certain Focus Groups are filmed on video as the visual image may provide supplementary material for analysis <gestures, mimicry and expressions). However, use of this medium may bias the discussion and make it like a television broadcast" with all the attendant narcissism and self-restraint. The advantage of the tape recorder is that people forget it is there and it interferes less with the spontaneity of the contributions. The observer must also keep an eye on the time and prevent any intrusions which might disturb the group. At the end of the session the observer gives the moderator his/her comments on the discussion. Two approaches are used to analyze the data collected in the Focus Groups: (1)
a superficial, intuitive analysis of the verbal material recorded or noted down after which conclusions are drawn; a more scientific approach: content analysis.
(2)
The material collected from the Focus Groups is mainly semantic, communicated by language (word and gesture) so the aim of the content analysis is to classify and categorize symbolic material.
29
WHOIMCH-FPPJMEPI93.1
For research purposes, content analysis uses systematic rules, based on quantification and on the principle that even verbal content, which is qualitative by nature, can be measured. Patterns of analysis have been developed along the lines of mathematical models, to guarantee as much objectivity as possible. Content analysis focuses on a theme, i.e. a more complex set of signs, which is also identified by a signposting reference and a context reference. This is known as thematic content analysis and consists of breaking down the material into themes and subthemes which have a precise significance for the research. These segments are then arranged rationally, either by hand or by computer, in major categories of themes subdivided into subthemes. The true benefit of the Focus Group technique is seen when the content analysis does not stop at distinguishing and enumerating themes and subthemes, but also sets out to combine the units of meaning, to relate the themes and subthemes to each other and investigate the links between them. A good content analysis should reveal an underlying structure of characteristic attitudes, beliefs and values of very specifically identified groups, by means of relevant indicators (age, status, religious affiliation, etc.). In conclusion, the quantitative approach does not necessarily give the research scientific standing, indeed there is a danger that it will lead us to miss the prime objective: discovering the real significance of the attitudes and behaviour of the study population, which is prerequisite for establishing truly effective communication and actions appropriate to the population's needs and expectations. THE STUDY REPORT The study report will be structured according to the initial objectives and will try to answer the questions raised at the outset. A report on a qualitative study does not generally present numerical data in the form of statistical tables, diagrams, etc., but tends to be a written text using psychosociological, anthropological and economic concepts. It will contain many quotations of Significant remarks, grouped according to theme. These quotations are of great value in drafting the messages to be directed at the target .population as part of the IEC activities: because they reproduce the actual words of the participants, they are more intelligible and have a greater impact. The quality of the report depends on the accuracy of the analysis and on the correcl interpretation of the subjective data. The writer should not only report and describe the attitudes and practices of the target group, but must also achieve a synthesis of the collected data, by going beyond a fragmented analysis or anecdotes to encompass a complex set of factors of more general scope. The text must help to understand the motivations, expectations and resistances, to clarify the reasons why people think and act as they do, and provide a basis for taking concrete decisions in terms of actions. CONCLUSIONS The Focus Group is a swift and inexpensive technique which requires highly qualified and specialized moderators. It is characterized by a detailed and thorough analysis of the semantic material collected, which is where its unique value lies. The aspect makes the Focus Group a valuable complement to quantitative research. The technique permits levels of interpretation and explanation that cannot be attained by an extended survey. It has many applications: forming and verifying (or invalidating) hypotheses, gaining a better understanding of attitudes, beliefs and practices, drafting questionnaires that are better suited to the objectives and to the target population, and compiling a body of terms, images and slogans for use in IEC campaigns. ,.. ,.. ,.. ,.. ,..
30