(African Programme for Onchocerciasis Control Programme africain de lutte contre l'onchocercose The World Health Organrzation Guidelines for conducting a sustainobility evuluation of on APOC APOC Management,2002 - 0 rE t. t G. I3 This guideline should be read carefully by all members of evaluation teams (especially the leader) before embarking on an evaluation. It contains the following sections: 1. Background information 2. Preparing for the evaluation visit 2.I The 'John the Baptist' visit Managing the evaluation visit 3.1 Sampling 3.2 The sequence of the visit 3.3 The timetable for the visit 3.4 Using the instrument in the field 3.5 Data from the document study Analysing the data collected and making recommendations 4.1 Summarising the field data 4.2 Grading each indicator 4.3 Grading groups of indicators 4.4 Describing the main findings verbally 4.5 Makingrecorunendations 4.6 Grading a whole project 4.7 Mistakes to be avoided in the analysis 4. 5 The'Feedback/ planning' workshops 5.1 The higher level (project/ State) workshop 5.2 The middle level (health district/ LGA) workshop 5.3 General comments about the workshops 6. Writing the report Appendix 1 Appendix 2 Appendix 3 Appendix 4 Appendix 5 Appendix 6 Information for persons to be interviewed Questionnaire for health districU LGA staff attending 'feedback/ planning' workshops Programme for 'Feedback/ planning' workshop with the projecV State level team Programme for 'Feedback/ planning' workshop with the health districU LGA level team Detailed instructions for group work in 'Feedback/ planning' workshops Handout on sustainability for 'Feedback/ planning' workshops ACKNOWLEDGEMENTS The Management of APOC would like to sincerely thank the members of the core group and the following persons for their immense contributions in the preparation of the final document - Richard Gibson, Detlef Prozesky, Eleuther Tarimo,Oladele Akogun , Edith Nnoruka, Joseph Okeibunor, Chukwu Okoronkwo 3 T4 I a Background information What is'Sustainability'? By'sustainability' we mean the likelihood that a project will continue to function effectively after APOC support comes to an end. Under'sustainability' there are two further concepts: * Self-sul/icienqt: the ability of a project to continue functioning effectively, using only resources generated within the country itself. * Sustainability: the ability of a project to continue functioning effectively, using both its own resources and those provided from outside - provided the latter are dependable. In this exercise we are going to be working with'sustainability' as our guiding concept. Aspects of sustainability We judge that projects will be more sustainable if the following aspects are present: * Effectiveness: projects that are functioning effectively are more likely to be sustainable. * Efftciency: projects that are run cost-effectively are more likely to be sustainable. * Simplicity: projects that use simple, uncomplicated routines and procedures are more likely to be sustainable. * Integration: projects which have become integrated into the routine running of the health services are more likely to be sustainable. * Attitude: projects are more likely to be sustainable if staff have accepted CDTI as a routine activity, which they will continue to do even in the absence of additional material reward; and if they accept that they will have to adapt their routines to become more efficient. * Resources: projects are more likely to be sustainable if they have enough resources (human, material, financial) to support what they are trying to do. o Community ownership: Projects are more likely to be sustainable if there is active community panicipation in planning and execution of control strategy You will note that the different indicators of sustainability embody these elements. In particular the last three groups of indicators embody the 'Resources' aspect. Levels of sustainability We are going to evaluate sustainability at four levels, in the organisation of the health services of the countries concerned: A Nutiqqul headquarters of the Ministry of Health Use Instrument no.l B Province/ region/ State - a group of health districts C Health district/ local govemment authoritv GGA) Use Instrument no.2 D Health sub-district/ first line health facility Use Instrument no.3 E Village/ community Use Instrument no.4 Please note that the situation regarding these levels varies from country to country: * Level A is of course common to all projects in that country. It should be evaluated by itself, regarding sustainability. In Nigeria this level includes both 'Federal' and 'Zorral' components. * We evaluate the projects themselves at levels B to E. In some countries the project is based at level B, in others at level C. In the latter case there may be no level B. The important thing is to collect information at each level of the health service, where CDTI activity takes place. 5 I * We have developed 4 instruments for data collection about sustainability. In the table above it shows which instrument is to be used at which level. Indicators of sustain ability These indicators were decided upon by the team which met in Ouagadougou, and all appear in the instrument. They are grouped into 9 categories. These categories relate to: * The results achieved in the project (1 group of indicators) * The routine activities which produce the results (5 groups of indicators) * The resources provided for these activities to take place (3 groups of indicators). This results in the following groups of indicators at the four levels: Level of operation Groups of indicators Headquarters/ province/ region/ State District ILGA Sub-district/ First line facility Village/ Community l. Coverage t t t t Indicators of results achieved 2. Planning t I I t Indicators of activities which support CDTI 3. Providing leadership t t t t 4. Supervision and monitoring t t t t 5. Mectizan procurement and distribution t t t t 6. Training and sensitisation/ mobilization t t t t 7. Financing/ funding t t t t Indicators of resources provided 8. Transport and other material resources t t t t 9. Human resources t t t t 6 2. Preparing for the evaluation visit We aim to complete as many project'sustainability' evaluations as possible, by the middle of June 2002 - this refers of course to projects which are in their fifth year of APOC funding. The reason for the deadline is that there is a meeting of NOCs and NGDO coalition chairpersons scheduled for June in Nigeria, and 'sustainability' is going to be high on the agenda there. The two team members in Pretoria have particulars of the dates on which team members are available. They will plan provisional dates for visits in close cooperation with Dr Amazigo in Ouagadougou, and between them they will then contact the team members and NOCs in the countries concerned to check if the dates suit them as well. The sequence of preparing for each visit will take the following form: . Dates are agreed and teams allocated (including team leaders) APOC ,uranges for the 'John the Baptist' visit (see below) a week or two before the evaluation. This person will draw up a provisional plan for the fieldwork, in collaboration with the team leader (they communicate electronically or by fax). The team leader sends the provisional timetable to Dr Amazigo at Ouagadougou. The APOC office in Ouagadougou: * lnforms team members of the coming evaluation and obtains their consent. * Aranges travel, subsistence etc. for the team. * Sends the Guidelines, instruments and provisional timetable to the team members, electronically. 2.1 The 6John the Baptist' visit The role of this person is to make all the arrangements necessary for the visit of the team. A suitable person should be appointed by the APOC office in Ouagadougou to undertake this preparatory visit about two weeks before the visit by the team. This person should be conversant with the instruments and the requirements of the team, and should be in touch with the team leader before and during the visit. The tasks of this support person include: . Going personally to the project site, and planning the practical details described below with the project directors (State and NGDO). . Taking the sample of the sites to be visited, in accordance with the instructions set out in the next section. . Planning the courtesy calls and report back meetings (attempting to go as 'high' as possible in the Ministry/ political hierarchy) . When all of this has been done, drawing up a tentative timetable for the visit. . Giving advance warning to all sitesi persons sampled, of the dates and times when they will be visited by the team - either personally, or by sending them an introductory letter (see Appendix 1). . Ensuring that all necessary documentation is available to the evaluation team. 7 ! I Ensuring that sufficient transport will be available to the team. Ensuring that suitable accommodation will be available to the team. Ensuring that arrangements have been made to pay subsistence and travel allowances to team members, and also to those who will be travelling to attend the 'feedback/ planning' workshops in Week 2. Planning the site and participants for the 'feedback/ planning workshops in Week 2; ensuring that participants are invited in good time. Selecting local team members to accompany the evaluating team throughout the field work. The 'John the Baptist' person may participate in the actual evaluation. 8 3. Managing the evaluation visit 3.1 Sampling The sampling is done purposively, not randomly. This is because a random sample from a small population may easily be non-representative. We use the following criteria: . The primary criterion is coverage (geographical and therapeutic). This is because it is a measure of the performance of the whole system - and it is the sustainability of that system that we are evaluating. . The following secondary criteria are also taken into consideration: " Endemrclry: ensuring that, as far as possible, the sample contains both hlper-endemic and meso-endemic areas (more or less in the same proportion as that of the REMO results for the project area). * Geographical spread: ensuring that, as far as possible, the sample contains areas which represent the different zones where the project operates, and communities which are closer to towns and some which are more isolated. For proiects containine many districts/ LGAs (e.g. Nigeria) ' For each project, you sample at least three districts (one with good coverage; one with medium coverage; one with poor coverage). . For each district/ LGA, you sample two sub-districts (one with good coverage, one with poor coverage) . For each sub-district, you sample two villages (one with good coverage; one with poor coverage). For projects containins 1-2 districts (e.g. Uganda) . For each project, you sample the one or two districts. . If there is only one district: * You sample four sub-districts (one with good coverage, two with medium coverage and one with poor coverage). * For each sub-district you sample trvo villages (one with good coverage and one with poor coverage) . If there are two districts: * You sample three sub-districts in each (one with good coverage, one with medium coverage and one with poor coverage). * For each sub-district you sample two villages (one with good coverage and one with poor coverage) In both of these examples you also you endeavour to satisfy the secondary criteria. This means that, if there are two items with more or less the same coverage, you choose between them in such a way that the total sample also satisfies the secondary criteria. Since we only have a limited time to do the fieldwork, we also have to take into account accessibility and convenience. If there are two or more areas which are very similar in every way, we select the one that is easiest to get to. 9 3.2 The sequence of the visit Team members travel to the site of the project headquarters, arriving on the Saturday before the field work starts (which will be on the Monday following). They are expected to familiarise themselves with the following documents before arrival: * This'Guideline' document. * The four instruments. * The provisional timetable. These documents will be forwarded electronically to each team member. The team spends the Sunday planning for the coming work. They go through the guidelines and instruments together, adjust the provisional timetable if necessary, and constitute the teams which will be doing the fieldwork. Field visits take place from the Monday to the Friday: * Project data is collected at all the levels, at the sampled sites. * The NOCP office in the Ministry Headquarters also needs to be evaluated. This office serves all the projects in the country, so it only needs to be evaluated once. This evaluation will be the job of the first evaluation team to visit that country. They have to create a small team that will go back to the Ministry HQ at some time during the first or second weeks, to do the necessary data collection (interviews and document study). Note that this report is written up separately (except in countries with only one project, where it will form part of the project report). The weekend following is used to: * Analyse the data collected systematically - drawing the necessary conclusions and making recommendations. * Write the draft report. * Plan for the 'feedback/ planning' workshops of the coming Monday to Wednesday. On the Monday to Wednesday of the second week, the 'feedback/ planning' workshops are held: * Monday: workshop for the project level * Tuesday and Wednesday: workshops for the health district/ LGA level. The idea is not to have too many people at one workshop, otherwise the workshop becomes difficult to handle. So if a project includes many health districts/ LGAs, half of these are invited to attend on the Tuesday, and the other half on the Wednesday. It is important to note that the evaluation team will gain insights during these workshops, that they did not pick up during the fieldwork. This additional information will have to be included in the data analysis, since it may well influence the recommendations that the team will make. Throughout the visit we recommend that the team meet every evening, to discuss the day's progress and problems, and to plan the following day's work. On the last evening they evaluate the whole evaluation process, and make recommendations about future recommendations. These are forwarded to all members of the core team (including Dr Amazigo). Another important aspect of the visit is the political one, of doing advocacy with decision makers at the local and national levels. The fact is that for the projects to be sustained, they are going to need regular and substantial financial contributions from the Ministry of Health and/ or local government. Outsiders are often able to state this obvious truth with more authority than local workers. Each team must therefore pay courtesy calls to the relevant civil servants and political heads early during the visit, and if possible give them personal feedback towards the end of the visit. They should try to go as high as possible: to the Minister of Health her/ himself, or to the State governor (in Nigeria). The point to be emphasised during such visits is this: the l0 I a Ministry has within itself a project which is really working well; a relatively small amount of money will keep it doing so; and failure to fund it will mean the return of misery to thousands of people. 3.3 The timetable for the visit We suggest you spend 11 days at each project - we found at Taraba (the second evaluation, after the trial at Kaduna) that this was sufficient. There will be six of you in each team. You have to divide yourselves into three teams - let's call them I, II and III: 't Team I: This should be the team leader and one other. This team has to work with staff at Ministry Headquarters level (in case of this being the first evaluation in that country). When s/he has finished this task s4re joins the other teams for the district level work. Teams II and III: Two persons each, with a mix of expertise and experience.i. Here is a possible timetable: You will clearly need to adapt this timetable according to the specifics of each situation. * The number of visits and interviews a team can handle per day will clearly vary from country to country and project to project. * Note again: in each country the Ministry Headquarters only needs to be visited once. This mearls that if there are two or more projects to be evaluated in a country, the Headquarters level only needs to be evaluated during one of these two evaluation visits. The information obtained there will then be made available to the second evaluation team. Saturday Travelling day - all teams go travel to the project headquarters Sunday Orientation day at the project headquarters Monday All teams: courtesy visits to MoH and NGDO management Team I: data collection at project headquarters (MoH and NGDO) Teams II and III: data collection in the field: health district/ LGA level and below Tuesday I Team I: data collection at project headquarters (MoH and NGDO) Teams II and III: data collection in the field: health district/ LGA level and below Wednesday - Thursday ! All teams: data collection in the field: health district/ LGA level and below In case this is the first visit to the country, Team I travels to the Ministry HQ to collect data for the national level Friday All teams: feedback to the health districts/ LGAs visited Saturday- Sunday All teams: . analyse the data, work out the recommendations, write the draft report . plan for the goming 'feedbacU planning' workshops Monday 'Feedback/ planning' wgrkshop for the project/ State level Tuesday 'Feedback/ planning' workshop for the health district/ LGA level (first round) Wednesday 'Feedback/ planning' workshop for the health district/ LGA level (second round, ifneeded) Thursday Departure for home 3.4 Using the instrument in the field l1 I On the Saturday and/ or Sunday, immediately after arriving, the team members should go through the instruments together, to arrive at a common understanding of what they mean and what they require. Note that this exercise is not formal research - it is an evaluation of each project with respect to its sustainability. The instrument should therefore be seen as a guide, which will help the evaluation team members to make a series ofjudgements about the sustainability of the project: * Each judgement is made against one of the indicators/ criteria of sustainability that have been identified. * Each judgement is made based not only on the information gathered at that level, but also on the supporting evidence gathered at other levels. * Note that additional information will be obtained during the 'feedback/ planning' workshops. This should also be taken into account when making the judgements. * [n making these judgements team members use information from as many sources and operational levels as they can. Such triangulation strengthens the validity of the judgements. For example, the report that district level staffgive about their supervision activities can be verified by asking sub-district staff about their experiences of the same supervision. * Whenever written evidence is used as supportive evidence, it is important that the document be carefully scrutinised by the evaluators, so that they understand it to mean what it purports to mean. When a team arrives at a site, they do the normal greetings and introductions. They explain what the visit is about, and discuss confidentiality and voluntary participation. They request the respondents to provide the necessary documentation. As soon as possible the interview starts. The answers to the questions are written in the boxes on the instrument. Where necessary information from documents is added to the interview data. It is best to use a pencil, so that corrections can easily be made. The judgement: 'How good is this for sustainability?' is left for later, when the whole team looks at all the evidence together on the Saturday and Sunday. We agreed in Ouagadougou that each team should invite at least one person from the project team to join it during fieldwork. This is to enable these persons to gain a deeper understanding of sustainability at first hand. However in Kaduna we encountered a specific problem with project staff, during field visits. They wanted to use the opportunity as a routine management exercise, whereas the real reason for the fieldwork is research/ data collection. [n short, they spoke too much, didn't listen and generally tended to dominate the proceedings. Before leaving for the field this potential problem must be discussed with the colleagues. They should be asked to let the evaluation team take the lead; to listen rather than to speak; and to limit their questions to those on the instrument. When interviewing a person from a certain level (especially government employees) it is not a good idea to have her/ his superior present. The reason is obvious: sftre may not feel free to express her/ his real opinion. Upon returning from the field every day the instruments have to be checked, to see that they are fully filled in. The completed instruments are handed to the team leader for safe keeping. 3.6 Data from document study At the beginning of each instrument is a list of the documents that you should try to get hold of, when you are at that particular level. You will not always be able to get all of it, but as much as possible of it will be helpful. As you go through the document you keep a blank copy of the instrument for that level next to you. Whenever you come across a relevant piece of information, you write it in the relevant box. S I I r2 I Note that you will also be making observations as you go around. Write these on a piece of paper, and enter them into an instrument for the relevant level as soon as you have a moment. 4. Analysing the data collected and making recommendations The whole team meets together to analyse the data. This mainly happens on the Saturday and Sunday, but may begin earlier, in the evenings during the week. The team leader chairs the discussion. Colleagues from the project should be encouraged to be present. The following process is followed: 4.1 Summarising the field data . You do the analysis one level at a time. The first step is top assemble all the information that has been collected at that level - from interviews, document study and observations. ' The team tackles the indicators one by one. Each piece of information that has been collected for that indicator is presented by the person who collected it. . The chairperson uses a blank copy of the instrument (paper or electronic) and fills in each piece of information that is presented. This data will eventually have to be typed into the instrument template for each level, and will form part of the final report. . Note that, due to time constraints, it may be necessary to divide up this work between two teams by giving each that data from a different level to summarise. Try doing one together and see how fast you go. If you do split forces, each team must have the data from all sources about that level available, Each team then presents its findings to the other to check. 4.2 Grading each indicator . Once all the evidence has been led for a particular indicator at a particular level, the team makes a joint decision on: 'Is this good for sustainability?'. The following grading is used: Grading Meaning Numerical value Fully The criterion is fully satisfied/ met. 4 Highly The criterion is almost fully met - there is a small deficit. 3 Moderately The criterion is about halfway fulfilled. 2 Slightly The criterion is marginally fulfilled. 1 Not at all The criterion is not fulfilled at all. 0 Not applicable This criterion is not relevant to this particular case. " effis@'n"si"slwl Indicators in Group t have a modification of this system, as follows: * For Grading Numerical value Fully (100% coverage) 4 Highly (problem with nomads only) aJ Moderately (a few villages/ hamlets not covered) 2 Slightly (many villages/ hamlets not covered) 1 Grading Numerical value Fully (>85% coverage) 4 Highly (7 5 -84% coverage) 3 Moderately (65 -7 4% coverage) 2 * For eutic 13 Slightly (less than 65oh coverage) 1 4.3 Grading groups of indicators Once all the indicators have been graded by the team, a score can easily be worked out for each goup of indicators. This is done by getting a mathematical average - for example: For the'Planning' group of tndicators at District/ LGA level there are three indicators. Suppose the scores for these three are 3, 3 and 2. The overall score for 'Planning' at this level is therefore (3+3+2)/j : 2.7 Note though that the 'average numerical value' is only a representation of the ordinal series: 'fully', 'highly' .. etc. The scores for all the groups of indicators at a certain level may then be illustrated in atable, or graphically - for example: In this way the main problem areas for sustainability are made clear. 4.4 Describing the main findings verbally Having graded a group of indicators (e.g. 'Leadership') the factual content of that group must be summarised in a paragraph. Both positive and negative findings are given, in such a way that all agree that the totality of the data is represented. The result of this process is nine paragraphs of varying lengths. Note that this is a critically important step of the evaluation. The grading exercise point to areas of major weakness - a skeleton - but this analysis produces the meat of the report. The recommendations that follow will be principally based on it. t4 Kaduna Project: self€ustainability at LGA leve! "a,t"* ^."""*ja.o ""t' group of lndicators 4 3.5 A3t .9 oiz o P r.s oe1 0.5 0 ,"* -.."" *.o*" ^ro'"'- 4.5 Making recommendations Finally the group has to make recommendations, about steps that projects and countries need to take to achieve sustainability. These recommendations are made for each of the four levels, and should have the following characteristics: . They must be strictly based on the findings of the field work. They address the areas of poor sustainability that the research has uncovered. . They must be practical and achievable: * For each recommendation a suggestion must be made, about who should be responsible for carrying it out. * For each recommendation a deadline or time line should be suggested. * One or more indicators of achievement must be given for each recommendation. . Recommendations should be prioritised, as follows: * HIGH: this recommendation is critically important. If it is not carried out the project will not be sustained. * MEDIUM: this recommendation is important but not critical. Carrying it out will enhance the sustainability of the project though. * There should not be any LOW priority recommendations. 4,6 Grading a whole project Finally the team may grade the whole project, in terms of its overall sustainability. The expert Soup meeting in Ouagadougou considered that it could be useful to grade each project, for the following reasons: . It gives an immediate idea of the sustainability of the project as a whole. . It sets a baseline against which to measure progress in achieving sustainability. This evaluation is done using all of the available information gained during the course of the evaluation. It is recommended that teams resist the urge to allocate a numerical value to the sustainability of a project - the grading is an overall impression, and not a mathematical summative scoring of individual elements of sustainability. When grading the project as a whole, it is important to bear in mind that there are a number of critical elements of sustainability, without any of which it is unlikely that any project will be sustainable: ' Money - there should be sufficient money available to undertake strictly necessary tasks which have been carefully thought through and planned (absolute minimum residual activities). ' Transport - provision must be made for the replacement and repair of vehicles. There must be a reasonable assurance that a vehicle will be available for minimum essential activities. Note that 'vehicle' does not necessarily imply '4x4' or even 'car'. . Supervision - the project will not be sustained without timeous mobilisation and targeted supportive supervision. . Mectizan supply - the supply system must be dependable. The bottom line is that enough drugs must arrive in villages at the time selected by the villagers. . Political commitment - effectively demonstrated by awareness of the CDTI process among policy makers (resulting in tangible support); and a sense of community ownership of the programme. Other indicators remain important and should be taken into account together with the more critical elements listed above. Against this background, the following schedule may be used to grade the project as a whole l5 Level of sustainability Description Excellent This proiect is completely sustainable. High This project is not far from being sustainable. With feedback from the team before departure, the project staff should be able to undertake the required remedial action. Moderate This project is potentially sustainable, but will require rethinking and mobilisation of high-level support to get it on the road asann Low This project is seriously unsustainable - there is some doubt as to whether it ever will be. It needs a lot of immediate expert guidance from outside. 4.7 Mistakes to be avoided in the analysis The following problem areas were noted in the analysis of the first two projects: . The analysis (and recommendations) could focus on the functioning of the project, rather than its sustainability. . The analysis could be incomplete, not taking significant pieces of field data into account. ' The recommendations could be superficial, non-specific and unrealistic. . Sometimes findings at one level were not consistent with those of another level. The team leader must ensure that these mistakes are avoided. l6 5. The 'Feedback/ planning' workshops This evaluation exercise aims to set the projects firmly on the road to sustainability. This means that the team has to do more than evaluate - it also has to set a process of planning in motion, based on its findings. This is done through two or more 'feedback/ planning' workshops, to be held at each level where major decisions about the CDTI programme are made. These workshops have the following aims: . Feedback In the workshop we present the evaluation team's findings to all the relevant project staff and political figures, who have responsibility for the CDTI prograrnme, The team has to inform them what happened - it has to present them with its findings. . Plannine We also have to enable project and political staff at these levels to work together to develop a realistic progrirmme for the coming years, based on the findings of the evaluation. This plan should as far as possible ensure the sustainability of the participants' own CDTI prograrnme. The process described below is based on experience gained so far in the Taraba and Kaduna evaluations. From them we have learnt the following: . Preparation for the workshops should be done by the project directors (Ministry and NGDO) and the 'John the Baptist' person. They have to invite those who will be attending, in good time. Written invitations to those who should attend, at least a week before the event . Separate workshops are held for the project/ State level, and the health district/ LGA level. The higher level workshop is held first, since it acquaints staff at that level with the workshop process - which they will help to facilitate on the following day(s), for the middle level. The workshops should be held at the project headquarters. . If considered necessary a thild workshop may be held for the national level. . Each workshop lasts for td(s 7:8 hours. 5.3 The higher level (project/ State) workshop The fo should attend this The programme for the workshop is designed to fulfil the two objectives given above. An example of such a programme is given in Appendix 3. This programme may be used as an example, but will obviously need to be adjusted according to the circumstances. More details about the programme are given in 5.3 below. Members of the evaluation team function as facilitators for this workshop. 5.4 The middle level (health districU LGA) workshop t From the official health service: - The project leader/ SOC. - Senior officials in charge of the project (e.g. director of the section in which the project leader works; the permanent secretary). - Field workers implementing the project at this leveV SOCT. From the NGDO: the officer allocated to the project.* * * An other stakeholder who at this levelcontributes to From the official health service: - The management team responsible for running the health service at this level; including the person responsible for disease control programmes. - The CDTI programme manager at this level. Relevant civil authorities at this level.* * * other stakeholder who at this levelcontributes to I The ersons should attend this t7 One or two of these workshops may be held - this depends on the number of persons who will be attending. It is difficult to run a workshop if there are more than 40 participants. In Nigeria for example one project may contain up to 16 LGAs, who are each asked to send 4-5 delegates In such a situation it is clearly necessary to run two identical workshops on consecutive days. Once again, the programme for the workshop is designed to fulfil the two objectives given above. An example of such a programme is given in Appendix 4. This programme may be used as an example, but will obviously need to be adjusted according to the circumstances. More details about the programme are given in 5.3 below. Note in particular that the workshops at this level provide us with the opportunity to gather more important information about sustainability. This is because we are able to meet those health districts/ LGAs which were not included in our sample. A questionnaire may therefore be administered to them, at some time during the day's prograrnme. An example of such a questionnaire is given in Appendix 2. The data obtained in this way should be used to revise the findings and conclusions that were made, following the initial analysis of the data collected during the field work. The facilitators at this workshop are: * The evaluation team * The persons who attended the higher level workshop 5.3 General comments about the workshops In some situations the health district and the project are the same. In that case only one workshop needs to be held. It is important that the workshop should be led by someone with good workshop skills - someone with experience of running workshops. You should appoint the team member who is most experienced in this way, to be in charge of the workshop programme. The following practical arrangements need attention, for both kinds of workshop: I I * * Those who attend the health district/ LGA level workshop will need travel and accommodation allowances. APOC management has agreed to fund this. However the team leader has to make sure that the funds are applied for, and that the cash will be available for participants at the end of each workshop. In this he will have the assistance of the project leadership and the NOC, who have experience in these matters. Participants have to be provided with tea and lunch. Again. APOC has agreed to pay, but the team leader has to make sure that the project leadership/ NOC arrange this. It is very important that the meals should arrive on time. must be carefu It must be big enough to accommodate the expected number of persons attending. It should be at some distance from the project office, so that there are no unnecessary intemrptions such as people being asked to take telephone calls. It should be quiet - e.g. no busy road or electricity generator nearby. fans be needed.It should not be too hot - if there is no air + The room where the takes l8 chosen: * The fo materials should also be ared APOC will foot the * Note that the more care is taken with practical details, the better the workshop will run. Note the following about the workshop programme: At the beginning of the workshop the concept of 'sustainability' needs to be discussed. We have been doing that as a presentation, followed by a discussion. We also gave participants a handout on the subject (see Appendix 6). * About the feedback sessions: The team gives feedback about their findings at all levels - but they give more time to the level of the persons for whom the workshop is being run. They first give the positive findings/ the achievements, before going on to the negative ones/ the problems. It will be useful if the team prepares a handout on the main findings (the section of the report prepared as describedin4.4 above). It is important to allow enough time for discussion of the findings. It is best to give detailed instructions for the group work. An example of such instructions is given in Appendix 5. At the end of the workshop each group presents its plan. This can be very boring if there are a large number of plans to present. We suggest that you only ask each group to present one or two items from their plan, as an example of their work. * IC tt * {t The last item on the programme is to work out the way forward. The main question is: what is going to happen to the plans that the participants have just made? Work out with each team of participants what they will do with it. To whom will they present it, and when? Where will they stick it up? How will they ensure that they keep to it? About the role of the facilitators: The participanls should be as active as possible. This means thefacilitators have to be as quiet as possible. They should resist the temptation to give workshop participants the benefit of their vast wisdom! It is the participants who have to make the plan - it is their programme, after all. Make clear they are now making their plan for sustainability. It is very important that they should feel that they own this 'sustainability plan'. When the participants are drawing up their plan, facilitators should help them to fill in their first planned activity (using the suggested format - see Appendix 5). Thereafter the facilitator's role should mainly be to praise and to encourage. For many participants this kind of planning will be new, and they can easily be flustered if the facilitator insists on having everything done'correctly'. A folder for each participant, containing the workshop programme, handouts (e.g. the main evaluation findings; also the one on 'sustainability'), some sheets of writing paper, a pen or pencil. The necessary equipment, depending on what the team wants to use: flipchart, overhead ectormarker etc. * l9 I I Make sure that participants go home with their plans. You should however keep copies for the person responsible for the programme, at the level above those who attended the workshop; and for your evaluation report. At some stage during the second week you will be giving feedback to the senior managers in the Ministry (or even the government). This means that a small team must be ready to leave the workshop, at a time which is suitable to the people you will be visiting. And remember to pack a tie for the occasion! You never know, you may get to see the Minister of Health, or the Govemor of the State. Project staffwill have the following responsibilities following the workshops: * The plans will have to be followed up - to make sure they are really implemented. * The project and district level teams will have to be encouraged and enthused. * Resources may have to be found, for additional activities - e.g. further training to prepare everyone optimally with the skills they need to cope with the departure of APOC. t 20 6. Writing the report Once the findings have been analysed as suggested above, a report must be written. This is done on the Saturday/ Sunday, and each team member may be given the responsibility for completing part of it. The following sections are recommended: . Introduction: a short description of the project, and its history to date. . Methodology: who the team members were, and how they set about performing their task; problems they encountered. and recommendations: for each of the four levels the fo is to be used: . Reports of the feedbacH planning'worl<shops: a short description of each - who attended, the degree of participation, planned next steps . A report on advocacy visits caruied out: who was seen, their response . Appendices: e.g. data sources, the data summary sheets for the four levels; the plans produced during the workshops It is suggested that the team leader consult a report from a previous evaluation (e.g. the Taraba project) to guide her/ him in writing the report, Note that a separate report has to be made for the National level of each country (except in cases where there is only one project per country, when National data is joined to the project report). The same layout is recommended. The final responsibility for completing the report rests with the team leader. Copies of the final report are sent electronically to: . The WHO representative in the country. . The APOC office in Ouagadougou. . The project concerned. . The NOC/ NOTF of the country concerned. . The teafil secretariat in Pretoria. A graph of the average overall grading (on a scale of 0-4) of the nine groups of indicators The main positive and negative findings within the nine groups, relating to sustainability, comprehensively described Detailed and specific recommendations, based strictly on the findings, prioritised, with indicators, with assigned deadlines and responsible persons * * .tt * ' for the ect.An overall'sustainabili 2t
Appendix I Information for persons to be interviewed Information about the coming evaluation of the Onchocerciasis (River Blindness) control programme (for persons whom the team would like to interview) For the past few years you have been helping to run a programme to control river blindness, by distributing the drug ivermectin (Mectizan) to villagers in your area. This programme has been supported by the African Programme for Onchocerciasis Control (APOC) as well as your Ministry of Health, and some non-governmental organisations. These funders now want to know whether the programme they have started has become strong enough to continue. Within the next week or two APOC is going to send a team, to investigate the sustainability of the river blindness control programme in your area. Since you play an important role in the programme they would like to meet you, to talk to you about the following issues concerning the control programme: . Coverage: how well is your area being covered by the programme? . Planning: how is the programme being planned yearly? , Providing leadership: who is providing leadership for the programme, and how? . Supervision and monitoring: what supervision and monitoring is being done? . Ordering and distributing Mectizan: what process is being followed to order and distribute the Mectizan? . Training and sensitisation/ mobilisation: what training and mobilisation activities are taking place? ' Financing/ funding: what funding is currently available, and what is likely to be available in future? . Transport and other material resources: what is available now, and what is likely to be available in future? . Human resources: who is available to run the programme, and how skilled are they? It is not necessary for you to make elaborate preparations for the visit. The team members need to find out what the present situation is, and you do not need to change anything you do before they arrive! They simply want to talk to you, to hear from you what is happening in your part of the programme. We would be most grateful if you could spare them a bit of your time, on an agreed date. The person handing you this document may also ask you to get some documents ready for the team which is coming to see you. The information in these documents will help them to understand your situation correctly. We would be grateful if you could assist us by having these documents ready on the day when the team comes to visit you. It may be that you do not have some of these documents - that is not a problem. Whatever you are able to show the team will be useful. Thank you very much! 23 Documents needed: nationall zonaV provinciaU State level . Ministry of Health policy documents and strategic plans regarding CDTI . National/ zonaU state/ regional/ provincial onchocerciasis control plans for current and previous years . REMO reports, including lists of all endemic communities/ target populations . Nationall zonallstate/ regional/ provincial budgets and financial reports for the current and previous years . The year plans for the country/ zone I State / region / province for the past two years: * Including the plans for CDTI (integrated with the main plan, or separate) * Also quarterly or monthly plans (if these exist) . The yearly budget for onchocerciasis control, for the last three years, for the country/ zone I State / region / province: * From all sources: Ministry of Health at this level, APOC, NGDO partners * Particulars of staff salaries, capital costs, running costs. . Financial control documents: expenditure authorisation slips, journals, ledgers etc. (note that these will supply information about the delivery of promised funding) . Reports of the last round of CDTI in the country/ zonelState/ region/ province . Reports (if these exist) of: * Routine supervision visits by employees at this level, to the level(s) below them * Intemal self-monitoring by employees at this level. . All the forms related to ordering, issuing and control of Mectizan at the country/ zonaV State/ regional/ provincial level . All curricula, training materials, timetables, attendance lists and reports relating to training for CDTI and in-service training in the country/ zonel State/ region/ province over the last two years ' All reports of mobilisation/ sensitisation activities in the country/ zonelState/ region/ province in the past two years. . All documentation related to control and use of official transport at this level, over the last two years: trip authorisations, fuel allocations, log books etc. Documents needed: communl level CDDs' treatment registers Summary forms, showing coverage data and Mectizan use for the village for the past three years. Copies of Mectizan orders for the village, for the past three years.I 24 D Documents needed: health district/ LGA level . The coverage reports/ tables for the health district / LGA for the past three years (per sub- district; per village) . REMO reports, including lists of all endemic communities/ target populations . The year plans for the health district / LGA for the past two years: * Including the plans for CDTI (integrated with the main plan, or separate) * Also quarterly or monthly plans (if these exist) . The yearly budget for the last three years for the health district/ LGA (from all sources: the LGA itse[ higher levels of the govemment, APOC, the NGDO partner) . Financial control documents: expenditure authorisation slips, joumals, ledgers etc. . Reports of the last round of CDTI in the health district/ LGA . Reports (if these exist) of routine supervision visits by DMT members, to the sub-districts . All the forms related to ordering, issuing and control of Mectizan at the health district/ LGA level . All curricula, training materials, timetables, attendance lists and reports relating to training for CDTI and in-service training in the health district/ LGA over the last two years . All reports of mobilisation/ sensitisation activities in the health district/ LGA in the past two years . All documentation related to control and use of official transport at this level, over the last two years: trip authorisations, fuel allocations, log books etc. Documents needed: sub-district/ first line health level . The coverage reports/ tables for the sub-district for the past three years (per village). . The year plans for the sub-district for the past two years: * lncluding the plans for CDTI (integrated with the main plan, or separate). * Also quarterly or monthly plans (if these exist). . The yearly budget for the last two years for the sub-district - from all sources: the health district/ LGA, own sub-district income, other (note that this budget may be kept at the health districU LGA level). . Financial control documents: expenditure authorisation slips, journals, ledgers etc. . Reports (if any) of the last round of CDTI in the sub-district. . Reports (if these exist) of routine supervision visits to the villages, undertaken by sub-district team members. . All the forms related to ordering, issuing and control of Mectizan at sub-district level. . All curricula, training materials, timetables, attendance lists and reports relating to training of CDDs in the sub-districts over the last two years. . All reports of mobilisation/ sensitisation activities in villages in the sub-district in the past two years. . All documentation related to control and use of official transport at this level, over the last two years: trip authorisations, fuel allocations, log books etc. 25
Appendix 2 Questionnaire for health district/ LGA staff, about their CDTI work Name of your health district/ LGA: About coverage: ' What is the rate in health district/ LGA? About plannins: . How do you make the plan for CDTI in your district ILGA every year? . Is the plan written down? . Who contributes to the ? Abau! leadership: Who is responsible for CDTI in your health district/ LGA? Does this person have enough time for the CDTI work? Do you think that the political leaders in your health district/ :LGA support CDTI? Why do you think so? 27 II I About supervision and monitorine: How frequently do you visit the health centres/ the sub-districts, to monitor them? How do you plan your supervision visits to the health centres/ sub-districts? How are the statistical collected form the health centres/ sub-districts? About Mectizan: . How does your yearly supply of Mectizan get to you? . How does the Mectizan get from you to the health centres/ sub-district? .Do Mectizan or do sometimes run short? About trainine and sensitatisation/ mobilisation: . Whom do you train every year? . How do you decide whom you are going to train? . Describe the mobilisation that out Whom do vlsl and how often? About financine/ fundine: . Do you have ayearly written budget for your CDTI work? . Who supplies the money that you need to carry out your CDTI work? . What lans have 28 made to obtain more funds when the APOC funds run out? About transport and other material resources: . Describe the transport available to you to do your work. . Does it ever happen that you do not have transport when you need to go out? How often does this happen? . Do the different About human resources: Are your LOCT members properly trained? Do they really know how to carry out CDTI? How stable is LOCT? Are its members transferred to other units, or not much? at level share each other's 29
Appendix 3 Programme for 'Feedback/ planning' workshop with the project/ State level team The following example is from the Kaduna evaluation. You will of course adapt it to your particular circumstances. Assessment of the sustainability of the Kaduna CDTI Project PLANNING WORKSHOP FOR STATE/ PROJECT LEVEL MAY 9,2002 AGENDA S/N ACTIVITY TIME FACILITATOR I Opening prayer 10.00 - r0.05 2. General introduction 10.05 - 10.1 5 Mr Sankwai 3. Introduction to the workshop programme 10.25 - 10.35 Dr Fayomi 4 Evaluation obj ectives: What is sustainability? 10.35 - 10.50 Prof. Prozesky 5 Methodology & limitations 10.50 - I 1.00 Mr Okoronkwo 6. Brief summary of findings: ' Community level . Health facility/ district level . LGA level (followed by time for open discussion) 11.00- r1.05 11.05-11.10 11.10- 11.15 11.15-11.25 ProfProzesky Dr Fayomi Mr Okoronkwo 7 Summary of findings: . State/ project level (followed by time for open discussion) 11.25-11.45 1 1.45 - 12.00 ProfProzesky 8 BREAK 12.00 - 12.20 9, Brainstorm: what could be the solutions to these problems? (group work) . Group l: Planning, Monitoring and supervision . Group 2: Finances, Training and mobilisation . Group 3: Transport, Mectizan (each group appoints a rapporteur) 12.20 - 13.20 Mr Okoronkwo 10. Report back from group work (followed by time for open discussion) 13.20 - 14.00 Dr Fayomi 1t LUNCH 14.00 - 15.00 12 Planning for sustainability at State/ project level Group work - members from each LGA work by themselves. Each group has thefollowing tasl<s: . Group 1: What do we need to do, between now and the end of this year? . Group 2: What will our CDTI programme be 15.00 - 16.00 ProfProzesky 3l for 2003? Group 3: What will our resources be for 2003, for CDTI? 13. Report back from group work (followed by time for open discussion) 3.45 - 4.45 Mr Okoronkwo 14. The way forward: implementing sustainability in the Taraba CDTI project . Practical steps to take the process forward (open discussion in the whole group) 4.45 - 5.30 Dr Fayomi 15. General matters/ administrative information 5.45 - 6.00 Mr Okoronkwo, Mr Sankwai 16. Conclusions/ closing 6.00 - 6.15 ProfProzesky Facilitators for small groups in Session 9 Group 1 Mr Okoronkwo Group 2 Prof Prozesky Group 3 Dr Fayomi Facilitators for small groups in Session 12 Group 1 Dr Fayomi Group 2 Mr Okoronkwo Group 3 ProfProzesky I 32 a Appendix 4 Programme for 'Feedback/ planning' workshop with the LGA/ health district level team The following example is from the Kaduna evaluation. You will of course adapt it to your particular circumstances. Assessment of the sustainability of the Kaduna CDTI Project PLANNING WORKSHOPS FOR LGA LEVEL MAY 10-11,2002 AGENDA S/N ACTIVITY TIME FACILITATOR I Opening prayer 10.30 - 10.35 SOCT member 2 Welcome 10.35 - 10.40 Dr. Fayomi 3 Introductions 10.40 - 10.45 Mr Sankwai 4 Introduction to the workshop: Why are we here, and what are we planning to do? 10.45 - 10.50 Mr Okoronkwo 5, Background about the evaluation: Why it was done; what is 'sustainability'? 10.55 - 11.05 Prof. Prozesky 6. Background about the evaluation How it was carried out 11.05- 11.15 Mr Okoronkwo 7. Brief summary of findings: . Introduction: LGA achievements . Community level . Health facllity/district level . State/ project level (followed by time for open discussion) 11.15-11.20 11.20- 11.25 1r.25 - t 1.30 11.30- 11.35 11.35 - 1r.45 Mr Okoronkwo Dr Fayomi Mr Sankwai ProfProzesky 8 Summary of findings: . LGA level (followed by time for open discussion) il.45 - 11.50 I 1.50 - 12.1 5 Mr Okoronkwo 9, Completing the'LGA questionnaire' (group work, in LGA groups) 12.15 - 12.45 ProfProzesky 10, BREAK 12.45 - 13.00 11 Brainstorm: what could be the solutions to these problems? (group work) . Group 1: Planning, Monitoring and supervision . Group 2: Finances, Training and mobilisation . Group 3: Transport, Leadership (each group appoints a rapporteur) 13.00 - 13.30 Mr Okoronkwo 12 Report back from group work (ideas are written up on flipchart paper) 13.30 - 14.00 Mr Sankwai 33 t3 LUNCH 14.00 - 14.30 14 Planning for sustainability at LGA level Group work - members from each LGA work by themselves. Each group has thefollowing tasl<s: . What do we need to do, between now and the end of this year? . What will our resources be for 2003, for CDTI? . What will our CDTI programme be for 2003? 14.30 - 16.00 Dr Fayomi 15. Report back from group work (followed by time for open discussion) 16.00 - 16.30 ProfProzesky 16. The way forward: implementing sustainability in each LGA CDTI project . Practical steps to take the process forward (open discussion in the whole $oup) r6.30 - 16.45 Mr Okoronkwo 17, General matters/ administrative information 16.45 - 16.55 Mr Sankwai 18, Word of thanks and closing 17.25 - 17.30 LGA member Facilitators for small groups in Session 10 Group l: Mr Okoronkwo Group 2: Mr Sankwai Group 3: Dr Fayomi Facilitators for small groups in Session 13 LGA 1. 2: Dr Fayomi LGA 3.4: Mr Okoronkwo LGA 5. 6:Prof Prozesky LGA 7. 8: Mr Sankwai I 34 Appendix 5 Detailed instructions for group work in 'Feedback/ ' worksho s TAffift' this year There are three kinds of activity that we need to perform: 1. We have to do everything that is needed to make sure that this year's distribution is completed successfully. 2. We have to correct problems that exist, while we still have some extra money to do this. 3. We have to undertake activities which will mobilise extra resources for our work next year, since we will no longer have APOC funds then. We are going to write our plan in the format given below: gsffitl*.. 1 3 There are two kinds of activity that we need to perform: 1. We have to do everything that is needed to make sure that this year's distribution is completed successfully. However we will have less money, so the activities we plan must cost as little as possible. 2. Again, we have to undertake activities which will mobilise extra resources for our work the following year. We write our plan using the same format as above. U1$ffisfi rvill need next year ,i,,,|'l.f; To carry out our work programme for 2003 we need to have resources: staff transport, money, stationery etc. We need to think carefully where we can get such resources. We make a list, using the following format: 3 No. Activity/ task (very specific) Why do we need to do this? Who should carry out the task? When must the task be carried out? What result do we expect to see? 1 2 No. The kind of resources we need Where can we obtain this resource? How can we obtain this resource? How sure are we that we will get this resource? 1 2 35 Appendix 6 workshops Handout on sustainability for 'Feedback/ planning' Sustainability - what is it? To 'sustain' something means 'to keep it going' or'to make sure it continues to work' 'Self-sufliciency' and'Sustainability' An organisation is 'self-sufficient' if it only relies on its own resources for the work it does. If however its own resources are not enough, the organisation may succeed in finding resources from outside. If these resources are dependable we say the organisation is 'sustainable' - which is acceptable for CDTI programmes. Elements of sustainability We judge that projects will be more sustainable if the following aspects are present: Elfectiveness: Projects that are functioning effectively ile more likely to be sustainable Elliciency: Projects that are run cost-effectively are more likely to be sustainable. Simplicity: Projects that use simple, uncomplicated routines and procedures are more likely to be sustainable. Integration: Projects that have become integrated into the routine running of the health services are more likely to be sustainable. I a Attitude; Projects are more likely to be sustainable if staff have accepted CDTI as a routine activity, which they will continue to do even in the absence of additional material reward; and if they accept that they will have to adapt their routines to become more efficient. Resources'. Projects are more likely to be sustainable of they have enough resources (human, material, financial) to support what they are trying to do. Community ownership: Projects are more likely to be sustainable if there is active community participation in planning and execution of control strategy 36 I I