Guidelines for conducting a self-sustainability evaluation of an APOC project

fAfrican Programme for Onchocerciasis Control (APOC) Programme Africain de lutte contre l'Onchocercose National Onchocerciasis Task Force (NOTFs) of Nigeria Second meeting of the National Onchocerciasis Task Forces (NOTFs) Representatives, Abuja, 17-22 June 2002 GUIDELINES FOR CONDUCTING A SELF-SUSTAINABILITY EVALUATION OF AN APOC PROJECT DIR/APOC 04t06t02 World Health Organisation African Programme for Onchocerciasis Control Guidelines for conducting a self-sustainability evaluation of an APOC project JThis guideline should be read carefully by all members of evaluation teams (especially the leader) before embarking on an evaluation. It contains the following sections: 1. Background information 2 Preparing lbr the evaluatiorr visit 2.1 The 'John the Baptist' visit Managing the evaluation visit 3.1 Sarnpling 3.2 The sequence of the visit 3.3 The timetable for the visit 3.4 Using the instrument in the field 3.5 Data from the document study Analysing the data collected and making recommendations 4.1 Surnrnarising the field data 4.2 Grading each indicator 4.3 Grading groups of indicators 4.4 Describing the main findings verbally 4.5 Makingrecommendations 4.6 Grading a whole project 4.7 Mistakes to be avoided in tlie analysis The' Feedback/ planning' workshops 5.1 The liigher level (projecU State) workshop 5.2 The middle level (health district/ LGA) workshop 5.3 General comments about the workshops 6. Writing the report 4 5 Appendix 1 Appendix 2 Appendix 3 Appendix 4 Appendix 5 Appendix 6 Information for persons to be interviewed Questionnaire for health district/ LGA staff attending'feedback/ planning' workshops Programme for 'Feedback/ planning' workshop with the project/ State level team Programme for 'Feedback/ plaruting' workshop with the health districV LGA level team Detailed instructions for goup work in 'Feedback/ planning' workshops Handout on sustainability for 'Feedback/ planning' workshops 2 I nd information What is'self-sustainabilityr?I By'sustainability'we mean the likelihood that a project will continue to function effectively after APOC support comes to an end. Under'sustainability' there are two further concepts: * Self-sufJiciency: the ability of a project to continue functioning effectively, using only resources generated within the country itself. * Self-sustainability: the ability of a project to continue functioning effectively, using both its own resources and those provided from outside - provided the latter are dependable. In this exercise we are going to be working with'self-sustainability' as our guiding concept. Aspects of sustainability We judge that projects will be more sustainable if the following aspects are present: * Effectiveness: projects that are functioning effectively are more likely to be sustainable. * EfJiciency: projects that are run cost-effectively are more likely to be sustainable. * Simpltcity: projects that use simple, uncomplicated routines and procedures are more likely to be sustainable. * Integration: projects which have become integrated into the routine running of the health services are more likely to be sustainable. * Attitude: projects are more likely to be sustainable if staff have accepted CDTI as a routine activity, which they will continue to do even in the absence of additional material reward; and if they accept that they will have to adapt their routines to become more efficient. * Resources:projects are more likely to be sustainable if they have enough resources (human, material, financial) to support what they are trying to do. You will note that the different indicators of self-sustainability embody these elements. In particular the last three groups of indicators embody the 'Resources' aspect. Levels of self-sustainability We are going to evaluate self-sustainability at four levels, in the organisation of the health services of the countries concerned: A \atiqnal headquarters of the Ministry of Health Use Instrument no.1 B Province/ region/ State - a group of health districts C Health districU local government authority (LGA) Use Instrument no.2 D Health sub-district/ first line health facility Use Instrument no.3 E Village/ community Use Instrument no.4 Please note that the situation regarding these levels varies from country to country: * Level A is of course common to all projects in that country. It should be evaluated by itself, regarding self-sustainability. In Nigeria this level includes both 'Federal' and 'Zonal' components. * We evaluate the projects themselves at levels B to E. In some countries the project is based at level B, in others at level C. In the latter case there may be no level B. The J n important thing is to collect information at each level of the health service, where CDTIactivity takes place. t * we have developed 4 instruments for data collection about self-sustainability. In the table above it shows which instrument is to be used at which r"""r. ^- Indicators of self-sustainability These indicators were decided upon by the team which met in ouagadougou, and all appearin the instrument. They are grouped into 9 categories. These categories relate to:* The results achieved in the project (l group of indicators) * The routine activities which produce the results (5 groups of indicators) *- . The r?sources provided for these activities to take pfu." 1f groups of indicators).This results in the following groups of indicators at the four levels: Level of eration Groups of indicators Headquarters/ province/ region/ State District /LGA Sub-district/ First line facility Village/ Community l. Coverage t t t t Indicators of results achieved2. Planning t t t t3. Providing t t t t 4. Supervision and monl t t t t 5. Mectizan procurement and distribution t t t t 6. Training and sensitisation/ mobilization t t t t Indicators of activittes which support CDTI 7. Financing/ funding t t t t 8. Transport and other material resources t t t t 9. Human resources t t t t Indicators of resources provtded 4 the evaluation visit3. Man 3.1 Sampling The sampling is done purposively, not randomly. This is because a random sample from a smallpopulation may easily be non-representative. we use the following criteria: ' The primary criteri-on is coverage (geographical and therapeutic). This is because it is a measure of the performance of the whole system - and it ii the sllf-sustainability of that system that we are evaluating. ' The following secondary criteria are also taken into consideration: * Endemicr'ty: ensuring that, as far as possible, the sample contains both hyper-endemic and meso-endemic areas (more or less in the sa-e proportion as that of the REMO resultsfor the project area). * Geographical -spread: ensuring that, as far as possible, the sample contains areas which represent the different zones where the project operates, and communities which are closer to towns and some which are more isolated. (e.g. Nigeria) ' For each project, you sample at least three diJtricts (one with good coverage; one with medium coverage; one with poor coverage). ' For each district/ LGA, you sample two sub-districts (one with good coverage, one with poor coverage) ' For each sub-district, you sample two villages (one with good coverage; one with poor coverage). Forprojects containing l-2 districts (e.g. Uganda) . For each project, you sample the one or two districts. . If there ts only one district: * You sample four sub-districts (one with good coverage, two with medium coverage and one with poor coverage). " For each sub-district you sample two villages (one with good coverage and one with poor coverage) . If there are two districts: * You sample three sub-districts in each (one with good coverage, one with medium coverage and one with poor coverage). * For each sub-district you sample two villages (one with good coverage and one with poor coverage) In both of these examples you also you endeavour to satisfu the secondary criteria. This means that, if there are two items with more or less the ,u*..ou..age, you choose between them in such a way that the total sample also satisfies the secondary criteria. Since we only have a limited time to do the fieldwork, we also have to take into account accessibility and convenience. If there are two or more areas which are very sinrilar in every way, we select the one that is easiest to get to. 5 I3.2 The sequence of the visit Team members travel to the site of the project headquarters, arriving on the Saturday before the field work starts (which will be on the Monday following). They are expected to familiarise themselves with the following documents before arrival: * This'Guideline'document. * The four instruments. * The provisional timetable. These documents will be forwarded electronically to each team member. The team spends the Sunday planning for the coming work. They go through the guidelines and instruments together, adjust the provisional timetable if necessary, and constitute the teams which will be doing the fieldwork. Field visits take place from the Monday to the Friday: * Project data is collected at all the levels, at the sampled sites. * The NOCP office in the Ministry Headquarters also needs to be evaluated. This office serves all the projects in the countqr, so it only needs to be evaluated once. This evaluation will be the job of the first evaluation team to visit that country. They have to create a small team that will go back to the Ministry HQ at some time during the first or second weeks, to do the necessary data collection (interviews and document study). Note that this report is written up separately (except in countries with only one project, where it will form part of the project report). The weekend following is used to: * Analyse the data collected systematically - drawing the necessary conclusions and making recommendations. * Write the draft report. * Plan for the 'feedbaclc/ planning' workshops of the coming Monday to Wednesday I On the Monday to Wednesday of the second week, the 'feedback/ planning' workshops are held: * Monday: workshop for the project level * Tuesday and Wednesday: workshops for the health district/ LGA level. The idea is not to have too many people at one workshop, otherwise the workshop becomes difficult to handle. So if a project includes many health districts/ LGAs, half of these are invited to attend on the Tuesday, and the other half on the Wednesday. It is important to note that the evaluation team will gain insights during these workshops, that they did not pick up during the fieldwork. This additional information will have to be included in the data analysis, since it may well influence the recommendations that the team will make. Throughout the visit we recommend that the team meet every evening, to discuss the day's progress and problems, and to plan the following day's work. On the last evening they evaluate the whole evaluation process, and make recommendations about future recommendations. These are forwarded to all members of the core team (including Dr Amazigo). 6 I Another important aspect of the visit is the political one, of doing advocacy with decision makers at the local and national levels. The fact is that for the projects to be sustained, they are going to need regular and substantial financial contributions from the Ministry of Health and/ or local government. Outsiders are often able to state this obvious tmth with more authority than local workers. Each team must therefore pay courtesy calls to the relevant civil servants and political heads early during the visit, and if possible give them personal feedback towards the end of the visit. They should try to go as high as possible: to the Minister of Health her/ himself, or to the State governor (in Nigeria). The point to be emphasised during such visits is this: the Ministry has within itself a project which is really working well; a relatively small amount of money will keep it doing so; and failure to fund it will mean the retum of misery to thousands of people. 3.3 The timetable for the visit We suggest you spend 11 days at each project - we found at Taraba (the second evaluation, after the trial at Kaduna) that this was sufficient. There will be six of you in each team. You have to divide yourselves into three teams - let's call them I, [I and III: * Team I: This should be the team leader and one other. This team has to work with staff at Ministry Headquarters level (in case of this being the first evaluation in that country). When s/he has finished this task slhe joins the other teams for the district level work. Teams II and III: Two persons each, with a mix of expertise and experience. Here is a possible timetable: :* Saturday Travelling day - all teams go travel to the project headquarters Sunday Orientation day at the prqject headquarters Monday All teams: courtesy visits to MoH and NGDO management Team I: data collection at project headquarters (MoH and NGDO) Teams and III: data collection in the field: health district/ LGA level and below Tuesday Team I: data collection at project headquarters (MoH and NGDO) Teams II and III: data collection in the field: health district/ LGA level and below Wednesday - Thursday All teams: data collection in the f,reld: health districU LGA level and below In case this is the first visit to the country, Team I travels to the Ministry HQ to collect data for the national level Friday All teams: feedback to the health districts/ LGAs visited Saturday- Sunday All teams: . analyse the data, work out the recommendations, write the draft report plan for the coming 'feedback/ planning' workshops Monday 'Feedback/ planning' workshop for the project/ State level Tuesday 'Feedback/ planning' workshop for the health district/ LGA level (first round) Wednesday 'Feedback/ planning' workshop for the health district/ LGA level (second round, if needed) Thursday Departure for home '7 I You will clearly need to adapt this timetable according to the specifics of each situation. * The number of visits and interviews a team can handle per day will clearly vary from country to country and project to project. * Note again: in each country the Ministry Headquarters only needs to be visited once. This means that if there are two or more projects to be evaluated in a country, the Headquarters level only needs to be evaluated during one of these two evaluation visits. The information obtained there will then be made available to the second evaluation team. 8 the data collected and recommendations4. The whole team meets together to analyse the data. This mainly happens on the Saturday and fyndaV, but may begin earlier, in the evenings during the week. The team leader chairs thediscussion. Colleagues from the project should be en-couraged to be present. The followingprocess is followed: 4.1 Summarising the field data ' You do the analysis one level at a time. The first step is top assemble all the information thathas been collected at that level - from interviews, document study and observations. ' The team tackles the indicators one by one. Each piece of information that has been collectedfor that indicator is presented by the person who collected it. ' The chairperson uses a blank copy of the instrument (paper or electronic) and fills in eachpiece of information that is presented. This data will &entually have to be iypeo into theinstrument template for each lever, and will form part of the final report. ' Note that, due to time constraints, it may be necessary to divide up tt i, work between two teams by giving each that data from a different level io summarise. Try doing one together and see how fast you go. If you do split forces, each team must have the data from all sources about that level available. Each team then presents its findings to the other to check. 4.2 Grading each indicator ' Once all the evidence has been led for a particular indicator at a particular level, the team makes a joint decision on: 'Is this good for sustainability?'. The ftllowing gr.ading is used: Indicators in Group I have a modification of this system, as follows: * For 9 Grading Meaning Numerical value Ful The criterion is full satisfied/ met. 4 The criterion is almost met - there is a small deficit 3 Moderatel The criterion is about tulfilled. 2 S The criterion is tulfilled. I Not at all The criterion is not fulfilled at all. 0Not icable This criterion is not relevant to this case. Gradin Numerical value Ful rc0% 4 blem with nomads aJ Moderatel a few hamlets not 2 hamlets notvl I :* For therapeutic coverage Grading Numerical value Eqlly (>85% coverage) 4 Highly (7 5 -84% coverase) 3 Moderately (65-7 4% coverage) 2 Sliehtly (less than 650/o coverage) I 4.3 Grading groups of indicators Once all the indicators have been graded by the team, a score can easily be worked out for each group of indicators. This is done by getting a mathematical average - for example: For the'Planning'group of indicators at District/ LGA level there are tlree indicators. Suppose the scoresfor these three are 3, j and 2. The overall scorefor'Planning'at this level is therefore (3+3+2)/3 : 2.7 Note though that the 'average numerical value' is only a representation of the ordinal series: 'fully', 'highly' .. etc. The scores for all the groups of indicators at a certain level may then be illustrated in a table, or graphically - for example: In this way the main problem areas for sustainability are made clear. 4.4 Describing the main findings verbally Kaduna Project: self-sustainability at LGA level group of indicators 4 3.5 3 S z, 2.5 .9 o =zo P r.sL ok1 0.5 0 ./ o*"'"" "."t 's"" ^c "-.*t .""r-""t" ""/ -"""" "".".'t "^.'"'"- .'C ou"" ."""" l0 Having graded a group of indicators (e.g. 'Leadership') the factual content of that group must be summarised in a paragraph. Both positive and negative findings are given, in such a way that all agree that the totality of the data is represented.Theresult of this process is nine paragiaphs of varying lengths. Note that this is a critically important step of the evaluation. The grading exercise point to areas of major weakness - a skeleton - but this analysis produces the meat of the report. The recommendations that follow will be principally based on it. 4.5 Making recommendations Finally the group has to make recommendations, about steps that projects and countries need to take to achieve self-sustainability. These recommendations are made for each of the four levels, and should have the following characteristics: ' They must be strictly based on the findings of the field work. They address the areas of poor self-sustainability that the research has uncovered. . They must be practical and achievable: * For each recommendation a suggestion must be made, about who should be responsible for carrying it out. * For each recommendation a deadline or time line should be suggested. * One or more indicators of achievement must be given for each recommendation. . Recommendations should be prioritised, as follows: * HIGH: this recommendation is critically important. If it is not carried out the project will not be sustained. * MEDIUM: this recommendation is important but not critical. Carrying it out will enhance the self-sustainability of the project though. * There should not be any LOW priority recommendations. 4.6 Grading a whole project Finally the team may grade the whole project, in terms of its overall sustainabitity. The expert group meeting in Ouagadougou considered that it could be useful to grade each project, for the following reasons: ' It gives an immediate idea of the self-sustainability of the project as a whole. ' It sets a baseline against which to measure progress in achieving self-sustainability. This evaluation is done using all of the available information gained during the course of the evaluation. It is recommended that teams resist the urge to allocate a numerical value to the sustainability of a project - the grading is an overall impression, and not a mathematical summative scoring of individual elements of sustainability. When grading the project as a whole, it is important to bear in mind that there are a number of critical elements of sustainability, without any of which it is unlikely that any project will be self-sustainable: ' Money - there should be sufficient money available to undertake strictly necessary tasks which have been carefully thought through and planned (absolute minimum residual activities). 1l . Transpor, - provision must be made for the replacement and repair of vehicles. There must be a reasonable assurance that a vehicle will be available for minimum essential activities. Note that 'vehicle' does not necessarily imply '4x4' or even 'car'. . Supervtston -the project will not be sustained without timeous mobilisation and targeted supportive supervision. . Mectizan supply - the supply system must be dependable. The bottom line is that enough drugs must arrive in villages at the time selected by the villagers. . Poliltcal commitment - effectively demonstrated by awareness of the CDTI process among policy makers (resulting in tangible support); and a sense of community ownership of the programme. Other indicators remain important and should be taken into account together with the more critical elements listed above. Against this background, the following schedule may be used to grade the project as a whole: 4.7 Mistakes to be avoided in the analysis The following problem areas were noted in the analysis of the first two projects: . The analysis (and recommendations) could focus on the functioning of the project, rather than its sustainability. . The analysis could be incomplete, not taking significant pieces of field data into account. . The recommendations could be superficial, non-specific and unrealistic. . Sometimes findings at one level were not consistent with those of another level. The team leader must ensure that these mistakes are avoided. Level of sustainability Description Excellent This project is completely sustainable. High This project is not far from being sustainable. With feedback from the team before departure, the project staff should be able to undertake the required remedial action. Moderate This project is potentially sustainable, but will require rethinking and mobilisation of high-level support to get it on the road again. Low This project is seriously unsustainable - there is some doubt as to whether it ever will be. It needs a lot of immediate expert guidance from outside. l2 5. The'Feedback/ plannin 'workshops This evaluation exercise aims to set the projects firmly on the road to self-sustainability. This means that the team has to do more than evaluate - it also has to set a process of planning in motion, based on its findings. This is done through two or more 'feedback/ planning, wJrkshops, to be held at each level where major decisions about the CDTI programme are made. These workshops have the following aims: . Feedback In the workshop we present the evaluation team's findings to all the relevant project staff and political figures, who have responsibility for the CDTI programme, The team has to inform them what happened - it has to present them with its findings. . Planning We also have to enable project and political staff at these levels to work together to develop a realistic programme for the coming years, based on the findings of the evaluation. This plan should as far as possible ensure the sustainability of the participants' own CDTI programme. The process described below is based on experience gained so far in the Taraba and Kaduna evaluations. From them we have leamt the following: ' Preparation for the workshops should be done by the project directors (Ministry and NGDO) and the 'John the Baptist' person. They have to invite those who will be attending, in good time. Written invitations to those who should attend, at least a week before the event ' Separate workshops are held for the project/ State level, and the health district/ LGA level. The higher level workshop is held first, since it acquaints staff at that level with the workshop process - which they will help to facilitate on the following day(s), for the middle level. The workshops should be held at the project headquarters. ' If considered necessary a third workshop may be held for the national level. . Each workshop lasts for takes 7-8 hours. 5.3 The higher level (project/ State) workshop The fol should attend thisI ' The programme for the workshop is designed to fulfil the two objectives given above. An example of such a programme is given in Appendix 3. This programme may be used as an example, but will obviously need to be adjusted according to the circumstances. More details about the programme are given in 5.3 below. ' Members of the evaluation team function as facilitators for this workshop. 5.4 The middle level (health district/ LGA) workshop - Senior officials in charge of the project (e.g. director of the section in which the project leader works; the permanent secretary). - Field workers implementing the project at this level/ SOCT. From the NGDO: the officer allocated to the project.* .t( * contributes toother stakeholder who at this level From the official health service: - The project leader/ SOC. l3 tThe fol should attend this One or two of these workshops may be held - this depends on the number of persons who will be attending. It is difficult to run a workshop if there are more than 40 participants. In Nigeria for example one project may contain up to 16 LGAs, who are each asked to send 4-5 delegates. In such a situation it is clearly necessary to run two identical workshops on consecutive days. Once again, the programme for the workshop is designed to fulfil the two objectives given above. An example of such a programme is given in Appendix 4. This programme may be used as an example, but will obviously need to be adjusted according to the circumstances. More details about the programme are given in 5.3 below. Note in particular that the workshops at this level provide us with the opportunity to gather more important information about self-sustainability. This is because we are able to meet those health districts/ LGAs which were not included in our sample. A questionnaire may therefore be administered to them, at some time during the day's programme. An example of such a questionnaire is given in Appendix 2. The data obtained in this way should be used to revise the findings and conclusions that were made, following the initial analysis of the data collected during the field work. The facilitators at this workshop are: * The evaluation team * The persons who attended the higher level workshop 5.3 General comments about the workshops In some situations the health district and the project are the same. In that case only one workshop needs to be held. It is important that the workshop should be led by someone with good workshop skills - someone with experience of running workshops. You should appoint the team member who is most experienced in this way, to be in charge of the workshop programme. I From the official health service: - The management team responsible for running the health service at this level; including the person responsible for disease control programmes. - The CDTI prograrnme manager at this level. Relevant civil authorities at this level.* * * other stakeholder who at this levelcontributes to t4 f T Appendix 5 Detailed instructions for group work in 'Feedback/ lanni 'worksh s TASK'tiTirffifrttiitg.,f,pfan for the rest of this year There are three kinds of activity that we need to perform: 1. We have to do everything that is needed to make sure that this year's distribution is completed successfu lly. 2. We have to correct problems that exist, while we still have some extra money to do this. 3. We have to undertake activities which will mobilise extra resources for our work next year, since we will no longer have APOC funds then. We are going to write our plan in the format given below: TASK 2: Mdliing'a plan for next year There are two kinds of activity that we need to perform: 1. We have to do everything that is needed to make sure that this year's distribution is completed successfully. However we will have less money, so the activities we plan must cost as little as possible. 2. Again, we have to undertake activities which will mobilise extra resources for our work the following year. We write our plan using the same format as above. TASK 3: Making a list of the resources we will need next year To carry out our work programrne for 2003 we need to have resources: staff, transport, money, stationery etc. We need to think carefully where we can get such resources. We make a list, using the following format: 3 3 No Activity/ task (very specific) Why do we need to do this? Who should carry out the task? When must the task be carried out? What result do we expect to see? 1 2 No. The kind of resources we need Where can we obtain this resource? How can we obtain this resource? How sure are rve that we will get this resource? I 2. l5 Appendix 6 Handout on sustainability for 'Feedback/ planning' worksho s Sustainability - what is it? To 'sustain' something means 'to keep it going' or'to make sure it continues to work'. 'Self-sufficiency' and'self-sustainability' An organisation is 'self-sufficient' if it only relies on its own resources for the work it does. If however its own resources are not enough, the organisation may succeed in finding resources from outside. If these resources are dependable we say the organisation is 'self-sustainable' - which is acceptable for CDTI programmes. Elements of sustainability We judge that projects will be more sustainable if the following aspects are present: ' Effectiveness: Projects that are functioning effectively are more likely to be sustainable. Elliciency: Projects that are run cost-effectively are more likely to be sustainable. Stmplicity: Projects that use simple, uncomplicated routines and procedures are more likely to be sustainable. Integration: Projects which have become integrated into the routine running of the health services are more likely to be sustainable. Attitude: Projects are more likely to be sustainable if staff have accepted CDTI as a routine activity, which they will continue to do even in the absence of additional material reward; and if they accept that they will have to adapt their routines to become more efficient. Resources: Projects are more likely to be sustainable of they have enough resources (human, material, financial) to support what they are trying to do. a l6