Evaluation of the sustainability of the Kaffa-Sheka CDTI project, Ethiopia: October 2003

World Health Organisation African Programme for Onchocerciasis Control FINAL REPORT Evaluation of the Sustainability of the Kaffa-Sheka CDTI project, Ethiopia October 2003 Evaluation Team Members: Eleuther Tarimo (Team Leader) Elizabeth Elhassan Atabe Andrew Cyrille Evini Uwem Ekpo Abraraw Tesfaye Kora Tushune 2TABLE OF CONTENTS Page Table of Contents ...................................................................................................... 2 Abbreviations/Acronyms ........................................................................................ 3 Acknowledgements ................................................................................................. 4 Executive Summary ................................................................................................. 5 1. Introduction ........................................................................................................... 9 2. Methodology ......................................................................................................... 9 2.1 Sampling ........................................................................................................... 9 2.2 Levels and Instruments ................................................................................ 9 2.3 Protocol ...................................................................................................... 11 2.4 Team Composition ..................................................................................... 12 2.5 Advocacy Visits and ‘Feedback/Planning’ Workshop................................. 13 2.6 Limitations .................................................................................................. 13 3. Major Findings, Discussions and Recommendations .................................... 14 3.1 State/National Level ....................................................................................... 14 3.2 Local Government Area/Zonal Level .............................................................. 20 3.3.District/Woreda Level............................................................................................................. 3.4 Front Line Health Facility Level ...................................................................... 24 3.5 Community/Kebele Level ............................................................................... 32 4. Conclusions ..................................................................................... … ……….37 4.1 Grading the Overall Sustainability of the Kaffa-Sheka CDTI project ............... 37 4.2 Grading of Project as a Whole ........................................................................ 40 ANNEXES ................................................................................................................. 42 Interviews .............................................................................................................. 50 Schedule for the Evaluation, Advocacy ................................................................. 52 Feedback and Planning Workshop Agenda .......................................................... 54 Report of the feedback/Planning Workshop .......................................................... 54 3Abbreviations/acronyms APOC African Programme for Onchocerciasis Control CDD Community Directed Distributor (of Ivermectin) CDTI Community Directed Treatment with Ivermectin EPOC Ethiopia Programme for Onchocerciasis Control FLHF First Line Health Facility H/Q Headquarters H/C Health Centre H/P Health Post HSAM Health Education/ Sensitization/ Advocacy/ Mobilization IEC Information, Education, Communication MoH Ministry of Health NGDO Non-Governmental Development Organization NGO Non-Governmental Organization NID National Immunization Day NOTF National Onchocerciasis Task Force PHC Primary Health Care REMO Rapid Epidemiological Mapping of Onchocerciasis RHB Regional Health Bureau TCR Therapeutic Coverage Rate WHO World Health Organization WHD Woreda Health Desk ZHD Zonal Heath Desk ZOTF Zonal Onchocerciasis Task Force 4Acknowledgements We would like to express our appreciation and gratitude to the following institutions and persons for their help:  Dr Sékétéli, Dr Amazigo and other staff at the Headquarters of the African Programme for Onchocerciasis Control (APOC) in Ouagadougou  His Excellency Vice-Minister of health, NOTF Chairman and other policy makers in the Federal Governmentof Ethiopia  WHO Country Representative and all staff, Ethiopia  The National Coordinator for Onchocerciasis Control programme, Ethiopia  The management of Jimma University, for the valuable support provided for the duration of the evaluation  The Country Representative of the Carter Center/Global 2000, Mr. Teshome Gebre, for logistical and technical support  Dr Assefa Worku, Programme Officer, Global 2000, drivers and other staff from Global 2000 and MoH, for undertaking all the necessary arrangements.  Political and traditional leaders, health workers and community members of areas visited in Kaffa and Sheka zones as well as all political leaders at woreda/district level (Masha, Yeki, and Gimbo woredas)  The scout, Mr. Abraraw Tesfaye, for having planned for the evaluation. 5Executive Summary The Kaffa-Sheka Zone CDTI project was started in 2001. Soon after the project started the zone was divided into two, Kaffa and Sheka; with 10 and 3 districts (woredas) respectively. Mectizan distribution started in one district of Kaffa Zone and three districts of Sheka Zone. The programme has now been expanded to cover all the 13 districts (woredas) of Kaffa- Sheka CDTI Project. A mid-term evaluation (in Kaffa and Sheka, the only zones where treatment had been carried out for three full years) was carried out early in October 2003. The evaluation exercise was carried out both at national (federal and region) and project (zonal, woreda, FLHF, Kebeles) levels. Results of the evaluation were presented to a feedback and Planning meeting. Table 1 below shows achievements (scores) for groups of indicators for different levels within the Zones. These scores show that the project has been very successful in many areas. Table 1, Scores on sustainability indicators at Project level. P la nn in g Le ad er sh ip M on ito rin g & S up er vi si on M ec tiz an s up pl y Tr ai ni ng a nd H S A M Fi na nc es Tr an sp or t H um an R es ou rc es C ov er ag e A ve ra ge Zonal level 3.0 4.0 3.6 3.5 2.6 1.5 2.7 2.0 4.0 3.0 Woreda level 2.0 2.0 3.0 4.0 2.5 1.0 2.0 2.0 4.0 2.5 FLHF level 1.0 1.0 3.0 3.5 1.5 1.0 1.5 3.0 4.0 2.2 Community 3.0 3.6 3.0 3.5 4.0 2.0 3.3 3.3 4.0 3.3 Average 2.6 2.7 3.2 3.6 2.7 1.4 2.4 2.6 4.0 Coverage (geographical 100% and therapeutic well over 65% in all woredas) is the area of most success, indicating a situation which fully supports sustainability. Mectizan supply and Monitoring/supervision scored highly, next to coverage. Mectizan® collection, storage, delivery to lower levels and control are carried out effectively and in an integrated manner within the government system. Mectizan® is generally available in time and in adequate amounts. There was a delay in delivery during the last round of treatment following recommendation by MDP who had received information that Loaloa might exist in Ethiopia. However, following investigation, it was established that the level of loaloa was not a threat to mass treatment with Mectizan® in the area and treatment was carried out two months later. In all kebeles visited, census updates and Mectizan® distribution were carried out at different periods. 6Monitoring and supervision by zonal oncho coordinator was carried out mostly in the month of Mectizan® distribution. These supervisions mainly target woredas/FLHFs. Visits to kebeles were carried out randomly. Supervision checklist exists. Supervision at each of the levels was not integrated with other programmes. Both financial and technical reports on CDTI activities were communicated within the HMIS. Performance for other indicators was moderate with the exception of 'finances' which was slightly. On Planning it was noted that three-year strategic health plans for health sector (which include onchocerciasis) had been developed at the Zonal and Woreda levels. Half of the FLHFs visited had timetables for CDTI activities. Planning was weak at the woreda level and even more so at the FLHF. Zonal level provided strong leadership for CDTI. There is a focal point for CDTI at the Zone and Woreda levels. Woreda and FLHFs have not taken full ownership and responsibility for CDTI. Community leaders played leadership role in CDTI; they select CDDs and change them as necessary and mobilize communities for CDTI. All individuals interviewed expressed their wish to take Mectizan® for as long as it takes to get rid of the disease (a figure of 12 years was often mentioned). Many advantages for taking Mectizan were mentioned; however, at times it was seen as a ‘wonder’ drug curing sexually transmitted infections, high blood pressure, paralysis and other diseases. Training and HSAM were carried out well. Staff who were interviewed (at all levels) had adequate skills, valued their work and expressed strong willingness to continue serving in their present position. Issues in this area included inadequate targeting and integration of training. It was claimed that each program had its own funding source and timetable and thus there was no need for integration. CDDs with one or two exceptions had all been trained well. HSAM materials were adequate. HSAM activities have led to increased awareness / acceptance of CDTI within communities and support from higher authorities. Budgets at the Zonal level contained all CDTI key activities with estimates clearly spelt out. Budgetary contributions of partners were clearly spelt out .Budgeting for Woreda level was essentially carried out and managed at the Zone. As a result personnel at the woredas and FLHF did not know the amount of funds provided for activities. There was no evidence of cost containment. The Zonal level was of the opinion that funds received were adequate for CDTI activities. However delays in release (or loss of funds) due to delays in the retirement of funds resulted in shortages in the field. For the first time, financial contribution by the government for CDTI was included in health plans (2003 – 2005 strategic plans) both at the Zone and Woreda levels. Support to CDDs by communities varied from place to place. Apparently rumors that CDDs were being paid by external donor agencies and that money earmarked for those who take Mectizan® was being withheld by government worked against support for CDDs by some communities. With regard to transport, vehicles at all levels were operated in a pool. Funds for maintenance were provided from pooled resources. With the expansion of CDTI to 13 Woredas (from an initial 5) there was shortage of transport and essential equipment. Though management was aware of the necessity to replace existing transport means, there was no realistic plan to do so due to scarcity of resources from Government. Management was hoping to get outside support. Control of vehicles was considered adequate. But there was no evidence of routine maintenance schedule for motorcycles at the woreda level. Staff’s coping mechanisms when motorcycles breakdown include using other motorcycles from the pool. Thus CDTI activities have not been disrupted because of vehicle breakdown. 7High turnover of staff was an important issue. The ratio of 2 CDDs for 250 persons as recommended by APOC was not followed. The routine is 2 CDDs per village irrespective of the size of the population and type of settlement. Drop out CDDs were immediately replaced. All the CDDs interviewed expressed willingness and dedication to continue with their CDTI activities. Some of them even remarked that it was a moral obligation to serve the people who selected them. On the overall grading of the project the evaluation team found that only one of the seven aspects of sustainability, ‘resources’ was blocking sustainability. With regard to ‘critical elements’ it was found out that only one ‘money’ was not satisfied. In accordance with the Evaluation Guidelines the team concluded that Kaffa-Sheka project is making satisfactory progress towards sustainability. Assessment at national level. Table 2 Scores at national and Regional levels P la nn in g M on ito rin g /S up er vi si on M ec tiz an or de r an d su pp ly Tr ai ni ng an d H S A M In te gr at io n of su pp or t ac tiv iti es Fi na nc ia l R es ou rc es Tr an sp or t a nd ot he r re so ur ce s H um an re so ur ce s C ov er ag e 2.5 1.7 4.0 2.7 3.0 2.3 2.5 2.5 4.0 Onchocerciasis control is included in the national and Regional strategic plans (2003 - 2005). Detailed plans with specific objectives and key CDTI activities for each of the three years 2001- 2003, both at the national and Region levels were available. There was close collaboration between MoH and the Carter Center in the development of the project and activities. MoH owns the project and empowers lower levels through training and technical support. As a result of under staffing at the MoH, the NGO partner was asked to be secretary of NOTF. There is no specific sustainability plan (neither at the national nor regional level) for the period after APOC funding is withdrawn. National and regional staff monitored and supervised zonal level but carried out spot checks at other levels. The zone has been empowered to supervise activities at the woreda level. Annual review meetings (involving key staff from regional and woredas) provided an opportunity to identify and solve problems. Mectizan® was ordered by the NOTF within the government system. Clearance and storage of Mectizan® is by WHO. The control system is within the government system along side the supply of other drugs. The region collects Mectizan® from WHO and notifies the zones for collection. The zones notify the woredas to effect collection. After collection, woredas deliver to FLHFs. Sufficient Mectizan® tablets were being ordered based on the need generated from the CDDs by the NOTF. 8Staff at the national level (National Coordinator, NGO and WHO) and regional level only trained staff at the region and zone and had empowered these levels with adequate skills to train staff at woredas level. CDTI staff were responsible for several interrelated projects (CDTI, malaria), information was obtained from relevant sources during monitoring and supervision of one project. The evaluation team was impressed by the extent of integration in Ethiopia (at both the national and regional levels), for example transport was pooled and no staff member was responsible for only a single project. Budget for different activities (maintenance, monitoring and supervision) was also pooled. On finances the cost of each activity in the year plan was clearly spelt out in the budget at both levels. There was no evidence of a cost containment strategy. The budgetary provisions by government and NGO partner were clearly spelt out in the revised project proposal and annual agreements. The government contribution was in four budget lines (salaries, capital, communication and operating costs). The actual disbursements from government were payment of staff salaries. Documentation for the disbursements for the other three budget lines was not available. The team was informed that disbursements were from pooled funds (government and donor funds). The NGDO partner was unable to make a commitment beyond 2005. As government does not allow separate project accounts, project funds are in WHO account. The NOTF manages the funds from the WHO account. On transport, vehicles were serviced regularly, costs of maintenance were met through pooled funds.The use of transport was properly controlled. Management was aware of the need to replace transport and other materials. The plan for replacement was the use of pooled vehicles from other projects (malaria, TB & Leprosy). Therapeutic coverage was over 65% in all woredas. A three-day Feed-back/planning meeting was held after the evaluation. A SWOT analysis of the CDTI project was carried out after presentation of the evaluation findings. Participants worked in two groups, (National, Region and Zone) and (Woreda) to develop sustainable CDTI plans. The evaluation team welcomed the move to give more visibility and ownership to Woreda level. The move which will greatly enhance sustainability of CDTI calls for considerable development and strengthening of installed capacity in the Woredas. A recurring issue in the meeting, perhaps reflecting the strong mindset for integration in Ethiopia centered on the need for the government to provide increasing resources for CDTI. It was argued that on one hand a call is made for integration and on the other advocacy is made for separate allocation of funds to CDTI. Was this not a contradiction? It was pointed out that it was critical to have some assurance that basic funding would be available to sustain CDTI. There was always the danger that big programs like malaria might take most of the funds in a basket. Others argued that Malaria with its large budget could be the lifeline to sustain CDTI. A related issue was that integration of training seemed to be difficult even in Ethiopia despite her solid commitment to the concept of integration. Ethiopia has a good environment and context to find solutions through operational research to these and related issues. 91. INTRODUCTION Kaffa-Sheka was one of the 12 zones of Southern Nations, Nationalities and Peoples Region, SNNPR. CDTI was introduced to Kaffa-Sheka Zone in 2001. Immediately after the CDTI launching workshop, Kaffa and Sheka zones became administratively independent. Kaffa Zone consists of 10 districts (woredas), while Sheka Zone has 3 districts (woredas). Mectizan distribution has been going on for the last three years in one district of Kaffa Zone and three districts of Sheka Zone. In fact, in the second and third years, the programme has been expanded to cover all the 13 districts (woredas) of Kaffa-Sheka CDTI Project. That is why samples for the mid-term evaluation have been taken only from the four CDTI districts. 2. METHODOLOGY While taking the samples at various levels, the scout team faced some difficulties in corresponding terminologies in the APOC instruments to what are actually in the project area; there are some differences in administrative/political structures in Ethiopia compared to those indicated in the instruments. Taking the flexibility option into consideration, the team has finally tried to make some adjustments. Accordingly, the three evaluation instruments (2, 3, and 4) are to be used systematically to carry out the evaluation at three levels. • Instrument 2 is to be used for zonal level – equivalent to LGA level in the instrument. • Instrument 3 is to be used for both woreda and FLHF levels together – equivalent to Sub-district/FLHF level in the instrument. • Instrument 4 is to be used for kebele level – equivalent to Community/Village level in the instrument. 2.1 Sampling of Districts (Woredas) Sampling was done according to APOC guidelines, using primary and secondary criteria. Primary criterion is geographical and therapeutic coverages; accordingly, three districts/woredas (with highest, medium and lowest coverage) were selected. Moreover, secondary criteria were considered. The secondary criteria include: endemicity, geographical spread, and accessibility/convenience. Ultimately, three districts/woredas, namely Masha, Gimbo and Yeki were sampled; Andracha district was excluded due to 10 accessibility and convenience criteria. Instead, Masha District (Woreda) was selected for evaluation. (See annex ). 2.2 Sampling of Sub-districts/FLHFs The sub-districts/FLHFs were also sampled taking the aforementioned criteria into consideration. Accordingly, two sub-districts/FLHFs (one with highest and the other with lowest coverage) were selected from each district/woreda. However, some sub- districts/FLHFs were excluded based on secondary criteria.  Masha District: o Masha Health Centre and Yina FLHF  Gimbo District: o Wushwush Tea Plantation FLHF and Argoba FLHF  Yeki District: o Zinki FLHF and Kubito FLHF 2.3 Sampling of Communities (Kebeles) Two communities (kebeles) from each sub-district/FLHF (one with highest and the other with lowest coverage) were selected based on primary and secondary criteria.  Masha Health Centre: o Abello and Gembeka Communities (kebeles)  Yina FLHF: o Yina and Gada Communities  Wushwush Tea Plantation FLHF: o Bubba and Shuka Communities  Argoba FLHF: o Choba and Argoba Communities  Zinki FLHF: o Zinki and Tsanu Communities  Kubito FLHF: o Ermichi and Kubito Communities 11 2.2 Protocol  Research Question: How sustainable is the Kaffa-Sheka CDTI Project?  Design: Cross-sectional, descriptive.  Population: The Kaffa and Sheka CDTI project, its NGDO partner (Global 2000/The Carter Center), the staff involved in onchocerciasis control at woreda and FLHF levels, the project communities, with their leaders and CDDs.  Instrument: * A record sheet, structured as a series of indicators of sustainability. The indicators were grouped into nine categories/ groups. These groups represent critical areas of functioning of the program. * The instrument assesses sustainability at four levels of operation. * The instrument guides the researcher to collect relevant information about each indicator from a variety of relevant sources.  Sources of information: ∗ Documentary evidence and observations. ∗ Verbal reports from persons interviewed.  Analysis: * Data from all sources was aggregated, according to level and indicator. * A qualitative summary of the situation regarding each indicator at each level was made. This was aggregated and summarized for each category of indicator for each level. * Based on the information collected, each indicator was graded on a scale of 0-4 in terms of its contribution to sustainability. * The average ‘sustainability score' for each group of indicators was calculated, for each level. * Finally an overall assessment of sustainability was made, by considering the 7 aspects and 5 critical areas of sustainability.  Recommendations: ∗ These were strictly based on the findings of each program evaluated. 12 2.4 Team composition The core team members were the following: 1. Dr Eluether Tarimo (Team leader) ∗ Box 33277 Dar es Salaam, Tanzania ∗ T (home) +255 222 775 891 T(cel) +255 744 315485 ∗ Email: eleuther@ud.co.tz 2. Dr. Elizabeth ElHassan ∗ Sight Savers International ∗ 1 Golf Course Road ∗ P.O. Box 503 Kaduna, Kaduna State, Nigeria ∗ T (home): (234) 62 210 723 T (office): (234) 62 248 973 or 248 360 ∗ Email: ssing@infoweb.abs.net, elizabethelhassan@yahoo.co.uk 3. Mr. Cyrille Evini ∗ Helen Keller International, Cameroon ∗ P.O. Box 14227, Yaoundé, Cameroon ∗ T (mobile): 237 771.07.21; T (office): 237 220.97.71 ∗ Email: cevini@hki.org, cyrilleevini@yahoo.fr, evini_cyrille@hotmail.com 4. Dr Andrew Atabe ∗ Sight Savers International, Cameroon ∗ P.O. Box: 4794 Yaounde ∗ T (Home): +237 767 27 29; T (office): +237 221 12 33; Fax: +237 221 79 43 ∗ Email: atabe_andy@yahoo.co.uk , ssicam@iccnet.cm 5. Mr. Uwem Ekpo ∗ Department of Biological Sciences, University of Agriculture, PMB 2240 Abeokuta, Nigeria ∗ T (mobile): 234 803 335 1706 ∗ Email: ufekpo@hotmail.com 6. Mr. Abraraw Tsefaye * Jimma University ∗ Tel. 251-07- 117689 (Home), 111461 (0ffice) ∗ E-mail: abrarawt@yahoo.com 7. Mr. Korra Tushune * P.O. Box 378 Jimma Ethiopia * Tel.(home) 251-7-114616, (off) 110951 * E-mail: ktushune@yahoo.com 13 The evaluation team members were grouped into three sub-teams for the purposes of fieldwork. Each sub-team had one or two persons that served as guides, facilitators, and translators. 2.5 Advocacy Visits and ‘Feedback/Planning’ Workshop Advocacy visits were paid to relevant people at each level as possible. Debriefing was done at the feedback and planning workshop. During the workshop, the evaluation team gave feedbacks on its findings and guided the zones and woreda teams in the development two-year sustainability plans taking the evaluation findings into consideration. 2.6 Limitations Reports of annual financial expenditures (APOC and MOH) by item were not available for review; thus financial efficiency of the project could not be assessed. Integration seems to be limited to “internal” integration of CDTI. There is a need for a separate indicator related to integration of CDTI with overall PHC. No indicators for “leadership” at national level. Thus no provision is made for assessing this important and critical aspect of sustainability. 14 3. MAJOR FINDINGS, DISCUSSIONS AND RECOMMENDATIONS 3.1 Zonal Level 3 4 3.6 3.5 2.6 1.5 2.7 2 4 0 0.5 1 1.5 2 2.5 3 3.5 4 Score/4 Pl an ni ng Le ad er sh ip M on ito rin g /S up er vi si on M ec tiz an o rd er a nd s up pl y Tr ai ni ng a nd H S AM Fi na nc ia l r es ou rc es Tr an sp or t a nd o th er m at er ia ls H um an re so ur ce s C ov er ag e Groups of indicators Scores at Zonal Level PLANNING (Highly, 3): There was a CDTI plan included in the overall strategic three-year plan for health programmes. The plan made provision for all key CDTI activities such Mectizan® supply, training, monitoring and supervision. Although the management team at zonal level drew up the plan, it evolved from Woredas. The existence of an overall health plan incorporating CDTI activities was made possible due to the responsibility of the health desk officer who also supervises other disease control programs such as malaria, EPI and TB. LEADERSHIP (Fully, 4): The management team at this level has the capacity to initiate key CDTI activities evidenced by the presence of a detailed timetable for the last and even coming treatment 15 rounds. There is a focal person in charge of CDTI at this level. This focal person is also responsible for all other health programmes. MONITORING/SUPERVISION (Highly, 3.6): Both financial and technical reports on CDTI activities were communicated within the HMIS. The reports contained all the necessary data concerning technical and financial details of CDTI activities. However, in some circumstances, reports were also sent to the NGDO when the need arose. During Mectizan® distribution campaign, the zonal oncho coordinator carried out supervision twice a month. Supervision visits mainly targeted woredas and FLHFs. However, visits to the kebeles were carried out randomly. During CDTI supervision visits, the zonal oncho coordinator exclusively oversaw CDTI related activities. Each supervision visit lasted three to four days. In most cases, CDTI supervision visits were made routinely. Occasionally, however, they be motivated by particularities, such as high rates of refusals and/or absentees in a woreda. There was supervision checklist. Sometimes, the checklist was actually used. However, reports of supervision were not written. The personnel at zonal level indicated that CDTI supervision visits were made during specific time, and therefore, would not be integrated into those of other health programmes that were carried out frequently. It was also reported that, due to time constraints, and in order to put greater emphasis on CDTI, there was no need for integration with other programmes. Problems identified during CDTI supervision were addressed promptly in conjunction with the persons in charge. Supervisees were encouraged and commended for their performance. In one of the two zones visited, the best performing CDD during the previous round of treatment was given an award (radio set). In previous treatment rounds, all CDDs were given certificates of commendation. MECTIZAN® PROCUREMENT AND DISTRIBUTION (Highly, 3.5): Mectizan® orders were based on needs and requests from woredas. In one zone, collection of the drug from the regional office took not less than five days excluding the time required for administrative procedures. Not withstanding, Mectizan® was available on time and in 16 adequate quantities. No shortage had been reported. However, Mectizan® delivery was delayed during the last round of treatment. Mectizan® destined to zone was already packaged from MDP. In the two zones visited, there was good documentation of drug management. Mectizan® collection, storage and delivery to lower levels was carried out effectively and in an integrated manner within government system. Woredas collected their supply at zonal level. Funding for transportation was shared between APOC sources and government. TRAINING AND HSAM (Moderately, 2.6): In one of the two zones, zonal personnel trained woreda and FLHF staff at the same time. When Woreda/FLHF personnel were found to be capable, they served as trainers during the session. In most cases, FLHF personnel were in charge of training CDDs. Training was routine and sometimes not targeting specific needs. Training curricula did not include other health programmes. In some cases, training lasted three days for each episode. The reason given for not integrating training episodes was that each programme had its own funding source and timetable for training. Due to the high turnover of political leaders, briefings were frequently carried out as soon as new people were appointed. For the next round of treatment, the council has specifically allocated funds for CDTI activities. HSAM activities were targeted and effective. FINANCIAL RESOURCES (Slightly, 1.5): The budgets contained all CDTI key activities with estimates clearly spelt out.  The budgetary contributions of the partners were clearly spelt out.  There had been no direct allocation of funds by government.  There was no plan for the government to fund the bulk of CDTI expenses by the end of Year5 of APOC funding.  Given that year4 activities were yet to start, it was not possible for the evaluation team to assess government budgetary contribution to CDTI programme. 17  There was no indication of increasing proportion of government expenditures on CDTI activities. The reason given was that financial resources in both zones were limited.  In order to address this issue, provision had been made from zonal government to support CDTI for the coming year in one zone. In the other, the team was informed that funds had actually been allocated to CDTI for the coming year  Although the zonal management has been aware of the APOC funding coming to an end in two years time, there was no evidence of a plan to assess the budget needed to sustain CDTI activities and find ways to mobilize funds from dependable sources. Managerial activities regarding funds for both zones include: o allocation of funds for each line item by NOTF and zonal management committee, and o approval of expenditures by the zonal health desk or management committee. Accounting details like regular calculations of residual amounts and funds released annually were not examined. TRANSPORT AND OTHER MATERIAL RESOURCES (Moderately, 2.7): Transport facilities at the zonal level were functional and adequate. However considering the terrain, the car and motorcycles will need to be replaced. The available car and motorcycles were maintained on a regular basis. The zonal health desk head authorized the deployment of transport facilities. Maintenance costs for the existing vehicles were covered from pooled funds. It was reported that government could undertake maintenance and repairs from pooled funds. All the available transportation facilities were pooled and deployed for health service, including CDTI. Trips were properly authorized. There was an integrated log book for all vehicles. Trips were made to supervise woredas and monitor activities being implemented at that level. Although the zonal management was aware of the necessity to replace the existing vehicles, there was no realistic plan to do so due to scarcity of resources from the government. 18 The management team at this level were hoping to get more motorcycles from donors and, with the integration approach, these would be available for CDTI. HUMAN RESOURCES (Moderately, 2): At the zonal level, staff turnover was generally high, and the main reason was reported to be restructuring. The personnel at this level expressed great interest and willingness to continue serving in their present positions. They valued the work they have been doing. However, there was no evidence of staff having received any sort of reward except salary. COVERAGE (Fully, 4): In 2003 treatment round, the two zones had 100% geographical coverage. This coverage has been constantly increasing. In this treatment round, the overall therapeutic coverage for the two zones was well over 65%. 19 Recommendations for the Zonal Level Recommendations Implementation Managing Mectizan ® : A less time consuming mode of collecting Mectizan® from regional level need to be put in place. Priority: MEDIUM Indicators of Success: Mectizan® is now collected from regional level in less time Who takes action: Regional Coordinator Deadline for completion: By next distribution and subsequently Leadership and Ownership: Zonal personnel ought to empower the woreda staff to carry out CDTI activities more efficiently. Priority: HIGH Indicators of Success: Empowerment of woredas staff to carry out routine CDTI activities efficiently. Who takes action: Zonal Coordinator Deadline for completion: January 2004 Finance: 1. There should be a clearly spelt out budgetary provision from the government for CDTI activities. 2. The financial contribution of the government to CDTI should show an increasing trend over time. Priority: HIGH Indicators of Success: 1. Clear budgetary provision for CDTI programme 2. Financial contribution of government to CDTI is showing an increasing trend. Who takes action: Regional Health Office Deadline for completion: 1. July 2004 2. January 2004 – December 2005 Transport: The zone ought to be aware of the need for replacing transportation and other material resources. Priority: MEDIUM Indicators of Success: Plan in place for replacement of transportation and or to strengthen vehicle pool Who to take action: Regional & Zonal health offices Deadline for completion: October 2003 20 3.2 Woreda Level 2 2 3 4 2.5 1 2 2 4 0 0.5 1 1.5 2 2.5 3 3.5 4 Score/4 Pl an ni ng Le ad er sh ip M on ito rin g /S up er vi si on M ec tiz an o rd er a nd s up pl y Tr ai ni ng a nd H S AM Fi na nc ia l R es ou rc es Tr an sp or t a nd o th er m at er ia ls H um an re so ur ce s C ov er ag e Groups of indicators Scores at Woreda Level PLANNING (Moderately, 2): In two of the three woredas visited, there was a three-year strategic plan that included CDTI. These plans contained all the key CDTI activities of the coming year. There was also a provision of 28,000 birr for CDTI activities. LEADERSHIP (Moderately, 2): In all the woredas visited, CDTI was implemented as part of their routine activities. Supervision and monitoring were initiated at this level with support from political and administrative leaders. However, training, Mectizan® ordering and distribution as well as management of funds were initiated at higher levels. This was because the woreda management team has not taken full responsibility for CDTI due to the existing health system management structure in the country which gives parts of the responsibilities for health service delivery to the zonal and regional councils. 21 SUPERVISION/MONITORING (Highly, 3) CDTI reports were submitted to the zonal level within the government system as part of Health Management Information System (HMIS). At times reports were submitted to the zonal office using APOC funds, separately from other health reports. These reports incorporated all the key CDTI activities. Supervision was carried out just twice a month during Mectizan® distribution, due to shortage of health staff. In some cases, supervision was extended right down to the communities. A supervision checklist was being used in one of the woredas. Supervision visits were usually integrated except during epidemic outbreaks. Supervision visits were reported to be problem-oriented in some other cases. Problems identified during supervision were addressed in conjunction with personnel from FLHFs and other relevant authorities. The major problems include CDD attrition, refusals of education and agriculture authorities to allow their staff involvement in CDTI activities. Well performing personnel were encouraged with extra provision of materials, such as stationery. They were also given verbal commendations and priority for training whenever such opportunities arose. MECTIZAN® SUPPLY (Full, 4): Supply of Mectizan® at the woreda level was sufficient and ordered in good time. The team was impressed with the level of documentation in the ordering, storage and distribution of Mectizan®. In one of the woredas visited, there was a report of late supply in the first year. However, this situation has been improving over time. Mectizan® was collected, stored and distributed within the government system. Although Mectizan® was collected separately from other drugs, it was kept in the same store. TRAINING/HSAM (Moderate, 2.5): In two of the three woredas visited, FLHFs and woreda personnel were trained by staff from the zonal level. In a few cases where woreda personnel were training personnel of the 22 FLHFs, training was targeted. In some cases, woreda staff trained CDDs routinely. In this case, woreda personnel were found out to be unaware of the need for empowering staff at FLHF level. New personnel were taken into consideration during training; these persons were routinely given full training. Generally, training was not integrated. HSAM was also routinely done yearly. In fact, HSAM activities have led to increased awareness / acceptance of CDTI within the communities and support from higher authorities. FINANCIAL RESOURCES (Slightly, 1): There was no budgeting for oncho control activities at the woreda level. Money was disbursed in block for each health programme, including CDTI. There was no cost breakdown per activity for the oncho programme at woreda level. Evidence for cost containment did not exist. Details of sources of funds were not known at this level. Funds received by the woreda personnel were insufficient for CDTI activities carried out at this level. Moreover, there was no evidence of increasing contribution from the government. Except for 2003-2005 strategic plan, no budgetary allocation was made for CDTI activities. TRANSPORT AND OTHER MATERIALS (Moderately, 2) The available transport facilities were not from government sources. They were inadequate for CDTI activities, and need to be replaced within the next 5 to 10 years. In order to tackle problem related to transport facilities, woreda personnel had already presented verbal request to the regional office. An encouraging aspect is that other government offices/sectors would render assistance whenever the woreda personnel requested for transportation service. On the other hand, there was no evidence of routine maintenance schedule for motorcycles at the woreda level. Funds provided for this purpose from the zonal office were reported to be inadequate. Besides, spare parts for these motorcycles were not available locally. 23 Staff’s coping mechanisms during motorcycles breakdown include using other motorcycles from the pool. CDTI activities have not been disrupted because of motorcycle breakdown. Even though the government has limited funds for repair and maintenance of vehicles at this level, it is generally capable of undertaking maintenance and repairs from pooled funds. The available transportation facilities were used in an integrated manner for all health programmes. No log books were being used. Trip authorisation forms were reported to have existed, but none could be seen. As a result of limited resources, no replacement plan for vehicles was available at this level. In order to ensure support for CDTI activities and increase effectiveness, more training and HSAM materials need to be reproduced and distributed. HUMAN RESOURCES (Moderately, 2) There has been a high turnover of personnel at the woreda level. Most of the woreda personnel have been there for less than 2 years. COVERAGE (Fully, 4) Apart from the first year when geographic coverage was 87.5%, it has always been 100%. Moreover, therapeutic coverage has been well above 65%. 24 Recommendations for the Woreda Level Recommendations Implementation Planning and Management: 1. The new initiatives with regard to developing strategic and operational plans need to be enhanced and materialized. 2. The woreda health personnel should take full responsibility for CDTI activities. Priority: HIGH Indicators of Success: 1. Woreda health personnel develop plans with clear budgetary provision. 2. Woreda personnel now become fully responsible for CDTI Who takes action: Woreda health desk, Zonal health desk Deadline for completion: October 2003 Supervision/Monitoring: 1. Supervision activities should be integrated and targeted as much as possible 2. Supervision checklists should be used in an integrated manner. Priority: HIGH Indicators of Success: 1. Supervision is integrated and targeted based on justification and need assessment. 2. Presence and use of supervision checklist. Who takes action: Woreda health desk (WHD) Deadline for completion: Immediately Training: Training should be targeted and integrated so as to increase efficiency. Priority: MEDIUM Indicators of Success: Training episodes are now targeted and integrated based on justifications and need assessment. Who takes action: Woreda health desk Deadline for completion: Immediately Finance: Priority: HIGH 25 Recommendations Implementation 1. Details of budgeting and disbursement of funds should be documented. 2. Woreda personnel should be encouraged to participate in budgeting and other financial matters. Indicators of Success: 1. Document is available on budgeting and disbursement of funds. 2. Woreda personnel are now involved in budgeting and financial matters concerning CDTI. Who to take action: Woreda and zonal health desks Deadline for completion: Immediately Human Resources: There should be some mechanisms for reducing the high staff turnover and improve staffing at this level. Priority: HIGH Indicators of Success: Turnover of staff is reducing. Who to take action: Regional health bureau Deadline for completion: January 2004 26 3.3 Front Line Health Facility Level 1 1 3 3.5 1.5 1 1.5 3 4 0 0.5 1 1.5 2 2.5 3 3.5 4 Score/4 Pl an ni ng Le ad er sh ip M on ito rin g /S up er vi sio n M ec tiz an o rd er a nd s up pl y Tr ai ni ng a nd H SA M Fi na nc ia l R es ou rc es Tr an sp or t a nd o th er m at er ia ls H um an R es ou rc es C ov er ag e Groups of indicators Scores at FLHF Level PLANNING (Slightly, 1): In half of the FLHFs, timetables for CDTI activities were seen. In all cases, CDTI activities were carried out according to directives from the woreda level. CDTI forms part of the “minimum” package of health care at this level. Nevertheless, the health personnel of the FLHFs have not been empowered in terms of planning. LEADERSHIP (Slightly, 1): In most cases, FLHF staff did not initiate CDTI activities. CDTI activities were initiated at the woreda level. In some cases, the staff at FLHF discussed with woreda personnel to decide on timing of CDTI activities. In some other cases, the opinions of the local administrative and political leaders were sought for deciding on the timing of CDTI activities. 27 Generally, instructions and directives tended to come from above and activities were carried out as instructed. MONITORING AND SUPERVISION (Highly, 3): In all cases, CDTI data from FLHFs were taken to the woreda health office by the health personnel in charge of the FLHF. This was carried out either by using pooled transport facilities, such as motorcycles and horses, or by using public transport. In some cases, however, they had to trek. Reports of other health programmes were taken to the woreda office separately from CDTI reports but using the same means of transportation indicated. Generally, CDTI supervision visits were made routinely, and they were not problem- oriented. In many cases, these supervision visits were not integrated with those other health programmes. In a few cases, health personnel claimed to have carried out integrated supervision visits. However, no written documentation regarding these was available. The existing system did not enhance integrated and problem-oriented supervision visits. Moreover, health personnel at this level did not see the importance of carrying out integrated and problem-oriented supervision visits. Whenever there were problems, health personnel at FLHFs would solve them on the spot together with kebele / village leaders. With regard to incentives, some of the personnel at FLHFs reported to have made recommendations for good performing CDDs to be rewarded at woreda level. MECTIZAN® PROCUREMENT AND DISTRIBUTION (Highly, 3.5): Mectizan® was requested by the FLHF staff but the actual quantity of tablets needed was calculated by woreda staff based on the current population figures provided by the former. Mectizan® was available in time for distribution in line with the time set by personnel at higher levels. In a few cases, the communities preferred Mectizan® to be distributed in other periods. No shortages were reported in the last round of treatment. However in the first year, brief Mectizan® stock-outs were reported in some kebeles due to wastage. Additional tablets were later sent and the problem was resolved. 28 In all cases, FLHF heads collected Mectizan® from the woreda health office. This was done either by using pooled motorcycles of the FLHF, or public transport. Funds for fuelling these motorcycles or paying for public transport were provided by the woreda office but the health personnel did not know the actual source of these funds. Other drugs required by the FLHF were brought separately from the woreda office using the same means of transportation indicated. Occasionally, these drugs were transported from the woreda office to the FLHF free of charge by some benevolent individuals. At the FLHF level, Mectizan® was stored together with other drugs. Documentation on the ordering, reception and disbursement of Mectizan® at this level was very good. This has highly impressed the evaluation team. TRAINING AND HSAM (slightly, 1.5): FLHF staff in collaboration with woreda health personnel carried out training of CDDs. The training episodes which usually lasted three days each were routine and not targeted to particular problems or deficiencies. They were not integrated either. Health personnel at this level did not see the need of carrying out problem targeted and integrated training episodes. Separate training for CDTI or other health programme was thought to be less burdensome, shorter and easier to assimilate by trainees. The available training and HSAM materials could easily be reproduced and used for CDTI activities in the years to come. In few FLHFs, stationery materials were purchased with government funds. In most cases, these materials were supplied by other partners. FINANCIAL RESOURCES (Slightly, 1): Budgeting was not carried out at the FLHFs. Health personnel at this level did not have any idea about budget breakdown and sources of funding. They had not been exposed to such financial matters. They reported that financial issues were dealt within higher quarters. Funds at the FLHFs were insufficient for CDTI activities. Moreover, health personnel at this level did not know the amount provided by the different partners. As a result of limited financial resources, no cash contribution, other than salaries, has come from the government in support of key CDTI activities. 29 TRANSPORT AND OTHER MATERIALS (Slightly, 1.5): The available motorcycles were old and always breaking down. They were not adequate for CDTI activities. They cannot be used even in the coming few years. In some FLHFs, there was routine maintenance of vehicles from pooled resources but government funds allocated for this purpose was usually inadequate and not released in time. Staff’s coping mechanisms in case of motorcycles breakdown include trekking and using public transport. In spite of the aforementioned problems, CDTI activities have not been disrupted. It was also noted that government could pay for maintenance, repairs and tire replacement from the limited pooled funds. FLHF staff did not manage the available transport facilities properly since neither travel authorizations nor log books were utilized. FLHF personnel were not aware of the fact that travel authorizations and log books are necessary and good practices. The health personnel at this level were aware of the need for replacement of transport facilities. However, there was no evidence of a realistic plan for replacement. HUMAN RESOURCES (Highly, 3): Personnel at the FLHFs personnel were given annual routine in-service training. In the FLHFs visited, most have served for less than 5 years. There was a general problem of understaffing. This problem was aggravated by the ongoing decentralization of health service structure. COVERAGE (Fully, 4): Geographical coverage was 100% in all of the 6 FLHFs visited. Therapeutic coverage rates were well over 65%. 30 Recommendations at the Front Line Health Facility Level Recommendation Implementation Planning: 1. Proper planning need to be carried out at the FLHFs. 2. CDTI activities should be initiated by FLHFs taking into consideration the specific characteristics of their communities. Priority: HIGH Indicators of success: 1. FLHF staff now have their own CDTI plans 2. FLHF staff now initiate CDTI activities based on specific needs of their communities Who to take action: FLHFs & WHDs Deadline for completion: immediately Monitoring and Supervision: Supervision of CDTI activities should integrated and problem-oriented Priority: HIGH Indicators of success: Supervision is integrated and problem oriented Who to take action: FLHFs & WHDs Deadline for completion: immediately Human Resources: Mechanisms to ensure stability of staff at FLHFs should be put in place. Priority: MEDIUM Indicators of success: Staff at FLHFs are now stable Who to take action: WHDs Deadline for completion: October 2003 Finance: FLHF staff should be aware of budgetary provisions and sources of funds for CDTI Priority: HIGH Indicators of success: FLHFs staff are more aware of budget and sources of funds for CDTI activities 31 Recommendation Implementation activities Who to take action: WHDs Deadline for completion: October 2003 Transport and other materials Transport facilities at FLHFs level should be well managed with provision for log books Priority: MEDIUM Indicators of success: Availability of log books to monitor transport usage Who to take action: FLHF staff & WHDs Deadline for completion: October 32 3.4 Community Level 3 3.6 3 3.5 4 2 3.3 4 0 0.5 1 1.5 2 2.5 3 3.5 4 Score/4 Pl an ni ng a nd m an ag em en t Le ad er sh ip a nd o w ne rs hi p M on ito rin g /S up er vi si on M ec tiz an o rd er a nd s up pl y H SA M Fi na nc ia l R es ou rc es H um an re so ur ce s C ov er ag e Groups of indicators Scores at Community Level PLANNING AND MANAGEMENT (Highly, 3): In all the kebeles visited, census updates and Mectizan® distribution were carried out at different periods. This was because CDDs were instructed by health personnel from the FLHFs to do census updates differently from Mectizan® distribution. CDDs also believed it was necessary to do census updates separately and use the current population figures to request for appropriate quantity of Mectizan®. In many of the communities, kebele administrators participated in carrying out sensitization and mobilization. In most communities visited, the community members decided on the visiting time. In some others, the CDDs decided. In all cases, the timing was accepted by the entire community. 33 LEADERSHIP AND OWNERSHIP (Highly, 3.6): The community leaders took the responsibility of sensitizing and mobilizing their people, choosing CDDs and in some cases deciding on the mode of Mectizan® distribution. The active community leadership and sense of ownership would contribute a lot to having good coverage rates. Moreover, the community leaders actively participated in managing problems related to Mectizan® distribution. The timing for Mectizan® distribution was mostly decided at higher levels than the community; in some communities visited, community members would like distribution to be in December or January instead of the usual April. All the communities visited passionately expressed their wish to take Mectizan® for as long as it is available. They did not want Mectizan® distribution to be discontinued at all. The communities were aware of the need for long-term treatment and most of the interviewed people mentioned 12 years. All the community members interviewed knew at least one advantage of taking Mectizan®. However, there were some misconceptions about the drug, such as treating sexually transmitted infections, high blood pressure and paralysis. MONITORING/SUPERVISION (Highly, 3): During CDTI activities, CDDs reported weekly to the FLHF. In few communities, CDDs were provided with horses; in others, they had to trek. Some CDDs had to trek for up to 2 hours to reach FLHF. In most communities, CDDs were not provided with transport facilities. They submitted detailed reports to the FLHFs. Personnel from the FLHFs and woreda office too often carried out monitoring and supervision activities at the community level. This indicates that CDDs lacked some sort of empowerment. 34 MANAGING MECTIZAN® (Highly 3.5): Mectizan® was distributed to all eligible members of the community based on census updates made in the same year. After distribution, Mectizan® was left within the communities for 1 month to take care of absentees and the temporarily non-eligible persons. All CDDs collected their Mectizan® stock from the FLHFs. In one of the woredas visited, a few communities got their stock from the FLHF on a weekly basis, while in other cases, they collected at once. HSAM (Fully, 4): CDDs and the kebele administrators/traditional leaders had been actively involved in HSAM activities. In some communities, health committees had been formed and these also took part in carrying out HSAM on CDTI. FINANCIAL RESOURCES (Moderate, 2): Regarding financial resources, situations varied from place to place. In some woredas, CDDs were supported by their communities; in others (about 50%), they were not. In areas where they were supported, the CDDs got compensation with free labour, or they were exempted from work. In some communities, rumors were spread about CDDs being paid by external donor agencies. There were also rumors about money being allocated even for the community members who take Mectizan® and the government withholding the money. Stationery materials for CDTI all came from the woreda health office. Of course some communities expressed their willingness to provide such materials if need be. Nonetheless, it was revealed that the communities had not been adequately mobilized with regard to sharing costs. On the other hand, it was indicated that discussions on how to mobilize supports for the CDDs were going on in some communities. 35 HUMAN RESOURCES (Highly, 3.3): In all villages, the recommended ratio of 2 CDDs for 250 persons was not attained. In some cases, 2 CDDs were assigned to serve between 900 and 2400 persons. The routine was 2 CDDs per village irrespective of the size of the population and type of settlement. In most areas, CDDs walked for about 30 to 40 minutes from their homes. In other cases, they had to walk for 1 to 2 hours. All CDDs had been trained, and they were skilled in carrying out CDTI activities. Side- effects were reported to the FLHF personnel as soon as they appeared. In case of CDDs failing to fulfil their duties correctly, they were replaced with new ones who would receive training subsequently. Drop-outs were also immediately replaced in like manner. All the CDDs interviewed expressed their willingness and dedication to continue with CDTI activities. Some of them even remarked that it was a moral obligation to serve the people who elected them. Very few CDDs had dropped out within the 3 years of the project life. COVERAGE (Fully 4) In the second and third rounds, all the eligible households were treated in all the communities. The geographical coverage rate was 100%. Therapeutic coverage rates were well above 65%. In the first year, however, a few households were not treated because of refusals. In one of the kebeles visited, there was shortage of Mectizan® which led to some persons being left without treatment. In general, high coverage rate has been one of the strongest aspects of the Kaffa-Sheka CDTI programme. 36 Recommendations at the Community Level Recommendation Implementation Planning: Census updates and Mectizan distribution should be combined for efficiency purpose. Priority: HIGH Indicators of success: Census update and treatment are now carried out simultaneously Who to take action: CDDs, FLHFs & Community leaders Deadline for completion: During distribution Monitoring: The frequency of CDDs reporting should be reduced as much as possible. Priority: HIGH Indicators of success: There is reduction in CDDs reporting. Who to take action: FLHFs & WHDs and Zonal health desk Deadline for completion: immediately Managing Mectizan® All CDDs should collect their Mectizan® stock at once. Priority: MEDIUM Indicators of success: CDDs now collect enough Mectizan® for distribution at once. Who to take action: FLHFs and WHDs Deadline for completion: January 2004 Human Resources: The ratio of 2 CDDs to 250 persons per community should be considered to reduce fatigue on CDDs. Priority: HIGH Indicators of success: More CCDs are selected for CDTI activities Who to take action: Community members, FLHF & WHDs Deadline for completion: October 2003 37 4. CONCLUSIONS 4.1 Grading the Overall Sustainability of the Kaffa-Sheka CDTI Project A judgment of the sustainability of the Kaffa-Sheka CDTI was made according to the following “aspects of sustainability” and “critical elements”. Aspects of Sustainability: Aspect Judgment: to what extent is this aspect helping or blocking sustainability of this project? Integration Very much helping Resources Potentially blocking Efficiency Not blocking Simplicity Helping Attitude of Staff Very much helping Community Ownership Very much helping Effectiveness Very much helping Integration: Very much helping sustainability CDTI activities have been highly integrated with other health programmes in Ethiopia. This aspect has great contribution to the sustainability of Kaffa-Sheka CDTI Project. Resources: Potentially blocking sustainability This aspect has potentially negative impact on the sustainability of the CDTI project. There had been no direct allocation of funds by government. There was no plan for the government to fund the bulk of CDTI expenses by the end of Year5 of APOC funding. Unless the government or any other partner takes prompt actions, it will negatively affect the sustainability of the project. Efficiency: Not blocking sustainability Regarding efficiency, strengths have been observed in such areas as Mectizan® distribution, using pooled vehicles, and good data management. However, there was lack of budgetary allocation and cost breakdown for CDTI activities. The latter requires some action for improvement. 38 Simplicity: Helping sustainability CDTI activities have been fairly simplified. Reporting and data collection systems need to be simplified more. The way Mectizan® is distributed to zonal level should be simplified. Attitude of Staff: Very much helping sustainability Personnel at the different levels of the project have shown great enthusiasm and interest to carry out CDTI activities, and this is one of the driving forces towards the sustainability of the CDTI project. Community Ownership: Very much helping sustainability The evaluation team found out that community members were playing active role in CDTI programme. They participated in selecting CDDs, changing CDDs when necessary, and some times they provided compensation to CDDs. Moreover, the CDDs were highly committed. The communities included in the Kaffa-Sheka CDTI Project are moving towards ownership. However, the community members still need to mobilize more supports for the CDDs. Effectiveness: Very much helping sustainability The project has been very effective. It is achieving its overall goal to cover 100% of meso- and hyper-endemic communities with a therapeutic coverage rate well over 65%. Critical Elements of Sustainability: Critical Element Yes/No Money: Is there sufficient money available to undertake strictly necessary tasks which have been carefully thought through and planned? (absolute minimum residual activities). No Transport: Has provision been made for the replacement and repair of vehicles? Is there a reasonable assurance that vehicles will continue to be available for minimum essential activities? (note that ‘vehicle’ does not necessarily imply ‘4x4’ or even ‘car’). Yes 39 Critical Element Yes/No Supervision: Has provision been made for continued targeted supportive supervision? (the project will not be sustained without it). Yes Mectizan® supply: Is the supply system dependable? (the bottom line is that enough drugs must arrive in villages at the time selected by the villagers). Yes Political commitment: Effectively demonstrated by awareness of the CDTI process among policy makers (resulting in tangible support); and a sense of community ownership of the programme. Yes Money: No There have been critical problems with regard to release of budgets and flow of funds that has a blocking impact unless some measures are taken soon. Transport: Yes Although transportation facilities have not been sufficiently availed for CDTI activities at some levels, the traditions of using pooled transport and providing transportation assistance across different sectors of the government help the sustainability of the project. Supervision: Yes This element is potentially helping the sustainability of the project. Supervision visits were integrated and targeted in some cases. The evaluation team feel that some improvement is required in this respect. Supervisors at all levels need to make such visits both targeted and integrated as in many cases as possible. Mectizan® Supply: Yes The supply system for Mectizan® distribution to the communities has been simple and reliable. In most cases, Mectizan® was available in villages in good time for planned distribution. 40 Political Commitment: Yes The government has accepted CDTI as one of the health programmes. Administrative and political leaders at all levels have shown their supports for the project. Hence, this element helps the sustainability of the Kaffa-Sheka CDTI Project. 4.2 Grading of project as a whole On the overall grading of the project, the team have found that six of the seven “aspects of sustainability” were helping or not blocking the project moving towards sustainability; only resources at the woreda and FLHF levels was seen as potentially blocking. In relation to “critical elements”, it was found that “Money” was not sufficient at all levels. The Evaluation Guidelines indicate that where “one or two aspects are not fulfilled and one or two critical elements are not satisfied, the project is making satisfactory progress towards sustainability”. Therefore, the Independent Evaluation Team concludes that the Kaffa- Sheka CDTI Project is making satisfactory progress towards sustainability. 41 5. MAJOR FINDINGS, DISCUSSIONS AND RECOMMENDATIONS AT NATIONAL AND REGIONAL LEVELS 42 2.5 1.7 4.0 2.7 3.0 2.3 2.5 2.5 4.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Score/4 Pl an ni ng M on ito rin g /S up er vi si on M ec tiz an o rd er a nd s up pl y Tr ai ni ng a nd H S AM In te gr at io n of s up po rt ac tiv iti es Fi na nc ia l R es ou rc es Tr an sp or t a nd o th er r es ou rc es H um an re so ur ce s C ov er ag e Groups of indicators Scores at National Level PLANNING (Moderately, 2.5) Onchocerciasis control is integrated in the national and Southern Nations Nationalities and Peoples Region (SNNPR) strategic plans (2003 -2005) as a component of malaria and vector borne disease control. CDTI plans of action were detailed with specific objectives and key activities for each of the three years 2001- 2003, both at both levels. There is close collaboration between all partners (MoH, WHO and The Carter Center). The region is responsible for overseeing CDTI implementation. The NOTF provides a forum for reviewing annual plans and sharing of tasks. The National Coordinator spends about 25% of his time on CDTI. As a result of under staffing at the MoH, the NGO partner serves as secretary of NOTF. WHO provides technical support, clears and stores Mectizan® and manages APOC funds in partnership with the NOTF. Funds for implementation of activities are transferred by WHO to the region for disbursement to the zones and accounting. There was no specific sustainability plan (neither at the national nor regional levels) for the period after APOC funding is withdrawn. 43 MONITORING AND SUPERVISION (Slightly, 1.7) During the past 2 years staff at the National and Regional levels having been supervising all the lower levels. This has led to increasing skills at the zonal and woreda levels. It was therefore expected that in the 3 year supervision should have been mainly targeting the zonal level, with spot check at lower levels if necessary. However, finding show that there was a gap between this policy and practice. Problems identified in the field were sometimes dealt with in a timely manner. However, a serious problem affecting the project is delays in accountability of funds. This problem is still unresolved. For instance, in the first year, the project lost about 36% of the funds approved by APOC while in the third year, 50% of project-approved funds were lost due to delays in accounting for funds already used. Staff at this level supervised only the next level and also carried out spot checks at other levels. The staff empowered the zone to supervise activities at their own level and at the level below them. The regional level, made three visits per year, two of which were integrated. Supervision checklist was said to be used at this level. At the national level at least one supervisory visit per year was made during the treatment period. Supervisory visits for CDTI and other activities for malaria and vector borne diseases were integrated. MoH and the NGO partner sometimes carried out supervision jointly. Supervisory checklist was not used. Annual review meetings were used to identify and solve problems effectively. Examples of such instances are: • Non-release of APOC vehicle for CDTI activities at zone – Corrective action was taken to ensure release of vehicle as required. • Refusal of treatment by some members of the community who considered CDDs to be uneducated –Health education was intensified. • Non-appreciation of CDDs –Approached NOTF in a meeting to recognize good performance. The NOTF agreed to provide awards (certificate and radios). There was no process of recognizing successes for health workers. 44 MECTIZAN® PROCUREMENT AND DISTRIBUTION (Fully, 4.0) The NOTF ordered Mectizan® tablets based on the need generated from the CDDs within the government system. Clearance and storage is by WHO. The control of supply is within the government system along side the supply of other drugs. The region collected Mectizan® from WHO and notified the zones for collection. The zones notified the woredas to effect collection, after which they delivered to FLHFs. TRAINING AND HSAM (Moderately, 2.7) Staff at the national level (National Coordinator, NGO and WHO) only trained staff at the region and zone. As a result of high staff turn over, it had been necessary to provide additional training. Integration of CDTI training with other training was not carried out at either the national or regional levels. The NOTF assessed situations in the implementation of CDTI and carried out corrective action including HSAM. The HSAM materials developed were based on and in response to problems. Evidence of the effect of HSAM was the cascading effect of their work, which had led to increased geographic and therapeutic coverages. INTEGRATION AND SUPPORT ACTIVITIES (Highly, 3.0) MoH staff members were responsible for several interrelated projects (CDTI, malaria). Information was obtained from relevant sources during monitoring and supervision. The evaluation team was impressed by the extent of integration at both the national and regional levels e.g. transport and logistic support was pooled and no staff member was responsible for only a single project. Budget for different activities (maintenance, monitoring and supervision) was also pooled. FINANCIAL RESOURCES (Moderately, 2.3) There was no deficit between estimated costs and the amount provided by partners. 45 The cost of each activity in the year plan was clearly spelt out in the budget and the Project manager was aware of funds available for the subsequent year. There was no evidence of a cost containment strategy. The budgetary provisions by government and NGO partner were clearly spelt out in the annual agreements. Government contribution was only for four budget lines (salaries, capital, communication and operating costs). There was no annual increase and evidence of how government would fund core CDTI activities at the end of Year 5 of APOC support. The actual disbursements from government were payment of staff salaries. Documentation for the disbursements for the other three budget lines was not available. The team was informed that the disbursements were from pooled funds (government and donor funds). In the event of a shortfall, pooled funds through integration of health activities, and support from APOC and the NGDO partner will be used. The NGDO partner was unable to make a commitment beyond 2005. It therefore remains unclear as to how the project will be sustained. The project plans to use to enhanced efficiency through integration of the core CDTI activities with other health activities. Government does not allow ministries to have separate accounts; as a result APOC funds are in WHO account. The NOTF manages the funds from the WHO account and releases funds following written requests by MoH/NOTF. WHO allocates expenditure against budget lines and provides regular insight on balances to the NOTF. The Project staff expressed concern with loss of project funds as a result of delays in the accounting and reimbursement from APOC. This was discussed during the feedback and planning meetings. TRANSPORT AND OTHER MATERIAL RESOURCES (Moderately, 2.5) There was routine maintenance as government regulations require that vehicles, be serviced regularly (after 10,000kms). Staff could cope with vehicle breakdown through use of other vehicles from the pool. Costs of maintenance could also be met by funds from the pool. 46 The use of transport was said to be properly controlled as written authorization and approval was required for trips. Neither the vehicle nor the authorizations or logbook were seen as the former were said to be in maintenance. Management was aware of the need to replace transport and other materials. The plan for replacement is the use of pooled vehicles from other projects (Malaria, TB & Leprosy). HUMAN RESOURCES (Moderately, 2.5) The use of spare time, overtime without payment for project activities by staff members and cancellation of vacation by a staff member (because of the evaluation) is evidence of commitment. The team observed that the staff members had a heavy work load as a result of staff shortage. There is a high turn over of staff mainly because of on going restructuring in the civil service. Two key members of staff interviewed had only been in office for two years. New staff assigned to CDTI was trained. COVERAGES (Fully, 4.0) The geographic coverage was 100% while the therapeutic coverage was well above 65% in the two zones. Both coverages were stable over the period of evaluation. 47 Recommendations for National and Regional levels Recommendations Implementation Planning: (1) MoH has the leadership role among partners for CDTI. To strengthen this role, MoH should appoint a secretary for the NOTF. He/She should be skilled and able to devote adequate time to the assignment. (2) On-going restructuring gives key role to woreda level in CDTI. Need to focus 4th year plan (sustainability plan) on woreda as the implementation unit. Need to develop and implement plan for ‘installed capacity’ at this level. Need to define clearly role of other levels Priority: HIGH Indicators of Success: (1) NOTF now has a skilled and available secretary as required for the position. (2) Focus on sustainability plan is now on woreda level Who takes action: National Coordinator Deadline for completion: Immediately Monitoring and Supervision: Integration is cornerstone of CDTI (and health services) in Ethiopia, with considerable success in a number of areas. Difficult issues with regard to integration of supervision, monitoring and training (particularly the later). Recommend operational research (in one woreda) to find ways of solving relevant issues (including integration of learning materials, supervision checklist and project planning). Priority: HIGH Indicators of Success: Operational research issues identified and conducted Who takes action: National Coordinator Deadline for completion: December 2004 48 Finance: (1) Realistic amounts should be budgeted for planned CDTI activities (sustainable plan) (2) Provide guidance (from central/regional level) on cost reduction/containment etc. (3) Guidance/memo on ensuring disbursed funds are efficiently managed, using skilled personnel (4) Ministry/APOC/WHO to appoint someone to sit in WHO to follow up on financial issues related to CDTI. Priority: HIGH Indicators of Success: 1. Realistic amount is budgeted for planned activities 2. Availability of specific guidance on cost reduction. 3. Skilled personnel employed to managed fund disbursement. 4. Recruitment of APOC financial assistance at WHO office Who takes action: National, APOC Deadline for completion: 1. October 2003 2. October 2003 3. January 2004 4. January 2004 Human Resources: 1. Need for commitment and stability to staffs. 2. Institute a reward and recognition system for good performance for individual and institutions; CDDs/Villages, Health workers (FLHF, woreda) Priority: MEDIUM Indicators of Success: 1. Commitment and stability of key staff for CDTI. 2.Reward for good performance implemented. Who to take action: National, Regional & Zonal health offices Deadline for completion: October 2003 49 ANNEXES 50 Annex 1 : INTERVIEWS National Level 1. Dr Daddi Jima National Oncho Coordinator 2. Gezahegen Tesfaye Malaria and other vector-borne DPC and Co-chair of NOTF 3. Teshome Gebre Country Representative, Carter Center Regional level 1. Esey Batisso Oncho Coordinator, SNNP Regional State Zonal Level 1. Mr. Modo Atto Executive President of Sheka Zone 2. Mr. Gemeda Bongolom Vice Executive President of Sheka Zone 3. Mr. Sharifo Kessito Head of Zonal Capacity Building Department 4. Mr. Habtamu Beyene Head of zonal Health Department 5. Mr. Ousman Ibrahim Health Service, delivery and human resources team leader 6. Fashaw Dubale Zonal Health Desk Representative, Gimbo Zone 7. Bekele Kindare Zonal Oncho Coordinator, Gimbo Zone 8. Bedruzeman Abdello Zonal Chief Administrator, Kaffa Zone 9. Mesfin Mengesha Head, Zonal Dept of Finance and Economic Development, Gimbo 10. Akilu Woldericaal Head Capacity Building, Gimbo Zone Woreda Level 1. Mr. Mitiku Haile Head of Masha Woreda Health Office 2. Mr. Abiot Asres Head of Masha Health Center 3. Mr. Adamu Ayenew Team leader of DPC, Masha Woreda 4. Shiferaw Hailemariam Representative, Gimbo Woreda Health Desk 5. Tamene Mamo Oncho Coordinator, Woreda (Gimbo) Front Line Health Facility Level 1. Mr. Abiot Asres Head of Masha Health Center 2. Mr. Eshetu Gebrehiwot Head of Yina Health Center 3. Fanaye Tesmma Oncho Coordinator, Argoba Health Station 4. Tekle Serba Head, Argoba Health Station 5. Bekele Daka Head, Wushwush Tea Plantation Clinic Community Level 1. Mr. Behanu Mamo Leader and CDD of Yina Kebele 2. Mr. Dago Desalegu CDD of Yina Kebele 3. Mr. Abate Wullo CDD of Abello Kebele 4. Mr. Behanu Dino CDD of Abello Kebele 5. Mr. Angulo Alalo Leader of Abello Kebele 6. Mr. Endeshaw Mamo Leader of Gembaka Kebele 7. Mr. Abera Achomo Vice Leader of Abello Kebele 8. Mr. Gudeta Gallo Vice Leader of Gembaka Kebele 9. Mr. Dakito Dino CDD of Gembaka Kebele 10. Mr. Shegito Vice Leader of Gada Kebele 11. Mr. Kassaye Shegito CDD of Gada Kebele 51 12. Bezabih Demeke Chairman, Argoba Kebele 13. Abate Alemu CDD/Chairman Korke Village 14. Molla Goddebo CDD, Shuka Settlement, Wushwush Tea Plantation 52 Annex 2: Schedule for the Evaluation, advocacy Day/Date Team Members Activity Monday 6th October 2003 All team members Visits to WR/Addis-Ababa, interviews of CR GB2000 and NOTF authorities Tuesday (7th, October 2003) All team members Arrival at Jimma Town, Orientation and Sub-teams Formation (at Jimma University) Wednesday (8th, October 2003) All team members Travel to the Project Site Thursday to Monday (9th to 13th, October 2003) All team members Advocacy visits, data collection: interviews and document studies at different levels Tuesday (14th, October 2003) All team members Travel back to Jimma Town Wednesday to Friday (15th, to 17th, October 2003) All team members Conduct Sustainability Planning Workshop Tentative timetable for sub-team A (Dr Tarimo, Dr Elhassan, Mr. Kora Tsuhune) Day/Date Sub-team members Activity Wednesday (8th, October 2003) Sub-team A Travel to the Project Site, make advocacy visit to decision makers at Kaffa zone Thursday (9th, October 2003) Sub-team A Data collection: interviews and document study (Kaffa Zone) Friday to Saturday (10th, to 11th , October 2003) Sub-team A Data collection: interviews and document study (Gimbo District as well as Wushwush Tea Plantation and Argoba FLHFs) Sunday to Monday (12th to 13th, October 2003) Sub-team A Data collection: interviews and document study (Bubba, Shukka, Chobba and Argoba communities) Tuesday (14th, October 2003) Sub-team A Travel back to Jimma Town 53 Timetable for sub-team B ( Mr. Ekpo, Mr. Evini, Dr Assefa Worku) Day/Date Sub-team members Activity Wednesday (8th, October 2003) Sub-team B Travel to the Project Site (Sheka Zone) Thursday (9th, October 2003) Sub-team B Make advocacy visit to decision makers, and proceed with data collection: interviews and document study (Sheka Zone) Friday to Saturday (10th, to 11th , October 2003) Sub-team B Data collection: interviews and document study (Masha District as well as Yina and Masha FLHFs) Sunday to Monday (12th to 13th, October 2003) Sub-team B Data collection: interviews and document study (Abello, Gembeka, Yina and Gada communities) Tuesday (14th, October 2003) Sub-team B Travel back to Jimma Town Tentative Timetable for Sub-team C (Dr Atabe, Mr. Abraraw Tsefaye) Day/Date Sub-team members Activity Wednesday (8th, October 2003) Sub-team C Travel to the Project Site (Yeki District) Thursday (9th, October 2003) Sub-team C Make advocacy visit to decision makers Friday to Saturday (10th to 11th, October 2003) Data collection: interviews and document study (Yeki District as well as Zinki and Kubito FLHFs) Sunday to Monday (12th, to 13th , October 2003) Sub-team C Data collection: interviews and document study (Zinki, Tsanu, Ermichi and Kubito communities) Tuesday (14th, October 2003) Sub-team C Travel back to Jimma Town 54 Annex 3: FEEDBACK/PLANNING MEETING FOR NATIONAL/REGIONAL/ZONAL AND WOREDA LEVELS AT WOLDE ARGAW HOTEL, JIMMA, 15TH – 17TH OCTOBER 2003 AGENDA DAY 1 S/No ACTIVITY TIME FACILITATOR 1 Opening ceremony and welcome 9.00 – 9.10am D. Jima 2 Introducing the participants 9.15 – 9.20am A. Worku 3 Presentation: • The objective of the evaluation • What is sustainability • The Evaluation Methodology 9.20 – 10.00am E. Tarimo 4 Presentation of Main Findings & Discussions • Community (Kebele) level • FLHF level • Woreda level • Zonal level 10.00 – 11.00am A. Atabe A. Atabe A. Tesfaye C. Evini 5 Coffee break 11.00 – 11.30am 6 Group work (in 2 groups) 1. SWOT analysis – ‘What is the situation regarding sustainability in our project?’ Group 1: The community and FLHF levels Group 2: The Woreda and Zonal levels 11.30 – 1.00pm E. Elhassan 7 Lunch break 1.00 – 2.00pm 8 Presentation of group work followed by plenary discussion 2.00 – 3.00pm U. Ekpo 9 2. ‘What changes do we need to make in order to address the weaknesses and threats regarding sustainability of our project considering the strengths and opportunities identified in the SWOT analysis?’ Group 1: Planning, Monitoring and Supervision Group 2: Finances, Training and HSAM Group 3: Transport, Mectizan supply Instructions: focus on trimming expenses, mobilising, resources, integration 3.00 – 4.00pm C. Evini K. Tushune 10 Coffee break 4.00 – 4.30pm 11 Presentation of group work followed by plenary discussions 4.30 – 5.00pm U. Ekpo 12 Summary of day’s work 5.00 – 5.15pm ZC - Kaffa 13 House keeping matters 5.15 – 5.30pm A. Worku 55 DAY 2 (PLANNING) 16TH OCTOBER 2003 S/No ACTIVITY TIME FACILITATOR 1 Introduction of the chair 9.00 – 9.05am T. Gebre 2 Summary of previous day’s workshop proceedings 9.05 – 9.15am A. Tesfaye 3 Discussions: • What is a sustainable CDTI Plan? Features? Format? Characteristics? 9.15 – 11.00am E. Elhassan 4 Coffee break 11.00 – 11.30am 5 Group work: What minimum resources do we need to make our programme sustainable in the course of the next 2 years? Issues to consider: Money, Human resources, Transport and equipment/materials • Group 1: National / Region / Zone • Group 2: Woreda / FLHF/Community 11.30 – 1.00pm E. Elhassan C. Evini A. Atabe 6 Lunch break 1.00 – 2.00pm 7 Group work continued 2.00 – 3.00pm K. Tushune Evini 8 Coffee break 3.00 – 3.30am 9 Group work continued 3.30 – 5.15pm K. Tushune Evini 11 House keeping matters 5.15 – 5.30pm A. Worku 56 DAY 3 (PLANNING CONT’D) 17TH OCTOBER 2003 S/No ACTIVITY TIME FACILITATOR 1 Introduction of the chair 9.00 – 9.10am E.Batisso 2 Group work continued 9.10 – 10.00am K. Tsuhune 3 Group work (in 2 groups) Development of sustainability plans for 2004 -2005 • Each group to compile a tabulated plan (what is to be done, why; by whom; when; indicator; cost) • The plan must fit into available resources 10.00 – 11.00am E. Elhassan 4 Coffee break 11.00 – 11.30am 5 Presentation of group work, followed by plenary discussion 11.30 – 1.00am Chair 6 Lunch break 1.00 – 2.00pm 7 Plenary discussion: Making a master ‘sustainable plan’ for the project, for the coming year 2.00 – 3.00pm C. Evini 8 Plenary discussion: • Practical steps to implement the required in the ‘sustainable plan’ 3.00 – 4.00pm E. Elhassan 9 Coffee break 4.00 – 4.30pm 10 Review/Endorsement of Plans 4.30 – 5.00pm Chair 11 The Way Forward (What is going to happen to our plans?) 5.00 – 5.30pm NC 12 Closing remarks 5.30 pm CC, WHO 57 Annex 4: REPORT OF FEEDBACK/PLANNING WORKSHOP FOR KAFFA-SHEKA CDTI PROJECT A three-day feedback and planning workshop was held for National, Regional, Zonal and Woreda level personnel involved in CDTI programme. The aim of the workshop was to give feedbacks on the findings and recommendations of the APOC Evaluation Team. The workshop also served as a forum for the development of a two-year sustainability plan for each level. The workshop was held in Jimma Town, Ethiopia, from 15th to 16th, October 2003, and 40 participants from different organizations involved in implementing CDTI programme attended the workshop. The workshop started with opening remarks made by the National Coordinator and representatives from WHO and Carter Centre. This was followed by a self-introduction of participants. Then, Dr Tarimo, Team leader of APOC Evaluation Team made briefings on the objective of the evaluation and the concept of sustainability. A briefing was also made on evaluation methodology. Following this, presentations of major findings and discussions were made. At the end of these presentations, Kaffa-Sheka CDTI project was judged to be satisfactorily progressing towards sustainability. After the coffee break, Dr Elhassan, one of the evaluation team members, made a presentation on SWOT analysis. The other activities of the day were group formation and working in groups on the problems identified by the APOC Evaluation Team relating to SWOT analysis. Below are the results of the SWOT analysis and addressing solutions to weakness and threats identified in Kaffa-Sheka CDTI project. RESULT OF SWOT ANALYSIS ZONAL LEVEL STRENGTH Good Planning/Strategic Plan The management team has the capacity to initiate key CDTI activities There is focal person for the programme Both financial and technical reports are integrated in HMIS Reports contain all relevant data Mectizan order are based on needs and requests from woredas High geographical and therapeutic coverage WEAKNESS No direct allocation of budget by government High staff turnover Training is routine and sometimes not targeted OPPORTUNITIES Having good leadership High acceptance of the programme by the community New health extension package TREATS Budget shortage Shortage of trained personnel WOREDA LEVEL STRENGTH Allocate budget for CDTI on strategic plan CDTI is included in routine activity Supervision has conducted up to community level Reporting system of CDTI activities is strong Mectizan is collected, stored and distributed in the government system Supply of adequate Mectizan and ordered in good time Available transport facilities are used in integrated manner High geographical coverage WEAKNESS No satisfactory involvement of FLHF and woredas in planning Training, Mectizan ordering, distribution and fund were initiated at zonal level There was no enough budget, no government contribution and the budget have no break-down 58 Means of transport are not provided by government There is high turnover of human resource OPPORTUNITIES Develop strategic planning Support form political and administrative leaders Woreda empowerment on activity THREAT No timely release of budget/resource Shortage of trained personnel in financial management Many competing priorities other than oncho FRONT LINE HEALTH FACILITY LEVEL STRENGTH High coverage, both geographical and therapeutic Mectizan Ordering and management Human resources Monitoring and supervision WEAKNESS Planning is poor Leadership is weak Finances is poor Training and HSAM is inadequate Transport is inadequate OPPORTUNITY Integration Expansion of FLHF Human resources Promise for continuous Mectizan supply THREAT High turn-over of health man power Finance Planning is poor Leadership Transport Training and HSAM COMMUNITY LEVEL STRENGTH High coverage HSAM Mectizan Supply and management Leadership Planning and Management Monitoring and Supervision WEAKNESS Financing OPPORTUNITY Decentralization Acceptance of Mectizan/broad spectrum antihelminths Integration Ownership Health seeking behaviour Willingness of CDDS THREAT Support to CDDs by communities Long distance travel by CDDs to Household and Health Facilities Bad rumours (“CDDs are paid”) 59 Annex 5 : ADDRESSING WEAKNESSES AND THREATS IN OUR PROGRAMME GROUP I REPORT – PLANNING & LEADERSHIP, MONITORING & SUPERVISION ITEM WEAKNESS/THREAT SUGGESTION PLANNING & LEADERSHIP Planning was not well practice at woreda and FLHF Adopting bottom-up planning Lack of training on CDTI planning Training of woreda and FLHF staff on CDTI planning Woreda and FLHF were not empowered Making empowered leadership in particular on CDTI MONITORING & SUPERVISION Checklist were not used for supervision and monitoring Train staff on the use of checklist Supervision not problem oriented Supervision to targeted and problem oriented Not integrated with other health programme Supervision should be integrated with other health programme No written feedback Feedback should be implemented after monitoring and supervision GROUP II REPORT – FINANCES, TRAINING & HSAM ITEMS WEAKNESS/THREAT SUGESSTION FINANCES Absences of government contribution Government should allocate budget for CDTI activities Shortage of trained manpower on financial management Capacity building for concerned personnel Non release of budget on time Quick retirement of funds Lack of support for CDDs by the community Initiate the communities to support CDDs TRAINING & HSAM Training is not integrated nor targeted Integration and targeted of trainings High turnover of staff GROUP III REPORT – TRANSPORT AND MECTIZAN SUPPLY ITEMS WEAKNESS/THREATS SUGESSTION TRANSPORT Shortage of vehicles and motorbikes Integration of transport resources. Use of community transport means – horses, donkey No attempts to maintain/replace vehicle Plan to have regular maintenance/replacement Secure adequate budget for maintenance/replacement Long distance travel by CDDs Targeted supervision Mectizan Supply Long journey to collect Mectizan Delegation of zones to collect Mectizan from MoH Day Two On the second, Mr Teshome, from Carter Centre chaired the workshop. After a brief summary of the previous day’s activities was presented for recapitulation, Dr Elhassan, made a presentation on features, characteristics and format of a sustainable plan. Discussions and comments were entertained. Then, two groups were formed; one group to work on sustainable plans at national, regional, and zonal levels, and the other on a sustainable plan at woreda level. Group works on sustainable plans continued for the rest of the day. 60 Day Three On the final day, Mr. Esey Batisso, Oncho coordinator, SNNPR chaired the workshop. The agenda of the day started with participants concluding their previous day group work on the development of sustainable plans. This was followed by presentation and plenary discussion of plans developed for each woreda, zone and national levels. The workshop continued with presentations on practical steps to implement these plans as well as the way forward for the prepared plans. After these, the evaluation team presented the findings at national/regional levels followed by discussions. The one but last item on the agenda was the presentation of the recommendations for each level. Finally, the workshop came to a close by closing remarks from the evaluation team leader, Carter Center Country Representative, WHO, the team leader of the malaria and vector borne diseases prevention and control. 61 Annex 6: Sampling Table 1: Selected Districts District Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicity Geographical situation Accessibility/Convenience Masha (79.6%)  Yes Gimbo (78.68%) Yes Yeki (75.6%) Yes Andracha  (82%) X No Table 2: Selected Sub-districts/FLHFs from Masha District District Sub- district/ FLHF Primary criterion (Coverag e) Secondary criteria Selection (yes, no) Endemicit y Geographical situation Accessibility /Convenience Masha Masha Health Center  (80.6%) Yes Yina Health Facility (75.9%)  Yes Kanga  (72.8%) X No According to the primary criterion, Kanga FLHF has the lowest therapeutic coverage. However, it has been excluded due to secondary criteria, i.e. inaccessibility/inconvenience. Table 3: Selected Communities from Masha Health Centre Sub- district/FLHF Community /Kebele Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicity Geographical Situation Accessibility /Convenience Masha Health Center Abello  (82.6.1%) Yes Shibo (77.7%) X No Gatimo 74.7% X No Atteso 76.6% X No Gembeka 78.2 %  Yes 62 Abello community has been selected as it is with the highest therapeutic coverage. All the other three communities have been excluded because of secondary criterion (geographical spread); instead, Gembeka Community (Kebele) has been replaced as a community with the lowest therapeutic coverage. Table4: Selected Communities from Yina FLHF Sub- district /FLHF Community /Kebele Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicity Geographical situation Accessibility /Convenience Yina FLHF Yina Kebele  (76.2%) Yes Gada Kebele  (75.6%) Yes Two communities (kebeles) have been selected automatically based on the primary criterion. Table 5: Selected Sub-districts/FLHFs from Gimbo District Distric t Sub-district /FLHF Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicit y Geographic al situation Accessibility /Convenience Gimbo Gojeb State Farm FLHF  (86.1%) X No Wushwush Tea Plantation FLHF (84%)  Yes Argoba FLHF (77.81%) Yes Two sub-districts/ FLHFs from the Gimbo District (Woreda) have been selected using both categories of criteria (primary and secondary). Gojeb State Farm FLHF has been excluded due to geographical spread criterion; and Wushwush Tea Plantation FLHF has been substituted. 63 Table 6: Selected Communities from Wushwush Tea Plantation FLHF Sub- district /FLHF Community /Kebele Primary Criterion Secondary Criteria Selection (yes, no) Endemicity Geographical Situation Accessibility /Convenience Wushwush Tea Plantation FLHF Bubba Kebele  (107.5%) Yes Shukka Kebele  (73.67.6%) Yes Two communities have been selected based on primary criterion. Table 7: Selected Communities from Argoba FLHF Sub- district /FLHF Community /Kebele Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicity Geographical Situation Accessibility /Convenience Argoba FLHF Choba Kebele  (82.14 %) Yes Argoba Kebele  (73.48% Yes Argoba FLHF catchment area has only two communities, and both have been selected according to the primary criterion. Table 8: Selected Sub-districts/FLHFs from Yeki District District Sub- district/FLHF Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicity Geographical Situation Accessibility /Convenience Yeki Zinki FLHF  (78.1%) Yes Kubito FLHF  (73.6%) Yes Both sub-districts/FLHFs have been selected automatically based on the primary criterion. 64 Table 9: Selected Communities from Zinki FLHF Sub- district /FLHF Community /Kebele Primary Criterion (Coverage) Secondary Criteria Selection (yes, no) Endemicity Geographical Situation Accessibility /Convenience Zinki FLHF Zinki Kebele  (83.5%) Yes Tsanu Kebele  (72.6%) Yes Two communities have been selected according to the primary criterion. Table 10: Selected Communities from Kubito FLHF Sub- district /FLHF Community /Kebele Primary Criterion Secondary Criteria Selection (yes, no) Endemicity Geographical Situation Accessibility /Convenience Kubito FLHF Ermichi Kebele  (81.5%) Yes Kubito Kebele  (71.2%) Yes Two communities have been selected automatically based on the primary criterion. 65 Annex 7: List of Participants No Name Address Position 1 Gezahegn Tesfaye FMOH Team Leader, NOTF Co-chair 2 Dr. Daddi Jima FMOH National Oncho. Coordinator 3 Fekade Balcha IPB, Addis Ababa University NOTF Member 4 Mr. Kora Tushune Jimma University External Evaluator 5 Mr. Abraraw Tesfaye Jimma University External Evaluator 6 Bedruzeman Abdella Kaffa Zone Chief Administrator 7 Gashaw Dubale Kaffa Zone Health Desk Head 8 Bekele Kidane Kaffa Zone Coordinator 9 Naser Abdella Kaffa Zone Finance Head 10 Mesfin Tekle Kaffa Zone Administrator 11 Ayele Gebresilassie Kaffa Zone Council Head 12 Bezabih Bayu Kaffa Zone Administrator 13 Samson Melese Kaffa Zone, Bita Wereda Chief Administrator 14 Solomon Tesfaye Kaffa Zone, Bita Wereda Woreda Health Office 15 Atrsie Gebre Kaffa Zone, Chena Wereda Wereda Health Office Head 16 Addisu Alemayehu Kaffa Zone, Cheta Wereda Wereda Health Office Head 17 Asrat Abebe Kaffa Zone, Decha Wereda Chief Administrator 18 Admassu Ayele Kaffa Zone, Decha Wereda Wereda Health Office Rep. 19 Admasu G/Medhin Kaffa Zone, Gesha Wereda Wereda Health Office Head 20 Tariku Bezabih Kaffa Zone, Gesha Wereda Administrator 21 Asrat Sahile Kaffa Zone, Gewata Wereda Wereda Health Office Head 22 Shiteraw H/Mariam Kaffa Zone, Ghimbo Wereda Wereda Health Office Head 23 Endargachew Debebe Kaffa Zone, Gimbo Wereda Administrator 24 Tekle Abebe Kaffa Zone, Menjo Wereda Woreda Health Office Head 25 Fekadu Mekuria Kaffa Zone, Menjo Wereda Administrator 26 Hussen Mohamed Kaffa Zone, Silem Wereda Wereda Council Head 27 Mesfin Chido Kaffa Zone, Silem Woreda Wereda Health Office Head 28 Habtamu Assefa Kaffa Zone, Tello Wereda Vice Administrator 29 Tigistu Zerga Kaffa Zone, Tello Wereda Wereda Health Office Head 30 Osman Ibrahim Sheka Zone Programm Coordinator 31 Wodo Atto Sheka Zone Administrator 32 Habtamu Beyene Sheka Zone A/Zonal Health Department Head 33 Mekonen Inno Sheka Zone, Anderecha Wereda Woreda Health Office Head 34 Gezahegn Garefo Sheka Zone, Andracha Wereda Administrator 35 Agezegn Mammo Sheka Zone, Masha Wereda Administrator 36 Mitiku Haile Sheka Zone, Masha Wereda Wereda Health Office Head 37 Demeke Assefa Sheka Zone, Yeki Wereda Administrator 38 Esayas Kinfe Sheka Zone, Yeki Wereda Wereda Health Office Head 39 Esay Batiso SNNP Regional Health Bureau Regional Onco. Coordinator 40 Dr. Aseffa Worku The Carter Center Program Officer 41 Teshome Gebre The Carter Center Country Rep., NOTF Secretary 42 Ambachew Medhin WHO NPO, NOTF Member 43 Dr E. Tarimo c/o WHO Tanzania External Evaluator and Team Leader 44 Dr E. Elhassan SightSavers International, Nigeria External Evaluator 45 Mr U. Ekpo Univ of Agriculture, Abeokuta, Nigeria External Evaluator 46 Mr C. Evini Hellen Keller International, Cameroon External Evaluator 47 Dr A. Atabe SightSavers International, Cameroon External Evaluator