Mid-term evaluation of the sustainability of the North-West CDTI project, Cameroon: March 2008

World Health Organisation African Programme for Onchocerciasis Control REPORT Mid-term evaluation of the Sustainability of the North-West CDTI project, Cameroon March 2008 Evaluation Team Members: Gaba Julius (Team Leader) Akoh-Arrey Matilda Nkeng Alexander Oponde Peter Evini Cyrille Thierry. 2TABLE OF CONTENTS Page Table of contents ............................................................................................................. Abbreviations/Acronyms .............................................................................................. Acknowlegments ............................................................................................................ Executive summary ....................................................................................................... 1. Introduction ............................................................................................................... 2. Methodology ............................................................................................................... 2.1 Sampling .............................................................................................................. 2.2 Source of information .......................................................................................... 2.3 Analysis ............................................................................................................... 2.4 Team composition ............................................................................................... 2.5 Advocacy meeting and Feedback/planning workshop ........................................ 3. Majors Findings, Discussions and Recommendations ........................................... 3.1 Provincial level ..................................................................................................... 3.2 Health District level .............................................................................................. 3.3 Front Line Health Facility level ........................................................................... 3.4 Community/village level ………………………………………………….......... 4. Grading the whole project ........................................................................................ 4.1 Qualitative judgement …………………………………………………………. 4.2 Quantitative judgement ………………………………………………………… 5. Conclusion .................................................................................................................. Annexes ........................................................................................................................... Feedback and planning workshop agenda................................................................... Report of Feedback and planning workshop............................................................... Contacts of evaluators ................................................................................................ 2 3 4 5 9 9 9 10 10 10 11 12 12 17 22 26 30 30 32 32 33 34 36 42 3ABBREVIATIONS/ACRONYMS APOC African Programme for Onchocerciasis Control C/BAFA Chief Bureau of Administration and Financial Affairs CBH Chief Bureau of Health CDD Community-Directed Distributor (of Ivermectin) CDTI Community-Directed Treatment with Ivermectin COC Chief of Centre COP Chief of Post CSM Community Self-Monitoring DMO District Medical Officer EPI Expanded Programme on Immunization FLHF Front Line Health Facility GCR Geographical Coverage Rate HA Health Area HC Health Centre HD Health District HSAM Health Education, Sensitisation, Advocacy and Mobilization MoH Ministry of Health NGDO Non-Governmental Development Organization NOTF National Onchocerciasis Task Force NWP North West Province OPC Onchocerciasis Provincial Coordinator PCSGA Provincial Chief of Service for General Affairs PCSP Provincial Chief of Service for Planning PCUSME Provincial Chief of Unit for Supervision, Monitoring and Evaluation PDPH Provincial Delegate of Public Health REMO Rapid Epidemiological Mapping of Onchocerciasis SAEs Severe Adverse Effects SSI Sights Savers International TB Tuberculosis TCR Therapeutic Coverage Rate WHO World Health Organization 4ACKNOWLEDGEMENTS We would like to express our appreciation and gratitude to the following institutions and persons for their contributions in various ways to the successful completion of the assignment.  Dr U. Amazigo and other staff at the Headquarters of the African Programme for Onchocerciasis Control (APOC) in Ouagadougou  WHO Country Representative and all staff, Cameroon  The National Coordinator for Onchocerciasis Control Programme, Cameroon  The Country Representative of Sight Savers International, Cameroon  The Provincial Delegate of Public Health, North-West Province  Political and traditional leaders, health workers and community leaders and members of all areas visited in Mbengwi, Santa, and Fundong. 5EXECUTIVE SUMMARY North West Province (NWP) is one of the ten provinces of the Republic of Cameroon. The Community Directed Treatment with Ivermectine (CDTI) was introduced in NWP in February 2004, and carried out for four years. At the request of APOC, a team of evaluators carried out an evaluation of the sustainability of the project between 17 March and 02 April 2008. The evaluators were assigned three specific tasks:  To evaluate the sustainability of the project.  To present and discuss the results with provincial and health districts staffs, NGDO.  To support the provincial and health districts levels in developing sustainability plans using the guidelines for sustainability planning developed by APOC. The sampling was done randomly, in a stratified way. Information was collected from interviews, verbal reports and documents. Analysis was made level by level and based on the information collected. Project performance in respect of each indicator is graded on a scale of 0-4, in terms of its contribution to sustainability Findings Provincial level There was no overall plan for 2007. The 2007 CDTI detailed plan as also a plan for 2004 – 2005 existed, but they did not have costs for activities. The planning was done during provincial appraisal meetings, during which all partners participate. The appraisal meeting of 2007 was attended only by partners of government and NGDO. The roles of all partners were defined even those of the stakeholders (councils, communities) who were absent. There was no precise plan for sustainability, but some thought had been given to sustainability. It was reported that activities were carried out in an integrated manner, but there was no evidence seen. Some members of the leadership team were relatively aware of some aspects of the CDTI project philosophy, progress, successes and problems of the project. At lower levels the responsibilities are delegated to the district teams. There was an annual technical report (2007) which contained all key activities of CDTI. Appraisal meeting report and activity reports of one zonal OPC were also available. The OPCs carried out supervision in the districts and exceptionally at the FLHF and community in case of problems. But no copy of supervision report was kept for filing. There was a checklist for supervision and a supervision plan for 2006. Problems were solved. But the processes of management of problems and successes were not documented. The Mectizan supply is controlled within the government system. The system is effective, uncomplicated and efficient. The Mectizan was supplied in time and in sufficient quantities. Training is done level by level. Training is routinely carried out every year. An advocacy meeting was organised in November 2007 for CDTI and other health programmes. HSAM activities were not effectively planned. However, before the commencement of each round of distribution, sensitisation messages were broadcast over the local radios. Church announcements were also made. The letter of agreement and approved budget for 2007 with various resources and sources clearly spelt out were seen. Budgets of 2005 and 2006 were not seen. There was very little follow up of the budget, hence gaps were not known. Though no written document was 6seen, the NGDO (SSI) has expressed the intention to continue supporting all projects which are controling blindness. Apart from the vehicle given by APOC for CDTI activities, there was a pool of appropriate vehicles and motorcycles from government and other health programmes used in an integrated manner. There are also other materials: computer, photocopier, fax machine, overhead projector, training modules, HSAM materials. The CDTI vehicle was maintained as needed from APOC funds, there was no particular plan to replace existing materials, but the ongoing elaboration of health district development plans gives room to progressively replace old materials. Apart from the pooled staff at the Provincial Delegation, there were 04 OPCs which were qualified for the job. However, it was reported that they needed much capacity building in numerous aspects of their job. At the last distribution (2007): GCR was reported at 97%. Examination of the technical reports for the year showed error of calculation. In 2005 and 2006 GCR were 98%. Health District level There were yearly plans drawn following a stereotyped format provided by the Provincial team. The plans were integrated into the overall plan of the HD. The plans included key CDTI activities. The plans sometimes took into consideration community desires/period of distribution. Plans were developed in a participatory manner. Minutes of planning meetings were most of the time absent. Staffs integrated more than one activity. There were no written work plans. Reports of supervision were seen for individual programmes eg EPI, CDTI. HD management teams are initiating key CDTI activities. There was a focal person at this level for CDTI activities. The reporting system is through the government system. Supervision was carried out by DMOs and CBH at FLHF level, they visited communities identified as being weak. Visits were integrated as they dealt with all programmes being implemented at this level. Problems were addressed as soon as they arose. SAEs cases were promptly referred to the appropriate hospitals. FLHFs staffs solved their own problems; however, they referred some to the higher levels. Feedback was provided to FLHFs staff during periodic meetings or during visits. No evidence of recommendation was seen as no written document was available. No ordering form was seen but a table showing the number of tablets used during the last round of distribution and the number needed for the next one was seen. Mectizan® arrived on time for distribution and no shortage was reported. The system is effective and uncomplicated. Training was routine and carried out every year. FLHFs staffs have been capacitated and empowered to train CDDs. HSAM activities were planned and carried out. HSAM activities were carried out at this level and FLHF were left to do their own at their level. There was no evidence of HSAM activities having led to action. In rare instances (30%), the cost for each CDTI activity was spelt out in the plans. There was no evidence of cost reduction strategy. There was no clear estimate of the funds that will be made available to them in the coming year. Funds allocated by the Government were in a lump sum and used in the execution of all health programmes. The funds have been increasing but there was no clear estimate as to how much the Government has been contributing in CDTI implementation. The amounts provided by the Government were 7somehow less than what was expected. However, the gap was bridged by APOC and Sights Savers International. There was no written commitment from any source to continue sponsorship of the programme once APOC funding ceases. Out of the three HDs visited, two had vehicles that were in bad shape but could be repaired while one was parked. Out of the 6 HAs visited, none had a motorcycle. Some HSAM materials were seen. Training materials were scarce. Existing vehicles were run and maintained from Government sources. Maintenance costs were supported by the Government. Sometimes, staffs used their personal transport means. Transport is usually used in an integrated manner. Generally, staffs were stable in their work. Staffs were equipped to implement CDTI activities effectively. Most staffs expressed their satisfaction in carrying out CDTI activities. There was however no system put in place to reward well performing staffs. Staffs were very committed in doing their job. All the health districts sampled had satisfactory GCR (100% from 2004 to 2007). All the 3 HDs sampled had TCR > 65% in 2007. Front Line Health Facility Most of the HC (70%) visited had written plans of action, of which 30% were integrated. It was generally observed that CDTI is part of the minimum package of health activities. Staffs carry out CDTI activities in an integrated manner. In 85% of HC, it was also reported that staffs combined CDTI activities with those of other programmes including EPI, malaria, and TB. However, there was no supporting document. CDDs are visited as need arises. Most of the time, data is collected and sent at the end of a given activity. Copies are not kept at the HC level though the reports were seen in the district office. Each community where Mectizan® is being distributed is visited 2-3 times during the distribution round. In most of FLHF visited supervision visits were said to be integrated. However, there was very little written evidence. Problems identified during visits were treated appropriately. There is very little or no feedback given to the community. Ordering of Mectizan® was carried out based on previous year’s treatment data. Mectizan is collected at the HD level during training. The system is effective, simple and uncomplicated. The staff trained CDDs yearly in a routine manner at the health facility in one day. Training material was very rarely seen. Health facility staffs carry out sensitization activities in a routine manner. The last round of treatment budgeting was not carried out at the FLHF. Health personnel at this level had no idea about budget. Funding disbursed is barely enough to enable carry out essential CDTI activities. It was not clear as to how much various sources have contributed and staffs could not state how much CDTI activities have costed. In most cases there was Government contribution to the implementation of CDTI but amount could not be clearly estimated. HA staffs lacked transport materials. The ones they had were either private materials or completely not adapted given the distances to cover and the terrain. In spite of these problems CDTI activities have not been disrupted. There was generally lack of training and HSAM materials. Staffs either use their private means of transport. 8Staffs were generally knowledgeable and skilled in most areas of CDTI activities and management of side effects. Staffs are stable, but inadequate. And when transfers occur, immediate training of new unskilled staff is undertaken. The GCR was100% in all the HDs visited, and the TCR was more than 65% in all villages. Community level CDDs systematically organized door-to-door visits to households to take census and distribute the drug at the same time. In most cases CDDs reported problems to the Chief of Centre to help in problem solving. No or very little initiative was reported from community leaders with regards to Mectizan distribution. Community leaders had very little knowledge of the problems encountered by CDDs. In 50% of communities visited, CDDs were directly selected from the health committee by the Chiefs of posts. CDDs decided on the timing of distribution. There was no report of community common initiative to support CDDs. There were few reports of CSM. Mectizan® is very much valued among the community members. They all expressed their readiness to take it as long as necessary. Once the distribution is over, the reports were given to the HC about a week or two after the last sunlight hours of distribution. In all cases, CDDs either trekked or used personal means to go to the COP. Census data were good as no shortage was reported. No transport was arranged by the community for Mectizan® to be collected. CDDs engaged in health education. There were no precise plans to educate the population and community authorities were sometimes actively involved in some HSAM activities. In general, there was no community common initiative to support CDDs. In general, CDD ratio to population ranged from 1/60 to 1/211, CDDs worked at a range of 1 to 2.5 km. All the CDDs met were very knowledgeable about their job including identification of SAEs and prompt referral. Out of the 12 communities that were sampled, 8 had a therapeutic coverage >65% in 2007. Overall grading of the project Two aspects of sustainability (integration and ressources) over seven were not helping the programme. Only one critical element (money) was not sufficient. Average of scores of indicators at the four levels was 2.5 over 4. In accordance with the evaluation guidelines, the evalators’ team concluded that the North West CDTI Project was making satisfactory progress toward sustainability. 91. INTRODUCTION North West Province (NWP) is one of the ten provinces of the Republic of Cameroon. It is bordered in the south by South West Province, in the east by West and Adamoua Provinces, in the west and north by Federal Republic of Nigeria. North West Province covers a surface area of 18,100 km2. The total population is estimated at 1,9 millions inhabitants. It is composed of 7 administrative divisions that are divided into 17 health districts. The 18 HD are divided into 188 health areas (HA). Each HA has one or more health facilities called health centers. The bio-climatic zone of the NWP is mainly sudan – savannah grassland with plains, hills and valleys. This Province is irrigated by several streams. The Community Directed Treatment with Ivermectine (CDTI) was introduced in NWP in February 2004, and has been carried out for four years. Funding of onchocerciasis control activities is provided in part by the NGDO partner Sight Savers International (SSI). In December 2004, independent monitoring had been conducted in the province. At the request of APOC, a team of evaluators, from Cameroon and Benin carried out an evaluation of the sustainability of the project between 17 March and 02 April 2008. 2. METHODOLOGY Evaluation question: how sustainable is the North West CDTI project? Design: Cross-sectional, descriptive. Population: the North West CDTI project, includes Provincial Delegation of Public Health team, DMOs and their teams, chiefs of FLHFs, communities leaders, communities members, CDDs. Sample: 12 communities/villages, 6 health areas, 3 health districts were selected. 2.1 Sampling: The sampling is done randomly, in a stratified way. Eighteen Health Districts of the Province were arranged in a descending order from highest to lowest, and divided into three strata. From each stratum, one HD was selected. Thereby 3 HD were, Fundong, Mbengwi and Santa. Then, FLHFs of each of the HD were arranged in a descending order from highest to lowest, and divided into two strata. In each group a FLHF was draw up. Six FLHF were thus selected. Lastly, 12 villages/communies were selected in the same manner. 10 Tab. 1: Distribution of HDs, FLHFs and communities sampled S/N Health Districts S/N FLHF S/N Villages/community 1 FUNDONG (CT: 72.30%) 1 Mbessa (CT: 78.43%) 1 Nbessa 2 Ajung 2 Belo (CT: 46.80%) 3 Fuli 4 Njinikijem 2 MBENGUI (CT : 74.70%) 3 Njindom (CT: 38.00%) 5 Njindom 6 Barakwe 4 Abebong (CT: 89.90%) 7 Njembeng 8 Akuandong 3 SANTA (CT: 68.70%) 5 Akum (CT: 88.02%) 9 Ntamadam 10 Nilap 6 Ndapang (CT: 51.21%) 11 Kwenegli 12 Ndapang 2.2 Source of information Information was collected from interviews, verbal reports and documents. Various categories of people were interviewed at the provincial level (PDPH, 2 OPC, Chief of Unit of Supervision, Monitoring ad Evaluation, Chief of Service of Planning and Chief of Service General Affairs, staff of the North West Special Fund for Health), health district level (DMO, CBH, C/BAFA), front line health facility level (chiefs of centre), communities/villages level (CDDs, community leaders, community members). Information was recorded on the evaluation instruments and discussed before grading the performance of the relevant level on the indicator. 2.3 Analysis Analysis was made level by level and based on the information collected. Project performance in respect of each indicator is graded on a scale of 0-4, in terms of its contribution to sustainability. The average sustainability score for each group of indicators is calculated, for each level. The evaluators discussed also qualitative description of findings. 2.4 Team composition This evaluation was carried out by the following persons: 1. Dr Julius Gaba (Team Leader) 2. Mr. Cyrille Evini 3. Dr. Matilda Akoh-Arrey 4. Dr. Alexander Nkeng 5. Mr. Peter Oponde 11 These team members were divided into three sub-teams, and each sub-team was accompagned by a guide and a translator. 2.5 Advocacy visits and ‘Feedback/Planning’ Workshop Advocacy visits were paid to relevant people of the District. At the provincial level the Governor was not seen. During the workshop, the evaluation team presented its findings and guided the provincial and health districts teams in the development of a three-year post-APOC sustainability plan which took the evaluation findings into consideration. 12 3. MAJOR FINDINGS, DISCUSSIONS AND RECOMMENDATIONS 3.1. Provincial level 3.1.1 Findings and discussions Planning (1.0): There was no overall plan for 2007 but it was reported that there was one in 2006 which was not seen. It was also reported that each district elaborates its plan and the provincial level does the synthesis. The 2007 CDTI detailed plan existed and was seen. The plan contained activities to be implemented by all actors from the provincial down to the community level. It is worth-noting that this same plan was seen in the district and health area levels except for the period of execution of activities that differed. It was also reported that the 2006 plan existed but no evidence was presented to the team. There existed a plan for 2004 – 2005. The two plans seen (2004-2005 and 2007) did not have costs for activities. Hence it was not possible to assess the variation of specific needs by year. The activities were the same for the 2 plans. It was reported that planning was done during provincial appraisal meetings during which all partners participated. However, the list of attendance of the appraisal meeting of 2007 showed that only partners of government and NGDO were present. The minutes of the appraisal meeting in 2007 showed that roles of partners were defined even those of the stakeholders (councils, communities) who were absent. Though some thought had been given to sustainability during an advocacy meeting held on the 21st November 2007 (minutes of the meeting showed that the participants discussed issues on the post-APOC assistance), there was no precise plan for sustainability. Intregration (2.0): It was reported that activities were carried out in an integrated manner, but there was no evidence seen. A supervision checklist which contained CDTI and other health programmes existed and was seen but there was no report or filled checklist on the integrated supervision reported to have been carried out in August 2007. During supervision of distribution of mectizan which took 5 weeks only CDTI activities were supervised by the team in charge of CDTI. Leadership (2.0): Some members of the leadership team were relatively aware of some aspects of the CDTI project philosophy, progress, successes and problems of the project. As concerns Coordination of the Project at this level, it is done by the Provincial Delegate of Health. There are 04 zonal OPCs each responsible for a number of health districts. At lower levels the responsibilities are delegated to the district teams. Monitoring/supervision (2.5): There was annual technical report (2007) which contained all key activities of CDTI. The treatment summary, equipment and financial reports were all available. As for periodic reports, appraisal meeting report and activity reports of one zonal OPC were also available. However, some of the reports were not dated nor signed by the responsible officers. Supervision was reported to be done by this level. The OPCs reported that they carried out supervision in the districts and exceptionally at the FLHF and community in case of 13 problems. The reports of supervision were submitted individually to the financial officer with justifications, but no copy was kept for filing. It was reported that routine supervision was done once a year primarily by the 04 zonal OPCs and the PDPH. There was a checklist for supervision and a supervision plan for 2006. Problems were reported to be routinely solved. But the processes of management of problems and successes were not documented. The major problem reported was that of geographical coverage which is not yet attained because of difficult accessibility to some zones (Furu Awa) where CDTI activities have not yet started. The DMOs have been trained to manage problems at their level. Mectizan (3.5): The mectizan supply is controlled within the government system. The system is effective, uncomplicated and efficient. The mectizan was supplied in time and in sufficient quantities. Training/HSAM (2.5): Training is done level by level. Higher levels train lower ones. Training is routinely carried out every year at the provincial level for 2 days. A report on training was available for 2005. An advocacy meeting was organised in November 2007 for CDTI and other health programmes (minutes of meeting were available). HSAM activities were not effectively planned. However, before the commencement of each round of distribution, sensitisation messages were broadcast over the local radios and it was also reported that church announcements were made. Financial resources (1.0): An advocacy meeting was organised in November 2007 for CDTI and other health programmes (minutes of meeting were available) The letter of agreement and approved budget for 2007 with various resources and sources clearly spelt out were seen. Budgets of 2005 and 2006 were not seen. As evidenced in the 2007 annual Technical Report of the Project, government contribution to project implementation has been reported at:  $237,522 in 2005  $208,123 in 2006  $194,137 in 2007 It was generally estimated that government contribution was used to fill gaps, apart from CDDs motivation. There was very little follow up of the budget, hence gaps were not known. Though no written document was seen, the NGDO (SSI) has expressed the intention to continue supporting all projects which are controling blindness. 14 Transport /other material resources (3.5): Apart from the vehicle given by APOC for CDTI activities, there was a pool of appropriate vehicles and motorcycles from government and other health programmes used in an integrated manner. As concerns office equipement, there is a computer, photocopier, fax machine, overhead projector given by APOC. Training modules and materials, registers were seen.HSAM materials (2 flip charts, a few flyers, some posters, 2 T-shirts) were seen. The T-shirts were supplied by MoH and SSI in 2005. A sample of a flyer was seen and it was reported that copies were being produced. The CDTI vehicle was maintained as needed from APOC funds but there was no schedule for maintenance. A log book was seen and it was filled. Though there was no particular plan to replace existing materials, the ongoing elaboration of health district development plans gives room to progressively replace old materials. 9. Human resources (2.0): Apart from the pooled staff at the Provincial Delegation, there were 04 OPCs which were qualified for the job. 3 of the 4 OPCs also served as CBH in specific HDs. However, it was reported that they needed much capacity building in numerous aspects of their job. 10. Coverage (3.5): At the last distribution (2007): GCR was reported at 97%. Examination of the technical report for the year showed errors of calculation. In 2005 and 2006 GCR were 98%. Fig 1: Sustainability at provincial level 1 2 2 2,5 3,5 2,5 1 3,5 2 3,5 0 0,5 1 1,5 2 2,5 3 3,5 4 Pla nn ing Int eg rat ion Le ad ers hip Mo nit ori ng /su pe rv Me cti za n Tr ain ing /H SA M Fin an ce Tr an sp ort /ot he r m at res Hu ma n r es ou rce s Co ve rag e group of indicators av er ag e w ei gh t(/ 4) 15 3.1.2 Recommendations at Provincial level Recommendations by Group of Indicators Implementation Planning 1.The Provincial level should elaborate an overall plan with costs that contains CDTI activities Priority: HIGH Indicator of success: an overall plan available Who to take action: PDPH Deadline for completion: june 2008 2.The CDTI plan for the provincial level should contain only activities to be executed by the provincial team Priority: MEDIUM Indicator of success: plan containing activities only for the provincial team Who to take action: PDPH Deadline for completion:june 2008 3.There should be a follow up of the execution of the approved CDTI plan Priority: HIGH Indicator of success: evidence of follow up Who to take action: OPC Deadline for completion: november 2008 Supervision/Monitoring 4.Activities should be appropriately documented and copies kept at this level for action and future reference Priority: HIGH Indicator of success:reports of activities available Who to take action: OPC Deadline for completion: a week after implementing each activity 5. There should be a documented routine system to reward well performing staffs. Priority: HIGH Indicator of success: well performing staffs are rewarded. Who to take action: PDPH Deadline for completion: at the end of every year. Financial Resources 6. Government funds should be allocated and effectively disbursed at this level for CDTI activities. Priority: HIGH Indicator of success:evidence of funds allocated and disbursed for CDTI Who to take action: PDPH Deadline for completion:july 2008 7.Staffs should document Government contribution with regards to CDTI Priority: HIGH Indicator of success: Goverment contribution documented Who to take action:Finance and Administration Officer of Provincial health delagation Deadline for completion:july 2008 Human Resources 8.There is an immediate need to appoint a provincial coordinator to closely follow-up on CDTI activities Priority: HIGH Indicator of success:an OPC in place Who to take action:PDPH Deadline for completion:june 2008 Transport /Other Resources 9. Staff should ensure that adequate training and HSAM materials are available for the entire Province. Priority: MEDIUM Indicator of success: adequate training and HSAM materials available in entire province Who to take action: PDPH Deadline for completion: December 2008 16 10.Staffs should document the use of transport and other material resources at their disposal Priority: HIGH Indicator of success: log sheet and material inventory available Who to take action:PCSAFH Deadline for completion: july 2008 11.Staffs should continue to advocate for transport materials for the District and levels below Priority: HIGH Indicator of success:transport/other materials available at districts and lower levels Who to take action:PDPH Deadline for completion: mai 2009 Training/ HSAM 12.The provincial level should ensure that all stakeholders are involved in appraisal and advocacy meetings Priority: HIGH Indicator of success:stakeholders are involved in appraisal and advocacy meetings Who to take action:PCUSME Deadline for completion: december 2008 13. HSAM activities should effectively be planned, implemented, and followed-up in an integrated and targeted manner. Priority: MEDIUM Indicators of success: - targed HSAM plan available - Reports of HSAM activities implemented in an integrated manner exist. Who to take action: PCSP Deadline for completion: December of each year 14.Written report on HSAM activities have to be made to document successes Priority: MEDIUM Indicator of success: written report available Who to take action:OPC Deadline for completion: a week after each success Integration of Support Activities 15. Staffs should have integrated written plans and reports showing that integration of activities is effective. Priority: HIGH Indicators of success: integrated written plans available. – Reports of integrated activities available. Who to take action:PCSP Deadline for completion:july 2008 Mectizan 16.Involve the NWPSFH in the ordering, supply and follow-up of Mectizan Priority: HIGH Indicator of success:Mectizan ordering,supply,follow-up done effectively by NWPSFH Who to take action: PDPH Deadline for completion: next round of Mectizan distribution Coverage 17.Ensure that all communities/villages have at least 65% TCR annually Priority: HIGH Indicator of success:all village has TCR>or=65% Who to take action : PDPH,DMO,commuunities Deadline for completion: next round of Mectizan distribution. 17 3.2 Health Districts level 3.2.1 Findings and discussions 1- Planning (2.0): There were yearly plans drawn following a stereotyped format provided by the Provincial team. The plans were integrated into the overall plan of the HD. The plans included key CDTI activities and had details about training, Mectizan supply, mobilization, treatment and advocacy, etc. The plans sometimes took into consideration community desires/period of distribution. Plans included period of execution but lacked dates. Plans were developed in a participatory manner during appraisal meetings involving the DMT, a representative from the Provincial team, chiefs of posts and community members. Minutes of planning meetings were most of the time absent. 2- Integration (1.5): Staffs reported that they integrated more than one activity when they went out for supervision. Integrated activities included EPI, HIV-AIDS. Some mission orders were seen.Two or more activities were usually combined during monitoring and supervision. There were no written work plans. Reports of supervision were seen for individual programmes eg EPI, CDTI. 3- Leadership (3.0): HDs management teams are initiating key CDTI activities. In some instances, the DMOs were ready to launch training without even waiting for the project funds, in order not to run late on planned activities. There was a focal person at this level who is the CBH who initiates activities at this level. 4- Supervision (2.5): The reporting system is through government system. Data collected are transmitted from the health centre, to the HD and to the Province. Annual appraisal meetings are held at the end of the distribution period and all reports including financial justifications are collected during the said meeting. Data transmitted included training report, Mectizan ordering and supply and distribution. Supervision was carried out by DMOs and CBH. Normally, their visits stopped at HA level; however, it happened that they visited communities where HA staffs had been identified as being week. Visits were integrated as they dealt with all programmes being implemented at this level. Problems were addressed as soon as they arose. However, it occurred that specific problems were not dealt with promptly (REA not conducted on time). Church announcements were made to sensitize communities where coverages were found to be low. SAEs cases were promptly referred to the appropriate hospitals. HAs staffs solved their own problems, however, they referred some to the higher levels. Feedback was provided to HA staff during periodic meetings or during visits. No evidence of commendation was seen as no written document was available. 18 5- Mectizan (3.5): Though no ordering form was seen from the field, the evaluation team was presented with a table showing the number of tablets used during the last round of distribution and the number needed for the next one. The ordering form was later on evidenced at the provincial special fund for health (Essential Drugs Programme). Mectizan® arrived on time for distribution and no shortage was reported. DMT collected Mectizan® at the special fund for health based on their needs and using their own means. Management of Mectizan® was then closely followed by the DMT. Once at the HD level, it was kept by the CBH. Unlike the other drugs which are managed at local pharmacies of the Health Centres, Health facilities collected Mectizan from the CBH. The system is effective and uncomplicated. 6- Training/HSAM (1.5): Training was routine and carried out every year. DMT trained HA nurses, District Hospital staff and laboratory technicians. At least three DMT were used as trainers of trainers. HA staffs have been capacitated and empowered to train CDDs. Training objectives were redundant and sometimes unclear. On average, 3 trainers were used to train 27 persons for a one-day session. HSAM activities were planned and carried out one week before the distribution was launched. Sensitization messages were routinely sent to churches. Sensitisation activities were not targeted as staffs missed the opportunity to advocate and convince administrative authorities. HSAM activities were carried out at this level and FLHF were left to do their own at their level. There was no evidence of HSAM activities having led to action though it was strongly believed that HSAM activities had apparently led to the acceptance of the programme at the FLHF and in the communities (high coverages). Minutes of meetings of HSAM activities and yearly annual report were not seen. 7- Financial resources (1.0): In rare instances (30%), the cost for each CDTI activity was spelt out in the plans. There was no evidence of cost reduction strategy. There was no clear estimate of the funds that will be made available to them in the coming year. Funds allocated by the Government were in a lump sum and used in the execution of all the health programmes. Allocations and approvals were done at higher levels. Government funds were mainly used to fill up the gaps and the bigger the gap, the more the Government contributed. The funds have been increasing but there was no clear estimate as to how much the Government has been contributing in CDTI implementation as no regular insight was done with regards to budget line. The amounts provided by the Government were somehow less than what was expected. However, the gap was bridged by APOC and Sightsavers. As of now, no plan is available as to how resources will be raised to continuously implement CDTI. There was no written commitment from any source to continue sponsorship of the programme once APOC funding ceases. In general documentation was very weak. 19 8- Transport/other material resources (2.0): Out of the three HDs visited, two had vehicles that were in bad shape but could be repaired while one was parked. Out of the 6 HAs visited, none had a motorcycle. Some HSAM materials were seen. Training materials were scarce. Existing vehicles were run and maintained from Government sources. Existing vehicles were said to be regularly maintained but there was no documented evidence of routine maintenance. Maintenance costs were supported by the Government. Staffs even used their personal transport means when official ones were not provided. Transport is usually used in an integrated manner, but no log book was seen. Where official transport was available and there was a need, trips were authorised by the Senior Divisional Officer or Divisional Officer and documentation was sometimes seen. Though there is no particular plan to replace existing materials, the ongoing elaboration of HD Development plans gives room to progressively replace old materials. 9- Human resources (3.0): Generally, staffs were stable in their work. Staffs were equipped to implement CDTI activities effectively.Most staffs expressed their satisfaction in carrying out CDTI activities. There was however no system put in place to reward well performing staffs. Salaries are regularly paid. Staffs were very committed in doing their job. 10- Coverage (3.5): All the health districts sampled had satisfactory geographical coverage rates (100% from 2004 to 2007). Out of the 3 HDs that were sampled, none had a therapeutic coverage rate <65% in 2007. Meanwhile out of the 17 HDs that were contained in the project area visited 16(94%) had a TCR >65% in 2007. In 2006 and 2005 respectively, 94% and 82% of the HDs had a TCR>65%. Fig 2: Sustainability at health districts level 2 1,5 3 2,5 1,5 1 2 3 3,53,5 0 0,5 1 1,5 2 2,5 3 3,5 4 Pla nn ing Int eg rat ion Le ad ers hip Mo nit ori ng /su pe rv Me cti za n Tr ain ing /H SA M Fin an ce Tr an sp ort /ot he r m at res Hu ma n r es ou rce s Co ve rag e group of indicators av er ag e w ei gh t(/ 4) 20 3.2.2 Recommendations at health districts level Recommendations by Group of Indicators Implementation Planning 1. Staffs should make their own situation analysis and elaborate their plan based on it. Priority: HIGH Indicator of success:Plan elaborated Who to take action:DMO Deadline for completion:june 2008 2. Staff should include costs on their yearly activity plans Priority: HIGH Indicator of success:Budget plan available Who to take action:DMO Deadline for completion:june 2008 3. There should be a follow up of the execution of the approved plan. Priority:MEDIUM Indicator of success: follow up of the plan done Who to take action: DMO Deadline for completion:through out the year Supervision/Monitoring 4. Staffs should have integrated plans for supervision written out Priority: HIGH Indicator of success: integrated plan of supervision available. Who to take action:DMO Deadline for completion: june 2008 5. Staffs should ensure that activities are documented and copies are kept at their own level for action and future reference Priority: MEDIUM Indicator of success: reports of activities available Who to take action: DMO, CBH, C/BAFA Deadline for completion: a week after each activity Financial Resources 6. Once the approved budget from external sources is made available, staffs should look for ways to bridge the gaps between money expected and money approved Priority: HIGH Indicator of success: Budget gap bridged Who to take action: DMO Deadline for completion: july 2008 7. Staffs should document Government contribution with regards to CDTI Priority: HIGH Indicator of success: Goverment contribution documented Who to take action: DMO, C/BAFA Deadline for completion: june 2008 Human Resources 8. DMT should put in place a routine documented system to reward well performing staffs Priority: MEDIUM Indicator of success: well performed staffs routinely rewarded Who to take action: DMO Deadline for completion: december avery year Transport /Other material resources 9. Staffs should document the use of transport and other material resources at their disposal. Priority: HIGH Indicator of success:Log book, material inventory available Who to take action:DMO Deadline for completion:during the use of transport; June and dec every year for inventory. 21 10. Staffs should continue to advocate for transport materials for the District and levels below Priority: HIGH Indicator of success:transport material available Who to take action: DMO Deadline for completion:june 2009 HSAM/Training 11. Staff should organise targeted training that is train only those in need and/or only on relevant subjects. Priority: HIGH Indicator of success:Target training carried out Who to take action:DMO Deadline for completion: before each training 12. Staffs should effectively plan, implement, follow-up and make written report on HSAM activities. Priority: HIGH Indicator of success:-plan of HSAM activities avaible - written report on implementation, follow-up of HSAM activities available. Who to take action:DMO/CBH Deadline for completion: before each activity (for the plan); a week after each activy (for the report). 13. Written report on HSAM activities have to be made to document successes. Priority: MEDIUM Indicator of success:written report on successes available Who to take action:DMO/CBH Deadline for completion:a week after a success 14. Staff should ensure that appropriate and adequate training and HSAM materials are available for the entire health district. Priority: MEDIUM Indicator of success:-training on HSAM carried out -HSAM materials available in health districts Who to take action:PDPH Deadline for completion:December 2008 Integration of Support Activities 15. Staffs should have integrated written plans and reports showing that integration of activities is effective. Priority: HIGH Indicator of success:-written integrated plan avaible; - written reports of integrated activities available. Who to take action:DMO Deadline for completion:june 2008(for integrated plan); a week after each activity (for reports) Mectizan 16. Staffs should set up a stock forms for stock control. Priority: MEDIUM Indicator of success:Stock forms available and used Who to take action:DMO Deadline for completion:july 2008 Coverage 17. Staff and all stakeholders should ensure that all communities/villages have at least 65% TCR annually Priority: HIGH Indicator of success:TCR is at least 65% in all villages Who to take action : DMO, chef of health centers Deadline for completion : from 2008 22 3.3 Front Line Health Facility Level 3.3.1. Findings and discussions Planning (3.0): Situation varied from one HC to the other. 70% of the HC visited had written plans of action (showing period, activity, place, and person in charge, indicator and funding source for each activity), of which 30% were integrated. It was generally observed that CDTI is part of the minimum package of health activities. In 30% of H C visited, no plan was available. Staffs however reported that they implemented CDTI according to the plan adopted during appraisal meeting at HD level. Integration of Support Activities (3.0): Staffs carry out CDTI activities of Monitoring, HSAM, training, fetching of records and Mectizan® delivery in an integrated manner. In 85% of HC, it was also reported that staffs combined CDTI activities with those of other programmes including EPI, malaria, and TB. However, there was no supporting document. Leadership (3.0): In most HC visited the management team was made of one person even when there were more than one staff. Once the yearly plan was adopted at the HD level, the staff took responsibility in key CDTI activities implimentation and follow up. They initiated HSAM activities, training of CDDs, and monitoring of the distribution of Mectizan. In some (60%) cases, village councillors were involved and they could describe main CDTI activities as they had actively been involved in sensitisation and advocacy activities. Monitoring/Supervision (3.0): CDDs are visited as need arises. Data collected during field visits include number of people treated, number of tablets used, and adverse effects information. Most of the time, data is collected and sent at the end of a given activity. Data collected are directly transmitted to the HD level through the ‘usual’ government system. However, copies are not kept at the HC level though the reports were seen in the district office. Each community where Mectizan® is being distributed is visited 2-3 times during the distribution campaign. In most FLHF visited supervision visits were said to be integrated as staffs took advantage of being in the villages to also look at other health programmes. Visits sometimes included households’ visits and in most cases, education and mobilisation of community members were included in visit agenda. However, there was very little written evidence. Problems identified during visits such as lack of incentives to CDDs and cases of drug refusals were treated appropriately, and when necessary referred to the community representatives. There is not much record keeping taking place. Most problems are dealt with at the community level except those of CDD motivation and financial support to CDTI implementation. There is very little or no feedback given to the community, CDDs are commended verbally and staffs reported having motivated them personally by giving them their personal money, giving them drinks or food. 23 Mectizan Supply (3.0): Ordering of Mectizan® was carried out based on previous year’s treatment data. Order are made by CBH and in most cases, delivery was timely. There were no reports of shortages. Mectizan is collected at the HD level during training. It is stored separately from the other drugs and distributed to CDDs during their training. The system is effective, simple and uncomplicated. Documents showing stock control were sometimes seen. Training/HSAM (1.5): The staff trained CDDs yearly in a routine manner. Generally training was done at the health facility in one day. Training material was very rarely seen (1/6). Health facility staffs carry out sensitization activities in a routine manner. There was neither planning nor report of HSAM activity. There was no evidence that HSAM activity had been effective and had led to action. Financial Resources (1.0): The last round of treatment: Budgeting was not carried out at the FLHF. Health personnel at this level had no idea about budget breakdown but sometimes choose not to include costs on the budget. Funding disbursed is barely enough to enable carry out essential CDTI activities. It was not clear as to how much various sources have contributed and staffs could not state how much CDTI activities have costed. In most cases there was Government contribution to the implementation of CDTI but amount could not be clearly estimated. Transport/Other Material Resources (2.0): HA staffs lacked transport materials. The ones they had were either private materials or completely not adapted given the distances to cover and the terrain. In spite of these problems CDTI activities have not been disrupted. Staffs either use their private means of transport (pay public transport or trek). Although there appears to be no plans for the provision of transportation at this level, staffs entirely depended on the HD levels to advocate for transport materials to be provided. They seemed to be discouraged following numerous unsuccessful attempts to see the problem solved. But with the ongoing elaboration of Health District Development Plans, there is possibility that government will make transport means available. There was generally lack of training material. Posters were seen pasted in all health facilities. Flip charts were not seen. Human Resources (2.0): Staffs at the HA received routine training at the beginning of each campaign. They were generally knowledgeable and skilled in most areas of CDTI activities. There is a general lack of skills in areas of planning and HSAM. Staffs were well knowledgeable in the area of management of side effects. Though stable, staffing at the FLHF is inadequate, and when transfers occur, immediate training of new unskilled staff is undertaken. Staff training in the private FLHF visited is only reserved for focal person 24 Coverage (3.0): The geographical coverage rate was100% in all the health areas visited. The therapeutic coverage rate was more than 65% in all villages. Fig 3 : Sustainability at front line health facility level 3.3.2. Recommendations at FLHF level Recommendations by Group of Indicators Implementation Planning 1.All health center staffs should elaborate integrated work plan including CDTI, plans should include costs Priority: HIGH Indicator of success: integrated plan including CDTI with cost of activities available Who to take action: chief of centre Deadline for completion: may 2008 Integration 2. Health center staffs should have integrated written plans and reports showing that integration of activities is effective. Priority: HIGH Indicator of success: integrated written plans and integrated reports available Who to take action: chief of centre Deadline for completion: june 2008 Leadership 3. The management team at health center should fully engage in CDTI implimentation Priority: MEDIUM Indicator of success: Who to take action: management team at this level is engaged is CDTI activities Deadline for completion: may 2008 4. Staff should empower Priority: HIGH 3 3 3 3 3 1,5 1 2 2 3 0 0,5 1 1,5 2 2,5 3 3,5 Pla nn ing Int eg rat ion Le ad ers hip Mo nit ori ng /su pe rv Me cti za n Tr ain ing /H SA M Fin an ce Tr an sp ort /ot he r m at res Hu ma n r es ou rce s Co ve rag e group of indicators av er ag e w ei gh t ( /4 ) 25 community leaders so as to anable them take ownership of the programme. Indicator of success: community leaders take ownership of the programme. Who to take action: chief of centre Deadline for completion: june 2008 Supervision/Monitoring 5. Health staffs should ensure that activity reports are written and copies are kept at own level for action and future reference. Priority: MEDIUM Indicator of success: all activitiy reports available Who to take action: chief of centre Deadline for completion: may 2008 6. Health staffs should have integrated plans for supervision written out. Priority: HIGH Indicator of success: integrated plans for supervision exist Who to take action: chief of centre Deadline for completion: may 2008 7. Health staffs should ensure that feedback is given appropriately to the communities. Priority: MEDIUM Indicator of success: feedback is given to the communities. Who to take action: chief of centre, community leaders Deadline for completion: may 2008 Mectizan supply 8. Health staffs should set up a stock form for Mectizan management. Priority: HIGH Indicator of success: stock form for Mectizan exist Who to take action: chief of centre Deadline for completion: may 2008 Training /HSAM 9. The chief of centre once trained should involve all other staffs so as to insure that activities are not discontined in case of his/her absence. Priority: MEDIUM Indicator of success: all other staffs are trained on CDTI by chef of center Who to take action: chief of centre Deadline for completion: next training of chef of center in 2008 10. Health staff should organise targed training that is train only those CDDs in need and/or only on relevant subjects. Priority: HIGH Indicator of success: targed training is organise every year. Who to take action: chief of centre Deadline for completion: from may 2008 11. Health staffs should effectively plan, implement, follow-up, and make written reports on HSAM activities. Priority: HIGH Indicator of success: - written reports on HSAM activities available. - written plan for HSAM exist Who to take action: chief of centre Deadline for completion: - a week after activity (for the report) – june 2008 (for the plan) Financial Resources 12. Health staffs should document health centres’ contribution to CDTI implementation. Priority: HIGH Indicator of success: document of contributions to CDTI available Who to take action: chief of centre Deadline for completion: july 2008 26 Coverage 13. Chiefs of centres and community leaders should ensure that all communities/villages have at least 65% TCR annually. Priority: HIGH Indicator of success: all villages have at least 65% TCR annually Who to take action: COC, community leaders Deadline for completion: from next distribution (2008) 3.4 Community Level 3.4.1. Findings and discussions Planning (3.0): In about 60% of the communities visited, CDDs met with village/quarter leaders after training to discuss when to distribute Mectizan®. CDDs then systematically organized door-to-door visits to households to take census and distribute the drug at the same time. However, in 3 of the communities visited CDDs carried out census before distribution. In most cases (75%), CDDs reported problems to the Chief of Centre to help in problem solving such as refusals, absenteeism, but not to the quarter/village heads as they said no help came from them. In few cases (30%), problems were reported to the community leaders, this usually happened when CDDs fed-back or when leaders asked to know how the distribution was going. Leadership (2.5): In about 40% of communities visited, CDDs and community leaders reported that leaders organised sensitization activities to try to convince reluctant people. Apart from that, no or very little initiative was reported from community leaders with regards to Mectizan distribution. As feed-back was very rare, community leaders had very little knowledge of the problems encountered by CDDs and it was generally believed that CDDs were heavily sponsored from Government and other external sources. Problems were generally met with by CDDs themselves and in some cases, the Chiefs of Posts. In 50% of communities visited, CDDs were directly selected from the health committee by the Chiefs of posts; in 25% by communities and 25% by village/quarter leaders. As for the timing of distribution, in some instances, quarter/village heads were consulted and in most cases, CDDs met among themselves and decided on the timing. Distribution was mainly carried out door-to-door as instructed during the training. There was no report of community common initiative to support CDDs. There were few reports of CSM Mectizan® is very much valued among the community members. They all tied Mectizan® with numerous advantages including de-worming and prevention of blindness. They asked to take it even more than once a year. They all expressed their readiness to take it as long as necessary. Monitoring (3.0): Once the distribution is over, about 70% of the CDDs prepared summary reports which included number of people censured, number of people treated, number of tablets used. The reports were given to the HC (about a week or two after the last day of distribution). The other 30% of CDDs brought the registers to the COP who prepared the report. In all cases, CDDs either trekked or used personal means to go to the COP 27 Mectizan (3.0): The CDDs are given tablets by the chief of health centre. Tablets received were more than enough to treat all those who were willing to take it. In all cases, CDDs were not always conversant with the need calculation. Some however said it was based on previous’ year treatment figures. Census data were good as no shortage was reported. CDDs usually collected the first batch of Mectizan® during their annual training session at the HC. When they ran out of stock, they either came back at the HC at their own costs or received more when the COC came down for supervision. No transport was arranged by the community for Mectizan® to be collected. HSAM (3.0): When carrying out door-to-door visits, CDDs engaged in health education. There were no precise plans to systematically educate the population and community authorities were sometimes actively involved in some HSAM activities. It was sometimes reported that church announcements were made but this was mainly on COP or HD initiative. Financing (1.0): There was very little support from the community to the individuals providing services as it was strongly believed that they received payments from Government and other sources. Some CDDs however reported that food was given to them by some family members but, in general, there was no community common initiative to support CDDs. Human Resources (3.0): CDD ratio to population ranged from 1/60 to 1/211, but in one community it was 1/754. CDDs were mostly chosen among the health committee by COPs. CDDs worked at a range of 1 to 2.5km. All the CDDs met were very knowledgeable about their job including identification of SAEs and prompt referral. New CDDs were coupled with old ones and work was usually shared among them. CDDs received one day training at the beginning of each year. Coverage (3.0): Out of the 12 communities that were sampled, 8(67%), had a therapeutic coverage >65% in 2007. Meanwhile at project level, 75% of the communities had >65% therapeutic coverage in 2007. In 2006 and 2005 respectively, 69% and 80% of the communities had >65% therapeutic coverage. 28 Fig 4: sustainability at community level 3 2,5 3 3 3 1 3 4 0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 pla nn ing lea de rsh ip mo nit ori ng me cti za n HS AM fin an cia l re so urc es hu ma n co ve rag e group of indicators av er ag e w ei gh t(/ 4) 29 3.4.2. Recommendations at the Community Level: Recommendations by Group of Indicators Implementation Planning 1.CDDs should plan and carry out distribution in close collaboration with village/quarter leaders Priority: HIGH Indicator of success: leaders are closely involved in planning and distribution of Mectizan. Who to take action: CDDs Deadline for completion: from next distribution of Mectizan(2008) 2. Community as a whole should look for local ways to facilitate CDDs work and CDDs motivation Priority: HIGH Indicator of success:CDDs are motivated and their work facilitated Who to take action: chief of centre, community leaders Deadline for completion: from next distribution of Mectizan(2008) Supervision/Monitoring 3. Community should engage in CSM as it will help them better understand CDTI problems and encorage ownership Priority: HIGH Indicator of success: CSM is conducted annually Who to take action: community leaders Deadline for completion: june 2008 Training /HSAM 4. Community leaders and CDDs should plan to effectively educate the population Priority: HIGH Indicator of success: population is educated according to a plan. Who to take action: CDDs/community leaders Deadline for completion: june 2008 Financial Resources 5. Community leaders should organise their communities to devise means to provide funds to CDTI activities such as transport to collect Mectizan, material purshase, incentives for CDDs. Priority: HIGH Indicator of success: Communityies provide funds for CDTI activities Who to take action: community leaders Deadline for completion: june 2008 Transport/other material 6. Community should provide transport for the collection of Mectizan when necessary. Priority: MEDIUM Indicator of success: transport provided by communities Who to take action: community leaders Deadline for completion: may 2008 Human resources 7. Community should sellect CDDs suited for the job after the express concern of the individual. Priority: HIGH Indicator of success: Motivated CDDs sellected for distribution Who to take action: community leaders Deadline for completion: as need arised 30 Coverage 8. CDDs and community leaders should ensure that communties/villages have at least 65% TCR annually. Priority: HIGH Indicator of success: all villages have at least 65% TCR annualy Who to take action: chief of centre/CDDs Deadline for completion: from next distribution (2008) 4. GRADING THE WHOLE PROJECT 4.1. Qualitative judgement The qualitatives judgment is based on the 7 aspects and 5 critical elements of sustainability: 4.1.1. Aspects of sustainability ASPECT JUDGEMENT Integration Potentially blocking sustainability Resources Potentially blocking sustainability Efficiency Slightly helping sustainability Simplicity Highly helping sustainability Health staff acceptance (Attitude of the health staff) Very much helping sustainability Community ownership Potentially helping sustainability Effectiveness Highly helping sustainability Integration: Potentially blocking sustainability The CDTI activities form part of the minimum package activities at operational level (district and health areas). At the provincial level it is no so evident. Transport means at all levels irrespective of the source, if functional are used for CDTI and other health programmes. Documentation (work plans and reports of activities like integrated supervision) at all levels are necessary to enhance the integration. Resources (Human, financial and transport/material): Though staffs are stable, and committed they are inadequate quantitatively especially at the health area level.So far, the bulk of funds for the running of CDTI project still comes from external funding bodies. Unfortunately government contribution made at all levels cannot be assessed. There is adequate transportation means at the provincial level, but at the operational level (district and health area) there is very little or no transport. Staffs are coping with their personal means or use public transport. Efficiency: Although there is some degree of efficiency noticed in the running of most CDTI activities, a lot is still to be done in the areas of planning, integration,monitoring/ supervision, training and HSAM. Since the activities are planned but not costed, a lot still needs to be done to run the program cost effectively. 31 Simplicity: It was observed that Mectizan is managed effectively and simply from the provincial to the community level. Funds designated to the project are disbursed with respect to the budget lines through simple and uncomplicated procedure. Health staff acceptance (Attitude of the health staff) Generally, it was observed that the health staffs at all levels are motivated to carry out CDTI activities. However, the lack of means of transport at the health district and FLHF levels to facilitate their work can be a cause for de-motivation. Community ownership: Planning at this level is generally initiated by the health staff who requests the village/quarter heads to select CDDs for training. After training the CDDs meet with the village authorities to discuss when to distribute mectizan. Following the discussions the evaluators had with village/quarter heads in communities visited they were willing and even waiting to appropriate the project but they lack understanding of the ownership policy. Effectiveness: All the health districts sampled had satisfactory geographical coverage rates (100% from 2004 to 2007).Therapeutic coverage:out of the 3 HDs that were sampled, none had a therapeutic coverage rate <65% in 2007. Meanwhile out of the 17 HDs that were contained in the project area visited 16 (94%) had a TCR >65% in 2007. In 2006 and 2005 respectively, 94% and 82% of the HDs had a TCR>65%. 4.1.2. Critical elements of sustainability: Money: So far the project activities have not been blocked because of lack of finances. However most of the funding has been from APOC and NGDO (SSI). Transport: Though there is no particular plan to replace existing materials, the ongoing nationwide elaboration of Health District Development Plans gives room to progressively replace old materials. It is hope that with this plan of the government vehicles will be available for minimum essential activities. Supervision: Though supervision is not targeted, there is provision for supervision because funds for other programmes are used for integrated supervision. Mectizan supply: The supply system is simple, efficient and effective. 32 Political commitment: Onchocerciasis control is one of the priority government programmes in Disease Control. 4.1.2. Quantitative judgement : Quantitative judgement was based on grades given to individual indicators. Table 2 : Scores for group of sustainability indicators at the four levels Level Group of Indicators Provincial Health Districts FLHF Community Average Planning 1.0 2.0 3.0 3.0 2.3 Integration 2.0 1.5 3.0 - 2.2 Leadership/ownership 2.0 3.0 3.0 2.5 2.6 Monitoring/supervision 2.5 2.5 3.0 3.0 2.8 Mectizan 3.5 3.5 3.0 3.0 3.3 Training/HSAM 2.5 1.5 1.5 3.0 2.1 Financing 1.0 1.0 1.0 1.0 1.0 Transport/other materials 3.5 2.0 2.0 - 2.5 Human ressources 2.0 3.0 2.0 3.0 2.5 Coverage 3.5 3.5 3.0 3.0 3.3 Average 2.4 2.4 2.6 2.7 Average for the whole Project 2.5 5. CONCLUSION Two aspects of sustainability (integration and resources) over seven were not helping the programme. Only one critical element (money) was not sufficient. Average score of indicators at the four levels was 2.5 over 4. In accordance with the evaluation guidelines, the evaluators’ team concluded that the North West CDTI Project was making satisfactory progress towards sustainability. 33 ANNEXES 34 ANNEXE 1 FEEDBACK AND PLANNING MEETING FOR PROVINCIAL AND DISTRICT LEVELS BAMENDA, 30TH MARCH – 2ND APRIL 2008 AGENDA DAY 1: 30/03/08 ‘S/No ACTIVITY TIME FACILITATOR 1 Opening ceremony and welcome 9.00 – 9.10am PDPH-NW 2 Introducing the participants 9.10 – 9.20am PDPH-NW 3 Presentation: • The objective of the evaluation • What is sustainability • The Evaluation Methodology 9.20 – 9:30am Julius Gaba 4 Presentation of main Findings &discussions • Community level • Health area level • Health District level • Provincial level 09:30 – 11:30am - C. Evini - M. Akoh Arrey - A. Nkeng - P.Oponde 5 Coffee break 11:30 – 12:00 6 Group work (in 2 groups) 1. SWOT analysis – ‘What is the situation regarding sustainability in our project?’ Group 1: The community and HA levels Group 2: The Health District and Provincial levels 12:00 – 1:30pm Evaluation team 7 Lunch break 1:30 – 2:30pm 8 Presentation of group work followed by plenary discussion 2:30 – 3:30pm Evaluation team 9 2. ‘What changes do we need to make in order to address the weaknesses and threats regarding sustainability of our project considering the strengths and opportunities identified in the SWOT analysis?’ Group 1: Planning, Monitoring and Supervision Group 2: Finances, Group 4: Training and HSAM Group 3: Transport, Mectizan supply Instructions: focus on trimming expenses, mobilising, resources, integration 3:30 – 4:30pm Evaluation team 10 Presentation of group work followed by plenary discussions 4:30 – 4:30pm Evaluation team 11 Summary of day’s work Mrs. Akoh-Arrey 12 House keeping matters 35 DAY 2 (PLANNING): 01/04/08 S/No ACTIVITY TIME FACILITATOR 1 Introduction of the chair 9.00 – 9.10am 2 Summary of previous day’s workshop proceedings 9.10 – 9:30am C. Evini 3 Presentation of group work followed by plenary discussions 9:30 – 11.30am A. Nkeng 4 Coffee break 11:30 –12:00am 5 Group work: What minimum resources do we need to make our programme sustainable in the course of the next 3 years? Issues to consider: Money, Human resources, Transport and equipment/materials • Group 1: Provincial level • Group 2: Health District level • Group 3: Health Area level • Group 4: Community level 12:00 – 1:30pm Evaluation team 6 Lunch break 1:30 – 2:30pm 7 Presentation of group works followed by plenary discussion 2:30 – 4:30pm P. Oponde 8 Summary of the day’s work A. Nkeng 11 House keeping matter DAY 3 (Planning cont’d) : 02/04/08 S/No ACTIVITY TIME FACILITATOR 1 Introduction of the chair 9:00 – 9:15 2 Summary of previous days’ workshop proceedings 9:15 – 9:30 C. Evini 3 Presentation of the format of the plan 9:30 - 10: 00 Evaluation team 4 Coffee break 10:00 - 10:30 5 Group work (in Provincial and HDs’ groups) Development of sustainability plans for 2008 - 2010 • Each group to compile a tabulated plan (what is to be done, why; by whom; when;, indicator;, cost) • The plan must fit into available resources 10:30 – 12:30 Evaluation team 6 Lunch break 12:30 – 1:30PM 7 Presentation of group work, followed by plenary discussion 1:30 – 3:30PM J. Gaba 8 Plenary discussion: Making a master ‘plan for sustainability’ for the project, for the coming year 3:30 – 4:00PM J. Gaba C. Evini 9 Plenary discussion: • Practical steps to finalise the ‘plan for sustainability’ • Review/Endorsement of Plans 4:00 – 4:30PM J. Gaba C. Evini 10 The Way Forward (What is going to happen to our plans?) 4:30 – 5:00PM Evaluation team 11 Closing remarks 5:00PM PDPH-NW 36 ANNEXE 2 REPORT OF FEEDBACK AND PLANNING WORKSHOP Report of day 1 activities. Rapporteurs: - Dr Che Soh Kingsley – Provincial Delegation of Public Health – North West Province. - Mr Nnegue Jean Jacques – CBAF Ndu - Mr. Titus Chi – CBAF Mbengwi The meeting started at 9 am with the registration of the various participants and distribution of the didactic materials. At 10:38 am, the provincial delegate for public health in the NWP after an opening prayer from the chief of Nkambe Urban IHC, proceeded with an official opening of the workshop. In his opening address, the PDPH –NWP welcomed the evaluators of the CDTI project in the NWP and called on participants to share with the evaluation results. After thanking the organizers of the evaluation, he explained that the evaluation was done in 3 of our eighteen health districts chosen by the Random sampling technique, and saw the meeting as an opportunity to use the outputs of the evaluation as input for an immediate planning of CDTI activities for 2008. He regretted that he could not precise the contribution of the government in the implementation of CDTI activities but however hoped that at the end of this meeting which he termed as timely, participants will come out with a practicable and sustainability plan of action geared towards sustainability of the CDTI program. He added as an opportunity, the realization of the multi annual consolidated Health District plan which is currently being elaborated within the frame work of SQI with subsequent funding through a SWAP. The PDPH ended his word of welcome with a call on participants to be effectively present and participate actively during the meeting. He then proceeded with the coordination of the introduction of evaluators and participants. The evaluators included; Dr Gaba julius (Head of the evaluation team) and Dr Akoh-Arrey Matilda, Mr. Evini Cyrille, Mr. Oponde Peter, Dr. Nkeng Alexandra as members. The next activity was a presentation of the objectives of the evaluation, the methodology used and the grading of the various indicators by the head of the evaluation team, Dr Gaba In his presentation, he defined CDTI activities as sustainable, if they continue to function effectively for the foreseeable future, with high treatment average, integrated into available health care services, with strong community ownership, using resources mobilized by the community and the government. In the methodology, 3 health districts, 6 health areas / Front Line Health Facility and 12 communities were sampled. The evaluation team was then divided into 3 sub-teams each of them evaluating a district. Evaluation instruments were designed differently for evaluation at 4 levels; the provincial level, the health district level, the FLHF / Health area level and the community level. There were 10 indicators for evaluation at the provincial, health district and health area levels and 8 indicators at the community level. The grading of the indicators ranged from 0 when an indicator doesn’t support sustainability at all, to 4 when an indicator fully supports project sustainability. All the indicators evaluated included planning, leadership, supervision and monitoring, mectizan, training and HSAM, integration of support activities, financial resources, transport and material resources, and coverage (GCR and TCR). After this presentation, Dr Akoh-Arrey Matilda proceeded with a presentation of the main findings of the evaluation at the provincial level. According to these findings, the 37 provincial level scored ¼ for planning, 2/4 for integration, 2/4 for leadership, 2.5/4 for monitoring / supervision, 4/4 for mectizan, 2.5/4 for training / HSAM, ¼ for financial resources, 3.5/4 for transport / other material resources, 2/4 for human resources and 4/4 for coverage. Following these results, the following recommendations were made: - The need among others for elaboration of an overall plan with costs, the limitation of planned activities to those carried out at the provincial level, the involvement of all stakeholders in appraisal and advocacy meetings, availability of integrated written plan to staff, the appropriate documentation, more involvement of the NWPSFH in mectizan management, of HSAM activities, reporting of HSAM activities, availability of appropriate and adequate training material for HSAM activities to the entire province, allocation and effective disbursement of government funds for CDTI activities and the documentation of government contribution for CDTI. This was followed by a discussion in which the PDPH appreciated the elaborate work done by the evaluators and admitted that all what was presented reflected reality at his level, and called on the members of the provincial team to make up from slumber while calling on the evaluators to the provincial level on how suitable advice these recommendation could be implemented. In this light, it was advised that CDTI plan of activities be costed to precise the financial contribution of the government and the community, and that emphasis should be laid on the precision of the fraction of government credits allocated for CDTI activities.The low GCR was attributed to a problem of accessibility rather than a problem of mapping. It was explained a provincial coordinator was needed because his role goes beyond data synthesis to other managerial aspects of the program It was announced that APOC financial assistance to the NWP has been suspended due to lateness at poor quality of financial justifications and therefore timely and quality financial reporting when the suspension has been uplifted was recommended. The discussions were followed by a coffee break at about 12 noon. The meeting resumed at 1:15 with a presentation of the main findings of the evaluation at the District level, by Dr. Nkeng A. According to this presentation, the Districts evaluated scored 2/4 in planning, 1.5/4 in integration, 2.5/4 in leadership, 2.5/4 in supervision, 3.5/4 in mectizan management, 1.5/4 in Training/HSAM, ¼ in financial resources, ¾ in human resources and 3.5/4 in coverage. Following these results, a set of recommendations were proposed to the district level by the evaluation team. This presentation was followed by a discussion during which it was explained that, a stratified random sampling method, which is appropriate for generalisability of the results especially in the context of this evaluation, was used. It was also commented that the use of a planning matrix was not a bad idea but that it was supposed to serve as a guide for planning and activities defined according to the level of planning and that a costed plan was necessary to know how to make up for the gaps. It was added that other forums could be used in training like supervision and coordination meetings given the problem of financial resources and the prospects of sustainability. Then came the presentation of the main findings of the evaluation at the health area level by Mr. Oponde Peter. According to the presenter, the health areas scored ¾ in planning, ¾ in integration, ¾ in leadership, ¾ in monitoring/supervision, ¾ in mectizan supply, 1.5/4 in 38 training/HSAM, ¼ in financial resources, 2/4 in human resources and ¾ in coverage. These led to a series of recommendations for the has. Due to time constrains, this was immediately followed by the presentation of the main findings of the evaluation at the community level by Mr. Evini. The results of the evaluation show that the community scored ¾ in planning, 2.5/4 in leadership, ¾ in monitoring, ¾ in mectizan supply, ¾ in HSAM, ¼ in financing, ¾ in human resources, and ¾ in coverage. These results were also followed by a series of recommendations from the evaluation team. The overall average scores were therefore HD level - 2.4 HA level – 2.5 Community level – 2.7 Average Provincial for all levels = 2.5 Presenting the critical elements of sustainability which include integration, resources, efficiency, simplicity, health staff acceptance community ownership and effectiveness, Mr. Evini noted in the overall grading of the project that the team found 7 elements of sustainability helping or not blocking the later. The team therefore concluded that the NW CDTI project is making satisfaction progress towards sustainability. After all presentations, the participants were divided into 4 groups. Group 1, lead by DMO of Tubah, was assigned to work on the SWOT analysis for the Community level. Group 2, lead by the DMO of Wum to work on the SWOT analysis of the District level, Group 3, lead by the DMO of Nkambe to work on the SWOT analysis for the HA level, Group 4, lead by the DMO of Bali to work on the SWOT analysis for the provincial level. Activities for day 1 ended at 6:30 pm with all groups asked to prepare their presentations for the next day. Report Of Day 2 Activities. Rapporteurs: - Dr Che Soh Kingsley – Provincial Delegation of Public Health – North West Province. - Mr. Alenwi Nfornuh – CBH - Wum - Mr. Nkwain Emmanuel N. The meeting started at 8:00 with the arrival of the participants and facilitators. The various group members assembled in same order to complete the previous days work on the SWOT analysis, regarding sustainability of the CDTI project at the various levels (Provincial, District, Heal Area and community). At 8:50, the meeting proper startexdc with a word of prayer by one of the participants. Hightlights of Day 1 activities was presented, followed by an introduction of the methodology of the Day 2’s activities by one facilitator (C. Evini). 39 This was immediately followed by the presentation of the minuts of the day’s activies by Dr. Che Kingsley. Classification and corrections of the minutes were made and then minutes adopted. At 9:40 am, we proceeded to the presentation of the group work on SWOT Analysis regarding sustainability of the CDTI project, where Group I – Provincial level Group II – Health District Levwel Group III – Health Area level Group IV – community level. Each presentation at the different levels was followed by a plenary discussion whereby clarifications and corrections were made on the spot. (See copies of SWOT –Analysis of the present situations regarding sustainability at the different levels). At the end of the presentation, one facilitator recalled that the exercise was to help us at our different levels to identify SWOTs which will enable us to effectively plan, to sustain the CDTI project. Coffee break from 11:05 am for 30 mins. The various groups were maintained, to commence the next group work on the changes we need to make in order to address the weakness and threats gregading sustainability of the project, considering the strength and the opportunity identified in the SWOT analysis, the indicators were shared out as follows: GroupI – Training /HSAM Group II – Financial Resources Group III – Planniing and monitoring Group IV – Transport/Mectizan supply. While focusing on trimming expenses, mobilizing resources and integratrion, groups were expected to identify weaknesses and threats based on SWOT analysis and come out with solutions. The exercise lasted for about 2 hours and presentations followed thereafter as from 2:10 pm following same order. Classification and corrections were done on the spot after plenary discussions. (See copies of solutions to our projects weaknesses in relation to sustainability). Lunch Break 4pm This was closely followed by the 3rd group work, which was to assess minimum resources needed to make the program sustainable for the next 3years at all the levels. Issues that were to be considered as resources were: Money, Human resources, Transport and equipment/material and was shared out amongst the 4 groups. Group I – Provincial level Group II – Health District Group III – Health Area 40 Group IV – Community level. The exercise lasted 11/2 hr and presentations commenced at about 5:45 pm. The resources assessed at the different levels by each group were presented and discussions, comments, suggestions and amendments done on the spot. (see copies of minimum resources) Presentaions and plenary discussions till 7 pm. The facilitators thanked the participants for their hardwork and commitment then declared the day’s activities closed. Report Of Day 3 Activities. Rapporteurs: - Dr. Che Soh Kingsley – PDPH, Bamenda - CBH – Bali - Ngem bed – Bali (DHS) The day’s activities started at 8 am with the arrival of participants and facilitators. This was then followed by a word of prayer from one of the participants officially opening the day at about 8:58 am. The reporter for day 2 presented their report which was precise and concise thus the report was confirmed and appreciated. The facilitator then came up to give a summary of the previous days activities (day 1 & 2). To evaluate the degree of understanding of the workshop from participants, this was in the form of brainstorming exercise where participants were chosen at random with corresponding questions. This was then followed by the distribution of working tools. Thus: - SWOT analysis of all the ten indicators for sustainability at all levels - What changes were needed to do in order to address the weaknesses and threats with respect to all the indicators. - Our future resources at all levels. The presentation of the format and features of the plan was done by one of the facilitator who also distributed out the CDTI plan for sustainability. The participants then split into groups according to different districts and provincial level to draw up a sustainability plan for 2009 (integrated) by coffee break which lasted for 30 minutes. Group work continued after the coffee during which participants were called up individual (district teams) to the ONCHO unit to sign their transports and per diem. At 2:45, group work was stopped for presentation of group work and plenary discussions. The provincial level group started the presentation. This was followed by a number of corrections and remarks. 41 The next group presentation came up from Ndu and Mbengwi. Both presentations were followed by remarks, corrections and appreciations. This was closely followed by lunch break that lasted from 4:54 pm – 5:48. The last presentation of group work came up from Wum Health District. This like others had corrections and remarks. The team leader appreciated all for the good work done but concluded that the districts should go back hold meetings with the H/A teams to come out with a consolidated plan. That is from H/A level, District levels to the Provincial level. He then gave 3 weeks for the plan to be well drawn for the 3 years. 2009, 2010 & 2011, respectively. He also distributed a guide to help come out with the 3 years development plans. Aspects in the guide were explained as the paper (guide) given was in French paying particular attentions to the items not to be funded by APOC again from 2009 such as CDD training, vehicle running cost and small equipment. The other items that APOC will support were also readout and also expected support. He then added that this plan needed to be endosed by the local authorities (D.Os, Mayors etc) and should be signed by 2 or 3 persons and not only tables, justifying reasons for the plan. Thus the plan should be a whole document thus must be a 3 year plan, filed in within 3 weeks. One participant asked if the plans for the HA was going to be separate from the District plan. After a lot of deliberations it was agreed that the plans should be consolidated and not separate. So in drawing the plan, note the inputs of the H/A i.e. do the briefing of H/C teams. The précised date for the reports to be sent to the Province was on or before 30th April in hard and soft copy. Then came up the closing ceremony from the team leader who thanked all participants who have been very keen and participatory. He also said we should take this very serious and implement and we have learnt not to be as if we have wasted time. 42 ANNEXE 3 CONTACTS OF THE EVALUATORS Doctor Matilda Akoh-Arrey Provincial Delegation of Public Health SW Province, Buea (Cameroon) e-mail:echi54@hotmail.com echi_54@yahoo.com cell. Phone: (237)77748312 Doctor Alexander Nkeng District Health Service-Toimbel SW Province, Buea(Cameroon) e-mail: nkengalex@yahoo.com cell. Phone : (237)7779558 M. Peter Oponde Provincial Delegation of Public Health SW Province, Buea (Cameroon) e-mail: opondepeter@yahoo.com cell.phone: (237) 77734745 Doctor Julius Gaba P.O Box: 04 BP 872 Cotonou ( Rep. of Benin) e-mail: jecgaba@yahoo.fr cell. phone: (229) 95 05 35 81 Tel : (229) 2136 08 74 M. Cyrille Evini SSI Yaounde (Cameroon) e-mail: cevini@sightsavers.org cyrileevini@yahoo.fr cell. Phone: (237)77710721