Report of the external evaluation October 2005

World Health Organization African Programme for Onchocerciasis Control (APoc) Evaluators: Harald Burmeister Patrick Kayembe Kalambayi Georges Koulischer Olademeji Oladepo Bernard A Philippon Eleuther Tarimo (Chairman) October 2005 Report of the External Evaluation JAF11.10 JAFl1.10 Page i Table of contents Glossary Acronyms Acknowledgements Executive summary 1. Introduction .............. 2. The evaluation process....... 3. Findings and main recommendations 3.1 Implementation of the CDTI strategy 3. 1 . 1 Community directed treatment .. . . 3.1.2 Drug procurement and delivery 3.1.3 Capacitybuilding...... 3.1.4 Monitoring and supervision............. 3.1.5 Severe Adverse Effects (SAE) 3.1.6 Programmes inconflicVpost-conflict areas........... 3.L.7 Treatment coverage 3.1.8 Evaluation and sustainability............ 3.1.9 Adequacy of impact assessment. 3.1.10 Operational research 3.1.1I Implementation of projects: achievements and delays 3.1.12 Vector elimination projects in APOC 3.2 Programme management................. 3.2.1 Programmegovernance 3.2.2 Role of CSA 3.2.3 Role of TCC 3.2.4 APOC headquarters office: adequacy and effectiveness of programme management, financial management, and staffi ng................. 3.2.5 APOCfinancialmanagement.................. 3.2.6 APOC staff management 3.2.7 The country Ievel........... 3.3 Integration of CDTI into overall health services 4. Partnership................ 4.1 The various partners in synergy at all levels...... 4.2 Theinternational level........... 4.3 The national level 5. Conclusions and the Way Forward Annex 1: Terms of Reference (Document JAF10.10, October 2004) Annex 2: APOC projects approved as of July 2005...... Annex 3: Analysis of treatment results since APOC's inception (data as of October 11, 2005).... ,.11 iii ,iv .1 .8 .9 10 .10 .10 .13 .t4 .16 .17 .18 .19 .20 .21 3l 32 33 34 5/ 39 39 39 39 43 46 50 51 53Annex 4: APOC organization chart...... JAFl1.10 Page ii Glossary Geoeraphical coverage: Therapeutic coveraqe: Ultimate Treatment Goal (UTG): Participating countries : The Donor Community: Phase 1: Phase 2: Phasing out period: The programme: Number of communities treated in a given year over the total number of meso/hyper-endemic communities as identified by REMO in the project area (this should be expressed as a percentage). Number of people treated in a given year over the total population (this should be expressed as a percentage). Calculated as the maximum number of people to be treated annually in meso/hyper endemic areas within the project area, ultimately to be reached when the project has reached fulI geographic coverage (normally the project should be expected to reach the UTG at the end of the 3'd year of the project). Countries conducting onchocerciasis control progmmmes in partnership with APOC Bodies donating funds for APOC and its projects and activities The period of APOC's work from 1996 to 2001 The period of APOC's work from 2002 to 2007 The period of APOC's work from 2008 to 201.0" APOC as a whole JAF11. i0 Page iii Acronyms AFRO APOC CDTI CDD CSA CSM DEC EPI FLHF FLHWs HSAM IEC JAF LGA LF MACROFIL MDP MoH NGDO NOCP NOTF OCP PHC SAE SIZ TCC RAPLOA REMO TDR African Regional Office of WHO African Programme for Onchocerciasis Control Community Directed Treatment with Ivermectin Community Directed Distributor of lvermectin Committee of Sponsoring Agencies Community Self Monitoring Diethyl Carbamazine Expanded Programme of Immunization Frontline Health Facility Frontline Health Workers Health Education, Sensitization, Advocacy and Mobilization Information, Education, Communication Joint Action Forum Local Government Area Lymphatic Filariasis Macrofilaricidal Drugs for Onchocerciasis and Lymphatic Filariasis Mectizan Donation Programme Ministry of Health Non-Governmental Development Organization National Onchocerciasis Control Programme National Onchocerciasis Task Force Onchocerciasis Control Programme Primary Health Care Severe Adverse Effect Special Intervention Zones Technical Consultative Committee (of APOC) Rapid Assessment of Loaisis Rapid Epidemiological Mapping of Onchocerciasis UNDP/World Bank/WHO Special Programme on Research and Training Diseases World Health Organization in Tropical wHo JAFl l.l0 Page iv Acknowledgements The External Evaluation Team greatly appreciates the cooperation received from all partners actively involved in taking APOC to completion. Information received in thought-provoking exchanges with groups or individuals, supported by a wealth of documentation, have been the determining factors in shaping the evaluation process. The team wishes in particular to express gratitude to: o Staff of the Participating countries visited; notably, in each country, the Ministers of Health and the staff of national, district and local-level health services, and members of the National Onchocerciasis Task Force (NOTF) o Management and staff at APOC Headquarters, in particular Dr A. S6k6t6li, Director of APOC; Dr L. Yam6ogo, Coordinator, Office of the Director; Dr U. Amazigo, Dr M. Noma, and numerous other resource persons o AFRO, the WHO representatives and country office staff of the countries visited . Members of the NGDO Coordination Group for Onchocerciasis Control, and the NGDO Coordinator at WHO Headquarters, Dr T. Ukety . Staff of the Special Programme on Research and Training in Tropical Diseases (TDR) of WHO, Geneva . At the World Bank in Washington, DC, Dr O. Bangoura, Head of the Onchocerciasis Coordination Unit, Mr B. Benton, and Dr B. Liese o Members of the Committee of SponsoringAgencies (CSA) of APOC o Members of the Technical Consultative Committee (TCC) of APOC o Mr. A. Daribi, Onchocerciasis Liaison Office, WHO, Geneva. JAF11.10 Page 1 Executive summary A. Overview (a) This report assesses the ongoing African Programme for Onchocersiasis Control (APOC), and makes recommendations for further improving its effectiveness and efficiency. Specific objectives of APOC, which was launched in 1995 are: (a) to establish within a period of 12 to 15 years effective and self- sustainable Community Directed Treatment with Ivermectin (CDTI) throughout the endemic areas within the geographic scope of the prograrnme, and (b) if possible, in selected isolated foci, to eliminate the vector by using environmentally safe methods. (b) The programme became feasible in 1987 when ivermectin, the first safe drug for large-scale treatment of river blindness became available; and Merck & Co, lnc. generously agreed to donate the drug free of charge to affected countries for "as long as necessary." From the very beginning of APOC, its programme strategy relied on mobilization of communities to play a leading role in planning and overseeing their annual treatments with ivermectin; and on a robust partnership between communities and governments in affected countries, NGOs, International Organizations, donors and private sector groups active in these countries. (c) As part of its rigorous follow up of projects the Joint Action Forum (the APOC Governing Body), at its last session (Kinshasa, December 2004) established an external evaluation team of six people to assess progress of APOC towards meeting its objectives and whether these will be achieved, and to make recommendations on how best to sustain CDTI after 2010. The present evaluation follows a similar one carried out in 2000. Detailed Terms of Reference are contained in Annex l. (d) The evaluation team consulted widely and participated in meetings of APOC governing and technical bodies. The team visited five countries (Cameroon, Democratic Republic of Congo, Nigeria, Sudan, and Uganda) to assess the situation on the ground. The Countries and projects visited were carefullyselected to represent the situation of the 19 programme countries as much as possible. Thus projects selected for detailed assessment included those doing poorly, those nearing completion, and those in conflict situations (See Annex 2, APOC projects approved as of July 2005). Each country was visited by two members of the team. Discussions were held with MOH staff at different levels, NOTF, community leaders and CDDs. Reports of findings fbr each country were compiled. Country reports were analyzed together with information from other sources and consolidated into the report of the evaluation team. (e) The Team's main findings and recommendations are in Chapter 3 of the report, separately for technical and management aspects (respectively, sections 3.1 and 3.2)--followed by discussion of integration of programme activities into overall health services (section 3.3); partnerships (Chapter 4), and conclusions and "the way forward" after APOC (Chapter 5). These are based on extensive field visits and discussions with various partners and stakeholders of APOC. The report makes extensive reference to the 2000 evaluation findings as a benchmark to assess progress. In this Executive Summary, while key achievements are highlighted, emphasis is given to pending issues and rnajor recommendations. BJAFl1.10 Page 2 Findings and main recommendations Findings on Programme implementation (0 APOC's main strategy, i.e. CDTI, is based studies carried out by TDR. It addresses an important neglected problem that affects mostly the rural poor; and it does so in a relatively low-cost manner. Thus, CDTI has a sound moral as well as technical and economic appeal. The Team believes that APOC has a clear strategy, and has operationalised it through well-formulated objectives, plans and targets. (g) Mobilization of communities, procurement and delivery of drugs, and capacity building are areas of notable achievements though some pending issues remain. There is extensive satisfaction of communities with ivermectin treatment. Community assertiveness and demand for drugs is high-which is good for sustainability. A large number of well-trained community directed distributors of ivermectin (CDDs), the backbone of CDTI, has been trained and deployed. Extensive training and mobilization of key staff and leaders, for CDTI, at different levels of the health system has been carried out. Procurement, delivery and supply of Mectizan @ (the trade name for ivermectin) is increasingly being done through the countries' Ministries of Health (MoH) and their offices at various levels. Supplies are generally adequate. Pending issues include: inadequate mobilization of communities to increase the numbers of CDDs and to strengthen their feeling of ownership (of CDDs') and responsibility by providing them adequate incentives; lack of uniform policies by MoH and extemal agencies on provision of incentives to community volunteers within individual countries. (h) The 2000 evaluation was very concemed about the danger posed by severe adverse effects (SAEs) and the inadequacy of programme response. Since then, the team notes that exemplary work has been done by one MoH and the National Onchocerciasis Task Force (NOTF) in one country, and by APOC and the Mectizan Donation Programme (MDP) on the prevention and management of SAEs. The tools developed and experiences gained in the country, as well as new guidelines developed by APOC, are now being disseminated to other countries and projects. The challenge still remaining is one of implementation and continuous updating of the SAEs' prevention and mitigation programme. (i) The number of people treated with ivermectin and therapeutic coverage have increased steadily for most projects, except for those in conflict areas. Annex 3 shows that the number of persons treated yearly in Participating countries increased fiom 14.58 rnillion in 1997 to almost 22 million persons in 2000; and it has since increased to 28.45 million in 2002 and 31.31 million persons treated in2004. There has also been a steady increase in therapeutic coverage since the beginning of CDTI projects; such coverage has now exceeded 70"/, in 2004 in five countries. U) APOC has launched an "adapted CDTI" programme for countries in conflict/post-conflict situations. Annex 3 shows that DRC, has achieved a steady increase to 45.60/o therapeutic coverage in2004.In Sudan, Liberia and CAR, coverage has been generally lorv and variable. The CDTI programme has just started in Angola. The challenge in relation to countries in conflict includes adequate preparation for 'when peace comes'. In these countries, there has been considerable delay in implernentation of CDTI projects. As of March 2005, hve projects were still in the planning stage, and 24 projects had been approved but not launched. These projects will not have received basic APOC support by 2010, when APOC is planned to end. Other pending issues in the area of coverage include weak analysis of coverage data and occasional non-use of accepted definitions of therapeutic covcrage. JAFl1.10 Page 3 (k) To supplement routine monitoring, APOC has introduced independent monitoring and evaluations; and these have been successful in enhancing implementation of CDTL Out of 49 projects thus evaluated in 2002-2005, 35 (73%\ were judged to be moving satisfactorily towards sustainability. Pending issues include the synthesis and dissemination of lessons learned from evaluations carried out to date. (l) Impact assessment -to find out the extent to which ivermectin treatment prevents and or leads to regression of manifestations and damage caused by river blindness- has started at 14 sites in 9 countries. Pending issues include introduction of monitoring of human infection (some countries are already doing this), and expansion of geographical coverage of impact monitoring and economic assessment. (m) APOC provides substantive contribution to research via support to TDR, joint support (with TDR) to the Macrofil initiative (seeking a macrofilaricide) and through the APOC Director's and TCC's grants to small operational research projects. Based on support to TDR research, a number of important operational tools and strategies have been developed and used. Pending issues include: insufficient technical guidance by the TCC and APOC management of research funded by APOC; insuff,rcient capacity of APOC to support research; lack of standards for drawing up budgetsl; and concerns regarding relevance of the strategy adopted for the search of alternative drugs especially macrofilaricide and quality of research on possible macrofi laricide. (n) Vector elimination is being carried out over the last 8 years at four sites in three countries. Results from the Itwara and Mpamba-Nkusi focus (both in Uganda) show that elimination of the Simulium. neavei vector species at selected foci is an achievable objective, already attained in Itwara.. However, results from Tukuyu (in Tanzania) and Bioko (in Equatorial Guinea) show that elimination of S. damnosum s.l, which is notoriously more diffi cult, remains uncertain. (o) The evaluation team was pleased to note that APOC leadership has continuously examined the evolution of the concept of "self-sustainability" providing clarihcations to partners as necessary. Sustainability has been recently defined by APOC Management (in 2004) as follows: "CDTI activities in an area are sustainable when they continue to function effectively for the foreseeable furure, with high treatment coverage, integrated into available health care services, with strong community ownership, using resources mobilized by the community and the government." (p) Government contributions, in the form of salaries of staff, facilities and equipment have contributed enormously to the success of CDTI. The "sustainability plans" developed after the evaluation of projects depend in particular on substantial Government contributions after the cessation of support from APOC. (q) Key recommendations on Programme implementqtion . NOTF and CDTI projects should intensify sensitization and mobilization to encourage communities to select more CDDs to increase the CDD/population ratio, decrease workload and improve coverage. Community sensitization and mobilization for selection of women as CDDs should be consistently encouraged. NOTFs should organize special meetings to deliberate and reassess the present sifuation on the issue of incentives, the lack of which can have unfortunate effect on sustainability of CDTI. . Each NOTF should formulate a plan including a realistic budget overall budget needed to sustain capacity building. APOC and NOTF,AtrOCP should intensify their effort to ensure that all projects go to scale with Community self-monitoring as soon as possible. APOC should facilitate exchange of information and experiences between Countries/projects in conflict situations and those in post- conflict. APOC should synthesize experiences tiom reports of CDTI project evaluated to date, leading to lessons, which can be a source of inspiration to individual projects. APOC should I This issue rvas already raised by tlie 2000 Evaluation. JAF11.10 Page 4 analyze the process of developing sustainability plans and the resulting plans developed to date with a view to arriving at standards for drawing CDTI budgets. APOC should consider introducing into impact assessment protocols, monitoring of human infection. APOC should consider expanding the geographical coverage of impact assessment sfudies to all participating countries and in countries with only one sentinel site, the number should be increased. APOC management should give more priority to operational research. A significant asset should be the creation of one Senior Staff Research post at the HQ for the coordination of APOC research activities, active participation in APOC supported research, participation in inter-country and international activities and other activities aimed at improving APOC performance to achieve its objectives. CSA should alrange for an Expert review of strategy and quality of research on Macrofil.[n view of the delays that a number of CDTI projects have experienced and taking into consideration the importance of ensuring that these projects, some of which will operate under difficult post conflict environment, are brought to satisfactory conclusion, the external evaluation team recommends the extension of APOC Trust Fund support on a decreasing scale to 2015. On vector elimination, in Bioko, maintain entomological monitoring over another two-year period. Should biting fly reappear meanwhile, a protocol is being worked out for a national project. In Tukuyu, APOC should ensure that optimal technical assistance is provided to the project both for treatments and entomological evaluation. In due course, the procedure should be similar to that in Bioko by the end of operations. In Itwara the question raised is that of the certification; TCC has approved the request and also recommended a publication on the Itwara success story. In Mpamba- Nkusi it is recommended that one more annual larvicide round be performed followed by intensive entomological monitoring. Without embarking in nuisance control operations which are not within its mandate, APOC should provide technical assistance to Participating Countries intending to carry out such operations on their own. Findings on Programme management (r) CSA is providing increasing leadership on sustainability of projects. In this context, the participation of resource persons in CSA meetings might provide useful information for decision making. TCC has now been relieved from financial evaluations, and technical reviews have been standardized. TCC members felt that the situation has improved considerably. The challenge now is for TCC to be more innovative and guide operational research of strategic importance for the future of APOC. (s) APOC Programme management is good. An important issue noted by the team was a degree of weakness in communication between APOC and projects, and between levels inside the countries as well as with partners, particularly in the area of financial management. (t) On APOC f,rnancial management, planned improvements include: schemes to provide additional staff training at headquarters; more country visits to monitor and train personnel at country offices; and more project- and financial management reviews. Pending issues include: persistent backlog of financial statements to be reviewed and cleared and inadequacy of accurate financial reporting up the chain. The evaluation team discussed the above issues and concluded that more in-depth analysis of the impact of curent arrangements and development of options/ solutions is needed. (r) There has been considerable staffincrease, to 14 Professionals, and 52 General Service (2003). Staff now appears less overwhehned than was the case a ferv years ago. An important pending issue relates to the inadequacy of contractual arrangements lbr sorne staff. a JAFI 1.10 Page 5 (v) At the country level, the NOTF mechanism coordinates the effort of CDTI partners. Two recurring issues have emerged: What is the future of NOTF? Decentralization yes, but where is the strategy for doing so? Similar questions were raised in the 2000 evaluation. APOC needs to develop such a strategy as a matter of urgency on the lines indicated in the recommendations below. (w) The role of NGDOs has been critical to the success of CDTI. Most NGDOs at the country level expressed willingness to continue supporting CDTI; "We were here before APOC, we will be there after." Reaffirmation of the same message emerged from the NGDO Group meeting, at its 25th session2, which referred to "its role in supporting onchocerciasis control up till and after 2010." The challenge is for NGDOs to step up their presence even further and participate fully in planning for 'when APOC is no more'. (x) The role of WHO at country level varies, but includes logistical, administrative and financial management support. The need for increased support from WHO and other agencies like LINICEF will remain, particularly after APOC. (V) On funding at country level, the Memorandum for APOC stipulates that the NOTF will be responsible for 25o/o of the ivermectin distribution costs (in cash or in kind), which will not be available from the APOC Trust Fund. Data on contributions of the countries visited by the team was scanty. (z) Integration has taken different forms, including other programmes using the CDTI infrastructure. APOC has developed and uses a quantitative tool to assess the extent of integration. About half of the twenty-nine projects evaluated, mostly in 2003, achieved a level of integration considered by APOC as adequate to enhance sustainability (a score of 2.5+ out of 5.0). The evaluations indicated that while integration is accepted as the life line for CDTI sustainability, there were no strategies or plans for enhancing integration. Another pending issue is that integration of services is beyond the mandate of CDTI. A recent APOC initiative to support meetings of high-level decision makers for programmes involved in community-health interventions is a promising start to finding ways of improving integration; but to succeed, the initiative will require more investment and visibility. (aa) Key recommendations on Programme management APOC, in consultation with govemments, NGDOs, WHO and all other partners, should promote and oversee the early development and implementation of policies and programs to prepare stakeholders for the continuation of the CDTI programme after APOC. CSA and JAF should consult at an early stage with partners--governments, international organizations, donor agencies and NGDOs--on ways to support CDTI and to ensure the continued fight against onchocerciasis in a coordinated fashion after APOC comes to an end. The primary focus may be on setting up an international coordinating and surveillance mechanism to become effective by 2010 To enhance effectiveness and quality of its deliberations on sustainability, the CSA should consider inviting resource persons, possibly representatives of NOTFs on a rotation basis, to participate in deliberations on relevant agenda items. To alleviative the heary work load on Programme Management, CSA shouid rationalize and reduce statutory and other meetings. 2 Ouagadougou, 10-12 March 2005 JAFl1.l0 Page 6 a a APOC should conduct a detailed review of its financial management systems, possibly with help from WHO Headquarters or outside consultants. APOC should conduct a detailed review, possibly with help from WHO Headquarters or outside consultants, of the adequacy of staff and make appropriate recommendations to assist the Organisation in facing its considerable task, including the challenges emphasized by the present evaluation. TCC should give more attention to issues crucial for the future of APOC (e.g.: CDTI sustainability at various levels, and integration) and regularly include a working session on these topics in its annual meetings. TCC should reinforce its proactive role in proposals for new research activities, evaluation of protocols coming from outside, and follow-up of on-going research projects, especially those conducted with APOC's partial- or fuIl financial support. APOC management should prepare a position paper on decentralization of selected functions from Ouagadougou to the country level. The paper should draw on the experiences of other programmes and agencies and what can be expected. APOC management should prepare an analltical position paper, for review and decisions by TCC and CSA, on issues related to the NOTFs' responsibility for covering 25o/o of a project's ivermectin-distribution costs. . As overall integration of health services is beyond CDTI, APOC should encourage Ministries of Heath, in collaboration with partners (including donor agencies), to find ways of enhancing integration ofhealth services and develop an appropriate strategy and plan. C. Conclusions on findings and way forward. (bb) Throughout its review, and particularly in weighing qualitative evidence, the team has examined closely the sustainability of projects and delays in their implementation. For assessing the likelihood of sustainability, the factors considered included the adequacy of: policy; planning and leadership; integration of CDTI activities within and support of the national health system; financing/funding; human resources; monitoring and evaluation; and coverage and related aspects of APOC and CDTI systems. The picture that emerges from this assessment is mixed. (cc) Our findings, taken as a whole, show that APOC is moving steadily towards its objectives, but still facing substantial challenges to continued satisfactory completion Most of the issues identified are not new; and there is considerable ongoing and planned work aimed at finding solutions to these issues like SAEs. Recommendations of the evaluation team are aimed at enhancing achievement of the objectives. However there is a serious problem of delays, particularly in conflict or post-conflict situations where a number of projects have not even started. The evaluation team concludes that APOC will not be able to achieve its objective of establishing "effective and self-sustainable" CDTI "throughout the endemic areas within the geographic scope of the programme", by the year 2010. On the second specific obi ective of "if possible, in selected isolated foci, to eliminate the vector by using environmentally safe methods", experience to date shows that the goal has been attained in one foci and is achievable in another one. The vector in both foci is Simulium. neavei. However in the other two foci rvhere the vector is S. damnosunt s.1. elimination remains uncertain. (dd) Ensuring the success and sustainability of CDTI will be a local and national responsibility. But appropriate partnership and solidarity among different partners, as well as collective pressure and advocacy will help to eusure that onchocerciasis remains "on the screen" of public awareness and gets adequate funding frorn the government and other sources. JAFlt.10 PageT (ee) The team feels strongly that the key role and contribution of APOC "towards the elimination of onchocerciasis as a disease of public health and socio-economic importance throughout Africa and so contribute to improving the welfare of its people" should be continued and enhanced. At the Regional/Global level. Such a function will be needed to coordinate efforts in such areas as research (in areas such as the effect of CDTI on intemrption of the parasite, SAEs, resistance to ivermectin, and new experiences in organizing CDTI), project evaluation, impact assessment, advocacy, and continued exchange of information between countries. (f0 In addition, the team believes that a complementary financial mechanism needs to be devised, at the intemational level, to support the oversight and partnership functions. Though it is premature to propose any precise formula, the team is of the opinion that if broader mechanisms can not be put in place, CSA and WHO might consider the creation of an extra-budgetary "Onchocerciasis Fund". (gg) The team feels strongly that CSA and JAF need to squarely address this issue of enhancing sustainability of projects as a matter of urgency, so as to put in place innovative strategies for the post- APOC period. (hh) However before we move into the post-APOC period, APOC and its partners need to successfully complete the activities envisaged for the remaining life of the programme, as recommended in this report. APOC should therefore develop a plan of action for implementing whatever recorlmendations of the team are eventually approved by CSA and endorsed by JAF. JAFI 1.10 Page 8 1. Introduction Why APOC? 1.1 Five inter-related developments led to the birth of APOC. First was concern of the global community in the early 1990s that while the Onchocerciasis Control Programme (OCP) was on the verge of eliminating river blindness (through large scale larviciding) as a public health problem in ten West African countries, the disease remained rampant in endemic areas of 19 other countries in Africa. The second development was the availability in 1987 of ivermectin, the first safe drug for large-scale treatment of Onchocerciasis, which the manufacturers, Merck & Co, Inc, decided to provide free of charge for treating the disease as long as needed. Community trials carried out in affected communities between 1987 and 1989 showed that the drug was effective and with minimal side effects. Following these studies ivermectin was, and continues to be, used successfully in the former OCP countries. 1.2 About the same time, a number of NGDOs started ivermectin-treatment in some of the 19 countries, using different approaches. To enhance cooperation, an NGDO Coordinating Group for Ivermectin distribution was formed in 1992. Fourth, and very importantly, a tool for Rapid Epidemiological Mapping of Onchocerciasis (REMO) was developed by the Tropical Disease Research (TDR) Programme of WHO to identify areas and communities most in need of treatment. Fifth, a TDR multi-country study on community-based treatment of onchocerciasis in eight countries, in the early 1990s, established that treatment approaches in which the community plays a major role-i.e. "self treatment"; later termed "Community Directed Treatment with Ivermectin" (CDTl)-achieved higher levels of treatment coverage than approaches where the community just played a "passive" recipient role. 1.3 By 1995 it was clear that the task of providing effective mass treatment in affected areas of the 19 countries was huge, complex, and beyond the capacity of NGDOs. It needed more resources, more government commitment, and stronger partnership and research. (The number of people treated per year at that time had stagnated at around 8 million). APOC was formed at the end of 1995 to spearhead a prograrnme whose ultimate goal was to eliminate onchocerciasis as a disease of public health importance in the 19 countries. Specific objectives of APOC 3 are (a) to establish within a period of 12 to 15 years effective and self-sustainable Community Directed Treatment with ivermectin throughout the endemic areas within the geographic scope of the programme, and (b) if possible, in selected isolated foci, to eliminate the vector by using environmentally safe methods. 1.4 Another key feature of APOC, besides "Community Directed Treatment with Ivermectin" is strong partnership that unites the participating countries and the global community. This partnership has six formal components: Communities, the Joint Action Forum (JAF), the Committee of Sponsoring Agencies (CSA), Programme Management, the Technical Consultative Committee (TCC), and the NGDO Coordination Group. JAF, which includes Ministers of health from participating countries and ms annually, is the highest decision-making body. a 1.5 Other features of the APOC programme include: early introduction of independent monitoring and evaluation; surveillance and prompt action on serious side effects; and operational research and impact assessment. With regard to funding, APOC may pay up to 75o/o of the cost of ivermectin distribution projects, while National Onchocersiasis Task Forces (NOTFs) and the government defray the balance. APOC funding for each project is given for a maximum of five years, followed by funding of three years of "winding down" of the project. 3 As revised by the Joint Action Forum (JAF) in 2001. a African Programme for Onchocerciasis Control (APOC), Programme Document for Phase l, 1996, Ouagadougou JAFl1.l0 Page 9 1.6 The cumulative number of "CDTI projects" supported by APOC has increased enormously over the years, from 7 in 1997, to 48 in 2000 and 96 in 2005. There are also four additional projects in support of the complementary strategy of vector elimination in selected foci; and seven ("HQ-supported projects") have been established in some countries to support the Coordination of the Programme at the national level. Wy the external evaluation? 1.7 External evaluations at regular intervals are part of APOC's rigorous follow-up of projects. External Evaluation of Phase I of the Programme (1996-2001) was carried out in 2000. The APOC Programme Document for Phase Il (2002-2007) and Phasing Out Period (2008-2010) indicates that "Further external evaluations of APOC will be undertaken in 2004, 2007 and 2010, at the end of the Programme". [n this framework, the 10th session of JAF (at Kinshasa, in December 2004) established an external evaluation team of six experts to assess progress of APOC towards meeting its objectives and to make recommendations on how best to sustain CDTI after 2010. Annex I contains the detailed Terms of Reference for the evaluation team. 2. The evaluation process lf/ide consultattons and review of documents 2.1 The evaluation team met in Geneva on 25 and 26 February 2005 with representatives of the World Bank, WHO and the Director of APOC to discuss and clarifu their TORs, agree on data that should be obtained, and to draw up a work plan of activities, including the organization and timing of work planned to be done. 2.2 In March 2005 the team held discussions in Ouagadougou with APOC management and other key staff, during which the evaluation team was briefed on ongoing APOC activities and emerging issues. The team reviewed relevant documents, including the 2000 evaluation report, and allocated tasks to its members. The timing of the work in Ouagadougou provided an opportunity for members of the team to participate in two important meetings: the 25th session of the NGDO Group for Onchocerciasis Control, and the 20th session of APOC's Technical Consultarive Committee (TCC). 2.3 The team also took advantage of the TCC to hold talks with senior staff of the WHO Regional Office for Africa (AFRO, Harare andBrazzavllle). Field visits 2.4 A great deal of the team's attention focused on planning field visits to assess the situation on the ground. Selection of projects to be visited was based on infonnation provided by APOC Management related to the respective dates of launching the projects; their implementation stage and degree of sustainability; geographical and therapeutic coverage, whether the projects were on schedule or delayed, including in countries in post-conflict situations; and inclusior-r of one Vector Elimination project. 2.5 Based on these considerations, projects were selected in five countries: Cameroon, the Democratic Republic of Congo, Nigeria, Sudan, and Uganda. Each country was visited by two members of the team during the period March to May 2005. One tearn meruber attended the Mectizan Donation Programtne (MDP) tneeting in Atlanta (USA) in April. Prior to country visits, docurnents related to the JAF1l. t0 Page 10 countries to be visited (including Projects submissions, Letters of Agreement, Technical and Evaluation Reports, Sustainability Plans, as well as TCC comments and recommendations) were reviewed. Within countries, information was obtained from field observations, interviews with community leaders and Community Directed Distributors of ivermectin (CDDs), National Onchocerciasis Task Force (NOTF) members, policy makers, and CDTI staff at different levels of the health system. Data analysis 2.6 The main areas considered in data analysis included the trends over time (e.g. in geographical and therapeutic coverage), the path to sustainability, responses to important events (such as SAEs), capacity building, level of skills in relation to tasks to be performed, interest and motivation of staff, and innovation demonstrated by project staff. The data obtained was analyzed and compiled into a report by each country team. Key components of country reports were discussed and synthesized at Ouagadougou in June 2005, together with information obtained by the team from other sources. Conclusions and recommendations emerging from the process were presented in July to the Committee of Sponsoring Agencies (CSA). Feedback from CSA was an important input for preparing this final report of the evaluation team.. 2.7 The main constraint to the process of evaluation was the limitation of time, particularly in view of the extensive travel needed for field visits. In addition, two evaluations --of APOC and the Special Intervention Zones (SZ) of the ex-OCP area-- were running more or less concurrently, with some members participating in both. 3. Findings and main recommendations 3.1 Implementation of the CDTI strategy To assess the extent to which APOC has moved towards achieving its two objectives, this section examines progress in twelve key implementation areas: community directed treatment; drug procurement and distribution; capacity building; monitoring and supervision; severe adverse effects; programmes in conflicVpost conflict situations; treatment coverage; evaluation and sustainability; impact assessment; operational research, implementation of projects: achievements and delays; and vector elimination. 3.1.1 Community directed treatment Achievements 3.1.1.1 The hallmark of the CDTI strategy is community empowerment. Community leaders and members showed good understanding of the strategy, the burden posed by disease, and most of their responsibilities in the CDTI process, Communities everywhere have played an active role in planning CDTI, selecting and deploying Community Directed Distributors (CDDs), and collecting and distributing Meclizan. An increasing number of cornmunities are carryng out "self-monitoring" to assess progress in implementing CDTI. 3.1 .1 .Z The evaluation team also found widespread satisfaction and appreciatiou of CDTI benefits by community members--as evidenced by expressions such as. "itching and skin rashes arc gone" and "blindness is less." JAF11.10 Page 1 1 3.1.1.3 CDDs, the backbone of CDTI, were found to be well trained. The 2000 evaluation reported a high rate of attrition among CDDs, "up to 20o/o per year in some projects." Findings of the present evaluation suggest a low attrition rate of between 2 to 5o/o in most projects. Anecdotal evidence suggests that communities where CDDs were selected by open election at village meetings experienced less attrition compared to those selected by village authorities and health workers. A recurring issue over the years has been payment of CDDs. While CDDs assert their willingness to continue ivermectin distribution for as long as necessary, they often showed expectation for some form of recognition and incentives. CDDs in one country indicated to the evaluation team that incentives were not necessary because they are treating members of their own corlmunities. Selection of CDDs in this country is based on a system of clanship. Other communities have devised local incentives in the form of "token" monetary gifts, provision of free meals during the distribution activity, provision of overnight accommodation, and help in cultivation of the CDD's farm. 3.1.1.4 Findings from evaluations undertaken during the 3'd and 5'h year of the projects show that over 71/% of the projects evaluated scored higher than the APOC cut-off point of 2.5 (out of 5) and had a mean score higher than the scores received by other levels of project staff (FLHFs, and at district and national levels).5 3.I .1.5 The present evaluation team was pleased to note during field visits that staff of many First Line Health Facilities (FLHFs) were generally skilled, well-informed on CDTI, and provided considerable support to CDDs and communities. This finding shows considerable improvement on the concems of the 2000 evaluation report. 3.1.1.6 The evaluation team found that staff at district- and council levels were supporting CDTI activities reasonably well in many projects. Some district managers had electronically-retrievable data on CDTI, including treatment coverage by villages, and were conversant with the situation on the ground. 3.1.1.7 Over the years, APOC has provided a total of 140 vehicles,952 motorcycles, and 3397 bicycles for the programme. Issues o-f concern 3.1.1.8 In spite of the positive findings noted above, several instances of inadequate sensitization and motivation of community leaders and communities to take ownership of CDTI, as well as of poor motivation of CDDs were found during field visits. For example, when a community leader was asked why the community was not providing sufficient motivationL/incentives to CDDs as required under CDTI, he addressed this response to the local health worker: "But you have not informed us of this matter." Some communities visited believed that CDDs were well paid by NGDOs and government; while some CDDs thought that their supervisors pocketed their money. There were also instances where unsustainable incentives had been given, occasionally by CDTI projects but it was mostly by other programmes, which might work against CDTI in the long term. 3.1.1.9 The ratio of CDDs to population served is up to 1:450 in sonre colnmunities, and has resulted in reportedly hear,y workload for three to four weeks annually. A recurring recommendation of many CDTI monitoring and evaluation reports in the period 2002-2004 has been to provide adequate numbers of CDDs. In general, the cornmunities'response to these recommendations lras not been adequate. What was intrigurng in the field was the aversion of CDDs to suggestions for training and deployrng 1lew CDDs to T APOC:, Year 2004 Progress Report , document JAF 10.5, Septen.rber 2005. Ouagadougou. JAFl1.10 Page 12 decrease existing workloads! Their responses were invariably "no, we can do the work provided we get appropriate incentives." But community leaders, in general, were for having more CDDs. The evaluation team was pleased to learn that APOC management has since 2004 provided additional funds to about ten projects, with low CDD population ratio, to support training and deployment of more CDDs. The support needs to be intensified by APOC partners. 3.1.1.10 While the CDTI policy is that communities should be mobilized to take ownership and toprovide sustainable motivation to CDDs, some other programs, including those for Vitamin A supplementation and immunization, supported by donor agencies, pay comparatively large amounts as allowances to their community-based "volunteers". The differential treatmeni of community volunteers is confusing, and is working against CDTI. 3'1.1.11 There was no evidence of reviews and continuous rethinking within APOC on CDD/CDTI issues that may be encountered in the furure, and how such developments could be prevented. Questions that need to be addressed, are, for example: Does CDTI depend too much on CDDs *hor" motivation can range from strong to weak? Do PHC facilities feel less concerned with CDTI since so much responsibility rests on CDDs and the communities themselves? Are CDDs over-burdened or over-demurding, oi are they becoming "unionized"-and hence at times taking a stance against increases in CDDs, thus holding CDTI to ransom? 3.1 .l .12 Lack of simple transport facilities, such as bicycles, in many villages limits the ability of CDDs to provide treatment and to follow up absentees and refusals among communities where the population is very scattered. Inadequacy/lack of transport at First Line Health Facilities (FLHSs) is also an obstacle to the training, sensitization, and supervision of CDDs. 3.1.1.13 lndependent Monitoring data in APOC's reports shows that of the157 CDDs surveyed, 52.1oA did not receive any kind of incentive, 26.6oA got monetary incentives , and 20.60/o received transportation support. But surprisingly, "there was no association between giving monetary or in-kind incentives to CDDs and treatment coverage.".6 However, in the evaluation tearn's view, programme experience in the field shows that the issue of incentives is complex, and just does not go away. This is an area that needs continuous follow-up, surveillance, and rethinking. 3.1'1.14 An often-overlooked contribution of CDDs to overall health activities is the CDTI census carried out by CDDs in communities, registering each member of the target population. Some health workers in the field indicated that such data, for example relating to under-fives and expectant mothers, when aggregated for geographical areas, might be useful to other programmes as well. For example, information on the availability of the census data might be of interest to Reproductive Health and Child Health programmes, and could provide a basis for fulure collaboration between these prograrnmes and APOC/CDTI u"ii,riti"r. 3.1:1.15 Withregard to increased participation of women in programme activities, religion and age-old traditions of male dominance in many culrures, as well as illiteracy, have seemed to work against involvement of wornen. However, information frorn APOC reports and discussions with field staff indicate that selection of women CDDs is gradually gaining ground in some communities. Reports from communities where women have been selected as CDDs indicate that their performance is equal to that of their male counterparts. Another positive trend observed in some projects u,as that of "pairing" male and female CDDs, to the benefit of thc community served. 6 Amazigo. U V, Obone, O N, and Daclzie, K Y et al., "lr,lonitoring CDll fbr Sustainability: Lessons from ApOC," Annals of Tropical l\4edicine and Parasitology, g6 (Suppl.l). S75-S92 JAFl1.10 Page 13 3.1.1.16 Actions to be considered: MOH and NOTFs should continue to intensify effort to strengthen FLHFs to play a leading role in supporting CDTI at the community level. APOC might consider supporting a study to explore the feasibility, usefulness, and cost effectiveness of using for other health activities the census data collected by CDDs.. APOC, NOTFs, and other agencies should mobilize additional resources to provide support with transport (particularly bicycles) for CDDs, with emphasis on areas where populations are scattered. Recommendation I (a) APOC management and CDTI partners should intensify and expand support to ongoing initiatives to sensitize and mobilize communities to select and deploy more CDDs in order to increase the CDD/population ratio and thus decrease workload. As an added measure in support of the CDDs' role and of the wider CDTI social contract, simple messoges (written or pictorial according to degree of literacy of target groups) indicating clearly the role of different partners, should be made available to the communities and posted at appropriate places. (b) In light of the positive reports of their performance compared to male CDDs, communigt sensitization and mobilization for selection of women as CDDs should be consistently encouraged. This could be done during community meetings. (c) NOTFs should organize special meetings and consultations to dcliberate and reassess the present situation on the issue of incentives, the lack of which can have unfortunate effects on the sustainability of CDTI, and take prompt remedial action. Basic data on the present situation in different projects should be collected to facilitate discu.ssions. 3.1.2 Drug procurement and delivery Achievements 3.1 .2.1 Drug procurement and distribution is anarea of great success in APOC. Annual orders of the dmg, based on total population and treatment figures of the previous year, have been efficiently used in ordering for Mectizan supply. Issues identified in the 2000 evaluation, such as long hold-ups of drugs during custom clearance, have been largely resolved. The role of APOC in the procurement and distribution of ivermectin is now less prominent, which is good for sustainability. Governments are increasingly clearing Mectizan shipments at the point of entry. Further down the supply chain, Mectizan delivery to rural communities is good, even where APOC funding has been terminated in the last two years. FLHFs collect drugs at the district/council levels, and the community collects it from the FLHFs. Drug collection usually does not necessitate additional funds since it is collected when health workers travel to the upper level ofthe health system for other purposes. 3.1.2.2 In general, drug orders to and delivery by the Mectizan Donation Programrne (MDP) do not suffer delays. Annual drug procurement and delivery has on the whole been adequate and tirnely, even in conflict areas. a a a JAF11.10 Page 14 3.1.2.3 An important finding, based on the evaluation team's visits to the field, is the high interest and demand for ivermectin by communities. "Why is Mectizan late? When will it arrive?" were corlmon questions asked by communities in some of the projects visited. Issues of concern 3.1.2.4 LTNICEF and WHO have continued to provide support in a number of countries for the clearance of Mectizan from the port and its storage before collection by government or partners. However, it was understood by the evaluation team that these arrangements are temporary. 3.1.2.5 Delays in requesting Mectizan were documented in three of the countries visited. This had resulted in delayed drug distribution, and increased absenteeism of persons expected to come for treatment. There were reports of ivermectin circulating in areas non-targeted for CDTI. Treatment in areas not targeted by APOC-funded projects is provided only by health facilities. 3.1.2.6 Actions to be considered APOC should encourage NOTFs and governments to ensure that they use sustainable arrangements for customs clearance and storage of Mectizan where applicable. NOTFs should intensify efforts to ensure that drugs are ordered early enough to avoid delays in arrival and distribution. Recommendation 2 NOTFs should continue to be alert to and immediately investigate reports of "leakages" of ivermectin, and promptly take corrective action. 3.1.3 Capacity building Training, inclttding re-training 3.1.3.1 Training is another area of APOC success. A "cascading" training process, starting with the NOTF and moving down to communities through existing health structures at various levels, has been used to improve operational skills in many countries. The ripple-effect of training is a source of strength for the programme. At the local level, targets have been set for training health workers and CDDs. APOC records show that the number CDDs trained has increased enormously from 539 CDDs in 1998, when the training process was launched, to 242,826 CDDs in 2004 (of which, 178,808 were re-trained aDd 64,018 newly trained) in the whole programme. 3.1.3.2 APOC has developed severai sets of training material which are packaged according to target audiences and purpose, and are used both for training and enhancing sustainability of CDTL A practical manual titled "Community-Directed Treatment rvith Ivermectin (CDTI), issued in 1998, has proven a valuable tool. Leaflets and visual material on different aspects of onchocerciasis are also widely used. 3.1.3.3 Training, including re-training, has covered a u,ide range of technical needs, including: REMO; RAPLOA and handling of SAEs; Geographic Infomation Systems (GIS); computer use, data analvsis, and financial management of the APOCI Trust Fund; as well as development and use of instruments and guidelines to evaluate CDTI. APOC support has taken different fonns, including technical and financial support. a o JAF11.10 Page 15 3.1.3.4 With regard to financing of the above measures, APOC contribution is made under a specific budgetary line for "Training, Workshops, Mobilization & Advocacy'', and as separate budget lines within each project. It is estimated that, to date, a total of over US $ 7.5 million has been spent or firmly committed for these items under APOC Phase II. Two other sources of funding are the governments and NGDOs. Although overall data on contributions of NGDOs is not available, findings in the field showed that considerable support is provided by them. 3.1.3.5 At its Fifteenth Session (in September 2002), the TCC reviewed the issue of APOC support to projects after 5 years, and recommended that support to measures for capacity building (including advocacy, training and new capital equipment) should be provided for years 6 to 8 as well. This is intended to avoid a decline in performance following withdrawal of APOC support after 5 years, and to consolidate progress towards sustainability. (Before this decision was taken, APOC had not been expected to support programmatic activities, including CDD training, after the fifth year.) The evaluation team considers this TCC recommendation appropriate and timely. 3.1.3.6 Extensive health education, sensitization, advocacy, and community mobilization (termed HSAM in APOC reports) has taken place in communities, as evidenced by the knowledge of community leaders and communities on CDTI in such areas such as Mectizan benefits, dosage, people to be included or excluded, and whether the Mectizan was donated free of charge. lssaes ofconcern 3.I.3.7 A major challenge is how to sustain resources for training, both basic and re-training. Funding by governments is inadequate. 3.1.3.8 In general, the programme's epidemiological and analytical culture was found to be weak. For example, some of the treatment-coverage data seen in the field showed great disparities between sub- districts and villages--but there was no evidence of the data being properly analyzed and used by programme staff to target specific support/activities to areas with poor coverage. 3.1 .3.9 A recurring issue in the field was "Why does Mectizan need to be taken continuously, for additional number of years, by people who no longer experience skin and visual problems?" This issue was reported to accentuate absenteeism or refusal to take the recommended treatment. Increased sensitization on the issue is needed at all levels, targeting particularly the community, health services staff, and most importantly policy makers. 3.1.3.10 Despite the strength of the cascading-approach to training on the lines indicated above, weaknesses were reported in at least one country. It is clear that one level of the system may be by-passed, with detrimental effect on the other levels, and/or on qualityof trainingprovided. Immediate on-the-spot corrective action needs to be triggered when supervisors detect inadequacies in training. 3. I .3.1 I Actions to be considered APOC and NOTFs should analyse and disseminate findings from the ongoing operational studies on compliance with and continued use of ivermectin. The findings and lessons could provide evidence on which to develop nerv IEC materials that would motivate cornmunity members to continue taking Mectizan even after symptoms have disappeared a JAF11.10 Page 16 By supporting the training of community-selected monitors/supervisors and FLHWs, APOC and NOTF should rapidly expand community self-monitoring. NOTF should organize re-training of district- and FLHF-level health workers involved in CDTI on data analysis and use. The aim should be development of their capacity and building a culture of using data as input to decision making--for example, for targeting special support to villages or sub- districts that are outliers in therapeutic coverage. Recommendation 3 (a) Each NOTF should formulate a plan that includes the budget needed to sustain capacity building. The plan should include needs for training and retraining, and for other aspects of enhancing capacity, including review and revision of IEC materials.APOC and partners should ensure that adequate funds are available, as needed, for projects which have completed their fifth year. (b) NOTF and APOC-funded projects should intensify sensitisation of communities, health service workers, the public, and political leaders on the basics of CDTI, including the need for continuing with ivermectin treatment for many years after symptoms have disappeared and the progression ofthe disease has been halted. 3.1.4 Monitoring and supervision 3.1 .4.1 Monitoring and supervision are carried out by national, state/provincial, and district/local government levels in a cascading manner. This approach means that the national and provincial levels supervise and monitor CDTI at council and district levels respectively, while district-level staff supervise and monitor CDTI at the FLHF level. FLHFs in turn supervise and monitor CDDs and community level activities. The approach provides for spot checks carried out by higher-level supervisors to supplement information obtained fiom routine activities. Issues of concern 3.1.4.2 Supervision of community-level activities by district-levels staff and FLHWs was inadequate in some projects. Supervision checklists often were not used, or were poorly completed. Furthermore, opportunities provided for supervision of other programmes, such as vitamin A distribution, were not exploited for CDTI supervision. Other issues encountered in some projects include lack/inadequacy of simple transportation (e.9., bicycles/motorcycles; also see discussion of capacity building); inability of district-level authorities to pay field allowances; and the absence of community self-monitoring of drug distribution in most communities. Some district-level officers on onchocerciasis teams complained that the APOC/ CDTI reporting form was "too Iong," and measured "technicalities" rather than "performance." Finally, reports of superuision undertaken were often not available for review by the evaluation team during its field visits. 3.1 .4.3 In general, however, the independent monitoring introduced byAPOC in the earlystages of the programme has been successful in improving implementation of CDTI. The main issue has been high cost of the exercise. The Evaluation Team is pleased to note that nationals from other projects are now mostly carrying out the ongoing independent monitoring of CDTI. Thus, it is clear that independent monitoring is one of the functions that could be gradually devolved to NOCPs,[rtrOTFs. a 5 JAFl1.10 Page 17 3.1.4.4 Actions to be considered APOC should actively support NOTFs to set up effective monitoring systems to take full responsibility for project monitoring before the end of APOC. While the policy should apply to all projects and countries, the strategy should incorporate a learning-by-doing approach and intensiff support to selected projects. Experience and lessons gained from such projects could be ofuse to other CDTI projects as well. APOC should review and revise supervision guidelines, as necessary, to enhance simplicity of use in the field. Recommendation 4 APOC and NOTF/NOCP should intensify their efforts to ensure that all projects go to scale with community self-monitoring as soon as possible. 3.1.5 Severe Adverse Effects (SAE) A chievem ents and r'sszes 3.1.5.1 The 2000 evaluation report observed that "in two countries, fear of SAE had been a reason for people refusing to take ivermectin". [n one of these, at the time of the (current) evaluation teams's visit in mid-2005, CDTI was suspended in 12 out 14 CDTI projects (after 62 cases of SAE, including 19 deaths were recorded in one country in the previous years, in two of the projects; and in another country, 199 cases ofSAE were recorded in 1999-2000). 3.1 .5.2 The present evaluation team was pleased to note that the number of SAE cases in the country visited, Cameroon, had gone down considerably, and were almost entirely confined to the first yearlround of treatment. The most recent outbreak of SAEs occurred in early 2005 in one of the two countries affected in 1999, resulting in a total of 27 cases, but no deaths. However a new threat for sustainability of CDTI appeared on this occasion--the launch of a misleading media campaign, relayed through the internet, giving exaggerated numbers of SAEs cases, even deaths, and calling for legal action. Fortunately, a quick response by the country's MoH prevented further harm. 3.1.5.3 For management of SAEs, the recommendations made by the 2000 evaluation have been implemented to the letter. APOC guidelines have been prepared, updated, and widely disseminated to countries with projects that encompass high-risk areas. In Cameroon, management of SAEs benefited from the joint effort of MoH, APOC and MDP. The various partners developed a prograrnme combining prevention (RAPLOA and sensitization), patient surveillance, and emergency care, as well as physio- pathological investigations, under the supervision of a senior full-time national specialist. The experience of this expert is being used by APOC for training and building similar systems in countries with loa loa- infected onchocerciasis zones. 3.1 .5.4 The evaluation team learnt that there are sometimes strong demands for ivermectin in hypo- endemic areas. Thus, it is likely that some ivennectin is finding its way to such areas as well. 3.1 .5.5 Actions to be considered APOC should update the existing guidelines for SAE management and disseminate them in countries at risk of co-endemicity of loa loa ar.rd onchocerciasis. a a o JAI11.10 Page 18 APOC should develop a SAE surveillance mechanism in selected hypo-endemic onchocerciasis areas infected with loa loa. NOTFs should organize seminars on management of CDTI in loa loa-infected areas, with education messages on the SAEs targeted to key partners, including the media. APOC and NOTFs should develop an SAE surveillance mechanism in selected hypo-endemic onchocerciasis areas, especially for use during the post-APOC era. Recommendation 5 The system developed by MDP/MoH/APOC for dealing with SAEr should be rapidly made available by APOC to countries with high risk of SAEs. The system should be further updated as soon as validated new information and adyances are available. 3.1.6 Programmesinconflict/post-conflictareas Achievements 3.1.6.1 Field visits of the Evaluation Team included one country in conflict, and two in post-conflict situations. The team commends APOC for spearheading a series of strategies and activities to get CDTI started in conflict areas, under very difficult circumstances. 3.1 .6.2 In the two post-conflict countries visited, a number of favorable factors have now emerged. These include political support up to the highest level in the Ministry of Health; availability of trained human resources in NOTFs; effective training and retraining activities; well-organized Mectizan distribution (but still financed by APOC Programme, pending incorporation into the national drug procurement system); functional FLHFs with satisfactory degree of integration of CDTI in-built from the start; NGDO commitment; and adequacy of IEC documents. All this is the result of past well thought-out efforts, which must continue unabated towards consolidation. 3.1.6.3 Thus even under the difficult circumstances community leadership is strong as evidenced by involvement of communities in planning for CDTI, large number of CDDs trained, and good availability of drugs. Supervision although sporadic was sufhcient to provide basic support and encouragement to CDDs and communities. Experiences in these and other areas in the implementation of CDTI should be nurtured as basis for building future CDTI work. lsszes ofconcern 3.1 .6.4 One great concern is non-availability of sufficient government resources. In one of the countries visited, project staff received no government salaries and survived on APOC- and NGDO support. Elsewhere, salaries paid by State are below subsistence level and are topped-up by APOC and NGDOs. Furthermore, all at risk populations living in conflict or former conflict zones can still not be reached due to lack of roads, areas mined with explosives, and continuing military activities. Because extensive population movements take place during as well as after conflict situations, the difficulty of reaching persons requiring treatment with ivermectin is likely to continue for some time. 3. 1 .6.5 REMO had recently been refined in the country in conflict visited by the team. Exarnination of the REMO arrd treatment data for the part of thc country that has over 80% of onchocerciasis revealed that therapeutic coverage was less than 10%. Tl.rus, uncertainties in conflict situation make it practically impossible to talk about sustainability of CDTI projects at this time. a a a JAFl1.10 Page 19 Recommendation 6 (a) Regardless of the number of years a project has been in existence, in some conflict areas APOC should consider CDTI as in its initial phases, keeping in mind that most work cannot be done until peace comes. (b) APOC should facilitate exchange of information and experiences between countries/projects in con/lict situations and those in post-conflict. Such experience could help guide preparations for activities to be launched when peace comes. APOC should also document experiences and lessons on CDTI in conJlict situations. Such lessons could lead to guidelines which would be useful for countries that might move into conflict situations in the future, particularly when APOC is no more. 3,1.7 Treatment coverage Achievements 3.L.7.1 At present, 63 projects are being implemented in 14 countries.. In the early years of APOC (around 1995), coverage was around 8 million people treated annually. Annex 2 shows that the number of persons treated yearly in Participating countries increased frorn 14.58 million in 1997 to almost 22 million persons in 2000; and it has since increased to 28.45 million in 2002 and 37.31 million persons treated in 2004. There has also been a steady increase in therapeutic coverage since the beginning of CDTI projects; such coverage has now exceededT0o/oin2004 in Uganda, Tanzania, Cameroon, and Ethiopia. In Uganda the coverage declined from 12.9o/o in 2003 to 70.7oh in 2004; and in Nigeria, the therapeutic coverage reached 70o/, in 2003, but declined to 60.70/o in 2004.7 There has been a steady increase in Congo and Malawi, to 660/o and 60 o% respectively in 2004. Even the DRC, a country in a post-conflict situation, has achieved a steady increase to 45.60/o therapeutic coverage in2004. But in Sudan, Liberia and CAR, all of which are countries still in conflict, coverage has been generally low and variable. The CDTI programme has just started in Angola. Finally The Ultimate Treatment Goal (UTG) for 2004 was 72 92'7 89'7. About 5l% of UTG (37,927 897 persons) were treated in participating countries. But here again as in the case of therapeutic coverage there is a big variation between countries. lssues of concern 3.1.7.2 Analysis of data is generally not done by programme staft but even a cursory look at the numbers shows that "averages" hide a lot of inequities. Weakness in the "culture" of data analysis were found to be widespread (see under Capacity Building). 3.1 .7.3 The evaluation team also found that accepted definitions for estimating therapeutic coverage have not been consistently used in the field. The evaluation team recornmended that steps should been taken to ensure that standard denominators and methods for measuring coverage be used, so as to enhance consistency and comparability of data on progralnme achievements and constraints. However the tearn's observations from field visits shows that the problern remains. 7 The declines in Uganda and Nigena have been attributed to the "sustainability test" and to stoppage of NGDO financial support to projects w,here APOC had stopped. Details of the causes of stoppage are being studied. (Ref: NGDO Group Ir{eeting Reporl, March 2005). JAFl1.10 Page 20 3.L7 .4 Other issues observed during field visits include the fact that some CDDs do not record in their "log books" the reasons for those who are not treated-i.e. whether such persons were "absent," "refused", "pregnant," or "breast feeding." Another major area of concern relates to the need for undertaking a follow-up REMO (which is a good assessment tool). Some staff and communities visited by the team claim that some villages that were previously hypo-endemic are now hyper-endemic. Confirmation of this change in endemicity of onchoceriasis in the affected population requires data from follow-up REMOs in selected areas. 3.1.7 .5 Actions to be considered NOTFs and their partners should ensure that retraining programs include correct methods of recording Mectizan distribution. Urgent steps should be taken by APOCAJOTFs to enhance the use of standard denominators and methods for estimating coverage. Recommendation 7 Adequate investment should be made infollow-up REMO studies so os to catch up with the progrqmme's needfor updated information on endemicity of onchocerciasis in selected areas. 3.1.8 Evaluationandsustainability 3.1.8.1 The 2000 evaluation noted that there was considerable skepticism and divergence of opinion on the programme's sustainability, as well as on how sustainability should be assessed. The concept of sustainability has evolved over the years, both in APOC and other programmes. At the beginning of APOC, self-sustainability or self-sufficiency meant the ability of a project to continue functioning effectively using only the resources generated within the country itself. To facilitate understanding and meaningful debate, APOC has provided a clear definition of sustainability as follows: "CDTI activities in an area are sustainable when they continue to function effectively for the foreseeable future, with high treatment coverage, integrated into the available health care service, witl.r strong community ownership, using resources mobilized by the community and the govemment." The evaluation team commends APOC for its leadership in this area, but regrets that some APOC's partners tnay not have fully accepted the new definition put forward by the programme. 3.1.8.2 lndependent evaluations have been carried out by APOC in 49 of the 53 projects targeted during the period 2002-2005. The verdict was that 35 projects (73%) were moving towards sustainability. In these projects, a total of 442 district/local government authorities (LGAs) in their fifth year and 84 district/LGAs in their third year of operations were involved. 3.1.8.3 In addition, one NGDO monitored the performance of 20 projects in three countries which had received no extemal support for 2004 from any source, neither from APOC nor from the NGDO itself. The first year's results after cessation of extemal funding identified problems with provision of treatment-- resulting in a decline in numbers treated in two of the three countries. But in some projects, actions by local health system leadership and comrnunities u,ere effective in sustaining treatment. The NGDO, in consultation with all the country-based partners, rvill rnonitor the projects fbr a further year to know exactly what the problems are, and what can be done about ther.n. 3.1.8.4 An irnportant development spearheaded by APOC has been the development of "sustainability, plans" as part of the evaluation of its projects. Indepeudeut rnonitoring of the irnplernentation a a JAF11.10 PageZl of sustainability plans, recently initiated by APOC, is proving to be a powerful tool for improving the sustainability of projects. The evaluation team believes that development of sustainability plans and their evaluation is a breakthrough in resolving a chronic public health problem. Since June 2005, monitoring of the implementation of sustainability plans for CDTI projects has been carried out for such plans; and l2 of them will have been monitored by the end of 2006. Lessons of this monitoring experience may be of great relevance to for future CDTI devolution and integration with national health services systems. lsszes ofconcern 3.1.8.5 The 2000 evaluation report stated that: "There appears to be a lack of standards for drawing up budgets. APOC Headquarters staff and TCC members complain that projects over-budget considerably and staff have to spend considerable time revising budgets downwards. At country level concerns were expressed about a lack of guidelines/standards for projects to use in calculating amounts in budget -for themselves and for TCC/APOC Headquarters." The report also noted that "setting budget standards (with country adjustments if necessary) will make every one's life easier." The present evaluation team regrets to note that such standards are yet to be set. However, experience and lessons from the monitoring of sustainability plans could make the development of the above standards easier and more realistic. 3.1.8.6 Other issues identified by the evaluation team include inadequate synthesis and learning from the evaluations undertaken by APOC to date, and inadequate coordination of evaluation initiatives undertaken by partners. Recommendation 8 (a) APOC shottld synthesize the monitoring and evaluation experience of CDTI projects evaluated to date, and identifi lessons that could benefit individual projects. To enhance learning-by-doing, APOC should also convene a nteeting of key individuals involved in projects in their third year ofphasing-oLtt so as to pool experiences end draw lessons. These experiences and lessons should be widely shared with other projects. (b) APOC should analyze the process of developing and implementing sustainability plans, y,ith a view to determining standards for CDTI budgets. 3.1.9 Adequacy of impact assessment Hypothesis and process 3.1.9.1 The APOC hypothesis for impact assessment, is that regular treatment with ivermectin will reduce severe itching; prevent development of onchocercal skin disease and or regress early lesions; prevent/delay progression of onchocercal eye lesion and blindness and may regress early stages of ocular lesions; and lead to reduction in vector infectivity and to improvement in the socio-economic stafus of the community. Four intemational assessment teams, each comprised of an ophthalmologist, a dermatologist, a socio-demographer, and a medical entomologrst have been set up by APOC; and these teams are supported by a total of 147 national associates. Standardized evaluation protocols have been developed; and 14 sites in 9 countries have been selected for impact assessment. JAr11.t0 Page22 Achievement 3.1 .9.2 The baseline survey was carried out in 2000, a second survey is being conducted in 2005, and the final impact assessment survey is planned for about 5 years from now. Although analysis of data from the sites being surveyed in 2005 will require some more months, preliminary results from one project indicate significant regressions (10/1) of all skin lesions and of prevalence of microfilaria in the anterior eye chamber, while posterior eye lesions have remained unchanged as expected. Issues of concern 3.1.9.3 Without parallel assessment of the evolution of human infection in the populations treated with ivermectin, interest in monitoring entomological parameters alone is fading. It is almost impossible to find ivermectin-free areas in some countries, and areas with less than 25o/o of the population treated has had to be accepted for the impact assessment survey in 2005. Some studies have been carried out in projects with only one ivermectin treatment in the past 5 years. It is now clear that by year 2010 there will be many projects which will not have reached the appropriate "stage" for the final round of impact assessment. 3 .l .9.4 Studies are being based on prevalence of nodules, which is known not to reflect changes of infection over the short and long term. Measurement of microfilaria prevalence and of the mean microfilaria load in the communities would be more accurate indicators of the evolution of human infection in the affected populations. 3.1.9.5 There are no economists in the impact assessment teams; and hence socio-economic data is not being collected. 3.1 .9.6 At least five sites will be excluded from the results of the current (second phase) survey, for various reasons: one country had not been selected for CDTI; in another country, baseline data had not been collected because of local uffest; in a third country, annual treatments with ivermectin had not been given for the same reason; in the fourth country, treatments had started after a 2-year delay; and in the fifth country, preliminary analysis of survey data has raised doubts on the quality of treatment data available. Consequently, the number of sites available for the hnal survey (3rd round) has been reduced to 11. 3.1 .9.7 Actions to be considered a o APOC should expedite the analysis of data collected during the 2005 impact assessment survey (second round), payrng attention to information that can be obtained from sites where APOC should encourage partnership with research institutions in Participating countries so as to collect data that could help assess the socio-economic impact of CDTI. Since ivermectin-treatment is belicved to produce collateral benefits for addressing other health problems, APOC should consider supporting initiatives to collect and analyze data on health-status trends, particularly for children aged 4-5 coming for first treatment, so as to observe the irnpact of CDTI on the general health status of the population. Collection of data could be incorporated into school health-examination protocols. During its phasing-out process, APOC should mobilize funds to ensure that impact assessment of delayed projects is adequately funded. Requirements of the protocol were only partially met. Consideration should be given to using thc second round data as the " baseline"for sites where CDTI did not start in time or experienced long delays after the first impact assessment survey had been conducted. a JAFl1.10 Page23 Recommendation 9 (a) APOC should consider introducing impact assessment protocols for the monitoring of human infection, as is already being done on a national basis by some APOC-countries. @ APOC should consider expanding impact assessment studies to all participating countries; and in countries with only one sentinel site, the number of such sites should be increased. 3.1.10 Operational research 3.1.10.1 TDRisAPOC'smainresearchpartner.Duringlgg6-2004,APOCcontributedaboutUS$307,000 annually to TDR for operational research. The research projects supported by APOC include those that led to tools for rapid assessment of endemic areas eligible for CTDI (i.e., REMO - Rapid Epidemiological Mapping of Onchocerciasis ), and for rapid assessment of loa loa in areas of co-endemicity (RAPLOA). They have also resulted in the development of guidelines and tools for evaluating the implementation and progress of CDTI; for taking into account sociological factors that condition a population's participation in and adherence to the requirements of CDTI; and approaches for enhancing the motivation of CDDs, and for improving prograrnme efficiency so that various APOC-supported activities could become more sustainable. 3.1.10.2 Developments of methods like REMO and RAPLOA have proven to be major assets facilitating APOC implementation. TDR has also obtained funds from the Melinda & Bill Gates Foundation for a study of the impact of ivermectin treatment on the survival of O. volvulzs in humans (focusing on worm life- expectancy and female fertilelife-expectancy). A study of "extended" ivermectin distribution, undertaken over a long period of time in two countries, is also being undertaken. Its results could significantly improve the quality of estimates of the number of years that CDTI must be applied in a given onchocerciasis control programme. APOC und TDR support to MACROFIL 3.1.10.3 Under the MACROFIL initiative (Macrofilaricidal Drugs for Onchocerciasis and Lymphatic Filariasis; searching for a macrofilaricide), APOC and TDR each contributes US $700,000 annually. The approach of TDR is that of complementing the investment of a small pharmaceutical company in the development of an already-screened potentially-macrofilaricidal drug. Only one product (Moxidectin) has emerged from this research until now, and is expected to enter the phase of clinical study on onchocerciasis patients, if and when reservations raised by the US Federal Drug Administration (FDA) are addressed. 3.1.10.4 MACROFIL has been instrumental in the development of ivermectin protocols and the development of the DEC (diethyl carbamazine) patch-test for early detection of low macrofilarial loads. It is now involved in experimentation trials of associative protocols of treatment in areas of co-infection of onchocerciasis-loiasis-lymphatic filariasis, and in genomic studies of worms in patients not responding to ivermectin. Issue of concern 3.1.10.5 The evaluation team learnt of a corlceru expressed by a member of CSA that there has not been tangible outcome from the MACROFIL work undertaken thus far. The evaluation team wishes to re- affirm that, in the long run, large-scale use of a macrofilaricide would be the only alternative to treatment JAF11.10 Page24 with ivermectin for achieving onchocerciasis control. However, the team does not have the technical expertise in drug development to evaluate either the relevance of the strategy adopted thus far or the quality of the research being performed by the company. CSA might consider obtaining the services of individual/s with appropriate skills to review the strategy and quality of ongoing research on MACROFIL. The TCC grants and APOC Director's grants Achievements 3.1.10.6 Both TCC- and Director's grants are restricted; and provide up to US $10,000 and $5,000 respectively for the two tlpes of grant. They support relatively small operational research projects proposed by national investigators in Participating countries. The TCC has accepted 24 of the 72 projects received thus far, for a total budgetary commitment of approximately US $200,000. Almost 50"h of the projects proposed and accepted have come from three countries only, and the investigators have been mostly the same individuals. For the Director's grants, only l0 applications have been received thus far. 3.1 .10.7 The proposals submitted cover a wide range, and deal mainly with strategies for improving coverage and community motivation, evaluation of the impact of strike-action by PHC workers on CDTI coverage, participation of women in CDTI, examination of the relationship between epilepsy and onchocerciasis, and approaches for building the operational research capacity in Participating countries. Issues of concern 3.1.10.8 The evaluation team noted that grant proposals are often repetitious; and it appears that adequate stock of previous results and/or on-going projects is not adequately taken into account in the development of new proposals. There is also a big gap between on-going operational research and the urgent sustainability-related issues that need to be addressed. Examples of such issues include much-needed debate on the impact of cost-sharing on sustainability; the basic budgetary allocations required to sustain CDTI; the optimal nurnber of patients per CDD in a given population; and an examination of local variations in endemicity levels that could affect coverage. The Way forwardfor APOC 3.1.10.9 In this context, APOC needs to act on two fronts. First, over the years, APOC has supported research that has contributed to generation of knowledge fundamental to improving oncho- control strategies. Strategic areas needing future support include the study of longevity-fertility of adult wortns after annual rounds of ivennectin, repeated over different periods of time; genomic changes and their significance, if any, in adult worns submitted to annual ivermectin treatments over different periods of time; indications of reduced ivermecin efficiency through follow-up of known cohorts of patients treated over several years; estimation of the impact of mid-low level coverage on infection and disease in humans; in-depth physio-pathological studies of SAEs; and integration of CDDs and other levels of the health systems of Participating countries (building on previous studies). APOC has also facilitated meetings of operations staff at the country level, leading to exchange of information and learning from one another. A number of national staff have been utilized for evaluation-, monitoring- and impact-assessment missions. The resulting muitiplicity of contacts and exchanges with outsiders has resulted in the building of an impressive comrron reservoir of experlise for which APOC can be justifiably proud. 3.1.10. l0 The second set of future needs relates to operational research. APOC has done verywell in solving issues "internal" to CDTI; but CDTI is now increasingly facing difficult and persistent problems JAr'11.10 Page25 whose solution is often beyond the control and resources of CDTI and APOC. The best approach currently (and likely to be) available with APOC, its projects, and Participating countries for tackling these issues is to leam-by-doing and from one another. Specific options and solutions could emerge from field experience rather than from academic meetings. For example, in developing policies for devolution of management from central to lower levels (an acknowledged "hard" issue), a learning-by-doing approach-i.e., selecting a handful of suitable candidate countries, providing intensified support, monitoring progress, and pooling experience for discerning and disseminating lessons--is perhaps the only promising way forward. The capacity to undertake operational research will be critically needed in countries when APOC is no more. APOC thus needs to work on both fronts-undertaking and building capacity for strategic as well as operational research-and has to do it in a balanced manner. The evaluation team believes that APOC's capacity to address the issues outlined in this section needs to be considerably augmented. 3.1 .10.1 I Actions to be considered APOC management and TCC should make a distinction between projects of very local interest, aimed at finding solutions to local problems, and proposals for larger scale, possibly multi-centre projects--whose results could be of wider interest in participating countries, and which therefore should be eligible for bigger sums of money from APOC. APOC should make it widely known in Participating countries that it places high priority on support for operational research, so that it could reach a larger spectrum of countries and investigators and encourage them to become involved in the research supported by APOC. APOC should participate actively in strategic thinking and research aimed at finding new directions for the control and eventual elimination of onchocerciasis. APOC should strengthen collaboration with TDR, in terms of joint "reflection"on a synthesis of experiences, and review of the programme's and CDTI projects' needs for applied/operational research so as to address emerging problems. TCC should give greater emphasis to reviewing existing and emerging research issues, and together with APOC management, should provide more technical inputs to TDR, at various stages of the research process-from validation of protocols, follow-up, discussion of findings, evaluation of completed work, acceptance of research results, and implementation. Recommendation 10 (a) APOC management should give greater priority to operational research. A Senior Staff research position should be created at headquarters in Ouagadougoufor the coordination of APOC research activities, and for enabling active Programme participation in scientific dis cus sions at international meetings. (b) APOC should advocate and support the training cf key CDTI staff, particularly district coordinators and project manqgers, on methodologies and other key qspects ofoperational research. (c) The CSA should arrange for an expert review of the strateglt and quality of research on MACROFIL. 3.1.11 Implementation of projects: achievements and delays 3.1.1 1.1 The cumulative nurnber of approved "CDTI projects" has increased enormously over the years, from 7 in 1997 to 48 in 2000, 65 in 2003, and 96 in 2005. This number is expected to reach the final figure of 1 11 by 2007 . In addition, there are four projects that support the complementary strategy of vector a a a a a JAFl1.10 Page26 elimination in selected foci; and seven HQ-supported projects that have been established in some countries to support coordination of programme activities at the national level. As of March 2005, five projects (3 in DRC, one in Uganda, and one in Angola) were still in the planning stage and had therefore not yet been approved; and24 projects had been approved but not yet launched. Most ofthe projects not yet approved or approved but not launched were in conflict or post-conflict situations. Since the "basic" APOC support is for 8 years-i.e., the initial 5 years, followed by three years of "phasing out" during which sustainability plans are implemented - it is foreseen that a total of 49 projects will still be under implementation by 2010. 3.1.11.2 Other reasons for delay, besides conflict and post-conflict situations, include Serious Adverse Effects (SAEs) and managerial or procedural constraints. The latter include delays in HQ due to the heavy workload of APOC staff which has to cope with significantly more projects than initially expected. At the field-level, delays are the result of an uneven quality of project proposals, and non-compliance with WHO rules for financial accounting and management. 3.1.11.3 Thenumberof projects (49) stillunderimplementationby20l0 accordingtopresentforecasts, is expected to decline to 5 by 2015. The evaluation team had the opportunity to discuss these delays, and their implications for sustainability and potential solutions, with APOC management. The evaluation team commends APOC management for its foresight and sound analysis of this and related issues, and supports the Management's proposal that APOC support be extended by five years beyond 2010, i.e. up to 2015. Recommendation 11 In view of the delays that a number of CDTI projects have experienced, and taking into consideration the importance of ensuring that the remaining projects--some of which will operate under dfficult post-con/lict environments--are brought to satisfactory conclusion, the external evaluation team recommends extension of APOC Trust Fund support, on a decreasing scale, to 20 15. 3.1.12 Vector elimination projects in APOC Rationale, selection of projects and achievements 3.1.12.1 The hypothesis for focal vector-elimination operations in APOC is that the rapid, cost effective, complete eliminalion of the vectors in small isolated foci, when combined with ivermectin treatment and post-larviciding entomological surveillance over a few years, can put a time limit to the endless annual rounds of ivermectin treatment required for achieving onchocerciaiss control. A source of inspiration for the elimination strategy was the six successful operations carried out in confined foci in Kenya and Uganda in the 1940s to 1960s. The vector in five of these foci was S. neavei and in the sixth focal site it was S. damnosum. Accordingly, four foci were selected for APOC support (two in Uganda, one in Tanzania, and one in Equatorial Guinea), following technical assessments by TDR consultants. 3.1.12.2 Thus, the focal eradication strategy has been mainly aimed at the control of onchocerciasis in East Africa, where the distribution of the disease is notoriously patchy. This is especially so in areas where the vectors belong to the Simulium neavei complex, which show a restricted flight range and great lulnerability to ecological changes. Such is the case also in East Afiican areas where the vector species are members of the S. damno.sum complex, which is more diversified and apparently less migrant than their sister species fiom Western and Central Africa . The four foci selected (out of six) are discussed below, with emphasis on current developments and achievements thus far at each site. JAF11.10 Page 27 3.1.12.3 The Itwara focus (in Western Uganda) was taken over by APOC from GTZ (German cooperation) in 1998 for "completion of vector elimination." GTZ had successfully carried out vector control operations on the main focus area during the period 1994-97. Adult S. neavei s./. has not been observed since 1997 in the main focus, and the vector is considered eliminated. In the sub-focus area,by 2002 annual rounds of selective treatments had progressively brought fly- and larvae occurrence down to zero. Monitoring is now confined to three sites. 3.1.12.4 ln the Mpamba Nkusi focus (in Uganda), where no vector control had ever been attempted, the national plan for feasibility studies was accepted in August 1998, but because of delays in APOC funding, annual larviciding started only in 2002.The vector has not been recorded in peripheral areas since 2003. Monitoring is still being carried out after the last larvicide campaign ended in March 2004. Discussions at TCC 20 approved APOC's budget request for the 2005 campaigrr. 3.1.L2.5 In the Tukuyu focus, the first larvicide round took place at an inappropriate time (during the rainy season in 2001-2002 instead of the dry season in 2001), and flies reappeared a few months after cessation of larviciding. Although the second round was apparently properly performed, again biting flies reappeared shortly after the campaign. No appropriate action was iaken either in Tukuyu or in Ouagadougou for analyzing the results and planning for the next round--with the result that no treatment was carried out in 2004. The 2005 campaign had to be planned in a hurry following strong intervention by the TCC and APOC management. Treatments were to start by July 2005. 3 .l .12.6 In the Northern range of the Bioko island, after a ground larviciding campaign in 2001 , it was concluded that it was not possible to reach breeding sites fuIly. Aerial larviciding covered the whole island in March 2003. However, biting flies could not be fully controlled; and recrudescence was observed by July 2003. The 2004 campaign could not take place due to shortage of insecticide; instead, larviciding was carried out from February to March 2005. The fly has not reappeared since March, according to reports of the intensive monitoring undertaken throughout the area covered by aerial larviciding. Other Oncho-related vector control octivities Reasons for delays in implementation of activities include: Ecological and environmental difficulties of black-fly elimination, especially S. damnosum s.l; Project management at the country level has not been satisfactory for technical, administrative or financial aspects; and APOC's burdensome administrative systems, when routinely applied, are not suitable for operations needing flexibility and quick responsiveness. TCC could not always give vector elimination projects (VEPs) the priority they deserved. As a result, TCC has somewhat "under- reacted" to reports of slow progress, and has not made innovative proposals to resolve the problems faced. 3.1 .12.8 Results from the Itwara and Mpamba-Nkusi focus confirm that S. neavei elimination is an achievable objective. However, elimination of S. damnosum s.1,. which is notoriously more difficult, remains uncertain. 3.1.I2.9 Action to be considered The highest health authorities in countries implementing vector elimination programmes should be invited to take stock of the entomological knowledge brought in or supported by APOC, wrth the aim of developing of a national operational capacity, not only for 3.1.12.7 a a a JAF11.10 Page28 onchocerciasis control but also for transmission control of other vector-borne diseases which j eopardize development. Without embarking on nuisance control operations--which are not within its mandate--APoc should provide technical assistance to Participating countries intending to carry out such operations on their own. Recommendation 12 (a) On vector elimination, the evaluation team recommends that in Bioko the programme maintain entomological monitoring over another two-year period. Should biting Jly reappear during this period, a protocol should be worked outfor a national project. (b) In Tukuyu, APOC should ensure that optimal technical assistance is provided to the project, both for treatments and for entomological evaluation. By the end of operations, the procedure should be similar to that in Bioko. (c) In ltwara the pending question is that of the certification; TCC has approved the request and also recommended a publication on the ltwara success story. The TCC suggested that an ad hoc meeting ofexperts be convened in 2006 to decide about the relevqnce and procedure ofsuch action. The evaluation team endorses this recommendation. (d) In Mpamba-Nla$i, it is recommended that one more annual larvicide rounds be performed, and intensive entomological monitoring be continued. Considering that at the end of 2005, the Msmba-Nkusi will be the only APOC-supported VEP showing good sign of repeating the ltwara success, and that the tuvo projects could serve as pilotsforfuture elimination of S. neaveifoci in East Africa, APOC should give the projects all the support needed for a successful conclusion, nohvtthstanding the 2005 deadline for completion of larviciding. 3.2 Programme management 3.2.1 Programme governance Achievements and issues 3.2.1.1 Preparation of and participation in the CSA, JAF, TCC and other meetings within or outside the APOC institutional framework, are heavily taxing APOC staff. Many of these meetings seem essential, but the current governance structure and modalities place a severe strain on the programme's limited human resources, not to mention the expenditures involved. Reflection and change must come from within: the Executing Agency and Programme management need to steer the course toward suitable rationalization. Recommendation 13 In streamlining APOC governance, the Programme monagement, with the support of CSA, and JAF as required, should seek a rational reduction of statutory and other meetings, thus alletiating part of the heat'y'vvorkload of APOC Headquarters stalf. Role of CSA and the TCC in enhancing, consolidating, and ensuring sustainability o.f Programme operations a JAF11.10 Page29 3.2.2 Role of CSA 3.2.2.1, CSA is composed of representatives of four "sponsoring Agencies,"s and a representative of the NGDO Coordination Group as well as a representative of the sole donor of ivermectin (Merck & Co, Inc.) are "invited" to participate in CSA sessions--thus bringing together representatives of APOC"s partners at the international level. Meeting at regular intervals during the year, CSA reviews APOC's plans of action and budgets and examines reports before forwarding these to JAI' with its observations. An important function performed by CSA is the selection of TCC members. ln practice, under the strong impetus of the World Bank as Fiscal Agent of the Programme and WHO as the Executing Agency (which also appoints one of its senior staff as the Director of APOC), the CSA maintains the course set by JAF. 3.2.2.2 At CSA's 109th session at Geneva in March 2005, sustainabilityof projects, a cornerstone of CDTI, featured prominently in discussions. The focus on sustainability is critical as the Prografilme further evolves, and an increasing number ofprojects reach the phasing-out stage. [n this context, the presence of resource persons to take an active part in CSA debates on sustainability should be considered. These persons could be invited on an ad hoc basis, i.e. without changing the formal Memorandum of Agreement. One example of a possible resource person would be the NOTF representative of a Participating country, on a rotating basis, when sustainability or closely-related items are on the agenda. By so doing, NOTFs could bring to CSA's attention, in the light of APOC's gradual phasing-out of financial support for projects, the actual funding gaps that will remain for undertaking activities essential to sustainability (such as training and retraining, EC, advocacy, etc.). The evaluation team believes that the CSA is the appropriate body that could ensure the sustainability of the prograrnme in general, and of specific projects supported by APOC. 3.2.2.3 Although it has been repeatedly stated by CSA that there will be no more call on donors for onchocerciasis control beyond the existing APOC Trust Fund, the team believes that CSA could play a role either in authorizing the utilization of the existing budget beyond the initially-accepted APOC life span, or in initiating fund-raising aimed at other categories of sponsors (e.g., those interested in PHC, integration of health services, or CDTI promotion, for example). The CSA could also support multi-country initiatives by a coalition of APOC (and ex-OCP) Participating countries directed to specific programmes such as those devoted to neglected diseases or poverty alleviation in developing countries. 3.2.2.4 Action to be considered CSA, as long as it is in existence and with the post-APOC era in mind, should play a role in maintaining interest of specific sponsors and in prompting or supporting multi-country initiatives directed to programmes of relevance to onchocerciasis. To respond better to the changing challenges CSA should critically examine its composition and functions to reflect better interests of various partners. Membership should be enlarged to include representatives of countries, and other institutions like the African Development Bank. It is likely that an expanded role of CSA, involving poverty relief and neglected diseases would be of more interest to LINDP and FAO. 8 The World Bank and WHO, together with LINDP and FAO, are referred to as the four "sponsoring Agencies." Over time, one sponsoring agency (FAO) reduced its involvement in APOC (compared with the role it had played in OCP), and finally withdrew from CSA. LINDP interest has also weakened considerably since the inception of APOC. O a JAF11.10 Page 30 Recommendation 14 To enhance e/fectiveness and quality of its deliberations on sustainability, the CSA should consider inviting resource persons, possibly representatives of NOTFs on a rotation basis, to participate in deliberations on relevant agenda items. To alleviate the heavy work load on Programme management, the CSA should rationalize and reduce statutory and other meetings. 3.2.3 Role of TCC 3.2.3.1 [n7996, the APOC Programme Document for Phase I specified that "the main functions of TCC will be to review applications for funds for ivermectin distribution projects, including both their technical justification and financial feasibility, and to review the implementation of funded projects. The TCC will also be expected to advise the Programme on technical and research matters". At that time, TCC was comprised of 10 members: 5 independent scientists, two representatives of the NGDO Group, one representative of The Carter Center, one member of the Ivermectin Experts Committee, and one member of the Expert Advisory Committee of OCP. 3.2.3.2 The external evaluation report (2000), stated that the TCC's "overload mirrors the APOC overload at all stages before, during and after the review process". It recommended that TCC's role be urgently re-defined to align with its technical mandate; and suggested that APOC Phase II would be a suitable opportunity for formal restructuring of TCC. In 2001, following the evaluation, the Programme Document for Phase II stated that "APOC will realign TCC focus to that of technical, implementation and operational research considerations". It also indicated that the composition of TCC would be enlarged to twelve "scientists/experts" appointed by the WHO/DG upon recofirmendation by CSA, one of the twelve being selected by Merck & Co, and three of the remaining eleven being proposed by the NGDO Group. There are presently 1l members of the TCC. The evaluation team believes they are persons knowledgeable in onchocerciasis control and CDTI in APOC countries, are highly dedicated to the success of the Programme, and are very careful in fulfilling their statutory tasks. fssues ofconcern 3.2.3.3 TCC has now been relieved from undertaking financial evaluations; and the processes for conducting technical reviews have been standardized. However, the number of projects in APOC has increased significantly, which also entails the need to review many more monitoring and evaluation reports. Consequently, TCC's semi-annual sessions are stilI mirroring the notorious overload on APOC management. For example, at its 20th session in March 2005, TCC reviewed 29 technical reports, 8 new project proposals, and 10 operational research proposals. Because of this "routine" overload on the TCC, its members have very little time for prospective and in-depth analysis and reflection on topics such as strategic and operational research that are crucial for the future attainment and sustainability of APOC's objectives. As one example of this, the evaluation team notes that the TCC was able to include a discussion of sustainability only at its 21st session in September 2005. 3.2.3.4 To some extent, TCC's overload has impaired its ability to demonstrate initrative and innovation, as well as its capacity for in depth scientific analysis. The team believes that in the future, TCC's involvement in guiding APOC's strategy and its implementation should be more thorough and effective, especially since more attention and foresight is needed in dealing with such issues as decentralization and capacity building. As mentioned earlier, TCC's role in guiding operational research, by commenting on projects initiated and conducted by TDR, also needs to be strengthened. Although some JAFl1.10 Page 3 1 TCC members actively participate in country field visits, and in evaluation and impact assessment missions, the TCC tends to be viewed, at project level, as part of APOC management. Its role as APOC's pre- eminent Technical Consultative Committee needs to be strengthened and better recognized. 3.2.3.5 Action to be considered The scientific standing and capacity of the TCC should be reinforced through the process of selecting and renewing Committee members; and experts in community development should be included. Given the need to have a more participatory approach in the planning and management of APOC's activities, and the current heavy workload of TCC, NOTFs should seek support from a network of expertise at the country level as needed, to enhance local ownership and provision oftechnical support. Recommendation I5 (a) TCC should give more attention to issues crucial for the future of APOC (e.g., CDTI sustainability at various levels, and integration), and should regularly include a working session on these topics at its annual meeting. (b) TCC should reinforce its proactive role in encouraging proposals for new research activities, evaluation of protocols coming from outside, and follow-up of on-going research projects, especially those conducted with APOC's partial- or full financial support. 3.2.4 APOC headquarters office: adequacy and effectiveness of programme management, financial management, and staffing 3.2.4.1 The evaluation team believes that the overall functions of APOC headquarters--mainly administration of the programme; launching, follow up and evaluation of projects; support to countries; support to and attendance at meetings of statutory bodies--are carried out efficiently. APO C Programm e management 3.2.4.2 An important issue noted by the evaluation team was weakness in communication between hierarchical levels of the programme structure, and with APOC's external partners. Some reports prepared by APOC headquarters (with TCC's concurrence in some cases), refer to instances of inefficient project administration, and late submission and other shortcomings of project proposals, technical reports, and sustainability plans. On financial aspects, APOC reports note inadequacies in financial management despite clear written procedures, delays in submitting monthly financial reports, uneven quality of statements and attachments provided, and in some instances unauthorized expendilures using funds Ieft over under a prior Letter of Agreement (LoA) before the signature of the next one. 3.2.4.3 On the other hand--from the projects'perspective-the evaiuation team heard widespread complaints regarding the perceived slowness at APOC headquarters in approving and transferring funds, thus affecting implementation performance. Other complaints were that some projects have remained unfunded for up to several months, some of the requirements of APOC reports and documents are too bureaucratic and onerous; equipment purchased is not always suitable; the rationale for APOC's cuts in project budgets requested is not always clear, management decisions by APOC are made without sufficient consultation with field staff; and there is lack of feedback on reports sent, especially those related to financial reports, often leaving the staff in the field somewhat bewildered and unable to draw conclusions. Some NGDOs felt that funds were often not forthcoming in sufficient amounts, and might have been earmarked for the future extension of APOC. JAF1l.10 Page32 3.2.4.4 In a number of specific cases inquired into by the team acceptable clarifications were received from APOC headquarters staff, and there is reason to believe that early corrective action is being taken. For resolving some of the major pending issues related to financial management, regular supervisory visits by headquarters staff would help. Recommendation 16 APOC should intensifu communication with project staff at various levels and with partners. This could be done, for example, through enhanced field visits to promote a shared understanding of problems encountered and their possible solutions. 3.2.5 APOC financial management 3.2.5.1 Financial management covers, inter alia, accounting for headquarters operations; annual budget preparation and budgetary control; treasury, including receipts and payments; preparation of monthly and annual financial statements; financial analyses; disbursements of funds to projects; contribution to the preparation of annual LoAs for all projects; and control of projects' monthly expenditure submissions. The team believes that for undertaking these tasks, the financial staff at Ouagadougou is of excellent professional quality and is highly motivated. Achievements 3.2.5.2 Some measures for improving financial management have recently been planned, including additional training of staff at headquarters, more country visits to monitor and train personnel at country offices and projects, and financial management reviews. These measures, if and when implemented, would help reduce the complaints voiced by both APOC headquarters- and field staff. 3.2.5.3 Internal and external audits, lasting two to three weeks each, are carried out every year, usually by 3-person teams. Reports of internal monitoring are made available to the auditors; and APOC management is expected to respond to all audit queries. As an example of management's responsiveness, following-up on earlier internal queries, APOC has developed guidelines titled "Management of APOC Funds by NOTFs: Financial and Administrative Procedures (2004)" and has disseminated these to projects. 3.2.5.4 To reinforce its administrative and finance network, and to enhance decentralization of certain functions, APOC covers the costs of six administrative and hnance assistants. These staff are employed by the respective WHO Country Offices, are trained in APOC Programme matters, and remain in close liaison with APOC. They function in the framework of ongoing relationships between APOC headquarters, NOTFs, and WHO country offices. Normally, the work of these officers include such functions as: o Training/retraining of project staff and other capacity building measures in the field o Following-up on the processing of expenditure statements by APOC headquarters o Conducting intemal operational and financial audits of NOTFs and projects o Assisting in the preparation and processing of Letters of Agreemeut o Assisting the planning and budgeting of NOTF- and project activities . Control/supervision of contributions by governments and NGDOs, within the fiamework of the established responsibility for NOTFs to cover 25"h of the cost of ivermectin distribution projects. JAFI1.10 Page 33 Issue of concern 3.2.5.5 The evaluation identified a number of problems related to financial management: . Some administrative and financial assistants in the field attended to a wide range of responsibilities, while others only performed ad-hoc functions, such as the review of project expenditure statements. o Effectiveness of the financial management system appears to be affected by staff shortages at Ouagadougou, as well as, in some cases, by lack of language skills. o Continuing backlog of statements to be reviewed and cleared (a backlog of some 170 project expenditure statements with up to five months delay was awaiting review and clearance at the time of the team's visit). Reviews of monthly project financial reports seem cursory, and on a sampling basis; feedback provided by APOC is insuffrcient. o Delays in approval of LoAs sometimes leading to delays in disbursement of funds to projects and in project implementation o Training and supervision of country- and project-level personnel needs to be intensified. o There are bottlenecks in the flow of funds from the source (World Bank) to the ultimate beneficiaries (the communities). Complete and accurate financial reporting upward along this chain remains problematic. 3.2.5.6 The evaluation team discussed the above issues and their possible solutions; and concluded that in view of the limited time available for its work, more in-depth analysis of the impact of current arrangements and potential solutions is needed. Actions for implementing recommendation 17(a) should be consistent with steps taken as follow-up of recommendation 18(b) (see below). Recommendation 17 (a) APOC should conduct a detailed review of its financial management systems, possibly with help fronx WHO headquarters staff or outside consultants. The objective would be to strengthen the organization's financial management, aiming inter alia at improving oversight of country/project activities, and reducing delays in processing of expenditure statentents, disbursements, and letters of agreement. In view of the close relationship between work in this area and management of APOC staff, the implementation of this recommendation should be done along with that of Recommendation l8 (b) below. (b) APOC should clearly de/ine the responsibilities of all its .financial and administrative assistants stationed in LItHO country offices, to cnsure they can play a role conducive to decentralizing certain APOC functions, and to alleviate the qdministrative burden on APOC headquarters in these matters. In consultation with l4/HO, APOC should also consider expanding the responsibilities and authority of such fficers, inter aliq to screen and recommend approval or otherwise of project expenditure statements, advise on operational budgets, take part in operational audits and reviews, and provide training. 3.2.6 APOC staff management 3.2.6.1 There were 5 staff members when the Programme was launched in 1996; and 20 in 2000 when the evaluation was conducted. Until the end of 2002 (i.e. when OCP ended), APOC was relyng on OCP for information systerns and for administrative support (including personnel, budget and finance). A restructuring carried out in 2003 was aimed at improving management of APOC headquarters. In 2003, 32 JAFl1.10 Page 34 local staff were added: of whom, 25 were taken over from OCP upon its winding down, and 7 were new hires. From January 2005 onwards, all 32 APOC staff have become full-fledged WHO staff on fixed-term contracts. With this change in status, working conditions are now better and security of tenure has improved, with beneficial implications for continuity and staff morale. In 2005, there were a total of 66 approved positions at APOC headquarters (14 Professionals, and52 General Service staff), with 6l staff already in place, and 5 vacancies to be filled. 3.2.6.2 Staff now appears less overwhelmed with the workload, though there is still some concern about the quality of the ouput. In recent years, more attention of staff has been given to effectiveness and consolidation of APOC itself. More attention has also been given to capacity building within APOC Units, for example increasing their ability to produce computerized data. Issue of concern 3.2.6.3 Nine of the 14 professional posts (one was vacant at the time of the evaluation), are occupied by staff scientifically- or technically well qualified, working under a WHO short-term status (STP), to whom the more-costly fixed-term contracts have not been given for budgetary reasons. Keeping staff under STP contracts year after year, apart from being a source ofinsecurity for the staff, is bound to enter into conflict with WHO policies--according to which, after 4 years, STP contracts must be converted into fixed-term status or the incumbent's STP contract must not be renewed. Unless some exceptional ad hoc formula is agreed upon between APOC and WHO Headquarters (which seems unlikely), there is a serious potential problem in staffing just when APOC activities are their peak and a full complement of well-qualified and knowledgeable staff is required for maintaining programme performance. 3.2.6.4 The evaluation team has considered whether staff increases are needed to handle existing workload, especially of financial management staff. The fact that the programme is at an advanced stage and will soon develop plans for transfer of activities to the Participating countries, posed a dilemma: on the one hand, it may appear late to absorb new staff so close to a phasing-out period when the size of the organisation should gradually be decreasing; and on the other hand, some additional staff may be appropriate because specific and considerable effort is now needed for properly transferring programme activities to countries. The evaluation team thinks that the various options need to be examined in detail. Recommendation 18 (a) APOC and WHO headquarters should seek a responsive solutionfor retaining STPs whose contracts cannot be converted into fixed-term stqtus, including ways that permit the continued employment of es s entia I pr ofe s s i o na I s taff. (b) APOC should conduct a detailed review, possibly with help from WHO headquarters staff or outside consultants, of the adequacy of staff to implement the recommendations of this evaluation report; and should takefollow-up actions, as appropriate 3.2.7 The country level Programme management: current and prospective 3.2.7.1 NOTFs are the cornerstone of programme management at the national level. They bring all stakeholders together as an essential partnership for establishing major policies and for pianning programme implementation by their secretariats (the NOCPs). The NOTFs consists of staff of the Ministry of Health JAF1l.10 Page 35 (normally in the chair, at a senior level), NGDOs, WHO country offices, the NOCP, some MoH heads of departments, and resource persons, as needed. 3.2.7.2 With the rapid increase in the number of projects undertaken in recent years, the volume of work for the secretariats has increased immensely. Some NOCPs appear to be too tied up with routine work, such as writing reports and coordinating monitoring- and evaluations exercises, with little time left for the development and oversight functions. APOC headquarters has set up projects for supporting NOTFs; and most of these have decisively contributed to improvements in management of country-level activities. lsszes ofconcern 3.2.7 .3 There is as yet no clear plan for decentralising functions from APOC headquarters to the country level. It was understood from APOC management that attempts to decentralise selected functions on the lines recommended by the 2000 evaluation report had not succeeded. The team would like to reiterate that recommendations for decentralization should not be interpreted to cover everything from policy directives to strategies, plans and activities. Plans for decentralisation should be drawn out in a participatory manner. It might also be useful for APOC to prepare a review paper, based on available information, that will clarify some basic issues on decentralisation. For example, to which mechanism(s) at the country level can APOC decentralize, and why? 3.2.7.4 The pace of decentralization will vary in different countries, according to their technical and administrative capacity. Thus, in the selection for the initial NOTFs for intensification of decentralization, priority should be given to those that are likely to generate practical experiences for other countries. Candidate areas for decentralization include evaluation, monitoring and supervision of projects under implementation; review and approval of technical and financial management; and endorsement of research proposals, and their subsequent monitoring and supervision. 3.2.7 .5 Other issues related to the NOTF mechanism and its future are discussed under the section titled Partnership (Section 4 and recommendation 22 are relevant). Recommendation 19 (a) APOC management should prepare a position paper on decentralization of selected functions from Ouagadougou to the country level. The paper should draw on the experiences of other programmes and agencies on what can be expected. This paper would be an iruportant input to the participatory planning work suggested below. (b) APOC, in liaison wilh NOTFs, WHO country Offices, qnd other portners, should develop a plan for transfer offunctions to countries. These plans should define areas of activity to be transferred and ways for doing so, including the role of NOCP/NOTFs, WHO country Offices and other partners, as well as time schedules .for achieving an effective transfer before APOC is closed. I|'HO support to the process should be provided in response to individual country needs. The plan should incorporate a learning-by-doing approach, beginning with intensiJied support to a .few countries whose experience could be o source ofinspiration and practical use to other countries. In preparation/br the transfer of functions, APOC should advocate with governments to examine their role at each administrative level in the country, and identtfi progrqmn'Le-related qreqs that need to be strengthened. In general, APOC should establish programs to help NOTFs sensitize, (c) JAFl1.10 Page 36 advocate with, and support governments in an effort to pool experiences under APOC,. and also help determine the government budgetary allocations and other funds needed for undertaking onchocerciasis control at dffirent levels of the health system. Funding at the country level 3.2.7.6 The Memorandum for APOC Phase II (2002-2007) and the Phasing-out Period (2008-2010) stipulates that NOTFs will be responsible for 25o/o of the ivermectin distribution costs (in cash or in kind), which will not be available from the APOC Trust Fund. No formal rules or agreements, other than what is included in the original "Project Proposals" issued at project inception, have been established among the various parties spelling out their respective obligations or the specific modalities of the contributions to be made. Funds are usually provided on the basis of ad-hoc and arbitrary decisions; and it appears that only in rare cases are mechanisms in place to ensure or verify that the 25o/o stipulation for NOTF contribution is being adhered to. 3.2.7 .7 Very few records of contributions made by governments, NGDOs and other local partners are kept at APOC or NOTFs to facilitate a coordinated and regular follow-up on compliance with the ApOC Agreement. The evaluation team believes the requirement of 25o/o contrlbution by NOTFs needs to be reconsidered. To enhance sustainability, the contribution of the government to CDTI should gradually increase as that of APOC decreases, so that by the fifth year the government contribution becomes 1002". Thus by the 3d year of a project, the govemment should be cortributing about 50Yo of the total project budget. The issue is 50% of what level of budget, recognizing that much higher levels of budget arejustified during the development phase of projects. In this respect, data from the Independent monitoring of sustainability plans is revealing. First, levels of sustainability budgets provided by APOC are modest. One of the four projects evaluated had received APOC funding; the other three had received no funding for two years. All projects were found to be doing well; and each had received increased funding from the district level. Both geographical and therapeutic coverage remained high. However, there was concem in one of the three projects which had not received APOC support as to whether its achievements could be sustained for a long period. 3.2.7.8 Data on contributions of the countries visited by the team was scanty. It would be appropriate for governments to envisage covering or enhancing expenditures for such activities as training and capacity building; measures connected with CDTI, including its administrative requirements; and transportation of CDDs or NOTF staff through the provision of vehicles, motorcycles or bicycles. Support for CDTI is increasingly being requested and provided under different funds--including those related to Highly Indebted Poor Countries (HIPC; for eligible countries) and Sector Wide Approach (SWAPFand is a welcome development. 3.2.7 .9 Actions to be considered In countries becoming eligible for participation under the Heavily Indebted Poor Countries (HIPC) debt reduction programme, governments should be formally urged to include the onchocerciasis Programme in the list of funding priorities. Similarly, health lending under any Sector-Wide Approach (SWAP) project should be designed to include funding for the onchocerciasis programme. Recommendation 20 APOC management should prepare an anctlytical position paper for review and decisions by TCC and CSA on issues related to the NOTFs'responsibility for covering 25% of a project's JAFt 1.10 Page 37 ivermectin-dtstribution costs. Meanwhile, efforts should be stepped up by APOC in keeping track of contributions by governments and NGDOs in compliance with their obligations under the APOC Agreement. APOC should develop and implement, together wtth NOTFs, a mechanism whereby planned and actual contributions by local partners can be better monitored. This mechanism should produce triggers for corrective action on the part of NOTFs and, if needed, by APOC, such as targeted advocacy measures within or outside the country. 3.3 Integration of CDTI into overall health services 3.3.1 lntegration here refers to incorporating elements of CDTI into the country's health system. There are two expected benefits from integration. First is improved efficiency through synergism and economies ofscale in the use ofscarce critical resources. Second is convenience to users who can get several services in the same place and/or concurrently. In practical terms, integration enables as much fusion of services as possible at the level of the individual user. Both benefits increase the likelihood of sustainability of CDTI. CDTI and the health system 3.3.2 CDTI has been used as an entry point for Primary Health Care (PHC), meaning that the oncho- control infrastructure has been graduallyused to provide other services (such as treatment of malaria, e.g., in Sudan where the health infrastructure is practically non-existent). Integration in terms of combining selected components of other programmes with CDTI is another area of considerable achievements. Examples of such "add-ons" include provision of Vitamin A supplements, and prevention and treatment of filariasis. Other areas being introduced or under discussion include home-based management of malaria; prevention and treatment of schistosomiasis, intestinal worrns, and HIV/AIDS; disease surveillance; EPI; blindness prevention prograrnmes ; and reproductive health programmes. 3.3.3 The 2000 evaluation observed that while direct CDTI line managers were well informed, heads of related sections in ministries (e.g. Primary Health Care) in some cases knew very little on CDTI. Evidence in the field showed considerable improvement. Highest authorities in MOH as well as senior managers were interested in and knowledgeable on CDTI. Similarly heads of Departments particularly Primary Health care participated actively, sometimes as Chairperson for NOTF. Also, district health staff played an active role in CDTI. The CDTI staff in most of the facilities visited had other responsibilities, which is good for integration and sustainability. Community level 3.3.4 The current piecemeal-approach in the delivery of health care at the community level, with each agency or programme delivering its own package, makes it practically impossible for communities to integrate their own activities with those being directed from the outside. Even studies aimed at finding ways of enhancing "integration" come with their own packages to test which additional interventions can be added without rocking the boat. They forget that integration at the community level is not about selecting this or that intervention but about arriving at an effective and efficier-rt mix of services that can be combined with the community's own activities to improve health. JAFl1.l0 Page 38 3.3.5 To assess the extent of CDTI integration at the National and State/Provincial levels, APOC has developed and uses a quantitative tool based on responses to three questions.e The questions are whether "there is a written work plan and if it shows how implementation of activities in an integrated manner will be achieved"; whether "staff combines CDTI activities with those of other prograrnmes, where this is relevant"; and whether "staff combine two or more tasks on a single trip." Findings show that few plans specify how integration of services would be achieved. About half of the twenty-nine projects evaluated, mostly in 2003, achieved a level of integration considered by APOC as "adequate" to enhance sustainability (a score of 25+ out of a possible 5.0). Many pending integratton issues are outside CDTI 3.3.6 The lack of integration strategies and plans for integration on the lines indicated above and related findings in the field indicate that persisting issues for integration of CDTI are beyond CDTI. These issues fall to the overall health services, to MoHs, and sometimes even outside MoH, to donor agencies and intemational organizations. Recently, APOC has supported high-level meetings of programme managers in three countries involved in community interventions. This is a promising development. In one of the countries, the CDTI strategy was adopted as a national policy for all community-based programmes in the country. While this approach straddles the border between APOC and MoH mandates, it is likely that the momentum being built is unique. In the past, such integration could not be achieved by many programmes, including Primary Health Care whose progress was derailed by such technical aspects as developing essential health packages. The new APOC initiative has the potentials of being one of the most important contributions of APOC and CDTI to health systems, but it has to be nurtured. Impact of CDTI on the quality of local health services 3.3.7 CDTI has contributed a number of approaches and tools that have enhanced the improvement of health services. A number of other prograrnmes are also using tools developed under CDTI, modified as necessary. Examples include tools and instruments for monitoring and evaluation. The training of medical personnel and CDDs has contributed to improvement in quality of care. Nurses in the field indicated to the evaluation team that CDTI had given them a better understanding of communities and their needs, which has enabled them to improve services to communities. 3.3.8 Actions to be considered In view of the many operational problems being encountered in the integration of health services, Ministries of Health, together with partners, should identify critical problems and carry out operational studies in selected geographical areas (including districts and communities) to find solutions. APOC and NOTF should intensify effort to make CDTI experiences and lessons readily available to other programs. Each NOTF should organise a special meeting on integration of CDTI. The meetings should take stock of achievements to date as well as issues and define future strategies and activities. In view of 9 Parameters assessed included whether "There is a written work plan and if it show,s horv implernentation of activities in an integrated lnanner will be achieved"; "Staff combines two or more tasks on a single trip such as monitoring/ supervision, training for CDTI, HSAM, fctching records and delivering of Mectizan"; and "Staff cornbines CDTI activities with those of other programmes, where this is relevant (e.g. supervision, training, HSANI, Mectizan delrvery)." a a 5 aJAF11.10 Page 39 the importance of this item, it would be useful for APOC management and or a member of TCC to participate in the meetings..'o APOC should take stock of experience gained in CDTI integration in all participating countries and make lessons learned widely available. Recommendation 2I As overall integration of health services is beyond CDTI, APOC should encourage Ministries of Heath, in collaboration with partners (including donor agencies), to find ways of enhancing integration of health seryices and developing an appropriate strategt and plan. 4. Partnership 4.1 The various partners in synergy at all levels The APOC Programme rests on a wide partnership at the international and national levels, each covering one or more of the following areas, all of which ultimately promote sustainability: funding; planning; programme development, coordination, supervision and monitoring; health education, sensitisation, advocacy, and community mobilization (HSAM); training, re-training and capacity building in general; and research. 4.2 The international level 4.2.1 APOC has an international partnership network largely inherited from OCP (see Introduction). Several of the international partners are likely to phase out and finally withdraw when APOC winds up. But some assets are likely to remain. WHO is expected to continue to play a prominent supportive role, notably through the Regional Office for Africa (AFRO). Efforts are being made by AFRO to integrate onchocerciasis into their regional agenda, and to gradually cover programme needs for surveillance, oversight, and exchange of information. AFRO already keeps up-to-date with APOC activities, participates in CSA, and is invited to TCC and JAF. Also, within its "Integrated disease surveillance strategy," AFRO has included onchocerciasis in its priority list of communicable diseases for surveillance. 4.2.2 Furthermore, it is expected that the WHO Multi Disease Surveillance Centre (MDSC) established in Ouagadougou, which inherited the regional surveillance infrastructure that OCP had developed over the years, will gradually join the APOC partnership, and will define a post-APOC role for itself. Other United Nations organizations, such as IINICEF, may contribute in ways yet to be determined. Finally, on the intemational level, though the contributing parties will dissolve as an institutional group, resources may be sought from individual donors prepared to adopt bilateral strategies of assistance to CDTI in specific countries, or focus their assistance on particular target populations. 4.3 The national level 4'3 'l Within a Participating country, programme activities rely on staff of the Ministry of Health (MoH) and on providers of health services at different levels throughout the country--down to First Line Health r0 A second item that might be taken up by the special rneeting is the Future of NorFs. JAF11.10 Page 40 Facility (FLHF). There are several important mechanisms at the national level: National Onchocerciasis Task Force (NOTF), and its operational branch, the National Onchocerciasis Control Programme (NOCP); the NGDOs; country offices of international organizations; and others involved in making CDTI effective, such as churches and denominational missions, a few national NGDOs, as well as the communities themselves. NOTF/NOCP 4.3.2 While NOTFs function under government chairmanship, as a high-level coordinating body and a meeting point and forum for partners and stakeholders they are not a standing body, but instead work through periodic meetings. ln order to give concrete expression to their deliberations and conclusions, they must fully rely on their operational arms, the NOCPs. Issue of concern 4.3.3 The 2000 evaluation had already called for urgent rethinking on the future of NOTFs. It noted: "It is clearly essential that when APOC ends there should be a mechanism which continues to promote partnership in planning, monitoring and research around onchocerciasis control. The nature of such an arrangement is however far from clear. There is therefore an urgent need for NOTFs to plan for the future of partnership. Since countries and NGDOs differ substantially, it is logical to expect different models to be developed in different situations. Each NOTF should be asked to develop a strategic five-year plan. This plan should indicate whether they see their role as phasing out and gradual integration with MoH, or whether they are going to continue to exist as a partnership, and to support onchocerciasis control activities."rl. There is no evidence of action having been taken to implement this recommendation (to which the evaluation team fully subscribes). NGDOs 4.3.4 Within NOTF, representatives of international NGDOs are active partners with a substantive role. In general terms, they contribute to and have a catalyic effect in various programme aspects. Concretely, depending on the circumstances and needs of different countries and agencies, the NGDOs offer various forms of support to NOCPs and to projects. These include management and financial assistance and equipment; logistics, including facilitation of procurement, storage and distribution of ivermectin; and, as an important contribution, capacity building. In the latter, they deploy substantive activities, especially in the form of training in a wide range of fields: health education, sensitization, advocacy, community mobilization (HSAM); data management and management information systems, monitoring and supervision. tl APOC Repo(: Extemal Evaluation, Septernber 2000 JAFl1.10 Page 4l 4.3.5 The NGDOs involved, in a joint endeavour, meet twice a year in an NGDO Coordination Group for Onchocerciasis Control.r2 Their presence in the field is not linked to whether APOC exists or not. In partnership with governments, long before APOC started its own operations, they were active in health progralnmes. ln a number of countries, they were involved in onchocerciasis, including the ivermectin delivery aspects. For the future, at its 25th session (in Ouagadougou, on 10-12 March 2005), the Group reaffirmed the need to maintain "its role in supporting onchocerciasis control up till and after 2010." Some individual NGDOs have already reasserted their long-term commitment, while the Coordination Group can see a role for increased training and surveillance. ln the Group's deliberations, the need to avoid onchocerciasis becoming a "disease of the past" was emphasized, as well as the need to engage as a Group in advocacy at all levels. Operationally, the evaluation team believes in the importance of NGDOs stepping up their presence even further, to act as implementing partners, including for new projects that might be launched in coming years. Moreover, they should actively participate in future planning exercises for the "after APOC" phase. Financially, each agency has its own resources and funding capacity; and works independently of others in the Group. One agency, though strongly committed for the long-term, clearly stated that it had no intention of filling any post-APOC financial gap. The evaluation team believes that this might be the general trend among members of the NGDO Coordination Group. Issues of concern 4.3.6 Some degree of dissatisfaction has been expressed by some NGDOs at the functioning of the NOTF. [n one country with a large number of projects, NGDOs complained about instances of insufficient coordination with them by NOCPs when sending written communications to other partners, or of NOCP's failure to provide the NGDOs concrete information on government contributions of counterpart funds. In another instance, an NOCP complained about NGDOs sending funding requests to APOC without govemment clearance. 4.3.7 NGDO contributions, in cash and kind, at any level in the countries, are not always presented with accuracy, leaving doubts as to what exactly is being contributed. Some agencies expressed concern about their long-term funding and the motivations and financial commitment of donors in the long-run. Though there is a general will to continue activities beyond APOC, for some agencies their future commitment beyond 2010 is not firm. The NGDOs involved are international. (National NGDOs are not part of--or represented in--the NGDO Group.) It remains unclear to the evaluation team whether individual NGDOs or the Coordinating Group has primary responsibility for identifuing local NGDOs with the right potential (after training and capacity building) for undertaking onchocerciasis control activities after APOC ends. 12 The members of the Group are: Christoffel-Blinden mission, The Carter Centre, Helen Keller International, Interchurch Medical Assistance Inc. (IMA), Lion's Club International Foundation (LCIF), Mectizan Donation Programme (MDP), Organisation pour la Pr6vention de [a C6cite, Sight Savers Intemational, and the US Fund for LINICEF. The only Associate member is Merck & Co., Inc. The Group is open to new members; and interested observers from the health field attend its meetings. JAFl1.10 Page 42 International organizations at country level 4.3.8 Among country offtces of international organisations, WHO has a specific role to play. At present, WHO offices provide various forms of assistance, e.g.: support to APOC and APOC-connected missions; clearing entry of Mectizan into some countries, and forwarding it inland; purchasing equipment; and channelling APOC funds to NOTFs. Also, administrative and finance assistants are out-posted by APOC in some WHO country Offices. ln the framework of post-APOC arrangements, the evaluation team considers that there will be a need for increased support from WHO; and furure requirements will have to be defined as functions are devolved to countries. WHO is well established and deployed in many countries, with no time limitation, and is close to health authorities, not only at MoH but at all levels. Also, many WHO offices are involved in helping integrate field health activities; and in one country visited, they had mobilised CDDs for vaccination campaigns. In the future, the WHO network in the Participating countries, with support from AFRO and HQ, could be extremely useful for organizing, or incorporating into their agendas, inter-country exchanges on experiences of mutual interest in the onchocerciasis field. One other potential partner in CDTI is UNICEF. In one of the countries visited, UNICEF carries out customs clearance of Mectizan; and in some of the countries visited, LTNICEF expressed an interest in participating more in CDTI. In one country, LINICEF is gradually "revitalising" health areas, which should lead to beneficial interaction with CDTL Other partners qt national level 4.3.9 Churches and denominational missions offer various forms of support to onchocerciasis control activities. In one country, a church mission engaged in community development in collaboration with government, is the implementing agency for an APoC-supported project. In other instances, such agencies provide logistical help in manyways, and offer such assistance as safe-keeping of funds, or shelter to staff on mission. Recommendation 22 (") Since effective partnership will remain essential for sustainability of CDTI, and in view of the phasing-out and eventuql withdrawal of one major group of international partners (donors, World BqnUTrust Fund) as well as of APOC, it is recommended that the role, contribution, andwillingness of other partners--current qs well as those yet to join the partnership--be defined clearly and in good time, within the overall objectives of post-APOC activities. In particular, with the assistance of WHO at appropriate levels, APOC should formulate plans for increesed involvement of WHO country offices in CDTI activities. Similar plans should be formulated with any other organisation, such as UNICEF, which could make a substantive contribution to the sustainability of CDTI (b) With APOC qssistance and in close liaison with governments, plans should be preparecl for the continuation, and strengthening where required, o.f NOTFs or similar structures tct ensure their uninterrtrpted activities for as long os necessaty. NOTFs, or any successor stakeholder group, should take the lead in the discussions on future partnerships; and with APOC support, inclttde the international level (such as WHO, UNICEF and other such potential partners) in these discussions, with a view to enlisting as much supportfor CDTI as possiblc. JAF11.l0 Page 43 (c) International NGDOs, who have thus far played an important role in APOC's development, should actively tcike part in future j oint planning exercises dealing wilh the longer-term future of CDTI . They should envisage strengthening their support even further, as required by current and emerging projects. They should, jointly with NOTFr, identify local NGDOs with the potential to act effectively on their own after training and capacity building. (d) Communication among partners should be strengthened through the establishment of country onchocerciasis-websites funded by APOC to enhance information sharing among all stakeholders. (e) APOC management should prepare an analytical paper on issues related to NOTFs'responsibility for covering 25% of the ivermectin distribution project costs. This paper should include issues related to monitoring of sustainability plans, and should be submitted for review and decisions by TCC and JAF. Meanwhtle, APOC should make greater efforts to keep track of contributions by governments and NGDOs, and seek their increased compliance with financial obligations already made. APOC should develop and implement, together with NOTFs, a mechanism whereby planned and actual contributions by local partners can be better monitored. This mechanism should incorporate triggers for corrective action on the part of NOTFs and, f needed, APOC, such as targeted advocacy measures from within or outside the country. (n With the help of APOC, the NOTFs should undertake detailed analyses of partners'contributions, and take strides to ensure that adequate funding is available, including from governments, for the orderly implementation of CDTI in all its aspects, particularly with a view to the phasing out of APOC. 5. Conclusions and the Way Forward Three key findings of the team are that: (a) APOC has made substantial progress in many areas, and its staff and partners deserve commendation for their hard work and sincere commitment to programme objectives; (b) progress towards sustainability of projects is uneven within and between countries; and (c) a number of projects will still be under APOC implementation by 2010, and it would be advisable to continue APOC's involvement, though on a gradually-decreasing scale, beyond that year. This section also suggests preparatory measures that need to be taken for "when APOC is no more." 5.1 Will APOC achieve its objectives? 5.2 There is no simple composite measure suitable for answering the above question. Throughout our review, and particularly in weighing qualitative evidence, the team has examined closely the sustainability of projects and delays in their implementation. For assessing the likelihood of sustainability, the factors considered include the adequacy of: policy; planning and leadership; support of the national health system, integration of CDTI activities with other health services and support systems; adequacy of financing/funding; human resources, monitoring and evaluation, coverage and related aspects of APOC and CDTI systerns. The picfure that emerges from this assessment (as reflected in Chapters 3 and 4) and extensive deliberations of the team , is unrnistakably mixed. JAF11.10 Page 44 Balance sheet 5.3 Review findings show clearly the soundness of the CDTI slatem. Most of threats identified are not new; and considerable ongoing and planned work is aimed at finding solutions to these issues. A big worry is unexpected events like SAEs and delays/stoppage of funding. The findings show that countries with appropriate support systems are able to overcome emerging threats. Overall, our findings, taken as a whole, show that APOC is moving steadily towards its objectives, but still facing substantial challenges to continued satisfactory completion. There is a serious problem of delays, particularly in conflict or post-conflict situations where a number of projects have not started, putting in jeopardythe achievement of APOC's objectives by the year 2010. 5.4 Hence, the evaluation team is convinced that winding up APOC in 2010 would be premature. The proposal for extension of APOC on a decreasing scale beyond 2010 is meant to ensure that the phasing out is conducted in a planned manner with adequate resources, and that all necessary measures are put into place in the next ten years to ensure that APOC, national health systems, and their partners have jointly built adequate capacity to support and sustain key CDTI and related activities after APOC is no more. 5.5 What after APOC? 5.5.1 As outlined in Chapters 3 and 4, considerable effort by APOC has gone into finding ways of improving the performance and sustainability of CDTI projects "when APOC is no more." The main mechanism has been the development of "project sustainability plans" under government leadership and relying on increased funding by the government. However, it can be hoped--but cannot be assumed or taken for granted--that such sustainability plans will continue to remain viable in a rapidly-changing national and global context. The team feels strongly that CSA and JAF need to squarely address this issue of enhancing sustainability of projects as a matter of urgency, so as to put in place innovative strategies for the post- APOC period. 5.5.2 Ensuring the success and sustainability of CDTI will be a local and national responsibility. Governments would have to lead the mobilization of resources from national and international sources, based on sound budget estimates. Appropriate partnership and solidarity among all concerned, as well as collective pressure and advocacy could help ensure that onchocerciasis remains "on the screen" of public awareness and gets adequate funding from the government and other sources. The need for such partnership is now recognized by other programs including the "big three" (HIV, malaria, and TB) and others. Nevertheless, there is always the danger that present and emerging health priorities might relegate onchocerciasis to a position of little importance in terms of national and international funding, thus endangering decades of successful efforts. Hence, the currently-strong "Oncho partnership" will have to continue to provide effective oversight and serving as the flag-bearer for what OCP and APOC have stood for. This type of function will be needed to coordinate efforts in such areas as research, project evaluation, impact assessment, advocacy, and continued exchange of information between countries. The information exchange would need to include such in areas as the effect of CDTI on intemrption of the parasite, SAEs, resistance to ivermectin, new experiences in organizing CDTI and their impact, and incentives for CDDs; 5.5.3 In addition, the team believes a complementary financial rnechanism needs to be devised, at the intemational level, to support the oversight and partnership functions. This will help ensure that OCP's and APOC's core activities of research, dissemination of infonnation, measures to tackle SAEs, and many of the other activities outlined in this report are carried out. Thougli it is premature to propose any precise formula, the team is of the opinion that if broader mechanisms similar to the Trust funds can not be put in place, CSA and WHO might consider the creation of an extra-budgetary "Onchocerciasis Fund". JAFl1.10 Page 45 5.5.4 Before we move into the post-APOC period, however, APOC and its partners need to successfully complete the activities envisaged for the remaining life of the prograrnme, as recommended in this report. APOC should therefore develop a plan of action for implementing whatever recorlmendations of the team are eventually approved by CSA and endorsed by JAF. 5.5.5 The external evaluation team would like to conclude this report with its broad vision of what might exist "after APOC" as follows: "CDTI is fully integrated in health s)6tems; High therapeutic coverage (65+) is maintained; Assertiveness and demand of communities for ivermectin treatment provides/maintains political pressure for CDTI; CDTI is seen as a component of poverty elimination; Adequate budget is provided by government; Solidarity and global interdependence ensures that all eligible populations are getting treatment; There is active support to find and prompt use of new drugs and technologies, particularly macrofilaricides and alternative drugs to Mectizan; Operational research is enhanced; Effective monitoring and evaluation is undertaken, with critical analysis of data and surveillance for equity; More synthesis and dissemination of experience is the norm; and Enthusiasm is generated and maintained at all levels." JAFl1.10 Page 46 l3 Annex 1 Terms of Reference for the External Evaluation @ocument JAF10.10, October 2004) 1. Introduction and Backqround (i) The primary purpose of the African Programme for Onchocerciasis Control (APOC) is to establish sustainable, community-directed ivermectin (Mectizan) delivery approaches covering approximately 90 million people in 19 countriesr3 which fall outside the scope of the earlier Onchocerciasis Control Programme (OCP) in West Africa. These systems are being established to become sustainable 'o by 2010. It is estimated that a minimum of 30 million people living within the APOC countries are currently heavily infected with onchocerciasis. (ii) The formal objectives of APOC approved by the Joint Action Forum in 2001 in Washington reads as follows: "To establish, within a period oI 12 to 15 years, effective and self-sustainable, community-directed ivermectin treatment throughout the endemic areas within the geographic scope of the Programme, and, if possible, in selected and isolated foci to eradicate the vector by using environmentally safe methods. The attainment of this objective is expected to contribute towards the elimination of onchocerciasis as a disease of public health and socio-economic importance throughout Africa and so contribute to improving the welfore of its people. It is worth stressing that the duration of APOC operations is limited in time and that the Participating Countries will assume, without major external support and well beyond the end of APOC, full responsibility for the continuation of community-directed treatment with ivermectin, set in course by the Programme." (iii) The principal tool for controlling and eventually eliminating onchocerciasis as a public- health problem is Mectizan, which is being given free-of-charge by the producer, Merck & Co., Inc., for "as long as needed." One dose of this drug given annually reduces the load of microscopic, larval worms in the human body by up to 95oh, without serious side effects. It thereby relieves intense itching and prevents occurrence of blindness. It is estimated that at least one billion free Mectizan tablets will be distributed over the life of APOC, having a value exceeding one billion US dollars. Angola, Burundi, Cameroon, Central African Republic, Chad, Democratic Republic of the Congo, Republic of Congo, Equatorial Guinea, Ethiopia, Gabon, Kenya, Liberia, Malawi, Mozambique, Nigeria, Rwanda, Sudan, Tanzania, Uganda Sustainability has been defined by an APOC special task force in2004 as follows: "CDTI activities in an area are sustainable when they continue to function effectively for the foreseeable future, with high treatment co\/erage, integrated into available health care services, with strong community ownership, using resources mobilized by the community and the government." JAFl1.10 Page 47 (iv) The total cost of APOC over the l5-year period is estimated to be US$135 million, to be contributed by the donor community and an additional of 25% of this amount are bome by the Non-Governmental Development Organization (NGDO) Coalition and the African countries.The average cost per person treated per annum after establishment of sustainable community-directed treatment with ivermectin (CDTI) is approximately US$0.25. APOC will protect the donor community's substantial investment of US$600 million in OCP. OCP has succeeded over the past 25 years in virtually eliminating onchocerciasis from a ten- country subregion in West Africa. By controlling onchocerciasis in Nigeria, APOC will protect the entire eastern flank of the OCP subregion from re-invasion by infective blackflies, which could otherwise re-establish the disease in the OCP countries. Twenty donors and all l9 African countries signed a multilateral agreement, bringing the program legally into force as of April 1996. Twenty donors have committed US$65 million in financing the Phase lI (2002-2007) and the Phasing-Out Period (2007-2010) program, and I I 7 projects in I 6 countries had been approved for financing by the end of 2004 to alleviate suffering and prevent blindness among a total population of nearly 50 million people per annum. 2. Obiectives of the Review The objectives ofthe external review are To assess progress towards meeting the Program's objectives, including an evaluation of operations, and to make appropriate recommendations on the Program's strategies in order to fulfill its objectives by the year 2010. (ii) To make recommendations on how best to sustain CDTI post-2010 in order to eliminate onchocerciasis as a public-health problem throughout the APOC countries. (iii) To review the program's progress in light of the impact data. 3. Terms of Reference 3.1. Programlmplementation (i) To review and evaluate the performance of the Program to date, with particular emphasis on the Community-directed Treatment (ComDT) strategy and the effect on coverage and sustainability. (ii) To review treatment and evaluate also the performance and impact of focal vector elimination projects. (iii) To make recommendations on actions to be taken in areas where peripheral health systems are weak or non-existent. (iv) To examine the strategic prospects for achieving the Program's objectives by the year 2010. To assess the adequacy of monitoring procedures and to suggest improvements, which might be made. (i) (u) JAF11,10 Page 48 (Yi) To assess the adequacy of the impact assessment of APOC operations and make recommendations for future impact evaluations. (vii) (viii) To evaluate operational research and application of its findings. To assess the role of the Program in conflict/post-conflict oncho-endemic areas and among displaced-person populations. 3.2 Coverage and Sustainability of the Program (i) To review treatment coverage achieved by the Program since its inception, as well as methods to expand coverage further. (ii) To review progress made towards enhancing the long-term sustainability of comDT including the role of partnerships and community involvement and make suggestions for improvement. 3.3. Integration of ComDT into the Health Systems (i) To evaluate progress towards integrating ComDT into existing health slstems in the Participating Countries and, within this context, the utilization of the ComDT system for the integrated delivery ofother health interventions. (ii) To examine APOC's contribution to health-systems development in Participating Countries, particularly the achievements of the Program in local capacity building. (iii) To review the impact of the Program on the quality of local health services, including Primary Health Care, especially in the poorest communities. 3.4. Financial integration of APOC (i) To assess APOC financial management systems and propose ways to progressively integrate them into national health accounts and general national public financial management systems. To review present APoc financing systems and propose alternative ways of financing APOC under programmatic and general budget support, i.e. SWAP and PRSC respectively. (iii) Carry out a program expenditure tracking survey in a proper sample of APOC projects. (iv) To provide a preliminary assessment of the participating governments' willingness to continue financially supporting APOC program operations after 2010 and to propose steps which could facilitate the program integration. (ii) aJAF11.10 Page 49 3.5. Program Management (i) To assess the adequacy and effectiveness of management and staffing at the Program Office in Ouagadougou and the role of the Technical Consultative Committee (TCC) and the Committee of Sponsoring Agencies (CSA) in light of consolidating and ensuring sustainability of Program operations. (ii) To examine the financial management and controls of APOC vis-d-vis projects in the field. (iii) To review program management at the different levels of the national health systems and make recommendations for better integration and effectiveness. 3.6. Partnership (i) To assess the technical, financial, and operational contributions of the partners at the country level, namely the Ministries of Health, APOC, NGDOs and other partners. (ii) To assess the scope of the partnership and how it works at the country level. 3.7. OrganizationalAspects (i) The evaluation will be organized by the Committee of Sponsoring Agencies (CSA) (ii) The evaluation will be carried out by independent experts. The CSA, with input from the donors and the NGDOs, will finalize the composition of the Evaluation team, insuring that the required expertise is represented, and arrange for secretarial support. (iii) The evaluation shall take place during 200412005. APOC staff will provide information to the Evaluation Team, and the Mectizan Donation Program (MDP), the NGDO Group, and members of the Technical Consultative Committee may be invited to act as resource persons where required. (iv) Evaluations of the roles of the various partners, 9.g. NGDOs, Participating Governments, and local communities will be grouped together and separated from the evaluations of APOC pel se in the body of the report. (v) Field visits to the APOC area will be arranged as required. (vi) The evaluation report will be considered in final draft form by the CSA and presented to the December 2005 session of the JAF for adoption. 3.8. Skills Requirement The six members of the Evaluation Team would have the following skills: Epidemiologist, Social Scientist, Entomologist, Economist, Public Health Specialist, and an expert with experience in Administration and Finance. N Xq) CD g EA e!:sffit 5{{ :ii 3tl, IaE IiT{tiri!!tr ilii i33[ 3E!E g (N" 6 i5rqLi ltFA .. ,:i. g1t;r -" It{! I iIii;rE 1l =t(J TI T II E I I Zt I I ! I t E E tt $B IBi IE c t I T 3 i E E Jta F o oFotll ar o ul occt In ra N do U1 CE c) oL o1 o]d 0 9l 0Jl ot r.l ol c.rl ol Crl <l a E T H >! e^E !EE ?EEert IET I I'l EFE :?E tI6aa6!vgt3ia>lJr 0OdcE;o. (D> g" i! E,llld elil ,l,lJ, v. U2) lllJ, JJJJJ, g JJJ, IG Iz UE az 7 ts li, 1 !I Io2 ri elf nl II E a ti { I I I I bltlol l+tx, J;iflflru I l II : a I obbpD 444+ lllir ffiff lflldri l It x{tff I @ 3tt I I I *I 0 t E I 0 q I -E 0 i I at : I I E 0 ct T I C , I E I ,{ -t l t{ T T 3 aI t -l T'IIFG'IIEattll b I it llJ, -bL l1', ilili J,l, !i+ d'ilr t ,i I ,9 II I V I & E I I I ! , a I a , '!- 1 I, a . I t T : I -E 0II it , a 3 Itt =I I I 0 ! I 5 -i I I slJlr ffi illli $ l. ti t I . rit I t; rj L t I I a I v t E I I c I I .t t I I I I :erl er ! g r!lllll t!5! II llrl a I I a ! I Itt -; Ii !: .! i I , ! j l lit 1 8 I l3 $a 0 ;gE,-:L; -,a-t!,') Y<.,o!=o :EO tr6 JAFl1 Page 10 5l Annex 3 a Analvsis of treatment results since APOC's inception (data as of October 11.2005) a Communities Population Coveraqe (%) Country Year Total Treated Total Treated Geographical Therapeutic Anqola 2005 122 46 90 527 6 062 37,7 6,7 Cameroon 1999 2 460 2013 1 379 405 529 967 81,8 38,4 Cameroon 2000 3 387 3 107 1 599 760 727 921 91,7 45,5 Cameroon 2001 5 684 4 069 1 868 278 1 018 117 71,6 54,5 Cameroon 2002 5 852 5 071 2 801 915 1 550 385 86,7 55,3 Cameroon 2003 5 864 5 843 3 161 522 2 482 672 99,6 78,5 Cameroon 2004 7 532 7 489 4 334 712 3 180 769 99,4 73,4 CAR 1999 5 014 4 584 1 375 637 934 158 91,4 67,9 CAR 2000 5 014 5 014 1 415264 1 076 292 100,0 76,0 CAR 2001 5 014 4 594 1 454 891 778 989 91,6 53,5 CAR 2002 5 014 4 682 1 151 864 492 287 93,4 42,7 CAR 2003 5 014 4 653 1 176 572 890 747 92,8 75,7 Conqo 2001 748 425 578 812 228 220 56,8 39,4 Congo 2002 748 468 583 480 195 950 62,6 33,6 Conqo 2003 748 719 569 652 353 281 96,1 62,0 Congo 2004 770 765 580 582 383 302 99,4 66,0 DRC 2001 10 020 2 636 5 536 453 741 370 26,3 13,4 DRC 2002 12 394 4 949 6 516 142 1 816 898 39,9 27,9 DRC 2003 17 507 11 782 I 958 813 4 403 838 67,3 44,2 DRC 2004 15 760 I 469 8 613 655 3 898 234 60,1 45,3 Eq. Guinea 1 999 129 15 63 115 1 236 11,6 2,0 Eq. Guinea 2000 129 68 64 601 7 882 52,7 12,2 Eq. Guinea 2001 129 67 66 122 10 797 51,9 16,3 Eq. Guinea 2002 129 95 67 679 10 874 73,6 16,1 Eq. Guinea 2003 129 Eq. Guinea 2004 129 128 70 903 48 551 99,2 68,5 Ethiopia 2001 3 277 494 902 687 233 309 15,1 25,8 Ethiopia 2002 3 277 2 088 928 691 516 217 63,7 55,6 Ethiopia 2003 5 244 4 250 1 563 635 1 025 859 81,0 65,6 Ethiopia 2004 20 368 16 364 4 965 717 2 960 696 80,3 59,6 Liberia 2000 2 596 281 2 083 435 497 662 10,8 23,9 Liberia 2001 2 596 1 988 2 148 021 900 000 76,6 41,9 Liberia 2002 2 596 1 628 2 214 610 900 000 62,7 40,6 Malawi 1 999 660 342 533 057 228 014 51,8 42,8 Malawi 2000 2 174 553 1 383 824 293 320 25,4 21,2 Malawi 2001 2174 596 1 412 841 310 056 27,4 21,9 Malawi 2002 2174 86'l 1 445 362 473 506 39,6 32,8 Malawi 2003 2170 759 1 477 153 477 103 35,0 32,3 Malawi 2004 2 186 2 186 1 604 453 1 095 530 100,0 68,3 JAF11.10 Page 52 Communities Population Coverage (%) Country Nigeria Year 1998 Tota! 2 531 Treated 1 333 Total 1 605 693 Treated 1 175271 Geographical 52,7 Therapeutic 73,2 Nigeria 1999 20 301 2 890 21 108 420 12 434 247 14,2 58,9 Nigeria 2000 32 530 23 286 23 779 019 15 154 631 71,6 63,7 Nigeria 2001 32 530 28 170 24 618 844 16 539 589 86,6 67,2 Niqeria 2002 35 783 31 801 25 930 91 1 18 389 925 88,9 70,9 Niqeria 2003 35 299 32344 25 170 612 19 511 290 91,6 77,5 Nigeria 2004 34 939 31 482 26 683 209 19 511 819 90,1 73,1 Sudan 1999 311 219 1 316 046 431 329 70,4 32,8 Sudan 2000 549 478 1 384 188 423 824 87,1 30,6 Sudan 2001 2 600 890 1 805720 789 935 34,2 43,7 Sudan 2002 2 600 349 1 877 412 495 970 13,4 26,4 Sudan 2003 326 238 586 625 248 180 73,0 42,3 Sudan 2004 618 357 557 813 253 089 57,8 45,4 Tanzania 1999 322 317 636 769 504746 98,4 79,3 Tanzania 2000 442 433 850 41 1 540 757 98,0 63,6 Tanzania 2001 592 588 1 258 154 806 136 99,3 64,1 Tanzania 2002 592 583 1 311 047 866 527 98,5 66,1 Tanzania 2003 633 633 1 523 578 1 058 352 100,0 69,5 Tanzania 2004 884 874 1 785 133 1 271 453 98,9 71,2 Uganda 1999 3 422 3 422 1 760 358 1 226 494 100,0 69,7 Uqanda 2000 3 613 3 613 1 776 241 1 402018 100,0 78,9 Uganda 2001 3 966 3 966 1 960 935 1 543 926 100,0 78,7 Uganda 2002 3 936 3 936 2 120 609 1632987 100,0 77,0 Uganda 2003 4 186 4 186 2 125 314 1 756 418 100,0 82,6 Uqanda 2004 4 785 4 785 2 416 898 1 761 700 100,0 72,9 D a U;lEIEI <l O c.t -(.)HO -ooG-d <A a a tr o oJ i6 o ooo .ic .6€ =o9'. e5 =Eul 6o o c6fr ot)9trFlr-roo= ke0 '6 0- B3 B6 o ooo b o o Ioo oo c .E . c =o(, ooEg5Q.OEo:ig o= 'q-o o(J o o(, o o oo F6= >f0-q6 !=; H=U Ed- 6EI(J() lrJ G- E" E . >o EEPttloC OG<E b6 E ooo u. uJo r l! o ulo-27\ <Ezo 06 F UI(9 o3o t ul9Lr o lrJ(.) 2 =4 d 6 E6 o ooo ooo I E! 6 o ooo dE .9g oo o .E oo =o 'in{ t oFa<oJ<otZF E,F a6 eooth i,EHA Eg= = 4o o o Tso B G o ooo g 6 = .: .2o-Ao<o .o EE doo ao h( o ooo 6 o ooth =gEr' 2 9Du EUaarr<(2*- -o oft.otr!=2=- ,=d < 5,4E9= =GaDl- -u,r=usi() €= g;ttg =r60f rrJ I I I I I I I dt, . eloE .gE6o doEE >!9o-= E{EO5:EEEh Eo e ooo !2 oo dE tg g b tro 6 6 o !io. =6 =oyotr< @ aD a tr o 6 6o *g o zI F =t oGz =luFo 1r, 9 .!2 oo o Eoo od o E ot o:<: Jo .9 oo o tr ooo ,9 oo I o Go g 6 tr ooo .: .9oo z>o(, - alz =dHFEb =uro EDc oooo = Eos o Ig o E c9otrFrs<E =aEB 8Uot o € o '6 o dE "9trooo(n coooo = oo0oI o o o .!!o !g(, o EEo =tTFo o ,:6E>E6<i .EuEZ E& (, .z o teoo6+ o o EE# oOEgoE6 oc :q60 E ,ctt *oicgOOA Egoa E O,i T E.E a6 o ooo u UJ tt.oz EF r-l6l ()l cl ol EI .!l nt6l hn r-l ol g,l oltrl <t