Report: external mid-term evaluation

\::'i. t t t :... ,-;r.."...i , . .)-:.., -'.':-:: .'."*:i;,'1.i ..'_t_:,,. . ''..'i'.-' i'". +.'' ?.j?. - ,.^ ' :'.: "i ;, /' ,-., .- !' ".,. .ti . ; ,1\,,-.1,| l,i i! Report: External mid-term Evaluation ,.-..1,, :.*;l ,... , :'' 4: "lr,',i'..y'i" .-.' !.rt; .. ;. . .--:nil' . .. ' :a-;. " -.ij -' : - ")i $,...'-r +_ . .,.. ] a.'^,."\'/' ,t ,l { ther-T t September 2000 ,4 JAF6.9 Page i Index I I Page B. C. Glossary and Acknowledgements Executive summary A. Introduction Appendices Appendix I Appendix 2 Apper-rdix 3 Temrs of Reference The evaluation team Data collection instrument The evaluation process Findings l. Programme management 1.1 APOC Headquarters management 1.2 Financialmanagement 1.3 TechnicalConsultativeCommittee 1.4 The NGDO Co-ordination Group 1.5 Synergies between APOC and OCP 2. Implementation 2.1 Coverage 2.2 Drug procurement and delivery 2.3 The role of national structures in Programme implementation 2.4 Community involvement 2.5 Monitoring 2.6 Operational research 2.7 Impact assessment 2.8 Sustainability 2.9 Vector elimination projects 2.10 The impact of current and future therapeutic developments3. Integration 3.1 Integration at community level 3.2 Integration at health district level 3.3 lntegration at higher levels 3.4 Contribution to health systems and capacity building3.5 Future co-operation with other health related programmes4. Partnership 4.1 Success of the present partnership formula 4.2 The contributions of the partners 4.3 National and international NGDOs ConclusionD. E. ll iii 10 1 44 7 9 10 t2 t4 15 t7 18 2l 22 24 27 28 29 30 32 34 35 35 36 37 38 40 4t 43 45 48 49 I .a JAF6.9 Page ii Glossary the Donor Community the Group Headquarters Management Participating Countries Phase I Phase 2 the Programme Abbreviations bodies donating funds for APOC and its projects and activities the NGDO Co-ordination Group the APOC base in Ouagadougou senior APOC professional staff at Ouagadougou countries conducting onchocerciasis control programmes in partnership with APOC the period of APOC's work from 1996 to 2000 the period of APOC's work from 2000 to 2007 APOC as a whole t AFRO APOC CAR CDD CDTI CHW CSA DEC DRC EPI FAO GS IBRD IEC JAF KAP LF MDP MoH NGDO NOCP NOTF OCP REMO SOCT SSE STP TCC TDR ToR UNDP UNESCO WHO African Regional Office of WHO Africa Programme for Onchocerciasis Control Central African Republic community directed distributor community directed treatment with ivennectin community health worker Committee of Sponsoring Agencies diethyl carbamazine Democratic Republic of the Congo expanded programme of immunisation United Nations Food and Agricultural Organisation general staff Intemational Bank for Reconstruction and Development (the World Bank) information, education, communication Joint Action Forum of APOC knowledge, attitude, practice lymphatic filariasis Mecti zan@ Donation Programme Ministry of Health non-governmental development organisation national onchocerciasis control programme national onchocerciasis task force Onchocerciasis Control Programme in West Africa rapid epidemiological mapping of onchocerciasis State onchocerciasis control team (in Nigeria) serious side-effects short-tenn professional Technical Consultative Committee of APOC Tropical Disease Research Programme of WHO Tenns of Refereuce United Nations Dcvelopment Programme United Nations Educational, Scientific and Cultural Organisation World Health Organisation r a t aa t JAF6.9 Page iii Acknowledgements The evaluation team would like to express its wann appreciation to the followir-rg groups and persons, for their help in planning and carrying out the evaluation, and providing infomration lor it: The staff at APOC Headquarters in Ouagadougou. Mernbers of the TCC. Staff at TDR, the NGDO Liaison Office and the Onchocerciasis Liaison Office in Geneva; Dr Ole Christensen, Mr Abdulai Daribi and Dr Janis Lazdins. Mr Bruce Benton and his staff at IBRD in Washington. Programme and Project staff in Cameroon, CAR, Equatorial Guinea, Nigeria, Tanzania and Uganda: * other national onchocerciasis co-ordinators and other members of the NOTF secretariat; NOTF members * staff concerned with onchocerciasis control at MoH headquarters * staff of NGDO project partners * staff concerned with onchocerciasis control at State/ provincial level * district level staff in districts that were visited by team members: district medical officers, health centre staff and supervisors + members of village communities visited by team members. The WHO representative and the WHO country office staff in Cameroon, CAR, Equatorial Guinea, Nigeria, Tarlzaria and Uganda. a a a a 'l tJAF6.9 Page I Executive summary About APOC The Afrrcan Programme for Onchocerciasis Control (APOC) was launched in 1995. It is a globat partnership whrch brings together l9 African countries, donors, non-governmental development organisations (NGDOs), the private sector, and the affected communities. The ultimate goal of APOC is to eliminate onchocerciasis as a disease of public health and socio-economic importance throughout the Participating Countries. Through the Mectizan@) Donation Programme (MDP) ivermectin is provided free of charge to APOC countries by Merck, Inc. for as long as needed. The objective of APOC is to establish sustainable, community-directed ivermectin delivery systems, serving 50 million people in l9 countries - these systems are to be established and become entirely self-sustaining within the 1996-2007 period. A subsidiary objective is to eliminate the blackfly vector of the disease in four selected foci. It is estimated that a minimum of 15 million people living within the APOC countries are currently heavily infected with onchocerciasis. About the mid-term evaluation The Committee of Sponsoring Agencies (CSA) drew up the Terms of Reference for a mid- tenn evaluation. The overall objective of the evaluation was to assess progress to meeting the Programme's objectives, including an evaluation of its operations; and to make appropriate recorllnendations on the Programme's strategies in order to fulfil its objectives by the year 2007. The evaluation was carried out by an international, multidisciplinary team of seven experls in the period May to November 2000. The team visited six APOC countries, attended key rneetings of the various committees involved in the organisation of APOC, interviewed people at all levels of operations and carried out an extensive docurnent review. Their findings and reconrmendations are described in the main repoft under the headings: Programme management, implementation, integration and partnership. In this summary the major findings and recommendations are highlighted and general conclusions presented. Overall APOC achievements APOC lias succeeded in generating enthusiasm and commitment on a wide scale. The leadership has proved exceptionally able and hard working. Al effective Headquarters has been built up in Ouagadougou, which communicates well with the projects it funds and supervises. The Technical Consultative Committee (TCC) which screens and monitors projects has proved very productive. The allocation, distribution and control of funds has proved to be a complex task, which has been handled well. APOC has been remarkably active and effective in stimulating the formation of u,orking projects in rts 19 partrclpatlng countries. To date 63 projects in l4 countries are operational, some of thenr already in their fourth year: 53 CDTI prolects, 6 headquarters strengthening projects and 4 vector ehmuration projects. Since its rnceptlon ut 1995 the number of persons treated yearly rn participating countries has doubled, to nearly 17 mrllion rn 1999. APOC's strategy of community directed treatment wrth ivermectrn (CDTI) has been a timely and tuuovatrve shategy for fighting a widespread scourge. Communities have been deeply involved in their own health cate on a massrve scale. The evaluation team concludes that CDTI could be sustainable if it rs fully irttegrated into the district and national levels of thc health services, which also demonstrate full commitment to tt. CDTI is a strategy which could be used as a model in developing other community based programmes, ald rs also a potential entry point in the fight agalnst other diseases of public health in-rportance. The Programme lras further developed useful new tcchniques, tools and models wrth the potential for wider apphcation - for cxar-nple extertral monttoring, cost recovery, and handlurg serious srde-effccts lu mass distributron programnles. Through its efforts the Progranme has positively influenccd the health services of participatitrg countries in a variety of ways. Capacity has been built; resources provided, a JAF6.9 Page 2 enthusiasm and motivation generated. The new programme has begun to be integrated into the official health service at different levels. Health service staff are closely involved in its planning and execution. In order to achieve this level of partnership and involvement the Programme has executed an impressive process of sensitisation and training. APOC has invested in establishing a large number of partnerships at different levels, between Ministries of Health (MoH), NGDOs, donors and civil society. Here stakeholders pool their ideas and resources in planning and executing the common programme. Workable models of public-private partnership have been pioneered at national and international level, which should serve as an example for other public health programmes. The establishment of such a quantity and variety of successful partnerships is a major Programme achievement. The National Onchocerciasis Task Forces (NOTFs) in each country deserve special attention in this regard. NGDOs, internationally and nationally, have through their financial and human resource contributions been a pillar of these partnerships, as have members of the donor community. Programme management The evaluation team felt that in order to consolidate achievements in the area of Programme management, and move forward into Phase 2, several areas needed to be reconsidered and adjustments made: . The burden on Headquarters staff is very high and cannot be sustained. Plans have been made to expand the staff, and to deal with some of the problems caused by short term insecure contracts. The governance structure of the Programme is complex and time-consuming, especially in terms of the statutory and general management meeting schedule, which Headquarters staff has to arrange and attend. Much rnanagement is still centralised and it appears to the evaluation team that opportunities for delegation are being missed. . There are a few, specific, problems related to the flow and control of finances. Projects find it difficult to provide the required proofs of expenditure. The main problem however is late disbursement of funds, due to sorne inefficiencies at the project level, delays in transferring information between levels, and bottlenecks at the WHO country office. There have been cases where such delays have caused project activities to be put on hold, which has been bad for credibility in the community and for staff morale. . The Technical Consultative Committee (TCC) is overloaded and its present input is not sustainable. [t has been necessary to call additional meetings to cope with a workload which should in many respects be managed by APOC Headquarters staff. The TCC is far too involved in project supervision and managernent, having gradually assumed a role much greater than that of giving technical advice to projects. Due to the volume of project review work, strategic, policy and operatiorls research issues do not always receive the attention they deserue. Priority recommendations relating to Programme management The staffing plan for 2001 should be implemented and vacancies filled without delay. Tlie APOC govemance structure should be critically reviewed and rationalised for Pliase 2 of APOC: to inculcate the principle of 'community directedr-ress' at all levels; to reduce costs of administration; to avoid duplication ol rvork; to reduce thc administrative burden on both APOC staff and national and project staff; and to rl a a a aJAF6.9 Page 3 position the Programme so that it links effectively with other emerging global or regional disease control programmes. The composition of the CSA should be reviewed in view of evolving changes in the Partnership. Plarrning procedures should be streamlined and wherever possible responsibility devolved to country level. In order to develop sustainability and accountability further, the role of the NOTFs and the WHO country office in financial management should be strengthened: as projects move through their five year cycle responsibility should increasingly be delegated to the national level, wherever possible. The role of the TCC should urgently be re-defined, to align it with its technical mandate. Some of the work that it currently does should be taken over by NOTFs, particularly six monthly project reviews. Prolonged uncertainty about the location of Headquarters should be avoided. Adequate administrative support must be ensured for Headquarters in Ouagadougou after 2002 (when OCP ends). If it is necessary to consider other locations then costed scenarios should be considered in the first half of 2001 and a decision made. t a a Programme implementation Concerns identified by the evaluation team relating to Programme implementation include: . There is a lack of information on coverage in some projects. Some CDTI projects are still failing to use recommended denominators and standardised formulae to calculate coverage rates. o I further significant problem is that the TCC guidelines regarding serious side-effects are not being adhered to in some projects. - . The four vector elimination projects appear to the evaluation team to be somewhat out of step with CDTI, the principal activity of the Programme. Two of the four vector elimination projects have experienced delays in taking off. Important questions that were not addressed by feasibility studies still remain to be answered in these projects. . The rate of progress of projects has varied widely, since they operate in a wide variety of political, social and geographical situations. Some have taken much longer to take off than others. . The security situation in at least three Participating Countries is very unsatisfactory, and in several others regional threats exist. It is unlikely that Programrne objectives will be achieved in these countries by 2007. Priority recommendations regarding implementation Steps must be taken to ensure that standard denominators and methods for estimating coverage are used, and that census data is updated where necessary. In this way better evrdence of Programme achievements and constraints may be collected. ../. - { Countries should be encouraged to integrate CDTI fully rnto the routine functioning of their health districts and regions. The danger of usrng APOC fundu.rg to create temporary, unsustainable structures is real. Accordingly, a key activity at the start of Phase 2 should be to ideutify which shuctures will be left to cany CDT1 forward after Year 5, and to use APOC funding primarily to strengthen those. -,. The Programme should be flexrble regarding the funding period it allorvs for indivrdual projects. Criteria should be developed to decide whether pro1ects can be eligible for additional fundin-e ur order to achicvc srrstarnabilrty. Tlarning and supervision are critrcal to the success of APOC and to rts sustarnabrlity. Each project or. drstnct should be encouraged to n.rake a plan for sustainable trainrng and supervisron, ancl to start irrrplenrentrng it be/brc APOC funding dries up. lndependent/lourt rnonltoring rs a key way to nrarntain the qualrty of project work and the motivation olthose involved. APOC should evaluate the new in-cotrnl.ry nronrtonng carefully, to see if rt produces the samc benefrcral results. Fundrng for its cor.rtrnuation should be arranqcd belorc ApOC's nrandate a a I a a a aa a a a JAF6.9 Page 4 exprres. Continued, sustained Sensitisation should be conducted, to clarify APOC's present and future role to stakeholders at all levels. The commitment of MoH is critical and constant advocacy is needed to ensure that this is forthcoming. Regarding operational research, topics requiring urgent attention include reasons for low coverage and reasons for CDD attrition. The relative role of men and women as CDDs needs investigating. The cost recovery initiative in Cameroon needs to be closely monitored, as a possible model for ensuring sustainability. An urgent investigation is needed, to determine to what extent TCC policy is being applied in areas where loiasis and onchocerciasis are co-endemic. Projects in these areas must be very strongly advised to adhere to policy fully. The future development of the vector elimination projects needs to be reviewed. This process should begin with a high-powered meeting of experts, called by APOC, where the vector elimination strategy of the Programme is critically discussed. APOC should continue its support for research into the development of a macrofilaricide, and potential non-response to ivermectin. Integration The evaluation team looked at the degree to which APOC activitres were integrated into national, district and community health care infrastructures. They also examined collaboration and integration between disease control activities. In spite of impressive achievements much work still remains to be done in this area, if APOC actrvihes are to be sustainably embedded in national health systems. CDTI is an appealing strategy for delivering other public health interventions but there is a danger that both the strategy and the delivery could be overwhelmed with new public health priorities and programmes in an unco-ordinated way. Care needs to be taken that onchocerciasis (which remains a public health problem) continues to receive adequate attention in an integrated community level programme, and that the coming lymphatic filariasis (LF) programme does not lead to unnecessary and costly duplication of structures at country level. Priority recommendations regarding integration A clear plan to empower national strucfures to handle nrajor management tssues must form part of APOC's plannrng for Phase 2 of its operations. Thrs again highlights the need to plan systernatically for the replacement of APOC funding by other, sustainable sources. The Programme rnust also build experlise in onchocerciasis contrrol at management level in each Participating Country, through a targeted scholarship programme. i The inclusion of excellent, relevant material on onchocerciasis control in basic curricula must be negotiated. APOC should promote this activity through the NOTFs. CDTI has entered communities on a large scale, often for the first time, and its success may lead to the inclusion of further activities. Care must be taken that CDTI takes place in harmony with other health and development related activities at community level (past and future). MoH and NGDO staff need to be committed to 'community empoweffnent' model, if other activities are included. The future of the NOTF needs to be decided upon. Its conturued existence should depend on its usefulness, which may vary with time. The body/ bodies which wrll drive CDTI after 200'l have to be officrally approved, strengthened, and funded. This is a task fol thc current NOTFs, with technical support fi'om APOC. It is recommended that CSA adjust APOC's tems of relerence for Phase 2, to allow the Programme to enter into partnership with other progral-ltmes utilising the conrmunity directed approach. q t /\ a a a a ) a aJAF6.9 Page 5 Partnership In spite of APOC's achievements the future shape of the partnership at national level is far from clear. It has not yet been decided whether the NOTF model of partnership is to continue, and if not, what will take its place. The role played by different partners varies considerably from country to country. Each chain is only as strong as its weakest link, and there are several examples of problems which arise when one or more of the partners is weak or lacks the necessary commitment. In some countries the MoH is so weak that the NGDOs have to bear almost the entire burden of the Programme. In other countries NGDOs experience problems and fail to fulfil some of their responsibilities. The role of the WHO country office also varies considerably from country to country. There is widespread agreement that it is desirable to promote local NGDOs as partners, but it has not always been easy to identify partners with sufficient capacity to mobilise resources. Prioritv recolxmendations in relation to partnership ' The overall aim of partnership must be the integration of CDTI into the routine functioning of the health service of each member country. While APOC funding remains there should be a planned and orderly process of task allocation to each partner, that will lead to sustainable CDTI. ' When APOC ends there should be a mechanism which continues to promote partnership in planning, monitoring and research around onchocerciasis control. Different rnodels need to be developed for different situations. Each NOTF should be asked to develop a strategic five-year plan, for submission to the MoH. This plan should indicate whether they see their role as phasing out and gradual integration with MoH, or whether they are going to continue to exist as a partnership to support onchocerciasis control activities. o NOTFs should identify local NGDOs with the potential to be subcontracted to take on work rvithin prolects, after due training and mentoring. Each international NGDO should in fact be encouraged to develop a partnership with an appropriate national or local NGDO, to build capacity and expeJrse rn fund-rarsing, programme implementation and financial management. General conclusions In the first four years of its life the APOC Programme has made significant and satisfactory progress towards meeting its objectives. A unique prograrnme has been created. The groundwork has been done: management systems and admrnistrative processes are in place, which on the whole are sound and effective. Working parlnerships have been established at many levels, from international to community, which are co,tributing srgnificantly to the success of Programme efforts. The capacity of the health services n participating count.es has been srgnificantly enhanced, so that they are actively involved in implementing oncliocerciasis cortrol - as are rnyriads of village communities. In 1999 l6 million persons were treated with ivermectin, and work has begurl otl four vector elimination projects. Progress is being scientifically monitored. The model of community directed treatment that has been developed and tested holds great promise for other existing and future disease control prograrrunes. Sustainabrlrty rs the key issue for Phase 2 of APOC, and the ma.;or challenge for the ftrture of o,chocerciasrs contro[. The evaluatron team concludes that the Programme should: ' Couttnue expandrng at the prcsent rate, so that overall Programtne objectives may be achieved by 2007. 'Ihis will requirc more streaurlined governance and administratrv. pro."drr.s, a11d illovative approaches rn countries wrth significant security problems. o Itrtensify its efforts to integrate onchocerctasis control fully rnto the health servlces of the par-ticipating Couutrtes, tu order that the Plogrammc's achrevements may bc fully sustaiped rvhen its i,pLrts conre to i JAF6.9 Page 6 an end. To this end it needs to embark on systematic devolution of responsibility and capacity to courltry level; to place major emphasis on capacity building; and to identify means of continuing support to essential activities after APOC comes to an end. f),\r a JAF6.9 Page 7 A. Introduction l. The objective of APOC At the first session of the Joint Action Forum in Washington, in December 1995, the following objective of APOC was approved: 'To establish, within a period of twelte years, effective and self-sustainable, corumunity-based ivermectin treattnent throughout the endemic at'eas in the geographic scope of the Programme, and, if possible, to elininate the vector and hence the disease by using environmentally safe methods in selected foci.' It was further stressed that 'the attainment of this objective will ultimately realise the goal of elintination of onchocerciasis as a disease of public health and socio-economic importance throughout Africa and so contribute to improving the welfare of its people.' APOC will end in 2007, but the activities of ivermectin distribution and treatment will cotitinue for as long as required to eliminate onchocerciasis as a public health problem. 2. The background for launching APOC Sir-rce Merck, [nc. announced in October 1987 that 'it would suppllt h[ectizan@ for the treatmcnt of river blindness to everyone who needed it, for as long as necessar!, at no clrut'ge ', Mectizan@ (ivemrectin) has become a mainstay in the control of the disease, pioneered by the Onchocerciasis Control Programme in West Africa (OCp). Tl-re decision to extend onchocerciasis control to the endemic countries outside West Africa. was principally based on two factors: . The success of ivermectin treatment in OCP. t Multi-country studies conducted by the TDR Task Force led to the conclusron that community-basect (later developed to communfiy-directcd) treatment would secure a satisfactory coverage rate, as well as potential sustainabilrty. When APOC came into being it was estimated that there were 50 rnillion people at risk of infection with Onchocerco volvulus in the l9 Programme countries. Of these l5 million were estirnated to be heavily infected: 6.4 million in areas with severe blinding and cutaneous onchocerciasis, and 8.6 million in areas with parasitic strains less damaging to the eye, but responsible for severe skin disease. 3. Main features of APOC According to the April 1996 memorandum which fomrally constituted APOC, it has four sponsoring agencies (WHO, F'AO, UNDP and IBRD - with IBRD being the fiscal agent and WIJO the executing agcncy). Various contributing parlies fund the Progranrme, wl-rich is intplernented in l9 Participating Countries. A particular feature of the Programme is the strottg partttcrship that unitcs the Participating Countries and the donor community witft the NGDO Co-ordination Group, private industry (specifically Merck, Inc.), the scientific cotrlnlunity and the Unitcd Nations system. All thc partners arc represcnted at the sessions of i JAF6.9 Page 8 tl-re Joint Action Forum. APOC operations are oriented essentially towards support to field operations which at the national level are conducted under the leadership of National Onchocerciasis Task Forces (NOTFs), usually with membership of the Ministry of Health, the NGDO(s) participating in projects financed by APOC, and WHO country offices. Tlre Programme may pay up to 75o/o of the ivermectin distribution projects while the balance is defrayed by the NOTFs concerned. APOC funding for each project is given for a maximum of five years, subject to annual technical and budgetary reviews by the TCC. The emphasis on support to field operations is clearly borne out in the budget of the Programme, in which management and administration account for less than 10% of the total allocation. The close collaboration between OCP and APOC, facilitated by the location of their headquarters in the same compound, has been of mutual benef,rt to both Programmes. APOC has received assistance on the administrative side from the OCP infrastructure and the two Programmes have shared their experiences in the fields of project implementation and operational research. 4. Developments of APOC to date and its achievements During 1996, the first year of the Programme, efforts centred on building up and putting in place the APOC structure, working out operational procedures, and sensitising the Parlicipating Countries regarding preparations and submission of project proposals. Since then the number of APOC supported project has accelerated, so that today 63 projects in l4 countries are operational, some of thern already in their fourth year. A special feature of the Programme has been the early institution of independent monitoring of projects, now also being developed into participatory monitoring at country level, and to community self-monitoring. The collection of baseline data for future impact assessments of APOC operations constitutes another achievement of the Programme. The institutionalisation of community directed treatment as the cornerstone of the Programme has been another critical feature. !"', a JAF6.9 Page 9 B. The evaluation process Early in 2000 a team of seven persons was appointed to conduct the evaluation, according to Terms of Reference which had been prepared by the Cornmittee of Sponsoring Ager-rcies (Appendix 1). Team members were drawn from a variety of backgrouuds, all having been exposed to onchocerciasis control programmes but in different capacities (Appendix 2). The team met for the first time in Paris in May 2000, to plan the evaluation. At this meeting APOC Headquarters staff and a representative of the NGDO Group made presentations to the evaluation team about developments to date, and the present situation as they saw it. Team members then proceeded to plan the evaluation as follows: . The points in the ToR were first clarified, eliminating overlaps and clarifying ambiguities. A series of agreed sub-questions was the result. . Working in groups, team members worked out which information would be needed for them to be able to make the required judgements on each sub-question in the ToR. . The sources of such information were then determined, as follows: * APOC projects in selected countries: Cameroon, CAR, Equatorial Guinea, Nigeria, Tanzanta and Uganda. These countries were selected in consultation with APOC Management on the basis of language, and project maturity, quality and nature. Within countries three types of project would be investigated: 'CDTI', 'vector elimination' and 'headquarters strengthening'. * APOC Headquarters in Ouagadougou, Burkina Faso. * Meetings related to APOC: the May 2000 APOC Parhrers'Meeting, the l0'l'meeting of TCC, the l6'l'meeting of the NGDO Co-ordination Group. * Relevant sources at IBRD in Washington, WHO in Geneva, donor country Minish'ies of Overseas Development. o At these sites information would be collected by means of interviews, document study and observation. (Following the Paris meeting the secretary prepared and distributed data collection instruments prepared according to the Team's decisions - Appendix 3). o Specific data collection tasks were allocated to each team member by mutual agreement. A time frame was set for data collection and producing country reports. Data collection proceeded according to schedule. Prior to country visits APOC headquarters provided each team member with relevant documentation. There was only one major hitch: due to lack of time only the Equatorial Guinea (Bioko Island) vector elimination project could be inspected in site - however discussions were held with those in charge of the Tanzania and Uganda vector elimination projects. Team members compiled country reports, as well as reports of other information gathered, and circulated these to each other. In Septen-rber all seven team members met in Geneva to draft a provisional reporl. TIte Tenns of Reference wcre further rationalised, since the experience of data collection had shown significant areas of overlap. Specific headings for the final evaluation reporl were agreed upolt. For each iter-n in the ToR, data from various sources were brought together. These were discussed, and agreement reached on achievements; on issues of concem; and ou suggestions , attd recommendatiotrs. Based on these discussions thc secretaries compiled a draft report, which was circulated to team members and some stakeholders, Aftcr two weeks of intensive further consultation and a telephone conference the final report was produced. I JAF6.9 Page l0 C. Findings 1 Prg g ram me'martggg.mgnt . iyi,,, " : ;i ..' t |,/lj :, 1.1 APOC Headquarters management F indings and o b s e rvatio ns Staffing /workload/ motivation t a a In general the evaluation team feels that tribute should be paid to the hard work and dedication of a small and very effective team which manages to maintain a high standard of work under enormous pressure. At all levels of the Programme their work is highly valued and appreciated. Nonetheless the staff,rng situation at Ouagadougou has been a cause of considerable concem. Vacancies have not been filled and key staff have been employed on short-term professional contracts, which give them relatively poor secondary conditions and little security. This has resulted in a situation in which the pressure of work is considerable, even unsustainable, and Headquarters has not always been able to meet the demands placed on it. Under these difficult conditions staff have maintained a rernarkable level of motivation, productivity and quality of work. The evaluation team were pleased to note that steps have been taken to increase the number of staff at Headquarters by filling vacancies and splitting one administrative post. In addition several members of staff have now been allocated fixed term posts which should help to guarantee continuity of staffing as well as improving staff morale. The evaluation team is satisfied that if the staffing situation planned for 2001 rs realised this wrll lead to a considerable improvement. WHO fixed term staff STP SSA Total Professional GS staff situation as of April 2000 1 3 4 8 t6 authorised posts for 2001 8 J 10 2t GS : general service staff STP : short-temr professional SSA: special service agreement Communication with countrv programmes and proiects In general the quality of communications belween country programmes, projects and Headquarters appears to be good. E-mail, telephone and fax communication is generally fast although some projects have only limited access to telephones and computers. Responses to queries are generally timely and the high quality of the technical input of Headquarters staff was appreciated. The numerous visits that Headquarters staff undertake to Participatrng Countries (for a variety of reasons) also contnbute to communrcation. 'lhe Partuers Meetrng organtsed at APOC Headquarters in June was well organrsed and very constructive ul ternts of communication between partners. Thc govemance structure The structure of AI'OC is one which has been built up to allow for, and encourage, full partrcrpation and partnership at various lcvels. One of the programme's strengths rs the rvay rn rvhich it has been ablc to gellerate and sustain commrtment and mobihse consrdcrable resoulces. On the other ltand the operation and tnatntenance of the structures, as thcy uorv stand, ar.e costly and a a a a a a ,JAF6.9 Page 1 I time consuming. A number of international meetings are held each year and these place a considerable administrative and financial burden on the programme: Meetings of APOC in 2000 Although the APOC governance structure is not unique among WHO programmes, there are other externally supported disease control programmes within WHO which have lighter and less rigid structures. Issues ofconcern The organisation of international meetings and support of the APOC governance structure places a heavy load on Headquarters staff and accounts for a disproportionate amount of its time. The style of Programme management is such that relatively few responsibilities are fully delegated to professional staff. Headquarters staff do not always have the time for necessary dialogue with national programmes and projects, and thus sometimes appear high-handed. At the national level there were complaints that I{eadquarters makes management decisions which have implications for the other levels, without due consultation. This is seen locally to be in contrast with the spirit of 'community directedness', rvhich APOC seeks to institute at community level. Mid-term planning within countries has sometimes been sacrificed to the pressure of deadlines from Headquarters. This can lead to sihrations in which country prograrrunes are faced with a succession of additional activities which disrupt project work, leading to delayed achievement and frustration at district and community level. An example is the additional census whicll had to be carried out across the board in early 2000, which effectively pushed planned distribution into the unfavourable wet season in some sifuations. There were many reports of money for prescribed or agreed activities arriving late, leading to their postponement in cases where NGDOs could not supply pre-financing. These concerns are to a large extent a reflection of the extreme pressure of work on Headquarters staff, and to some extent it is expected that increased human resources will help to solve them. However they may also reflect pressures put on APOC staff by CSA and donors, which are then passed on to the countries. Yet the quality of consultation and planning between APOC Headquarters and country programmes and projects remains crucial: the volume of projects is increasing, and the need to delegate responsibility for projects to the country level becon-res increasingly urgellt. Several stakeholders expressed concern about the present cornpositron of tlre CSA, beheving it not to be representatrve of the partnership as it has evolved. a a a t a a a a S uggestions and recorttruendation s date event venue 28-29 February TCC 9 Ouagadougou 8 May CSA 88 Paris 8-11 May TDR Task Force on Filariasis Intervention Geneva 22-27 May APOC Partners Ouagadougou 26-27 Ilune TDR Joint Co-ordinating Board Geneva 26-30 June TCC 10 Ouagadougou 17-18 July CSA 89 London 14-15 September ad hocTCC Geneva 18-21 September TDR Task Force on Filariasis Intervention Geneva 23-24 October CSA 90 Washington 10 and 16 December CSA 91 Yaound6 11-13 December JAF 6 Yaound6 a The staffrng plart for 2001 should be implemented and vacancies filled as soon as possrble. Since JAF6.9 Page 12 a however only three professionals are to be added to the staff complement, the situation regarding Headquarters staff workload should continue to be monitored closely. In order for the increased staff to operate optimally, the Programme's Executive Director should consider delegating more final responsibility for operahonal work to his professional staft thereby adopting a more strategic management style. ln view of the increasing number of projects and the continually expanding workload, efforts should be nrade to streamline planning procedures and place as much responsibihty as possible at country level. I{eadquarters should pay increased attention to the need to involve country programmes in decision makrng about the timing of visits, missions etc. The APOC governance structure should be critically reviewed and rationalised for Phase 2 of ApOC: to inculcate the principle of 'community directedness' at all levels; to reduce costs of administration; to avoid duplication of work; to reduce the administrative burden on both APOC staff and national and project staff; and to position the Programme so that it links effectively with other emerging global or regional disease control programmes. When this review takes place information should be gathered about the formats of governance of other externally supported progranmles in WHO, with a view to learning from their examples. 1.2 Financial management Firtdtrt gs and o bservution s The financial management system used throughout the Programme rs the tried and trusted WHO system, based on imprest accounts. Plojects and NOTF secretariats have been properly trained in WHO financial procedures, including the preparation of the monthly and six monthly financial reports (which all agree are easy to prepare, and for which guidelines exist). Although the system rs somewhat cutnbersome and time consuming it does appear to be effective. All planned projects receive their funding in due course. There appears to be a high degree of financial conhol and accountability, though perhaps too little transparency - in the sense that those who need to know the state of budgets for their planning often do not have that knowledge. Projects all have access to bookkeeping expertrse, to a greater or lesser extent. The bookkeepers/ accountants have to run two parallel sets of accounts: one for APOC (using the WHO accounting systern) and one for the NGDO concerned (usually using a computerised system, unique to that NGDO). Issues of concern Bottlenecks and delays a These can occur at all levels, but particular hold-ups seem to be located at the WHO country offices, and at project level. The payment of money into the WHO country office account causes a number of problems. The NOTF is not always informed by the WHO country office of the an'ival of money which has led, in some situations, to serious delays in transfer of funds. When projects receive n.roney it is very often not made clear what the money is for - for which budgetary year, or for whrch activities. The ceiling on the WHO country office imprest account sometimes leads to money for projects being leleased in small amounts, which conrplicates matters further. The evaluation team is infomred by APOC Management that it is possrble for APOC funds for projects to be sent directly to NOTF accounts, without going through the WHO courltry office. Whrle such a step would eltmrnate the bottleneck at the country office, it would mean that APOC could not turn to the WHO country representative for help in case of problems or suspected uregularities. There appear to be two prrncipal problems at project level. The first is that proJects have great drffrculty in collectrng the necessary recerpts/ documents of proof ol expenditure. Such documents often have to be provrded by thc most peripheral level (e.g. where trarnrng has taken place), and accountants have to rely cln numbers of different people to supply them. There can therefore bc a delay of rnonths belbre all the docur.nentation that is requrred for the monthly financral report (to APOC Fleadquarters) has been plepared. -l'he second problem (much less comrnon) rs a lack of accounting abrlrty - in terms ol both a a a I C,\ a a a aJAF6.9 Page 13 discipline and efficiency. This situation has been improved by training pro.ject staff in the necessary procedures, which is ongoing. Although monthly financial reports are delayed in a number of projects (from 2 to l9 months) ttris has seldorn resulted in funding being withheld by APOC Headquarters. This is because it is possible for APOC Headquarters to make disbursements on the basis of the completed 'statement of reconciliation of account', which may be forwarded pending the collection of all the necessary documentation. Projects seem unaware that this is possible, which at least partly accounts for the delays in reporting. In some instances these hold-ups in disbursement of funds from WHO have led to delays in project activities. Project and district level staff stressed how this has damaged project credibility. Delays also demoralise staff, who have to change their plans and respond to community queries and demands about the delay. The situation is mitigated somewhat by pre-financing of activities by NGDO partners, but this is not always possible, nor is it encouraged by APOC Management. In a few instances pre- hnancing has been disallowed, because of a breach ofthe procedures for the process. This has caused son-re friction between NGDOs and APOC. APOC Headquarters staff also express an increasing problem in coping with the mass of documentation that arrives in the form of monthly financial reports. This has meant that they are no longer able to check each report in detail. , a a Tlrere appears to be a lack of standards for drawing up budgets. APOC Headquarters staff aqd TCC members complai,n that projects over-budget considerably, and staff have to spend much time revising budgets downwards. At country level concems were expressed about a lack of guidelines/ standards for projects to use in calculating amounts in budgets - for themselves and for TCC/ APOC Headquarters. The issue of 'topping up' came up in several country visits, salary top-ups being catered for in some countries and not in others. Many instances were reported from the field where the funding allowed by APOC for an activity was inadequate, so that other partners had to chip in for an expense that ApOC had agreed to underwrite - for example, nurses in health centres themselves paying for taxi fares to go to villages for trainrng and supervision. The sus'lainability of current arrangements is a matter of concern. The WHO accounting procedures are inflexible and have been seen as 'non-negotiable'; therefore solutions have been four1d in the form of extra resources to deal with the volume of work which rs required. In four out of 14 WHO country offices APOC is paying for an accountant to assist in preparing and rnonitoring project accounts. It is hoped that this will resolve some of the delays and problems. The evaluation team is concemed that this solution is one which, in effect, creates another layer of accounting work but does not necessarily contribute to national capacity in terms of the post-APOC situation. ht view of the possibility of sending APOC funds into directly to the NOTF account, APOC Management should carefully consider this option in each individual case. If it is well supported it would build local capacity (rn the NOTF). In sonre cases there is a lack of transparettcy at project level about budget allocations. In one project visited lack of transparency about project budgets and a patemalistic management style within one NGDO has led to a climate of suspicion and distrust. In several instances project accounts are held in banks far removed frorn project sites. This may be because it is close to the NGDO partner's office, or because smaller local branches of banks are unable to cope. This means that cheques have to be transported physically to the project area for signing, then back again to be cashed. Finally large sums of cash have to be transported back to the project area. Project staff are worried about the security risk that this entails. S u ggestio rr s an d reco n, nten datiorts Other issues a a a t a h.r order to develop sustainability and accountabrlity at country level the role of the NOTFs and the WHO country office in financial management should be strengthened progressrvely, through the neccssary trainrng and sensitrsattou. As projects move through their five-year cycle the responsrbility should increasingly be left to the national level. NOTF and the WIIO office should play an increased role rn controlling pro.;ect expendihrre. Settrng budgetrng standards (with country adjustments rf nccessary) u,ill make everyone's hfe easier. There should also bc a clear pohcy rn relation to topplng-up of salaries. The evaluatron team is of the opinion that, in the tnterests of sustainabilrty of projects, the practrce of topping-rrp has to be phased out as raprdly as possible. The inevitable cousequences regardrng sta[f motrvatron will have to be dealt wrth rtt other ways, so that solutions are in place by the trmc APOC fundrng ends. -l'he follou,rng are a aa a a JAF6.9 Page 14 suggested: * Administering a well-run programme, with clear objectives. * Good, supportive supervision, integrated with routine supervision of other activities at each level. Projects should be made aware that some delay in providing receipts is acceptable, provided that they are supplied within a reasonable time frame (e.g. three months). Every transfer of funds by APOC Headquarters should be accompanied by letters to both the NOTF secretariat, the project itself and the WHO country office, stating exactly rvl-rich funds are being transferred, and for what purpose. The ceiling on the WHO country office imprest accounts must be raised, rvhere this has been a problern. To achieve a progressive hand-over of project management to national progralrrmes, APOC Management and TCC should identify projects which perform well in temrs of financial reporting. These projects should not be subject to continuous monthly review of expendrtures - rather, random spot checks should be applied to them. 1.3 TechnicalConsultativeCommittee Findings and o bservations 'The main purpose of the TCC will be to review applications for funds from rvennectin distribution projects, rncluding both their technical justification and financial feasibilrty, and to review the inrplenrentatron of funded projects' (Programme Document, 1996, p. 29). The TCC consists of ten members. Membership is defined by the Programme Document, and includes five independent scientists; two technical representatives of the NGDO Co-ordination Group; one representative from the Carter Center; one representative of the Mectizan@ Donation Programme (MDP); and one lcpresentative of the Expert Advisory Committee of OCP. The TCC meets twice a year, usually in Ouagadougou. APOC staff serves as a secretariat for the meetutgs. TCC members take their work very seriously and are incredibly dili-eent. The level of review they offer is very high and is greatly appreciated at all levels of the APOC Progran.rme. As one member of the TCC put it, 'I have never served on a committee, especially an intemational technical corruDrttee, that works so hard and so productively as the TCC.'There are now 63 projects in 14 countnes: 53 CDTI projects, six headquarters support projects and foul vector eliminatron projects. The total is expccted to lise to 78 rn 2001. During TCC 10, for example, four new projects or natronal plans were revierved, and 40 continuing projects were reviewed. Each project receives a thorough review by two TCC members - one serving as primary reviewer, the other as second reviewer. All other members revle\\' each project, but less thoroughly. During TCC 10, each TCC member had the responsibility, as erther prirnary or second reviewer, for about ten proJects. Reviews of pro.;ects include technical revieu,of project activities, on a six monthly and annual cycle, and a detailed budget review. The TCC meetings are therefore long and require concentrated attention to detail. The work of review occurs in the eventngs in preparation for the next day's meeting. TCC and APOC Headquarters staff have both proposed strategres to reduce the workload of the TCC and the APOC staff as well as to restructure the TCC process. TCC believes that there must be srgnrficant changes in the structure and organisational structure of APOC, and that the transitron from Phase 1 to Phase 2 rs the approprlate tinre to make these changes. t r.\\i a a a a l.ssrras of concern The TCC is ovcrloaded and its present input rs not sustainable. TCC menrbers believe that rts rvork is 'at a clisrs pornt.' At the end of thc mcetrug week, both TCC membels and APOC staff are exhausted. TCC overload rs r.nirrored by APOC staff overload dunng the project revieu' process. The volume of pro.lcct review nreans thal stratcgic, policy and operatrons research tssues that need to be addressed by TCIC are rclcgated to the end of thc nreetrng week, and may not recelve the attentrorl they deserve. BLrdgets, plans of action alld progress reports arc llot always in sequence, making it clifficult lor TCC to synchronise budgets with activities. TCC overload is mirrored by ) a a JAF6.9 Page t5 APOC overload at all stages - before, during and after the project review process. The TCC is far too involved in project supervision and management. The TCC has gradually assumed a role much greater than that of giving technical advice to projects. It has become closely involved in work requiring financial and administrative expertise that most of the TCC members do not have - they were selected for their technical expertise. Thus there appears to be something of a mismatch between the expectations for technical project review, the need for managerial skills, and the expertise brought by the TCC members to the task. The evaluation team is concerned that not enough is being done to develop capacity at country level for the work currently being done by the TCC. This is particularly important as the TCC will no longer exist after 2007. The role of the TCC is not well understood by projects and national headquarters staff. It is sometimes seen as having powers beyond that ofa consultative/advisory body. There is some confusion between the TCC and APOC staff in terms of how much discretion APOC staff should have in nraking decisions regarding budgetary allocations and financing issues for projects. The issues surrounding governance and integration of onchocerciasis and lymphatic filariasis (LF) are contentious, particularly at the international level. It is unclear whether the role of the TCC will expand to include LF in countries where onchocerciasis and LF are co-endemic. The structure and value of co- operation are discussed in section 3.5. S ugg es t io rt s a n d re co n, n, e n datio rts The role of the TCC should urgently be re-defined, to align it with its technical mandate. Some of the work that it currently does should be taken over by NOTFs, particularly six monthly project reviews. Sorne of the work should be done by APOC Headquarters staff (which is about to be expanded). At the same time TCC should advise that decision-making be delegated as far as possible to other partners, including NGDOs and MoH at appropriate levels. TCC should review the degree of scrutiny needed to assess projects adequately, in order to streamline the process ofreview and to burld capacity for review and decisron-ntaking at the national level. Phase 2 of APOC offers an opportunity for formal restructuring of TCC, as well as a reconfiguration of TCC membership to include members with skills in management and finance. The proposals developed by the TCC and APOC staff are a useful starting point to initiate a trial of new procedures. 1.4 The NGDO Co-ordination Group Ftndings and observatiorts a a The NGDO Co-ordination Group, while not one of the goveming bodies of APOC, plays a significant role in the Programme. It represents the NGDOs involved in ivermectin distribution in both APOC and OCP. The NGDOs have been active in developing the state of the art with respect to CDTI and work in close collaboration with MoHs in the APOC structure. The NGDOs are committed to building capacity at the local level for all aspects of CDTI, including the goal of CDTI serving as an entry point into the development of primary health care. Thcre is a strong corlsensus from the NGDO Group that onchocercrasrs and LF must be dealt with togethel at the cot.trt.uunrty level, regardless of the shape of governance issues at the national and tutetuattoual Ievels. The Group recogntses that there are opportunitrcs for more NGDO involvement in the new dtsease cotrtrol and ehminahon inrtiattves of WIIO, and desires to bc a significant player in the drscussrons about these progranxnes takrng placc at all levels. Replesetrtattves of partuer NGDOs serving on the Croup arc alnrost all Amerrcan or Eulopean, as are thc t'eprcsentattvcs of thc Group to the TCC. 1'his appears contrary to the desrre to build capacity of natronal stafl wrtlrrn international NGDOs. a a a a 5 aJAF6.9 Page 16 Issues of concern Tlte NGDO Gloup has been principally composed of blindness prevention NGDOs. If CDTI stars being used by other programmes which also desire an operational partnership with NGDOs, it is not clear whether the co-ordination of NGDO participation in such partnerships will also fall to the present group to deal with. If it does the membership of the Group will necessarily have to expand to include more broadly based NGDOs, i.e. those not exclusively dealing with blindness prevention. The Group does not contain representatives from the natronal staff of the NGDO country programmes, nor does it have any local NGDO representatives. The financral expectations of membership may preclude membership by local NGDOs. S uggestio rts and reco,rrtnendatio rts The Group should considel nominating a national staff member of one of rts member NGDOs, and a representative of a local NGDO undergoing mentoring, to serve on TCC. a f\ a JAF6.9 Page 17 1.5 Synergies between APOC and OCP Findings an d observatiorts A thorough enquiry was made in July 1998, which went into the question of the location of the APOC Headquarters in some detail. At that time an analysis of the situation was made and a questionnaire sent to all members of the CSA and to all countries with programmes running. The majority of those asked were in favour of maintaining the current location in Ouagadougou adjacent to the OCP Headquarters. Not only does OCP provide considerable logistical support to APOC but also there are regular opportunities to share experiences, skills and scientific knowledge. A move would be costly, and would disrupt the functioning of Headquarters (already overstretched) for at least ayear. The conclusion of the 1998 report was that at the tirne the 'the benefits of remaining in Ouagadougou are very substantial and greatly outweigh those of moving.' The evaluation team endorses this conclusion. Issues ofconcern At tlre beginning of 2002 OCP will close its offices and the services which it provides (e.g. purchasing, transport, communications) will have to be replaced. In addition the expertise on vector control which OCP provided to APOC will have to be found elsewhere. S ug g estio tt s an d reco rnmendatio rt s The Programme Management should develop a plan (early in 2001) which demonstrates how the technical and administrative support which they need can be provided in Ouagadougou, after 2002. This plan should of course take into account the fact that a subregional Centre for Multi-Disease Surveillance and Control may be in operation by that time, in the same Ouagadougou compound, which could continue to provide the support that OCP currently provides. If APOC Management cannot find this support in Ouagadougou then they should consider other locations and develop costed scenarios, so that a definite decision can be taken iri tlre first half of 2001. a JAF6.9 Page 18 2.Impl9mentation,",i,,,|.,o;,....l.;:i,-l|..'.'.,,-,..., The objective of APOC at its inception was to establish sustainable, community-directed ivermectin delivery approaches covering 50 million people in 19 countries. These systems will be established and become entirely self-sustaining by 2007. APOC also aims at eliminating the vectors in selected foci. This section of the report examines coverage, drugs distribution and delivery, the roles of various actors at country level and aspects of implementation. 2.1 Coverage Coverage achieved Coverage is defined by APOC as the ratio of the total number of people treated in an area that should be under CDTI (as determined by REMO), to the total population in that area. Fir tdirtgs and obscrvatiorts APOC started its activities at a time when total treatment in APOC countnes had stagnated around eight million people being treated annually. This figure has now doubled - coverage in individual countnes has increased considerably since 1998 (for example from 50% to 70o/o in the case of Nigeria). This is a substantial achievement. The evidence of all the country visits leads to the conclusron that a great deal of eltergy, commitment and lesources is resulting in considerable successes in terms o[ providing access to treatment to those living rn areas designated for treatment. On the evidence available treahnent will have been extended to the majority of communities living in designatedproject areas, after five years'support. APOC's target is for each proJect to achieve and sustain a coverage rate of 65o/u irt each populatron that slrould be under treatment, for at least 20 years. APOC now estlmates that, by 2007, 59 million persons will need to be treated per year. Currently 17 million people are being treated annually (1999 figure). This number will have to increase by 5 millions per year, if the target of 59 millions per year is to be reaclred by 2007. The availability of data on coverage varied greatly from country to countly. ln some all data was fully available at NOTF level (whrch could therefore aggregate it), but in othels it was only obtainable at project level. The denomrnator used in calculations also varied between proJects (see below). APOC faces a partrcular problem in achieving its goals, in member countries where there is significant civil unrest and even war. In several of these the Programme is going ahead, but it is necessarily constrained - not only by the threat ofphysical danger to health workers, but also by the breakdown of the officral health service. It is not realistic therefore to expect the same good results in these situations tl.rat are berng achieved in other, more nomral ones. It is unlikely that Programme ob.;ectives will be achreved in these countries by 2007 . a a a a a a '1" Factors favourable to high coverage factors likely to decrease coverage * If people know about ivermectin before the start of the project there is likely to be less suspicion. * Efforts to sensitise communities and village leaders appeared to have a direct positive effect on coverage. * Motivation and strong leadership at community and district level. * Leaving a supply of pills locally makes it possible for some communities to follow up absentees. * High prevalence and severity of clinical symptoms of onchocerciasis. * Establishing reliable census data makes it possible to follow up defaulters more carefully - and also improves quality of coverage data. :t Lack of previous experience with vermectin * programmes may lead to more initial suspicion of mass treatment - e.g. is it a contraceptive? The side-effects of previous treatment with DEC (or with loiasis co-endemicity) made people unwilling to be treated with the new d-g. Delays in payments to projects led in some cases to delayed fi'eatrnent and lower coverage (e.g. if treatment happened at a less suitable time for farming communities). Cost recovery systems may adversely effect coverage of poorer groups, if there is no effective exemptions policy. * * JAF6.9 Page 19 a a a A number of factors emerged from the country visits which influence coverage: a a Issues of concern Data presented to the APOC Partners' Meeting showed that six out of ten countries discussed have less than 65%o coverage. Possible reasons are given in the table above. The evaluation team reached the conclusion that many of the projects will not have reached the point of sustainabilityr after five years, unless very specific remedial action is taken (this is discussed in greater detail in section 2.8). Unless this is done coverage will be hard to maintain when APOC support comes to an end. There is considerable diffrculty in getting reliable and comparable data on coverage and coverage trends. At the Partners Meeting it was stated that in 22 out of 27 projects the denominator was inconsistent and unreliable. The decision by TCC to request all projects to conduct a new census in early 2002 may lead to more reliable coverage rates in some sihtations, but not ir.r others (see below). Various factors were found which may confuse coverage data: extrapolating population figures from an outdated census; families refusing to take part in the census (being suspicious that they were being 'controlled'), yet wanting to be treated; self-treatment with stolen tablets bought on the open market(resulting in treatrnent not being repeated during the 'official' distribution).. It appears to be the practice in the majority of APOC projects for CDDs to conduct the census and the distribution on two separate occasions. In OCP countries both are done together. Advantages of the Iatter method are that it saves time and effort (which has been shown to affect CDD motivation and turnover), and that people understand why the census is being done (thus calming suspicions). A disadvantage is that CDDs may not include families who refuse treatment in the census, giving falsely high coverage rates. It has long been OCP policy not to distribute ivermectin in towns wrth populations of over 2000, since sfudies have shown that the vector avoids larger human settlements, and in consequence transmission there is very low. As a result OCP recommends passive treatment in such situations. Some APOC projects however are conducting CDTI in fairly large towns, with populations well rn excess of 2000. The rationale for such a step is not clear. The evaluation team takes note of an attempt to renew the debate around the possibilrry of rntemrptrng transt.uission of onchocerciasis, and even eliminating the disease, by means of rvermectin distribution alone. It is noted that, according to the Onchosim simulatron, such rnterruption rs only possible in hypo-endenuc areas, where APOC does not operate. a a a ' ltor the purposes of thrs report an activity is considcred 'sustarnable' rf the polrtical decisions, adr-ninistrative measures and resources (financral, human, matenal) needed to support it are in place, attd can rcasonably be expected to continue being so for the loreseeable future. t'1.: n JAF6.9 Page 20 a a a Participating countries with significant problems of security and health service breakdown should be treated as a special case. [t is highly unlikely that they will achieve full coverage when APOC comes to an end in 2007. Since none of these countries was visrted it is not clear what provision is being made for them, to con-rplete the work once APOC inputs cease. S ug g cstio rts a nd reco m nte ndotio ns Steps must urgently be taken to ensute the use of standard denominators and methods for estimating coverage at all levels. It is recommended that TCC immediately reaffirm its preferred options. In doing so it should consider the tried and trusted OCP formats: * Number of persons treated as a percentage of the total population of that area. * Number of communities/ villages treated in an area, as a percentage of the total number that has been indicated (by REMO) for treatment in that area. Projects must be required to commit themselves to using only these methods and denominators, at all levels. There is an urgent need for operational research in projects with low rates of coverage, to identify causes of low coverage and to identify strategies to address this. The evaluation team notes with appreciation that studies of this type are a current TDR priority. However projects and district level staff should also be encouraged and empowered to undertake research on this topic, as it pertains to their situations. When the process of negotiation with communities is underway, it is suggested that the advantages and disadvantages of conducting separate or simultaneous census and distributron be discussed with them. In this way communities can make a more informed choice of method. In either case it should be routine for the cellsus to be updated with each distribution, which keeps the denominator accurate. For each participating country with significant problems of security/ health service breakdown, very specific planning is needed to make provision for a continuation of onchocerciasis control after 2007 . :\a a a Tl-re effect of CDTI on coverage Firtdirtgs and ohservations The evaluation team was asked to draw conclusions about the effect of the introduction of CDTI on covetage rates. The strong initial study carried out by TDR upon which the CDTI strategy was based still stands. However the anecdotal evidence which the evaluation was able to uncover is ltnited in quality and con.rparability, makrng it impossible to draw clear conclusions. Limited data from one country suggested that the move to CDTI led to an lncrease in coverage of about 10%o. However the rntroduction of CDTI coincided with extra money, extra resources and a strong programme, making it impossrble to say whether it is the community directed approach which has achieved the results, or other factors. Qualitative data fronl visits suggest that the community directed approach has some clear advantages. It strengthens community participation and understanding; rt makes it possible to link treatment closely to a census, thus enhancing rehability of rates; and it makes it easier to follow up absentees and those who refuse. A few disadvantages of the community directed approach were also detected. It takes longer to set up - and as a result some NGDOs report finding it hard to post results of expanding covel'age as expected by their donors. It may also engender initial mistrust because of the fact that '1a1,' people are seen to be handrng out'mcdical' treatment. ln general the evaluation team is of the vtew that CDTI rs an appropriate and effectrve strategy for the drstribution of ivermectin in conrnrunities. Issues of concern There l-ras been little additronal evidence about the eflect of CD'II on coveraqe, since the initral TDR shrdy. Lrttlc lcsearch has been donc into the reasons for success and farlure of CD-I'I. a , a a 1i;t - JAF6.9 Page 2l S uggestiorts and recont,rrendatiotts Operational research is needed which would build up a stronger evidence base on which to develop CDTI projects. The results of such research should also be used to modify the CDTI strategy where appropriate. These studies should be conducted by projects, in consultation with the respective NOTF, and the TCC. The Programme needs to build operational research capability at the local level. 2.2 Drug procurement and delivery Fin dings and observatio rts a a a a a a a a a a The donation of Mectizan@ (ivermectin) by Merck, Inc. to the countries in the Programme 'for as long as it is needed' is an outstanding example of international humanitarianism. All countries visited have found a working model for distributing ivemrectin. None reported major current problems in obtaining, clearing and distributing supplies of ivermectin to health facilities. There are major differences in country capacity to deal quickly and efficiently with drugs through public sector mechanisms. As a result procurelnent and delivery procedures vary greatly from country to country. In some the ivennectin supplied by MDP goes through the same channels as other public health sector drug requirements, fi'om the national down to the district level, whereas in others all or part of the national system is at present being bypassed (with implications for sustainability). If shipping agents other than MDP shipping agents are used (usually in countries which route ivermectin through channels used for public sector drug supply) clearance and handling charges may be imposed. This can lead to financial constraints. Some pro..;ects reported problems in distributing the drug to cornmunity level. CDDs are expected to collect the drug from health facilities, but quite often health service staff takes it out to them. In one country cars were sent around to villages telling CDDs to come and pick up the drug - thrs was cited as an example of a rather dockinarre application of the APOC philosophy of 'comn'runity directedness'. Several projects visited mentioned problems in relation to the packaging and storage of ivermectin. The drugs should be kept at a temperafure lower than 30'C and this can lead to problems at conrmunrty level. It was reported that the quantlty of tablets in a container made CDDs reluctant to open new bottles towards the end of the distribution period, and the rapid collection of unused drugs made it difficult to follow up people who had been absent for the main distribution. Exprred drugs are a problem - opened packs have a short shelf life. In some countries unused drugs are stockpiled at district hospital pharmacies. Country capacity for safe drsposal is limited or absent and may be expensive. However Merck, Inc. recently announced that studies are showing that open bottles have a shelf hfe of eight months. Studies are contrnuing to find out whether stability is maintained up to twelve months. Issues of concern Soure country programmes are partly or entirely dependent on mechanisms funded by APOC for the supply and distribution of ivermectin. These arrangements will not be sustarnable when the programme ceases or cuts back on it operations. Clearance through MDP shrpping agents is not conducive to rnte-qratron or sustainability. Clearance fees when using other agents can equally constltute a threat to sustainabilrty. Thcre are different ways of dealing wrth distrrbution to CDDs and to pcople in the conlnunity. In some countries collectror.r of rvermectrn is tred to CDD trainrng, and drstnbutrou u,ould be cornpromised rf regular training was stopped or reduced. In a I'ew countnes substantial losses have been reported. It appears that these take place at community levcl - the drugs find their way onto local markets, where they have commercial value. Such losses appear to be due to lack of comn-runity participation on the one hand, and effective supervision by drstrrct [eve[ stafl on the other. A contributing factor nray be that record systerrs ale sometimes a a a Il aJAF6.9 Page 22 haphazard. S uggestio n s and reco rrt me n datio rt s An rmportant feature of Phase 2 should be the planning and establishment of a delivery system which is completely independent of APOC funding. Wherever feasible drug delivery should be integrated into MoH drug delivery systems., but where these are not functioning other channels will have to be used. ln any plan the record keeping systems need to be firmly established and standardised. The possibility of dealing with the drug as a'donation of no commercial value'should be explored, if this will save clearance and handling costs. Research should be carried out to establish whether alternative package sizes would reduce wastage and/ or increase coverage. As always communities should be encouraged to identify the most effective and sustainable way of ensuring distributron to community level. This means however that they need to be aware of the advantages and disadvantages ofthe different options. 2.3 The role of national structures in Programme implementation Findings and observatiorts a a a , f',1L.f a The challenge in each country has been to develop structures and procedures rvhich will fulfil several simultaneous aims. They have to bring together the resources of the State and NGDOs; they have to be sustainable in the long term; and most of all they have to deliver ivermectin regularly and successfully to all the villagers who need it. In general countries have responded to this challenge in a positive and effectrve way. The structures that have arisen vary from country to country. Each has a NOTF (discussed in detail in section 4.1 below) and a national onchocerciasis co-ordrnator, who are together responsible for the Progranrme in the whole country. The NOTF is responsible for the development of a national plan and for setting up and guiding projects, so that they are able to apply for funds to APOC. Below this level there are varying structures at State/ provincial/ regional level, and at the district level. The principal difference between these shuctures is the degree to whicl-r they are integrated into the malnstream of the official health service, or separated from it in parallel vertical structures. Another drfference concems the degree of commitment of key indrviduals at different levels - particularly their fulfilment of promrses of resource allocation to the Programme. There are instances where this commitment appears weak. There appears to be no consistent (standardrsed) method to monitor govenlment counterpart funds. Funds car.r be reported as rn-kind contributions, e.g. salaries for NOCP staff, or actual funds allocated to the onchocerciasis programme in a given country or district. It is difficult to get an accurate picture of the ach-ral counterpart funds involved in the Programme ovel time - and the availability of such funds consistently and over time is a useful indicator of the likelihood of financial sustainability of the CDTI strategy in a country. A conrmon feature of national programmes is that salary top-ups are provrded for of State (and sometimes NGDO) employees rnvolved in projects. This may take place at all levels, from national headquartels to the health centre, but drffers from country to country and even frorn project to project within countries. The srx 'Headquarters support' projects that were approved by TCC appear to have been successful. Sufficient capaclty has been built to manage the pro.;ects tn the countries concemed, and the necessary resources have been madc avarlable for the respective tealrs to be able to functron satisfactorily. lssrrcs of concern Somc countries have appornted distnct level staff who are solely r.esponsrble for CDTI. Thrs phenomenou nceds to be very carclirlly thought through. It rs unlikely that any country can afford cledrcated staff for each progranrure at dtstrtct level for any length o[time, and there rs the ever-present dangcr olveltrcalisatron of progranlr.nes. In this respect APOC nray be burldrnq capacrty in the wrong placc, or ul a way wltrch canuot be sustatued. a a a a a a a JAF6.9 Page 23 The policy of providing salary top-ups is equally a matter for concern. On the one hand they motivate health workers who receive sub-subsistence salaries, and get the Programme going. On the other hand they are not sustainable - when they come to an end after the five years of APOC support, the initiative and hard work they have generated may well slacken: 'When I don't have tl.rat [amount] a month extra do you think I'n-r going to work like this?' The Programme's provision of logistics (such as dedicated trausport and computers) is similarly worrying - if projects depend on these inputs for their routine functioning, it may be hard for them to cope once the additional help is not forthcoming. There appears to be little serious discussion in projects of what will happen in 2OO7 when ApOC ceases operations, as well as little sense of urgency about developing scenarios for sustainability. S uggestio ns and recomruendatiort s a a a a a t a a a a a and ervlslon Findings and observatiorts Tlaining aud supervision are critical to the success of APOC and to its sustainability. A phenomenal atnount of training and capacily building has taken place since the inceptior-r of the Programme, from international to village level. This training has in general been thorough, but also costly. Some countries have developed a cascade system whereby training starts at the top and each level trains the level under them. lu some countries the NGDOs have assumed an important role in training and supervrsion. In other countries training has been ahnost completely integrated into the health care system. In other countries trained schoolteachers are being widely used as supervisors. Supervision of the CDTI related activities of each level by the level above rt rs certainly taking place, but not always equally thoroughly. Much of this supervision is specifically CDTI related and vertical - not integrated into routine supervisory activities. The limited number of bicycles and motorcycles provided through APOC have been tremendously useful in this regard. Issues ofconcern Countries should be encouraged to integrate CDTI fully into the routine functioning of their health districts and regions. Clear indicators are needed to show progress in integration (see also section 3). Dedicated onchocerciasis personnel are probably only sustainable at national level, and at State or provincial level in large countries with many projects. The danger of using APOC funding to create temporary, unsustainable structures is real. Accordingly, a key activiry at the start of Phase 2 should be to identify which struch.rres will be left to carry CDTI forward after Year 5, and to use APOC funding primarily to strengthen those. There is little doubt that the district level will be a prominent player. Expertise should be developed in such a way that it is the districts which assume responsibility and have the necessary capacity. Further important recommendations concerning sustainability are made in section 2.8. In tenns of sustainability training and supervtsion are heading for problems. Tl-rey are to a very large degree dependent on APOC funding, and when funding is not available superlision does not take place. In most sifuations there are few concrete plans about how to fund them after APOC inputs come to an end. In some proJects the style of training appears relatively costly - the h'arning lastrng longer than rs absolutely necessary; training being done by large teams, including 'outsrders' (as opposed to sin-rply uslllg one or two health district staff); and re-training being routinely dorre with each new distribution (rvrthout fir'st ascertainrng whether it rs operationally necessary). If this style becomes the norm it has obvious in.rphcations for sustainability. The ma.ior constralnt to supeF/lslon is lack of transport. 'lhrs is a gerleuc probler-n, altbctilg all progratrxnes in the district. Another constraint is the fact that ur many cases CDTI is 1ot yct seen to be a routiue part of drstrict and sub-district health care delrvery, to be superviscd routrnely wrth the other acttvtties included in the 'basic package'. A further constraint is that supervlsors, especrally at hrgher levels (APOC I{eadquarters and the national onchocerciasis co-ordrnator) are so takcn up rvitS cmergency marlagel-nent rssues that supervision sometlmes takcs a back seat. l'o date APOC has uot invested tn techntcal training of natronal staff-, at a levcl rvhrch cqurps them to a a a a -* l-: aJAF6.9 Page 24 manage a national onchocerciasis control programme. This is in contrast wrth OCP, which has through the years built up a corps of expertise in disciplines related to onchocerciasis control, in each of its member countries. S uggcstiotts and reco rumendatiorts Each project or distnct should be encouraged to make a plan for sustainable trarning and supervision, as soon as possible - and to move into the implementation of that plan beforc the APOC subsidies dry up. Training must become 'leaner and meaner'. Innovative approaches should be encouraged and shared - for instance, sharing transport between programmes (already being done); training school teachers or other members of the community to be involved in supervision (already being done); conducting training during routine supervisory visits for other purposes. There is a need to ensure that each country has a reservoir ofthe technical expertise that onchocerciasis control needs, at senior management level - especially epidemiology/ disease control, biostatistics, medical entomology and medical sociology. Available expertise in the country should be investigated, and shortages that are identified should be remedied before 2007, by a carefully managed scholarship scheme. It rs recommended that APOC should budget yearly for this exercise during Phase 2. a 2.4 Community involvement CDDs: selectiqn, mqtivation and ince Fi r t dtn gs an d o bservatiort s The sustainability of CDTI will depend on the extent to whrch communities take responsibility for the organisation of activities and take necessary decisions. The success of the Progranrme in involving thousands of communities in an aspect of their own health care (CDTI in this case) has been phenonrenal. Technical repofis of 21 projects in 1999 rndicate thatl9%o of communities selected their own CDDs. The method of selecting CDDs takes various forms: open electron at village rneetings; approval at village rneetings of people who had volunteered their services; or selectron by village authoritres. In a number of cases health workers selected the CDDS from among people known to them rn the village - this was especially the case when previously selected CDDs had dropped out. One project had compared the perforrnance of the various categories, and found that those appointed by village authorities (rather than being selected at an open meeting) showed less commitment to the job, achieved lower coverage rates, and were more likely to resign. Implicit in the concept of 'community directedness' is the understandir.rg that CDDs, being members of a close-knit village community, should not ask for remuneration for rendering a service to their kin. This aspect is being strongly emphasised in some countrtes, to good effect. Many community members expressed satisfaction with the Programme, and were wrlling to continue drstributing and taking the drug for a longer period. On the other hand most cornmunities are only partially aware of the CDTI philosophy - 'ownership' is a tenn which is not well understood, and many conllllullltles regard rt as a progranxne which is funded and which wrll contrnue to be so. The idea that fundrng will be phased out is difhcult to comprehend, and even more difficult to accept. Iu all the countnes visited by the evaluation team the payment of incentives to CDDs was an issue. In rnost they received an incentrve - otten in the form of a per diem allowance for training days. Some few plojects were still paylng them regularly. In son're communities CDDs were released frorn community duties, or a comnunity levy was paid. Where they received no tncentive and few signs of community apprecration therr motivation to continue was badly affected. Some CDDs crted other programmes which pard village level workers. The inconsistencies across similar community based or community drrected programmes with regard to paymcnt of incentives is a touchy issue which cl'eates divisivetress and competitlon across progralttmes. ln Canreroon a full 'cost recovery' systenl had been dcvcloped, by chargrng a small amoullt pcr treatntellt. 'lhe flunds gencrated rn this way were beitrg used to build up a cash reservc fol the luture, ancl also to pay a frxecl proportron of the lncorne as a caslt iucenttve lo all those rnvolved rn CDTI, hom plovincral officials to CDDs. In thrs srtuation there are higlt expectatrons that thrs rlteasure wrlI ensurc sustarnabilrty, although the administration of the systenr ts complex and not wrthout problen-rs. Thc { --, a a a a a , a a aa JAF6.9 Page 25 evaluation team had reservations however about payments to higher level officials. Issues of concern There is a varying but definite rate of attrition among CDDs - up to 20Yo per year in some projects - for which the reasons are as yet poorly understood. Per diem allocations during training are one of the most corrrmon incentives given to CDDs. It is not clear how these incentives will continue after the end of the project period. It cannot be assumed that district healtli budgets will provide for these training costs. If they cease then it will be hard to train new CDDs, and it will be increasingly difficult to keep them motivated if there are no rewards. Some CDDs complain that the decision by their communities to opt for door-to-door distribution effectively passes the buck - to the CDDs. Centralised distribution on the other hand would require all to pull together - and be much less onerous for the CDDs. S uggestio rts and recontmendatiorts Operational research is needed on a number of topics. The phenomena of attrition, community involvement, CDD motivation and incentives need to be more thoroughly understood - as well as the interaction between these issues. This is needed if appropriate corrective measures are to be taken and sustainabiliry achieved. For example, the most appropriate forms of incentives in different contexts need to be identified. Such operational research needs to be undertaken by the projects themselves. The existing example of cost recovery should be closely researched, and compared with the experience of cost recovery/ sharing in other community based health seruices, e.g. revolving drug funds, EPI and family planning. Health educatiorV IEC Findtngs and observatiorts a a a a Prolects are aware that, if CDTI is to become sustainable, the rationale for treatment has to be widely known and understood. Accordingly APOC Headquarters, NOTFs and pro.jects have carried out a large amount of sensitisation of all levels of the health service, and locally in comr.nunrties. In some countries much work has also gone into sensitising political leaders at all levels. At the community level health education is often largely oral and is dependent on effectrve flace to face comt.uunication. In the projects visited knowledge and understanding varied considelably. In many of the communities there was a surprisingly high knowledge of the drug and its action, including the need for repeated treatment. On the other hand there was little understanding of the fact tl.rat the 'project' was of limited duration, and that communities would have to take on an increasing role. The situation concerning IEC materials varies widely. In some projects and NGDOs these have been developed scientifically, based on extensive KAP studies, and shared with other projects - these were reported to be extremely useful. In some situations materrals were too ferv in quantity, or not produced rn the local language. In other sihrations materials were either absent ol of poor quality: unattractive and containing unclear messages. Issues of conccrn In spite of all the efforts that have been made there were wrdespread reports that health administrators (at district, regionaI or national level) and polrtical leaders (fund holders) rvele not well informed about APOC if thcy were not drrectly involved in the Programme. 'lhe partlclpatlon of the Minrsters of Ilcalth of Partrcrpating Countries in JAF, whrle an excellent rdea, is not alu,ays consistent. 'lhe development of IEC materrals has often been haphazard, and rnsuflicient anloullts have been printed and distributed. Although they have usually been tested for relevancc, thcrc have beep no studres about therr effect on coverage. Little has been donc to integrate teaching about onchocerclasrs and/or (lD'l'l into curncula lor predrcal students or health professronals. Where it is rncludcd rnaterral may be outdatcd and rmpractrcal. a a a a a a aJAF6.9 Page 26 S uggestio rts and recommendatio rt s Continued and sustained sensitisation about the Programme is needed at all levels, both in the health service and also in political struchrres like provincial, district and local govemrnent. It should be a major priorify for APOC and national prograrnmes. IEC materials are needed but they should be carefully developed (on the basis of a KAP study wherever possible) and pilot tested. APOC could draw on known expertise in this field - one NGDO is particularly skilled - to facilitate this process in national programmes which are lagging behind. Modules for inclusion in the curricula of health professionals should be developed by APOC for adaptation at country level (there is evidence that this would be welcomed). Serious side-effects Firtdirtgs and observations It is now known that particularly serious side-effects occur where loiasis and onchocerciasis are co- endemic, and a small number of deaths resulted in Cameroon in 1999. The reaction of all partners to these cases was prompt, strong and effective, and is highly to be commended. The Ministry gave full support to APOC and assumed responsibility. TCC drew up a policy to guide future distribution, and guidelines for medical treatment of SSE were drawn up. Training of all relevant health care staff in the management of SSEs has also being conducted. Efforts have been made to ellsure that units at sub-district and district level stock the necessary drugs to treat SSEs. Srde-effects, being conrmon (although seldom serious) are one of the reasons cited by those who refuse to take the drug. One advantage of CDTI over mass distribution is that rt makes it easier to follorv up, explain and treat side-effects. CDDs and health workers who are properly trained to recognise and treat side-effects are generally effective in dealing with them. Issues of concern There is evidence that distribution is being carried out in areas of knorvn loiasrs co-endemicity, wrthout all the requirements that TCC has laid down being observed. Thrs is a cause for grave concem. In particular: * The risk to villagers is being played down - as one CDD said, 'How can I tell people that what I am about to give them could kill them?' + Surveillance of each village after treatments is not taking place as rrgorously as it should. In fact the requirement that a health centre nurse should be present rn the village from the third to the fifth day is highly problematical - one 'health area' may have up to 40 villages in it. At best nurses will be able to do a rapid visit daily to each village, and even that will require motorised transport. * Transport for urgent referral is not readily available in a number of villages. It appears that in principle a process of rapid epidemiological mapping for loiasis has been approved. However the technique (microscopic examination of blood) is at present cumbersome, and a suitable selological test has not yet been located. It is also not clear who should be financing the exercise. On several occasions fear of SSEs was mentioned as a reason for people refusrng to take ivermectin. On the other hand there were also reports of communities saying that they would take the chance, since the benefits so clearly ouhveighed the risks. 'fhe problem rs likely to fade fairly rapidly, since SSEs only occur with the first dose. However coverage rates are still low, which means tl.rat many people are still due for therr first dose. Vigilance tl.rerefore has to be maintarned for the immedtate future. S uggestio rr s and recorrtrrrendatio n s An urgent lnvestrgatlon rs needed, to deterurtne to what extent TCC polrcy rs berng applred rn co- endemic areas. Pro.;ects tn these areas must bc very strongly advised to adhcre to policy firlly. Ilealth centrc llurses in affected areas urgently need some fbrm of motorrscd transport, to enable them to carry out surverllarlcc as lecolnmended by the TCC drrectrve. If thrs rs uot vL-t available Al)OC rvrll a a {'} a a a a a a a a o 5 aa JAF6.9 Page 27 have to budget for it - it is unlikely that the MoHs in affected countries will be able to do so. A decision must be taken as soon as possible, to launch rapid epidemiological n-rapping for loiasis in suspected co-endemic areas. In otl.rer countries where SSEs are likely but have not yet occurred, the lessons learnt and policies developed must be applied immediately. Involvement of women Findings and observatio trs a The involvement of women in CDTI programmes varies considerably from country to coultry and from region to region. In general this mirrors the status of women and the cultural context - ultimately it is communities who select the CDDs, according to the 'community directed' paradigm. The principll of community directedness (which tends to sanctify a status quo of male dominance) is sometimes in conflict with the goal of greater female participation in the Programme; and in such cases it is not clear which takes precedence. As far as APOC Headquarters is concerned, there has been a major effort to involve women participants in the independent monitoring teams - this has been seen as olle way in which projects and communities can be encouraged to think about the involvement of women. At project level and in the health service the number and position of women in the Programme varies, but in general men predomirate. There is little evidence to date that projects are encouraging awareness of gender issues, or are particularly promoting the participation of women at village level. There is also lrttle evidence however that having female CDDs encourages greater coverage or treahnent among wonlen. In some sihrations there was anecdotal evidence that female CDDs, while being less available, were more diligent and commrtted and perform better than men. Issues of concern There is insufficient data about the impact of the gender of CDDs in different settings. Gender distinct data is not collected, which means that little is known about coverage and refusal rates for men and women. This distinction is therefore not taken rnto account in further development of the CDTI strategy. S u gg es t io rts an d reco m fi, endatio rts Operational research could be carried out to determine whether it is important to collect gender distinct data about coverage, and to look at the relative performance of men and women as CDDs. It is certainly possible for the Programme to be more active in promoting the recruitment of women to management, as well as encouraging them to become CDDs. 2.5 Monitoring Fin d ings and recont m en datton s Independcnt monitorirtg (i.e. monitoring by teams of independent experts) has been a feature of the Programme. Of the 48 CDTI projects which had been approved by December 1999, eleven were monitored in 1998 and 16 in 1999. These monitoring exercises have been highly valued in most countries. The teams produce high quality reports, and their recommendations are important in determining the strategy and approach of projects. They work in a highly participatory way and this means that they make a significant contribution to capacity building arld team building i1 the projects and national programmes. There is a urove to make the exercises ilcreasir-rgly participatory. These nlonitoring exercises could well serve as a useful n-roclel for other a a a a a a a a aJAF6.9 Page 28 programmes aiming to develop participatory monitoring methods. Srnce independent monitoring is very expensive, the decision has been taken that fuhrre teams will be nrade up of nationals from other projects, rather than using persons from otl.rer countries. This is to be welcomed as a more sustainable option. APOC has developed and is testing contmunity self-monttorirg tools and methods, which are complementary to independent/ joint monitoring. These tools will help communities to collect appropriate data and use it at community level, before sending it routinely to peripheral health facilities and to the APOC monitoring and evaluation unit. The examples observed by members of the evaluation team were pilot activities, from which it was not possible to draw conclusions about successes or constraints. Issues of concern Tl.re cost of the external monitoring exercises is high and this will not be sustainable wllen APOC fundrng ceases. Community self-monitoring may, in the future, be a valuable and affordable way to monitor projects at village level. It will not however provide the focus for the capacity development, exchange of skills, development of innovative ideas and programme advocacy which have all been important aspects of the monitoring exercises. The monitoring of CDTI by means of routine data collection within the r.rational health information system has only partly begun in some countries. The problems experienced with reliability of coverage rates have been dealt with in section 2.1. A key element of supervision is the linkage between expenditure and results. At present projects are supposed to calculate the 'cost per treatment' routinely in their reporting, but it is not clear if cases with high cost/ treatment ratios are followed up. S uggestio tts and recornmendattons lndependent/ joint monitorrng is a key way of maintarning quality of project work and the motivation of those involved in them. APOC should evaluate the new in-country monitoring carefully, to see if it produces the same beneficial results. Funding for its contrnuation sllould be alranged before APOC's mandate expires. The NOTFs must work actively towards the integratron of basic monitoring data into national and distrrct lrealth systenrs. The official health information system should include data on CDTI wherever appropriate. Reports on onchocerciasis control activitres (health education, training, treatment) should be regularly rncluded on agendas of district health management teams. NOTFs should begin arranging tranung for the different levels of the health service, in the utihsation of routine CDTI data for rranagement purposes. Projects wrth hrgh and low cost/ treatrnent ratios should be inspected on site, to clarify the reasons for such srtuations. 2.6 Operational research F in din gs an d o bservutio tt s a o a a a a a ft a :t a There has been a fruitful partnership between APOC and the TDR. In particular the REMO technique was developed by the TDR task force on operational research for onchoccrciasis, and has been a crucial tool in developing and implementing the APOC strategy. A nurnber of bodies have been couductrng operattonal research in mernber countries, related to drflcrent aspects of the drsease and it control. Topics for operattoual research are identified in a number of ways Nccds are rdentified during the deliberations of TCC; APOC Headquarters staff identify needs dLrrrng supcrvisory visits; proyects bnng forward rcsearch rdeas to TCC (some of which have been approved); other bodres such as research lrlstrtutes identrfy yet further ones. Although thc volume o[ rescarch conductcd is largc, rt also appears poolly co-ordrnated Thcre is no a a a JAF6.9 Page29 specific in-country mechanism for identifying operational research needs and responding to them, nor for co-ordinating such research, nor for distributing the results and optimising their utilisation. Issues of concern The lack of co-ordination in operational research is of particular concern. In one of the countries visited no fewer than seven different groups were known to be involved in onchocerciasis related research. In tlre countries visited there was a great variation in knowledge of and capacity for operational research. There was a demand for capacity building in this area - in particular in the area of health systems research. The evaluation team observed that few of the research questions which were being dealt with seemed to have been identified locally by projects. In stead, most questions seemed to have been generated at regional level. Decisions about funding such research are taken by APOC/ TCC, which is not sustainable. S uggestio tts and recomntendatio ns a a a a a a a a Countries which have little capacity for operational research should be encouraged and enabled to develop this with the support of APOC staff and the TCC. Key staff at the national level could be identified and trained, and national instihrtes encouraged to participate (see also section 2.3 above). There is a clear need for co-ordinating onchocerciasis related research in Participating Countries. At present the only body which could reasonably do so is the NOTF - perhaps through a suitably skilled subcommrttee. In the interests of sustainability, review of proposals could be delegated (or partially delegated) to such a body. Once the necessary capacrty has been built, NOTFs should process proposals, based on operational problems encountered. Several problems were repeatedly raised, as requiring elucidation through research. These particularly coucerned issues at the coalface: causes of absences and refusals, cost recovery; incentives; the link between onchocerciasis, epilepsy and ivetmectin treatment. Substantral researclt is also needed into questior.rs sur-rounding the issue of integrating other interventions into the CDTI progranxne, or linking them to rt Projects should be encouraged to bring forward their particular needs. 2.7 Impact assessment Ftndings and observatiorts ln 1997 rt was decided that impact assessment studies would be canied out that would pemrit the evaluation of the long term impact of CDTI on transmission, eye drsease, skin disease, and socio- economlc status. Progress since then has been steady. The protocol was drawn up and approved by TCC, and the study was piloted in a district in Cameroon. Subsequently different teams were trained together in the methodology. Baseline data has been collected in l3 sites in nine APOC countries. Socro-demographic data has been collected in all sites visited. The studies have established baselines for entomological indices, as well as for pattems and prevalences of different manifestations of skin and eye disease due to onchocerciasis. ,Issrrus of concern The studies wrlI grve an interesting account of changes/ developments in relationship to onchocerciasis prevalence and morbidity. Measuring the impact of CDTI on socio-economic status wrll be drfficult because of the ethical ir-npossibility of rncluding controls ur the study desrgn, rvhrch means that the elfects ol'other possible factors cannot be controlled for. Socio-econonllc status is likely to be more stror.rgly affected by polrtrcal and macro-ecotromic factors; by tl're current AIDS eprdemic; and by natural dctermrnants such as weather. The qualrty of thc prelin-rinary entomologrcal results of the baselrne studics rvas unsatisfactory - lor cxample grving averages fol different epidemiological zones together. These studies are by nature long-ternr, complex and expensive. It rs not clear rvho will take charge of thcnr, and bear the cost, when APOC ceases lts operations. a aa aJAF6.9 Page 30 S ugg estio rts and reco m nte n datio rt s In view of observed problems with the entomological aspects of study execution, it is recommended that a panel of experts be commissioned to review progress to date, and to suggest possible refinements for the next phase of the study. A home must be found for the study, when APOC ceases to function after 2007. Although the studies did not set out to measure the impact that the Programme rs having on the health services, it appears from data collected so far that such an effect is taking place ur project areas. Since it is too late to conduct baseline studies in this regard, other shrdies of impact will have to be commissioned. (e.g. post hoc, case control studies). A thorough review of the socio-economic impact studies done to date should be conducted to determine if appropriate and relevant data are available, without huge expense incurred in collecting it. 2.8 Sustainability The issue of sustainability appears under the rnajority of headings ir1 this report. In this section important strands are drawn together. F indings s tt d o bs ervatio rt s Tlrere ts considerable divergence of opinion about the format of partnership after 2001 , and about the future of NOTFs. There rs agreement though that a sustained partnership could well act as a motor providrng energy and support to CDTI. Much work has been done to make drug procurement and distnbution sustarnable. In some countnes there are however strll aspects of both procurement and drstrrbution whicl.r depend on APOC funding. There rs unrversal agreement about the need for health drstricts to take full responsibility for CDTI, and fol the activity to become rntegrated into distrrct routrnes. The efficiency and commrtment of the district is seen to be a key determinant of the sustainabrlity of CDDs and their community suppoft. It has become clear that projects move and develop at different rates, being influenced by a wide variety of factors of which many are not within their power to control. Earlier projects in particular were a learning experience for all involved, not least the NOTFs and APOC Headquarters itself. This indicates a need for flexibility in determining the duration of APOC support. It was noted that the original Programme document makes provision for short extensions of projects, if this rvill clearly enable them to reach their goals fully. Issues ofconcern A large amouut of energy and rnittative has been unleashed by the .;oint action of CSA, APOC Headquarlers staff, the NGDO Co-ordinatron Group, JAF and TCC, and the cascade of natronals u,ho have rn turn bcen emporvercd and enthused by them. This activity has been made possible by the avarlability lalge sums of donor money. Thc evaltuilron leom is dceplr concerncd aboul thc sus'taittabtlily of this acttt'rtv and entltusiosnt, once APOC.funditlg comas to ttn cnd There appears in r.rrarry instances to be a lack of undcrstandtng that CDTI may have to corltlnue to 2020 and beyond, wrth lrttle defir.ritive plannurg to mobilrse the resources and set rn place the routines whrch rvrll eltsure the continuatron ol' the proglanune. Son.re districts in sonte pro.lects ate acceptltlg CDTI as simply olle more actl\ lty for rvhrch they are responsrble. This rs hon'cver by no means utrtvet'sal. In many rnstances the commitmcnt to take in CDTI is not yet there, and Dl-lMTs are not sufficiently awarc ol'either APOC or CDTI. In several situattons the fact that NGDOs are involvcd at al[ "vas secn to bc an obstacle to a a a a a o a a t3 a , aJAF6.9 Page 3 I sustainability, since such involvement delays the eventual acceptance of full responsibility by the official health service. The fact that NGDO priorities may change with time further strengthens this point of view. S ugg estio rr s and reco m me n datio rt s a The desired sihration in which projects should find themselves at the end of APOC funding must be very clearly defined, at first in general terms. This should urgently be done by TCC, using irrler alia the t'ecommendations of this report, and the result given in the form of a checklist with timelines - a set of progressive indicators for each year of the project, which allow for a realistic appraisal of whether projects will achieve a level of sustainability by their fifth year of funding. The guiding principle of such a document should be intensified devolution of Headquarters functions to countries, by means of intensified advocacy, sensitisation and capacity building, leading to integration ofthese functions (see also section 4). Following this general definition existing projects (especially those in their second 2t/z fears offunding) must be enabled to make specific plans to comply with the requirements of such a checklist, taking into account the realities of their sih-ration. A similar process is required for national programmes and NOTFs - if the latter plan to disband they have to start building capacity elsewhere as soon as possible. Projects and NOTFs will need to be supported and monitored as they implement such plans. It is recommended that the responsibiliry for these processes be specifically given to one or more senior professional staff members at APOC Headquarters. To this end it will also be useful to develop a flow chart of projects which visually shows when projects started and when the five years of funding end. Some projects took an unexpectedly long time to take off, for a variety of reasons. These investments should not be destroyed for the sake of a relatively arbitrary cut-off point, when some additional support could reasonably lead to sustainability and success. In such cases it is aspects which are critical to the future of onchocerciasis control in that country that need to receive additional suppoft, namely capacity building leading to full integration into the health service. a n aJAF6.9 Page 32 2.9 Vector elimination projects Firtdings a nd observations APOC embarked on vector elimination following the advice of TDR consultants. This advice was based on attempts at focal and/ or intermittent eradication operations which had been carried out in the South-east extension area of OCP, as well as pre- control surveys performed by the same consultants in pre-selected foci imrnediately APOC had been launched. Criteria were developed for accepting vector elimination projects into APOC (size, degree of isolation, the availability of experienced operators, existence of baseline data, cost equivalence with CDTI operations). 'Practical feasibility' was however not considered as a criterion, and the corresponding factors were generally underestimated. It was decided at the outset that vector elimination should be complete, definite and fast. If one of these conditions were to be lacking the shategy would implicitly move from vector elimination to vector control - a more long-term activity contrary to APOC's declared intention. APOC would fully and exclusively support these short-term elimination projects. There has been progress in each of the four projects, to varying degrees. Baseline data has been collected; training has been conducted (with OCP assistance); equipment has been obtained; and practical feasibilily studies have begun to be undertaken (including experimental treatments). An important achievement has been the training of local health workers in the multiple tasks associated with vector control, which may be useful in future for other disease conhol programmes. There is a long-standing history of onchocerciasis control in the Itwara focus in Uganda, with APOC taking over the responsibility of seeing the project through to its final conclusion. Elimination seems to have succeeded in the main focus: no Simulium neavei larvae have been found there since September 1998, and treatments will be discontinued by October 2000. The experience built up in the Itwara main focus is likely to benefit the extensions of this focus and Mpamba-Nkusi focus, which is being initiated from scratch. Despite problems in financial management - shared by the other two vector elimination projects - the two Ugandan projects appear to be developing satisfactorily. In the light of past evidence elimination may well be achreved for the two Ugandan foci, where Stnruliunr neavet Ls the vector, there rs a high level of experlise and previous success, and pre-control data are exceptionally good. However experience has shown that elimination/ eradication is much more drfficult wlren the vector is S. dantnosum s./. (even endemic cytospecies ar.rd forms of this complex), as is the case rn the Bioko (Equatonal Guinea) and Tukuyu (Tanzania) foci. a a a a a a a Issues of conccrn In general the administrative and financial management of all four projects has left a lot to be desired, resulting in significant delays in irnplementatiou. There is conceln about the rnclusron of vector elimination projects in the Programme ln the first place. Exanrples of successful vector elimination projects are few (especially in the case of S. damnosum s.l.). Such prolects are usually lalge, expensive, complex and long-term, wrth lrttle cornmunity participation - very different from APOC's community directed approach with CDTI. It had been shown in the mid-90s that selective elimination strategies had not resulted in intermption of transmission in OCP South-east extensiou, and the recent APOC feasibility studies reveal that TDR-sponsored pre-control surveys had underestimated the need for complemelltary operational data. Concernrng the Broko pro.lect, most of the past and prcsent adminrstrative and comrnunication probler.r.rs wrll likely be overcome through iucreased NO'IF cormlitrnenl and rmproved collaboration between thc MoH, the NGDO and the WI{O country olfice, insrde thc NOTF. The organisational cal)acrty and teclrnical cxper-tise of the MoH is low, but is being burlt up. -l'he nra.lor collcern rn Bioko is ta a aJAF6.9 Page 33 the lack of access to all the sttes to be treated, because of the mountainous and forested terrain - and 100%o coverage everywhere is a sine qua non for vector elimination. The larvicide coverage will depend on the intensity of rainfalls in the Northern part of the island; full-scale ground or air larviciding in the Southem part seems impossible without damaging the dense primary forest which is under UNESCO protection. Due to these difficulties vector elimination in Bioko does not appear to be feasible by 2001 or even 2002, when OCP (which alranges the contracts for the aerial larviciding) ends. Concerning the Tukuyu project several problems have been noted. There was a year's delay in starting the three year planning phase, due to late disbursement of APOC funds and poor communications between the national onchocerciasis control office and the Tukuyu operational base. High quality entomological and transmission data are available, but they were collected during the seventies. Several considerations of strategic importance (e.g. geographical lirnits of the focus; sites for treatment and access to them; risks of recolonisation) have been underestimated or under-investigated during the feasibility study, and still have to be complemented. The first round of larviciding on operational scale was planned for July 2000, but had to be postponed because Tanzanian authorities delayed their clearance of ir-rsecticide use. S uggestio r$ and recomntendatio rts More entomological expertise is needed in TCC as the need arises, in view of the fact that it is charged with supervising the vector elimination projects as well as supervising the impact assessment studies (with their entomological component). It is fruitless to speculate about whether the inclusion of vector eradication projects in APOC was perhaps over-optimistic. APOC is the only programme with the potential to determine whether focal elimination of onchocerciasis vectors can be successfully achieved. Considering that the collection of pre-control data have been seriously implemented in the selected foci, with important national comrnitment and local capacity building, it is the opinion of the evaluation team that each existing APOC vector elimination project should be given a chance of demonstrating the feasibility of vector elimination over a short period of time. It is therefore necessary provide every possible assistance to the existing projects, to allow them to corne to a successful conclusion. In view of a general lack of experience in field operations around larviciding, it is absolutely essential that strong support by APOC/ OCP consultants to projects be continued and even strengthened. Equally importantly, the recommen- dations made by these persons, and by TCC, must be strictly followed in tlie field. Concerning the Bioko project it is recommended that the results obtained in 2000 should be carefully monrtored and used to optimise the planning and execution of the first round of treatments (2001). The decision to embark on aerial larviciding in 2002 should only be taken once the results of the 2001 campaign are known, so that vector elimination is planned in relation to results of previous ivermectin delivery operations and ongoing CDTI activities. In the meantime a detailed helicopter survey of the whole river network is a prerequisite. Concerning the Tukuyu project it is recommended that the first round of treatment in 2000 be very carefully monitored, and the results used to optimise the planning and execution of a second round (in 2001). The high level of scientif,rc expertise in Tukuyu should be reinforced at the level of field operations. The strategic planning meeting scheduled for 2001 should consider available entomological and epidemiological data together, so that of vector elrrnination is planned in relation to ongoing CDTI activities. Conceming the Itwara focus, the challenge is now to pursue close entomological monrtoring: to conturue observing the main focus following cessatron of larvrciding, and to adjust larvrciding strategy in the sub-foci until equally good results are obtained. Agarn, results of local CDTI activities should be taken rnto account. Concerning the Mpamba-Nkusi focus, updated maps of prospected rrvers, known breedrng sites and catching srtes should be urgently prepared. It could also be wortl.r while for APOC to sponsor a revlew oIongourg/ planned/ potential vector eltmination pro-;ects in Uganda and nerghbouring countrres. APOC is requcsted to convene a meeting in Tanzanra in 2001, to deterrnine APOC strategy in the field of vector elulination (as opposed to control) for Phase 2 of the Programme. Such a meeting should brrng togetl.rer all the available expertise in Partrcipating Countrres, APOC, OCP, TDR, TCC and elscr.vhele. In these deliberations duc consideratloll must be pard to thc nrcasurable and predrctable a a a a a a I a JAF6.9 Page 34 a impact of CBTI and CDTI operatlons on the epidemiology and transmission of onchocerciasis. It is likely that tlte answer to the question of the feasibility of vector elimination will be obtained in 2002. If it is 'yes' (likely in Uganda) the strategic consequences for APOC will have to be considered; if it is 'no' (likely in Bioko, uncertain in Tukuyu) the feasibility studies will have to be stopped, and recommendations will have to be rnade to the countries in terms of continuation of vector control in national or bilateral co-operation contexts. 2.10 The impact of current and future therapeutic developments F in d ings and o b servatio rts Recent animal studies have shown that a single dose of moxydectin may well be macrofilaricidal. Studies in humans are about to cornrnence. A study is ongoing to investigate apparent non-responsiveness to ivermectin in the Pru basin in Ghana. Issues ofconcern The advent of an effective macrofilaricidal drug, while very welcome, would make yearly distribution of a microfilaricide redundant. After one or two rounds of the new h'eatment the community directed treatment progranunes could cease, and only monitoring would remain to be done - for which no plan at present exists, since the decision was taken to monitor the disease by means of coverage rates. The absence of disease monitoring will also make it difficult to detect resistance, since it is discrepancies between coverage rates and disease incidence that give waming of it.. The possibility of fuhrre ivermectin resistance in APOC counh'ies is not far-fetched, given that the drug lras been used sporadically in many projects in the past. S u g gest io rr s a n d reco nt rnen datio rt s APOC should lend its full support to teams and centres which are monitoring possible uon-response to ivermectin (e.g. in the Pru basin in Ghana). TCC should consider developing criteria for the need to conduct research into this phenomenon in a given situation. APOC should lend its full support to the current trials for macrofilaricidal drugs, e.g. moxydectin (which is now at the human trial stage), and for the teams and centres which are performing this research. TCC should also give serious consideration to implementing a system of monitoring the impact of CDTI on morbidity (through determining disease prevalence and incidence in selected sites), to complement and monitor coverage rates. { Ia a /' .rt; a a a a a t JAF6.9 Page 35 3. Integr.elidil'v"i1''i: i,i:,i.r*si.*.*:ir.'.i::'.:",',;;;i.ii:," ,-''. '',;ii'1|CI., , : .:'i.i!, .7J ?/ .,,,,.';. .r')11:, .r, ii.:,,,.') 'lntegration' rn this section is taken to mean: ( I ) The degree to which CDTI becomes an effective part of the routine functioning of the health service in a country.(2) Collaboration between disease control initiatives at intemational level. Integration ofonchocerciasis control activities into existing health care structures at all levels is seen to be a prerequisites for sustaining these activities in the years after APOC suppoft comes to an end. 3.1 Integration at community level Firtdings an d observatiorts In places where the process of mobilisation of communities was either not done or ineffective, communities were not actively involved in health care work when the APOC Programme began. In communities not active in health care, the election and deployment of CDDs was a first step in community based activities, and, in some instances, served as the stimulus to revive defunct community health structures. hr communrties with some previous involvement in health care (CHWs, community health/ development committees etc.), CDDs were an addition to the communrfy base. In these situations, CDDs appeared to fit into the existing structure without problems. lssrres ofconcern The process of community mobilisation and empowerment was no effectively done in a few places, leading to poor community participation of the cornmunity in CDTI. The parallel development of a number of vertical health programmes at both the national and international level is often mirrored in the community. Communities as well as community workers may become confused about the respective duties of numerous vertical programmes. Programmes may in some instances be in competition with each other for appropriate community level workers - it was in fact found that CDDs often wore several hats, in terms of voluntary service to their communities. Suggestiorts and reco,rtrflendatiorts , .. ... j, ,! ).:i:'.: a a a a a a a CDTI has entered communities on a large scale, often for the first time, and very likely has the credibility to promote the inclusion of further activities. Care must be taken that CDTI takes place in harmony with other health and development related activities at community level (past and future). MoH and NGDO staff need to espouse the 'community empowerment' n.rodel when introducing other actrvities to communities. On a practical level rt is suggested that NOTFs lnltiate a dialogue with other disease control programmes in their countries, to clarify the situation regardrng utilisation of community level workers, and the incentrves gtven to them. Once the situation is clarrfied steps should be taken to harmonise and optrrnise the ntodus ope randi of the different progralnnles. a I 3.2 Integration at health district level Findings a nd obsert atio rts At the critically irnportant sub-district level staff generally appear to accept CDTI as sirnply D aJAF6.9 Page 36 an additional task, among their other routine duties. These nurses (sometimes doctors) are fully involved in training and supervising CDDs, although district and project level staff usually help with the training. In one case districts have been given permission to open budget lirres for CDTI. Issues of conccrn The fact that CDTI is not included in the 'minimum/ basic package' for some countries has caused problems, since there is then no budget for it. Even if it is included training, rnotivation and supervision of health centre staff will still be needed to make it a reality. This is especially important in view of reports that some nurses are getting tired of exerting themselves in the villages, for instance to replace CDDs who have fallen away. CDTI activities are carried out by distr-ict staff. However planning, informatron systems and budgeting for CDTI are in many cases done separately - i.e. are not included in the routines at district (and other) levels. In some countries district and provinciaU State level staff are given supplements for CDTI related activrties. This is a cause for concern, since it militates against integration - CDTI is reported to be seen as a 'separate' programme, and may be easier to ignore once APOC funding comes to an end. S uggestiorrs and recorflntendatiotrs APOC must spearhead advocacy for a specific plan at all levels, to lead to full integration of CDTI by at tl.re end of Year 5 of APOC support. By this time district authorities must have been enabled to plan for, train for and supervise CDTI fully. In some countries such planning will have to include civil authonties at district level, who are responsible for health budgets. It is hearter.ring to know that APOC has developed a checklist developed to monitor integration in its member countries; this may now be urgently and consistently used. It nrust be clear where the activities that CDTI requires (trarning, supervision) ale located rn budgets (whether there is a separate line item or not). Since CDTI coverage data is periodic (once a year) it may well have to remain distinct from other continuously collected health information. Progress reports on CDTI activities should form a routine part of district planning meetings. Supervision schedules must include CDTI. 3.3 Integration at higher levels F i n di ngs/ o b s e rvatio n s There is commitment at the highest levels to the tnclusion of CDTI (ar-rd sometinres vectol control) into natronal health programmes. To this end MoHs have allocated staff and resources such as offices to Programme actrvities, to varyir-rg degrees. The NOTFs have given shape to the concept of a national planning and executing agency u,ithin the Mrnrstry, rvhrch also includes outside paftners such as NGDOs and technrcal experts. They appear horvcver to have few, if any, countetparts in the Mo[{ structurc rn other programmes. lssncs ofconcern Although staff and facilities have bcen allocated, there is to date Iittle evrdence of specific rncremental budgetary allocatrons to support Programme shuctures at national and provrncral/ State level. Although dilect lrne marlagels appear well inforn-red, tt rvas noted that heads of related secttol.ls u1 lulrlrstnes (e.g. Prrmary I{calth Care) in some cases know very lrttlc. Therc rs evrdence rn solne cases that the pnmary loyalty of the NOTF rs to APOC, rather than to the Moll a a a a a a a a a a a aJAF6.9 Page 37 S uggestio n s/ reco m mendatio n s a Again there must be progress towards planning, budgetary allocations, supervision and information gatherrng - by MoH - in support of activities that are now known to be essential for the smooth running of the programme. Fresh thought must be given to advocacy, so that relevant groups inside and outside MoH are fully informed and, where relevant, involved. Again, the checklist that the Programme has developed to monitor integration in its member countries will be useful. 'Ihe future of the NOTF needs to be worked out. In spite of its official position it still exists in parallel wrth the'onchocerciasis desk' in some section of the Ministry. Its continued existence should depend on its usefulness, which may vary with time (e.g. NGDOs may change their priorities; other ministries hke Education or Local Government may be included). In either case the body/ bodies which will drive CDTI after Year 5 have to be officially approved, strengthened, and funded. This is a task for the current NOTFs, with technical support from APOC. 3.4 Contribution to health systems and capacity building F irt din g s an d o b s ervatio rts Programme activities resulted in numerous benefits and contributions to the health service. The following were reported and observed: * Technical improvements'. Training has resulted in greater technical and managerial ability (some of which has a wider application). The emphasis on accountability has introduced a new disciplined approach to resource management. The census has provided managers with information they needed, but had never had before. Additional transport has supported other progranunes, and decreased isolation of peripheral workers. * New attitudes and approaches: ln many cases the Programme appears to have led to cornrnunity empowerment. Communities were emboldened to ask for this treatment, and in solne cases were beginning to ask for even more. Staff relatronships wrth conmunities were strengthened. Successful treatmerts buttressed the credibility of the healtl-r system and boosted morale. Botl-r Ministry and NGDO staff benefited in these ways. lssrres ofconcern An overridrng concem about these contributions is that they are to a greatel'or lesser degree energised by the generous funding which APOC supplies - and which rs coming to an end. There is a clear assumption in some countries that the NGDOs will corrtinue to provide the bulk of the money needed after APOC funding has ceased. This is by no means a certainty, since donor priorities are known to be changing - for instance, in the direction of supporting health system refomr, rather than focusing on single disease entities. To date decision making regarding major issues hke project suitability, proJect perforrnance and has been largely undertaken by TCC and APOC Headquarters Management. The cornnritment of the Ministry (and NGDOs) to the 'empowemrent approach' rs not clear. Many of the initiatives whrch arose in the period immediately after the promulgation of the Declaration of Alma Ata rn 1978 were subsequently neglected or abandoned. ln general the basic training of health workers does not rnclude up-to-date informatron about onchocercrasrs control, and the role ofhealth professronals in rt S ugga st io rts a n d reco n, mendatio n s A cleal plan to empower national structures to har-rdle lnaJor uranagement lssues must form part of AI'}OC's planrring for thc second phase of rts operations. 'l'his agarn hrghlrghts thc need to plan systcmatrcally lbr the replaceureut oIAPOC funding by othe r', sus(ainable soulccs. LJrgertt dralogue rs needed at all levels, to explore thc commitnrcnt of thc natronal health services and NGDOs to thc 'community eutpou,ernletlt' approach. 'l-[re lact that the conlnlunrty dtrcctad approach a a a t a a a a a aJAF6.9 Page 38 was researched (by TDR) and found to lead to better coverage, should be a useful tool in this debate. Not everyone is convinced by the existing evidence. Additional evidence based shrdies should be undertaken to document the contribution of the community directed approach in integrated disease control initiatives. TDR's priorities in operations research may assist in funding some of these additionaI studies. The rnclusion of excellent, relevant material on onchocerciasis control in basic curricula must be negotiated. APOC should promote this activity through the NOTFs. 3.5 Future co-operation/ collaboration with other health related programmes F in din g s an d o bs ervatio rt s There is widespread acceptance of the fact that the large scale introduction of the 'communiry directed' modus operandi has opened the door for a number of other potential interventions at community level. This is a significant Programme achievement. lmplementation has already started in a number of situations, for instance for chloroquin distribution, recognising blindness due to cataract and mobilisation for childhood immunisation. At a higher level LF was mentioned as a programme which would do well to follow a 'community directed treatment'approach, since the technical aspects oftreatment ofLF and onchocerciasis have so much in cornmon. Tlrere are a variety of opinions about the degree of organuational co-operation that should exist between existing CDTI prograrrurles, and future LF treatment ones: * The advantages of closer co-operation are seen to be that duphcation of shuctures is rninimised (at district and national level), and that the lessons learnt by OCP and APOC in CDTI can be usefully employed in LF (at nafional and supranational level). * A disadvantage that was mentioned is that mistakes of APOC will be perpetuated - an over-rigid and top-heavy management structure was mentioned in this regard. * The fact that the duration of CDTI is much shorter in tlie case of LF, and that different countries are involved, makes it difficult to conceive of using the salne structures at supranational level. lssrres ofconcern a a a l) a Certain lessons from the past need to be borne in mind: * The possibilrty of vertical prograrrrnes entering communities heavy-handedly with specrfic ager.rdas and financial incentives is ever present. This presents a danger to the community directed approach (which relies on community members working for the good of their communities), and also to priority setting and systems development. * The multiplication of vertical programmes at district level (and even higher) leads to under- r.rtilisation of scarce resources. It is not sustainable to have a specialist uniquely allocated to one disease programme at district level. * Future disease control programmes which also use ivermectin lun a serious risk of provoking ser rous side-effects, if they operate in areas where loiasrs is endemic. TCC is very concerned about issues surrounding the integration of APOC and the LF elimination initiative. TCC believes that countries are receivillg mixed messages from WHO, and that the emergence of two separate programmes has the potential to be 'disastrous' at the country and community levels. The evaluation tearn shares these concenls - the emergence of multiple elimination and disease colltrol programmes at WHO has the potential to divert both resources and attention lrom onchocerciasis (especially at the uational level), and may confuse planning and resource allocation considerably. 'l'CC is conrnritted to thc CIDTI strategy for other disease control programmcs, rncluding LF, Vrtamrn A drstrrbutron, sonre actrvrties proposcd uuder Viston 2020. l'hc emergencc of- these proeran-rmes off-els thc possrbrlrty lor APOC to oflcr its expcrtise regardrng thc CDTI stratcsy to othcr <lrsease control a a a aJAF6.9 Page 39 prograrunes as a model. While the APOC Programme Document ( 1996) calls for integration generally, it does not give APOC or the APOC governing bodies (including TCC) a mandate to support activrties other than onchocerciasis. S u ggestio tts a nd reco m nt endatio rts Policy at national and district level should safeguard and build on the 'community directed' approach, for all health programme activities at village level - existing and new. An approach which allows the villagers themselves to handle new or additional activities as they see fit ntust be promoted. At provinciaV State and district level the model of an activity integrated into existing procedures and budgets is clearly the most sustainable. At national level the 'NOTF' model as a support for specific disease control programme 'desks' in MoH should be considered - and, where prograrnnes have a lot in common, shared. It goes without saying that the lessons learnt about loiasis co-endemicity should b.e applied pro-actively in any future progralnme which intends to promote mass distribution of filaricides. This would involve rapid epidemiological mapping exercises, and providing training and resources for handling the side- effects. In order to avoid possible expensive duplication of structures and dissipation of resources between different vertical prograrunes, CSA should urgently review the APOC Programrne Document and the Memorandum of Understanding for Phase l. Modifications or amendments should be made that will allow APOC to explore and work towards a sensible integration of onchocerciasis control with other disease control and elimination programmes (including that for LF, which is about to be operationalised) at all levels: international, national, district and community. a a a aJAF6.9 Page 40 4. Partnership , . 4.1 Success of the present partnership formula F in clirtgs an d o b s ervat io rt s The partnerships initiated or strengthened by the Programme embrace the Ministry of health, NGDOs, WHO, donors (including MDP and Merck, Inc.), bodies of experts and civil society. They exist at different levels, from supranational forums to the village, and at each level the different partners interact in different combinations. The establishment of such a quantity and variety of successful partnerships is a major Programme achievement. In the NOTF concept in particular the Programme has further developed a fornal model which brings together different stakeholders in onchocerciasis control, in planning and implementing a common plan for a country. This example of a private-public partnership is a significant and strategic achievement in several ways. It provides a flexible forum for non-governmental bodies to have a direct say in the planning and implementation of a national prograrnme; partners have managed to harmonise their methods and shategies; and partners have shared human, material and financial resources in working for a common goal. In most counh'ies a remarkably good relationship has developed between MoH and the NGDOs. This relationship is flexible, with partners accommodating themselves to eacl.r other, and exhibits a parlicular synergy: the NGDOs energise the MoH, while MoH helps the NGDOs to brtrg their practice into hne with accepted norns. NGDOs also provide forward funding at times, when APOC disbursements arrive late. The establishment and maintenance of the body of donors for the Plogranrme rs a signrficant achievement. It goes without saying that their contribution underpins everytl.riug that the Programme has achieved. The same is true of the generous donation of Mectizan@) (ivermectin) by Melck, Inc. a a f: a lssrres ofconcern The NOTF secretariat appears to play a key role in maintaining the partnershrps. Since much of the work of the secretariat is funded by APOC, the sustainabllity of this arrangernent rs in doubt. The view was also expressed that partners may be working together in the NOTF largely in order to gain access to APOC funds. While MoHs have clearly appropriated the Programme at a hrgh level, there is evidence that some staff at distnct and sub-district level see it more vertically, as a separate 'programme'. Although sorne NGDOs are entirely supportive of MoH policy and work entirely wrthur its framework, others have a more independent approach which sometimes runs counter to policy - for instance, paying CDDs a wage for each distribution. There is a spirit of competition between projects and NGDOs wrthin countrres, which is both healthy (e.g. trying to get the best coverage rates) and unhealthy (e.g. not sharing ideas and experiences). a a a I JAF6.9 Page 4 I Suggestiorts a nd recontrflendatiorts It is clearly essential that when APOC ends there should be a mechanisrn which continues to promote partnership in planning, monitoring and research around onchocerciasis control. The nature of such an affangement is however far from clear. There is therefore an urgent need for NOTFs to plan for the future of partnership. Since countries and NGDOs differ substantially, it is logical to expect different models to be developed in different situations. Each NOTF should be asked to develop a strategic five-year plan. This plan should indicate whether they see their role as phasing out and gradual integration with MoH, or whether they are going to continue to exist as a partnership, and to support onchocerciasis control activities. 4.2 The contributions of the partners Fin dings and o bservatio rts a The main areas of responsibility/ contribution are clear, although they vary: 'l'he scopc of activrty of NGDOs appears to be linked to thc abilrty of the MoH to deliver. Where the Statc rs reasonably effective the NGDOs take urore of a back scat, mclely supporting the efforts of MolI stafl'wrth the ir expertrse and resources. Ir.r other cases NGDOs are oblrqed to be very activc in opcratror.rs in the ficld. I MoH Accepting the programme; integrating it into the national systenr; making the necessary policies to support it; providing leadership for it; providing some of the resources for it (staff with their salaries; budgetary allocutions for running costs; office accommodation and overheads; the existing district health care system through which CDTI works). NGDOs Accepting the programme (with the CDTI approach); providing an administrative home for projects; providing some of the resources for projects (staff with their salaries; office accommodation; transport). WHO country office Technical and strategic support for NOTF and prdects; administrative support; assistance with bookkeeping; transmitting APOC funds to NOTF; purchasing of equipment from approved local suppliers; internal audit. donors As a body, providing funding for the Programrne until its end. MDP/ Merck, Inc. Providing Mectizan@ (ivermectin) free of charge to the Pro gramme, for as long as necessary; making arrangelnents for its movement from factory to country. communities Selecting and supporting CDDs; collecting ivennectin from certarn points; taking the medication. APOC Providing funding (maximum 75o/o) and resources; supervision/ n-ronitoring and advice; advocacy; resource mobilization; financial management; technical problem solving. a JAF6.9 Page 42 Issues of concern A chair-r is only as strong as its weakest link. There are several examples of problems which arise when one or more of the partners is not performing to standard. The severe problems caused by the frequent rnability of tlie WHO country office to cope with its financial responsibilities have been described. APOC Headquarters has caused delays in the disbursement of funds. In sorne countries the MoH is so dysfunctional that NGDOs have to bear almost the entire burden of the Programme. In others NGDOs experience problems - with staffing, financial control, planning - which make it difficult for them to fulfil their responsibilities fully. Problems at community level have led to a failure of volunteerism. The contribution of MoH varies. In some countries it is substantial - the initial commitment has been backed up by an allocation of resources (although it is difficult to see how these could ever approach the replacement level for present APOC funding). In other situations there appears to be a lack of understanding that the partnership requires MoH conhibutions as well, so these are never made. The mobilisation of district level funding has only begun in a few cases. In some countries both operational capacity and general PHC infrastructure are woefully inadequate, and a very large amount of sustained capacity building is required for the health service as a whole. A situation where the CDTI programme is the only properly functioning entity in a sea of inefficiency and demoralisation does not bode well for sustainability. In general NGDOs subscribe wholeheartedly to the partnership, and honour their commitment to projects. There have however been cases where they resist change, preferring to follow their pre-APOC modus operandi (for example by refusing to conduct a census to provide a denominator for coverage rates). There are projects in which the NGDOs are not financially transparent, even in their dealings with APOC. Also, although NGDOs have confirmed their commitment to seeing the projects through, there are clearly going to be many cases where their support after 2007 will still be required - but their priorities may have shifted by that time. The bottleneck caused by lack of capacity in the WHO country offices affects many more areas than financial disbursement. The potential for technical and administrative suppo( that could be offered to NOTFs and projects is very far fi'om being realised. t a a f'r\v a a Suggestiott s and reconxntenddtiotts The overall aim of partnership must be the integration of CDTI urto tl.re routine fur-rctioning of the health service of each member country. Where there is no functional health system, CDTI should serve as the entry point for its development. In each project the spirit of partnership that has developed must now be put to use at each level, while APOC funding remains - a planned and orderly process of task allocation to each partner that will lead to sustainable CDTI. The role of the donors is naturally crucial to the Programme - they have demonstrated commendable commitment. However their priorities are not static, and recent changes towards (for example) poverty alleviation have been noted. The Programme urgently needs to engage with donors in discussion about the future - if it fails to take a creative and forward thinking role in these discussions it will find it hard to maintain donor commitment. a I tJAF6.9 Page 43 4.3 National and international NGDOs Findings and observatiorts There is widespread agreement among partners that it is desirable to promote local NGDOs as partners, in view of the need for sustainability. There is also agreement that these will have first to be identified, and then to receive administrative and financial support (for example from international NGOs or the WHO country office). At present the NGDO partners in all projects except one are intematiolal. The one exception is a national NGDO which has been groomed by an international one, and which is now subcontracted to take responsibility for a few districts within a project. This is a yery promising and significant development. fssues ofconcern In one country a local NGDO has made two project submissions to TCC, but these have been turned down due to a perceived lack of capacity. The problem is not that skilled local staff cannot be found - the international NGDOs run mostly on local staff anyway - but that local NGDOs find it hard to raise substantial funds within their countries. The level of APOC funding at present is so high that, in order to qualify as partners, local NGDOs would be obliged to rely on international funders - thus becoming in effectjust another layer ofbureaucracy. Experience in other programmes has shown that many national NGDOs come into being when it is known that funds are available for a particular purpose. The few which succeed and last have tended to work in urban areas. There is evidence that progress in implementing some national plans is being retarded, because of a failure to find NGDOs (even international ones) willing to take on more projects. S uggestio16 and recontmerudatiorts a a a a a a The one successful example points the way: NOTFs should identify potential local NGDOs, who are then to be trained and mentored by international ones ur-rtil they are ready to be subcontracted to take on work within projects. Finally they will be able to function independently. Each international NGDO should in fact consider partnering with an appropriate national or local NGDO to build capacity and expertise in fund raising, programme implementation and financial management. It may be useful for NOTFs and TCC to accept a broader concept of what a 'suitable NGDO' Iooks like, beyond vision/ blindness related NGDOs. For example the possibility of working with local NGDOs dealing with general community development, or even village associations, should be investigated. At the same time NOTFs and TCC should continue to be vigilant in screening applications, ensuring that approved national NGDos are able to deliver the goods. Where NGDOs cannot be found who are willing to take on new projects, the tried and trusted OCp model of a bipartite partnership (APOC and official health service) should be considered - again, working towards full integration within a five year period. JAF6.9 Page 44 D. Conclusion In the first four years of its life the APOC Programme has made signif,rcant and satrsfactory progress towards meeting its objectives. The groundwork has been done: management systems and adn.rinistrative processes are in place, which on the whole are sound and effective. Working partnerships have been established at many levels, from international to community, which are contributing significantly to the success of Programme efforts. The capacrty of the health services in participating countries has been significantly enhanced, so that they are actively involved in implementtng onchocerciasis control - as are myriads of village communities. [n 1999 16 million persons were treated with ivermectin, and work has begun on four vector elimination projects. Progress is being scientifically monitored. A unique prograrnme has been created- The model of community directed treatment that it has been developed and tested holds great promise for other existing and future disease control programmes. As it enters Phase 2 of its operations the Programme is called upon to: . Continue expanding at the present rate, so that overall Programme objectives may be achieved by 2007. This will require, inter alio, more streamlined governance and administrative procedures, and innovative approaches in countries with significant securlty problems. o Intensify its efforts to integrate onchocerciasis control fully into the health services ofthe participating countries, in order that the Programme's achievements may be fully sustained rvhen its inputs come to an end. JAF6.9 Page 45 E. Appendices Appendix I .Terrns of reference-;,r.: TERMS OF REFERENCE The African Programntefor Onchocerciasis Control (APOC) External Mid-Term Evaluatiort 1. Introduction and Background The ob.yective of the African Programme for Onchocercrasis Control (APOC) is to establish sustainable, community-directed ivermectin (Mectizan@) dehvery approaches covering 50 million people in 19 countries' which fall outside the scope of the ongoing West Afrrcan Onchocerciasis Control Programme (OCP). These systems will be established and become entirely self-sustaining within the 1996-2007 time period. It is estimated that a minimum of 15 million people living within the APOC countries are currently heavily infected with onchocerciasis. The principal tool for controlling and eventually eliminating onchocerciasis as a public health problem is ivermectin, which is being given free-of-charge by the producer, Merck, Inc. for "as long as needed". One dose of this drug given annually reduces the load of rnrcroscopic, larval worms in the human body by up to 95%o, without serious side effects. It thereby relieves intense itching and prevents occurrence of blindness. It is estimated that as many as one billion fi'ee 3 n-rg. Mectizan@ tablets will be drstributed over the twelve year life of APOC, having a value runnlng into the hundreds of urillions of dollars. The total cost of APOC over the twelve year penod is estimated to be US$124 million to be contributed by the donor community and an additional of 25o/, of this amount will be borne by the Non-Governmental Development Organization (NGDO) Coalition and the African countries. The average cost per person treated per annum after establishment of sustainable community-directed treatment with ivermectin (CDTI) will be approximately US$0.25. APOC will protect the donor communitlz's substantial investment of US$560 million in OCP. OCP has succeeded over the past 25 years in virtually eliminating onchocerciasis from an eleven-country subregion in West Africa. By controlling onchocerciasis tn Ntgerta, APOC will protect the entrre eastem flank of tlie OCP subregion from re-rnvasion by infective blackflies to re-establish the disease. l7 donors and all 19 Afrtcan countries have signed a multilateral agreement, bringrng the programrne legally into force as of April 1996. 20 donors have committed US$ 56 mlllion in financing for the first srx years of the program; and 57 projects in twelve countries had been approved for frnancrng by July, 1999, to alleviate suffering and prevent bltndness aulong a total population ol up to 32 mrllron people. ' At.,gola, Buruudt, Canreroott, Central African Repubhc, Chad, Dcmocratrc Republic of the Congo, Republic of Congo, Eqtratorial Guinea, Ethiopia, Gabon, Kcnya, l-rbcr.ia, Malawr, Moza n.rbiquc, N rger-ra, Rrvanda, Sudan, Tanzania, [Jgancla o 3a JAF6.9 Page 46 2. Objectives of the Review The objectives of the external mid-term review are: To assess progress 's objectives, incl an eva operations; and to make appropriate recommendations on the Programme's strategies in order to fulfil its objectives by the year 2007. 3.1 Terms of Reference Programme Management To assess the adequacy of management and staffing at the Programme office in Ouagadougou and the continued effectiveness of the Technical Consultative Committee (TCC) in the light of increasing Programme operations. To examine the financial management and controls of APOC vis a vis projects in the field. To evaluate synergies, if any, gained by housing APOC and OCP together in Ouagadougou. 3.2 Programme Implementation To review and evaluate the performance of the Programme to date with particular emphasis on the operationalisation of its CDTI strategy and the effect on coverage and sustainability since its inception in 1996. To examine the strategic prospects for achieving the Programme's objectives by the year 2001 and assess the projects' ability to function after five years offunding. To assess the adequacy of monitonng procedures and to suggest improvements which might be made. To review the participation of women in CDTI and consequently identify areas of improvement if any. To evaluate operational research and application of rts findings. To assess the role of the Programme in conflict/post-conflict onchocerciasis endemic areas and on drsplaced person populations. a a a \i a a a a a a a a a 3.3 Coveragc and Sustainability of the Programme To revrew coverage achieved by the Programme since its inception, as well as methods to expand coverage further. To assess the impact of community involvement in CDTI on levels of coverage. To review initial progress made towards enhancing the long-term sustainability of CDTI and make suggestrons for improvement. To evaluate the effect of health educatron on attitudes and behaviours of local communities. 3.4 lntegration of CDTI into the Health Systems To evaluate the progress towards integrating Communrty-Directed Treatment with Ivermectin (CDTI) rnto existrng health systems in the Participating Countries. To exarnine APOC's contribution to health systems development rn Partrcipating Countnes, particularly the achievements of the Programme ln local capacrty building and assess its rnvolvement in the sector-wide approach to developn-rent. 'fo research the rrnpact o[ the Programme on the qualrty of local health servlccs, including Prrmary Health Care especrally in the poorest communrties. 'l-o assess the prospect of rntegrating the control of additional drseascs rnto APOC operations. a a o .I JAF6.9 Page 47 3.5 Partnership To assess the technical, financial and operational contributions of the partners at the country level, namely the Ministries of Health, APOC, NGDOs and other partners To examine the relationships between international, local NGDOs, and local communities in regard to capacity transfer. a a 4. Organizational Aspects The evaluation will be organized by the Committee of Sponsoring Agencies (CSA). The evaluation will be carried out by independent experts. The CSA, with input from the donors and the NGDOs, will finalize the composition of the Evaluation team, insuring that the required expertise is represented, and arrange for secretarial support. The evaluation shall take place during 2000. APOC staff will provide information to the Evaluation team, and the Mectizan@ Donation Program (MDP), the NGDO Group and members of the Technical Consultative Committee may be invited to act as resource persons where required. Evaluations of the roles of the various parbrers, e.g. NGDOs, Participating Governments, and local communities, will be grouped together and separated from the evaluations of APOC per se in the body of the report. Field visits to the APOC area will be arranged as required. The evaluation report will be considered in final draft form by the CSA and presented to the December 2000 session of JAF for adoption P :\ ! UNITS\AFTF3\APoc\revised TOR.doc a a a a :l 3 JAF6.9 Page 48 Appendi x.2 ;,' "''. ; ThC evaluatio i,t)Eam ;- ;i? : Dr Olikoye Ransome-Kuti (team leader) 20-A Elsie Femi Pearce Victoria Island, Lagos, Nigeria tel: **234 470 1544 (H) e-mail : ab frco@infoweb. abs.net Prof Detlef R Prozesky (secretary) 90,22nd Street Menlo Park, Pretoria 0081, South Africa tel:**27 123461875 (H); **27 12 354 1147 (W) fax:**27 12354 Il58 e-mail : dprozes@postillion.up .ac.za Ms Catherine Hodgkin (assistant secretary) Royal Tropical Institute PO Box 95001, 1090 HA Amsterdam, The Netherlands tel: **31 20 568 8458 (W) fax: **3I 20 568 8444 e-mail : c. hodgkin@kit.nl Dr Deborah A McFarland Rollins School of Public Health, Emory University 1518 Clifton Road, Atlanta, Georgia 30322, USA tel: +x | 404 727 7849 (W) fax: **1 404727 4590 e-mail : dmcfarl@sph.emory.edu Dr Bernard A Philippon Institut de Recherche pour le D6veloppement 213 Rue Lafayette, 75010 Paris, France tel: +*33 1 4803 7709 (W) fax: **33 1 4803 7806 e-rnai I : phi lippo(rDparis. i rd. fi' Dr Alexander S Muller 42 Zandpad 3621 NE Breukelen, The Netherlands tel: ** 31 346 263 793 (h) e-mail: mimuller@worldonline.nl Dr Eleuther Tarimo c/o WHO Olfice PO Box 9292, Dar-es-Salaanr, Tanzania tel: x*255 51 725051 73568 lax: **255 51 71720 e-nrai I : eleuther@u d.co.tz ..:.. a {'. JAF6.9 Page 49 Appqnd,rx3 ,i{ri/,ii:t 'TFei{1,!{1;c"6.11$fitrXii,HW,€!tf.,Nl; Z,',Ni,,:'";;i,i',.\W# The instrument uses document study, interviews and observations, at the following levels . level /.' supranational (CSA, headquarters staff of intemational NGDOs, NGDO Co-ordination Group, AFRO headquarters in Harare, Geneva stafQ . level 2.' Programme (APOC Headquarters, TCC, OCP Headquarters) . level 3.' nationaV provinciaV state (MoH headquarters, NOTF, state/ provincial level, external monitoring teams) . level 4: projectl district (project staff, Iocal NGDOs, district management teams) . level 5. sub-district (health centres/ clinics, clinic committees) o level 6: villagel community (CDDs, village authorities, villagers) Since the actual instrument is very long, only the section covering Level 4 is given below LEVEL 4 - Document scrutiny: project/ district level [3 4 A ut the effecl of health cducatiort ott reports on in projects education undeltaken uttitudes anrl lrchaviours of local cotttrttunitrcs * the oncho. related health education currently being done in projects health education materials available m projects to determind lookfor documents to examine [1.4J About the project financial system and how it works * sources of income * the present flow of money from donors to projects * nature and relative proportions of line items in budgets * the financial control system; inadequacies in it * purchasing, issuing, stock control procedures * control oftransport * training and other suppoft for staff handling finance flow charts of disbursement of funds manual of financial policies and procedures project financial files budgets for projects; the plans linked to these 6 monthly (or more regular) financial reports of expenditure per budget item documentation related to vehicle control: trip authorities, log sheets, service records plans for/ reports of training staff in hnancial procedures :t * * * * auditor's reports [2.laJ About operationalising CDTI; coverage and sustainabilitlt [3.1J About coverage achieved * progress with REMO; whether REMO results are being used in nranagement * plans to achieve wider coverage; actual coverage achieved; trends in coverage * what projects means by 'sustainability'; plans to achieve it; the success ofthese * struchrre and efficiency of the official health service and NGDOs; the human resources available; whether shucture and human resources can cope in fuhrre strategies and plans for CDTI: annual plans all data on REMO: proportion of country covered, number of areas identified as needing CDTI all data/ reports on CDTI treatment rates: proportion of regions, villages, village populations covered * * statements of [2.5] About women's participation in CDTI * the present situation: roles and proportions of women in the Programme at different levels * whether information systems are gender distinct; cases where drstinction seenls nccessary * rvhether there is a difference rn comphance rates between men and women in the villages reports with evidence of proportions of wornen as CDDs/ other levels of worker (e.g. project annual reports; surrunary reports of coverage) report forms at all levels rea lt [a.2aJ About APOC and health service det,elopment: local capacity building * previous (pre-1996) capacity building done by NGDOs * capacity building APOC/ NGDOs/ MDP has done for CDDs, health workers, managers (since I 996) * how this compares with the capacity they should have for thejob * the qualiry of the training being done * descriptions of competencies needed for CDTI at each level (if available) * staff development plans (if available) * course timetables and other training materials * reports of training events (including student assessments) [4 2b] About APOC and overall health sector development * generic skills that APOC training has imparted (i.e. transferable to other health system activities) ,* training course objectives/ timetables [5.1J About contributions of the dffirent partners at country level * who the partners are at each level (national, State/ project, district) * the technical, financial and operational contributions of partners at each level (including ivermectin distribution) * the rntegration of ivermectin into the countries' drug supply system * how well the partners work together * gaps in the partnership, which cause operational problems * Letters of Agreement for projects/ ToR for parb:rers * NOTF/ SOCT level plans, minutes of meetings * reports of monitoring groups * progress/ other reports to APOC and other partners. * criteria for NGDO participation [5 2] About relationships betvveen 'international' and 'national/ local' NGDOs + the number, nature, distribution of local NGDOs; their cornmunity base * the nature of contacts between international and local NGDOs + attempts by local NGDOs to become partners * attempts of international NGDOs to empower local ones * selection criteria for NGDOs * correspondence with local NGDOs * constitutions of local NGDOs * minutes of meetings betu,een intemational and localNGDOs JAF6.9 Page 50 LEVEL 4 - Interviews with project/ district level workers project melnbers - singly, or in a group (MoH, NGDO reps) [1.2] About management at country level: o perceptions of the effectiveness of NOTF/ SOCT management plannrng/ monitoring/ evaluation, human resources, ofhce management, communications . perceptrorls of successes and failure . suggestions for improvement fi.41 About financial control [also include project accountants in the discusstort]: . tl'arning they have recetved . theu' overall financial procedure; the role of each persou in tt . the rr budgetrng procedure; its hnk to planning r tlrerr. sources of lunding for oncho. control o the efficiency of disbursemcnts o pcr-ccptrons of successes and problerns f) \.,J 'I; I JAF6.9 Page 5l a solutions to the latter [2.1a] About operationalising CDTI; coveruge and sustaittability [3.1] About coverage uchieved . progress with REMO; use of REMO results . strategies they have been using to implement CDTI (sensitisation, training, supervision etc.) o plans to increase coverage, to full coverageby 2007 . what sustainabilify means; plans to promote it o perceptions on successes, problems . suggested solutions to the latter [2.1b] About vector eliminatiott projects: . their strategy for implementing the projects . perceptions ofprogress; oflikelihood ofsuccess; of . suggested solutions to the latter [2.2a] About the prospects of APOC achieving its aints by 2007 . final objectives for their project, as they see them o the strategies they have been using to increase coverage; progress made . perceptions on the likelihood of achieving objectives; problems foreseen . suggested solutions to the latter [2.2b] About the ahility of projects to functiort after 5 years . perceptions of progress made by the project r prosress towards integrating the project into the State health service r the nahlre and sustainability of the technology being used (for transport, reporting etc.; for vector control) . perceptions on the likelihood that the project will be functioning independently after 5 years of support; problems foreseen . suggested solutions to the latter [2.3J About the adequacy of n o,titoring procedures o the nahrre of all the components of monitoring in the Programme (Ouagadougou to village) . perceptions ofthe quality ofthese procedures . perceptions of the use that is made of the findings (at Ouagadougou and in the countries) [2.4J About the adequacy of the planned impact studies o aspects they would like the studies should address [2.5J About the participatiort of women in CDTI . perceptions of the present role of women in the project: numbers, reasolls for low numbers, examples of gender discrimination . suggestions for an expanded role for women in future [2.6J About operatiortal research and the applicotion of its findings . their perceptions of how the need for operational research is identified - whether they can do it themselves o their perceptions of how rt is commissroned and funded - what APOC's role should be . examples of what has been commissioned to date . the quality and usefulness ofthe research . how the findings have been acted upon . needs for future operational research [2.] About CDTI anel nonndic populatiorts . perceptlons of the problems posed by nontadrc populatrons; the scope of thrs phe nomerlorr -t! JAF6.9 Page 52 a perceptions of opporfunities, achievements and constraints; lessons learnt [3.2J About the intpact of community involvement on levels of coverage o their definition of 'community participation' . perceptions of/ information about the criteria (leadership, involvement of local infrastructure, cornmunity contribution, assessment of progress, community satisfaction) . perceptions of a link between community participation and coverage . perceptions of factors affecting participation [3.3J About progress towards lortg-term sustainability of CDTI . perceptions on/ information about criteria for sustainability: community participation (see 3.2); ability to perform the treatment (skilled, motivated CDDs; ivermectin available); strategic and contextual factors (sound policies; viable management; integration into PHC; wide support; finance available; monitoring) . perceptions on factors likely to affect/ currently affecting sustainability o reported steps to enhance future sustainability . suggestions to enhance sustainability [3.4] About the effect of healtlt educatiort on attitudes and behaviours of local con munities health education activities they have planned and implemented, materrals they produced/ distributed perceptions of awareness of oncho. and CDTI among health workers and villagers perceptions of what works and what not suggestions for improvement perceptions of how well onchocerciasis and CDTI are dealt with in medical and nursing school curricula [4.1J About progress in tntegrating CDTI irtto country lteoltlt systems perceptions ofeffectiveness ofeach level ofthe health system perceptions of the actual participation of each level of the health system in CDTI perceptions of successes and problems suggested solutions to the lafter [4.2a] About APOC and healtlt servtce development: local capacity builtlirtg . previous (pre- 1996) capaciry building done by NGDOs . perceptions of the nature/ amount/ qualrty of training they have had (CDTI/ vector controV managernent) . perceptions of the nature/ amount/ qualiry of training they have given (CDTI/ vector controV management) . perceptions ofsuccesses and problems . suggested solutions to the latter [4.2b] Abont APOC and overall health sector developnrent . perceptions of skills learnt from APOC training and activities, that have a rvider application in their work . perceptrons of skills they impart in 'APOC' training, that should have a wider application in the work of fiarnees [4.3] About impact on healtlt service quality/ PHC for the poorest . perceptions/ evidence of neu'/ tmproved processes/ structures at drfferer.rt levels (especrally PHC), resultrng from APOC's work . perceptions/ evrdence that APOC has rmproved health care . perceptrons of problems that CDTI work causes to workers at different levels r\\? aJAF6.9 Page 53 suggestions for how APOC can enhance health service quality/ stimulate health service development [4.4J About integrating the control of other diseases into APOC operatiotts ' opinion about whether APOC should widen its mandate, to include other donated drugs/ other activities/ even a basic PHC programme o ideas about which additions would be usefuV feasible ' ideas about how to integrate this with NGDO and health service policy and activities [5.1J About contributions of the diffirent partners at country level . who all the partners are o how they see their own contribution (technical, financial, operational; also ivemnectin distribution) - intended and achral o how they see the contributions ofthe other partners (intended and achral) . perceived advantages/ disadvantages ofworking with other partners ' perceptions of how well partnerships function at each level: successes, problems . suggested solutions to the latter [5.2J About relationships betweett 'international' and 'national/ local' NGDOs perceptions of the nature of local NGDOs; of applications they have made efforts made to empower local NGDOs successes, problems suggestions for the future LEVEL 4 - lnterviews with committee members of local NGDos [5.2J About relationsltips betweert 'international' and 'national/ local' NGDos therr objectives, their modus operandi, their membership and community base why they wrsh to be involved in APOC help they have received from international NGDOs successes, problems suggestions for the fuhrre LEVEL 4 - Observations at project/ district level offices/ services fi.2 - local management capacity; 1.4 -financial syste,nJ In the offices: information/ document retrieval system system for maintenance/ repairs system for postage, telephone, fax system for arranging transport the whole financial system [2.1a - CDTIJ In the offices: . nature/ efficiency of CDTI and other reporting systems o the operational capacity: particularly transport for supervisron, the dmg supply system o l.runlan resources available: suitability [2.1b - vector eliminatiortJ On the terrain: the nature ofthe terrain; the local infrastructure capacity of local field workers preparatlons undertaken to date activities on site: inspectton of breeding sites, method of ground larvrcidrng, entomologrcal surveillance a a a a a JAF6.9 Page 54 [2.2a] About the prospects of APOC achieving its aims by 2007 . strength of oncho. control structures at project level (MoH and NGDO) D , I ( r, t a