Assessment of the sustainability of the Abia state CDTI project, Nigeria, June 2003: report, volume 2 - instruments

Sustainability of the Abia State CDTI project, Nigeria. June 2003 World Health Organization African Programme for Onchocerciasis Control Assessment of the REPORT VOLUME 2 - INSTRUMENTS Ekanem Ikpi Braide (Team Leader) Charles Franzen Yisa A. Saka Sunday Isiyaku Obinna Onwujekwa RECU I 5 SEP. 2003 A?ocrDtt lnstrument 1: national/ State level NOTE: . This instrument is for national level, which coordinates programme imptementation in the country. lt could also be used (modified slighfly if necessary) for another level of support other than the main implementation level {districU LGA}. An example of this level is the'State' in Nigeria. p,reviseasnecessary,andoverseeimplementationofCDTl Policy. This level also provides support to the level below it: . Providing targeted training, HSAM and monitoring/ supervision. - rzen, Obinna Onwujekwe and Sunday lsiyaku Abbreviatio ns/ acronyms CDD CDTI FLHF HSAM NGDO NOTF community directed distributor community directed treatment with lvermectin first line health facility health education, sensitisation, advocacy, mobilisation - i.e. activities that are aimed at getting all the key players to participate wholeheartedly in the programme non-govemmental development organisation national onchocerciasis task force t .t Cfrecf wnether ttrere is a year plan for onchocerciasis control, appearing as part of an overall written plan for the health service at this level. has bec:ame integrated into the health service, management is accepting ownerchip of the programme - both good for sustainability. Characteristics of the indicator Sources of information a. Onchocerciasis control should be integrated into the overall written year plan of the health service at this level. Note that this plan is usually not very detailed. . Examination of: * Written plans: yearly, quarterly, monthly etc. * Minutes of planning meetings. . lnteruiew wilh senior heatth senrice staff at this level. Describe the oresent situation: There is an existing plan for onchocerciasis control activities and it is part of the overall annual plan of the department of public Health and Primary Health Carc and also of the State Ministry of Health services. There are also monthly plans for core CDTI aclivities at the project office. All programmes submit plans, which are consolidated into an annual plan for the MoH. There is no direct evidence thai plinning is integrated, as this is not the system within the Ministry. All programmes submit plans, which are consolidated into an annual plan. The planning process is not integrated, as this is not the policy within the Ministry. lf Olanninq and implementation of CDTI is not part of the overall vear plan: . Whv is this? Not Applicable . Which steps are beinq taken to improve the situation? Not Applicable Sustainability evaluation instrument no.1 - national level- March 2003 version Analysis . When writing the repft you have to summarise: * The evidence about how well fhis indicator is being achieved. * Reasons for poor pefiormance (if any). sfeps being taken to improve it, and how f/,is is likely to affect sustainability. . Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slighily I Not at ail | ruot appticable 1.2 Gheck whether there is a year plan containing details of all activities needed for CDTI at this level. This iMicator assesses whetherthe poject is functioning effectively, and whether management access ownerchip of the programme - both good for sustainability. Gharacteristics of the indicator Sources of information a. This more detailed plan should make provision for all key elements of onchocerciasis control: Mecitizan supply; targeted training ; ta rgeted HSAM; targeted monitoring/ supervision. (Note that in the case of a country, which contains many projects, the NOTF plan will be less detaited). The plan varies from yearto year, showing that it is targeted to the specific needs of each year. Examination of written plans: yearly, quarterly, monthly etc. lnterview with senior health service and project staff at this level. Sustainability evaluation instrument no.1 - national level - March 2003 version Find Describe the plan for the oresent vear: There is a written plan for CDTI activities at this level. The plan provides key elements of CDTI, is not a re-write of previous years plan and is targeted at specific activities. This plan exists within the overall Ministry of Health plan. Though a plan exists with a bulk budget amount for all activities, it does not include activities that would need to be funded by the various partners. lt does not specify the costs to be bome by each partner neither does it indicates activities that would be funded post APOC. A supplementary plan for 2003 was produced just before the evaluation team anived. Describe the plan for the orevious vear: There was a similar plan for 2002, which varied with respect to line item and bulk budget amount. lf the olan is incomolete. or simplv a re-write of previous plans: . WhY is this? Not Applicable . Which steps are beino taken to imorove the situation? Not Applicable Analysis t When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overall judgement: is this indicator for sustainability being achieved? FullylHighlylModeratelylstighty Check whether all partnens (govemme overall planning process. indicator assesses the gogrunme6 this is good for sustainabitity. - if each partner is about its role, Gharacteristics of the indacator- Sources of information o. /-\1l patruets snout(I ContnOute tO the foutine planning of a projeci.b. Partnerc should be clear about their own roles, and those of the other parlners. ' Examination of: * Plans: yearly, quarterly, monthly etc. * Minutes of NOTF meetings. . lnterview with senior staff at this level (p@ect, govemment, relevant UN agencies, NGDOs etc.) The team was informed that the main partners - Ministry of Health (oncho coordinator, Assistant chief planningofficer of the Ministry and state DPHIi'HC) and the NGbo fpioj"cfAdministrator) parl"iprte in the ptanning.NocP/NGDo review meetings are also heid where tne panl aii presented and adopted. There were however nominutes of these meetings. These plans are consolidated into the MoH ptans and sent to the state planning commission for approval. Therewas no evidence that participatory planning was done. This is not the practice in the MOH The Permanent secretary whg is strongly in.support of integration is planning to have a meeting with programmeofficers and supporting agencies to disluss intdjration. When writing th * The evidence about how weil this indicator is being achieved. - ::::**lJpoor performance (if anv), steps uein[ taren to i*pror" it, and how this is likety to affecr . , Yorr or"t lljudg", 1.4 Check whether specific nfunding is withdrawn. 7l,s assesses ownerchip of it and can mobilise the resources rf needs. management has begun to take Characteristics of the Members of NOTF h programrne sustainability. Thi.s planning should include: identifying resourcegaps, strategies to cut expenditure, and strategies to find oepenoante sourcesof resorrrc,es. There should be written evidence that such pranning has taken prace.There should be evidence that the ptans are being iuccessrutiylmpemented. Examination of tfre written susiainability plans. lnterviews with senior staff at this level (government, NGDO, NOTF etc.). sustainability evaluation instrument no.1 - national level- March 2003 version Describe the present situation The state oncho coordinator has produced a plan for sustainability post Apoc that coverc the period of october2003 - January 2004. This is yet to be agreed and finaliseo witn irri: MoH and the NGDo. There is no ptan for theperiod beyond. . Why is this? The Permanent secretary explained that the MoH is waiting for LGA chairmen to be appointed beforesustainability plans are developed. The state oncho coordinator is also waiting for the planning meeting. :Thereisaplantoho1dmeetingswffieyareappointed. When writing the reportffi * The evidence about how weil this indicator is being achieved.* Reasons for poor performance (if any), steps bein[ taken to i*proru it, and how this is likely to affect . Your overalljrdn" Check whether staff at this level ii assesses whether programme is functioning of the indicator Staff members at this tev level immediately berow them. Staff shoud nof supervise the FLHF.or community levels. ,Spot checks' rnay however be donefrom time to time. Staff members at this level should have empowered staff members at the level below them to supervise activities at their own level, as well as levels further down. * Supervisory checklists, plans and reports. * Visitor's books at allthe levels below this one. * Trip authorisations. . lnterviews with: * Staff at this level. 1 Staff at levels below this one. There are 5 SOCT members and each one is responsible for monitoring and supervision in 1 or 2 LGAs. A monthlyworkplan is prepared in which monitoring and supervision is inctuoeo ano canied out by the staff. Monitoring andsupervisory checklist are used and reports written. Trrougn wra;;pil;;;;;1;;i", there were no reports offollow up activities. sustainability evaluation instrument no.1 - national level - t t"."nloog ,*oi* Analysis I When writing the report you have to summarise: - * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve sustainability. it, and how this is likely to affect ' Youroverall judgement: isthis indicatorfors@Fully I xishty lffoderaCi, Slightly I Not at ail | Not appticable 2.2 Check whether monitoring/ supervision is being planned anC carriio out in an efficiena and indicator assesses whether the gogrartme is Characteristics of the indicator Sources of information a. one routine supervision visit per year m@ (as a separate entity, or as a district).b. Supervision visits for cDTl should be integrated where appropriate with supervision of other programmes. c. Resources for supervision (human, transport etc.) should be efficiently used: * Using as few staff members as possible. * Planning trips to cut down on distance travelled. * Not spending unnecessarily many nights out etc.d. Supervision visits should be thorough, using a checklist. . Examination of: * Supervisory checklists, plans and reports. * Visitor's books at the level immediately below this one. . lnterviews with: * Staff at this tevel. * Staff at the level. immecliately below this one. The SOCT monitor and supervise distribution based on planned activities. This is mainly at the LGA levelwith spot checks to communities. There is a minimum health package in the State and sometimei all programme officers work together- For example recently in Obingwa (UNICEF-assisted LGA) alt programme officers visited as a team and carry out separate interventions. Occasionally there are integrated monitoring and supervision, but this is usually tied to the Oncho programme schedule because of the project vehicle. Other programme officers oflen scheduled their monitoring tips to coincide Oncho activities. Describe the situation the previous vear: Same as above Not Applicable . Which steps are beinq taken to imorove the situation? Not Applicable When writing the report you nav- to-uffiarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability evaluation instrurnent no_1 - national level- March 2003 version . Examine the trend . ,Your or"t"ll 2.3 Check whetherthe* indicated by the monitoring systbm. management a. As soon as proUn supervision visits, or from coverage data (i.e. areas with row - :ovgrage).the appropriate manager should deal with them.o. uucn proDtems should usually be passed on to the appropriate managers at the next iever berow to dear with, with the necessary support _ thus empowering thesepersons. c. Successes should be noted and reported, and appropriatefeedback given.d. There should be evidence of aclion taken based on recommendations in the reports of previous monitoring exercises. . Examinationofttre@ ] l].ans: yearty, quarterly, monthly etc.* Minutes of staff/ planning meetings. * Reports of previous monitoring exercises. * Letters of commendation. * Letters with information and feedback . lnterviews with: * Staff at this level. * Staff at the levels below this one. . Whuj writin ] I[:r:9gl* about how wefi this indicator is being achieved.. ffi:l*t tlJ;:or performance (if anv), steps bein[ taren toimprove it, and how this is likety to affect Describe the presenGituation The system in place for.solving problems is the normal administrative channel. lf there is a problem at the firstinstance the socr in-charge or firat r-cn ii responsible for ensuring tne proolem is iesorveo. lf the problem is not t?S'r'rTJlis reported to the state oncho cooniinator ano wneie'she cannot handte it, she passes * on rhe where successes are recoded, they are noted and the officer or team is commended. For exampte in 2001 H'[YA':"ffif ,f,ff.Xfi.in therapiutic coverase and the coordinatorwas nominateo to attend an NorF/Apoc . Whv is this? Not Appticable Not Appticabte . ,Yorr or"rattjrOg sustainability evaluation instrument no.1 - national level- March 2003 version 3.1 Check whether sufficien stlstem at this level, in good time. This indicator asses becoming more integrated into the fov6mment sysfem. The Mectizan su i::r".**t}:p ,!e.t|r"-.11"t9* routinety.useo ror t[e supprv oi o:tn", drugs.The system should be effecrive, uncomprieated and eniciini.This system should use dependable, sustainable resources for its operation. ltis.desirabte that the government should supply these resou*r.The system should supply sufficient Mectizan-for the needs or att fire projectsconcemed, ln good lime. . Examination of all Mectizan ordering and slock control documentation at this level. . lnterviews with staff at this level (govemment, once the state is notified of the availability of Mectizan@ it is collected by the state from the NGDg office in owerriand kept in the oncho coordinator's offid from where the LeAs are quickry informed to come and collect theirallocations' Though this is considered as the project system antt is seen .. orp"norite tor quick delivery to theLGAs, it is not within the drug supply system of the MoH. Same as above The medicar store is not safe that is why the drugs are not stored there. :.None yet Whenwritingtneffi + The evidence about how weil this indicator is being achieved. - ffj:ilt^tl#oor performance (if any), steps oein! taten to improve ir, and how this is tikely to affecr . What fras Ue . ,Yorr orarrlljuOgt - national level- March 2003 versionSustai nability evaluation insttmntao. 1 4.1 Check whether staff members at this level are being used appropriately as trainers. This indicator assesses whether the ffogramme is funclioning Characteristics of the indicator Sources of information a. Staff at this tevel should routinely only train staff at the level immediately below it. b. Staff membes at this level should have empowered the level immediately below them to train lower levels. Examination of training materials, plans/ programmes, reports: * At this level. * At the levels below this one. lnterviews with: * Staff at this level (the trainers). * Staff at the very next level below (the trainees). * Staff at the districU LGA level. * Staff at the FLHF level. The SOCT train the LOCTs on CDTI and they in tum the CDDs. The SOCT members are assigned topics during training. Training materials like reporting forms, flipcharts, posters and brochures were seen. Describe the situation the vear before: ln 2002 the situation was the same as above, but new LOCT members transfened from non-endemic areas were trained in addition to existing members. lf staff members are not beinq used efficientlv as trainers: . Whv is this? Not Applicable . Which steps are beinq taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Examine the trend in the way in which staff are being used as trainers: ' Your overall judgement: is this indicator for sustainability being achieved? FullylHrghtvlModeratelylslignit 4.2 Check whether training is being planned and carried out in an efficient and integrated manner assesses whether Wrymfire is fmctioning efficienily an integrated manner. Characteristics of the indicator Sources of information a- There should be an objective need for each episode of training. This means there should be evidence that slaff to be trained lack knowledge and skills to perform the job, and the training should then focus on this deficiency only. Repeat traininq of alreadv skilled . Examination of training materials, planV prograrnrnes, reports: * At this level. * At the levels below this one. sustainability evaluation instrument no.1 - national level- March 2003 version b. c. staff should not happen. lf circumstances permit training for CDTI should be integrated with othertraining, e.g. in in-service training programmes. Resources for training (human, transport etc.) should be efficienily used: * Using as few staff members as possible. lnterviews with: * Staff at this level (the trainers). * Staff at the very next level below (the trainees). * Staff at the districU LGA level. * Staff at the FLHF level. * Using as little time as possible (without sacrificing quality) * Choosing the most cost-effective site etc. Describe the oresent situation: Staff have acquired and adequate training skills. Training is sometimes targeted at specific aspects of the CDTI. There is minimal integration of CDTI with other trainings. Describe the situation the vear before: Training of LOCTs covered the whole CDTI processes because of the LOCTs were transfeped from other non- endemic areas. lf tralnlno.ir not b"ino *rri"d ort in ,n "ffiri"nffi. Whv is this? There is no clear policy on integrated training. . Which steps are being taken to improve the situation? None Analysis . When writing the report you have to summar * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Examine the trend in training activities - . Your overall judgement: is this indicator for iu@ FullyIHighlylrvrooei 4.3Checkwhetherstaffatthislevelisplanninganocarry Thi s i n dic ator assesses the yogramme is are taking ownerchip of the programme. and effectively, whether managers Characteristics of the indicatoi Sources of information Staff members identify situations information abouu cornmitment to cDTl, and undertake ac'tivities to inform and pesuade these persons. HSAM activities are properly planned. They are only carried out where there is an objective need for them, and not as a matter of routine. such activities should only be camed out at the national level, and at times at the level immediately below (but only when staff at that level asks for help). There is evidence that these HSAM activities have been effective and have led to action. b. d. . Examination of HSAM plans/ programmes and reports. . lnterviews with: * Staff (programme and management) at this level. * Civil authorities at this level. * Staff and civil authorities at the next leveldown. sustainability evaluation instrument no.1 - national level- March 2003 version 10 Fi Describe the present situation fin relation to efficiencv and outcome): Briefings are canied out from time to time for decision makers and other staff in the Ministry to advocate, health educate, sensitize and mobilise them. For example when the cunent Permanent Secretary was appointed, HSAM was canied out and as a result she visited LGAs for advocacy and spot checks. ln other instances when planning division require clarification on issues HSAM is providecl. Descilbe the situation the vear before (in relation to efficiencv and outcomeL Advocacy visits were to LGA Chairmen and council members. The concept of CDTI is explained to them and support for CDTI activities solicited. lf HSAM activities are not beinq canied out efficiently: . Whv is this? Not Applicable . Which steps are beinq taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Examine the trend in HSAM activities - is it becoming more efficient and effective? . Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I tlot applicable sffi4s Describe the present situation: There are no indications that support programmes are planned and executed in an integrated manner. There are however instances where SOCU memberc are involved in other programmes. For example, one member of the SOCU team is a health educator for CDTI and combines this with other diseases. The Data Manager also data for other 5.1 Check whether the various programme support activities are being planned and carried out in an efficiently and in an integrated manner. Gharacteristics of the indicator Sources of information a. Staff combines two or more tasks on a single trip: x Monitoring / supervision for CDTI (and other projects, if staff is responsible for them as well). * Training for CDTI (and other p0ects, if staff is responsible for them as well). * HSAM. * Fetching records. * Deliverino Mectizan. . Examination of documents: trip authorisations, log books, trip reports etc. . lnterviews with: * Staff from this level (managers, administrators, drivers etc.). * Staff from the next level below. Sustainability evaluation instrument no.1 - national level- March 2003 version Describe the situation the vear before: Same as above lf inteoration between support activities is poor: . Whv is this? There is no policy on integration . Which steos are beino taken to imorove the situation? Permanent Secretary intends to hold meetings of all programme managers to plan for integration Analysis r When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and sustainability. how this is likely to affect . Examine the trend in the integration of aAiv . Your overalljudgement: is this inOicato Fully l Highly I F.:."::":".::.fn{.lqwgt{itr:ryqqlf4f F:_fi fi ffi E*rt'."::: 6.1Checkwhetherappropriateamountsarebudgetedforp Characteristics of the indicator - Sources of information b. d. The costs for each onchocerciasis control a should be clearly spelt out in a budget. There is evidence of a const reduction/ containment slrategy (e.g. targeted training, HMAS and monitoring/ supervision; training conduCted it thsnext level below etc). Project managers should have a clear estimate of the funds that will be available to them for onchocerciasis control in the coming year, and should be able to justify this belief. The total amount budgeted for in the year plan should fall within this estimated income. . Examination of the budget documents. . lnterviews with the health service managers at this level. F The budget and estimated income: . For this year: The budgeting is canied out following the govemment's procedure. There is a budget line for onchocerciasis control activities, which is listed under subhead 12 in the recunent budget of the MoH. The budget estimate for 2003 is Nl00, 000.00. There are olher CDTI activities, which are covered under other budget lines like travet,tllling stationery etc. This is to ensure that the project has other budget lines from which funds can be sourced forCDTlactivities. . For the previous year: No amount was budgeted in 2002 for Onchocerciasis control activities, the State Govemment released N500, 5OO from the 2001 budget. lf budoetino has been inapprooriate: . Whv is this? ln 2002 there was no budget allocation for CDTI. Other sub heads within the MOH budget covered CDTI activities like training, travel, and stationery. This also applies to the 2003 budget. . Which steps are beino taken to imorove the situation? Advocacy will be intensified sustainability evaluation instrument no.1 - national level- March 2003 version l2 Analysas . When writing the report you have to summari * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. ' Examine the trend in the budgeteo amouni@ ' Your overalljudgement: is this incticator FullyIHightylrullo Checkwhetherthegovernmentisbudgetinganddisbursingiu controlyearly, and in good time. This iMicator assesses whether the pr , ownerchip of the gagramme and can mobilise the resourC* it needs. and wfietherthe govemment is acceptinig Gharacteristics of the indicator Sources of information a. The relative budgetary contributions of tfre to onchocerciasis control should be clearly spelt out. b. The amount that the govemment has budgeted in one or more specific onchocerciasis control budget lines (e.9. cunent ancl capital) should be increasing yearly, as a proportion of total expenses. By the end of year 5 o.f APoc funding the bulk of onchocerciasis control eipenses at this level should be met from govemment funds; by the end of year 3 at least half of it. c. The amounts actually disbursed from such budget lines shoutd be increasing yearly, as a proportion of total expenses. (Note that actual disbursement is more important than budgeting, and is a real sign ofpolitical commitment). . Examination of: * Budget documents (govemment, NGDO etc.) * Records of disbursement and expenditure (ledgers, oderc, approvals for expenditure etc.) . lnterviews with managers at this level (govemment, NGDO etc.). Budgetary contributions of the govemment and other partners to Onchocerciasis control are not clearly spelt out.The amount budgeted by govemment for CDTI has not increased yearly, as a proportion of total expenses. Closeto the end of Year 5 of Apoc funding the bulk of expenses on cDil is met ny nebc. The budget and disbursements from Govemment are as foilows . For this year: (2003) The sum of Nl.5 million was requesled fnom the state govemment but there has been no approval yet. Howeverthe MoH has so far expended N93, 930 on spares, fueJ, and vehicle maintenance in 2003. . For the previous year: (2002) The sum of N76, 200 was used for stationery and entertainment during meeting and briefing. ApOC . Forthe year before that: (2001) The sum of N550, 500 was requested for the project in 2001. N370, 100 was used for coilection of capital equipments from WHO Lagos, allowances, repairs, fuel and other miscellaneous expenses. From NGDO records contributions on APOC-WHO sustainability programme is given as follows for lmo Abiaproject YEAR APOC NGDO STATES1 $139,274.44 *$112,347.002 $87,940.53 $108,069.25 $5,244.083 $8s,032.0S $63,531.00 $1,093.s04 $69,779.s8 $197,202.26 $4,1s6.795 $26,275.63 $70,370.48 TOTAL $408,000.00 $550,000.00 $10,500.0042o/o 56% 1.Zo/o * inclusive of capital equipment COMMUNITIES $2769.96.00 0.3olo Sustainabilityevaluationinstrumentno.1-nationalffi 13 lf the government proportion of expenditure is not increasing proportionately: ' Why is this? This is due to dwindling allocations from the govemment. E.g. in 2002 there were no budget allocations for CDTI. Other sub heads within the MoH budget covered activities like training, travel, etc. . Which steps are being taken to improve the situation? A supplementary budget has been prepared and submitted to the govemment but there has not been any response Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Examine the trend in govemment budgeting and disbursements: . Your overall judgement: is this indicator for sustainability being achieved? FuilylHishtylModeraretylstishily[ffi 6.3 Check whether in case of a deficit between estimated costs and the amount provided by the provision is being made to meet it. fhis commitment to ownerch i p. assesses whether the goventment is to mobilise the resources rf needg as well as rfs Characteristics of the indicator Sources of information a. Project management at this level should be aware of the shortfall, if one exists, and of its size.b. Project management should have specific and realistic plans to bridge the shortfall. c. lf it is planned that non-governrnent souroes of funding are to be used after APOC funding ends, written commitment for this should have been obtained at the highest level in these donor organisations. Projects in Year 3 of APOC funding should also be well on the way to achieving such commitment. . lnspection of: * The budget documents. (Govemment, NGDO etc.). * Records of expenditure (ledgers, orders, approvals for expenditure etc.). * Letters of agreement. . lnterviews with managers at this level (govemment, NGDO etc.). There are no plans to source for any dependable funding post APOC. lt is however believed that budget shortfalls or deficit could be bridged from other sub heads within the state Ministry of Health budget and the 50/o the MOH keeps from all revenue it generates. Describe the situation the orevious vear: Deficit bridged from other subheads in Ministry budget lf the shortfall cannot be met: ' Whv is this? There has been no plans for the post APOC period . Which steps are beino taken to imnrove the situation? The DPH/PHC is considering approaching the supporting NGDO to support some core activities Sustainability evaluation inslrument no.1 - national level- March 2003 version t4 Analvsis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Examine the trend in shortfall and how it is to be supplemented: . Your overall judgement: is this indicator for sustainability being achieved? Futty I Highly I Moderately I Slightly I Not at all I Not applicable 6.4 Check whether funds disbursed for onchocerciasis control from the budget at tbis level are Characteristics of the indicator Sources of information The budget holder should be using a control system with the following elements: * Approval of each item of expenditure. * Allocation of expenditure against specific budget headings. * Regular calculation of residual amounts under budget headings. All the funds released yearly should be spent as budgeted. a. b. . lnspection of: * The budget documents (government, NGDO etc.). * Financial control records (ledgers, orders, approvals for expenditure etc.). , lnterviews with managers at this level (qovemment. NGDO etc.). Describe the oresent situation: . Approval of expenditure: The State Oncho coordinator requests for funds through the DPH/PHC. Funds from the State for Onchocerciasis control activities are approved by the Permanent Secretary and go through the accounting system in the ministry before it is released. However there were no clear guidelines as to how this is done because there is no project accountant for the State. ln February an APOC account was opened in the State and funds transfened. The APOC financial is due to end in September 2003. . Allocation of expenditure: Funds are allocated based on budget sub heads and is limited to available funds . Regular insight into budget line balances: Expenditures made with reference balances lf the funds are not beinq well manaqed: . Whv is this? Not Applicable . Which steps are beino taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overall judgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not applicable Sustainability evaluation instrument no.1 - national level- March 2003 version 7.1 Check if adequate and appropriate transport and other material resources are available for necessary CDTI activities at this level. l5 Thi s indicator assesses renources it needs. ffogramme is functioning effectively, and whether it is able to mobilise the Characteristics of the indicator Sources of information a. There are adequate numbers of functional vehicles available for necessary CDTI activities.b. The vehicles are appropriate for the purpose they are intended to fulfil - tough but not luxurious. c. There is sufficient office equipment available, in working order: computers, printers, photocopiers - also stationery and materials for training and HSAM.d. The running costs for these vehicles and equipment are met from dependable. sustainable sources. . lnspection of: * Each vehicle in the pool, each piece of equipment: its source; its functional status. * Training materials and stationery stocks. . lnterviews with managers at this level (govemment, NGDO s.). Describe the availability/ suitabilitv/ functionality of the present vehicles. equioment and materials. considerino tfie work still to be done in the coming 5-10 vears: Available CDTI vehicle will not last for the next 10 years. lt will require replacement in the next two yearc. Computer and accessories will need to be replaced in the next two years. Adequacy for CDTI tasks* Adequacy for CDTI tasks** * APOC, MoH, NGDO, other * /s rf working? ls there enough ot it for the job? ls it suitable for the iob? lf transport. equipment and materials are inadequate and/or funded from sources which are not dependable . Whv is this? . Which steps are beino taken to imorove the situation? Analysis I When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overall judgement: is this indicator for sustainability being achieved? FullylHighlylModeratelylS sustainability evaluation instrument no.1 - national level- March 2003 version 7.2 Check if transport and other material resources in use at this level are adequately and appropriatelv maintained. This indicator assesses whether the programme ii@ Characteristics of the indicator Sources of information b. There is a routine maintenance schedule for each vehicte, wtrictr is adhered to and recorded. This incluctes weekly driver maintenance, scheduled garage servicing, and replacement of wom tyres. Equipment such as photocopiers and generators is regularly maintained according to a schedule, and this is recorded. staff members have ways of coping when vehicles break down or are not available, so that CDTI activities are not disrupted. The costs for vehicte and equipment maintenance and repair are met by from dependable/ sustainable sources. Repairs to vehicles and eguipment are rapidly and efficienflv done. d. e. . lnspection of: * Vehicle and equipment maintenance schedules. * Vehicle and equipment maintenance records. ' lnterviews with managers at this level(govemment, NGDO etc.), drivers Routine maintenance is canied out for the vehicle and other equipment, but there is no record of maintenance (no maintenance book). The MOH and NGDO provide vehicle when needed to ensure that CDTI activities are not disrupted. The MOH has been providing funds for vehicle maintenance Analysis I When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overall judgement: is this indicator for sust@ Fully I Hishly I Moderatety I 7.3 Check if the transpgrt available at this level is appropriatety managed and used. Gharacteristics of the indicator Sources of information b. Transport is used at this level, and to undertake sup@ next level. lt should not be used for cDTl implementation activities at lower levels. The use of transport is properly controlled: * Trips made for cDTl purposes should be properly authorised in writing by the relevant official. * Each trip undertaken for CDTI purposes should be recoded in a logbook. * Trip authorities and log book entries and regular.ly reconciled, and action taken if there are discrepancies. . lnspection of vehicle control documents: * Copies of trip authorities (also noting destination and purpose). * Log books. . lnterviews with managers at this level (govemment, NGDO etc.). sustainability evaluation instrument no.1 - national level- March 2003 version t7 Describe the present situation: Transport is adequately utilised for CDTI activities at this level. Authorizations for the use of the vehicle are however verbal based on approved workplan. There is no abuse of vehicle usage lf the transoort is not being well managed: . Whv is this? Not Appticabte . Which steps are beino taken to imorove the situation? Not Applicable Analysis I When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), sleps being taken to improve it, and how this is likely to affect sustainabiliU. ' Your overalljudgement: is this indicator for sustainability being achieveOZ FullylHighlylModeratelyls 7 .4 Check if there are appropriate and rcalistic plans for the replacement of transport and other materia resources, when APOC support comes to an end. This indicator assesses whetherthe Wogrcmme manage@ to find resources for it. the gogramme, and are able Gharacteristics of the indicator Sources of information Management should know that replacements will be needed berore tne eno of the programme, and have specific, realistic plans to meet the need at that time. It should be planned that the govemment will: * Provide replacements for vehicles and equipmeni. * Maintain existing vehicles and equipment. * Provide stationery and materials fortraining and HSAM. lf it is planned that replacement will be from non-govemment sources, written commitment for this should have been obtained at the highest level in these donor organisations (end of Year 5), or negotiations should have started (end of Year 3). . lnspection of letters of agreement. . lnterviews with: * Programme managerc at this level (govemment, NGDO etc.). * High-ranking Ministry officials and other decision makers at this level. Vehicle is functional and will serve the proiect only for the next 1-5 years. The MoH is committed to replacing the vehicle and equipment when necessary. Replacement from other sources have not been explored lf the plans for replacino vehicles. equioment and materials are unsatisfactorv: . Whv is this? The issue of APOC pulling out afier five years not seriousty considered and therefore no plan made for reptacing capital equipment. . Which steps are being taken to improve the situation? None yet Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, sustainability. and how this is likely to affect ' Your overall judgement: is this indicator for sustainability neingrechieved 2 Fully I Hishly I Moderately I sustainability evaluation inslrument no.1 - national level- March 2003 version l8 8.1 Check whether staff at this level is skilled, knowledgeable and committed, regarding the implementation of GDTI in its area of operation. This indicator assesses whether the gogramme has been able to develop sufficient resources for itself. Characteristics of the indicator Sources of information a. The number of staff members in the govemment health service at this level should be appropriate to the task in hand: not too many or too few.b. Team members should have enough knowledge and skill to undertake all the key CDTI activities themselves, without help: * Planning * Data management * Report writing * Computer skills * Training and HSAM * Mectizan ordering/ distribrution * Monitoring/supervision c. There should be evidence that the team is committed to the success of the programme (from the evidence of the partners, as well as workers at the next level below: from written reoorts and timetables). . lnspection of: Staff files. * Training reports and timetables. . lnterviews with * Managers and other staff at this level (government, NGDO etc.). * Staff at the next level below. lf the staff at this level lack skills and commitment: . Whv is this? Not Applicable . Which steps are being taken to improve the situation? Not Applicable Analysis When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and sustainabilitv. I how this is likely to affect . Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Stigntty I Not at all I Not appticable . Number of persors wJto show that they are committed to thier work, and perform it wetl : No. of persons quatifred in this area No. of persons competent enough to 8.2 Check whether staff at this level is stable, and whether provision is made for passing on CDTI skills when a trained person moves away. This indicator assesses whether the gogramme has been are able to maintain r'fs resources. Characteristics of the indicator Sources of information Staff at this level should remain in one post for at least five years. There should be immediate orientation (in CDTI) of new, unskilled project staff members. a. b. . lnspec-tion of staff files. . The table in 8.1. . lnterviews with managers and other staff at this level. Sustainability evaluation instrument no.1 - national level- March 2003 version t9 Describe the oresent situation: Members of Staff are stable and would remain in their posts. Some of them have been in the programme for 13 years. There plans to train more people on CDTI in case the cunent members of staff are moved. Describe the situation two vears aqo: Same as above Describe the situation when APOC fundinq started beino oiven: There were 5 staff members then and they have demonstrated commitment to the project because of the motivation and job satisfac{ion. lf the staff is not stable. and new staff is not beinq trained: . WhY is this? Not Applicable . Which steps are beinq taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . What is the trend in the number and quality of staff? . Your overall judgement: is this indicator for suslainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not applicable 9.1 Check whether all projects in the country (or districts in the project) have a satisfactory therapeutic assesses whether the programme is effec-tive - if the rates are poor the project is clearly struggling, Gharacteristics of the indicator Sources of information a. Alt projects in the country (or districts/ LGAs in a project) should have a therapeutic coverage rate of 650/o or higher (the denominator being the total population). b. These rates should be stable or increasino. . lnspection of distribution reports and statistics at project level, forthe past 3 years. . lnterviews with staff at: * This level. * The next levelbelow. ind The theraoeutic coveraqe situation in the proiects (or districts/ LGAs): . At the last distribution: 2003 distribution still going on. Final reports were not available . The vear before: Sustainability evaluation instrument no.1 - national level- March 2003 version Sustainability evaluation instrument no.1 - national level- March 2003 version 20 ln 2002 therapeutic coverage was 640lo - 81o/o. However it was discovered that therapeutic coverage in some communities was calculated using eligible population as against total population. . The vear before that: ln 2001 coverage was 610lo - Elo/o lf the therapeutic coveraoe rates are ooor: . Whv is this? Not Applicable . Which steps are being taken to improve the situation? (lf such steps are already being taken that is good for sustainability) Not Applicable Analysis . When writing the report you have to summarise the reasons for poor performance (if any); steps being taken to improve it: and how this is likelv to affect sustainabilitv. . What is the trend in therapeutic coverage? . Your overall iudoement: is this indicator of sustainabilitv beino achieved? Fully (100o/o of projects have a therapeutic coverage rate 2650/o - stable or increasino) Highly (90- 99o/o of proiects) Moderately (80-89o/o of oroiects) Slightly (/0- 79o/o of oroiects) Negligibly (<70o/o of oroiects) lnstrument 2: districu LGA level NOTE: This instrument evaluates the level, which actually takes responsibility for implementing CDTI in its area of operations. lt is that level where health services are planned and provided. We are going to refer to it as the districu LGA level. The level below this one is the one where the health centres/ clinics/ dispensaries are located. We are going to refer to this level as the 'FLHF' (front line health facility) level. Geographical name of this districu LGA: lkwuano, Ukwa East and Umunneochi LGAs Project: Abia State CDTI Proiect, Nigeria ffikpiBraide,YisaSaka,charlesFranzen,obinnaonwujekweandSundaylsiyaku Date: 17 -20 June 2003 Abbreviations/ acronyms CDD CDTI FLHF HSAM NGDO *,'.".."."-:.'.. community directed distributor community directed treatment with ivermectin first line health facility health education, sensitisation, advocacy, mobilisation - i.e. activities that are aimed at getting all the key playerc to participate wholeheartedly in the programme non-govemmental development organisation hsq&lqffrqffig{#ffiI;::: ' ttre miin function of this level is to tafte responsibility for the implementation of CDTI in its area of operation. . However its function is still largely one of support of the FLHF level: * Providing targeted training, HSAM and monitoringl supervision. * Arranging for an adequate supply of Mectizan. . The FLHF level is the one that is for workinq with the CDDs in the communities. 1.1 Check whether the year plan for CDTI appears as part of an overall written plan for the activities of the districU LGA. management is beginningto accept ownerchip of the gogramme - both good for sustainability. Th i s i nd icator assesses the programme has become integnted into the heafth service, whether Gharacteristics of the indicator Sources of information a. CDT!should be integrated into the overallwritten plan (showing that staff at this level consider CDTI to be part of their yearly routine, like any other programme). b. The plan should make provision for allkey activities: Mectizan supply; targeted training; targeted HSAM, taqeted monitoring/ supervision. c. Year plans should be drawn up in a participatory way. d. Year plans must take into account community requirements for the timino of distribution. . lnspection of: * The written year plans. * Minutes of planning meetings. . lnterviews with: * Staff at this level: managers and others (pharmacist, transport officer etc.). * Staff at FLHF level. Sustainability evaluation instrument no. 2 - districll LGA level - March 2003 version 2 Findin Describe the present situation: The comprehensive plan for CDTI activities is separate and is included as part of disease control or miscellaneous (subhead 10 & 13) under the PHC in the overall LGA plan. The CDTI plans do not in most cases have budgets estimates for CDTi activities. The plans are not drawn up in a participatory manner. . Whv is this? No explanation . Which steps are beino taken to improve the situation? Steps are being taken by the PHC coordinators to include CDTI control in the overall plan Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overalljudgement: is this indicator for sustainability being achieved? Futly I Highly I Moderately I Slightly I Not at all I Not applicable 2.1 Check whether the districU LGA health management team is taking full responsibility for the imolementation of CDTI at this level. indicator assesses whether management is takrng Gharacteristics of the indicator Sources of information a. lt should be the rnanagernent tearn at this level, and not higher levels/ NGDO leadership, which is initiating the key CDT|ac{ivities: planning, targeted monitoring/ supervision, targeted training and HSAM, Mectizan ordering/ distribution. b. There should be a focal person for CDTI activities. . lnspection of year plans. . lnterviews with: * Management team at this level. * Person responsible for CDTI at this level. * Staff at the project level * NGDO leadership. * Staff at FLHF level. Descilbe the present situation: The LGA Oncho Coordinator is in charge of activities at this level and initiates activities. There are 3 LOCT members who assisl the coodinator. lf leadership at this level is nottakinq the initiative in implementino CDTI: . Why is this? Not Applicable . Which steps are beinq taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overall judgement: is this indicator for sustainability being achieved? FullylHightylModeratetylSrignit Sustainability evaluation instrument no. 2 - districU LGA level - March 2003 version indicator a s sesses whether gogramme become more integratea into@f u n ctio ning effectively. 3.1 Check whether routine data conceming Gharacteristics of the i-dicator Sources of information a. tne reporting process should take place within the govemment sysiem, not using other resources. Data being transmitted includes: coverage reports; Mectizan statistics; training reports ; dislribution reports; financia t reports. . Examination of reports and report forms. . lnterviews with: * Managerc and staff at this level. * Staff at the project level. Reporting is done on a monthly basis and sent to the State oncho Coordinator. This is however not withingovemment system, i.e. through the monitoring and evaluation unit. . Why is this? This was not the initialsystem adopted by the programme . Which steps are beinq taken to improve the situation? None Whenwritingtnerep@ * The evidence about how welr this indicator is being achieved. * Reasons for poor performance (if any), steps bein! taken to improve it, and how this is likely to affect . Your overall judgement: is th 3'2 9l?.:[yHP'jl,:.s.:tgpioregg!y!!y at the FLHF tevet in an integrated manner. indicator assesses whe@ Characteristics of the staff at this level shourd routinery onty sufffi community level. 'spot checks' may however be done from time to time. Although one routine supervision visit per FLHF per year is necessary, supervision visits should focus more on FLHFs where there are provenproblems - each supervision visit must be justified. supervision visits for cDTl should be integrated with supervision of otherprogrammes (e.9. through a shared checklist). Transport for supervisory visits should be shared with other programmes. . Examination of: * Supervisorychecklists, plans, itineraries and reports. * Visitor's books at FLHF level. . lnterviews with: * Staff at this level. * Staff at the FLHF level. supervision is canied out al this level to the FLHF with spot checks to communities. These trips are in most casesroutine without definite schedules or targeted to solve problems. There are no suoervisorv checktists or wriiten and no deliberate efforts are r p y l r ritt made to inteorate. Sustainability evaluation instrument no. 2 - districU LGA level- March 2003 version 4lf suoervision is not beino done in an inteorated and efficient manner: . Whv is this? This is not the practice in the LGAs . Which steps are beinq taken to imorove the situation? None yet Anrlvcie a When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and sustainabilitv. how this is tikely to affect dicator for sustainability being achieved? Fully Highly Moderately Slightly Not at all Not applicable tineprocessofmanagementofproblemsandsuccesses,whichare fris'nd'tcator assesies whetherthe programme is running is beginning to accept ownerchip of the programme. Characteristics of the indicator Sources of information a. As soon as problems are identified as a result of supervision visits, orfrom coverage data (i'e. areaswith low coverage) the appropriate rnanager should deal with them. b. Where relevant such problems should be passed on to the appropriate FLHF staff to deal with, with the necessary support - thus empowering these persons' c. Successes should be noted and reported, and appropriate feedback given. d. There should be evidence of action taken based on recommendations in previous monitoring exercises. . Examination of the following documents: * Year plans and annual rePorts. * Minutes of staff/ planning meetings. * Reports of previous monitoring exercises. * Letters of commendation. * Memos. . lnterviews with: * Staff at this level. * Staff at the FLHF level. Describe the present situation: The Oncho coordinator deals with problems when they arise, but in instances where the officer cannot solve such problems, they are reported to the appropriate higher authorities. Such problems only get to the officer if they are beyond the FLHF. There was no evidence that successes are recognised and commended and feedback given. There is also no evidence of action taken based on recommendations from previous monitoring exercises. lt was gathered that this is done but not documented. . Whv is this? System is not weak in terms of capability of personnel but there is not much documentation and planning. . Which steps are beino taken to improve the situation? Sustainability evaluation instrument no. 2 - districU LGA level- March 2003 version Analysis I When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Youroveralljudgement: isthis indicatorfo@ Fully l Hishly 4.'l Check whether sufficient Mectizan is being ordered and re This indicator assesses whether gagra mme i s f u nc:t io n i n g Gharacteristics of the indicator Sources of information The order forms for the districU LGA exist, and should be based on FLHF and community requests. The Mectizan should be available at this level in time for distribution at the time requested by the csmmunities. There should be no reports of shortages andl or late supply. lf there have been shortages, there should be specific plans to remedy them. b. Examination of all Mectizan ordering and stock control documentation at this level. lnterviews with staff at this level (managerc and pharmacist). The request for Mectizan is based on community requests indicated in the previous years community summaryforms. Mectizan was found to be adequate and no shortages were reported. They w-ere also suppliei in good time(January/February of each year) Describe the situation the orevious vear: Same as above Describe the situation the year before that: Same as above ttttg=r9 r!" qrgOt"*.*itn oUt"inino tnffi . Why is this? Not Applicable . Which steps are beino taken to improve the situation? Not Applicable When writing the report you have to sunrrnarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect What is the trend in Mectizan suppty at ttris teven . Your overalljudgement: is tnis inOica Sustainability evaluation instrument no. 2 - districU LGA level - March 2003 version This indicator assesses whetherthe gogramme is efficiently, rfs processes are sr'mple, and it is 4.2 Check whether Mectizan is being collected, stored and efrectively delivered within the government becoming more integrated into the govemment system. Characteristics of the indicator Sources of information a. c. The Mectizan should be controlled within a govemment system. This does not have to be the system routinely used for the supply of other dngs. The system should be effective, uncomplicated and efficient. The districU LGA should ideally fetch its Mectizan from the project level itself (although use of the routine MoH drug supply system to dislricts is acceptable). ln either case transport should be supplied and paid for by govemment at this level. . Examination of: * All Mectizan ordering and stock control documentation at this level. * Vehicle log books and/ or trip authority forms. . lnterviews with slaff at this level (managers, pharmacist, drivers). Fi Describe the present situation: The LGA Coordinator goes to collects Mectizan from the State and stores it in his office. There was no clear indication that govemment is providing transport for the collection of the drugs from the project level. The drug is not controlled within the govemment system. lf the Mectizan supoly is not being administered within a qovemment svstem: . Whv is this? This is to facilitate quick and easy delivery to FLHF , Which steps are being taken to improve the situation? None Analysis r When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. ' Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderatety I Slighily I Not at ail T Not appticabte 5.1 Check whether staff members at this level are being used as trainers. indicator assesses whether the propmme is functioning efricienfly. Characteristics of the indicator Sources of information b. Staff should routinely only train siaff at the FLHF level, and not at the community level. Staff should have ernpowered staff at the FLHF level to see to their ov*n training needs as much as possible, and to conduct training activities at the community level independently. Wherever possible staff at this level should conduct their own training for CDTI, if they have need for such trainino. . Examination of training materials, plans/ programmes, reports: * At this level. * At the FLHF level. . lnterviews with: * Staff at this level (the trainerg. * Staff at FLHF level (the trainees). * CDDs. Sustainability evaluation instrument no. 2 - districu LGA level - March 2003 vercion Describe the present situation The oncho coordinators and Locrs train th.e District Health supervisors (DHS) and FLHF staff who in turn trainthe cDDs under the observation of the coordinator and the loCir. Describe the situation the vear before: Same as above . Whv is this? Not Applicable . Which steps are beinq taken to improve the siluation? Not Applicabte When writing tne reportffi * The evidence about how wefl this indicator is being achieved. - ::j:*"1?Ipoor performance (if any), steps oein! taken to improve it, and how this is tikely to affect . Examine the trend in tn . Your overalljrdg"r"n ? ?= = ,9!:"=k=!gEItq' There should be an objeciive n there should be evidence that staff to be trained ract tnowlea-ie and skiils toperform the job, and thetraining should then focus on this oeR"ciency only.Repeat training of already skilled staff should not frappen. lf circumstances permit training for cDTl should oe integrated with othertraining, e.g. in in-service training programmes. Resources for training (human, transport etc.) should be efficienfly used:* Using as few staff member:s as possible. * using as little time as possibre (without sacrificing quarity)* Choosing the most cost-effective site etc. . Examination of training materials, planV programmes, reports: * At this level. * At the FLHF level. . lnterviews with: * Staff at this levet (the trainers). * Staff at FLHF level (the trainees). * CDDs. Describe the oresent sitG,,or.* Training is more of a routine than targeted with sel objectives this is because of the frequent transfers within thesystem' New Locrs, DHS and FLHF staff a.re lrainedannuaily. irainrno! are noiint{r"t"o with any trainingprogramme' There are sufficient human and materiar reiouili for training at this level ) Same as above lf trqi_nino.is not efficient and inteorate* . Why is this? This is not a policy in the LGAs a None sustainability evaluation instrument no. 2 - districu LGA level - March 2003 version 8 Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Examine the trend in training ac{ivities - is it becoming more efficient and integrated? . Your overall judgement: is this indicalor for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicabte 5.3 Check whether staff at this level is planning and carrying out HSAM activities in an efficient manner. This indicator assesses whether the gogramme is functioning efficiently and effectively, and whether managers are taking ownership of the gagramme. Gharacteristics of the indicator Sources of information a. Staff members identify situations where decision makers lack information abouV commitment to CDTI, and undertake activities to inform and persuade these persons. b. HSAM activities are properly planned. They are only canied out where there is an objective need for them, and not as a matter of routine. c. Such activities should only be canied out at this districu LGA level, and at times at the FLHF level (but only when staff at that level asks for help). d. There is evidence that these HSAM activities have been effective and have led to ac,tion. . Examination of: * HSAM plansl programmes and reports. * Year plans and annual reports. * Minutes of planning meetings. ' lnterviews with; * Staff (programme and rnanagernent) at this level. * Givil authorities at this level. * Staff and civil authorities at the FLHF level. Describe the present situation: HSAM is canied out to policy makers and decision makers at this level. This is usually targeted at new decision makers to solicit for their support and commitment. Evidence of HSAM was noticed during interviews with decision makers at this level. Describe the situation the year before: Same above lf HSAM activities are not beinq caried out efficientlv and effectivelv: . Whv is this? Not Applicable . Which steps are being taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Examine the trend in HSAM activities - is it becoming more efficient? . Your overall judgement: is this indicator for sustainabilily being achieved? Highly I Moderately I Slightly I Not at all I Not applicableFully Sustainability evaluation instrument no.2 - districU LGA level - March 2003 version Check whether appropriate amounts are nned GDTI activities at this level. Characteristics offie The costs for each "rr, "O,,spelt out in a budget. There is evidence of a cost reducrion/ containment strategy (e.g. targetedtraining, HMAS and monitoringl supervision;training coniucieiat FLHF reveretc). Managers at this level should have a clear estimate of the funds that will beavailabre to them for cDTr in the coming year, and shourd be abre to justifythis belief. The totar amount budgetecl for in the year plan should fallwithin thisestimated income. . Examination of the budget documents. . lnterviews with health service and local govemment managers at this level: * Technical managers. * Treasurer/ administrator. me.ouooeiffi . At the orevious distribution: cDTl is usually budgeted u.nder PHC (e.g. wodq bank projects in Umunneochi, capital expenditure in ukwa East ::Bi?;'."';ffiil:'S"ff:jJ::A;:f proi-ectt under tne ireirtn iector subheao inis is not irearry spert in ine ouoset . For the previous year: Same as above . Forthe year before that: Same as above tf nqqqetinq nas Ueen inaoprooil;.o* . Whv is this? No explanation a None When writing tne r : I$y11".!9" about how weil this indicator is being achieved.- *:?:iff llil:oor performance (if anv), steps uein[ taren to improve [, and how this is likety to affect . Examin" rf,",t . Your overalljudgem sustainability evaluation instrument no. 2 - districu LGA level- March 2003 version 10 This indicator assesses whether the programme is and whether government is beginning to assesses whether management is aDIe fo the resources ff needs, as weII as 6.2 Check whether the government at this level is budgeting and disbursing increasing amounts for CDTI yearly, and in good time, accept ownership of the programme and can mobilise the rsources it needs. Characteristics of the indicator Sources of information a. The relative budgetary contributions of the local govemment and other partners to CDTI should be clearly spelt out. b, The amount that the govemment has budgeted in one or more specific CDTI budget lines should be increasing yearly. By the end of Year 5 of APOC funding the bulk of CDTI expenses at this level should be met from local govemment funds; by the end of Year 3 at least half of it.' c. The amounts ac{ually disbunsed from such budget lines should be increasing yearly, as a proportion ol total expenses. (Note that actual disbursement is more important than budgeting, and is a real sign of political commitment). . Examination of: * Budget documents (government and NGDO) * Records of disbursement and expenditure (ledgers, orders, approvals for expenditure etc.) . lnterviews with health service, local government and NGDO managers at this level: * Technicalmanagers. * Treasurer/administrator. There were disbursements of funds, though it could not be ascertain if this was disbursecl ctirectly from the budget. ln 2002 Umunneochi LGA spent N135, 000 for core CDTI activities. ln Ukwa East N130, 000 was expended for training, transporl and stationery in 2A02. . For the previous year: No information . Whv is this? No explanation was given . Which steps are beino taken to improve the situation? Analysis I When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and sustainability. how this is likely to affect . Examine the trend in govemment budgeting and disbursements: . Your overalljudgement: is this indicator for sustainability being achieved? Fully HighlylModeratetylstighttyl@ 6.3 Check whether in case of a deficit between estimated costs and the amount provided by the provision is being made to meet it. commitment to Gharacteristics of the indicator Sources of information a. lf there is a shortfall the management should have specific and realistic plans to bridge it. b. lf it is planned that non-govemment sources of funcling are to be used afier APOC funding ends, written commitment for this should have been obtained at the highest level in these donor otuanisations. lnspection of: * The budget documents (government and NGDO) * Records of expenditure (ledgers, orders, approvals for expenditure etc.). * Letters of agreement. lnterviews with health service and local government managers at this level: * Technicalmanagers. * Treasurer/administrator. Sustainability evaluation instrument no. 2 - districU LGA level - March 2003 version There is no evidence of any dependable resources to be used after APoc even though the LGAs claim they can support CDTI post APOC Describe the situation the preyious vear: Same as above lf the shortfallcannot be met: . Whv is this? No explanation . Which steps are beino taken to improve the situation? None When writing the report you have no sunrmarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect . Examine the trend in shortfallano h@ . Your overalljudgement: is this indicato 6.4 Check whether funds disbursed for CDT| fio fhis inclicator assesses whether the ls Characteristics of the indicator Sources of information a. The budget holder should be using a control system with the following elements: * Approval of each item of expenditure. * Allocation of expenditure against specific budget headings. * Regular calculation of residual amounts under buclget headings. b. Allthe funds released yearly should be spent as budgeted. . lnspeclion of: * The budget documents (government and NGDO). * Financial control records (ledgers, orders, approvals for expenditure etc.). . lnterviews with health service and local govemment managers at this level: * Technical managers. * Treasurer/administrator. . Approval of expenditure: No budget documentation was obtained from the coodinators, however the finance department hact approvedbudgets and some records of expenditure for CDTI activities. . Allocation of exoenditure: No supporting documents . Reqular insioht into budoet line balances: None. Releases are made based on the availability of funds without due consideration to budget. . Whv is this? No explanation . Which steps are beino taken to improve the situation? None Sustainability evaluation instrument na. 2 - dislricu LGA level - March 2003 version t2 Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable 7.1 Check whether adequate and appropriate transport and materials arc available for necessary CDTI activities at this level. This indicator assesses whether the ffogramme is functioning effectively, and whether it is able to mobilise the resources it needs. Characteristics of the indicator Sources of information a. There are adequate numbers of appropilate, functional vehicles available for necessary GDTI activities. b. The governmenl meets the running costs for these vehicles. c. There are sufficient materials available for training and HSAM. . lnspection of: * Each vehicle in the pool: its source; its functional status. * Stocks of materials for training and HSAM. . lnterviews with managers at this level: transoort officers. orooramme manaoers. Describe the availability/ suitability/ functionality of the present vehicles and materials, considering the work still to be done in the coming 5-10 years: They are available and functional but would not be functional for the next 5 - 10 years Adequacy for CDTI tasks* Training/ HSAM material * APOC, MoH, NGDO, other (spectfy) * Is ff working? ls there suitable for the iob? lf transoort and materials are inadequate and funded from non-oovemment sources: . Why is this? lnadequate number of motorcycles and bicycles provided at the onset of the project. . Which steps are beino taken to imorove the situation? None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Highly I Moderately I Slightly I Not at all I Not applicableFully Sustainability evaluation instrument no. 2 - districU LGA level - March 2003 vesion 7.2 Check whether tran This l3 assesses t at this tever is ad"!g3!g!t "nffi Characteristics of the There is a routine mainte ::*HP j:l j:1I*. rhis includes weekty driver maintenanc",scheduled garage servicing, and repracemeniof ,nom ty*r. '"-"'The govemment meets the costs for vehicle and equipment maintenance and repair. Repairs are rapidry ano eficientiy done.Members of staff have ways of coping when vehicres break down or arenot available, so that CDTI ac{iviti'es ire not disrupted. * Vehicle and equipment maintenance schedules.x Vehicle and equipment maintenance records. . lnteryiews with managers at this level: transport officers, There is routine maintenance of the motorcycles at the LGA level, but there is no evidence of schedule or record ofmaintenance' LGA bears the costs of maintenance and altematives are provided when vehicle breaks down. Themotorcycles are under the custody of the coordinator and not in fi," pool of vehicles f"1H:l["*"""""*t ffit:H:l"r$i*:#rnotorcycles and Bamako tnitiative vehicte (atso used for cDl) but woutd . Which steps are being taken to improve the situation? Not Appticabte Phg writ ; :i::r]::T"-:prt how welt this indicaror is beins achieved.* Reasons for poor perfornance (if any), steps nein6 taren tolmprove it,sustainabilitv. and how this is likely to affect . Yorroyuo 7 3 f,?::LYI:'.I:L'h Transport is used at ttr activities at the FLHF rever. n shourd not G used for cDTrimplementation activities at the commil,tl'rcvet. Trips made for CDTI purposes should h property authorisedin writing by the relevant official. Each trip undertaken shouldbe recoded in a lqbook s of the indicator . lnspectionorvffi * Copies of trip authorities (also noting destination and purpose) * Log books. * The supervision plan/ matrix. , lnterviews with managers at this level: transport officers, prograrnrne managers. sustainability evaluation instrument no.2 -districu LGA level - March 2003 version Findinos Describe the oresent situation: other programme officers to support activities at FLHF level also use lhe motorcycles. The motorcycles are underff,:l'::fl:jj:::::l:.:.:fT:p3 9y1 ".fi*T ;f itl ;; ;"",ti p,og,.**es have access to it. rhere are noftd"'.i,ffiffi;,ffi;;J",_[ol lf thg_transoortisnoffi . Why is this? No explanation given a Anangement is being made to use logbooks anct make it t4 )n writing the report It^"-r]9"1.e about how welt this indicator is being achieved. l:ili*Xll,lloor performance (if anv), steps beind tai;;io-irprou. it, and how this is tikety ro arrect . Yorr or"r"lljudg" 7.4 Check whether th^e5^ materials when APOC support comes to an end. is indicator assesses the gogramme managersere mng of the programme, andare anteta find resources for it. Management should tnow t e1! of the programrne, and have specific, rearistic prans to meeiilre neeoat that time. It should be planned that the govemment will:* Provide replacements for vehicles. * Maintain existing vehicles. *- . Provide stationery and materials for training and HSAM.lf itis planned that repracement wiil be from nongovernrnent souroes, y313 : ::lT'p:ll l"j !!': shoutd _have been obiainuJ .i in. nig h""tlevet in these donor organisations (end or veaiil, oi *glii"iiJri shourdhave started (end of year 3). . lnspection of letters of agreement. . lnterviews with: * Programme managers at this level : administrators, technical managers. * NGDO project managers. * High-ranking local government officials. There are no plans in place to replace motorcycles when Apoc support comes to an end. . Whv is this? I Wh"l writing tne I I[^"_rf!g!o about how wefl this indicator is being achieved.* Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect . Yort or"rrllj sustainability evaluation inslrument no. 2 - districu LGA tevel - March 2003 version 15 8.1 Check whether staff at this level is skilled and knowledgeable, regarding the implementation of CDTI in its area of ooeration. This indicator assesses whether the s been able ta sufficient resources for itself. Characteristics of the indicator Sources of information a. Staff should have enough knowleclge and skillto undertake all the key CDTI activities themselves: planning, training, HSAM, ensuring Mectizan supply, monitoringl supervision. b. Staff at this level should remain in one post for at least five years. c. There should be immediate training (in CDTI) of new, unskilled project staff members who have CDTI responsibilities. . lnspection of: * Staff files. * Monitoring reports. * Activity reports. . lnterviews with * Managers and other staff at this level. * Staff at the project level * Staff at the FLHF level. Describe the present situation: . Particulars of cunent staff . lnformation about staff stability The LOCTs are more stable while the DHS and FLHF staff members are transfened frequently. . lnformation about in-service trainino: No information No. of persons competent enough to perform the job lf the staff at this level lack skills. and are often transfened: . Whv is this? Frequent transfers are unavoidable . Which sleos are beino taken to imorove the situation? None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicabte Sustainability evaluation instrument no. 2 - districu LGA level- March 2003 version 8.2 Check whether staff members at this level are committed to their CDTI work. This indicator assesses whether the pragramme has been able ta develop resources for itself. Characteristics of the indicator Sources of information a. Staff members express satisfadion with their present responsibilities b. There is evidence of specific motivational practices and rewards within the programme: awards, financial incentives, compensation in cash or kind. c. Salaries/ wages and allowances are paid regularly. d. Staff members mention non-financial rewards inherent in CDTI work- e. There is evidence from partners and workers at the FLHF level that staff members are committed to their CDT work. . lnspection of: * Staff files (for performance records and awards). * Documentation about incentive schemes. . lnterviews with * Managers and other staff at this level, * Localgovemmentofficials. * NGDO project staff. * Staff at FLHF level. Fi Describe the present situation: The staff members are satisfied with their work and are committed to their responsibilities. Motivation is however poor because salaries are delayed. lf staff members appearto have little commitment to CDTIwork: . Whv is this? No material motivational practices because of lack of funds . Which steps are beino taken to imorove the situation? Will plan for motivational practices Analvsis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable This indicator assesses whether the programme and less sustainable. Characteristics of the indicator Sources of information a. Alt sub-.districts and communities identified by the latest REMO should be under treatment (i.e. the geographical coverage rate is 100%). b. This rate should be stable or increasino. ' lnspection of: * Distribution reports and statistics at community level, for this districU LGA, for the past 3 years. * REMO list of endemic comrnunities for this districU LGA. . lnterviews with: * Staff at district/ LGA level. * Staff at FLHF level. Sustainability evaluation instrument no. 2 - districU LGA level- March 2003 version 17 . At the last distribution: 100o/o . The year before: 100o/o . The vear before that: lQOo/o lf thegeograptrica@ . Why is this? Not Applicabte . Which steps are being taken to improve the situation? Not Applicabte When writing tn" ruO improve it; and how this is likely to affect sustainabitilvWhatisilretren@ . Iel|.r oyelelrtggFully (1007o of communities are doing Negligibly(<85% of communities -if therateispoorth@, Sources of infonnation All communities in the districi/ LGA should have a therapeutic coverage rate of 65% or higher. These rates should be slable or increasing. lnspection of: * Qislribution reports and statistics at community rever, for thisdistricU LGA, for the past 3 years. * REMO list of endemic cornmunities for this dislricu LGA.lnterviews with: * Staff at districU LGA levet. * Staff at FLHF levet. 680/o (Umunneochi), 64yo (tkwuano) and glolo(Ukwa East) . The vear before: (2001) 83o/o (Umunneochi) 61yo(tkwuano) and 73olo (Ukwa East) . The vear before that: (2000) 657o (Umunneochi) 52% (tkwuano) and 690/o (Ukwa East) lf thetherapeutic@ . Why is this? Not Appticable . Which steps are being taken to improve the situation? Not Appticabte Sustainability evaluation instrument no. 2 - districu LGA level - March 2003 version l8 Sustainability evaluation instrument no. 2 - districU LGA level- March 2003 version Analysis . When writing the report you have to summarise t improve it; and how this is likely to affect sustainability. . What is the trend in therapeutic coverage? your overail Juqgemenr: ts rnts tnotcaror ot sugglnqqlly lqing achieved? Fully (100% of communities have a therapeutic coverage rate 265% - stable or increasinq) Highly (90- 99% ot communities) Moderately (80- 89o/o of communities) Slightly (/0- 79o/o ol communities) NEgligibly (<70o/o ot communities) lnstrument 4: community level NOTE: This instrument evaluates the CDTI programme at the level of villages and communities, where the actual distribution of Mectizan takes place. We use the term 'community' to refer to both villages (in societies where there are welldefined villages) and communities where family groups are fairly isolated from each other, and do not live in a 'village' in the accepted geographical sense of the word. tn these communities we find the following persons are involved in the GDTI programme: * The community or village leadership - both traditional and elected. * The community dirccted distributors (GDDs) - the persons who have been selected by the community to do the distribution of Mectizan. * The other, 'ordinary' community members, who take the Mectizan yearly. ln this document these persons will be referred to as'community members'. When coltecting information from 'ordinary'community members discussion groups should be conducted. The main function of this level is to disTribute the [tectizan yearly to the communityt memberc: . Communities select CDDs, who are supported by the leadership and the other community members. . The CDDs update the community census every year; distribute the Mectizan appropriately; and send a on the distribution to the FLHF level. I I Eeographicafname of this community/ village:Aroayama, Omuigu, Azunchayi, Upa, Umunwankwo, Amaoba, Obozu, Umuogo, Uhude, Umuada, Obiagu-Amoii-Lekwesi and Ama ogidi Project: Abia State CDTI Proiect, Nigeria Researcher: Ekanem lkpi Braide, Yisa Saka, Charles Franzen, Obinna Onwujekwe and Sunday lsiyaku Date: 17 - 20 June 2003 Abbreviations/ acronymsCDD community directed distributorCDTI community directed treatment with ivermectinFLHF first line health facilityHSAM heatth education, sensitisation, advocacy, mobilisation - i.e. activities that are aimed at getting all the key players to participate wholeheartedly in the programmeNGDO non-govemmentaldevelopment organisation assesses whether the progrcmme is efftcient and mare streamlined and time-efficient 1.1 Check whether CDDs are planning and managing their CDTI work efficiently. the job, the higher its sustainability. Characteristics of the indicator Sources of informataon a. CDDs choose visiting times and routes, whieh will make the work less burdensome. b. CDDs arrange with the community leadership for helpwith specific problems, such as families who are not willing to participate in the programme. c. CDDs carry out census and distribution during the same visit (using this census data for the followino yeafs order). . lnspection of community treatment registerc. . lnterviews with: * CDDs. * Community members. * Community leaders. * FLHF staff. Describe the present situation: On notification of the availability of the drug at the FLHF level the CDDs collect the drug after consultation with the community leaders and members. They also agree mode and timing of distribution. These are usually done at community meetings, in churches and other meetings. Community census is ufiated before distribution Sus{ainability evaluation instrument no. 4 - community level- March 2003 version 2 commences lf CDDs are not workino efficientlv: . Whv is this? Not Applicable , Are anv steps beino taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully I Hignly I Moderately I Slightly I Not at all I Not applicable 2.1 Check whether community leadership is managing problems with the distribution. Characteristics of the indicator Sources of information b. The community leadership should be taking responsibility for the distribution of Mectizan within the community. lf coverage (geographical and therapeutic) is not adequate or not being maintained, the leadership should undersland the reasons for this. Together with the community at large, the leadership should identifu and solve problems related to the dislribution. . lnspection of minutes of community/ council meetings (where available). . lnterviews with: * CDDs. * Community members. * Community leaders. * FLHF staff. This indicator assesses whether it. programme , and whetherthe community is taking ownership of . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect Describe the present situation: ln most of the communities visited, the community leaders are responsible for ensuring dis{ribution in their communities. There are adequate coverage (therapeutic and geographic) except in a few inslances were low therapeutic coverage was found to be due to non commitment from the CDDs. Community leaders, when informed of problems, try to solve such problems with the CDDs lf the community leadershio is not involved in the distribution: . Whv is this? Not Applicable . Are anv steps beino taken to improve the situation? Not Applicable Sustainability evaluation instrument no. 4 - community level- March 2003 version -) r Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly Moderately Slightly Not at all Not applicable 2.2 Check whether the community at large has been involved in taking decisions on the distribution This indicator assesses whether the Characteristics of the indicator Sources of information a. The community should have taken responsibili$ for decisions such as: * The selection/ changing of CDDs. * The timing and mode of distribution. . lnspection of minutes of community/ council meetings (where available). ' lnterviews with: * CDDs. * Community members. * Community leaders. * The persons supervising CDDs: FLHF staff, lay supervisors etc. Describe the present situation: All community members are involved in decision making in CDTI. They select their CDDs and decide on timing and mode of distribution. There are a few instances whEre the community leaderc and council members choose the CDDs. There are also areas where the community memberc wrongly hlieve that CDDs must be females. ln most cases distribution is from house to house. lf the communitv is not sufficientlv involved in takino decisions: . Whv is this? Not Applicable . Are anv steps beino taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overalljudgement: is this indicator for sustainability being achieved? Fuily I Highty I Moderately I Slightly I Not at all I Not applicable 2.3 Check whether the community members value and annual treatment. assesses whether the community is taking Gharacteristics of the indicator Sources of information Community members should be able to mention one ot more advantages of taking Mectizan. Community members should express the need for annual treatment with Mectizan. People should show underslanding of the need for, and express interest in longlerm treatment with Mectizan. c. lntervievus with: . CDDs. . Community members. . Cornrnunity leaderc. . The persons supervising CDDs: FLHF staff, lay supervisors etc. Describe the oresent situation: The community members are aware of the advantages of taking Mectizan. They indicate that it improves their induces exoulsion of worms and clears scabies. They are aware that the druo should be taken annually but Sustainability evaluation instrument no. 4 - community level- March 2003 version indicator assesses progrcmme is effective. lf such reprting is not taking place Mectizan supply 4 most do not know for how long. ln a few cases where the communities knew the duration of treatment, they do not know the reason for long-term treatment. lf communitv members do not value and acceot the treatment: . Whv is this? Not Applicable . Are anv steos beinq taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at att I Not applicable 3.1 Check whether CDDs are reporting appropriately to the FLHF level. willbe compromised, which is bad for suslainability. Characteristics of the indicator Sources of information a. b. Reports to the FLHF level should get there on time. Reports may be summary reports, or the original community distribution record, depending on the level of skill of the CDD. Adequate transport should have been arranged for distribution records/ reports to be handed to the appropriate person. . lnspection of community distribution reports. . lnterviews with: * CDDs. * The percons supervising CDDs: FLHF staff. lav supervisors etc. Describe the oresent situation: At the end of treatment, CDDs collate their treatment figures and submit immediately to the DHS. Most of the communities do not provide transport for the submission of reports because the FLHF are close to the communities lf the reportino bv CDDs is ooor: . Whv is this? Not Applicable . Are any steps beino taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightty I Not at all I Not applicable Sustainability evaluation instrument no. 4 - community level- March 2003 version This indicator assesses whether the programme right amounts are received it will foster 4.1 Check whether the amount of Mectizan is received. community ownerchlp, whrbh is good for susfarnaDiltfy. Characteristics of the indicator Sources of information a. Allcommunity members who were eligible for treatment got it, and some Mectizan was left over for absentees and those who were temporarily non-eligible. b. There should be a rational explanation about how the amount ordered forthe community is calculated (on the basis of population). . lnspection of treatment register for the community (held by CDDs; or at higher levels) . lnterview with: * CDDs. * Comrnunity members. * Community leaders. * The persons supervising CDDs: FLHF staff, lay supervisors etc. What hapoened at: . The last round of treatment? The right amount of Mectizan is usually received and distributed. CDDs are given one tin of Mectizan at a time to distribute and this is replenished by the DHS. Most CDDs do not know how to calculate the number of drugs to order for their communities and depend on the supervisor to determine the number of drugs to be given to them. . The rounds before that? Same as above lf the wronq amount of Mectizan was received: . Whv is this? Not Applicable . Are anv steos beino taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. ' What is the trend in Mectizan supply? . Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not applicable 4.2 Check whether the CDDs or community members themselves fetch the yearly supply of Mectizan. This indicator assesses whetherthe project fosfers community ownerchip. Characterastics of the indicator Sources of information The CDDs or community membem fetch the Meclizan they need every year, from a designated and mutually acceptable place. Adequate transport should have been ananged for Mectizan to be collected from such a place. a. b. lnterviews with: . CDDs. . Community members. . Community leaders. . The persons suoervisino CDDs: Sustainability evaluation instrument no. 4 - community level- March 2003 version FLHF staff, lay suPervisors etc. DistricU LGA staff.c. tn tfre case of remote communities, the districU LGA, in collaboration with FLHFs, should ensure that supplies reae! Jsc!-glgllps. What haopened at: . The last round of treatment? The CDDS usually collect the drug from the central collection point. ln most cases no adequate transport arangements are made by the communities for the CDDs to collect the drugs, because the FLHF are close . The rounds before that? Same as above ffiCDDs have not been collectinq the Mectizan: . Why is this? Not Applicable . Are anv steos beinq taken to imorove the situation? Not Applicable Analysis I Wnen writing the report you have to summarise: * The evidence about how well this indicator is being achieved- * Reasons for poor performance (if any), steps being taken to improve it and how this is likely to affect sustainabilitv.;-Wfat is the tren ectizan theY need? inabilitY being achieved? ignttY | ruot at att l Not aPPlicable Describe the oresent situation: CDDs and community leaders have been involved in HSAM particularly during community meetings. lnformation is also provided by CDDs to persons who refuse treatment to persuade them to take Mectizan. lf CDDs and communitv leaders are not involved in HSAM: . Whv is this? ffiDDsandcommunityauthoritiescontinuetobeengagedinHSAMofother e goiect ts effective, and whether the has taken ownershiP of it. Characteristics of the indicator Sources of information CDD9 community authorities identify situations where community members require information. CDDV community authorities take necessary steps to provide required information; encoumge community members to provide resources; promote acceptance and ownership (meetinos. sanctions, communitv by-laws). lnterviews with: . Community leaders. . Comrnunity members. . CDDs. ' The persons supervising CDDs: FLHF staff, lay supervisorc etc. Sustainability eraluation in$rument no. 4 - community level- March 2003 version 7Not Applicable . Are anv steps beino taken to improve the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. ' Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not applicable 6.1 Check whether the community has made arangements to fund local costs of distribution. Characteristics of the indicator Sources of information a. The community should support individualswho are providing CDTI services for them. b. The community should make provision for the supply of record books, pencils, transport and other expenses incuned during CDTI. lnterviewswith: . Community leaderc. . Community members. . CDDs. . The persons supervising CDDs: FLHF staff, lay supervisors etc. This indicator assesses whether the project can mobilise fhe resources ownership. needg and fosfers community Describe the oresent situation: Most of the communities provide incentives for their CDDs excefl in a few cases where the community said they did not know they should provide incentives for their CDDs. Most communities give an average of Nl, 000 - N2, 000 to their CDDs. Few communities do not give any incentives at all. ln all cases, communities provide treatment registers and writing materials. lf the communitv is not supportino or helpino to defrav costs: . Why is this? Not Applicable ' Are any steps beino taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performancr (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I stigntty I Not at all Jttot appticaote Sustainability evaluation instrument no. 4 - community level - March 2003 version This indicator assesses the project can mobilise the resources if needs, whether the community has 7.1 Check whether there is a satisfactory ratio of CDDs to households. taken ownerchip of the programme. Characteristics of the indicator Sources of information a. A ratio of at leasl one CDD to 20 households (or 2 CDDs per 250 population) is recommended. b. The households for which CDDs are responsible should be close to their otrvn homes. lnterviews with: . Cornrnunity leaders. . Community members. . CDDs. . The persons supervising CDDs: FLHF staff, lay supervisors etc. The ratio of CDDs to population is inadequate. On the average each CDD treats between 500 - 600 persons. . The averaqe distances that CDDs have to walk to oet to homes: The households the cDDs visit are relatively close (average of 2 - 3 kilometres) The community members aid they did know they could have more than one CDD and the ideal CDD: population ratio was not explained to them. . Are anv steps beinq taken to imorove the situation? More CDDs will be selected Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. ' Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I 7.2 Check whether all CDDs have received appropriate training. Characteristics of the indicator Sources of information a. CDDs should be skilled at their work: doing the census; giving the right dose; knowing who is not eligible; knowing what to do with side-effectsb. There should be a plan in place for training CDDs to replace those who drop out, or when new ones are elected for other reasons. . lnteryiews with: * Community leaders. * Community members. * CDDs. * The persons supervising CDDs: FLHF staff, lay supervisors etc. . Observinq CDDs at work. Sustainability evaluation instrument no. 4 - community level- March 2003 version Describe theltesent situation : Most of the CDDs have good reporting skilts as reflected in their treatment registers. Exclusion criteria are known and observed, while cases of side effects are usually refened to health faciliti-es. There are plans to train newCDDS where they are replaced and the communities did not consider this a problem. lf CPPs appear rnskilled. or if there is . Why is this? Not Applicable . Are any steps beinq taken to imorove the situation? Not Applicable When writing the report you have to sunrrnarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect . Your overall judgement: is tnis inOic 7.3 check whether cDDs arc willing to continue iheir work in cDTl. This indicator assesses wnefn@ Gharacteristics of the indicator Sources of information a. b. CDDs should express willingness to continue with distribution in the long term, given the conditions, which prevail in the community. Few CDDs in this community have dropped out from the distribution work. lnterviews with: . Cornrnunity leaders. . Community members. . CDDs. . The persons supervising CDDs: FLHF staff, lay supervisors etc. Describe the present situation: ln all communities visited except one, the CDDs express willingness to continue distribution for a long term. There were very few cases of CDD attrition r wnv ts tnts? Not Applicable for most communities ln.the one community where the CDD expressed unwillingness to continue the problem was identified as wrong criterion for CDD selection. The CDDs here are young school leavers who are keen on leaving for skill acquisition. . Are any steps beinq taken to improve the situation? Not Applicable for most communities CDD will be replaced by a suitable CDD in the only community where it is a problem and appropriate criteria will be applied for future selection of CDDs. Sustainability evaluation instrument no. 4 - community level- March 2003 version 10 Analysis t When writing the report you have to summarase: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, sustainabilitv. and how this is likelY to affect eOz Fully I not at att | ruot aPplicabte AJ Check whether the household coverage in the community is satisfactory. Characteristics of the indicator Sources of information a. All households and areas in the community are being treated. This includes the hamlets for which the community has agreed to be responsible. b. lf this household coverage is not 100%, it should be improving. lnspection of: * CDDs'treatmentregisters. * Yearly distribution reports for that community. lnterviews with: * Community leaders. * Community members. * CDDs. * FLHF levelstaff. This indicator assesses whether the /ess susfainable. is effective- if rate is poor the Proiect is struggling, The household coveraqe situation: . At the last distribution: Household coverage is 100o/o as reported . The vear before: 1O0o/o . The year before that: 100o/o in the treatment register. lf household coveraoe is ooor: . Whv is this? Not Applicable . Are anv steos beino taken to improve the situation? Not Applicable Analysis nsforpoorperformance(ifany);stepsbeingtakento imorove it: and how this is likelv to affect sustainability. . What is the trend in household coverage? indicator of sustainability being achlqyqq? Fully (100o/o coverage of all households) Highly (only nomads in the sunounding area were missed) Moderately (the outlying hamlets were also missed) Slightly (some wards of the community were also missed) Negligibly (no-one got treated, or only a few families) Sustainability evaluation instrument no. 4 - community level- March 2003 version 8.2 Check whether the community has a satisfactory therapeutic coverage rate. This indicator assesses whetherthe programme is effective- rate is poor the project is clearly struggling, and less susfarnable. Characteristics of the indicator Sources of information a. The community overall has a therapeutic coverage rate of 650,6 or higher. b. This rate should be stable or increasing. lnspection of: * CDDs'treatmentregisters. * Yearly distribution reports for that community. lnterviews with: * Community leaders. * Community members. * CDDs. * FLHF level staff. The therapeutic coverage situation in the communities . At the last distribution: (2002)... figures obtained from SOCT records Aroayama - 96% Umunwankwo - 78o/oOmuigu - 50o/o Amaoba Azunchayi - 98o/o Obozu Uhude - 39o/oUmuada, - 97o/o Obiagu-Amoji- Lekwesi - 91o/o Ama Ogidi - 960/o Uhude - o/o Umuada, - o/o Obiagu-Amoji- Lekwesi - o/o Ama Ogidi - o/o Uhude - YoUmuada, - o/o Obiagu-Amoji- Lekwesi - o/o Ama Ogidi - o/o ' 82o/o - 55o/"Upa - 89a/o Umuogo - 83o/o Figures not available for 2001 & 2000 . The year before: (2001) Aroayama - o/o Umunwankwo Omuigu - o/o Amaoba Obozu Azunchayi - o/o ObozuUpa - o/o Umuogo . The year before that: (2000) Aroayama - o/o Umunwankwo -Omuigu - o/o Amaoba Obozu - o/o o/o o/o o/o o/o o/o a/o o/o Azunchayi - o/o Upa - o/o ObozuUmuogo No village specific coverage figures were available for 2000 and 2OA1 lf the therapeutic coveraoe rate is ooor: . Whv is this? Coverage figures are inaccurate in most cases because eligible population rather than total population is used as denominator in calculation of therapeutic coverage. . Which steps are beino taken to imorove the situation? Training will be canied out to conect the mistake. Analysis . When writing the report you have to summarise th improve it; and how this is likely to affed sustainability. a What is the trend in therapeutic coverage? ' Your overalliudgement: is this indicatlgr of sustainability ueing:chieveoz Fulty (the community has a therapeutic coverage rate a65% - stable or increasing) Moderately (the therapeutic coverage rate is 2650/o, but it is unstable or decreasins) Negligibly (the therapeutic coverage rate is <65%) Sustainability evaluation instrument no. 4 - community level- March 2003 version I lnstrument 3: first line health facility (FLHF) level NOTE: . This instrument evaluates the level, which finally interacts with the villages and communities, in ensuring that CDTI takes place in all the communities in its area of operations. This level has different names in different countries. ln most countries there is a health centre - a clinic, or health centre, or dispensary, which we call a'first line health facility'. The health workers who work there are the ones who are responsible fortraining and supporting the GDDs in the villages. . When we speak of a FLHF we therefore mean: * The health facility and its staff. * The accompanying political/ administrative mechanisms between the districU LGA and community levels. . By 'FLHF team' is meant the group of persons working in the first{ine health facility and in its catchment area. . The level below this one is of course the community, the villages. Here the CDDS -'community directed distributors' - live and work. We refer to this level as the 'community' level. The main function of this level is to wort with the village communities, so that CDTI is established in them: . Mobilising them to become involved in CDTI, by selecting CDDs. . Training the GDDs and supporting them in their work. . Arranging a dependable supply of Mectizan for them, at the right time each year. . Helpinq them to collect and forward the coveraqe data for their communitv. Geographical name of FLHFs: Oboro, Ariam/Usaka, Umuigubeachara, Azumini, Ngodo and Umuchieze Project: Abia State CDTI Project, Nigeria Researcher: Ekanem lkpi Braide, Yisa Saka, Charles Franzen, Obinna Onwujekwe and Sunday lsiyaku Date: 17 - 20 June 2003 Abbreviations/ acronyms CDD CDTI community directed distributor community directed treatment with ivermectinFLHF first line health facilityHSAM health education, sensitisation, advocacy, mobilisation - i.e. activities that are aimed at getting all the key players to participate wholeheartedly in the programmeNGDO non-govemmentaldevelopment organisation indicator assesses whether the programme is being an integrated manner, and whether management ls begrnnrng to accept ownerchip of the gogramme. 1.1 Check if there is a written year plan for CDTI in the FLHF arca. Characterastics of the indicator Sources of information a. There should be a written plan or timetable in existence, for the most recent round of CDTI (this is recommended). b. ldeally the plan should be integrated into the overall year plan for the FLHF area. c. ldeally CDTI should form part of the 'minimurn' or 'recommended' packaqe for this level . lnspection of: * The written year plans. * Minutes of planning meetings. . lnterviews with staff at this level. ln most of the FLHF there are no written plans or timetable for CDT activities. This was seen in only two of the six visited. Where there is a plan it was not integrated into the overall plan of the area and is not part of a minimum at this level. The olans seen were however not into the overall plan of the FLHF. Sustainability evaluation instrument no.3 - FLHF level- March 2003 version 2lf there is no written plan for CDTI: . Why is this? No explanation . Are anv steos beino taken to imorove the situation? None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. , Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not applicable 2-'l Check whether the FLHF management team is taking full responsibility for CDTI at this level, in an integrated manner. This indicator assesses whether the CDTI projed r's integrated into the heafth sysfem, and whether management is beginning to accept ownership of the programme. Characteristics of the indicator Sources of information The FLHF management team and all health staff at this level consider the program as theirs and are initiating the key CDTI activities: planning, monitoring/ supervision, training, HSAM, Mectizan ordering/ distribution. The political head/ senior politician at this level should know about CDTI and have participated in some CDTI activities. a. b. . lnspection of year plans. . lnterviews with: + Management team at this level. * Senior political figures at this level. * Staff at the districU LGA level. Describe the present situation: The officers in charge of the FLHFs are also in charge of CDTI at this level and are responsible for initiating CDTI activities like training of CDDs, monitoring and supervision. Most of the political leaders at this level are aware of and have participated in CDTI activities. lf DMT is not takino full responsibility for CDTI: . Whv is this? Not Applicable . Are anv steos beino taken to imorove the situation? Not Applicable . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect Sustainability evaluation instrument no.3 - FLHF level - March 2003 version J . Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly Moderately Slightly Not at all Not applicable 3.1 Check whether routine and necessary data concerning CDTI activities at this level are being This indicator assesses whether the Nryamme is becoming more into the national health sysfem. Characteristics of the indicator Sources of information a. The reporting process should be within the government syslem, not using other resources. Data being transmitted includes: coverage reports; distribution reports ; Me dizan statisiics, tra i ning re po rts. ' Examination of reports and report forms. . lnterviews with: * Manages and staff at this level. * Staff at the districU LGA level. Describe the present situation: The reporting system is not part of the standard reporting process of the health system i.e. it not part of the monitoring and evaluation minimum reporting package. Reports are sent to LGA coordinator for collation. Copies of the reports are kept at this level. lf CDTI data at this level are not beino orocessed within oovernment svstem: . Whv is this? Not the practice in the LGAs . Are any steps being taken to imorove the situation? None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overalljudgement: is this indicaior for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not appticable 3.2 Check whether health service staff at this level is routinely and efficiently supervising GDTI activity at the communities on site in an inteqrated manner. This indicator assesses whether the CDTI programme Characteristics of the indicator Sources of information a. b. Although one routine supervision visit per community per year is necessary, supervision visits should focus more on communities where there are proven problems - each supervision visit must be justified. During visits to communities FLHF staff should turn their attention to as many health related programmes and problems as possible. . Examination of: * Supervisory checklists, plans, itineraries and reports. * Log books. . lnterviews with: * Staff at this level. * Villaoe heads and CDDs. Sustainability evaluation instrument no.3 - FLHF level- March 2003 vercion Describe the present situation: Supervisory visits are made to each community at least once a year. The visits are mainly routine, not targeted and not in an integrated manner. . Whv is this? No explanation . Are any steos beinq taken to improve the situation? None Analvsis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not appticable 3.3 Check whether there is a routine process of management of problems and successes, which are indicated by the monitoring system (coverage data, visits and This indicator assesses whetherthe programme is berng implemented efricienfly and management is beginning to accept ownerchip of the programme. Characteristics of the indicator Sources of information a. As soon as problems are identified through supervisory visits, coverage data etc. (e.9. communities with low coverage) health staff at this level should deal with them. b. Where relevant such problems should be passed on to the relevant community to dealwith, with the necessary support - thus empowering c,ommunities to make decisions on CDTI and cope with problems. c. Successes should be noted and reported, and appropriate feedback given to communities. d. There should be evidence of action taken based on recommendations in previous monitoring exercises. , Examination of the following documents: * Year plans and annual reports. * Minutes of staff/ planning meetings. * Reports of previous monitoring exercises. * Letters of commendation. . lnterviews with: * Staff at this level: in-charge and others. * Community leaders and CDDs. Describe the oresent situation: Staff at this leveldealwith problems identified from monitoring visits in consultation with the communities. Where the problems cannot be solved they are refened to the LGA coordinator. There is no evidence of success recorded or actions taken based on recommendations from monitoring visits or problems solved. lf there is no routine process of managing problems and successes: . Why is this? There is a process but no documentation . Are any steps being taken to improve the situation? None Sustainability evaluation instrument no.3 - FLHF level- March 2003 version 5 Analysis ' When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . Your overalljudgement: is this indicator for sustainability being achieved? Futty Highly I Moderately I Slightly I Not at all I Not applicable 4.1 Check whether sufficient Mectizan is being ordered annually, and in good time. This indicator assesses whether the programme is Gharacteristics of the indicator Sources of information a. The order forms for the FLHF area exist, and orders should be based on the reguests from the community. b. The Mectizan should be available at this level in time for distribution at the time requested by the communities. c. There should be no reports of shortages andlor late supply. lf there have been shortages, there should be specific plans to remedy them. . Examination of all Mectizan ordering and stock control documentation at this level. . lnterviews with: * Staff at this level (managers and pharmacist). * Village leaders and CDDs. F What haopened at: . The last round of treatment? Requests for Mectizan are based on information on population generated from the community summary forms. This is used to determine the quantity of drug to order. The drug is usually available in time for distribution (January/February). There were no reports of shortages of or late supply of Mectizan. . The round ofthe vear before? Same as above . The round the year before that? Same as above lf sufficient Mectizan is not beinq obtained annuallv: . Whv is this? Not Applicable . Are anv steos beino taken to imorove the situation? Not Applicable Analysis t When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. + Reasons for poor performance (if any), steps being taken to improve it, sustainability. and how this is likely to affect . What is the trend in Mectizan ordering and supply? . Your overalljudgement: is this indicator for sustainability being achieved? Highly I Moderately I Stightly I Not at ail i trtot appticabteFully Sustainability evaluation instrument no.3 - FLHF level- March 2003 version 4.2 Check whether Mectizan is being collected, stored and effectively delivered within the government becoming more integrated into the govemment system. indicator assesses the gogramme is functioning efficiently, its ffocesses are simple, and it is Characteristics of the indicator Sources of information b c. The Mectizan should be controlled within a govemment system. This does not have to be the system routinely used for the supply of other drugs. The system should be effective, uncomplicated and efficient The FLHF level should fetch its Mectizan from the districU LGA level itself, by means of transport supplied and paid for by government at this level. Communities should fetch their Mectizan from the FLHF themselves. However in situations where villages are very far from health centres, or where it is easy/ practicable for FLHF staff to deliver the Mec{izan, they may help with the delivery. d. ' Examination of: * All Mectizan ordering and stock control documentation at this level. * Vehicle logbooks and/ or trip authority forms. . lnterviews with staff at this level (managers, pharmacist, drivers). Fi Describe the oresent situation: Mectizan is stored at the storage facility in the health centre. lt is controlled within the programme system where the drug is collected from the LGA and managed by the supervisor. There are no specific transportation arangements to collect the drugs from the LGA. Communities come to the facility to collect their drug allocations and this is recoded in an inventory for stock control. No shortages were recorded at this level. lf Mectizan is not beinq received and stored within a oovemment svstem: . Whv is this? Not Applicable . Are anv steos beino taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable assesses whether the Nogramme i s f u nctioning effic i ently. Gharacteristics of the indicator Sources of information a. There should be an objective need for each episode of training. This means there should be evidence that CDDs to be trained lack knowledge and skills to oerform the iob. and the trainino should then focus on this deficiencv onlv. . Examination of training materials, plans/ programmes, reports at Sustainability evaluation instrument no.3 - FLHF level- March 2003 version 7 perfolrn the job, and the training should then focus on this deficiency only. I ttris tevet.b. Resources for training (human, transport etc.) should be efficiently used: I . lnterviews with: * Using as few staff members as possible. | * Staff at this level * Using as little time as possible (without sacrificing quality) I ftne trainers). * Choosing the most cost-effective site etc. | * CDDs (the Findi Describe the present situation: The staff at the facility train CDDs and these trainings are mainly routine. ln a few instances they were targeted e.g. training of new CDDs and introduction of new operation strategies like the community self-monitoring. There is adequate manpower and material for training at this level. Describe the situation the vear before: Same as above lf traininq is not efficientlv done in an inteorated manner: . Whv is this? Not Applicable ' Are anv steps beinq taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . What is the trend in the way training is done - the method and conGnt? ' Your overall judgement: is this indicator for sustainability being acnieveOe Fully HighrylModeratelylsrightrylW 5.2 Check whether staff at this level is planning and carryins out HSAM activities in an efficient tnanner. This indicator assesses whether the gogramme are taking ownership of the programme. Characteristics of the indicator Sources of information a. Staff members identify situations where decision makers lack information abouU commilment to CDTI, and undertake activities to inform and persuade these persons.b. HSAM activities are properly planned. They are only canied out where there is an objective need for them, and not as a matter of routine. c. There is evidence that these HSAM activ'rties have been effective and have led to adion. . Examination of: * HSAM plans/ programmes and reports. * Year plans and annual reports. . lnterviews with: * Staff at this level. * Civil authorities at this level. * Community leaders and CDDs. sustainability evaluation instrument no.3 - FLHF level- March 2003 version 8 Findi Describe the present situation: HSAM is canied out in this level by the health staff based on needs after the initial HSAM canied out at the beginning of the programme. HSAM, targeted at problem solving, is canied out during community meetings. There are however no clear indications how these have led to effective actions Describe the situation the vear before: Same as previous year lf staff is not effectively engaged in HSAM: . Why is this? Not Applicable . Are any steps being taken to improve the situation? Not Applicable 6.1 Check whether the costs involved in planned CDTI activities at this level are clearly defined and budgeted for. This indicator assesses whether the prog amme is functioning efficiently. Characteristics of the indicator Sources of information a. The costs for each CDTI related activity in the year plan should be clearly spelt out in a budget. These activities include monitoring/supervision, training, HSAM, and ananging Mectizan supply. b. The staff should be able to justify the amount they plan to use. There should be evidence of cost containment (e.g. by targeting training, HMAS and supervision). . Examination of the budget documents (government and NGDO). . lnterviews with: * FLHF team managers (leader, treasurer). * Local government managers at this level (chairperson, administrator, treasurer). Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Examine the trend in HSAM activities - is it becoming more efficient? . Your overalljudgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightty I Not at all I Not applicable What haooened at: . The last round of treatment? There is no budget at this level for CDTI activities . The round ofthe year before? Same as above Sustainability evaluation instrument no.3 - FLHF level- March 2003 vercion 9 lf costs involved in CDTI related activities are not clearly defined ' Whv is this? It is not clear why costs related to CDTI activities are not available at this level, but indication are that this is not the routine practice in the syslem . Are anv steps beino taken to improve lhe situation? None Analysiq I When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainability. . What is the trend in the way costing for CDTI related activities is done? . Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not appticabte 6.2 Check whether sufficient funds to cover these costs are being disbursed from FLHF and/or districU LGA resources. accept ownerchip of the programme and can mobilise the resources rt needs. This i ndicator assesses the programme is becoming and whether management is beginning to Characteristics of the indicator Sources of information a. Funding disbursed is enough to enable targeted, essential CDTI activities at this levelto be canied out. b. The relative contributions of allsources of funding should be clearly spelt out. c. The proportion provided by the govemment (FLHF and/ or districU LGA levels) shoutd be the major one by now (end of Year 5) or covering at least half of expenditure (end of Year 3). . Exarnination of: * Budget documents (government and NGDO) * Records of disbursement and expenditure (ledgers, orders, approvals for expenditure etc.) . lnterviews with: * DistricU LGA level managers (technical and adminislrative). * NGDO managers at this level. * FLHF team managers (leader, treasurer). * Local govemment managers at this level (chairperson, administrator. treasurer). Funds are not usually disburced from this level directly for CDTI activities. One FLHF uses funds generated from drug revolving scheme to defray costs for CDTI activities. ln one instance application for funds by a supervisor was approved by govemment and funds released. . The round ofthe vear before? Same as above . The round the vear before that? Same as above sustainability evaluation instrument no.3 - FLHF level- March 2003 vercion l0 - Are anv sleps beino taken to improve the situation? None Analvsis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . What is the trend in the relative proportion of resources contributed by the official health service? . Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable This indicator assesses whether the programme is resources rf needs. effectively, and whether it is able to 7.1 Check whether adequate and appropriate transport and materials are available for necessary CDTI activities at this level. Characteristics of the indicator Sources of information a. There are adequate numbers of appropriate, functional vehicles (of any type) available for necessary CDTI activities. b. The running costs for this transport are met by a dependable source (e.9. the govemment). c. There are sufficient materials available for training and HSAM. . lnspection of: * Each vehicle being used: its source; its functional status. * Stocks of materials for training and HSAM. . lnterviews with the FLHF management team. of the present vehicles and materials, considering the work still to be done in the coming 5-10 years: The transportation at this level is inadequate Adequacy for CDTI tasks** Not adequate, high cost of maintenance2 facilities had 1 each Not adequate Adequacy for CDTI tasks*Training/ HSAM material ls it working? ls there enough of it far the job? ls it suitable for the iob? * APOC, MoH, NGDO, other (specfi) lf transport and materials are inadequate and funded from non-qovemment sources: . Why is this? No reasons were given . Which steps are beinq taken to imorove the situation? Sustainability evaluation instrument no.3 - FLHF level- March 2003 version 11 None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable 7.2 Check whether transport at this level is adequately and appropriately maintained. This indicator assesses whether the progrcmme is functioning effectively and efficiently. Characteristics of the indicator Sources of information a. There is a routine maintenance schedule for vehicles (where relevant), which is adhered to and recorded. b. The costs for vehrcle and eguipment maintenance and repair are met by dependable sources (e.9. the government). c. Repairs are rapidly and efficiently done. d. Staff members have ways of coping when transport breaks down or is not available. so that CDTI activities are not disruoted. . lnspection of: * Vehicle and equipment maintenance schedules. * Vehicle and equipment maintenance records. . lnterviews with the FLHF management team. Sustainability evaluation insirument no.3 - FLHF level- March 2003 version Describe the present situation: There are no records of maintenance and maintenance costs is bome by the staff. There are no reliable altematives except to hire vehicles at high costs Make particular enquiries about the abilitv of the oovemment to oav for maintenance. reoairs and U Govemment has not been forth coming in maintenance and replacement of motorcycles lf the vehicles are not being well maintained. and/ or the oovemment is not payinq: . Why is this? No reasons were given. . Which steps are beino taken to imorove the situation? None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overalljudgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable 7.3 Check whether the transport available at this level is appropriately managed and used, in an integrated way. This indicator assesses whether the ffogramme is functioning efficiently. assesses whether the programme managprs are taking ownerchip of the programme, are able s Describe the present situation: There are no realistic plans to replace transport and materials Characteristics of the indicator Sources of information a. Trips made for CDTI purposes should be properly authorised in writing by the relevant official. Each trip undertaken should be recorded in a logbook. b. Transport provided for CDTI, and that provided for other programmes, should be combined as a pool to be used for legitimate activities of all programmes at this level. . lnspection of vehicle control documents: * Copies of trip authorities (also noting destination and purpose) * Log books. * The supervision plan/ matrix. . lnterviews with the FLHF management team. Describe the present situation: There is inadequate transport and where there are, no logbook or authorizations were found. lf the transport is not beino well manaoed: . Whv is this? No reasons were given . Which steps are beino taken to improve the situation? None Analvsis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable 7.4 Check whether there are appropriate and realistic plans for the replacement of transport and materials when APOC support comes to an end. to find resources for it. Characteristics of the indicator Sources of information a. Management should know that replacements will be needed before the end of the programme, and have specific, realistic plans to meet the need at that time. b. lt should be planned that the govemment will: * Provide replacements for vehicles. * Maintain existing vehicles. * Provide stationery and materials for training and HSAM. c. lf it is planned that replacement will be fmm non-govemment sources, written commitment for this should have been obtained at the highest level in these donor organisations (end of Year 5), or negotiations should have started (end of Year 3). . lnspection of letters of agreement. . lnterviews with: * The FLHF management team. * NGDO project managers. * High-ranking local govemment officials. Sustainability evaluation instrument no.3 - FLHF level - March 2003 version 13 lf the plans for replacing vehicles and materials are unsatisfactorv: . Whv is this? No reasons were given . Which steps are beinq taken to imorove the situation? None Analysis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affecl sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully Highly I Moderately I Slightly I Not at all I Not applicable 8.1 Check whether the team at this level is skilled and knowledgeable, regarding the implementation of CDTI in its arca of operation. This indicator assesses whether the gogramme fias been able to develop sufficient resources ffiffiB Characteristics of the indicator Sources of information a. Staff should have enough knowledge and skill to undertake all the key CDTI activities themselves: planning, training, HSAM, ensuring Mectizan supply, monitoring/ supervision. b. Staff at this level should remain in one post for at least five years. c. There should be immediate training (in CDTI) of new, unskilled project staff members who have CDTI responsibilities. . lnspection of: * Staff files. * Monitoring reports. * Activity reports. . lnterviews with * Managers and other staff at this level. * Staff at the districU LGA level. * Village leaders and CDDs. Describe the present situation: . Particulars of current slaff . lnformation about stability and in-service tra Staff members have enough knowledge and skills to undertake CDTI activities. Staff can be transfened at any time thus cannot be said to be stable. There are no plans for in service trainings Level of skill: is it adequate to perform the job? lf the staff at this level lack skills. and are often transfened: . WhY is this? Not Applicable . Which steos are beino taken to improve the situation? Sustainability evaluation instrument no.3 - FLHF level- March 2003 version l4 Not Applicable Analvsis . When writing the report you have to summarise: * The evidence about how well this indicator is being achieved. * Reasons for poor performance (if any), steps being taken to improve it, and how this is likely to affect sustainabilitv. . Your overall judgement: is this indicator for sustainability being achieved? Fully I Highly I Moderately I Slightly I Not at all I Not applicable {,;:.:''ffiftffid,i:i-iifaiffii*'ffirfrtie 9.1 Check whether the in the FLHF area is satisfactory. This indicator assesses whether the project is effecfive - if the rate is poor the project is clearly struggling, and less sustarnab/e. Characteristics of the indicator Sources of information a. All villages identified by the latest REMO should be under treatment (i.e. geographical coverage should be maintained at 100o/o). b. The rate should be stable or increasino. . lnspection of: * Dislribution reports and statistics at community level, forthis FLHF area, for the past 3 years. * REMO list of endemic communities forthis FLHF area. . lnterviews with staff at FLHF level. The oeoqraphic coveraoe situation: . At the last distribution: 100o/o . The year before: 1A0o/o . The vear before that: 10006 lf qeoqraohical coveraoe is poor: . Whv is this? Not Applicable . Are anv steps beino taken to imorove the situation? Not Applicable Analysis . When writing the report you have to summarise the reasons for poor performance (if any); steps being taken to improve it: and how this is likelv to affect sustainabilitv. . What is the trend in geographical coverage? . Your overall iudqement: is this indicator of sustainabilitv beinq achieved? Fully (100o/o of comnrunities are doing CDTI) Highly (95-99olo of communities - stable or increasino) Moderately (90-94o/o of communities - stable or increasino) Slightly (85-89% of communities - stable or increasino) Negligibly (<85o/o of communities) Sustainability evaluation instrument no.3 - FLHF level- March 2003 version