Evaluating the sustainability of CDTI projects: progress and prospects

JOINT ACTION FORUM Ollice of the Chairman JAF-FAC: EIGHTH SESSION Gatineau, 3-5 December, 2003 FORUM D'ACTION COMMUNE Bureau du Pr6sident t, I African Programme for Onchocerciasis Control Programme africain de lutte contre l'onchocercose EVALUATING THE SUSTAINABILITY OF CDTI PROJECTS: PROGRESS AT{D PROSPE CTS By Eleuther Tarimor Detlef Prozesky2 September 2003 I Dr Eleuther Tarimo, PHC & Health Services Expert WHO/APOC & WORLD BANK Consultant on Sustainability 2 Professor Detlef Prozesky, PHC & Community Health Expert WHO/APOC & WORLD BANK Consultant on Sustainabilitv. JAF 9.8 ORIGINAL: ENGLISH September 2003 TABLE OF CONTENTS Page EVALUATING THE SUSTAINABILITY OF CDTI PROJECTS: PROGRESS AND PROSPECTS... 1 INTRODUCTION: A BRIEF HISTORICAL OVERVIEW ............. 1 DEVELOPING THE PROCESS OF 'SUSTAINABILITY EVALUATION' OF APOC PROJECTS..2 FTNDTNGS OF THE EVALUATTONS (AS AT AUGUST 2003)........ ............. 3 BACKGROLIND TO THE EVALUATION FINDINGS ..................4 ON-SITE ACTIVITIES TO PROMOTE CDTI PROJECT SUSTAINABILITY.... .............5 POST-EVALUATION PROGRESS TOWARDS PROJECT SUSTAINABILITY .............6 THE WAy AHEAD.... .................7 JAr9.8 Page 1 EVALUATING THE SUSTAINABILITY OF CDTI PROJECTS: PROGRESS AND PROSPECTS The aim of this presentation is to give an overview of APOC's present drive to ensure that the projects it supports become sustainable, and continue to function after APOC support comes to an end. Introduction: a brief historical overview In 2000 APOC commissioned a Mid-term External Evaluation. The evaluating team identified the many accomplishments of APOC, but noted that it rvas 'deeply concerned about the sustainability of APOC related activity at all levels within projects and countries, once APOC funding comes to an end.' It further reported that'much core activity (training, supervision, NOTF meetings etc.) is directly funded by APOC. When the existing widespread salary top-ups are inevitably discontinued, decreases in motivation and productivity are likely to occur. There is as yet little definitive planning to mobilise the resources and set in place the routines which will ensure the continuation of the projects.' In response, APOC Management in 2001 commissioned two consultants to start working on this problem, with the following terms of reference:(i) To develop a simple framework with realistic and measurable indicators of sustainability.(ii) To evaluate the curent situation of seven CDTI proj ects in twelve districts/states which would be completing their fifth year of APOC support by Decemb er 2002.(iii) To assist national managers to develop plans for achieving sustainability for these projects, based on careful prioritising, and more careful use of ApOC funds. (iv) To prepare a short manual for the evaluations, using the frarnework developed. In consultation with APOC management, and with the assistance of a core group of experts from APOC member countries, these tasks were systemattcally tackled. The report of the Extemal Mid-term Evaluation of APOC (2000) considers that 'an activity is sustainable if the necessary political decisions, administrative measures and resources (finaniial, human, material) are in place, and can reasonably be expected to continue being so for the foreseeable future.' In the present context 'sustainability' means the likelihood that a project will continue to function effectively after APOC support comes to an end. Here two related concepts that were taken into consideration: Self-suffic'iertc1,: The ability of a pro.jecl to continue firncticlning cffcctivcly, using only rcsoLrrccs scncratccl u,ithin thc country itsell. S u s ts i rt uh il itl:' 7y" ab i lity o1- a pro.j cct to coltin rr c lirnctio; i pg cff-cctivcly, usir-rg both thc counlry's oil'rr rcsourccs ancl thosc lrclnr outsic'lc' - pror,'ided both. in parlrculiLr qovenrnrcnt resoLu'ces are clepcndable . In this exercise the evaluating teams used the definition of project "sustainability'as our guiding concept. JAF9.8 Page 2 Developing the process of 'sustainability evaluation' of APOC projects The first set of evaluation instruments was tested during a pilot evaluation of the Kaduna CDTI project in Nigeria. This was followed by furlher evaluations, with regular reflections on the evaluation process. The evaluation process presently in use is illustrated in the diagram below: Judgement of sustainability Community ownership For each indicator For groups of indicators For the whole project4 Effectiveness Staff acceptance Resources lntegration Simplicity Efficiency validated instruments for 4 levels CDTI projects use resources to undertake specific activities, leading to results - at different levels (usually nationally/ regionally; health district; first line health facility; and community). Particular aspects of these activities, resources and results are selected as indicators, since they give information about sustainability at each level. Instruments are designed for each level, which collect information about each indicator, so that evaluators arrive at a clear understanding of the seven aspects of CDTI project sustainability: efficiency, integration, resources, health staff acceptance (Attitude of health staff), effectiveness, community ownership and simplicity. Judgements can then be rnade of contributions to sustainability in the case of individual indicators, or of groups of indicators, or of the project as a whole. Guidelines for the use of the instruments have been developed and revised. It covers the following: planning the evaluation; data collection in the lield; analysing the data; writing the report; and managing the feedback/ planning events afterwards. The instruments and guidelines are constantly being fine-tuned, with the experience gained from evaluations in different countries. Training of evaluation teams has been done'or.r the job'. Teams are usuallymade up of six members, and should contain two experienced evaluators, who teach ner,v members what to do at the start of each evaluation. A workshop was held in February 2003 in Ouagadougou to train francophone evaluators, and to validate the French translation of the instruments and guideline. Both English and French versions are currently in use in the field. All team members working together make judgements about the degree to which individual indicators for sustainability are being achieved. These judgements form the basis for recommendations, and also for making an overall judgement of a project's progress towards sustainability. Various models have been tried to arrive at a 'sustainability score', e.g. by differential weighting of indicators, but this process has not yet been finalised. Since 'project sustainability' has been found to be a complex, the ' GDTI project ,'i R"tlrities (prannins, leadership, training, supervision, drug supply) Resources (financiat, material, human) Results (coverage) at 4 or more levelslndicators of sustainability JAFg.8 Page 3 details which lead to specific and sensible recommendations on all fronts seem more important than overall judgements at this stage. Findings of the evaluations (as at August 2003) The process of conducting sustainability evaluations is continuing more or less on schedule, the process being expertly supported by APOC management. To date 1.7 projects have been evaluated during their fifth year of APOC funding, and 8 during their third year. Overall 'sustainability judgements' were made of these twenty-five (25) projects evaluated. Of these 20 were judged to be making satisfactory progress towards sustainability, whereas five (5) were not. A clear pattern ltas emerged of areas of success and weakness (a) Concenring the four levels of operation, the community level was generally performing sustainably. Once communities understand the benefits of the treatment they organise themselves to carry it out, even though there is some attrition of community distributors. The problems encountered in the higher levels (which all involve health care health staff) are discussed below. (b) The seven aspects of sustainability: (i) 'Community ownership' generally scored well - this is in line with the finding above, and reflects the hard w,ork that projects have put into mobilisation, leading to community empowerment. (ii) 'Effectiveness' (of planning, training, Mectizan supply, supervision and coverage) also scored well - and effective projects are more likely to be sustainable. (iii) 'Simplicity' too scored highly: because the strategy is simple and feasible, the routines that the CDTI projects have established are generally easy to carry out. (ir) 'Health staff ucceptance (Attitude of the health staff)' of the new programme did not score quite so well: at district and first line health facility level CDTI is an additional burden to poorly paid health staff members that already have many other responsibilities. The three ng aspects were howev er more problematical. (u) 'Efficiency'did not score well because mobilisation, training and supervision by the health staff do not taper off as they should, and salary top-ups and other financial incentives (such as (per diem allowances) are widely used. (ui) 'Integration' suffered, since many projects started off before the launching of the CDTI strategy relatively vertically (needing specific mobilisation, training, monitoring etc.) - and some have not yet fully made the transition to becoming routine programmes, like any other in the life of the health districts. (vii) 'Resources' finally were a mixed bag. Hurnan resources were generally available and skilled (although also motivated by various financial incentives). On the other hand govemment at different levels has often been slorv in contributing its share of financial JAF9.8 Page 4 and material resources (e.g. transport), so that many key CDTI activities (training, mobilisation, supervision) still depend to a large extent on APOC and NGDO funding. This showed a worrying lack of political commitment in many situations, at the higher levels of government. Although these weaknesses exist they are by no means universal - there have been frequent examples where efficiency and integration are improving, and resources provided by government at various levels. Background to the evaluation findings It is important at this stage to consider the CDTI projects' efforts at achieving sustainability, in the context of general trends in the health services of member countries: (i) Before and duringthe'PHC period'great strides were made in improving service access to populations: more facilities, more health staff, more services at all levels. Around 1990 however this changed to a phase characterized by loss of confidence, shortage of resources, poor management of resources, dilapidated health facilities, scarcity of drugs and demoralized health staff. Earlier achievernents in integration have been reversed. Major underlying reasons for these changes have been greatly decreased budgets and donor factors (pressure for tangible results tending to encourage support to verlical programmes). A serious obstacle to the integration of services is the 'box synclrome' of vertical programmes: 'This is the package, take it or leave it.'Budgets promote vertical, unintegrated activity (training, supervision etc.) - and yet expect the programmes somehow to become integrated into mainstream activity by the time funding is withdrawn. People involved in such projects take as their standard Year 1 funding. By Year 5 it should be much less - this is when Govemment should be contributing a good proportion, but there is a problem of mindset: those involved appear unable to liberate themselves from the 'box' and previous levels of budget. (i i) (iii) Together these factors have resulted in a situation where findings from ongoing assessments of quality of health care from more than one member country shor.v clearly that programmes that are of good quality are those receiving extemal support - like EPI, reproductive health, TB/leprosy, t'nalaria, and HIViAIDS. Programlnes that do not receive such support, like clinical services, environmental health and mental health, are moribund. (iv) Governments have realized that given the srnall salaries, implementation of activities is likely to be minimal. Many health staff members turn to bribes over or under the table, or have to spend considerable tirne each day on a second job, to make ends meet. A recent study (on quality of care) in one country revealed that programmes that are doir-rg well are those that provide some fonn of incentive. This being the case, it would be unwise to ignore or wish away the issue of incentives. The problems they raise though are significant. Even if they were to be used rationally and transparently, the fact is that donors fund them, which is not sustainable. (v) At community level there are other dangers. Some other prograrlmes provide direct remuneration to 'volunteers', thereby bypassing community decision-making. Different programmes provide different incentives; managers may insist on vertical programmes when communities are known to prefer holistic ones which are easier to use. JAF9.8 Page 5 (vi) On the positive side, international partners have decided to support the health sector in a number of countries through the mechanism of 'basket' funding - a move from supporting individual programmes to more integrated donor support. This move is to be welcomed since it fosters integrated health service development. (vii) In order to deal with the lack of motivatior.r caused by extremely low salaries, Ministries of Health in many countries have introduced different types of incentive. An example is the Selective Accelerated Salary Enhancement Scheme (SASE), which has the intention of remunerating a selected number of health staff at all levels based on their performance. There are also proven quality improvement systems, lvhich programmes like CDTI couldjoin: 'Quality Improvement and Recognition Initiative' (QIRI) supported by USAID, and 'Comprehensive Quality Management' (CQM) supported by GTZ. ht eaclt country tlterefore APOC has to operate within a unique and complex system. Some member countries have well developed peripheral health systems and others not; some are experiencing internal conflict; yet others have been labelled as'failed states'. As has been pointed out above all share some structural and organisational weaknesses, which one programme cannot be expected to remedy all at once. The future of CDTI has to be planned by embracing these realities and working within them. This process has already begun, for example by the decision to continue support (at a much reduced level) for up to three additional years, for projects which for various reasons have not yet achieved, but are judged to be making progress towards sustainability. On-site activities to promote CDTI project sustainability In accordance with the third article of the evaluation team's Terms of Reference, evaluation teams have routinely undertaken activities to promote project sLrstainability. The most important of these have been workshops or meetings, where project stakeholders and decision makers are brought together to receive feedback on the evaluation team's findings. This is followed by a planning exercise, where the stakeholders together make a 'sustainable plan' for the coming yearl years. This process involves identifying the resources that are likely to be available to the project for the coming year; scaling down the routine activities of past years so that they can be accommodated within this budget. In the process, priority activities have had to be carefully identified; additional or alternative sources of funding and materials sought; and ways found of integrating CDTI into existing routines, and ofsharing scarce resources such as transport. These events have been arranged for three different levels, depending on the circumstances: the national level, the project level (e.g. the State and Provincial levels in Nigeria and Cameroon respectively), and the health district level. The plans that have been developed at these workshops have varied somewhat in quality, but all projects, and in most cases every district within them, have prepared at least one plan. During the approximately two weeks that it takes to complete the evaluation, the team members also undertake as many advocacy visits as possible. These are made to community leaders and members, political heads and senior health service and NGDO marlagers (in the areas served by the projects, as well as nationally) and also to stakeholders such as the WHO Representative (WR) for the country. The team members point out the very real achievements of the CDTI projects, and the need for sufficient future resources for them to be able to continue functioning. At national level managers are reminded of the undertaking they made in the original Letters of Agreement, to provide increasing amounts of resources so that sustainability would have been achievecl by Year 5 of APOC funding. Future technical support for the CDTI programme is discussed rvith the WR's office. JAF9.8 Page 6 It is clear that these activities should only be seen as a step (in some cases a first step) in the difficult process of achieving project sustair-rability. The difficult part remains: to develop the 'sustainability plans' further, and to implement them fully, right to the situation where each project is really as sustainable as possible. Post-evaluation progress towards project sustainability Upon departure of the evaluation teams it is then left to the CDTI project management (government and NGDO) first to complete the development process of the sustainability plans and then to ensure implementation of these plans. APOC Management is able to monitor the process through its routine mechanisms, such as receiving progress reports as well as new plans and requests for support. Tltere ltave been botlt successes and challenges in understanding the process of developittg sustainable plans with measurable targets. The serious attention that was given to sustainable planning at the workshops is a major achievement in itself - as is the resulting existence of numbers of plans to achieve sustainability. Other successes have includecl thefollowing: (i) Activities such as training, n-robilisation and supervision have been more carefully targeted and to be carried out by smaller nuntbers of health personnel. (ii) Government at different levels has committed increased level of budgetary support to the programme - and in some cases the actual disbursement of the budgeted amounts were made immediately after the planning rvorkshops. (iii) NGDO partners have clarified the nature, extent and duration of the supporl they will be able to provide in future. On tlte otlter hand tlrere have been challenges:. (iv) Projects evaluated have rather been slow to submit sustainability plans. (v) in spite of the evaluation process, project managers are still wedded to the 'box' and have not yet changed their mindset to embrace sustainability. This is seen lvhen, APOC receives requests for support for Year 6 which are sirnply a continuation of those for previous years. (vi) Most of the issues raised by the evaluators are yet to be properly addressed by the projects. APOC management hus responded to such sitttqtiorts irt the following ways: (vii) It has developed and circulated clear guidelines to projects, on how to develop sustainable plans. (viii) It has adapted the protocol for the external monitoring visits that new projects undergo, so that these focus strongly on the issue of sustainability. (ix) The plans and budgets that APOC receives for support after Year 5 are carefully scrutinised, and allocations made in such a way that projects are strongly guided into efficient and integrated activity. This is in line with the decisions taken by the countries themselves when they met in Abuja, Nigeria in June 2002 to discuss the results and recommendations of the first sustainability evaluations. aJAF9.8 Page 7 (x) High-powered advocacy teams have been dispatched to member countries and to individual projects, to lobbygovernment at different levels to make good its promises of financial and material support. (xi) Repeat workshops have been arranged, where administrative and political authorities and project managers revisit the plans they rnade during and after the evaluation, refine them, and re-commit themselves to their implementatiorr. It is clear that such support to projects needs to continue and even to be intensified, as they struggle to achieve sustainability. In the process all stakeholders, frorn APOC and its supporters to national programme managers to project health staff, are gaining in experience about how this should be done - and a body of generic experience is being built up. The way ahead APOC finds itself at a critical juncture, with respect to the sustainability of the projects it has been supporting. It has taken great strides in understanding a complex phenomenon; in evaluating project sustainability; and in moving projects forward on the path to achieving it. But it is also clear that there is still some way to go. APOC has two important cards to play for enhancing sustainability: the evaluation process it has developed, and advocacy to leaders and decision-makers. A good understanding of the context and predicament of countries is essential for evolving appropriate tactics for playing the cards. Admittedly this approach is cumbersome and uncharted ground - but without such understanding it is likely to be business as usual, leading to unrealistic demands being made and possible project failure. CDTI is a gold mine with a lot of potential still to be exploited, but realism is needed on the issue of sustainability We would like to present the following issues to the partners at this meeting, as they consider the future and the sustainability of the CDTI projects they have been sr-rpporting: (i) Is 'basket funding' considered a sustainable mechanism for funding CDTI? If so partners should take measures to ensure that CDTI is included iu basket funding (in all countries where such a mechanism has been established). (ii) Should CDTI be included under incentive schemes such as SASE? What about its inclusion in QIRI and CQM or other relevant quality improvement initiatives? (iii) Given that CDTI sustainability will be enhanced by rewards, are there specific sustainable suggestions emerging from the experiences of partners that should be considered? (iv) Is it possible for the CDTI 'box' to be n:rodified earlier on in the process of project development (e.g. in the letter of agreement) so that issues like integration, efficiency and government resource allocation are attended to sooner? And in the same breath, would partners see pragmatic approaches to establishing levels of 6'h year 'sustainability budgets' as worth trying out? (for projects that meet required conditions). (v) How can partners ensure that the basic concepts of CDTI are widely known particr.rlarly after changes in leadership (for example by including up-to-date material on CDTi in the basic and ongoing training of health workers)? (vi) Is it possible for APOC to take a lead in advocating integrated health care at the community level? A number of programmes have been involving community members in their JAF9.8 Page 8 activities, but their approach may be top-down and laden with unsustainable incentives - which is both inferior to, and a danger to, APOC's 'community directed' approach. Partners may feel that taking the lead in this way does not fall within APOC's mandate -but unless they are pro-active for CDTI, the concept may well be hijacked and comrpted, j eopardizing sustainability. (vii) Can partners spearhead and support specific research to support sustainability? Two areas of research specifically commend themselves: community level activity (e.g. the organisation of community volunteers and other resources; the integration of services at community level; attrition of CDDs) and the process of achieving sustainability (e.g. the reasons why 'sustainability plans' are not implemented; mechanisms whereby government support could be enhanced; sustainable reward systems). APOC and the projects it is supporting liave come a long way in dealing with the issue of sustainability, but much remains to be done. It is irnportant to recognise here that each of the partners in this great movement has to play its role in achieving project sr-rstainability: (i) APOC management can only do so much. It can monitor and supporl each project as it attempts to reach sustainability; it can coordinate the research mentioned above; it can take the lead in advocating an integrated approach at community level. It cannot however continue to supply resources - nor, critically, can it force other paftners to do so. (ii) Government, the national Ministry of Health, has the inevitable responsibility of supplying resources (including transport), consistently and at a sufficient level to keep th; CDTI programme running. It is also finally responsible for monitoring, supervision and training. It has to take and implement decisions on issues like basket funding, quality improvement, and sustainable health staff incentives. These are responsibilities that only Government can undertake, and failure to do so will critically endanger project sustainability. (iii) Communities have demonstrated their willingness and ability to play their role in the projects - but they too need support, and the Mectizan needs to continue reaching them. (iu) The NGDO partners can continue their commitment to each project, through moral and technical support, and possibly filling ir.r small resource gaps and encouragirrg decentralization of program management to districts/LGAs, and with more emphasii on greater involvement of local NGOs. (v) The WR's office in each country can provide useful and consistent technical and moral support to each national CDTI programme. It is through hard work and partnership that APOC has achieved so much in such a short time - alleviating much suffering, and at the same time providing the world with an example of a successful multi-national disease control programme. Ensuring the sustainability of th.r. achievements will require the full commitment of each partner to its agreed role. t