Increasing access to health workers in underserved areas: a conceptual framework for measuring results

357 Luis Huicho et al. Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920 Introduction Health workers’ willingness to practise in underserved areas, such as rural, remote or poor areas, is a recognized challenge in achieving equitable access to health services. Many countries have developed strategies to attract and retain qualified health workers in these areas. But the evidence on the successes or fail- ures of such interventions is scarce and weak, and it is difficult to compare lessons and measure results from the few evaluations that are available.1–3 There has been significant progress in generating common understanding and debate on ways to evaluate the impact of development interventions, and on using consistent monitoring and evaluation terminology, such as “outputs”, “outcomes” and “impact”.4–6 However, this has yet to be applied to the evaluation of human resources for health interventions and, specifically, to those seeking to increase access to health workers in under- served areas. There is an urgent need to achieve an agreement and strengthen the evidence that would underpin sound policy recommendations in this area. Seeking to address this gap, this paper proposes a conceptual framework to guide managers, policy-makers and evaluators in the assessment of interventions to increase access to health workers in underserved areas. The framework aims to support all stages of policy development. It suggests a logical sequence that can be followed when deciding on any intervention that ad- dresses attraction and retention issues and assists in formulating key questions to be answered when designing, monitoring and evaluating such interventions. It includes examples of indicators to inform the process, which can be adapted to a specific context. A comprehensive consideration of indicators to monitor the health workforce in general is offered elsewhere.7 Challenges in evaluation Policy- and decision-makers need to know whether interventions work or not, why they work and in which context. Therefore it is important to have information about the effects of the inter- ventions, but also about the factors that made the intervention succeed or fail (the questions dealing with “when, why, how, and in what circumstances such interventions work well or fail to work”).8 Evaluating interventions to improve human resources for health is complex for different reasons. First, relating these interventions to health status is very difficult due to broad socioeconomic, cultural, political and health systems factors that influence health. For instance, although improved health outcomes, such as reduction in maternal mortality, are directly correlated with increased availability of health workers, it is dif- ficult to attribute the improvement directly to a certain health workforce intervention.9,10 Illustrative country case studies showed that Afghanistan and Ethiopia have implemented Une traduction en français de ce résumé figure à la fin de l’article. Al final del artículo se facilita una traducción al español. .ةلاقلما هذهل لماكلا صنلا ةياهن في ةصلاخلا هذهل ةيبرعلا ةمجترلا Abstract Many countries have developed strategies to attract and retain qualified health workers in underserved areas, but there is only scarce and weak evidence on their successes or failures. It is difficult to compare lessons and measure results from the few evaluations that are available. Evaluation faces several challenges, including the heterogeneity of the terminology, the complexity of the interventions, the difficulty of assessing the influence of contextual factors, the lack of baseline information, and the need for multi- method and multi-disciplinary approaches for monitoring and evaluation. Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluating interventions for human resources for health. This paper proposes a conceptual framework that offers a model for monitoring and evaluation of retention interventions taking into account such challenges. The conceptual framework is based on a systems approach and aims to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its design phase through to its results. It also aims to guide the monitoring of interventions through the routine collection of a set of indicators, applicable to the specific context. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of the interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It promotes the use of a common understanding on how attraction and retention interventions work, using a systems perspective. Increasing access to health workers in underserved areas: a conceptual framework for measuring results Luis Huicho,a Marjolein Dieleman,b James Campbell,c Laurence Codjia,d Dina Balabanova,e Gilles Dussaultf & Carmen Doleag a Department of Paediatrics, Universidad Peruana Cayetano Heredia, Lima, Peru. b KIT Development, Policy and Practice, Royal Tropical Institute, Amsterdam, Netherlands. c Instituto de Cooperación Social, Integrare, Barcelona, Spain. d Global Health Workforce Alliance, Geneva, Switzerland. e London School of Hygiene and Tropical Medicine, London, England. f Intituto de Higiene e Medecina Tropical, Universidade Nova de Lisboa, Lisboa, Portugal. g Department of Human Resources for Health, World Health Organization, 20 avenue Appia, 1211 Geneva 27, Switzerland. Correspondence to Carmen Dolea (e-mail: doleac@who.int). (Submitted: 5 October 2009 – Revised version received: 19 November 2009 – Accepted: 5 January 2010 ) Policy & practice Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920358 Luis Huicho et al.Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas comprehensive health sector strategies with multiple co-existing interventions, like the recruitment of community health workers, which may have promoted access to life-saving services, thus improving health status.11 Another problem is that many evalu- ations lack a baseline against which to assess the results, particularly in countries with a major health worker deficit,3,10,12 as well as a specific intervention logic that clarifies the expectations of the inter- vention designers (S Kane, B Gerretsen, R Scherpbier, M Dal Poz, MA Dieleman, unpublished data, 2009).13 Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluation of human resource interventions.14,15 Thus, the main challenges which evaluators face are related to the multidi- mensional nature of interventions, and the difficulty of assessing the influence of contextual factors. To be able to assess such interventions in complex systems, there is a need for multimethod and mul- tidisciplinary monitoring and evaluation approaches, inclusive of all relevant stake- holders.16 Another challenge is generat- ing lessons across regions and countries to inform the global health agenda and promote a meaningful dialogue at both national and international level. There is clear need for a framework that offers a model for monitoring and evaluation of health worker retention interventions taking into account such challenges. Conceptual framework The proposed framework is based on a systems approach and differentiates between “inputs”, “outputs”, “outcomes” and “impact”, with regards to the results of interventions to attract and retain health workers in underserved areas. It builds on the proposed common framework for monitoring performance and evaluating progress in the scale-up for better health,17 and proposes indicators to measure prog- ress in implementing various strategies, allowing users to determine what works or not, and to explore contextual factors influencing their success or failure. The framework has two aims: (i) to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its inception/ design phase through to its results, by sug- gesting key questions about the relevance, efficacy, efficiency, effectiveness and sus- tainability of the intervention; and (ii) to guide the monitoring of interventions, through a focus on a routine collection of a set of indicators, applicable to the specific context. We used the following definitions for evaluation and monitoring:4 We define “evaluation” as the system- atic and objective assessment of an on- going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. We define “monitoring” as a con- tinuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing develop- ment intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Later in this paper, we examine the application of the framework to the stages of policy development: design, implementation, monitoring and evalu- ation (Fig. 1). The sequence in design and implementation phases does not necessarily imply a linear policy process, as monitoring and evaluation may help identify a need for new interventions. They are, in effect, part of an ongoing policy cycle. Design The evaluation of the design stage should identify the extent to which the chosen intervention was relevant and adequate to the need, the expectations of the popula- tion and the health system context, and the reasons for selecting a particular op- tion.18 By relevance, we mean the extent to which the objectives and the elements of the intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.4 The relevance issue can and should also be addressed retrospec- tively; and then the question is whether the objectives of an intervention or its design were and are still appropriate given changed circumstances.4 Questions that an evaluator should ask are: did the intervention respond to a documented need, was it appropriate for the given context and how was it selected? Is the choice of the intervention based on evidence or at least on robust arguments? Fig. 1. The conceptual framework for measuring efforts to increase access to health workers in underserved areas Design Situation analysis Labour market Organization and management capacity Regulatory systems Resources needs Criteria for choosing interventions Feasibility analysis Di m en si on s - Total graduates - Total health workers - Budget for human resources for health strategy/plans In di ca to rs (e xa m pl es ) Implemen- tation Interventions Education Regulatory Financial incentives Management and social support - Policies on education and recruitment - Career pathways - Regulatory frame- works - Type/costs of incentives Attractiveness Intentions to come, stay, leave - Intention to stay/leave - Number of health workers recruited - Funded positions - Stability index - “Survival” rates Outputs Engagement Deployment Effective contracting and posting Retention Duration in service Job satisfaction Workforce performance Availability Competence Productivity Responsiveness - Staff ratios - Waiting lists - Absence rates - Coverage rates patient satisfaction Outcomes Accessibility Coverage of interventions Productivity Service utilization Responsiveness Patient satisfaction Improved performance health service delivery - Millennium Development Goal indicators - Health status - MMR / IMR Impact contributing to Improved health status Context: Social determinants, political situation, stakeholders power and interests, economic issues (fiscal space, fiscal decentralization), individual level factors (marital status, gender) IMR, infant mortality rate; MMR, maternal mortality rate. Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920 359 Luis Huicho et al. Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas The assessment of the situation analysis should check if the problem of limited access to health workers has been clearly described. Is it due to an insuffi- cient number of students willing to work in underserved areas, or an insufficient production of health workers, or were there other reasons? The evaluation needs to assess if a health labour market analysis has been conducted, to identify whether the lack of access to health workers in underserved areas was a matter of insuf- ficient demand, insufficient supply, or insufficient remuneration and incentives, and unattractive working conditions.19 The situation analysis should also identify whether the elements in place to support the implementation of the intervention were assessed and if a feasi- bility analysis preceded the choice of the intervention. Strategies to increase access to health workers in underserved areas require participation of stakeholders from different sectors. Therefore, questions to be asked include: were mechanisms for multi-stakeholder engagement in place and used, were regulatory frameworks analysed before the intervention, were the intervention costs calculated and considered affordable (availability of funds), was the organizational capacity to support its implementation analysed? At this stage, the Human Resources for Health Action Framework is a recognized tool that can be used to guide the situa- tion analysis (Fig. 2).21 It comprises health system dimensions such as policy, finance, education, partnership, leadership and human resources management systems. At the end of this phase, a strategy can be proposed to address the perceived need of lack of access to health workers in underserved areas and of maldistribution of health workers. Such a strategy can be in one or more of the following thematic areas: education, regulatory, financial, and/or management and support, de- scribed elsewhere.3 Implementation When the implementation of an inter- vention is evaluated, questions might include: did the activities take place ac- cording to plan, did any changes occur in implementation capacity or in the context that influenced the implementation of the intervention components? Did any changes occur regarding the involvement and commitment of stakeholders? Which ones and why? Was any action undertaken to adapt the intervention to the changes? What was done and why? Or why was no action taken? Monitoring the progress in implementation can be informed by the systematic collection and analysis of a set of key indicators, which are further discussed in this paper. Results Intervention results can be evaluated at different levels (Fig. 1): outputs, interme- diate outcomes, outcomes and impact. Questions to be asked include: did the intervention achieve its intended results at the level of outputs, outcomes, or impact? Is the intervention sustainable? What were the contextual factors (political, economic, social, or organizational) that have influ- enced the results? The level of “outputs” is the direct ef- fect, attributable to the intervention, such as increased attractiveness of the areas/or the profession, improved recruitment and deployment, and increased retention of health workers in those areas. The level of intermediate “outcomes” is that of improved health workforce performance, the dimensions of which have been described in The world health report 2006 as “availability, competence, responsiveness and productivity”.10 Some attraction and retention interventions can have a direct effect on these dimensions too. For example, changes in rural curricula can improve competencies to address rural health problems and performance-based financial incentives can improve productiv- ity. However, often improved health work- force performance cannot be attributed to retention interventions alone, as there are many other factors that contribute, such as education strategies or health sector management reforms. The level of “outcomes” is that of increased health systems performance, measured by increased coverage and utilization of services, and improved re- sponsiveness. Improved health workforce performance could contribute to improved health system performance but a host of other factors can contribute as well, such as the availability of drugs, infrastructure and equipment.22 Finally, the level of “impact” is understood as the effect on health out- comes and an improved health status, and Fig. 2. The adapted Human Resources for Health Action Framework20 Country specific context including labour market Critical success factors Improved health workforce outcomes Better health services Equity Effectiveness Efficiency Accessibility Implement- ation Other health system components Leadership Policy Finance EducationPartnership Preparation and planning Situation analysis Monitoring and evaluation Better health outcomes HRMS HRMS, human resources management systems. Table 1. Dimensions and indicators to assess health workforce performance11 Dimension Indicators Availability Staff ratios Absence rates Waiting times Competence Individual prescribing practices Institutional level: readmission rates; live births; cross-infections Responsiveness Patient satisfaction Assessment of responsiveness Productivity Occupied beds Outpatient visits Interventions delivered per worker or facility Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920360 Luis Huicho et al.Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas other unexpected effects on underserved communities (e.g. attraction/retention of other professionals or services). This cannot be directly attributed to retention interventions, as the simple availability of health workers cannot directly improve health outcomes in the absence of effec- tive components such as drug supply, functioning facilities and good clinical practices.9 But a well performing health workforce is a significant component of a strong health system that, together with other social determinants, can lead to improved health status.22 Evaluations of any attraction and retention intervention need to consider the contextual factors that may explain why and how such interventions have achieved the intended outputs, outcomes and impact, i.e. the social, economic and political context. This includes a stake- holder analysis including their respective power bases and interests, as well as an analysis of the capacity of the health sec- tor to implement. Evaluations should use both qualitative and quantitative methods to measure outputs and impacts, but also to analyse formal and informal policy pro- cesses accompanying the design and imple- mentation of interventions. Ideally, to find out if an attraction and retention interven- tion works or not, a cluster randomized controlled trial with an adequate control group would be needed, but this may not be feasible or ethical in the “real life” policy scenario. Moreover, causal chains are often complex in public health interventions, making results of randomized controlled trials subject to effect modification in different populations.20 Alternative obser- vational methods such as before-after and plausibility designs can also be considered, taking care to document and control the contextual factors appropriately.20,23 Stud- ies with adequacy and plausibility designs can improve both the internal and external validity of randomized controlled trials, and plausibility studies are often the only feasible option for getting valid evidence of impact when attempting the evaluation of large-scale interventions.23 Monitoring The framework depicted in Fig. 1 proposes a tool to guide the monitoring of interven- tions to increase access to health workers in underserved areas. At each stage of the implementation process, indicators are proposed to assess whether the implementa- tion is on track. Indicators to monitoring at the design stage should be those that will allow for a detailed description of the stocks and flows of health workers across sectors, age and sex categories, geographical areas and professions. Detailed description of these indicators, as well as data sources for health workforce monitoring, are described elsewhere.7 When monitoring the implementation of interventions, systematic data collection is required on the planned activities, by con- stantly reviewing the production capacity, as well as the organization and management capacity. Monitoring implementation also requires tracking changes in context and stakeholders, so as to be able to adapt the interventions when required. Regarding the monitoring of results of the intervention, at the level of “outputs” it is important to monitor whether the di- mensions of “attractiveness”, “engagement/ recruitment”, “deployment” and “reten- tion” are achieved. Indicators to monitor progress on these dimensions can include, for example, changes in the intentions of students or health workers to relocate, reside in or leave underserved areas (attrac- tiveness). The available funded positions in underserved areas can be an indicator of effective engagement, whereas the number of health workers recruited and changes in vacancy rates can be indicators of effective deployment and recruitment. Finally, the effects of an intervention on retention can be measured by duration in post, turnover, absenteeism, stability index or even survival rates.24,25 At the level of “outcomes”, it is pro- posed to monitor the progress on improved health workforce performance and health systems performance, through dimensions and indicators proposed by The world health report 2006 (Table 1).10 As said before, as- sessing the longer-term impact of human resource interventions on health outcomes should be done within the broader context of factors influencing health status. The effectiveness of attraction and retention interventions can only be mea- sured at the level of outputs and, to a certain extent, at the level of outcomes. In addition, implementing human resource retention interventions can take many years before any results can be seen, for example in the case of creating new schools in rural areas or rolling out financial incentive schemes. Table 2. Proposed questions and indicators for the evaluation and monitoring of interventions to increase access to health workers in underserved areas Stage Questions Indicators Methods Design • Did the intervention respond to a documented need? • Is the choice of the intervention based on evidence or robust arguments (situation analysis)? • Were the actual reasons for limited access to health workers clearly documented (i.e. insufficient number of students willing to work in rural areas or insufficient number of funded positions? • Human resources for health-costed plan, including situation analysis • Stocks and flows of health workers (for more details see reference)8 • Labour market analysis • Survey of intentions • Demographic analysis (health workforce stocks and flows) • Stakeholder analysis • Review of policy documents Implementation • Did the activities take place according to plan? • Did any changes occur during implementation? • Process indicators (changes in the programme elements) • Surveys • Stakeholder analysis • Review of policy documents Results • Did attractiveness of profession/rural/remote areas improve? • Did recruitment of health workers in underserved areas improve? • Did retention improve? • Total health workers recruited • Changes in preferences for rural/ remote areas • Reduced turnover • Reduced vacancies • Stability index • Survival rates (or years in post) • Surveys • Survival analysis • Analysis of registries data or facility data Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920 361 Luis Huicho et al. Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas ءاقبتساو بذج لىإ فدهت تايجيتاترسا دادعإب نادلبلا نم ديدعلا تماق نأ يرغ ،تامدخلا نم ةمورحلما قطانلما في ينلهؤلما ينيحصلا ينلماعلا ةردان تانيب يه تايجيتاترسلاا هذه لشف وأ حاجن ىدم حضوت يتلا تانيبلا ةقثبنلما جئاتنلا سايقو ةدافتسلما سوردلا ةنراقم بعصلا نمو .ةفيعضو اهنم ،ةيرثك تايدحت هجاوت مييقتلا ةيلمعف .ةرفاوتلما ةليلقلا تماييقتلا نع ةنراقلما لماوعلل ام مييقت ةبوعصو ،تلاخدلما دقعتو ،تاحلطصلما ةيرياغت قئارطلا ةددعتم بيلاسأ لىإ ةجاحلاو ،ةيساسلأا تامولعلما صقنو ،يرثأت نم سيايسلاو يعماتجلاا ىوتحلما نإف كلذ لىع ةولاعو .مييقتلاو دصرلل مظنلاو في ذخؤي ام ًاردان تلاخدلما هذه هللاخ نم تذفنو تممص يتلا يداصتقلااو في ةلماعلا ةيشربلا دراولما تلاخدم مييقتو دصرب ةصاخلا تايلمعلا في رابتعلاا دصرلل ًاجذونم مدقي يميهافم لمع راطإ ةقرولا هذه حترقتو .يحصلا لاجلما هذه لثم عضو عم ينلماعلا ءاقبتساب ةينعلما تلاخدلما لجأ نم مييقتلاو ماك ،مظنلل بولسأ لىع يميهافلما راطلإا زكتريو .ةساردلا ديق تايدحتلا لوصولا ةبسن ةدايز لجأ نم ةلخادلما مييقت وحن يركفتلا هيجوت لىإ فدهي ةلحرم نم ًاءدب ،تامدخلا نم ةمورحلما قطانلما في ينيحصلا ينلماعلا لىإ تلاخدلما دصر هيجوت لىع ةولاع .جئاتنلا صلاختسا ةلحرم ىتحو ميمصتلا راطإ في اهقيبطت نكيم يتلا ،تاشرؤلما نم ةعومجلم ينيتورلا عمجلا للاخ نم دنع لماشلا بولسلأا مادختسا لىإ ةجاحلا ًاضيأ راطلإا حضويو .ينعم ىوتحم راطإ نوكي نأ دصقلا سيلو .اهمييقتو ،اهدصرو ،اهذيفنتو ،تلاخدلما ميمصت .ةدح لىع دلب لك قاطن في قبطي ثيحب ًانرم نوكي نأ لب ًادمجتم لمعلا ةينعلما تلاخدلما لمع ةيفيك لوح عئاشلا موهفلما مادختسا لىإ وعدي وهو .مظنلا روظنم مادختساب ءاقبتسلااو بذجلاب Strong human resource information sys- tems, linked to national health information systems, as well as clearly defined and agreed metrics or indicators are required to moni- tor the progress of such interventions in terms of increased attractiveness, improved recruitment and duration of stay, increased availability, improved competence, produc- tivity and responsiveness. Table 2 makes an attempt towards advancing a set of key questions and indicators to be used when evaluating and monitoring retention inter- ventions for human resources for health. The conceptual framework presented here needs to be extensively piloted and will be refined through further inputs from different sources. There are ongoing plans for applying it in several countries that are in the process of designing and implement- ing attraction and retention strategies in underserved areas. Efforts will also be made to make sure the evaluation tool is user-friendly. For the sake of argument, a hypothetical example of its potential use is presented in Box 1. The way forward This framework proposes a common approach to facilitate the evaluation and monitoring of interventions to increase access to health workers in underserved areas. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evalua- tion and review of such interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It is hoped that it will promote the use of a common un- derstanding/logic on how attraction and retention interventions work, using a sys- tems perspective. It starts with a common set of indicators, which enable compari- son between various cases, and facilitates reviews of published studies. As such it can be used to monitor and evaluate inter- ventions, either using a method-based or a theory-based evaluation approach with a specific set of indicators. ■ Funding: This work was funded by the World Health Organization, Health Sys- tems and Services, Department of Human Resources for Health, Health Workforce Migration and Retention Unit, Geneva, Switzerland. Additional support was received from the WHO Collaborating Centre on Human Resources for Health at the Royal Tropical Institute (KIT), The Netherlands. Competing interests: None declared. صخلم جئاتنلا سايقل يميهافم لمع راطإ :تامدخلا نم ةمورحلما قطانلما في ينيحصلا ينلماعلا لىإ لوصولا لبس ةدايز Box 1. A hypothetical use of the conceptual framework The framework can be applied to a hypothetical scenario of country X. The evaluation will explore: (i) Does the intervention respond to the problem identified and has the intervention achieved the intended results? (ii) What improvements in intervention and policy design are needed, or are alternative strategies required? (iii) What indicators should be used to measure results? Step 1: situation analysis For example, in country X, a situation analysis has identified a lack of health workers in rural areas. A study of intentions of young graduates to practise identified the unwillingness of health workers to go to those areas due to a lack of opportunities for professional development and low salaries. Thus, the main bottlenecks seem to be in initially attracting staff to rural areas. Step 2: choice of interventions The intervention that has been implemented sought to “create more schools in rural areas” (education category) and to implement a “rural allowance scheme” (financial category) for some categories of health workers, and improve support and supervision (management). The intended result of this multi-level intervention was to increase attractiveness of rural practice for health professions and improve access to professional training to students with rural backgrounds, as well as an increased duration of stay of the categories of health workers covered by the scheme. Step 3: measuring results using a selection of country-appropriate indicators The indicators chosen for monitoring progress and measuring the direct effects (“outputs”) of the intervention would be: “intentions to go to rural areas after graduation”, “number of applicants from rural background to the newly developed schools compared to urban”, “number of graduates choosing to work in rural areas”, “duration of stay in post of the health workers covered by the scheme”. The indicators to measure the indirect effects (“outcomes”) of the intervention would be: “changes in staff ratio within 3 years”, “increase in outpatient visits” before and after, “changes in staff satisfaction”, “user satisfaction” perceptions of community representatives of service responsiveness to need. The outputs and outcomes can be related statistically to take-up of training programmes, to the percentage of staff covered by financial incentive schemes and supported by an experienced manager, to establish correlation and formulate plausible explanations for the effect of the different types of incentives on the way the intervention has performed. Eventually, the final impact of the intervention can be estimated by improved “health outcomes”, bearing in mind that these changes cannot entirely be attributed to the intervention itself. Step 4: monitoring Analysis of longitudinal data from indicators to examine how the situation has changed, especially seeking to identify unintended consequences of implementing the intervention. For example, attracting staff in rural areas may have resulted in their recruitment by nongovernmental organizations and private facilities rather than working in the public sector. Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920362 Luis Huicho et al.Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas Résumé Améliorer l’accès au personnel de santé dans les zones mal desservies : cadre conceptuel pour la mesure des résultats De nombreux pays ont mis au point des stratégies pour attirer et retenir le personnel médical qualifié dans les zones mal desservies, mais les preuves du succès ou de l’échec de ces stratégies sont rares et faibles. Il est difficile de comparer les enseignements tirés et de mesurer les résultats à partir des quelques évaluations disponibles. L’évaluation de ces stratégies se heurte en effet à plusieurs difficultés, dont l’hétérogénéité de la terminologie, la complexité des interventions, la difficulté d’évaluer l’influence des facteurs contextuels, le manque de données de référence et la nécessité de démarches empruntant à plusieurs méthodologies et à plusieurs disciplines pour le suivi et l’évaluation. En outre, le contexte social, politique et économique dans lequel les interventions sont conçues et mises en œuvre est rarement pris en compte dans le suivi et l’évaluation des interventions en faveur des ressources humaines pour la santé. Le présent article présente un cadre conceptuel proposant un modèle pour le suivi et l’évaluation des interventions visant à retenir la main d’œuvre qui tient compte de ces difficultés. Ce cadre conceptuel repose sur une démarche systématique et a pour objectif de guider la réflexion dans l’évaluation d’une intervention pour améliorer l’accès au personnel de santé dans des zones mal desservies, depuis sa phase de conception jusqu’à ses résultats. Il vise aussi à guider le suivi des interventions par le recueil systématique d’une série d’indicateurs, applicable au contexte considéré. Il introduit l’idée qu’une approche globale est nécessaire pour la conception, la mise en œuvre, le suivi, l’évaluation et le bilan des interventions. Il n’est pas destiné à être prescriptif et s’applique de manière flexible à chaque contexte national. Il favorise l’utilisation d’une interprétation commune des modalités d’action des interventions pour attirer et retenir la main-d’œuvre, en recourant à une perspective systématique. Resumen Ampliar el acceso a los trabajadores sanitarios en las zonas subatendidas: marco conceptual para medir los resultados Muchos países han desarrollado estrategias para atraer y conservar a personal sanitario cualificado en zonas insuficientemente atendidas, pero la evidencia sobre el éxito o fracaso de esas iniciativas es escasa y poco robusta. Es difícil comparar las enseñanzas extraídas y cuantificar los resultados de las escasas evaluaciones disponibles. Las medidas de evaluación tropiezan con varios problemas, en particular la heterogeneidad de la terminología, la complejidad de las intervenciones, las dificultades para evaluar la influencia de factores contextuales, la falta de información basal, y la necesidad de aplicar enfoques multimetódicos y multidisciplinarios en la vigilancia y la evaluación. Por otra parte, en la vigilancia y evaluación de las intervenciones relacionadas con los recursos humanos para la salud rara vez se tiene en cuenta el contexto social, político y económico en el que se diseñan y aplican esas intervenciones. En este artículo se propone un marco conceptual que brinda un modelo de vigilancia y evaluación de las intervenciones de promoción de la permanencia que sí tiene en cuenta esos factores. Dicho marco está basado en un enfoque sistémico y pretende servir de orientación a la hora de evaluar una intervención de ampliación del acceso al personal sanitario en las zonas subatendidas, desde su fase de diseño hasta sus resultados. También tiene por objeto orientar la vigilancia de las intervenciones mediante la recopilación sistemática de un conjunto de indicadores, aplicables al contexto específico. Se sugiere que es necesario aplicar un enfoque omnicomprensivo al diseño, ejecución, vigilancia, evaluación y examen de las intervenciones. El marco no pretende ser prescriptivo y se puede aplicar con flexibilidad en el contexto de cada país. Promueve el uso de una perspectiva común sobre el funcionamiento de las intervenciones de fomento de la captación y la permanencia, aplicando un enfoque sistémico. 1. Grobler L, Marais BJ, Mabunda SA, Marindi PN, Reuter H, Volmink J. Interventions for increasing the proportion of health professionals practising in rural and other underserved areas. Cochrane Database Syst Rev 2009;1:CD005314. PMID:19160251 2. Wilson NW, Couper ID, De Vries E, Reid S, Fish T, Marais BJ. A critical review of interventions to redress the inequitable distribution of healthcare professionals to rural and remote areas. Rural Remote Health 2009;9:1060. PMID:19530891 3. Dolea C, Stormont L, Braichet J-M. Evaluated strategies to increase attraction and retention of health workers in remote and rural areas. Bull World Health Organ 2010;88:379–385. 4. Development Assistance Committee. Glossary of key terms in evaluation and results-based management. Paris: Organisation for Economic Co- operation and Development; 2002. Available from: http://www.oecd.org/ dataoecd/29/21/2754804.pdf [accessed on 15 February 2010]. 5. Leeuw F, Vaessen J. Impact evaluations and development: NONIE guidance on impact evaluation. Washington, DC: The Network of Networks on Impact Evaluation; 2009. Available from: http://siteresources.worldbank.org/EXTOED/ Resources/nonie_guidance.pdf [accessed 15 February 2010]. 6. White H. Theory-based impact evaluation: principles and practice. New Delhi: International Initiative for Impact Evaluation (3ie); 2009. 7. Dal Poz M, Gupta N, Quain E, Soucat ALB. Handbook on monitoring and evaluation of human resources for health – with special applications for low- and middle-income countries. Geneva: World Health Organization; 2009. 8. Mills A, Gilson L, Hanson K, Palmer N, Lagarde M. What do we mean by rigorous health-systems research? Lancet 2008;372:1527–9. doi:10.1016/ S0140-6736(08)61633-5 PMID:18984174 9. Anand S, Bärnighausen T. Human resources and health outcomes: cross- country econometric study. Lancet 2004;364:1603–9. doi:10.1016/S0140- 6736(04)17313-3 PMID:15519630 10. The world health report 2006: working together for health. Geneva: World Health Organization; 2006. 11. Peters DH, El-Saharty S, Siadat B, Janovsky K, Vujicic M, editors. Improving health service delivery in developing countries: from evidence to action. Washington, DC: The World Bank; 2009. 12. Terms of reference for an independent evaluation of the impact of Malawi’s Emergency Human Resources Programme. Lilongwe: Department for International Development (UK); 2009. 13. Dieleman M, Gerretsen B, van der Wilt GJ. Human resource management interventions to improve health workers’ performance in low and middle income countries: a realist review. Health Res Policy Syst 2009;7:7. doi:10.1186/1478-4505-7-7 PMID:19374734 References Bull World Health Organ 2010;88:357–363 | doi:10.2471/BLT.09.070920 363 Luis Huicho et al. Measuring efforts to increase access to health workers Special theme – Health workforce retention in remote and rural areas 14. Dieleman MA, Kane S, Zwanikken P, Gerretsen B. Identifying patterns in retention intervention studies: revisiting the evidence from a realist perspective. Geneva: World Health Organization; 2009. 15. Fogarty L, de Vries D, Reavely E, Bunch E. Indicators for evaluating human resources for health capacity-building. In: Evaluation 2009: context and evaluation, American Evaluation Association, Orlando, 11-14 November 2009. 16. Bristow G, Farrington J, Shaw J, Richardson T. Developing an evaluation framework for crosscutting policy goals: the Accessibility Policy Assessment Tool. Environ Plan A 2009;41:48–62. doi:10.1068/a4092 17. Monitoring and Evaluation Working Group of the International Health Partnership and related initiatives (IHP+). Monitoring performance and evaluating progress in the scale-up for better health: a proposed common framework. Geneva: World Health Organization; 2009. Available from: http:// www.internationalhealthpartnership.net//CMS_files/documents/a_proposed_ common_framework_EN.pdf [accessed 15 February 2010]. 18. Codjia L, Jabot F, Dubois H. Evaluation du programme d’appui à la médicalisation des aires de santé rurales au Mali. Geneva: World Health Organization; 2008. In French. 19. Lemiere C, Herbst C, Jahanshani N, Smith A, Soucat A. Reducing geographical imbalances of health worker distribution: what works, what does not and why? A labour market perspective of urban-rural health disparities, policy solution and experiences in sub-Saharan Africa. Washington, DC: The World Bank; 2009. 20. Habicht J-P, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999;28:10–8. doi:10.1093/ije/28.1.10 PMID:10195658 21. Human Resources for Health (HRH) Action Framework [internet site]. Available from: http://www.capacityproject.org/framework/ [accessed 15 February 2010]. 22. Everybody’s business: strengthening health systems to improve health outcomes. Geneva: World Health Organization; 2007. Available from: http:// www.who.int/healthsystems/strategy/everybodys_business.pdf [accessed 15 February 2010]. 23. Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health 2004;94:400–5. doi:10.2105/ AJPH.94.3.400 PMID:14998803 24. Buchan J. The benefits of AHP workforce stability. Edinburgh: Scottish Executive Health Department; 2005. 25. Rabinowitz HK, Diamond JJ, Markham FW, Rabinowitz C. Long-term retention of graduates from a program to increase the supply of rural family physicians. Acad Med 2005;80:728–32. doi:10.1097/00001888-200508000-00004 PMID:16043525