Learning through monitoring : lessons from a large-scale nutrition program in Madagascar

WPS4058 Learning through Monitoring: Lessons from a Large Scale Nutrition Program in Madagascar Emanuela Galasso and Jeffrey Yau Development Research Group Bureau of Economics World Bank Federal Trade Commission* Abstract: Monitoring data is generally collected as a by-product of the process of monitoring program implementation. Yet this rich source of data has not been exploited to assess the effectiveness of the program. In this paper we use detailed administered data from a large-scale, community-based nutrition program in Madagascar, to argue that this data can be used to estimate the differential effect of increased exposure to the program and study how these returns to exposure evolve over time. We find that the returns to exposure are positive: communities exposed for additional one (or two) years display on average lower malnutrition rates of around 7-9 percentage points. Moreover, we find that the returns are decreasing as time and duration increase, though they do not dissipate to zero. These results are consistent with the hypothesis that the returns to the program reflect learning effects from the intervention. Finally, the results show higher differential returns to the program in poorer areas and areas more vulnerable to diseases. These findings have important implications for how such programs should be scaled-up within a country. Keywords: impact evaluation, duration, nutrition intervention, community-based program, large-scale programs, propensity-score, matching JEL classification: I12, C14, C31 World Bank Policy Research Working Paper 4058, November 2006 The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Policy Research Working Papers are available online at http://econ.worldbank.org. This research has been sponsored by the World Bank Research Grant "Community Nutrition: Evaluation of Impacts in Africa", a comparative meta-analysis on the effectiveness of community-based nutrition programs in Madagascar and Senegal. We thank Claudia Rokx, the project task manager for her enthusiastic support to the evaluation effort, Harold Alderman for useful discussions throughout the project, Rinku Murgai, Martin Ravallion and seminar participants at the NEUDC at Brown University, the World Bank and Virginia Tech. We are extremely grateful to the Seecaline project management unit, to Malalanirina Ratomaharo in particular, and to all the provincial directors for providing the monitoring data and sharing their vast knowledge of the program. We thank Bart Minten for sharing the data on the Commune Census. We wish to thank Quinghua Zhao for providing technical help in the data entry. The findings, interpretations and conclusions of this paper are those of the authors and should not be attributed to the World Bank, its Executive Directors, or countries they represent, the Federal Trade Commission or any individual commissioner. Correspondence by emails to egalasso@worldbank.org or yyau@ftc.gov 1  Introduction Monitoring and supervision are intrinsic aspects of the management of any social programs. Monitoring over time how the program performs in the field allows administrators to understand whether the project is implemented according to its design. An essential data tool underlying the monitoring activities is usually referred to as `monitoring data', or MIS (monitoring and information system), which is a subset of the administrative data that is specifically linked to activities of a specific program. The data contained in the monitoring system generally produces simple descriptive statistics that, at different level of aggregation, provide feedback to the implementing agents to address eventual problems. Administrative data have been increasingly used as a tool to evaluate program performance of public assistance programs in developed countries, but there is still very limited evidence to date that uses administrative data to evaluate social programs in developing countries and hardly any that uses monitoring data for this purpose. 1 Why is it the case? The main reason constraining the use monitoring data for evaluation purposes is that, by design, it only collects information on participants, therefore making it impossible to estimate what would have happened in the absence of the program. Evaluating a public program requires data from which to construct participant groups and comparisons groups. Simple reflexive comparisons (before-after) using only data on participants would generally impose a very strong identification assumption (Heckman, Lalonde, Smith 1999): they implicitly assign all the observed change in the outcomes of interest to the program, and assume away the need to construct a counterfactual change in the absence of the program2. As such, evaluations using monitoring data would not allow quantifying the parameters of interest generally used in the evaluation literature, such as the average treatment effect or the average effect of treatment on the treated, both of which require information on non- participants. Yet, monitoring data have inherent (but un-explored) features that make them an attractive tool for analytical and evaluation work. First, it covers the entire universe of the participating population, allowing the results based 1 A notable exception is Ravallion (2000), who uses administrative data on the geographical allocation of expenditures of a workfare program in Argentina matched with a poverty map to implement a methodology to measure the latent differences in the average allocation of the program between poor and non-poor. 2 An interesting variation to using reflexive comparisons is proposed by Piehl et al. (2003), who show that sufficiently long time-series data on participants can be used to test for structural breaks and identifying impacts. 2  on this data to generalize to the entire target population and to study how the same program might have different outcomes in different geographic locations depending on their initial conditions. Second, it is created as a natural by-product of program implementation, making it a readily-available, low-cost source of data. Finally, it is collected since program inception and therefore has a time dimension that would be worth exploring when assessing the effectiveness of a program over time. In this paper we argue that monitoring data can be fruitfully analyzed in the context of assessing program impact to address a different set of questions, questions that are highly relevant for programs where the treatment impact crucially depends on duration of exposure to the program. Specifically, we examine the relationship between the duration of program exposure and nutrition outcomes. The returns to exposure is important from a policy perspective in programs where gains are cumulative over time and where (as in most cases) the program impacts are expected to be heterogeneous across socio-economic groups or areas in across the country. If the returns to exposure are positive and the program is gradually phased in within a country, it might pay to target sequentially groups/areas with the highest marginal return to exposure first, in order to maximize the total gains from the investment in the program. When estimating the differential returns to program duration on child nutritional outcomes, we use a community-based nutrition intervention in Madagascar. The program (SEECALINE)3 was initiated under World Bank support in 1999 and has been gradually scaled-up to cover about one-third of the total targeted population. Its gradual expansion provides substantial variation in program exposure. Its large scale feature allows us to exploit the richness of the monitoring data that was collected over time during its implementation. We quantify the marginal effect of increasing duration to the program by comparing at any given point in time the nutritional outcomes of the "comparable" communities that experience different durations of program exposure. We call this parameter of interest differential treatment effect (DTE). Figure 1, which plots the outcome trends for sites joining the program at different points in time, illustrates this scenario. The vertical distance between these curves is the observed difference in outcomes between the longer and the shorter exposure sites. These vertical 3 Seecaline stands for "Surveillance et Education des Ecoles et des Communaut