墨西哥的教学成果和学校的成本效益：PARE计划

POLICY RESEARCH WORKING PAPER 2128 Learning Outcomes and Roughly doubling theschcc resources allocated per School Cost-Effectiveness student overcame a in M exico 30 percent deficit in test scores among rural stu-,erts in Mexico's PARE program. The PARE Program Gladys Lopez Acevedo The World Bank Latin America and the Caribbean Region Mexico Country Management Unit May 1999  POLICY RESEARCHI WORKING PAPER 2128 Summary findings Past research often attributed most differences in student The PARE program increased learning achievement in learning to socioeconomic factors, implying that the rural and native schools, where students had typically not potential for direct educational interventions to reduce performed as well as other students (in Spanish). Not learning inequality was limited. only did students' cognitive abilities improve under the Acevedo shows that learning achievement can be PARE program, but the probability of their continuing in improved through appropriately designed and reasonably school improved. well-implemented interventions. In rural areas where the PARE design was fully She studies the impact of the Programa para Abatir el implemented, test scores for the average student Rezago Educativo (PARE), a program designed to increased considerably. A 30 percent deficit in test scores improve the quality and efficiency of primary education among rural students could be overcome by roughly in four Mexican states by improving school resources. doubling the resources allocated per student. This paper - a product of the Mexico Country Management Unit, Latin America and the Caribbean Region - is part of a larger effort in the region to understand the impact of program intervention in Mexico. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Michael Geller, room 14-142, telephone 202-45 8-5155, fax 202-522-2093, Internet address mgeller@worldbank.org. Policy ResearchWorking Papers are also posted on the Web at http://www.worldbank.org/html/dec/Publications/Workpapers/home.html. The author may be contacted at gacevedo@worldbank.org. May 1999. (23 pages) The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas ahout development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Produced by the Policy Research Dissemination Center  Learning Outcomes and School Cost-Effectiveness in Mexico: The Pare Program* Gladys Lopez Acevedo *Gladys Lopez is an Economist, the World Bank (Mexico Department) and an Economics Professor at CIDE and at ITAM. This paper was prepared with research assistance from Mireya Pasillas. 1  Learning Outcomes and School Cost-Effectiveness in Mexico: The Pare Program Gladys Lopez-Acevedo 1. Introduction To understand better the qualitative dimension of basic education, it is necessary to analyze student learning outcomes and school effectiveness. What factors influence them? How responsive is student learning to these factors? What impact can learning improvement interventions have? This paper attempts to address some of these questions. Empirical studies of student learning achievement in Mexico are scarce. Interest, however, regarding its determinants and the impact of interventions to improve it is increasing. In January 1995, the Ministry of Education presented the Programa de Desarrollo Educativo 1995-2000 (PED), which contains a series of targets and general guidelines in order to improve the coverage, efficiency and equity of the Mexican educational system. In fact, the PED recognizes the importance of research and evaluation in its strategy to improve quality of education. In view of this policy, the Ministry of Education has collected databases that can be useful for this purpose. These include Carrera Magisterial, PARE, and TIMSS (Third International Mathematics and Science Study) databases among others. Due to data constraints, this paper is unable to do a comprehensive and in-depth analysis of the learning achievement issues in Mexico. The following analysis, therefore, should be regarded as an exploratory rather than as a conclusive study. Available data are used to highlight certain ideas about learning improvement interventions. As mentioned, there has been very little study in Mexico that examined this issue. There are, however, many international studies that looked at this question. An excellent summary of this literature can be found in Fuller and Clarke (1994), and Hanushek (1995). The early studies on learning outcomes showed that the student's socio-economic and cultural background predominantly determines differences in test scores. These led to the conclusion that there was little that government can do by way of direct educational policy and government interventions to improve learning outcomes. More recent results and experience, however, indicate that school factors do matter and that they can play a more critical role than previously thought. Moreover, "education production function" studies indicate that the magnitude of production inputs varies substantially. Some inputs have larger marginal effects than others do and, in some places, the effects of some of the factors are not statistically significantly different from zero, while in others the same factors have shown substantial impact. Table 1 summarizes the various educational inputs that have been empirically analyzed, the number of studies reviewed and the "confirmation percentage" for each of 2  the inputs. Confirmation percentage is defined as the proportion of the reviewed studies showing positive and significant relationship between the specific input and test scores. At the primary level, it is clear that class instructional time, school library, textbooks, and class frequency of homework have the highest confirmation rates at 73.1 - 88.9 percent. On the other hand, teacher's salary level and school teacher/pupil ratio have the lowest confirmation rates at 36.4 and 34.6 percent. More recent studies also tend to stress the effectiveness of improving of physical facilities. Relating the cost of these inputs to their marginal effects on test scores, available estimates further show that in fact textbooks and other educational materials along with improvement of physical facilities have much higher cost-effectiveness than increased teacher salary, years of experience and teacher/pupil ratio. Table 1. Confirmation Percentages of Various Educational Inputs Sorted by Direct Importance to Teacher Utility Number of Positive and Confirmation studies sianificant relation Percentaae Primary Schools: Teacher's salary level 11 4 36.4 School teacher pupil ratio 26 9 34.6 Teacher's years of schooling 18 9 50.0 Teachees experence 23 13 56.5 Class instructional time 17 15 88.2 Class frequency of homework 11 9 81.8 School library 18 16 88.9 School textbooks 26 19 73.1 Secondary Schools: Teachers salary level 11 2 18.2 School teacher pupil ratio 22 2 9.1 Teacher's expenence 12 1 8.3 Class instructional time 16 12 75.0 School textbooks 13 7 53.8 Source: Fuller and Clarke (1994). Several lessons might be drawn from these studies. First, given the above- mentioned differential effects, it is not surprising that differences in aggregate education' budget does not appear to have a tight association with learning outcomes. It all depends on how budgets are allocated and used. Second, in the absence of local information about the relative effectiveness of inputs, improving availability of text books, workbooks, educational materials, school library, and physical facilities would be a prudent choice over other inputs such as increasing teacher/student ratio, teacher salary, and experience especially if schools have a shortage of the previous type of inputs. Nevertheless, in view of the findings that the relative impact and cost of particular inputs depends on the local conditions of schools and their student, it is important to collect local information about the issue. 3  Beyond the above issues, there is a need to understand the structures and processes needed to establish a motivating and enabling environment to ensure that highly cost-effective inputs and interventions are indeed chosen. It becomes also extremely important to think carefully about the appropriate program design and implementation strategy. This paper presents some empirical analyses of learning outcomes based on local data and experience regarding the impact of Programa para Abatir el Rezago Educativo (PARE). The paper is divided as follows. The next section describes the PARE Program and the database. Section 3 assesses the impact of the PARE program on learning and achievement. Section 4 evaluates the cost-effectiveness of the PARE program. The final section presents concluding remarks. 2. PARE background Programa para Abatir el Rezago Educativo (PARE), 1992-1997. The objective of the program was to assist the Government of Mexico in improving the quality and efficiency of primary education, focusing on four Mexican states (Oaxaca, Guerrero, Chiapas and Hidalgo) with the highest incidence of poverty and low education indicators. These objectives, considered as being of the highest priority within the Government's Education Modernization Program, would be achieved through; (i) reducing the high repetition and dropout rates; (ii) raising the level of cognitive achievement of children, and (iii) strengthening management of the primary education system, including program design and implementation, monitoring and evaluation of the system. The program consisted of giving schools additional resources (components) like libraries, better distribution of textbooks, academic material, training aid to teachers and principals, increased in official supervision of teachers and construction and repair of schools. From its inception its performance was monitored through statistical comparisons between the target, or experimental, population (schools in the states of Chiapas, Guerrero, Hidalgo and Oaxaca) and a control group formed by students in comparable schools in the state of Michoacan which falls outside the scope of the program. Special surveys were conducted yearly between 1992 and 1995. In addition, all students were given standardized achievement tests in Spanish and mathematics. PARE also provided the resources to evaluate the success of this program. To this end, two studies were conducted for two different research institutions. One study was made by the C.E.E, - mainly through quantitative variables on school, parents, community, teachers, inputs, supervisors, socioecomic and academic background, and the other by the D.I.E (Departamento de Investigaciones Educativas), through qualitative variables. These databases were developed to evaluate the effects of PARE (Programa para Abatir el Rezago Educativo) on student achievement. During the program several test on Mathematics and Spanish were applied to the students in three consecutive years, when they were in fourth, fifth and sixth grades. The scores of these tests give the outcome or output variables and at the same time allow us to use a value-added estimation. The C.E.E staff also evaluated school directives and school 4  characteristics. Students' parents and teachers answered a survey at the same time. This information was needed to control for teacher and socioeconomic characteristics. The C.E.E sample consists of students from 198 schools randomly chosen from four different types of schools: Urban, RURAL, NATIVE and CONAFE from five different states.' The participation of each school type, relative to the total is shown in Table 2.2 Table 2. Number of schools by type and state, 1992 State Urban Rural Native CONAFE Total Chiapas 6 13 14 5 38 Guerrero 4 14 12 4 34 Hidalgo 3 11 12 8 34 Oaxaca 7 17 15 12 51 Michoacan 7 15 10 9 41 Total 27 71 64 44 198 Source: PARE's database. 3. Impact of the PARE program on learning and achievement 3.1 Control and the experimental groups The literature generated by the PARE points toward a mixed conclusion about the impact of the program.3 This was partly due to incomplete and faulty implementation, especially in urban areas. By design, the program intended to provide a number of simultaneous actions (components), which together would impact on educational outcomes. For pedagogical reasons the total was to be greater than the sum of the parts. The actions were to affect the behavior of students, parents, teachers, principals and supervisors; they were to provide the target schools with supplies, didactic materials and 1 CONAFE stands for Consejo Nacional de Fomento Educativo. 2 The Native school refers to schools offering services to populations which mother tongue is not the Spanish. 3 The PARE program has generated a voluminous literature produced mainly by the Direcccion General de Evaluacion of the Secretaria de Educacion Publica and by the Centro de Estudios Educativos A. C. (CEE). The CEE was chosen by the executing agency of the PARE program (the Consejo Nacional de Fomento Educativo, CONAFE) to monitor and evaluate the program. Its conclusions were summarised in the document "Determinacion del Impacto del PARE en el Aprovechamiento y la Retencion Escolares," Tercer Infonne, Tomo IV, Mexico, D.F., March 1996. After an extensive analysis of the data the report concludes (my translation) on page 21: "... the variable PARE [a dichotomous variable identifying schools which had access to the program] had a significant impact in only two of the estimated equations. They are, first, the equation referring to performance in mathematics in urban schools of the states' capitals; second, the equation for performance in Spanish in rural schools closer to the states' capitals. ... only for schools in these two sub-samples did students achieve performance levels greater than those in comparable schools which remained outside the PARE program." 5  physical infrastructure. In fact, however, only a sub-set of schools benefited systematically from all actions what will be called from now on components 1358. To assess the probable impact of the PARE program we consider a number of experiments based on the following question: What would have been the program's historical performance if it had been implemented as envisaged without faults or delays. We construct counterfactual experiments based only on those schools, which received all of the main components of the program. Before going into the analysis, it's important to mention that the information available posed important constraints for building a panel data set. Table 3 shows the distribution of students by school type in the sample. Our analysis will focus on schools located in rural and native communities, the two most disadvantaged groups in the population with the lowest educational attainment, poorest test performance and highest incidence of school desertion. At the margin, the supplemental actions provided by the program should have the greatest impact amongst this population. Table 4 shows the resulting samples for analysis considering that, for the reasons already noted, we concentrate our attention on a sub-set of these schools -- those which benefited integrally from the program. Table 3. Distribution of students by school type, 1992. SA,cta: IVichoacan: CWapas Guetrem Fidalgo COxam T LIban 396 107 257 357 1,119 361 1,480 Rual 200 202 175 239 816 208 1,024 Nafive 197 114 122 259 692 205 897 Conru* 19 11 29 59 118 Z7 145 TOaW 814 434 583 914 2,745 801 3,546 Source: PARE's database Table 4. Students included in the analysis, 1992. Native & Native & Sub-total Native & Urban & Sub-total Rural. Rural. With Rural. With excluded included in Community TOTAL With comp. 1358 some from the comp. 1358 and other the analysis componente Schools analysis Experimental 585 624 1,209 299 1,237 1,536 2,745 Control 0 0 413 0 388 388 801 TOTAL 585 624 1,622 299 1,625 1,924 3,546 Of which: Native 769 Rural 853 Source: PARE's database. 6  We measure performance by the student's score obtained in the tests applied at the beginning of the 4th grade - before the program began- and at the conclusion of the 6th grade, when the program was already in its third year of implementation. The tests were designed and applied by the Direccion General de Evaluacion (DGE) of the Secretaria de Educacion. Notice that in the opinion of both the DGE and of the CEE, which conducted the impact evaluation of the program, the Spanish test provides a superior metric. Students' performance in mathematics was very low. Measured by their scores in Spanish, the performance of students in the experimental group of schools is significantly better in both the rural and native sub- samples. As shown in Table 5, before the program, students in native schools in the experimental group were markedly disadvantaged with respect to their peers in the control group. The program eliminated this difference. Students in rural schools were undifferentiated before the program; with the program, those in the experimental group showed significantly higher scores. The percentage change in performance is, on average, three times as large for students in the experimental group. However, in urban areas a retrocession in student's performance was observed probably because bad implementation or the wrong components. Table 5. Student's change in performance, 1994. Before (1992) After (1994) Difference Students Average test Students Average s core Total Percentage Native Experimenl 564 14.6 356 29.1 13.9 95.3 Control 205 23.2 125 26.8 4.1 17.7 Total - t/test 769 16.9 481 28.5 11.4 67.3 Rural Experimernal 645 20.7 421 32.9 11.6 56.0 Control 208 20.1 128 29.7 8.2 40.6 Total - ttest 853 20.5 549 32.1 10.8 52.5 Urban Experimental 337 26.9 238 39.7 12.0 44.5 Controi 361 26.9 221 44.3 15.9 59.3 Total - tttest 698 26.9 459 41.9 13.9 51.6 Source: Own calculations based on PARE's database. Note: Difference respect to control group. 7  3.2 Regression analysis on the impact of the PARE program on learning and achievement In this subsection, we assess the impact of the PARE intervention on students' scores controlling for supply and demand indicators. The results are shown in Tables 6 through 8. The indicators, constructed through principal components analysis, include: * Family's cultural capital: index based on parents' schooling, reading habits, television and radio programs listened, and number of books at home; * Teacher performance: index based on teacher's attendance and other practices; * Quality of school director: variable indicating favorable school conditions for teaching and learning, such as qualification of principal, his knowledge update, and the distribution students in the classroom. * Supervision quality: a composite indicator based on frequency of supervisor visits, duration, occupations of people interviewed, and themes discussed; and * Parents' participation: a measure of parents' attitudes to teachers' attendance, participation in school activities, and relevance of parents' school association. Table 6. Student's change in performance, 1992 and 1994. Native Rural Beta coefficient tvle Beta coefficient t-value t-valuet-au Control 0.245348 4.698a 0.11465 2.695a Teachefs Performance 6th grade -0.002794 -0.060 0.074814 1.691c Teachers Performance 5th grade -0.004594 -0.102 0.107404 2.485a Director's Quality 0.171017 3.709a 0.138362 3.040a Supervision Quality 0.121867 2.302b 0.013111 0.283 Parents' Participation 0.072675 1.565c -0.133568 -3.048a Child'spartAcademic Record 0.043542 0.984 0.061537 1.441d R2-adjusted 0.12097 0.06234 F 10.437a 6.205a N 480 548 Students self-esteem at 5th grade -0.088019 -2.032b -0.044318 -1.039 Availability & quality of urban infrstructure -0.166759 -3.120a 0.006836 0.153 Memorandum item: Maximum total contribution of PARE program 0.530844 0.448341 a - Significant atthe 1% level ormore b - Significant at the 5% level or more c - Significant at the 10-% level or more d - Significant at the 20% level or more Dependent vanable: Difference in normalized test scores between 6th and 4th grade 8  Source: Own calculations based on PARE's database. Table 6 shows a simple ordinary linear square model that captures only about 6% of the variance in the difference of scores (between 4th and 6th grades) amongst students in rural schools and 12% amongst students in native schools. No doubt this reflects an inadequate specification of the model be it in its functional form or inclusion of relevant explanatory factors. To the extent that the measured test scores fail to capture the true level of performance in the sample, much of the influence of variables such as parental background, the quality of teaching, etc., is lost in the model. The point to note, however, is that, even so, the explanatory variables behave as expected.4 More importantly, the coefficient of the experimental variable is large and significant. The PARE program has a large positive impact on student achievement in this counterfactual experiment by all means in the scenarios and specifications. The impact is larger for the native schools, a result that is consistent with the orientation of the program. As reported in Table 6, the marginal contribution of each explanatory variable is measured in terms of standard deviations of the dependent variable; i.e., of the percentage change in performance between 4th and 6th grades. This is in order to control for possible demand driven effects and hence simplify the analysis. For the average student at native schools, attendance at a school fully served by the program would, on average, increase the percentage change by 25%. The comparable percentage change for students attending rural schools is half as large. The variables- of school ".supply" (the performance of teachers, principals and supervisors) are partly an outcome of the program. Thus, the program, at its maximum effect estimated with the results of Table 6, could increase the performance of the average student by one-half of the standard deviation of the percentage change in test scores for the respective sub-ample. It should be noted that the variables measuring the characteristics of students, parents, school personnel and facilities are all numerical indices constructed by C.E.E analysts. Some indices aggregate answers to as many as a dozen questions in the original survey. The model in Table 6 is a simple, parsimonious representation. In particular, it could be argued that if the characteristics of the demand (family and community background, parental attitude towards and involvement in schooling, academic history, self-esteem, etc.) were adequately measured, the additional effect of the PARE program would be smaller, even insignificant. Alternatively, if the characteristics of the supply 4 Three observations may be pertinent. First, for students attending native schools it seems that self- esteem, measured at 5th grade and residence in a community with greater access to public services is negatively correlated with performance. One plausible explanation is due to the conflictual character of native education: Students that are positively self-selected may have a greater resentment in attending special schools. Second, and for the same group, while the performance of teachers does not seem to alter significantly the perfornance of students, the performance of principals and supervisors does. This result may be due to the generally poor quality of teaching in native schools. Finally, it is puzzling to note that, in the nrral sub-sample, parental involvement diminishes students' performance. One possible reason for this is the possibility that parental involvement increases as the quality of the school diminishes. Parents act only when the problems are large and apparent. 9  (teachers, principals and supervisors background, performance, attitudes, assiduity, pay, etc., as well characteristics of the school infrastructure and availability of textbooks, supplies, etc.) were captured more precisely, the impact of the program could be larger. The data allows us to do better than the simple model of Table 6; and to make use of the available information without introducing damaging multicollinearity in the results we constructed two sets of principal (orthogonal) components measuring respectively the characteristics of the demand and supply of schooling. Table 7 shows the results of the model built on this more complex structure captured through the two principal components. The results are very similar to those of Table 6. In fact, the impact of the program is greater and more significant. The factor capturing the conditions of supply is also significant and large, especially in the case of schools serving native communities. Table 7. Student's change in performance, 1994 Native Rural Beta coefficient Beta t-score t-score coefficient Control 0.273609 6.210a 0.127214 3.000a Factor - Characteristics of community & family -0.009075 -0.205 -0.161033 -3.815a Factor - Characteristics of school & system 0.201875 4.664a 0.074449 1,754c R2-adjusted 0.12376 0.035 F 23.599a 7.713a N 480 548 Memorandum item: Maximum total contribution of PARE program 0.475484 0.201663 a - Significant at the 1% level or more b - Significant at the 5% level or more c - Significant at the 10%/o level or more d - Significant at the 20% level or more Dependent variable: Difference in normalized test scores between 6th and 4th grade. Source: Own calculations based on PARE's database. An objection may be raised, nonetheless, about the measure of performance. What if small differences in test score are very imperfect measures of relative capabilities and/or achievements? To try to get around this issue, we perform a final experiment on the test scores. We stratify the samples in two sub-samples each: those of students with performance above and below their respective medians. These results are shown in Table 8. Once again the estimates are consistent. The program has a positive and significant impact, and especially so for the native population. 10  Table 8. Student's change in performance, 1994. Dependent variable: Probability of testing above the median in 6h grade. Coefficient Std. Error t-Statistic Prob. Native schools Constant -0.991 0.217 -4.572 0.0% Control 1.272 0.246 5.162 0.0% Factor - Characteristics of community & family 0.054 0.103 0.528 59.8% Factor - Characteristics of school & system 0.630 0.104 6.056 0.0% N 481 Log likelihood -295.562 F-statistic 15.024 0.0% Chi-square 60.095 0.0% Obs with Dep= 1 237 Obs with Dep=0 244 Ex-ante probability 49% Estimated probability (at means) 49% Estimated probability without PARE (control) 27% PARE contribution - percentage gain probability 45% Rural schools Constant -0.396 0.183 -2.161 3.1% Control 0.495 0.209 2.372 1.8% Factor - Characteristics of community & family -0.233 0.086 -2.713 0.7% Factor - Characteristics of school & system 0.107 0.083 1.279 20.1% N 549 Log likelihood -374.073 F-statistic 3.112 1.5% Chi-square 12.448 1.4% Obs with Dep= 1 271 Obs with Dep=0 278 Ex-ante probability 49% Estimated probability (at means) 49% Estimated probability without PARE (control) 40% PARE contribution - percentage gain probability 19% Source: Own calculations. 11  Table 8a. Probability of being in school in the 6th grade, 1994 (Being at school in the 4th grade) Variable Coefficient Std. Error t-Statistic Prob. Native schools Constant 0.499 0.148 3.379 0.001 Control 0.115 0.174 0.660 0.510 Factor- Characteristics ofcommunity & family 0.125 0.078 1.599 0.110 Factor - Characteristics of school & system -0.067 0.076 -0.876 0.381 N 769 Log likelihood -500.106 F-statistic 15.597 0.000 Chi-square 62.386 0.000 Obs with Dep=l 493 Obs with Dep=0 276 Ex-ante probability 64% Estimated probability (at means) 64% Estimated probability without PARE (control) 62% PARE contribution - percentage gain probability 3% Rural schools Constant 0.496 0.144 3.441 0.1% Control 0.271 0.168 1.613 10.7% Factor - Characteristics of community & family 0.184 0.076 2.419 1.6% Factor- Characteristics of school & system 0.121 0.075 1.617 10.6% N 825 Log likelihood -519.618 F-statistic 23.752 0.0% Chi-square 95.010 0.0% Obs with Dep=1 549 Obs with Dep=0 276 Ex-ante probability 67% Estimated probability (at means) 67% Estimated probability without PARE (control) 62% PARE contribution - percentage gain probability 7% Source: Own calculations. Table 9 summarizes the results on test scores. The PARE program - when adequate and fully implemented - could cause an increase in performance for the average student in the range of 19 to 38% amongst rural students. For native students, the percentage change could be much larger, anywhere from 45 to 90%. If consideration is taken of the factors affecting supply, such as the performance of teachers, principals and supervisors, on the plausible assumption that this performance is in part a product of the program, the total impact could be even larger. 12  Table 9. Marginal contribution of belonging to the experimental group, 1994. Mean of dependent Estimated coefficient Marginal contriibtion: Group Unit variable Exnerimental Exneximental Table 6 Rural 10.778 Gain in scores 4.043 0.375 Native 11.356 Gain in scores 9.998 0.880 Table 7 Rural 10.778 Gain in scores 3.644 0.338 Native 11.356 Gaininscores 10.259 0.903 Table 8. Rural 0.494 Probability 0.495 0.190 Native 0.493 Probability 1.272 0.450 Table 8a Rural 0.670 Probability 0.271 0.070 Native 0.640 Probability 0.115 0.030 Source: Own calculations. Note: For Tables 6 and 7, the percentage gained to the mean. For Table 8, the percentage gained to the initial probability of success, estimated at the means of the independent variables. Aside from increasing the student's cognitive achievements while at school the PARE program also increases the probability that the student will continue in school. The two outcomes are probably linked: children who perform better are more motivated to continue and their parents may be more inclined to allow them to continue in school. This is clearly the case for rural students, as shown in Table 10. The probability of school desertion is 20% lower amongst students supported by the program, and the effect is just as large for the broader group of students who benefited from only a partial application of the program. Surprisingly, however, the result does not seem to hold for the native population. One-third of the native students who received the full program from 4th grade onward abandoned the school before completing the 6h grade. Their probability of desertion was 12% greater than that of the comparable control group. Table 10. Desertion. Percentage of students who quit school by the end of the 6th grade, 1994. Complete Proaram* Partial orocram Native Rural Native Rural Experimental 32.9% 28.4% 36.2% 31.1% Control 29.4% 35.7% 36.0% 38.5% Difference 11.7% -20.5% 0.7% -19.2% N 698 809 841 1,006 Source: Own calculations based on PARE's database. * Students in school that received all PARE components simultaneously. This result deserves more analysis. An intriguing possibility is that high-achieving students in native communities move to rural schools where they are immersed in a Spanish-speaking environment. On the other hand, a multivariate analysis (controlling for "4supply" and "demand" variables) of the probability that the student was in school in the 6t grade (given that she had been at school in the 4t grade) indicates that the program had a positive impact on both rural and native schools, see Table 8a. The percentage change 13  in probability is small, however, and specially so for the native population (a mere 3 percent increase). Due to the lack of adequate and sufficient number of instruments we could not sort out the intriguing findings posed by the sign or significance of some of the variables. No doubt, in all the models the experimental variable was significant and positive. 4. Cost-effectiveness of the PARE program 4.1 Costs in the PARE program It is very difficult to estimate the true costs of the PARE program. The program, financed by CONAFE, is not independent of actions taken by SEP in its usual activities of finding and supervising basic education, as explained in Section 2. It could be, for example, that teachers in a school benefiting from the PARE program become more motivated and assiduous simply because they perceive the threat (or reward) of closer supervision by the educational authorities. The costs of the PARE program, as reported by the C.E.E, are shown in Table 11. Expenditure on native schools was nearly 60% higher compared to rural schools and 786% higher respect to urban schools.5 The largest cost items were infrastructure and materials. Expenditure on teacher training and wage incentives accounted for less than 14 % of total spending. Table 11. Per pupil expenditure, 1994. AlI schools* PARE** Cost increase Native Rural Urban Native Rural Urban Chiapas 1,983 605.7 338.1 210.7 30.5% 17.0% 10.6% Guerrero 2,253 749.1 764.2 62.8 33.2% 33.9% 2.8% Hidalgo 2,143 1,127 636.8 51.0 52.6% 29.7% 2.4% Oaxaca 1,770 624.1 229.7 23.8 35.3% 13.0% 1.3% Average 2,037 776.4 492.2 87.1 38.1% 24.2% 4.3% Source: PARE's database. * Unit cost for primary schools in native communities, SEP. ** See Table 12. 5 The percentages are obtained as follows: the difference in cost increase between native and rural areas (and native and urban areas) is divided by the cost increase in rural (urban) strata. 14  Table 12. Per pupil costs PARE program, 1994 BirIii ary Stares Training Inrasucture supervision is Toal teadhems Mtrasupriinntd Per pupil expenciture - Indgenous schools Chiapas 21.5 3.5 2.0 45.6 215.2 0.0 111.3 47.0 159.5 605.7 Guewrero 20.2 3.2 4.1 50.1 282.1 45.8 101.1 103.2 139.4 749.1 Hdalgo 25.2 7.4 3.3 62.0 635.2 123.4 78.3 82.3 109.8 1126.7 Oaxaca 9.2 4.6 3.5 50.3 279.7 52.8 139.3 29.2 55.6 624.1 Average cost 19.0 4.7 3.2 52.0 353.1 55.5 107.5 65.4 116.1 776A Per pupil expenditure - Rural schools Chiapas 0.0 7.3 3.2 32.7 32.6 21.2 136.4 39.4 65.4 338.1 Guerrero 0.0 4.8 6.4 40.8 333.0 4.0 97.9 139.9 137.5 764.2 Hdalgo 0.0 10.6 4.1 58.7 302.4 16.8 93.6 62.4 88.2 636.8 Oaxaca 0.0 6.1 4.4 42.9 0.0 0.0 94.2 46.5 35.7 229.7 Avewage cost 0.0 7.2 4.5 43.8 167.0 10.5 105.5 72.1 81.7 492.2 Per pupil expenciture - Urban schools Chiapas 0.0 5.0 4.0 48.1 0.0 0.0 - 20.1 133.5 210.7 Guerrero 0.0 1.6 1.0 27.3 0.0 0.0 - 9.8 23.1 62.8 Hdalgo 0.0 1.4 0.6 28.3 0.0 0.0 - 13.4 7.3 51.0 Oaxaca 0.0 0.8 0.3 16.4 0.0 0.0 - 5.8 0.6 23.8 Average cost 0.0 2.2 1.5 30.0 0.0 0.0 - 12.3 41.1 87.1 15  As shown in Table 11 and Table 12, the PARE program increased the average per pupil cost of education by 38% in native schools, by 24% in rural schools and by 4 % in urban schools. A simple comparison between the percentage change in average test scores and the cost of the supplementary pedagogical actions under the PARE program - for the subset of schools that received all of the actions and implemented them accordingly - suggests that the program was well implemented for the native population. Here we observe a 42% in average scores versus the 38% increase in cost, an elasticity of 11% (Table 13). However, the ratio is negative for the rural and urban population; the increase in cost is greater than the percentage change in performance. In particular, for urban areas the elasticity was - 445%, which may implied that the implementation of the PARE program was bad in this sector of the population.6 Table 13. PARE, Program: Cost Elasticity, 1994 Avera,e gain in test score Percentae Increase in Ratio Exs1ereal Control Difference ain cost Native 13.9025 4.1 9.8 42.3% 38.11% 11.02% Rural 11.5746 8.2 3.4 17.0% 24.16% -29.67% Urban 11.9755 15.9 -4.0 -14.7% 4.27% -445.00% Source: Own calculations. * With respect to base year - control group; see Table 5. ** See Table 11 and Table 12. Instead of using the observed outcomes as reported in Table 13 we could use the simulated outcomes as reported in Table 9. The results are better. Considering the maximum estimated impact for the native population (a maximum percentage change in performance of 90% estimated in Table 6) the benefit/cost elasticity is 137:100. The equivalent ratio for the rural population (with a maximum change in performance of 38% estimated in Table 7) is 58:100. 4.1 Cost-effectiveness estimates in the PARE program The previous analysis looks at the impact of PARE interventions as it was implemented on average, without limiting the assessment to cases where the program wag fully implemented as envisioned. Specifically, the present section seeks to directly relate the monetary value of the PARE assistance actually received by the schools regardless of the original amount originally planned for them. As explained earlier, ordinary least squares regression was initially used to estimate the relationship. However, the results shaw a "perverse" negative relationship between PARE expenditure per student and learning outcomes, strongly indicating that schools 6 Regression analysis was used to test for this hypothesis controlling for placement effects. The results support the initial hypothesis. 16  that were lagging behind in learning achievement were systematically being targetedfor more assistance. Consequently, a two-stage least squares methodology was used, where the monetary value of PARE assistance per student was modeled as a function of school characteristics and a dummy variable for being in the experimental group or not. This dummy variable is used to identify the learning achievement equation. The results, which are presented in Table 14, reveal that on average PARE assistance has had a significant positive effect on learning outcome in Spanish. Moreover, they show significant positive fixed effects for the quality of school management, supervision and teachers. The surprising result is that parental participation has a significant negative coefficient. Considering the importance that education reformers attached to this factor, further analysis is called for by this unexpected finding. A possible explanation for this "perverse" finding is that disadvantaged schools are forced to mobilize parents for additional resources. Or, it might be that when children perform badly, their parents take a more proactive role in student learning. The elasticity estimates appear reasonable. There are several things worth noting here. First, a 10 percent improvement in staff performance and quality as well as the family's cultural capital is associated with about one to two percent increase in test score. Second, a ten percent increase in per student expenditure that is devoted to finance to PARE program activities would likely raise Spanish learning achievement by about 3.3 percent. This is roughly half the above-mentioned full implementation cost-effectiveness estimate of PARE. Third, being in rural area reduces learning achievement by 31 percent. If a student is in a rural and native school, his score is about 75 percent less than that of others. 17  Table 14. Determinants of Sixth Grade Spanish Test Score: PARE, 1994. Average Coefficient t-value Means Std. Dev Elasticity Spending Elasticity Child's part Academic Record -0.1400 -1.6850 70.8100 19.6400 -0.3470 Per student cost of PARE assistance 0.0055 2.0810 165.9700 364.3400 0.0320 0.0000185 Score in 4th Grade 0.2518 9.9160 22.6800 11.5000 0.2000 Family's Cultural Capital 0.1052 3.9550 53.5900 18.2300 0.1970 Teacher's Performance 5th Grade 0.0953 2.3990 51.8600 7.6100 0.1730 Teacher's Performance 6th Grade 0.1266 2.9340 42.8300 5.4200 0.1900 Director's Quality 0.0928 2.2020 52.2700 7.0500 0.1700 Supervision Quality 0.0472 2.7690 63.6900 18.3500 0.1050 Parenfts Participation -0.0626 -2.8090 35.3400 12.2300 -0.0770 DUM:MYforRural -8.8827 -9.0870 0.3000 0.4600 -0.3110 DUM:MY for Native -12.4228 -8.8540 0.2600 0.4400 -0.4350 (Constant) 23.0799 4.7220 Adjusted R Squares 0.2565 F 67.2897 N=2114 Dependent variable: 6th Grade Spanish test score Estimation method: two-stage least squares Source: Own calculations. 18  5. Concluding remarks Exploratory analysis suggests the following ideas. First, students in rural and native schools are way behind others, at least in Spanish, even when school quality and family's cultural capital are taken into account. Second, this disadvantage could be overcome to some extent by providing those schools with PARE-type assistance, focusing on improvement in physical facilities, books and materials, teacher performance incentive, school management and supervision and teacher training. The cost-effectiveness estimates suggest that, despite their imperfection, a 30 percent deficit in test score among rural students can be overcome by roughly doubling the amount of resources per student allocated to those schools to finance the above-mentioned activities. On this point, it is plausible to think that less resources would be needed if school improvement programs were implemented more efficiently and fully. These conclusions need further verification. It is not clear to what extent these results are applicable beyond the five states under study. Furthermore, due to the limited sample of urban schools, separate analysis of urban children could not be done reasonably well. Finally, fuirther analysis of school effectiveness and parental participation is required. 19  Variables' Defimitions DESCRIPTION CONSTRUCTION SCALE NAME SCORE IN 6h GRADE ESPANOL6: Scores Scores. The exam has six parts, 0-100 obtained in the exam of reading comprehension, use of Spanish in 6th grade. graphics, writing, language interpretation, literature and writing expression. The grade is given by the percentages of correct answers. SCORE IN 4*l GRADE ESPANOL4: Scores Scores. The exam has six parts, 0-100 obtained in the exam of reading comprehension, use of Spanish in 4th grade. graphics, writing, language interpretation, literature and writing expression. The grade is given by the percentages of correct answers. DIFFERENCE IN DIFESP46: Difference Scores. 0- 100 NORMALIZED TEST between test scores SCORES BETWEEN 6th obtained in exam of AND 4h" GRADE. Spanish in 6hf and 4h. FAMILY EDUCATION CCFAM: Quantitative Includes average parents' 0-100 BACKGROUND indicator of family's schooling, lecture habits, cultural capital. television and radio programs and number of books in the house. FAMILY ECONOMIC NVIIDA: Family's standard Housing quality, purchasing 0-100 BACKGROUND of living index. power: transportation services and goods, number of household members. TEACHER DESEMP6: Quantitative Academic considerations in the 0-100 PERFORMANCE (6ff indicator of the teacher improvement of quality of grade) performance in 6th grade. education such as school objectives, teacher's practices in evaluation, attendance, etc. TEACHER DESEMP5: Quantitative Academic considerations in the 0-100 PERFORMANCE (5th indicator of the teacher improvement of quality of grade) performance in 6t grade. education such as school objectives, teacher's practices in evaluation, attendance, etc. DIRECTOR'S QUALITY DC ACA_1: Quantitative Favorable conditions for 0-100 indicator of director's academic activities, teaching and quality. learning processes. Directors' qualifications and actualization. 20  Distribution of students in the classrooms. SUPERVISION CALI_S_1: Quantitative Includes annual frequency of 0-100 QUALITY indicator of supervision's visits, duration, occupations of quality. interviewed people and themes discussed. PARENTS' APF6: Quantitative This indicator weighs the 0-100 PARTICIPATION indicator of parents' attitudes of parents with respect participation in the school to teachers' attendance, parents' process. participation in school activities and relevance of parents associations in the school. UNIT COST Unit cost per pupil Presents the fixed unit cost per pupil. CHILD'S PART HIST_ESC: Index of Total years in pre-school, total 0-100 ACADEMIC RECORD historical academic record repetition and dropout years. of the student. DUMMY FOR RURAL DUMMYR Dummy variable: If DUMMYR 0 & 1 = 1 then the observation is of rural areas. DUMMYR= 0 for other cases. DUMMY FOR NATIVE DUMMYI Dummy variable: If DUMMYR 0 & I = 1 then the observation is of native areas. DUMMYR= 0 for other cases. FACTOR - It's a compound index CHARACTERISTICS OF FACI_14 constructed by principal COMMUNITY AND components method. It includes FAMILY the characteristics of the demand such as family and community background, parental attitude towards and involvement in schooling, academic history, self-esteem, etc. FACTOR- It's a compound index CHARACTERISTICS OF FACI_15 constructed by principal SCHOOL AND SYSTEM components method. It includes the characteristics of the supply such as teachers, principals and supervisors background, perfornance, attitudes, assiduity, pay, etc., as well as characteristics of the school infrastructure and availability of 21  textbooks, supplies, etc. STUDENT' S SELF- SI_MISMO Student self-esteem index. ESTEEM, 5' GRADE Student's perception of his own school performance, of his own goals, of other peoples' opinion, and if he thinks that his success depends on himself 22  References CENTRO DE ESTUDIOS EDUCATIVOS (C.E.E.), Evaluaci6n del Impacto y Efectividad de Costos para Abatir el Rezago Educativo (PARE), Informe Ejecutivo, Mexico, D. F., 1994. CENTRO DE ESTUDIOS EDUCATIVOS (C.E.E.), Determinaci6n del Impacto del PARE en el Aprovechamiento y la Retencion Escolares, Tercer Informe, Tomo IV, Mexico, D. F., 1996. CENTRO DE ESTUDIOS EDUCATIVOS (C.E.E.), Impacto y Eficiencia del Programa para Abatir el Rezago Educativo (P.A.R.E.), Tercer Informe, Tomo IV, Mexico, D. F., 1995. CENTRO DE ESTUDIOS EDUCATIVOS (C.E.E.), Anexo Metodologico, Tercer Informe, Mexico, D. F., 1996. Fuller, Bruce and Prema Clarke (1994), Raising School Effects While Ignoring Culture? Local Conditions and the Influence of Classroom Tools, Rules, and Pedagogy, Review of Educational Research, 64(1): 119-157. Hanushek, Eric A., (1995), Interpreting Recent Research on Schooling in Developing Countries, The World Bank Research Observer, Vol. 10, no. 2, August. Ontiveros, J., Manuel, The Education Production Function: Simultaneous Interaction Between Students, Teachers and Bureaucrats, Ph.D. Thesis, Department of Economics, University of Houston, 1997. Secretaria de Educaci6n Publica (SEP), PARE, Anexo II, Informe de Conclusion (1992- 1996), Mexico, D. F., 1997. 23  Policy Research Working Paper Series Contact Title Author Date for paper WPS21 10 Life during Growth: International William Easterly May 1999 K. Labrie Evidence on Quality of Life and 31001 Per Capita Income WPS2111 Agricultural Land Reform in Postwar Toshihiko Kawagoe May 1999 P. Kokila Japan: Experiences and Issues 33716 WPS2112 Industrial Policy after the East Asian Ashoka Mody May 1999 S. Kpundeh Crisis: From 'Outward Orientation' 39591 To New Internal Capabilities? WPS2113 Wage Determination and Gender Stefano Paternostro May 1999 N. Nouviale Discrimination in a Transition David E. Sahn 34514 Economy: The Case of Romania WPS2114 Economic Reforms and Total Factor Pablo Fajnzylber May 1999 S. Holt Productivity Growth in Latin America Daniel Lederman 37859 And the Caribbean (1950-95): An Empirical Note WPS2115 Foreign Investment and Productivity Simeon Djankov May 1999 R. Vo Growth in Czech Enterprises Bernard Hoekman 33722 WPS2116 Does Child Labor Displace Martin Ravallion May 1999 P. Sader Schooling? Evidence on Behavioral Quentin Wodon 33901 Responses to an Enrollment Subsidy WPS2117 On the Size and Number of Regional Soamiely Andriamananjara May 1999 L. Tabada Integration Arrangements: A Political 36896 Economy Model WPS2118 Developing Countries and the Next Anne 0. Krueger May 1999 L. Tabada Round of Multilateral Trade 36896 Negotiations WPS2119 Tax Deductions, Environmental Ian Parry May 1999 R. Yazigi Policy, and the Double Dividend' Antonio Bento 37176 Hypothesis WPS2120 Can Africa Export Manufactures? Ibrahim A. Elbadawi May 1999 C. Bernardo The Role of Endowment, Exchange 31148 Rates, and Transactions Costs WPS2121 Fiscal Management in Federal William Dillinger May 1999 A. Lara Democracies: Argentina and Brazil Steven B. Webb 88148 WPS2122 Decentralization and Fiscal William Dillinger May 1999 S. Webb Management in Colombia Steven B. Webb 38680  Policy Research Working Paper Series Contact Title Author Date for paper WPS2123 Access to Land in Rural India Robin Mearns May 1999 G. Burnett 82111 WPS2124 Social Exclusion and Land Robin Mearns May 1999 G. Burnett Administration in Orissa, India Saurabh Sinha 82111 WPS2125 Developing Country Agriculture and Bernard Hoekman May 1999 L. Tabada The New Trade Agenda Kym Anderson 36896 WPS2126 Liberte, Egafit6, Fraternite: Monica Das Gupta May 1999 M. Das Gupta Exploring the Role of Governance 31983 In Fertility Decline WPS2127 Lifeboat Ethic versus Corporate Monica Das Gupta May 1999 M. Das Gupta Ethic: Social and Demographic 31983 Implications of Stem and Joint Families