Improving elementary mathematics education in Nicaragua : an experimental study of the impact of textbooks and radio on achievement

World Bank Reprint Series: Number 391 Dean T. Jamnison, Barbara Searle, Klaus Galda, and Stephen P. Heyneinan lmprovuig lexnientary M4athematics Education in Nicaragua An Experimental Study of the hnpact of Textbooks and Radio on Achievement Copyright (1981) by the American Psychological Association. Reprinted by permission of the authors from the Joumnal of Educational Psychology, vol. 73, no. 4 (August 1981), pp. 556-567, published by the American Psychological Association.  Journal of Educational Psychology Copyright 1981 by the American Psychological Association, Inc. 1981, Vol. . 3, No. 41, 556-56, 0022-0663/81/7304 0556$00 75 Improving Elementary Mathematics Education in Nicaragua: An Experimental Study of the Impact of Textbooks and Radio on Achievement Dean T. Jamison Barbara Searle The World Bank, Washington, D.C. Stanford University Klaus Galda Stephen P. Heyneman Stanford University The World Bank, Washington, D.C. Because widespread availability of textboolis in the United States preceded research on the effectiveness of instructional materials, there has been little systematic study of their impact on student achievement. The developing world provides an appropriate setting for such studies. This article reports an experimental study of the impact of textbook availability on mathematics achievement of students in Nicaraguan first-grade classes. This intervention is compared with control classes in which textbooks are relatively rare and with a radio-based instructional program that uses student worksheets but no other textual material. Classes were assigned at random to the three condi- tions. The control and two treatment groups scored similarly on a pretest of mathematical readiness. Both the textbook and the radio treatments had sig- nificant positive effects on achievement. Availability of textbooks increased student posttest scores by about 3.5 items correct, approximately .33 of a stan- dard deviation. Availability of the radio instructional program increased stu- dent posttest scores 14.9 items, about 1.5 standard deviations. Both interven- tions reduced the achievement gap between urban and rural students. How- ever, the question remains whether either radio or textbook use is sufficiently powerful to close the substantial achievement gap that exists between the schools of high- and low-income societies. Currently many children in the devel- Hannan, 1979). It seems likely that the oping world never enter school. However, central concern of educational systems and enrollments have been expanding rapidly, international agencies that contribute to and some projections suggest that mnost their support will shift from the difficulties children will have a place at the first-grade of getting pupils into classrooms to the level by the end of the century (Meyer & problem of what pupils find in the classroom after they have entered. Thus, it becomes The research reported here was supported principally crucial to identify factors that can increase by World Bank Research Project RP0671-60, entitled the quality of primary education at an af- "Textbook Availability and School Quality." It could fordable cost. not have been undertaken, however, if it had not been Primary schools in the developing world imbedded in the joint United States Agency for Inter- are typically of poor quality in both inputs national Development (U.S. AID)-Nicaragua Ministry and outcomes. On the average, teachers, of Education Radio Mathematics Project, supported in part by U.S. AID Contract AID/CM/ta-C-73-40 to furniture, equipment, and materials are Stanford University. below standards considered minimum in the. The authors are grateful to those who were part of the industrialized societies. In 1977 there were project, and to the agencies supporting it, for their col- 10 pupils for each available primary-school laboration. The views and conclusions expressed in this 0ppl o ahaalbepiaysho article are, however, those of the authors and do not textbook in the Philippines (Jamison & necessarily reflect those of any of the supporting agen- Montenegro, in press); in 1978 the value of cies. The authors also wish to thank Jamesine Friend furniture and materials in the average and Juan Jose Montenegro for their contributions to the fourth-grade classroom in Bolivia was 1% of work reported here. l- r ax l Requests for reprints should be sent to Dean T. tat Of Maryland classrooms (Heyneman Jamison, The World Bank, 1818 H Street, N.W., Note 1). A 1979 survey conducted in Mal- Washington, D.C. 20433. awi found one chair for each 8 pupils and one 556  IMPACT OF TEXTBOOKS AND RADIO ON MATHEMATICS ACHIEVEMENT 557 desk for each 88 pupils (Heyneman, 1980). balances exist with respect to the evidence Furthermore, the gap in classroom quality on textbooks, duplicating nrachines, audio- between high- and low-income countries is visual aids, radio, television-in fact, with widening; as more pupils enter school, there respect to all school resources. Further- is less available for each of them. In 1960 more, results from available studies are not the average country in the Organization for unambiguous. Smaller class size and longer Economic Co-operation and Development teacher-training programs are not alwavs invested 16 times more per pupil than did associated with higher achievement levels. any of the 36 countries with per capita in- One of the more consistent indicators of comes below $265 (U.S.) per year. By 1970 higher achievement is the availability of the difference had grown to 22: 1, and by textbooks and other printed materials 1975 to 31: 1 (World Bank, 1980a).' (Heyneman, Farrell, & Sepulveda-Stuarto, The existing evidence indicates that stu- 1978; Heyneman & Jamison, 1980). Table dents in the developing world do not learn as 1 lists 14 studies that report 19 assessments much in primary school as their peers in the of the relationship between the availability industrialized world (World Bank, 1980b, of printed material and student outcomes. pp. 52-53). A typical finding emerging from Although different methodologies were used the International Evaluation of Educational and the quality of the data is far from uni- Achievement, for example, suggests that the form, the positive relationship between input mean score on science tests for students in and outcome in 16 of the 19 cases strongly low-income countries is in the bottom 5% to suggests that textbooks are a potentially 10% of students from high-income countries important and consistent contributor to (Inkeles, 1977). improved quality in schools. However, the Evidence from the United States (Cole- nonexperimc!-ital design of these studies man et al., 1966; Jencks et al., 1972; Jencks often leaves open the possibility, indeed et al., 1979) has suggested that variation in plausibility, of alternative interpretation of the level of school inputs is not strongly as- the cause for observed correlations between sociated with levels of student attainment. textbook availability and achievement. For However, a growing body of work from the this reason we felt it would be desirable to developing world suggests that the situation undertake a carefully controlled experiment there is different. Differences in classroom quality, as measured by physical facilities, availability of materials, and levels of teacher I If one attempts, however, to adjust for international education, appear to be surprisingly robust variations in purchasing power that are not captured in o, a t computation of exchange rates, the ratio of spending in as predictors of student achievement. The industrialized to nonindustrialized societies would de- correlation between the proportion of vari- cline. For example, at the official exchange rate, the ance in science performance explained by 1975. per capita income of the UTnited States was ap- classroom qualityT and national per capita proximately *53 times that of India; after adjusting for differences in purchasing power, however, this ratio income is approximately -.37 (p < .05) declines to 14 (World Bank, 1979, p. 177). (Heyneman & Loxley, in press), suggesting 2 There is, moreover, some evidence that in devel- that the poorer the country, the higher the (ping countries the quality of a primary or secondary impact of classroom quality on achiev e- school is a robust predictor of a person's success in the ment.2 This may result from a much higher labor market, substantially more powverful, for example. than is an individual's socioeconomic status iHevne- variance in the quantity and quality of man, 1980: Heyneman & Currie, 19,9; Frv, 1980(: school inputs in 7ow-income countries than Schiefelbein & Farrell, Note 2). These studies are too in high-income ones. recent and too few to generalize these results with WN'hich inputs can be expected to raise confidence. 3 The Educational Resources Information Center student achievement in the developing sYstem lists :388 titles published on this subject in the world? The evidence is relatively scarce. A United States in 19717. A recent review of the evidence careful literature search located fewer than from low-income countries located 23 studies published 30 studies on the cognitive impact of teacher between 196;3 and 1977 (Husen, Saha, & Noonan. Note gin low-income countries, in com- 3. A subsequent review, which made a specific effort training to locate studies published in non-European languages, parison with almost 400 published in the found a slightly higher number (Avalos & Haddad, United States in 1 year,3 and similar im- 1981).  558 JANIISON. SEARLE. GALDA. AND HEYNEMAN Table 1 AVailability of Reading AMaterial and Student Outcomes in Lowt-Income Countries: Svnupnsi.s of the Literature .School Outcome Relation- Author (Countrv level measures ship- Comments Beebout iI9731 MIalaysia Grades Achievement gains + Random sample of 10. 11 in secondary school; 89 schools (N = 7.674) used national examinations Comber & Chile Primary Science + International Keeves (1973) 10-year- achievement Evaluation of India olds + Educational Achievement Iran Farrell & Chile Primary Math, language +b National sample of Schiefelbein Grades achievement primary schools (1974) 6-8 Educational + aspirations Same Type of secondary b cohort, school entered 4 years Survival to end of + later secondary school Fuller & Thailand Primary Language & math +b Chantavanish Grade 3 achievement (Note 7) Haron (1978) Malavsia 12- to 13- Bahasa Malaysia, +b National year- English, math, prTbability census olds science, history, sample (N = 6,000) geography Heyneman & Uganda Grade 7 National tests + Results stronger Jamison 198h0) in English, math, for English than general math or general knowledge; knowledge; Ravens random sample of Progressive primary schools in Matrices used as 8 districts abilitv control Jamison & Philippines Grades Filipino, math, +b National reduction of Montenegro 1, 2 science pupils to textbooks ratio (in press); achievement from 10:1 to 2:1; Heyneman & gains over 1 year Montenegro ranged from .18 to .51 (in press) SDs; national sample of primary schools Lynch (Note 8) Ecuador Grade 1 Reading, math. 0 88 purposively sampled science classrooms; assessed achievement new texts ODEPOR lNote 9) El Salvador Grades Spanish, math, + National probability 2, 3, 5, social & natural sample of schools; 6, 8, & 9 science household data Simmons & Tunisia Individual Arabic, French, + 44 village students; Askov (Note 1o) students, arithmetic 80 students from Grades 4-8 achievement urban suburb Smart (1978) Ghana Grades Reading + 40 rural experimental 8, 9 comprehension and control schools; used pre- and posttests over 2 vears; treatment was access to a school newspaper  IMPACT OF TEXTBOOKS AND RADIO ON MATHEMATICS ACHIFVEMENT 559 Table 1 (continued) School Outcome Relation- Author Country level measures shipa Comments Thorndike Chile Primary Reading + International l1973) 10-year- comprehension Evaluation of Iran olds - Educational Achievement Wolff (Note 11) Brazil Primary Promotion +b Random selection of Rio Grade 1 to Grade 2 5% of schools Grande based oi1 (N = 20,000) do Sul teacher assessment Note. ODEPOR Oficina de Planeamiento y Organizacion. a This column refers to relationship between availability of printed material and student outcomes (+ = positive relationship; 0 = no relationship; - = negative relationship). b Stronger with low socioeconomic status and/or rural students. assessing the impact of increased textbook the administrative capabilities and the re- availability on student learning. The op- sources available in Nicaragua. Thus, the portunity to do this arose during the course work does not assess the potential that of an experiment designed to investigate the textbooks or radio lessons have for improv- effectiveness of a radio-based instructional ing student achievement under optimal cir- program. This experiment, known as the cumstances. Rather, it attempts to assess Radio Mathematics Project, was conducted their impact as they might be adopted in the in Nicaragua from 1974 to 1978 (Friend, typical developing country. Searle, & Suppes, 1980). The Radio Mathematics Project provided Method an ideal setting for undertaking such an ex- periment. Not only were random assign- Subjects ment and testing procedures already in place, but inclusion of the textbook experi- In earlier work conducted by the Radio Mathematics e, bt i o o , te t k Project, all primary schools in three provinces of Nic- ment in the radio project offered the possi- aragua were classified as radio or control, using a ran- bility of comparing the two interventions dom sampling process stratified by urbanization (Searle with one another as well as with the status & Galda, 1980). For the present experiment 88 first- quo. Previous evaluations had shown that grade classrooms were selected at random, 48 from radio the radio instructional program has a strong schools and 40 from control schools. The experiment was conducted in the 1978 school year (February to positive influence on children's mathematics November). Half of the classrooms from control achievement in Grades 1-3. and particularly schools were assigned to the textbook treatment, and in Grade 1 (Searle & Galda, 1980). Thus the remaining 20 served as controls for both the text- first grade was chosen as the major focus of book and radio treatments. Close to 3,000 students participated in the two experiments, but because some the textbook intervention. This article children registered late and others dropped out, not all compares mathematics achievement levels students were enrolled for the entire school year. The of control classes with classes provided with fluctuation in numbers of students is discussed radio lessons and adequate textbooks. below. Educators have become increasingly Materials aware that placing new or additional in- structional materials in classrooms does not Textbooks. In 1977, 13 rural and 7 urban fourth- guarantee their use. Their research re- grade classrooms were surveyed to determ'ne the ported here attempts to assess the impact of number of mathematics texts present and the ways they materials under naturalistic conditions. were used by the teachers. The number of textbooks That is, the level of effort expended in en- available varied widely, ranging from 2 to 28 per class- room. However, the high number (28) was found only couraging teachers to use the new materials in one urban and one rural classroom. Four classrooms was limited to that considered feasible given had between 10 and 14 texts; 12 of the 20 classrooms had  560 ,JAMISON, SEARLE, GALDA. AND HEYNEMAN 3 or fewer texts. Most teachers reported that thev used mathematics section of the Spanish version of the kin- the textbooks as resources for themselves and did not dergarten-level Test of Basic Experiences (Moss, 1970). make them available to the children. In contrast, the The test booklet consists of 28 pages. one item per page: experimental treatment provided a text for each the questions are read aloud. All items are multipie child. choice, and students respond bv drawing a line through In choosing a textbook, all those approved by the the illustration of their choice. In adapting the *cst. Ministry of Education were examined. Only one text several questions were modified because either the i,- series conformed reasonably well so the official curric- lustration or the text was unsuitable for use in Nica- ulum and was available locally in sufficient quantities. ragua. This was Matemnatica Para ia Educaciont Primaria, an adaptation of ElementarN School Mathematics by Procedure Eicholz et al. (1968). The translation and adaptation were designed specificallv for use in Central America: The experiment WaS conducted during 1978 in the the first-grade text is in workbook form. ' context of a project that had been in operation in Nic- Teachers were given a textbook for each child as well aragua since 1974. Most procedures were those that as the teacher's editions of the texts for Grades 1 and 2, had been established for implementing and evaluating the latter because some topics in the official Nicaraguan the Radio Matherratics instructional programs (Searle curriculum appeared in the second-grade text. A chart et al., 1976; Suppes, Searle, & Friend, 1978); in partic- cross-referencing topics in the text and in the curricu- ular, the assignment of classrooms to treatment condi- lum guide was prepared to help teachers accustomed to tions (see previous Subjects section) was based on following the guide. In addition, a handout was pre- groups established for the Radio Mathematics Project pared that explained the differences between the text evaluation. and the official curriculum and provided suggestions for Teacher orientation. Teachers using radio lessons teaching the topics that did not appear in the text. attended one 3-hour training session before school Radio lessons. The first-grade radio lessons are opened in February 1978. The session described the described in detail in Searle, Friend, and Suppes (1976). instructional program and gave teachers guidance in The full course consists of 150 daily mathematics les- using the radio, the teacher's guides, and the student sons, each in two parts (i.e., a 20- to 30-minute broadcast worksheets. A session of similar duration was held for and a postbroadcast lesson taught by the classroom teachers of classes in the textbook group at which the teacher). Most Grade 1 lessons require individual handouts were distributed and discussed and teachers student worksheets; these were distributed to teachers were given suggestions about using the texts effectively. in three installments during the year. At the beginning No meeting was held for teachers of control classes. of the school vear teachers were given a guide describing activities for teachers before, during, and after the radio Table 2 lessons. The instructional program was designed to Variables LUsed in Regression Analyses cover the topics in the official Grade 1 mathematics curriculum. Variable n M SD Instrutnents Classroom level Mathematics achievement was assessed using a Pretest posttest, which is described in detail in Searle and Galda score" 87 21.9 1.9 (1980). The posttest uses a matrix-sampling design Posttest (Shoemaker, 1973) and has 84 items, one fourth of which scoreb 87 46.1 8.3 are taken bv each student. The test was constructed Urban by selecting items from pools defined from the topics schoolo 88 .477 .502 in the official curriculum. The number of items for a Radio topic was chosen to be proportional to the time allotted graupd 88 .545 .501 to that topic by the curriculum guide. Since all three Text instructional programs in the present experiment are groupe 88 .227 .421 based on the same set of objectives, the method for de- Control veloping the posttest assures a fair assessment of groupf 88 .227 .421 achievement across programs. (Although students taking radio lessons may learn test-taking skills that Individual level give them an advantage, this does not seem to explain the score differences reported below. For discussion Pretest of this potential difficulty, see Searle & Galda, 1980, p. score 2,070 21.9 4.62 112.) Posttest A test of mathematical readiness was administered score 1,699 46.6 11.85 as a pretest (see Searle & Galda, 1980). The pretest was used to obtain scores that could serve as covariates in a Maximum possible score = 28. b Maximum possible the prediction of the posttest score. mhe pretest scores score = 84. c 1 if school is urban, 0 otherwise. d l if were also used to check on the comparability of the class assigned to radio group, 0 otherwise. e 1 if class groups of classrooms randomly assigned to treatments. assigned to text group, 0 otherwise. f 1 if class assigned The test used for the pretest is an adaptation of the to control group, 0 otherwise.  IMPACT OF TEXTBOOKS AND RADIO ON MATHEMATICS ACHIEVEMENT 561 Table 3 Numbers of Classes and Students Tested Pretest Posttest No. ii students No. of students tested, class tested/class No. of No. of Group classes M SD classes M SD Breakdown by treatment Control 20 24.1 6.1 20 18.1 7.3 Textbook 20 24.3 5.3 20 20.7 6.0 Radio 47 23.6 5.3 47 19.6 7.6 Breakdown by urbanization Rural 46 20.0 3.0 45 17.4 5.6 Urban 41 28.2 4.1 42 21.8 8.0 .Note. One urban radio class was not pretested; one rural radio class was not posttested. SuperVision. Classes assigned to experimental on pretest and posttest can be attributed to any of treatments by the Radio Mathematics Project were not several factors: late registration, dropping out, absence supervised or visited by project staff members. This from school on the day of the test, and failure to be policy had been adopted in evaluating the radio lessons tested even though present because of overcrowding. to assess their effectiveness under normal operating conditions. The decision not to supervise treatment groups requires some explanation. An important Data Analysis Methods trade-off exists: It is necessary to monitor program use to assess the degree of implementation of an innovation. M a w c u t c However, classrooms in Nicaragua are not normallyv unt of analvsis. This IS appropriate, since both the visited bv supervisorv personnel during the school year. assignment to treatment and the implementation were at the level of the classroom (Cronbach, Deken, & Webb, Visits to check on implementation would have to be Note 4, chap. 1). Multiple linear regression was used made by project personnel, and disguising their purpose to estimate treatment effects on the posttest score. would be difficult. Thus, visits made to check on im- Reg re atmere etectsin the least plementation could be expected to influence stronglv Regressions were estimated using ordinary least the degree to which teachers used the radio lessons or squares. Variations in classroom size, which might have the textbooks. As a compromise, the project planned suggested the use of generalized least squares, were re- to make one surprise visit to each class assigned to the garded as unlikely to have an important effect textbook treatment to determine whether the books were being used. However, the visits were not made Results because of political turmoil at the time they were scheduled. Instead, the visits were conducted early in Data the following school year. Testing. Pretests were administered during the first Table 2 presents the variables used in the 3 weeks of school. Posttests were scheduled to be ad- analyses as well as their means and standard ministered in the last 3 weeks of school but were given deviations. The numbers of classes tested 2 weeks early because there were rumors that school would close early. All tests were administered bv and the mean number of students in classes project staff members, not by classroom teachers. To for different treatments are shown in Table minimize variation in procedures, taped instructions 3. The number of students pretested (per were used for the posttests. Thus, all students heard class) ranged from 9 to 32. For the posttest the same instructions, and timing for the tests was the range was 6 to 37. (The reader is re- precisely controlled. Not all students were tested. To reduce copying minded that in larger classes students may when a classroom was overcrowded, test administrators have been excused from testing sessions.) were instructed to excuse students ty using a systematic Means and standard deviations for pretest procedure that gave each child an equal probability of a * m being excused. For logistical reasons it was not possible and posttest are presented in Table 4, at to identify for posttesting those students who were both the classroom and individual level. pretested. Thus, fluctuations in the student population Pretest scores for the three treatments are  562 -JAMISON. SEARLE, GALDA, AND HEYNEMAN not significantly different.4 This result is Table 5 expected, since classrooms were randomly Regression of Treatment Variables on Class assigned to treatments. Pretest scores were Mean Posttest Score obtained primarily to serve as covariates in Re-reso results the regression analyses reported below; that Determini Regression results they are comparable across treatments variable b t confirms the effectiveness of the randomi- zation. All of the differences in posttest Radio 14.88 11.36 scores between treatment groups at the Text 3.93 2.52 classroom level are statistically significant. Urban 1.81 1.67 The textbook group scored significantly Pretest .24 .86 higher than the control group, t (38) = 2.74, Constant 31.02 p < .01; the radio group scored slgnifficantly R2 .67 higher than the control, t (65) = 10.63, p < .001, and significantly higher than the text- Note. n = 86. The Radio and Text regression coeffi- book group, t (65)= 8.54, p < .001. At the cients refer to the control group; the Control variable individual level, the textbook group scored was omitted from the regression. about .33 of a standard deviation higher than cantly to predicting posttest score can be the control group; the radio group scored 1.5 attributed to the low variability across standard deviations above the control classrooms in pretest scores. The standard group. deviation is 1.9, and all but six scores fall Determinants of Posttest Score between 19 and 24.9. (The variability of n e s pretest scores across individual students is Maln effects. Table 5 presents the resls substantial [see Table 2], and in regresrions of aelindea re ressio . The models o a run at the student level the significance of data quite well, accounting for amost fits Of the coefficient of pretest on posttest is ex- dat qute ell acoutin fo alost70%of tremely higyh, t > 13.) As expected, urban the variabilitv (R-2 = .67 ).5 The variable teeyhg,t>1. sepce ra e v t R = .6 T classes perform better on the posttest than with strongest effect is radio, the indicator rural classes. for assignment to the radio group. The re- Interaction effects. The presentation of gression coefficient for text is also signi.fi- findings in Table 5 fails to allow for the cantl different from zero, indicating a sig- possibility that either the radio or the text- nificant treatment effect, although it 's book intervention could work differentially weaker than that of the radio treatment. The coefficient on pretest score in Table 5 is small and not statistically significant. The failure of pretest to contribute signifi- 4 Because of attrition between the pretest and post- test, it is useful to examine the pretest scores of those Table 4 who did and did not take the posttest. There is a small Pretest and Posttest Means and Standard but statistically significant difference between the mean Deuiations or Three Treatments pretest scores for those who were present for the post- Deviations-for-ThreeTreatments test and those who were not present (22.4 vs. 21.7, t = Pretest Posttest 3.10). The difference is in the direction expected if less ostes, able students drop out, but in this case, it is very small. Similar results are discussed in Searle, Sheehan, Gon- Treatment n Ml SD n M SD zalez, and George (1978). However, there is no differ- ence in pretest scores across treatment groups for those Classroom level not posttested. The scores are 22.1, 22.0, and 21.5 for Control 20) 21.8 2.0 20 37.2 5.0 control, textbook, and radio groups, respectively. Thus, Textbook 20 '22.1 1.8 20 40.9 3.5 there is no differentiai attrition across treatments, as Radio 47 21.8 2.0 47 52.2 .54 measured by the pretest. o 47 28 2 We used a double-log (or Cobb-Dotiglas) specifi- a le cation of the relation between posttest score and its determining variables to explore for the possibility of Control 486 21.9 4.6 362 37,6 9.7 a stronger nonlinear than linear relationship between Textbook 486 22.3 4.8 414 41.1 9.4 the two main variables; both the R2 and the estimated Radio 1,098 21.8 4.5 923 52.5 10.1 effects of the determining variables were little changed.  IMPACT OF TEXTBOOKS AND RADIO ON MATHEMATICS ACHIEVEMENT 563 Table 6 Interaction of Urban and Treatment Variables, With Posttest as the Dependent Variable Regression runs Against rural control Against urban control No. of Variable Treatment classes b t b t Location Urban Text 5 5.67 2.33 1.13 .47 Urban Radio 24 17.93 9.66 13.38 7.29 Urban Control 10 4.54 2.06 - - Rural Text 13 6.08 2.96 1.54 .7 o Rural Radio 22 16.39 8.81 11.84 6.30 Rlural Control 10 - - -4.54 2.06 Other Pretest .22 .78 .22 .78 Constant 30.18 34.72 R .678 .678 Note. n = 86. The unit of analysis in these regressions is the classroom. Plausibly, the effects could differ among contain the same information, with different students with different pretest scores, for bases formed by deleting different indicator students in rural and urban schools, as well variables from the regression.) as for other possible student characteris- Table 6 shows, for example, that in com- tics. parison with the rural control group, the To test for a possible interaction between urban control group is expected to average treatment condition and students' compe- 4.54 additional items correct on the posttest, tence at entrance, as measured bv the pre- or the rural text group would average an test, we ran a series of regressions (with in- additional 6.08 items. This indicates that dividual students as the unit of analysis the availability of mathematics textbooks rather than classrooms) that included in- more than compensates for the disadvantage teraction terms between text and pretest and of being in a rural school. More than this, radio and pretest.6 All estimated interac- Table 6 indicates that both interventions tion coefficients were very small and did not reduce the difference in performance be- approach statistical significance. We con- tween urban and rural groups. Whereas the cluded that the effectiveness of both inter- difference between urban and rural control ventions is indlependent of the initial com- groups is 4.54 it4ifns correct, the difference petence of the student. between urban and rural radio groups is only Differences in effectiveness between rural about 1.5 items, and the urban text group and urban schools did emerge. Since both actually performed slightly less well than did the attribute (rural-urban) and the treat- the rural text group. Providing instruc- ment icontrol-text-radio) variables are tional media, whether print or audio, seems discrete, we formed six (0. 1) indicator vari- to reduce or even eliminate urban-rural ables to denote the group each classroom is differences in school quality. in. For example, if a classroom is in a rural Degree of implementation of treatment. schlool that received the textbook treatment, As indicated earlier, the implementation of the indicator variable for that class denoting the textbook treatment was not monitored rural and text would take on the value of 1, during 1978. Early in 1979 all schools in the and the other five indicators would take on the value of 0. These indicator variables are then entered, along with the average pretfest 6 See Delaney and Maxwell (1980) for a discussion score for the class, into regressions predicting and justification of examining Attribute X Treatment interactions through multiple regression analvsis in- posttest score. Table 6 shows the results. cluding discrete covariates tas our urban variable is in (The two sets of results columns in Table 6 this case).  564 .JAMISON, SEARLE, GALDA, AND HEYNEMAN textbook group were visited and information analysis of the extent to which treatment about textbook usage was collected from all efficacy differed between rural and urban but one of them. Eleven teachers reported schools showed both radio and textbooks to using the texts all year, and 4 reported 6-8 be relatively more effective in rural schools; months usage. In the four remaining both interventions, even if uniformly ap- classes, the texts were used little, if at all. plied, would close a substantial fraction of One would expect that the degree to which the existing differences in quality between teachers claim to use the texts would be re- rural and urban schools. It should be borne flected in posttest scores. This is not the in mind that the achievement differences we case. The mean posttest score for the 11 have reported resulted from interventions classes where teachers said they used the texts faithfully is 41.3, compared with 40.6 7 Other project results bear on the relationship be- for the remaining classes. However, from tween intensity of treatment and outcome. The results another point of view this finding is not reported here for first-grade radio and control classes surprising. The score difference between in 1978, the year of the revolution, replicate findings for control and textbook groups is only 3.7 1976, a more normal year. All the communities covered by the project were affected by revolutionary activities, points (see Table 4). With small numbers and battles were fought in several of them, resulting in of classrooms it is difficult to detect differ- school closures and other disruptions of the educational ential effects within this relatively limited process. Yet regression results on data pooled from range.7 both years showed that revolution caused less than a 2-point decrement in posttest scores, and the effect was the same for radio and control classes (Searle & Galda, Discussion 1980, pp. 126-128). In other (unpublished) work we investigated several The experiment reported here took place measures of radio usage and failed to find one that during the 1978 school year in the first provides more predictive informaticn than the dichot- omous radio variable. That is, any level of exposure to grades of 88 schools.8 Students in 48 of the radio-based instructional program results in in- these schools received radio lessons; students creased posttest score and variability in outcome asso- in 20 of the schools received individual ciated with different levels of exposure is too small to workbooks; and students in the remaining 20 be detected. These results suggest that reducing instructional time schools served as controls. Assignment of during the school year does not substantially diminish schools to treatment and control conditions achievement. (The pattern of missed lessons may, was random. All students were pre- and however, be quite important. That is, missing every posttested, and the resulting data were an- other day has a smaller impact than missing the first or alyzed at the classroom level, using linear second half of the year.) These results are inconsistent methd to te with the emerging learning-time literature (Wiley, 1976; regression methods to Dredict the posttest Wiley & Harnischfeger, 1974; Berliner, Note 5) and with score from indicator variables for treatment our own previous analysis of the effect of attendance on condition and location (rural-urban), with learning in Nicaragua (Jamison. 1978, pp. 209-210). pretest controlled. However, Cooley and Leinhardt (1980) reported similar pretesth c trolled. t kresults in a careful study of the impact of instructional Both the textbook and the radio treat- programs on first- and third-grade reading and math- ments had significant positive effects on ematics achievement. They examined a large number achievement; availability of textbooks in- of process variables (more than, 5) and found that for creased student posttest scores by about 35 first grade, the best predictors of mathematics items correct, and availability of radio les- achievement gain were pretest and "opportunity to learn," which was defined operationally as the inclusion sons increased the scores by about 14.9 items in the material taught by the teacher of the topics ap- correct. These differences are substantial: pearing on the posttest. The amovmnt oftime scheduled The standard deviation of the classroom for a subject bore little relation either to gain or to raw posttest means is 8.3, and the standard de- achievement. viation of the individual posttest scores is A parallel experiment was run at the fourth-grade viaton f th inividal osttst core is level. However, differences between treatment groups 11.8. and control were small and are difficult to interpret Both interventions, but particularly radio, because the control classes were tested in 1977, the year have the capacity to eliminate much of the prior to the experiment. Some evidence exists that- gap in achievemen;t across classrooms and a fourth-grade classes were more substantially disrupted gap tby the civil disorder than other grades. The fourth- noticeable amount of the more substantial grade results are discussed in Searle and Galda (1980) gaps among individuals. Moreover, an and are not reported here.  IMPACT OF TEXTBOOKS AND RADIO ON MATHEMATICS ACHIEVEMENT 565 lasting for 1 school year: ultimatelv it will be safer and more sure wav of obtaining the desirable for research to examine conse- desired results. However. the question re- quences of radio or increased textbook mains whether either radio or textbooks availability over the entire elementary school contain sufficient potential for closing the experience of a child. very substantial achievement gap that exists Why did the radio group improve at a rate between the schools of high- and low-income significantlv greater than that of the text- societies. book group? It seems likely to us that at least part of the difference can be attributed 9 The potential of textbooks for improving school to the more inconsistent application of the qualitv has led such agencies as the United Nations textbook treatment tin the hands of teachers Educational. Scientific and Cultural Organization and with relativeiv low levels of education) ii: the World Bank to examine carefully the production, comparison with the uniformlyr adminis- logistical, and cost implications of national textbook tered, high-quality radio lessons that em- programs;see Bolibaugh et al. (Note 6) and Neumann bodied learning principles such as active responding, immediate feedback about an- swers, and careful timing and systematic Reference Notes review of past lessons. The tradeoffs im- plied by this situation are interesting: The 1. Heyneman. S. P. Primary education in Bolivia: use of radio as a delivery medium makes What's w(rong? tnpublished manuscript. The Wvorld Bank, Education Department, 1979. possible standardized delivery of lessons, but . Schiefelbein, & Farrell, J P Education and the pedagogical value of the lessons can be occupational attainment in Chile: The effects of no better than the design and content of educational qualitv, attainment. and achieve- what is transmitted. On the other hand, ment. Unpublished manuscript. The World Bank, teachers can use textbooks poorlv or well and Education Department, 1980. H. Husen. T., Saha, L. .J., & Noonan, R. Teacher have the potential for making highly effec- training and student achievement i'n less well de- tive usE of even poorly designed learning veloped countries (Staff W orking Paper No. 310). materials. Washington, D.C.: The World Bank. 19.78. As we have noted, textbooks are far fr-)m 4. Cronbach. L. J... Deken. J. E.. & Webb, N. Re- univrsa inlow-ncoe cuntres,andthe search on classr'ooms and schools: Formulation universal in low-income countries, and the ,of question,. design, and analvsis (Occasional decision to increase their availability has cost Papers of the Stanford Evaluation Consortium). and logistical implications that can be sub- Paio Alto, Calif.: Stanford tUniversity, 1976. stantial for financially pressed ministries of 5 Berliner. D. C. The beginning teacher eLaluation education.' Thus. research into the extent studx OLerew and selected findings, 1974-,5. San Francisco: Far West Regional Laboratorv for of cognitive gains that could be expected Educational Research and Development. 1975.). from increased investment in textbooks has 6. B3olibaugh. J.. et al. (;Uidelines for the prepara- immediate p0licy relevance for low-income tuo, production, and distribution (of text books countries. iEducational Financing Occasional Paper No. 5). counties.Paris: United Nations Educational, Scientific and Our principal purpose in this article has Cultural Organization. 1980. been to examine experimentallv whether 7. Fuller. W.P.. &Chantavanish. A. Astudvyofpri- provision of textbooks can result in a peda- mary schooling in Thailand: The final report. In, gogically-significant improvement in school Fautors affecting scholastic achievement (oi the quality in an environment where an ade- primar sNchool pupil', Bangkok. Thailand: Of- .lice o(f the National Education Commission, 1977, quate supply of texts is lacking. We con- . Lvnch. P. 1). A national texthook program. elude that it can. WX e further conclude that F 'romise and fru.,tration. Paper presented to the instruction by radio. if the lessons are annual meeting of the American Educational Re- properly designed. can do even more for in- search Association, Chicago. April 1974. prhoperlv designed, candoeven mo than can- te . Oficina de Planeamiento y Organizacion. The school quality improvement than can the reiati,ns of somia,'. oonomic.( and cultural factor. provision of textbooks. But the provision of and national achietornent exoan rr(sltts El Sal- textbooks is a more straightforward opera- vador, San Salvador: Ministrv of Education. tion, both logisticallv and in terms of the 197.i demands made on skilled project manage- 10. Simmons, J& Askoy. . Schooiinfordevelop t, , ,,,, ment' Students. ard w'orkers in Tunisia. Un- ment; for this reason, in many circumstances published manuscript. Harvard University. De- expanding the supply of textbooks will be a partment of Economics, 1972.  566 JAMISON, SEARLE, GALDA, AND HEYNEMAN 11. Wolff, L. Why children fail first grade in Rio Hevneman, S., & Loxley, W. Influences on academic Grande do Sol: Implications for policy and re- achievement across high and low income countries: search. Rio de Janiero, Brazil: United States A re-analysis of IEA data. Sociology of Education, Agency for International Development, 1970. in press. Heyneman, S. P., & Montenegro, X. P. Social envi- ronmental factors which influence academic References achievement in the Philippines. In D. T. Jamison & G. Feliciano (Eds.), Improving school quality in the Philippines: Evaluation, research, and educational Avalos, B., & Haddad, W. A review of teacher effec- policv. Manila: University of the Philippines Press, tivene sq research in Africa, India, Latin America, in press. Middle East, Malaysia, Philippines, and Thailand: Inkeles, A. Review of the International Evaluation of Synthesis of results. Ottawa, Ontario, Canada: Educational Achievement. Proceedings of the Na- International Development Research Centre, 1981. tional Academy of Education, 1977, 4, 139-200. Beebout, H. S. The production surface for academic Jamison, D. T. Radio education and student repetition achievement: An economic study of Malaysian sec- in Nicaragua. In P. Suppes, B. Searle, & J. Friend ondarv education (Doctoral dissertation, University (Eds.), The Radio Mathematics Project: Nicaragua, of Wisconsin-Madison, 1972). Dissertation Ab- 1976-1977. PaloAlto,Calif.: SlanfordUniversity, stracts International, 1973, 33, 3099A-3100A. Institute for Mathematical Studies in the Social (University Microfilms No. 72-27,310) Sciences, 1978. Coleman, J. S., et al. Equalitv of educational oppor- Jamison, D. T., & Montenegro, X. P. Evaluation of the tunity. Washington, D.C.: U.S. Government Philippines textbook project: Multivariate analysis Printing Office, 1966. of data from grades 1 and 2. In D. T. Jamison & G. Comber, L. C., & Keeves, J. P. Science education in Feliciano (Eds.), Improving school quality in the nineteen countries. New York: Wiley, 1973. Philippines: Evaluation, research, and educational Cooley, W., & Leinhardt, G. The instructional di- policy. Manila: University of the Philippines Press, mensions study. Educational Evaluation and Policy in press. Analysis, 1980, 2, 7-25. Jencks, C., et al. Inequality: A reassessment of the Delaney, H. D., & Maxwell, S. E. The use of analysis effect of family and schooling in America. New of covariance in tests of attribute-by-treatment in- York: Basic Books, 1972. teractions. Journal of Educational Statistics, 1980, Jenck;, C., et al. Who gets ahead? The determinants 5,191-207. of economic success in America. New York: Basic Eicholz, R. E., et al. Matematica para la educacion Books, 1979. primaria (Libro 1). Reading, Mass.: Addison- Meyer, J. W., & Hannan, M. T. (Eds.). National de- Wesley, 1968. velopment and the world system: Educational, Farrell, J. P., & Schiefelbein, E. Expanding the scope economic, and political change, 1950-1970. Chi- of educational planning: The experience of Chile. cago: University of Chicago Press, 1979, Interchange, 1974, 5, 18-30. Moss. M. H. Tests of basic experiences. Monterey, Friend, J., Searle, B., & Suppes, P. (Eds.). Radio Calif.: CTB/McGraw-Hill, 1970. mathematics in Nicaragua. Palo Alto, Calif.: Neumann, P. H. Publishing for schools: Textbooks Stanford Univerrilv, Institute for Mathematical, and the less developed countries (Staff Working Studies in the Social Sciences, 1980. Paper No. 398). Washington, D.C.: The World Fry, G. WY. Education and success: A case study of the Bank, 1980. Thai public service. Comparative Education Re- Searle, B., Friend, J., & Suppes, P. The Radio Math- iew, 1980. 24. 21-34. ematics Project: Nicaragua, 1974-1975. Palo Alto, Haron, I. Social class and educational achievement in Calif.: Stanford University, Institute for Mathe- a plural society: Peninsular Malaysia (Doctoral matical Studies in the Social Sciences, 1976. dissertation, University of Chicago, 1977). Disser- Searle, B., & Galda, K. Measurement of the effect of tation Abstracts International, 1978, .38, 7262A- radio mathematics lessons on student achievement. ,263A. In ,J. Friend, B. Searle, & P. Suppes (Eds.), Radio Hevneman, S. P. The evaluation of human capital in mathematics in Nicaragua. Palo Alto, Calif.: Milalauz (Staff Working Paper No. 420). Washing- Stanford University, Institute for Mathematical ton, D.C.: The World Bank, 1980. Studies in the Social Sciences, 1980. Heyneman,S.P.,&Currie,J.K. Schoolingacademic Searle, B., Sheehan, J., Gonzalez, J., & George, E. performance and occupational attainment in a Patterns of promotion and wastage for Nicaraguan non-industrialized society. Washington, D.C.: first-grade students. In P. Suppes, B. Searle, & J. University Press of America, 1979. Friend (Eds.), The Radio Mathematics Project: Heyneman, S. P.. Farrell, J. P., & Sepulveda-Stuarto, Nicaragua, 1975-1976. Palo Alto, Calif.: Stanford M. Textbooks and achievement: What ue knowt University, Institute for Mathematical Studies in the (Staff Working Paper No. 298). Washington, D.C.: Social Sciences, 1978. The World Bank, 1978. Shoemaker, D. A note on allocating items to subtests Heyneman, S. P., & Jamison, D. T. Student learning in multiple matrix sampling and approximating in Uganda: Textbook availability and other factors. scandard errors of estimate with the jacknif . Comparative Educarion Revieu, 1980, 24, 206- Journal of Educational Measurement, 1973, 10, 220. 211-219.  IMPACT OF TEXTBOOKS AND RADIO ON MATHEMATICS ACHIEVEMENT 567 Smart, M. The Densu Times: Self-made literacy. Academic Press, 1976. Development Communication Report, 1978, 21, Wiley, D. E., & Harnischfeger, A. Explosion of a myth: 1-4. Quantity of schooling and exposure to instruction, Suppes, P., Searle, B., & Friend, J. The Radio Math- major educational vehicles. Educational Re- ematics Project: Nicaragua, 1976-1977. Palo Alto, searcher, 1974, 3, 7-12. Calif.: Stanford University, Institute for Mathe- The World Bank. World development report, 1979. matical Studies in the Social Sciences, 1978. Washington, D.C.: Author, 1979. Thorndike, R. L. Reading comprehension in 15 The World Bank. Education sector policy paper. countries. New York: Halsted Press, 1973. Washington, D.C.: Author, 1980. (a) Wiley, D. E. Another hour, another day: Quantity of The World Bank. World development report, 1980. schooling, a potent path for policy. In W. J. Sewel, Washington, D.C.: Author, 1980. (b) R. M. Hauser, & D. L. Featherman (Eds.), Schooling and achievement in American society. New York: Received October 13, 1980 *