Inside Indonesia's mathematics classrooms : a TIMSS video study of teaching practices and student achievement

 54936 KEMENTERIAN PENDIDIKAN NASIONAL I . Kingdom of the Netherlands BANK DUNIA I THE WORLD BANK THE WORLD BANK OFFICE JAKARTA Indonesia Stock Exchange Building, Tower ll/12-13th Fl. Jl. Jend. Sudirman Kav. 52-53 Jakarta 12910 Tel: (6221) 5299-3000 Fax: (6221) 5299-3111 Printed in November 2010 INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: A TIMSS video study of teaching practices and student achievement is a product of staff of the World Bank. The findings, interpretation s and conclusions expressed herein do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the government they represent The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomination and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement of acceptance of such boundaries. Cover photo credit: Dwi Agus Kurniawan (SMP Negri 1 Jatiluhur)  Report No. 54936-ID INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: A TIMSS video study of teaching practices and student achievement Human Development Department East Asia and Pacific Region  • Table of Contents Acknowledgments vi List of Abbreviations vii Executive Summary 9 Section 1 Background and Context 17 1.1. Background 17 1.2. The Teacher Law 18 1.3. Why do a Video Study 19 Section 2 Design 20 2.1. Conceptual Framework 20 2.2. Objectives 21 2.3. Research Questions 22 Section 3 Methodology and Scope 24 3.1. Justification of a Video Study 24 3.1.1. Interviews with teachers 24 3.1.2. Teacher questionnaires 25 3.1.3. Live observation of classrooms 25 3.1.4. The video study approach and its advantages 26 3.2. Collection of Data 26 3.2.1. Unit of Study and Ana lysis 26 3.2.2. Sampling 27 3.2.3. Data , 28 3.2.4. Data Cbllection Procedures 29 3.3. Data Coding 29 3.4. Data Affillysis 30 3.5. The ffesearch Team 30 3.6. Achieved Sample and Problems Encountered 30 3.7. Lessons Learned through the Implementation of the Video Study 31 Section 4 Video Results and Cross-Country Comparisons 32 4.1. Teacher Background 32 4.1.1. Teacher Education Level 32 4.1 .2. Teacher Certification 33 4.1.3. Teaching Experience in Mathematics 33 4.1.5. Teacher Gender 35 4.2. Lesson Structure 35 4.2.1. The Duration of the Lessons 35 4.2.2. Amount ofTime Spent Studying Mathematics 37 4.2.3. The Role of Mathematical Problems 38 4.2.4. Time Used to Review, Learn New Content, and Practice 40 4.3. Lesson Content 46 4.3.1. Level of Mathematics Evident in the Lessons - Complexity of the Problems 46 4.3.2. Type of Mathematics Evident in the Lessons- Problems with Proofs and Applications 47 4.4. Instructional Practices 49 4.4.1. Teaching Strategies 49 4.4.2. How Mathematical Problems Were Presented and Solved 49 4.4.3. Opportunities to Talk 53 INSIDE INDONESIA'S MAT HEMATICS CLASSROOMS: ll A TIMSS video study of leaching practi ces and student achievement  Table of Contents 4.4.4. Resources Used During the Lesson 55 Section 5 Classroom Patterns: Indonesia's "Lesson Signature" 57 5.1. Method of Constructing the Lesson Signature 57 5.2. Pattern of Mathematical, Non-mathematical and Mathematical Organization Time 57 5.3. Pattern of Purpose of the Lesson Segment 58 5.4. Problem vs. Non-problem 60 5.5. Public vs. Private Interaction 61 5.5.1. Pub lic Interaction Breakdown (Teacher, Teacher and Student, Student) 62 5.5.2. Private Interaction Breakdown 62 5.6. Teaching Strategy 63 5.7. Description of the Typical Pattern by the Study Team 64 Section 6 Regression Analysis to Identify Relationsh ips between Teaching Practices and Student Mathematics Scores 66 6.1. Methodology of Regression Analysis 67 6.1. 1. Steps in Framework Development 67 6.1.2. Use of Mu ltiple Data Sources 69 6.1.3. Recognizing the Limitations 69 6.1.4. Model Development 70 6.2. Regression Results 73 6.2.1. Identification of Control Variables 73 6.2.2. Ana lysis of Re lationship between Teaching Practices and Student Mathematics Scores 75 6.2.3. Summary of Regression Results 81 Section 7 Summary and Implications 83 7.1. Pos itive Aspect s of Mathematics Teach ing in Indonesia 83 7.2. Potential Areas for Improvements in Mathematics Teaching 83 7.3. Add itional Observation Notes from the Videos 84 7.4. Implications for Educational Policies 84 7.5. Implications for Teachers 85 7.6. Conc luding Remarks 86 Appendix 87 Appendix 1: Indicators for the Preliminary Research Questions 87 Appendix 2: Indicators and Data Sources for the Research Questions 88 Appendix 3: Summary of Definitions 91 Append ix 4: Comparison of Full Sample Results with Subset 95 Appendix 5: Reg ression Resu lts 98 Appendix 6: Study Costs 99 References 101 List of Tables Tab le 4.1 Duration of lessons (in m inutes) 36 Ta bl e 4.2 Time used for mathematical work, mathematical organization, and non-mathematical work (minutes) 37 Tab le 4.3 Time (in minutes and seconds) used for review, new content, practice and assessment 41 Table 4.4 Segment length (in minutes) for pub lic interaction and private interaction 42 Tab le 6.1 Grouping of teaching techniques and classroom practices 70 Table 6.2 Models used for regressions 72 Table 6.3 Legend for presentation of statistical significance and direction of variab les in the regression 73 111  Table 6.4 T-statistic results for home, student, schoo l, class, community and teacher background variables (with mathematics examination score as the independent variable) 74 Table 6.5 T-statistic results for structure of time 76 Table 6.6 T-statistic results for purpose of lesson segment 76 Table 6.7 T-statistic results for public (full class) vs. private (individual and group) Interaction 77 Table 6.8 T-statistic results for types of public interaction 77 Table 6.9 T-statistic results for types of private interaction (as percentages of total private interaction) 78 Table 6.10 T-statistic results for teaching strategies 78 Table 6.11 T-statistic results for problem and non-problem time 79 Table 6.12 T-statistic results for use of applications and proofs 79 Table 6.13 T-statistic results for methods for setting up problems 79 Table 6.14 T-statistic results for types of mathematics problem language 80 Table 6.15 T-statistic resu lts for the use of instruments during class 80 Table 6.16 T-statistic results for lesson planning 81 Table 6.17 T-statistic Results for Teacher Influences 81 Table A6.1 101 list of Figures Figure 1.1 Estimated impact of high vs. low performing teachers on student achievement 18 Figure 2.1 Conceptual Framework 20 Figure 4.1 Educational background of teachers: percent with a mathematics degree 33 Figure 4.3 Average class time of mathematics teachers in mathematics class (in hours) vs. other subjects (not including non-class workload) 34 Figure 4.2 Years of experience in teaching mathematics and comparison with other countries 34 Figure 4.4 Number of classroom teaching hours per week in mathematics 35 Figure 4.5 Length of lessons in mathematics 36 Figure 4.6 Ranking of class length in minutes, from low est to highest 37 Figure 4.7 Percentage of time used for learning mathematics 38 Figure 4.8 Mathematical time divided into Problem vs. Non-Problem Segments (percent of total) 39 Figure 4.9 Number of independent problems solved in the lessons 40 Figure4.10 Average number of independent problems solved in the lesson and average length of time in minutes 40 Figure 4.11 Duration for different activities in Indonesia and other countries 42 Figure 4.12 Percentage of time for public interaction and private interaction 43 Figure 4.13 Publi c interaction breakdown: teacher and student involvement 43 Figure 4.14 Private interaction breakdown: time used for group work and individual work 44 Figure 4.15 Percent of lessons w hich included at least one goal statement 44 Figure 4.1 7 Percent of lessons w ith at least one interruption from outside 45 Figure4.16 Percent of lessons w hich include at least one summary statement 45 Figure 4.18 Level of complexity of problems 47 Figure4.19 Average percentage of problems per Grade 8 mathematics lesson that were applications 48 Figure 4.20 Average percentage of problems per Grade 8 mathematics lesson that inc luded proofs 48 Figure 4.21 Time for different learning activities in mathematics lessons 49 Figure 4.22 Average percentage of problems per mathematics lesson of each problem statement type 50 Figure 4.23 Problems related to the real world and using mathematical language and symbols only 51 Figure 4.24 Percent of classes with a given number of problems with more than one solution method 52 Figure 4.25 Percent of lessons with at least one problem where more than one solution is presented 52 Figure 4.26 Percent of lessons that contained at least one examining-methods problem 53 Figure 4.27 Percent of lessons that included a lesson summary 53 . - INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: IV A TTMSS vid eo stud y of teaching practices and student achi evement Figure 4.28 Average number of words spoken by the teacher and students during a lesson 54 Figure 4.29 Average number of teacher words to every one student word per lesson 54 Figure 4.30 Average words per sentence spoken by teachers 55 Figure 4.31 Average words per sentence spoken by students 55 Figure 4.33 Percentage of lessons which used calculators 56 Figure 4.32 Use of various resources (proportion of videotaped classes where the given resource was used) 56 Figure 5.1 Lesson Signature of layer 1: Mathematical, non-mathematical and mathematical organization time 58 Figure 5.2 Lesson Signature of the purpose of segments: Review, new content, practice and assessment 59 Figure 5.3 Lesson Signature of the purpose of segments for other countries 60 Figure 5.4 Lesson Signature of problem vs. non-problem mathematics time 61 Figure 5.5 Lesson Signature: Public (full class) vs. private (small group and individual) interaction 61 Figure 5.6 Lesson Signature of publ ic interaction breakdown 62 Figure 5.7 Lesson Signature of private interaction breakdown 63 Figure 5.8 Lesson Signature of discussion, exposition, investigation, practical work and problem-solving 64 Figure 6.1 Estimated influence of key factors on student achievement 67 Figure 6.2 Illustration of teacher background vs. classroom instruction and teaching practices 68 Figure 6.3 Framework used in analyzing classroom instruction and teaching practices 69 v  • Acknowledgments This report is the result of a joint effort between the World Bank and the Ministry of National Education (MON E). The team of authors which produced this report is gratefu l to the officials and staff of the Ministry of National Education (MONE) for their overa ll support. Key counterpart units with in MONE include the Directorate General of Quality Improvement ofTeacher and Education Personnel (PMPTK) and the Testing Center (Puspendik) under the Research and Development Department (Bal itbang). Special thanks are in order to Prof. Dr. Fasli Jalal, the Vice Minister of Nationa l Education, w ho provided the vision and initial provisions to launch the study. The team is grateful to Arnold van der Zanden (First Secretary Education, Royal Netherlands Embassy, Indonesia) for his insightful comments and to Dr. Baedhowi (Director General for Quality Improvement ofTeacher and Education Personnel), Burhanuddin Tolla (former Head of the Testing Center, Puspendik) under the Research and Development Department (Balitbang), and Bastari (Head of PISA, TIMSS, and PIRLS in Puspendik) for their inputs into the report. It shou ld be noted that while inputs of various officia ls have been in corporated into the report, the policy recommendations in this document do not necessari ly reflect the policies of the Government of Indonesia or the Government of the Netherlands. The study was designed by Dr. Frederick Leung (Hong Kong University). The design was based heavily on the Teaching Mathematics in Seven Countries: Results from the TIMSS 1999 Video Study (Hiebert et al, 2003) in order to allow for comparison of Indonesia's results to those of the other seven other countries that conducted similar studies. The authors of this report were Frederick Leung and Andrew Ragatz (DPh il candidate, Oxford University). Important contribution s to the management of the study activities and preparation of this report were made by Ratna Kesuma (Operations Officer, Education Unit, East Asia and Pacific Region, World Bank) and Susie Sugiarti (Operations Assistant, Human Development Sector Department, East Asia and Pacific Region, World Bank). The Study Team consisted of 10 core team members who are sen ior mathematics instructors from the Center for the Development and Empowerment of Mathematics Teachers and Education Personnel (P4TK), the Educationa l Quality Assurance Institution (LPMP), and some junior secondary schools. The study team was led by Puji lryanti (P4TK) and Ratna Kesuma. Initial data analysis and reporting were performed with the assistance of experts from universities. Technical support was provided by a team from MoNE. Countless hours were put in by the study team wh ich tirelessly analyzed and reanalyzed the videos to provide the essentia l coded results and produced the initia l analysis that contributed to this report. This team was led by Puji lryanti and cons isted of Adi Wijaya, Budiharjo, Erwin Roosilawati, Krismanto, Rachmadi Widdiharto, Reubun, Su hendro, Suwarkono, Yuniarti and lsfarudi. The analytical work was generou sly supported by the Dutch Education Support Trust Fund under the technical leadership and management of Mae Chu Chang (Lead Educator and Sector Coordinator, Human Development Sector Department, World Bank). The report was prepared under the supervis ion of Mae Chu Chang and w ith the overall guidance and support of Eduardo Velez Bustillo (Education Sector Manager, East Asia Human Development, World Bank). The peer reviewers for the report included Harry Patrinos (Lead Economist, Human Development Network, World Bank), Helen Abadzi (Sr. Education Spec iali st, Human Development Network, EFA-FTI Secretariat, World Bank) and Carlos Ruano (Education Specia li st, Human Development Network, EFA-FTI Secretariat, World Bank). Government reviewers included Doctors Baedhowi, Dasuki, Suraranata, and Suryadharma of PMPTK. Indonesia Country Director: Stefan G. Koeberle East Asia Human Development Sector Director: Emmanuel Jimenez East Asia Education Sector Manager: Eduardo Velez Bustillo Indonesia Human Development Sector Coordinator: Mae Chu Chang • INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: VI A TIMSS video study of teaching practices and studen t achievement • List of Abbreviations Abbreviation Description BALITBANG Research and Development Department within the M inistry of Nationa l Ed ucation BAN-PT National Board of Accreditation for Higher Education BAPPENAS National Planning Agency BERMUTU Better Education through Reformed Management and Universal Teacher Upgrading BOS Major national government program that provides grants to schools for operational costs BSNP National Education Standards Agency CAR Classroom Action Research CPD Continuous Professional Development 01,2,3,4 Post-secondary diploma (1-year), (2-year), (3-year), (4-year) DIKTI Directorate General of Higher Education GOI Government of Indonesia GTT Non-permanent, school-hired teacher GTY Non-permanent, school-hired teacher in private schoo ls HEI Higher Education Institution (university, institute, school of higher learn ing, academy, or polytechnic) IDR Indonesian Rupiah IKIP Teacher and Education Stud ies Institute KKG Teacher Working Group for primary schools LPMP Institute for Educational Quality Assurance LPTK Teacher Training Institutions (faculties) within universities MGMP Teacher Working Group for secondary schools MONE Ministry of National Education M&E Monitoring and Evaluation OECD Organization for Economic Co-operation and Development PGSD LPTK course to upgrade elementary teachers to S1 PGSMTP Teacher tra ining co llege for junior secondary school teachers PIRLS Progress in International Reading Literacy Study PI SA Program for International Student Assessment PMPTK Directorate General for Quality Improvement ofTeacher and Education Personne l PNS Civil servant PPG Post-graduate professional course of one or two semesters to gain certification pp Government Regulation P4TK Center for Development and Empowerment ofTeachers and Education Personnel (a national agency) VII  Abbreviation Description QITEP Directorate General for Quality Improvement of Teachers and Education Personnel (also termed PMPTK) RENSTRA 5-year Strategic Plan RPL Recognition of Prior Learning 51 Degree equivalent to Bachelor's Degree 52 Degree equivalent to Master's Degree 53 Degree equivalent to PhD so Primary school SKS Credit points gained by university study or its equivalent SMA Senior secondary school SMP Junior secondary school SPG A now discontinued teacher training secondary school STR Student-Teacher Ratio TIMSS Trends in International Mathematics and Science Study UT Open University uu National Law INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: Vlll - A TIMSS video study of teaching practices and student achievement Executive Summary The following report is the first of a two-stage video study to examine teaching practices and activities in Indonesian classrooms. The first stage is linked to results of the 2007 Trends in International Mathematics and Science Study (TIMSS) where 100 of the 150 classes that participated in the TIMSS examination also participated in this additional video study component. The second stage will also involve 100 classes that will participate in the 2011 exam. The second phase will follow the same methodology, allowing for comparison across years, but will also involve more in-depth analysis of the links between teaching practices and student outcomes and how teaching practices are influenced by teachers' belief systems and subject content knowledge. Indonesia has been a committed participant in the TIMSS, Program for International Student Assessment (P ISA) and the Progress in International Reading Literacy Study PIRLS international standardized student examinations for many years and is one of the few non-OECD (Orga nization for Economic Cooperation and Development) countries to participate so fu lly. Indonesian student performance in these examinations has been relatively low, even when taking socio-economic levels into account. For example, for mathematics Indonesia ranked 36'h out of 48 participating countries in TIMSS 2007, and its score of 397 was more than one standard deviation below the international average (Mullis eta/, 2008). The results have been useful in providing an indication of Indonesia's relative standing in student ach ievement and its progress over time, but the real challenge is to take the next step and translate the results into an understanding of the factors leading to the test scores and what might be done to enhance student achievement in Indonesia. Teachers certainly play a key ro le in improving student outcomes, and since 2005 Indonesia has undertaken a major teacher reform effort. A cornerstone po licy of the reform is the requirement that all teachers have a four- year degree and become certified by 2015. As part of the certification process, teachers are required to submit a portfolio which is meant to capture information that demonstrates both teacher competency and performance. In the debate of how teachers should be evaluated for certification, a key question arises: What makes a high- quality, effective teacher? To answer this question, and in the larger context of the teacher reform, it is vital to understand Indonesia's current teacher situation in terms not only of teacher qualifications but also of teaching practices and teaching effectiveness. The Indonesia Video Study provides an in-depth analysis of teaching practices leading to insights that can be applied to Indonesia's teacher reform effort. The first phase of the study examined 100 classes across Indonesia 9  that participated in the 2007TIMSS international exam . It is said that a picture is worth a thousand words; similarly, a video study is able to go beyond traditional surveys or observational techniques to capture a rich set of both quantitative and qualitative information. The analysis derived from the first phase of this study has provided unique, comprehensive insights into what happens in Indonesia's classrooms and has led to the identification of relationships between teaching practices and student achievement. The study was designed to allow for analysis from three key angles. First, it followed the same methodology used in a video study of seven countries that participated in the 1999 TIMSS examination (including Australia, the Czech Republic, Hong Kong, Japan, the Netherlands, Switzerland and the United States). This provided not only a high-quality, proven coding scheme but also the context for a comparison of Indonesia's results with other countries. Second, because the video coding also gives a second-by-second breakdown of classroom time, the pattern of each activity throughout the lesson can also be seen, giving what has been labeled a "lesson signature" for Indonesia's classrooms. Finally, the fact that the classes involved in the study also participated in the TIMSS examination allowed for the analysis of the relationship of classroom practices and teaching techn iques w ith student mathematics scores. Additional data on students, teachers, schools and classrooms collected for TIMSS could also be used in this process. Cross-Country Comparisons The study focused on key dimensions that frame mathematics classroom practices: Structure of Lessons, Content of Lessons, Actions of Participants, Instructional Practices and Classroom Climate and Resources. The cross-country comparison highlighted similarities and key differences between Indonesia's classrooms and those of the seven other countries. The following is a summary of major findings: Structure of Lessons: The averag e duration of classes in Indonesia was significantly longer than in comparator countries, with lessons lasting 70 mi nutes compared to the next closest country average of 51 minutes. This is mainly due to Indonesia's practice of grouping t wo mathematics periods together and having classes only two or three days per week. 1 It does not translate to more mathematics time per week (with Indonesian students actually receiving fewer weekly hours than the other countries), and, as was evident in some classrooms, there are concerns that C)rade 8 students may have trouble concentrating for such an extended period of time. Classroom time was divided into three areas: (1) mathematics, (2) non-mathematics, and (3) mathematics organization time. While most countries had at least 96% of class time dedicated to mathematics, in Indonesia's case it was only 89%. Much more ofthe lesson time was spent on organizational work (8%) and non-mathematics time (3%) than in other countries. Mathematics time w as also broken into problem-solving and non problem-solving time. Only 76% of mathematics time was devoted to problems, whereas in other countries problem time accounted for between 81 % and 96%. Indonesia also had relatively few independent problems 2, w ith 3.3 problems on average per lesson. In other countries the number was between 3 and 13. Indonesia did tend to spend more time per independent problem, however, with an average of 6.6 minutes compared to most other countries spending between 3 and 5 minutes. Typically schools do one of two schedules: two periods are grouped together two days per week (2-2), or one two-hour class is held one day and two one-hour sessions are held on ot her days (2-1-1 ). 2 Independent problems include group and individual seatwork problems as we ll as problem s worked on w ith the whole class. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 10 - A TIMSS video study of teaching practices and student achi evement  Executive Summary The purpose of lesson time was broken into: (1) the review of previous content, (2) the introduction of new content, (3) practice, and (4) assessment. In Indones ia's classrooms the students were given relatively more opportunity to practice, with 37% of all mathematics time dedicated to practice compared to between 16% and 37% in other countries. Most time was spent on introducing new content (43% compared to between 22% and 60% in other countries), but Indonesia spent much less time reviewing material from previous lessons, with on ly 10% compared to between 24% and 58% in other countries. Content of Lessons: The percent of mathematics problems in Indonesia's classrooms that was determined to be of high complexity was on ly 3%, which was much lower than in other countries which ranged between 6% and 39%.1ndones ia also had relatively few problems involving applications but relatively more problems involving proofs. The cho ice of alternative solution methods was not stressed; few students, therefore, had a chance to examine different ways of solving problems. Actions of Participants: One of the most striking resu lts of the study was how few words were spoken by both teachers and students compared to the other countries. Counts of words spoken focused on only full-class interaction (rather than group or individual work) and were standard ized to indicate the number of words spoken over 50 minutes. Indonesian teachers spoke fewer than half the number of words as in other countries, w ith only 2,633 words in an average lesson compared to 5, 198-5,902. Student words were simi lar, with only 194 compared to 640-1,108 in other countries. A further striking feature was that the teacher-to-student speaking ratio was much higher than in other countri es, with teachers speaking 28 words for every word spoken by students, compared to 8-16 words elsewhere. This indicates that students in Indonesia tend to participate less in a verbal sense. While this is an indicator of student participation, it does not necessarily measure the level of student engagement. Analysis of the classroom videos revealed classes where the students were engaged but not necessarily speaking frequently. Stil l, the lower verbal communication for both teachers and students may signal less active and engaged participation. Time was also divided into public (fu ll -class) and private3 (small-group or ind ividua l) interaction. Indonesia's distribution of 57% for public vs. 43% for private interaction falls in the middle compared to other countries. When public interaction was examined, the most common method was teacher lecture which accounted for 59% of all public interaction time, while 19% of the time was devoted to student-on ly work (students presenting) and 22% of the time to student and teacher discussion. For private interaction, 55% was spent in small group work while t he remaining 45% was individual work. Indonesia tended to use the technique of working in smal l groups more than in other countries. Instructional practices: Compared to other countri es, Indonesia had relatively more lessons that included goal statements and lesson summaries. This, theoretically, should lead to improved clarity and flow of the lessons. Use of goal statements and lesson summaries is part of Indonesia's teacher training guidelines, and the video study results show that this training has permeated into the classroom settin g. 3 Private interaction is defin ed to be activities where all students work at t heir seats, ind ividua lly, in pairs, or in small groups, while the teacher often circulates around the room and interacts privately w ith the groups or individual students. ll  The teaching strategy most commonly used was exposition (teacher explaining w hile students listen and answer closed questions), which made up 52% of all teach ing strategy t ime. Problem-solving was the next most uti lized technique, at 20%, followed by discussion, practical work and investigation at 15%, 1Oo/o and 3%, respectively 4 The mathematical processes suggested by problem statements were divided into three types: (1) use procedures5, (2) state concepts6, and (3) make connections 7 Indonesia's use of stating concepts in 35% of problems was much higher than the comparison countries (between 5% and 24%). The use of procedures (41 o/o) was lower than in all but one of the other countries (between 41 o/o and 84%) Classroom Climate and Resources: The environment of Indonesia's classrooms was generally cond ucive to learning. The classes were mainly conducted with few outside interruptions. The quality of the classrooms varied, wi t h most being well-l it and well-resourced, but others operated in di lapidated classrooms with lim ited resources. Resources used in t he classroom varied, with only 9% of classes using projectors and 13% using ca lculators, but textbooks were used in 93% of the classes. Rea l-world objects were used in 28% of t he classes; this was higher t han all comparator countries wh ich only used them between 4% and 21 o/o of the classes. Indonesia's Lesson Signature By coding what takes place in the classroom second-by-second and evaluating the video over many different layers, a complete t imel ine of what happened in each ind ividual classroom emerged. By merging all t he classroom results, common patterns of the country cou ld be seen. This is known as the "Lesson Signature". All class times were standardized by breaking each lesson into percentiles from 1 (the beginning of class) to 100 (the end of class). Indonesia's lesson signature was formed using the dimensions ment ioned above, with the following striking features emerging: General Pattern: The general pattern identified by the lesson team was that classes were general ly segmented into t hree stages. The introduction stage typica lly involved reviewing homework from the previous class. This was followed by the development stage which conta ined introduction of new content. Teachers typically began this stage by motivating students with an explanation of the importance of studying the lesson, followed by questions of prerequ isite knowledge that was used in the development of the new materia l. For the closing stage, teachers (sometimes with the involvement of students) built summaries of the day's lesson and gave students tasks to work on as homework problems. The analysis of the data quantified this general pattern. The following points emerged from the study of t he second-by-second timeline analysis: 4 Definitions of these terms are given in the main text. 5 Problem statements that suggest the problem is typically solved by applying a procedure or set of procedures. These included arithmetic with whole numbers, fractions and decimals; manipulating algebraic symbols to simplify expressions and solve equations; finding areas and perimeters of simple plane figures; and so on. 6 Problem statements that call for a mathematical convention or an example of a mathematical concept. 7 Problem statements that imply the problem would focus on constructing relationships among mathematical ideas, facts or procedures. Often, the problem statement suggests that students will engage in special forms of mathematical reasoning such as conjecturing, generalizing and verifying. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 12 - A T!MSS video study of leaching practices and student ach ievement  Executive Summary • Lesson segment purpose: The review of previous materia l was almost completely performed within the first 20% of class t ime, and most classes spent fewer than eight minutes on it. Indonesian classes began work on new material earlier in the lesson than in most countries and spent the second ha lf of class conducting practice. • Problem vs. non-problem mathematics time: As mentioned earlier, Indonesian classes spent more t ime on non-problem mathematics work. This tended to take place at the beginning of class and often took the form of giving definitions, discussing concepts or describing the history of a mathematics problem. • Mathematics, non-mathematics and mathematics organization: While mathematics time made up 89% of all class time, the beginning and end of class tended to be dedicated to non-mathematics t ime. Often this came in the form of prayer, da ily rituals or discussion of other non-mathematics subjects. The class also often concluded w ith a dai ly ritua l. Mathematics organization time also tended to occur at the beginning and end of class, but there was significant time devoted to it in the middle of class, often du ring trans itions between interaction types (switching to group work, etc.). • Public and private interaction: While 60% of class time was publ ic, there was a distinct pattern of beginning class with public interaction as well as a less pronounced pattern of ending class with public interaction. Private interaction was most prevalent between the 40% and 80% marks of the class timeline. • Problem-solving strategy: Exposition made up over half of all t ime working on problems. The exposition tended to take place more in the earlier portion of the class, whi le problem-solving tended to take place in the latter portion of the class. Relationships between Teaching Practices and Student Mathematics Scores Regression ana lysis allows for the detection of relationships between various teaching techniques and student mathematics scores. Caution must be exercised in the interpretation of the regression results because the TIMSS results only provide a snapshot rather than a "before and after" result. Relationships cannot, therefore, be interpreted to show cause and effect. Still, after contro lling for key student, household, school and class characteristics, statistically significant relationships between teaching practices and student mathematics scores emerged to provide useful insights into what may be effective teaching in Indonesia. These included the fol lowing: • A constant theme from the regress ion results was that classes with higher student involvement (e.g., student presentations, teacher-student interaction, and student problem-solving) had higher test scores. Traditional teacher lecturing, on the other hand, had a negative relationship. Although cause and effect cannot be determined, these results indicate that more student-centered learning can lead to better learning outcomes. • The percent of time spent on problem mathematics time (as opposed to non-problem mathematics time) had a positive relationship with student scores. This is interesting to contrast with the fact that Indonesia had the lowest percent of problem mathematics time compared to other countries. • Although rarely used, assessment time and assessment-related activities such as quizzes had a pos itive relationship to test scores. • The process of setting up a problem w ith "use a procedure" had a negative relationship w ith student achievement. "Make a connection", on the other hand, was not as common but had a positive relationship with student mathematics scores. • Students in classes where more proofs were introduced tended to have higher mathematics scores. 13  • Classes where the introduction of problems used mathematics language tended to have higher mathematics scores than those that introduced problems w ith real-life contexts. • Use of projectors tended to have a positive relationship w ith mathematics scores whi le use of textbooks tended to have a negative relationship. • In lesson planning, students in classes w ith teachers that specified they developed the lesson plan with another teacher tended to have higher mathematics scores. Strengths of Teaching Practice in Indonesia's Classrooms Many positive results emerged from the ana lys is, indicating that Indonesia is emp loying many good practices. In some cases, the results surpass those of other countries. Among the results: • The classroom environment was often conducive to learning, and mathematics teaching in Indonesia was mainly conducted with few outside interruptions. • Students were given ample opportunity to practice what they had just learned in the lesson. • Compared to other countries, students had more time working in small groups. • Indonesia had relatively more lessons with goal statements and lesson summaries which should lead to im proved clarity and f1ow of the lessons. • There was more use of real-life objects in the lessons than in comparator countries. Focus Areas for Improvement Th e findings of this video study also point to some areas for improvement in mathematics classroom organization and instructional practices in Indonesia: • The duration of Grade 8 mathematics lessons was rather long compared to other countries. As a resu lt, students perhaps could not concentrate on the subject matter to be learned for the whole duration of the lesson. • Much of the lesson time was spent on organization work, with a res ult that less lesson time was spent on teaching and learn ing mathematics. • Not much time was spent on reviewing what had been learned in past lessons before introducing new content. • Relatively little homework was given, and much lesson time was consumed on practicing. • Both teachers and students spoke relative ly few words in the lesson, and their statements were generally short. • The ratio of student words to teacher words was very low compared to other countries. • Very few of the mathematics problems dealt with were of high complexity. • There were few problems involving appl ications. • The choice of different solution methods was not stressed. • Calculators were rarely used in classrooms. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 14 - A TIMSS video study of teaching practi ces and student achievement  Executive Summary Additional Observation Notes from the Videos Although the coding of the videos provided for objective data analysis, it could not always capture what the observers of the videos could see. The study team (who are mathematics experts and practitioners themselves) noted interesting patterns and felt that certain activities were not being properly conducted. Recommendations include: • There is a need to apply better time management in the classroom and to use the time effectively to teach relevant content. • More emphasis should be put on higher order thinking in instructional delivery. • There was often a mismatch in the level of content coverage (i.e., the level and the amount of the content covered is equal to the level and the amount understood by a student). • There is a need to create an environment of enjoyable learning to maintain student engagement, involvement and attention. Policy Implications While any policy measures to be taken need to ensure that the many strengths of mathematics teaching in Indonesia as listed above are not lost, the various deficiencies above point to some specific improvement measures. First, while Indonesia's current qualification upgrad ing exercise introduced by the Teacher Law passed in December 2005 is moving in the right direction and should be applauded, it is important to remember that the mere upgrading of qua lifications is not suffic ient for high quality teaching . In particular, the educational background of the teachers should match the subjects that they are teaching. In the event that this is not the case, effective in-service professional development activities (including activities in Teacher Working Groups known as MGMP) need to be provided to ensure that teachers are able to build on their qualifications to develop expert knowledge in the field that they are teaching. Second, the organization of lesson time shou ld be reviewed. The average of 70 minutes per lesson may be too long for children of Grade 8 (although the regression results indicate that longer cla sses actually have a positive relationship with mathematics scores). More importantly, measures need to be taken to reduce the org an izational work of the teacher during the lesson so that more time can be devoted the most important activity in the classroom- that of teaching. Third, the policy to not allow the use of calculators in mathematics examinations should be reviewed. The calculator is not merely a calcu lation device. When used properly, it is an extremely useful tool for learn ing (e.g., in exploring number patterns). Finally, the policy of promoting student-centered learn ing appea rs to be a val id approach, with the more student-centered classes tending to have higher mathematics scores. The relatively low number of both teacher and student words compa red to other countries, as we ll as the relatively high amount of teacher speaking time compared to student time, indicates that the student-centered approach is not being fully implemented in many classrooms. Methods to further promote such an approach in mathematics shou ld be pursued. 15  Implications for Teachers Many of the problems dealt with in the Indonesian classroom were not of high complexity. While the teacher should always pitch the level of difficulty and complexity of the subject matter towards the level of the students, care should be taken to ensure that the difficulty level of the content is not too low. Developing flexibility in the approach to the solution of problems is an important aim of mathematics educat ion. This can be enhanced by more discussion with students on different ways of tackling problems (examining methods) and by encouraging different solutions to the same mathematical problem. Communication is another important aim of mathematics education. A noticeable finding of this study is the reticence of both teachers and students in the Indonesian classroom . Whi le this may be rooted in the Indonesian culture itself, teachers need to realize the importance of communication in the learning of mathematics. Students need to be given the chance and the encouragement to express themselves verbally. Assessment activities are very rare ly used, but it appears to have a strong positive relationship with student mathematics scores. Increased frequency in the use of assessment may ass ist in enhancing student learning. Indonesian students have very little homework relative to other countries. At the same time, a large amount of class time is devoted to conducting practice activities. While practice in class enables students to directly discuss problems with the teacher and other students, it appears that class time is too often being used to conduct practice that could be done as homework. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 16- A TIMSS video study of teaching practices and student achievement Section l Background and Context 1.1. Background Despite Indonesia's great success in achieving near universal enrollment at the primary and junior secondary level, many students still have low literacy and cognitive skills (Filmer, 2006). Indonesian stude nt performance in international stu dies has been re lati ve ly low, even when taking soc io-economic levels into account. For example, for mathematics Indonesia ranked 36th out of 48 pa rticipant countries in TIMSS 2007, and its score of 397 was mo re than one standard deviation be low the international average (Mu llis eta/, 2008); in the PI SA 2006 study, Indonesia ranked 51 s t of the 57 participan t countries (OECD, 2007) For science, Indonesia's score of 427 in TIMSS 2007 was sli ghtly better than that for mathematics, but it stil l ran ked 35th of the 48 countries (Martin eta/, 2008). For reading, Indonesia's score was 405 in the PIRLS 2006 study, which was also low, and it ranked 36th of the 48 partic ipant countries (Mu llis eta/, 2007). Some scholars have attributed the poor performance of Indonesian students to the curriculum. For example, t he teac hers of the classes part icipatin g in t he TIMSS 2007 examination indicated that only 20 out of the 39 topics assessed were, in fact, covered 8 Other factors, of course, can be offered to exp lain t he unsatisfactory situat ion in Indonesia, but sin ce school chi ldren learn most of the ir mathematics w ith the gu idance of their teachers, it is reasonable to expect that t eacher competence in subject matter and pedagogy are major factors in influencing student achievement. As a result, t he relationship betwee n teacher competence and student achievement has attracted m uch attention in the literature (Wright eta!, 1997; Da rli ng-Hammond, 1999; OECD, 2005; Hiebert and Grouws, 2007). Teachers play a critical role in student outcomes. According to a meta-analysis by Hattie (2003), wh ich synthesized 51 studies, teacher variables account for approximately 30% of the va riations in student achievement, after stu dent characteristics (49%), and are much great er than variables relating to schoo l, home, and peers (approximately 5-10% each). As Ba rber and Mona (2007) remarked, "the quality of an education system cannot exceed the qual ity of its teachers'; and research has shown that w hat teachers know and are able to do does improve the academic performance of their students (H ill eta/, 2005). Ma (1999), for example, in a study of elementary schoo l mathematics teachers in the United Stat es and Shanghai, Chi na, found that many Chinese teachers possessed a "profound understanding of fun damenta l mathematics'; and this profound understanding enabled t hem to invoke rich and relevant pedagogy in teaching elementary mathematics. 8 Education officials noted that the full curriculum, in fact, included all topics in the examination, but the teacher responses are more indicative of what students actually covered by the time they reached 8th grade. 17  Figure 1.1 Estimated impact of high vs. low performing teachers on student achievement 1 OQth percentile After 3 years with high •••':T 90th percentile c ...... 0 quality teachers , ••••• •• ~ E ffi ~ .. .. .. ...... .. 53 percentile .:::....... Ew .g~ SQth point difference Q) N o..:o percentile c ro ·····························). Q)""O ""0 c After 3 yea rs of low 37th percentile :::J C1l 0505 quality teachers Qth percentile-- - - - - - - - - - - - - - - -- Age 8 Age11 Source: Sanders and Rivers on the Tennessee Value-Added Assessment System (TVAAS) Students exposed to effective teachers have been shown to outperform dramatically those with ineffective teachers. What is the d iffe rence between a good teacher and a bad teacher in terms of learning achievement? Groundbreaking research by Sanders and Rivers on the Tennessee Value-Added Assessment System (TVAAS) estimated the impact of the quality of teachers on student ac hievement. The study found that if average eight-year old students (t hose scoring in the 50th percentile in a standardized examination) are given teachers of varying abilities, their later achievement leve ls diverge d ramatically. Specifically, one group had high abil ity teachers (i n the top 20%), and the other had low ab ility teachers (in the bottom 20%) over a 3-year period. At the end of the three years, performance had d iverged by more than 53 percentile points. Thus, by age 11, the upper group was sco ring in the 90th percentile, and the lower group, in the 37th percentile. Their researc h also indicates that lower achieving students benefit most sign ificantly from having hig her ability teachers. Cross-country comparisons have highlighted deficiencies in Indonesia's level of teacher quality and teacher support. For example, in a comparative study of the performance of Indonesia, Ma laysia and Singapore in mathematics in TIMSS 2003 by Martin and Mullis (2006), it was found that Indonesian students had more instructional time in mathematics but that Indonesian teachers had a lower level of forma l ed ucation, were less likely to have a degree in mathematics and had less professional support in improving content knowledge and teaching skills. Cultural context plays a critical role in student learning so that teaching practices that are effective in one country may not lead to the same outcomes in another country. In the TIMSS 1999 Video Study (Hiebert eta!, 2003), it was found that instructional practices in class rooms in different countries were also sign ificantly different, and even among high achieving countries, there was no single"best"method of teaching mathematics. For example, Japan and Hong Kong SAR are both high performers in the TIMSS exam, but teaching practices and classroom environments differ significantly; this high lights the need to understand cultural contexts in determin ing teacher effectiveness. In the case of Indonesia, it is important to understand w hat teaching practices are effective within the Indonesian context. 1.2. The Teacher Law In December 2005, the Indonesian Government passed a law (hereafter referred to as the Teacher Law} that sets minimum academic and professional requirements for teachers. As a consequence of the INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 18 - A TIMSS video study of teac hing practices and student achievement --------------------------------------------~~================================================================== Background and Context Teacher Law, there is a pressing need to upgrade about 80% of the almost three million Indonesian teachers w ho cu rrently do not satisfy the minimum requirements. In this connection, the World Bank (hereafter, the Bank) launched a project, known as BERMUTU 9, to support the Indonesian government in fulfilling this mission. The scale of investment by both the government and the Bank in this endeavor is huge, and so it is of paramount importance that the effect of the project be closely monitored. A central aim of the Teacher Law is to upgrade the quality of teachers and their teaching. The ultimate yardstick of success for this initiative is, of course, the improved academic performance of students. Since Indonesia is participating in important internationa l studies such as the lEA TIMSS and PIRLS and the OECD PISA studies, Indonesian student performance in these studies over the years provides a relatively objective meas ure of any change in outputs of the system. However, the more direct outcomes of thi s teacher upgrading initiati ve are improved teacher competence in the classroom, and a study that gauges the classroom performance of teachers wi ll provide an important indicator of the effect of the Teacher Law. 1.3. Why do a Video Study Classroom performance of teachers may be studied in a number of different ways, but in recent years the use of videos has proved to be a highly effective means of studying classroom activities (Hiebert eta/, 2003; see discussion below in the Methodology section of the Report). The Bank, therefore, decided to launch a video studyl 0 in parallel w ith the BERMUTU project as a means to monitor and better understand the classroom compete nce of teachers. The design of the video study is such that it links Ind onesia's participation in TIMSS 2007 and its intended partic ipation in TIMSS 2011 . The coinc idence of Indonesia's partic ipation in these t wo rounds of the TIMSS and the Teacher Law provides a golden opportunity for monitoring the effect of the Teacher Law through exploring the relationship between teachers' classroom performance and their students' TIMSS achieve ment in 2007 and 2011. In this video study, the subject of mathematics at Grade 8 was chosen in order to study the effectiveness of the initiatives under the Teacher Law and the BERMUTU project. Mathematics teaching in a sub-sample of the 2007TIMSS classroom sample wa s studied using videos, and the results were linked to student achievement in TIMSS 2007. The same study will be repeated in 20 11. So that this replicati on study can be carried out in a manner as close as poss ible to the 2007 study, clear documentation of both the implementation and the results ofthe 2007 study should be kept. This documentation is provided in the remaining chapters of this report. 9 BERMUTU stands for Better Employment and Reformed Management for Universal Teacher Upgrading. The word "bermutu" in Bahasa Indonesian means "of good quality". 10 Arguments for the use of videos in studying classroom teaching over the use of other means such as questionnaires and live observat ion were presen ted in the report of t he Pilot of this Video St udy by Asrijanty eta/ (August 2006). 19  Section 2 Design 2.1. Conceptual Framework A central aim of the study is to analyze the changes in teaching practice over time and to measure their effects on student achievement. The Teacher Law sets minimum academic and professiona l requirements for teachers. The assumption is that teachers with higher qualifications, and following various professional development activities organized in conjunction wi th the Teacher Law and those provided t hrough the BERMUTU project wi ll teach better in the classroom. This wi ll in turn lead to gains in academ ic achievement of their students. According ly, this video study is based on a very simple conceptual framework represented in the diagram below: Figure 2.1 l Teacher Law: sets More qualified minimum requirements teaching force for teachers Video Study Video Study (Phase 1) (Phase 2) r--- Improved Teachers' Teachers ' Classroom Classroom Instructional Practices Instructional Practices (base-line) l Student Achievement Improved Student (base-line: TIMSS 2007 ) Achievement (TIMSS 2011) 20  Design The video study should be conceived of in the context of the Teacher Law and what it purports to achieve. As can be seen from the simple model above, phase one of t he study documents teachers' classroom instructional practices in 2007. This w ill provide for the first time in the history of Indonesia a representative documentation of teaching in the Indonesian classroom. A comparison of this documentation w ith similar studies in other countries wi ll provide an understanding of instructional practices in the current Indonesian classroom in an international context. More importantly, this documentation of teaching practice wil l act as base-line data w ith which a replication study in 2011 can be compared . By 2011, it is envisaged that the measures of the Teacher Law will have had an impact on some of the teachers in the sample (since it is a random sample), and so a comparison of the instructional practices of these teachers against the practices of (1) sim ilar teachers in 2007 and (2) those teachers in 2011 who have not yet benefited from the Teacher Law should provide very good evidence of the effects of the Teacher Law. The framework hypothesizes that teachers' classroom instructional practices, both in the base-line year of 2007 and in 2011, will be related to student achievement as measured by the TIMSS scores. For the 2007 exercise, this will provide va lu ab le data for curriculum developers on the appropriateness of the curricu lum and the kind of instructional practices needed for enhanc ing student achievement. The similar statistical analysis to be performed in 201 1 will further test the effectiveness of the improved instructional practices as a result of the Teacher Law. 2.2. Objectives As mentioned in the conceptual framework above, the purpose of the video study is to monitor the success or otherwise of the initiatives under the Teacher Law and the BERMUTU project. The data collected will enable stakeholders to gain a better understanding of t each ing practices in Indonesian classrooms, to see how these practices change over time and how these changes are related to changes in po licy, and to gain insight on how these changes have an impact on student ach ievement. A secondary purpose of the project is capacity building for relevant personnel in the country. BERMUTU is a project with partial funding from the World Bank and the Netherlands in support of the Indonesian government's initiatives under the Teacher Law, and, as such, the expectation is that the video study should be conducted main ly by personnel in the Indonesian government. The Bank is primarily playing an advisory and supportive role. This not only enhances owners hip of the project by local officials and ed ucators; it w ill hopefull y also help t hem acqu ire competence in the use of videos as a research tool. A th ird purpose of the study is to document teaching practices over time and to integrate video into teacher professional development activities. As a by-product of the video study, the data collected may be used for studying aspects of the Indonesian classrooms other than those focused on in this study. Th e classroom videos can be used to produce an archive for future use and in teacher profess ional development activities. The BERMUTU project is supporting teacher working groups made up of teachers from 6-10 neighboring schools who meet regu larly to conduct professional development activities. These teacher working groups provide an ideal environment for utilizing video as a means for self-assessment and teacher quality improvement. To summarize, the objectives for the 2007 component of the study are as follows: 1. To characterize classroom teac hing-learning behavior w ith reference both to curriculum intention and to classroom characteristics in other countries 2. To provide baseline data for comparison with data to be collected in 2011 3. To relate classroom teaching -learning behavior w ith student achievement in TI MSS 2007 and to determine which teaching methods are effective so as to inform ongoing teaching improvement programs 4. To produce an archive of classroom videos for use in research and teacher development in the future 5. To develop the capacity of relevant personnel in Indonesia. 21  2.3. Research Questions Sources of research questions The research questions were developed through analysis of the Indonesian mathematics curricu lum, documents related to the Teacher Law and a review of the literature on mathematics education and video studies. To characterize classroom teaching and to relate classroom teaching characteristics with student ach ievement, it is necessary to formulate clearly measurable research questions that can be answered from the video data. The research questions came from three sources: 1. The Indonesian mathematics curriculum A set of preli m ina ry research questions was formulated after a careful study of the Indonesian mathematics curricu lum. The curriculum issued by the Indonesian government sets out w hat students are expected to learn in the discipli ne of mathematics. These were translated into expected teacher behaviors through asking the question:"What teacher behaviors are needed for students to be able to learn this particular area of mathematics?" Research questions based on the mathematics curricu lu m document 11 were formu lated as follows: • Does the teaching help students understand mathematics concepts? • Does the teaching enhance student communication in mathematics? • Does the teaching enhance student ability in reasoning? • Does the teach ing help develop student ab ility in problem-so lving? • Does the teaching enhance student competence in applying mathematics procedures? The teacher behaviors were then operationalized to a set of indicators for each research question through asking the question: "What features should be observed in the video before we can say that such behaviors are present?" This set of indicators for the preliminary research questions can be found in Append ix 1. 2. Documents related to the Teacher Law Two Comm issions were set up after approval of the Teacher Law, and they produced t wo documents relevant to this study. One of these is on Teacher Competencies and the other on Certification Instruments for teachers. Because these documents define what teacher behaviors and competencies are expected in the country, they formed an important basis for determining the research questions. 3. Literature on mathematics education and video studies This includes literature on the teaching and learn ing of mathematics, curricu lum documents from other parts of the world and past video stud ies. To a large extent, Indonesian cu rriculum documents already reflect elements from the first two kinds of literature (e.g., the NCTM Standards in the United Stat es, 1989). Research questions from other video stud ies also provided good references for this study. In particu lar, reference was made to the TIMSS 1999 Video Study (H iebert eta/, 2003) and the Learners' Perspective Study (LPS) (Clarke eta/, 2006). After takin g into account the information gathered from the last two sources, the preliminary research questions arri ved at from the Indonesian math ematics curriculum were fine-tuned into the following set of research questions: 11 Under the Teacher Law, teachers are required to be proficient in four competency domains: pedagogical, professional, personal and social. The research questions here mainly address the pedagogical domain, with perhaps some coverage of the professional domain. However, the video data collected can be used to answer research questions in the social and personal domains as well, but these are not part of what this present video study intends to do. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 22 - A TIMSS video study of teaching practices and student achievement  Design ---- Characteristics of teaching-learning • How well are teachers prepared for their teaching? • What mathematics content is covered in the lessons? • How competent are teachers in teaching mathematics? • How are the lessons structured, and how is time managed during the lessons? • What types of mathematics problems do students solve7 • How are mathematics problems solved? • What teaching strategies are used by the teachers? • What types of questions do teachers ask? • How do teachers assess student learn ing? • Are teachers motivated to improve their teaching skills? • What learning resources are used for supporting teaching and learning? • What are the profiles of the teachers? • What are the students' attitudes toward mathematics? • Wh ich of the characteristics above have positive or negative relationships w ith high student achievement? The in dicators for these characteristics, together with the sources of data for answering the correspond ing research questions from the instruments of the study, can be found Appendix 2. Th ese research questions will guide this video study. The way the study was conducted will be described in the next section. 23  Section 3 Methodology and Scope 3.1. Justification of a Video Study A video study of Indonesia's classrooms was deemed to be of high value because of its advantages in providing insights that cannot be gained through other methods. As point ed out above, the purpose of this video study is to examine classroom performance of teachers in order to provide evidence for the effect of the Teacher Law and the BERMUTU project. Classroom performance of teachers may be studied in a number of different ways. The major alternatives are: 3.1.1. Interviews with teachers Interviews are limited by their reliance on a teacher's memory, honesty and perception of what happens in the classroom. In the literature on teaching practices, there are ample studies of instructional practices gained through teacher interviews. Wh il e such interviews may lead to the collection of detailed classroom behaviors and may even be used to probe beyond behaviors to the reasons behind the behaviors, this se lf-reporting method ha s obvious limitation s: a. Teachers in the interviews may have forgotten what exactly happened in the classroom under question, and so the responses may represent what should have happened rather than what really happened b. For one reason or another (for example, saving face, offering answers which they think the interviewer is looking for, etc.), the teachers may not be honest in responding to the interviewer's questions. Even w hen the researcher promises confidentiality, teachers may still fear that their answers w il l somehow be disclosed to people to w hom they don't want the information to be disclosed (for example, government officials or the principal of the school) and so may refrain from responding honestly. On the other hand, teachers may also be sub-consciously defensive, not wanting to admit to themselves and others that they are not performi ng as well as they want to. c. Even if the interviewees are being honest, they may be sincerely mistaken about their own performance in classroom teaching. In reporting on their classroom practices "honestly'; teachers may in fact subconscious ly report on what they wanted to achieve rather than what they actually did in the classroom. d. The final drawback of interviews is, of course, that they are time-consuming so that it is not possible for one interviewer to gather data on a large scale. If more than one interviewer is involved, then inter- 24  Methodology and Scope interviewer reliability is difficult to establish since the subjective element in the interview is difficult to detect. As a result, it is hard to genera lize the findings to the population under study. 3.1.2. Teacher questionnaires Teacher questionnaires also suffer from the limitations of memory, honesty and perception, as well as inflexibility in their administration. Studying instructional practice through administering questionnaires to teachers is another popular methodology in the literature. The advantage of using questionnaires, in con trast to interviews, is that they can be administered on a large sca le involvin g less manpower, and once the questionnaire is set the scoring and analysis of results are relatively more objective. If the sample is large enough, generalization of the instructional practices to the population under study may be achieved. However, the teacher questionnaire methodology, being a self-reporting methodology as well, shares most of the problems encountered in the interview approach discussed above. These include forgetfulness of the teachers, dishonesty and tea chers being sincerely mistaken. Furthermore, because of the inflexibility in administering a questionnaire, the cond ition s under wh ich the questionnaire is answered are not know n to the researcher; hence, unacknowledged ignorance and unconscious biases are difficult to detect. For example, because of the different background and experience of the respondents, they may have a different understanding of the same words used in the questionnaire. This makes generalization and comparison of the results dubious. 3.1.3. Live observation of classrooms Observations have the limitations of being intrusive, time-consuming, and challenging in terms of inter- observer reliability and do not allow for re-observation of lessons. A third popu lar method for studying classroom behaviors and practices of teachers is through live observation of lessons. Data collection in live classroom observation may take a more quantitative, "systematic observation" approach or a more qualitative, "ethnographic" approach. The former uses an observational system to reduce classroom behavior to small-scale units under pre-determined categories (e.g., Flander's interaction analysis categories, 1970) suitable for tabulation and statistical analysis. In the second approach, the observer is "i mmersed " in the situation be ing observed for a long duration, interacting with the subjects (ca lled informants) and taking detailed field notes. Words of the "informants" can be recorded down in fu ll with some of these vvord s then quoted verbatim in the research report (see, for example, Delamont and Ga lton, 1986). While live observation of cl assrooms overco mes some of the problems of using an interview or questionnaire approach in that it does not rely on self-reporting from the teachers, it has its own problems: a. It is an intrusive method, and as it is likely that the teacher and students are distracted by the presence of the observer, the observed instructional practice may not be typical of the regular behaviors of the teacher when he or she is left alone. b. It is time consuming . c. If only one observer is involved, it is not practical to study a la rge samp le, and so generalization of research findings is a problem. If more than one observer is involved, then inter-observer rel iability is difficult to establish. d. The most severe problem with live observation is that lessons cannot be re-observed, and so the foci of observation need to be decided beforehand. When important and interesting findings evolve from the data, it is not possible to go ba ck to the classroom again to collect further data on the same lesson. 25  3.1.4. The video study approach and its advantages The video study approach provides many unique advantages for understanding classroom activity. Video study is also an intrusive methodology, and some argue that it may be even more intrusive than live observation, especially when it is done in a community where video-taping is not common . But experience has shown that w hile students (and their teachers) may be distracted by the video-taping equipment in the beginning of the lesson, this distraction may lapse soon after the lesson begins; if video-taping is done in consecutive lessons, then the effect of the presence of the camera is negligible from the second lesson onward. In this sense, video- taping may be less intrusive than live observation, especially if the latter involves more than one observer. The advantages of vi deo study are many and make it an extremely powerful methodology for studying the instructional practices of teachers: a. Different observers may focus on the same video as the basis of a shared analysis. This in creases inter-rater reliability, and if the required level of reliability is not achieved initially, further tra ining of observers may be conducted to increase the reliability. b. The use of multiple cameras may allow different aspects of the classroom to be capt ured simultaneously, and synchron ization and the use of a mixer will enable the different aspects to be related to each other. c. Since the videos are permanent records of classroom activities, multiple analyses may be performed . The videos may be analyzed repeatedl y, at any time and in any place. d. The videos may be paused, rewound, fast-forwarded, etc., for further analysis. For thi s study, therefore, a video study approach was adopted to study the in structiona l practices of Indonesian teachers. Some methodological issues of the present study are discussed below. 3.2. Collection of Data 3.2.1. Unit of Study and Analysis The unit of analysis-- the classroom-- is the most applicable to the purpose of this study. The objective is to characterize mathematics teaching in Indonesia's classrooms and to id entify wh ich of the classroom characteristics are related to high student achievement. The purpose is not to characterize or evaluate performance of individua l teachers, and so individual teachers are not the unit of analysis in this study. Making ind ividual teachers the unit of analysis would, of course, have the advantage of enabling it to be more sensitive, but to do this, videotaping of multiple lessons of the same teachers (to obtain a reliable measure of the performance of individual teachers) would be needed. And to study the change in performance from 2007 to 2011, the teachers would need to be traced for videotap ing aga in four yea rs later. Thi s is obvious ly both too expensive and impractical. As an alternative, the methodology used by the TIMSS 1999 Video Study is followed in this study w here a representative sample of teachers is chosen and one lesson per teacher is videotaped. In this way, conclusions are made not about the performance of individual teachers but rather about the performance of teachers as a whole. Thus, comparisons can be made between particular groups of teachers (for example, teachers with S1 qualification ve rsus those w ho do not have such a qualification) in 2007 and again in 2011. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 26- A TIMSS video study of teaching practi ces and student achievement  Methodology and Scope 3.2.2. Sampling The video study uses a random sub-sample from the 150 schools chosen for the TIMSS sample, which itself is based on a rigorous sampling methodology. A representative sample is needed if we want to general ize the find ings to the population under study. There are at least two aspects of representativeness in a video study: whether the teachers chosen are representative of the teachers in the popu lation, and whether the lessons videotaped are representati ve of the teaching of the teachers concerned; i.e., the issue of t ypicality of the videotaped lessons. a. The Sample A random sub-sample of the TIMSS schools aims to have the selected teachers be representative of teachers in the population. Since we are to relate characteristics of classroom teaching to student performance in TIMSS, the sample for the video study should be linked to the sample of Grade 8 classrooms (and hence teachers) in the TIMSS study. To keep the scale of the study manageable while ensuring representati veness, 100 classrooms (and their teachers) were random ly sub-samp led from the TIMSS sample, and one lesson per classroom w as videotaped for study, follow ing the practice of the TIMSS 1999 Video Study. It should be pointed out that since the TIMSS sample was randomly draw n, the randomly drawn sub-sample of 100 lessons was also considered to be representative of the lessons in the country. The Indonesian TIMSS sample itself was drawn in two stages. First, a random sample of 150 schools was drawn from all the target popu lation schools in the country using a PPS method 12 Two stratifications were used in the sample: type of school (public or private) and quality of schoo l (based on the nationa l test scores: high, average, and low). Then one Grade 8 class from each selected school was randomly drawn for study, and all students of the chosen classes were asked to take the test and the student questionnaire. Following the practice ofTIMSS, the sub-sample for the video study was also drawn using a PPS met hod from the ma in sample. After the 100 schools were chosen, the teachers of the Grade 8 classes chosen in TIMSS were invited to participate in the video study. b. Exclusions Some schools in the TIMSS sample are in extremely remote locations, and it is difficult and expensive for equipment to be carried there for the video study. Such schools had been exc luded before the sub-sample was draw n. The decision of w hich schools were considered to be in "extremely remote locations" w as made by the relevant personnel from Puspendik, and such exclusions were kept to a minimum in order not to jeopardize the representativeness of the sub-sample 13 . On the typicality of the videotaped lessons, two measures were taken . First, teachers were asked in a short questionnaire administered to them (see below) how typical the lessons videotaped were. This, of course, is self-reporting, and the results may not be reliable. The second measure, taken to mitigate the problem of intru siveness due to the presence of the camera and the videographer, was for each teacher in the sample to be videotaped for t wo consecuti ve lessons. The videotaping in the first lesson was for the teacher, students and researchers to get accustomed to the presence of the videotaping personnel and equipment, and on ly the videotape for the second lesson was used in the data analysis. (The teacher and students were not told about this part of the design.) 12 PPS stands for "Probability Proportional to Size': The sample is chosen in such a way that the probabi lity of a school being chosen is proportional to the number of Grade 8 students in the school. 13 The Educationa l Assessment Cen tre suggested that two of the 150 TIMSS schools be excl uded from the video study, and this was considered acceptable by the international consultant. 27  3.2.3. Data A rich array of data was captured for the video study, including the video data, teacher questionnaires, student questionnaires, field notes from the observation team and TIMSS data, which included questionnaires and test results. The kinds of data collected and the standardized data collection procedures included are described below. a. Video Data Two cameras were used to videotape classroom sessions, with one tracing and focusing on the teacher and a stationary one focusing on the whole class of students. Because of the limitation of space in some of the classrooms, a wide ang le lens was used for the camera that captured the whole class image. Mixing of the two tapes (for both images and sound) was done by one of the researchers on site, so the vid eotaping for each lesson produced three sets of video data: the teacher image tape, the whole class image tape, and the mixed image tape. These tapes were ready immediately after the lesson, and so they could be used for the teacher and/ or student interviews (see below). Only the mixed image tape was used for the interview and data ana lysis, but the teacher and whole class tapes were kept in an archive in case there was a need to go back to the originals. b. Data Collected for the Video Study To provide background information for the analysis of the video data, many other forms of data were collected : Teacher questionnaire Immediately after videotaping, a questionnaire was administered to the teacher whose lesson had just been videotaped. Since the same teachers had taken the TIMSS questionnaire w here general questions on mathematics teaching and learn ing were included, this video study questionnaire on ly focused on two areas: details about the lesson videotaped and the teacher's experience in in-service professional development activities. The latter is important since in 2011, it w ill be important to compare the performance of teachers who have received in-service activities because of the Teacher Law w ith those who have not yet benefited from the Law. Issues asked in the questionnaire included qualifications of the teacher, recent in-service professional development activities, information on the lesson videotaped, teacher's ideas about the planning and implementation of the lesson, typicality of t he teaching, typicality of student behavior, etc. Student questionnaire A short questionnaire was administered to all the students in the class being videotaped. Again, since the same students were taking the TIMSS questionnaire, this video study questionnaire only focused on information about the lesson videotaped w ith the results then linked to the TIMSS student questionnaire. Questions were asked on the typicality of th e videotaped lesson, whether students understood w hat was covered in the lesson or not, whether they enjoyed the lesson or not, what they felt were important or interesting points in the lesson, etc. Interviews with teachers and students Interviews were carried out with the teacher and a random group of students after the videotaped lesson. Interviewees were asked to elaborate on w hat had happened in the videotaped lesson. Field notes or classroom observation records Field notes on events in the lessons or events that could not be captured by the video images were taken by an observer. The field notes also included a brief description of the stru cture of the lessons as well as points that the observer found significant or interesting. INSIDE INDONESIA'S MATHEMATICS CLASSHOOMS: 28- A TIMSS video study of teaching practices and studen t achievement  Methodology and Scope Lesson plans Teachers were asked to submit a copy of the lesson plan or learning Implementation Plan (RPP) for the lesson videotaped. These lesson plans provided information for making a judgment on how well prepared teachers were for the videotaped lesson. In general, the characteristics looked for in a good lesson plan include the following : • Carefully planned teaching-learning activities which w ill be a learning experience for students • Systematic steps in carrying out the activities to attain the learning objectives • Steps in the learning process specified in detail, so that the plan can be easily understood and used by other teachers without caus ing different interpretations of the process. c. TIMSS Data The fact that the schools involved in the video study also participated in TIMSS provided a rich set of additional data collected through the TIMSS survey instruments. These data sets included: • TIMSS results: mathematics scores broken down by question type (geometry, algebra, numbers, data and change) • Student surveys (questions on home characteristics, student background, student perceptions of mathematics and extracurricular activities) • Teacher surveys (background and perceptions) • School surveys (school conditions, resources, safety, parental involvement and perceptions) 3.2.4. Data Collection Procedures Data collection took place between mid-January and the end of April 2007 and involved significant logistical planning. In order to videotape lessons in 100 schools w ithin this relatively short span of time (given the diverse geographic areas where the schools are located), five teams of researchers were involved in conducting the videotaping simultaneously. Each team consisted of two technica l people and one member with expertise in the field of mathematics education. One techn ical person handled the teacher camera and followed the actions and movements of the teacher (the whole class camera was stationary and did not need to be staffed), and the other techn ical person did the on-site mixing of the two images. The third person in the team was the observer and was responsib le for taking field notes (see above). He or she was also responsible for administering the questionnaires and interview of the teachers/students. The tapes were digitized and compressed as soon as possib le wh ile the teams were still in the field so that immediately after the data col lection period, a digitized and compressed dataset was ready for coding and analysis. 3.3. Data Coding Data coding (and later analysis) was done using the StudioCode software, and a "data coding tree" was constructed based on the research questions. To ensure the reliability of coding, at the beg inning of the process, a selection of lessons was coded by all the coders under the supervision of a cons ultant, and the inter- rater reliability for the codes was calcu lated. The cod ing proceeded after an inter-rater reliability of 85% had been achieved. When about 50% of the lessons had been coded, this exercise was repeated one more time to ensure that the coding was done reliably throughout the whole coding exercise. After all the videotaped lessons were coded, they were transcribed to facilitate data analysis. 29  3.4. Data Analysis Data analysis was conducted by the Indonesian research team under the advice of the consultant who had conducted a similar study in Hong Kong. Firstly, the data was analyzed and presented in a descript ive form, usin g representations such as bar cha rts, percentages, medians, modes, standard deviations and ranges. Then the data was ana lyzed inferential ly using correlation, regression and SEM (standard error of the mean). In addition to analyzing the data using the statistics tools available in the StudioCode software, data analysis was also performed usin g standard statistical packages such as SPSS, Microsoft Excel, and LISREL. The data was analyzed with reference to the research questions based on the coded data. In th e first phase of the ana lysis, the video data and other data (questionna ires, interviews and classroom observation records) were analyzed, and the results were compared w ith those of seven countries of the TIMSS 1999 Video Study. In the second phase of the analysis, correlations between t he teaching-learning practices identified in the video study and TIMSS 2007 scores were comp uted. 3.5. The Research Team The Research Team consisted of local experts from various institutes and with diverse backgrounds. It had 10 core team members who are sen ior mathematics in structors from the Center for Development and Empowerment of Mathematics Teac hers and Education Personnel (P4TK), the Educational Quality Assurance In stitution (LPMP), and some junior secondary schoo ls. Dat a ana lysis and reporting were performed w ith the assistance of experts from universities. Technical support was provided by a team from MoNE. 3.6. Achieved Sample and Problems Encountered The target sample was 100 classrooms, but 101 classrooms were eventually videotaped (see reasons below). The 101 schools included both public and private schools and were spread out over 51 d istri cts in 17 provinces across the country (including six provinces in Java, five in Sumatera, three in Sulawesi, and the provinces of South Kalimantan, Nusa Tenggara Timur, and Nusa Tenggara Barat). In the process of conducting this video study, the team found several constraints with respect to the condition of the teachers and students in the research sample. These can be d ivided into four types, w ith the prob lems and the ways they were dea lt with summarized below: Type of Problem Handling of Problem Different teacher and students Excluded from sample 2 Different teachers Excluded from sample 3 Different students Excluded from sample 4 Repetition of lessons Not excluded from sample The sample classrooms where teachers or students were different from the TIMSS sample were removed from the TIMSS regression analysis linking student achievement to classroom practices. Dropping these samp les was necessary because the links between teach ing practice and student mathematics scores wou ld have been inva lid. For the understand ing of what goes on in Indonesia's classroom, though, th e 28 invalid classroom sessions we re considered useful in providing insights into teaching practices. In one class the students took an examination for the fu ll period and because there w as no interaction or variati on in activities, t he co re INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 30 - A TIMSS video study of teaching practices and student achievement  Methodology and Scope team decided to remove this class from the analys is on teaching practices. This left 72 valid schools for the analysis presented in this report. A comparison of the 72 valid schools and 28 dropped schools was performed and interesting differences were discovered. A summary of the comparison results can be found in Appendix4: Comparison of Full Sample Results with Subset. For the purposes of presentation within this study, though, only the results of the 72 valid classrooms are included. The number of cases for each type of instrument available for analysis is presented in the table below. Type of Instrument Total Cases Video data 101 Teacher questionnaire 84 Student questionnaire 3679 Teacher interview 101 Classroom observation record 101 Learning Implementation Plan (RPP) 88 3. 7. Lessons Learned through the Implementation of the Video Study Through the course of implementing the video study, many lessons arose which are important both for the analysis and for planning for the second phase of the study which will take place in 2011. The following is a summary of the lessons learned: Several schools attempted to change the sampled teacher with their best teacher when videotaping took place. Accurate data of students and the teacher in the class sampled are necessary for confirmation. Some schools tried to fix up classrooms (painting, replacing old tables, etc.) in order to have a good appearance. Prior to implementation of the study, a clear understanding by the teachers and their principals on the purpose of the study (portraying the actual situation in Indonesia classrooms) is necessary. The study requires involvement of people who are experienced in teaching mathematics, who know how to conduct research and who can devote significant time to the study. To maintain continuity of the st udy, four to five people are needed to form the core team to work full-time from the beginning until the end. To run the study smoothly, the study team should own and be well-trained in using the software and the equipment. The video data coding requires the use of special software (StudioCode) which is quite expensive, requires spec ial skills to operate and runs on ly on Apple computers which are uncommon in Indonesian institutions. 31  Section 4 Video Results and Cross- Country Comparisons Findings of the study are reported in this chapter in four sections: (i) teacher background, (ii) lesson structure, (iii) lesson content and (iv) instructional practices. First, the Teacher Background section present characteristics of the teachers videotaped to give readers an idea of the general profile of the teachers in the sample. Since the sample is a representative 14 one, the profile should tend to reflect the general profile of the population of Grade 8 mathematics teachers in Indonesia. The Lesson Structure section focuses on the length of the lessons; how much time is dedicated to mathematics, non-mathematics and mathematics organization activities; the purposes of various segments; and the type of interaction (full class, ind ividua l and group work) that takes place. The Lesson Content section focuses on the mathematics content of the lessons and assesses the complexity of prob lems and whether the problems involved appl ications or proofs. The Instructional Practices section focuses on how the mathematics problems were presented and worked on, the opportunities of teachers and students to talk and resources used du ring the lesson. 4.1. Teacher Background 4.1.1. Teacher Education Level Most teachers in the sample had some form of training in mathematics and had achieved a four-year degree. As far as educationa l background in mathematics is concerned, 87% of the teachers in the sample had some form of training in mathematics (e ither through a degree or mathematics certificate); 13% did not have a mathemat ical background. As for the teachers' highest level of education, 3% of the teachers were educated up to 02 level (a two-year college degree), 13% to 03 (a three-year college degree), and 80% to S1/04 (a four-year college degree), with fewer than 2% of the teachers graduated w ith an S2 degree. 14 To be precise, the study includes only a representative sample of schools in Indonesia and not a representative sample of Grade 8 mathematics teachers since the teachers were not draw n randomly from the population of teachers in the country. 32  Video Results and Cross- Country Comparisons Figure 4.1 Educational background of teachers: percent with a Compared to other countries, mathematics degree a relatively high proportion 96 of Indonesia's teachers had 100 90 mathematics as their college 90 80 74 major. Figure 4. 1 be low shows 70 64 the educational backg round of 61 57 teachers who were educated at ~ 60 e ~ 5o 41 S1 level mathematics or 0.. 40 mathematics education or 30 above. As can be seen from the 20 figure, 74% of the teachers in 10 the samp le had an S1 diploma 0 +-----.-----.-----.-----.- in mathematics/mathematics education or above, while the corresponding figures for teachers in the TIMSS 1999 [__ Video Study countries were Source: Indonesia results combined wi th data from table 2.1 in Hiebert, J. et. al, (2003), page 17 between 41 % and 96%. 4.1.2. Teacher Certification For the 2007 phase of the study, only one teacher in the sample had attained certification; this w ill be important for comparison w ith the 2011 phase of the study. In Law No. 14 of 2005 on teachers and lecturers, it is mentioned that the recognition of teachers as professionals is proven w ith a certificate in education, and teachers in Indonesia wi ll get certified in stages. By the end of 2007, of the approximately 53,000 junior secondary mathematics t eachers in Indonesia, 4,425 had been certified (Puspendik, 2008), amounting to approximately 8% of all junior secondary teachers. There were altogether 27 certified mathematics teachers in the sample schools. However, there was only one certified mathematics teacher in the videotaped lessons. The reason for such a low overall number of certified teachers is that at the time of data collection, Indonesia's new portfolio process for certification had been in place for less than a year. The goal under the Teacher Law is to have all teachers certified by 2015. When the second phase of the video stud y is conducted in 20 11 it is expected that at least half of teachers in the sample will have undergone certification. 4.1.3. Teaching Experience in Mathematics Indonesia's teachers had relatively fewer years of experience compared to other countries. Teaching experience is one of the aspects that may affect the performance of teachers in their teaching. Some research shows that the longer teachers teach, the more adequate they are in their teaching ability. In this study, only experience in teaching mat hematics, ra t her than general teaching experience in other subject areas, was taken into account. The lengths of teaching mathematics for teachers in the sample varied from 1 year to 32 years. Sixteen teachers (26%) were relatively inexperienced, w ith fewer than five years of teaching experience. Eighteen teachers (30%) had experience between five to 10 years; 17 teachers (28%), had experience between 11 to 20 years; and 10 teachers (16%) were very experienced, w ith over 20 years of experience. Figure 4.2 shows the experience of these teachers in tea ching mathematics at the junior secondary leve l. The average Indonesian teacher had been teaching for 11 years (with a median of 10 yea rs), which is much lower than teachers in the other seven countries (the average teaching experience of teachers in the TIMSS 1999 Video Study countries was from 10 to 21 years). The likely reason for Indonesia's teachers being re latively younge r is that while the other countries in the study have had universal enrollment at the junior secondary level for many years, Indonesia's junior secondary enrollment is much lower but has been increasing. At this level, the gross enrollment rate was 33  Figure 4.2 Years of experience in teaching mathematics and comparison with other countries Average Years of Experience by Country Years of Experience of Indonesian Teachers 25 21 19 20 17 15 14 13 11 10 10 5 0 Source: Indonesia results combined with data from table 2.3 in Hiebert, J. et. al., (2003), page 19 only 65.6% in 1995 but reached 82% by 2007. While Indonesia's older teachers tend to work in primary schools, many of the newer teachers tend to have been hired at the secondary school level to meet the demands of increased enrollment. 4.1.4. Teacher Workload The workload of Indonesia's teachers was relatively low compared to other countries. As for classroom workload, Indonesian teachers on average taught mathematics classes for 14 hours per week while those in other countries ranged from 11 to 20 hours. Teachers in all countries taught multiple subjects, and when taking other subjects into account, teacher workloads in Indonesia increased to 18 hours w hile in the other countries it increased to between 16 and 24 hours. When taking work outsi de of the classroom into account, the workl oads ranged between 36 and 42 hours per week. Figure4.3 Average class time of mathematics teachers in mathematics class (in hours) vs. other subjects (not including non-class workload) 3 4 8 13 6 4 20 Other Subjects 18 12 13 14 Mathematics 11 .}f ,..'S..,'b ~'b >::-"' *-0 >:-0, ~ 'f::...,c., .,~ '->'-"'"' 0~ .,,.,__.,<::- 0 .,,.,__.,<::- 00 «:-"'<:( '?' ""0, .,._.,o ;s- ~~ '"" '<:-0 iS' c.-"-"' .;:,<::-' ~., '? Source: Indonesia results combined with data from table 2.4 in Hiebert, J. et. al., (2003), page 20 INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 34- A TIMSS video study of teaching practices and student achi evement  Video Results and Cross- Country Comparisons Figure 4.4 Number of classroom teaching The mathematics teachers in the sample tended to hours per week in mathematics teach only mathematics. Most Indonesian mathematics teachers taught only the subject of mathematics, but there were 12 mathematics teachers in the sample w ho also taught non-mathematics lessons. The number of hours teaching non-mathematics varied from two to 20 hours. Almost all teachers fall short of the Ministerial Decree (Permendiknas) No. 18 of 2007 that requires that teachers have 24 class periods of mathematics in order to be eligible for the certification bonus (professional allowance). 15 Figure 4.4 indicates that 80% of the teachers in the sample taught fewer than 24 hours per week. Meanwhile, in Australia, the Czech Republic, Hong Kong SAR, the Netherlands, Switzerland and the United States, the average teaching load is between 36 to 42 hours per week and the number of hours teaching mathematics is between 11 to 20 hours (see Figure 4.3). 4.1.5. Teacher Gender The teachers participating in the video study were evenly split in terms of gender, with 51 % female and 49% male. This is very close to the national average for junior secondary mathematics teachers, with 49% female and 51% male. Interesting ly, 60% of female teachers were mathematics majors compared to only 43% of male teachers. There is little difference in terms of yea rs of experience, with female teachers having 12.7 years on average, compared to 13.3 years for male teachers. There is also little difference in terms of civil servant status, w ith 58% of female teachers being civil servants compared to 60% of male teachers. There is a slight difference in t erms of geographic location, w ith 64% of female teachers working in rural areas compared to male teachers at 69%. 4 .2. Lesson Structure 4.2.1. The Duration of the Lessons There was a large variation in the length of the mathematics classes videotaped in Indonesia. Table 4.1 below shows the duration and other descriptive statistics of the sampled lessons in the study. Indonesia's class length ranged from 39 to 97 minutes, w ith an average class length of 70 minutes and a standard deviation of 14. The number of minutes of mathematics class per week is estimated to be 140, which is lower than most other countri es. 15 Law No. 14 of 2005 states that teachers are only allowed to teach in the subject in which they are certified and that they are only allowed to be certified in one subject. In practice, though, many teachers teach more than one subject. 35  Table 4.1 Duration of lessons (in minutes) Country Est. Minutes/week 2007 hours total Hong Kong SAR 41 36 26 91 175 148 Czech Republic 45 45 90 13 179 128 Netherlands 45 45 35 100 7 127 Switzerland 46 45 39 65 3 Australia 47 45 28 90 13 174 131 Japan so so 45 55 2 200 105 United States 51 46 119 17 179 148 Indonesia 140 136 Source: Indonesia results combined with data from tables 3.1 and 3.2 in Hiebert, J. et. al., (2003), pages 37 and 41; 2007 hours is from the TIMSS 2007 report (Mu llis, 2008, exhibit 5.2) Indonesia's classes were significantly longer than in other countries. The mean duration of 70 minutes for Indonesia was much longer than the average duration of only 41 to 51 minutes in other countries (see Figure 4.5 below) . The curricu lum calls for four class periods of mathematics per week (with a period being 45 minutes), w ith the 70 minutes witnessed in the video often being t wo combined class periods. According to the TIMSS 2007 report, Indonesia's total hours per yea r is 136, w hich falls in the middle of the comparison countries. (M ullis, 2008, exhibit 5.2) The breakdown of each individual class shows that most classes last at least 1 hour, with 25% extending 1 Y2 hours or long er. In examining the classrooms by ranking them from the shortest to the longest classroom t ime, it can be seen in Figure 4.6 that approximately 30% of classes are an hour or less, w ith the shortest class being 40 minutes long. The median class time w as 68 minutes and the average class w as 70 minutes. Over 40% of the classes went beyond one hour and 15 minutes, with eight classes extending beyond the 90 minute mark. While there may be advantages of having longer classes, one area of concern is whether Grade 8 students can maintain an attention span for such an extended period of time. Figure 4.5 Length of lessons in mathematics 100 90 80 70 70 60 50 51 50 45 45 46 47 41 40 30 20 10 0 +-----,-----,-----.------,-----.-----.-----.- Source: Indonesia results combined with Hi ebert, J. et. al., (2003), page 37 Note: Indonesia's lessons are significantly longer due mainly to the fact that in most schools two class periods are combined into a sing le session. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 36 - A TIMSS video stud y of teaching practi ces and student achievement  Video Results and Cross- Country Comparisons Figure 4.6 Ranking of class length in minutes, from lowest to highest --------------------------~ 1 120 105 Average: 70 minutes 90 75 l 60 45 30 15 0 0 10 20 30 40 50 60 70 80 90 100 Classrooms (ranked from shortest to longest time) 4.2.2. Amount of Time Spent Studying Mathematics The initial layer of analysis broke down class time into mathematics, non-mathematics and mathematics organization time. Following the practice of the TIMSS 1999 Video Study, the time in the recorded lessons was classified according to three kinds of activities: mat hematica l work, mathematical organization, and non- mathematical work. The TIMSS 2009 Video Study defined these three kinds of activities in the following way: • Mathematical work: Time spent on mathematical content presented either through a mathematical problem or outside the context of a problem; e.g., talking or reading about mathematical ideas, solving mathematical problems, practicing mathematical procedures or memorizing mathematical definitions and rules. • Mathematical organization: At least 30 continuous seconds devoted to preparing materials or discussing information related to mathematics but not qualifying as mathematical work; e.g., distributing materials used to solve problems, discussing the grading scheme to be used on a test or distributing a homework assignment. • Non-mathematical work: At least 30 continuous seconds devoted to non-mathematical content; e.g., talking about a socia l function, discipl ining a student while other students wait or listening to school announcements on a public-address system. (p. 38 of Hiebert et al, 2003) The distribution of t ime for the three kinds of activities was as follows (Table 4.2): Table4.2 Time used for mathematical work, mathematical organization, and non-mathematical work (minutes) Structure Mathematical work 89% 62.2 60.9 14.1 35.0 90.1 Non-mathematical work 3% 1.8 1.3 1.6 0.2 10.9 Mathematical organization 8% 5.8 4.9 4.0 0.4 18.8 37  Figure 4.7 Percentage of time used for learning mathematics 100% 90% 80% 70% Non-mathematics 60% Mathematics Organization 50% 40% • Mathematics 30% 20% 10% Oo/o Source: Indonesia results combined with Hiebert, J. et. al., (2003), page 39 Indonesia's classes tended to have a much larger proportion of non-mathematical and mathematical organization time. The bar chart in Figu re 4.7 below shows the percentages of the average duration of the three kinds of activities in the recorded lessons. It can be seen that mathematical activities took up 89% of the time (3734 of the 4188 seconds), wh il e non-mathematical activities and organization activities took up 3% and 8% of the time respectively. Compared w ith other countries where the time used for mathematical activities ranged from 95% to 98%, the time that was used for mathematical work in Indonesia was lower. In contrast, the percentage of time used for mathematical organ ization work was higher. 4.2.3. The Role of Mathematical Problems a. Time spent on problems and non-problems Mathematics time can be further broken down into segments of either working on problems or non- problem time. Time spent on mathematica l work (89% of class time in Indonesia) was divided into either problem or non-problem time. The definition of this time is as follows: Working on problems: Problems are defined as events that contain a statement asking for some unknown information that can be determined by applying a mathematical operation. Simple questions asking for immediately access ible information were not counted as problems. Examples of mathematical problems include: • Add ing, subtracting, multiplying and dividing who le numbers, decimals, fractions, percents and algebraic expressions • Solving equations • Measurin g lines, areas, volumes and ang les • Plotti ng or reading graphs • Applying formulas to solve rea l-l ife problems Non-problem segments: A non-problem segment is defined to be mathematical work outside the context of a problem. Without presenting a problem statement, teachers (or students) sometimes engaged in: • Presenting mathematica l definitions or concepts and describing the ir mathematical origins INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: . 38 - A TIMSS video study of teaching practices and student achievement  I Video Results and Cross- Country Comparisons Figure 4.8 Mathematical time divided into Problem vs. Non-Problem Segments (percent of total) I Indone sia 24 I 18 15 15 15 13 United States - ·- ·• ,•;-c.· ··">-r 11 • Problem Non-problem Source: Indonesia results combined with resul t s from Hiebert (2003) using graph from page 42, but modified to only include mathematical time. • Giving an historical account of a mathematical idea or object • Relating mathematics to situations in the real world • Pointing out relations hips among ideas in the lesson and previous lessons • Providin g an overview or a summary of the major points of the lesson • Playing mathematica l games that did not involve so lving mathematica l problems (e.g., a word search for mathematical terms) . Indonesia spent a significantly larger proportion of time on non-problem segments. The resu lts in Figure 4.8 show that of the time classified as mathematics time, Indonesia spends much more time on non-problem segments than in other countries, w ith 76% dedicated to problem work and 24% to non-problem time. b. Independent problems Most ofthe lesson time in Indonesia, as in other countries, was spent on solving mathematical problems, either by the teacher or by the students. The TIMSS 1999 Video Study classified mathematical problems into three types according to the setting in which they were solved. Here we focus only on one type, ca lled independent problems, w hich were defined as those presented as single problems and worked on for a cl ea rly definable period of time. These problems might have been so lved publicly - as a w hole class- or they might have contained a private work phase w hen stud ents worked on them individua lly or in sma ll groups. (Hiebert, 2003, p.43) Most classes had only two independent problems per lesson, with an average of 3.3 problems. The number of independent problems solved over the cou rse of each videotaped lesson varied from one to 10 (as shown in Figure 4.9). The average time used to solve one independent prob lem was 6.5 minutes (393 second s). The shortest independent problem was solved in only 3 seconds w hile the most time-consuming one took 14.5 minutes (873 seconds). In comparison with other countries, Indonesia had fewer independent problems but spent more time in solving them. Figure 4.10 shows Indonesia compared to other countries. On one extreme, Japan averaged 39  Figure 4.9 Number of independent problems solved in the lessons 45 40 40 "' E Cll 35 :g 30 - 0 c. 25 0 Cll Cl 20 ~ c: 15 Cll ~ 10 Cll 11. 5 2 0 0 0 2 3 4 5 6 7 8 9 10 Number of independent problems Figure 4.10 Average number of independent problems solved in the lesson and average length of time in minutes 16 15 14 13 12 10 10 4 Indonesia Japan Switzerland Australia Hong Kong Netherlands United States Czech SAR Republic • Average Number of Independent Problems Average Problem Length (m inutes) Source: Indonesia results combined with results from Hiebert (2003) using table 3.3 on page 44 and figure 35 on page 46 three independent problems per lesson but spent 15 minutes per problem. On the other extreme, the Czech Republic had 13 independent prob lems per lesson and spent on ly four minutes per problem on average. Only Japan spent more time per independent problem. c. Problem solved in more than 45 seconds Most of the problems in Indonesia took longer than 45 seconds to solve. The length of time used to solve a problem is also an indication of the complexity of the problem. The average number of problems tha t was completed in more than 45 seconds was 4.8 wh ich constituted 65o/o of all the independent problems. 4.2.4. Time Used to Review, Learn New Content, and Practice Al l mathematics time can also be broken out into segments based on their purpose. Reviewing: This category, more techn ically called "add ressing content introduced in previous lessons;' focused on the review or reinforcement of content presented previously. These segments typ ically involved the practice INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 40 - A TIMSS video study of teach ing practices and studen t achievement  Video Results and Cross- Country Comparisons or application of a topic learned in a prior lesson or the review of an idea or procedure learned previously. Examples included: • Warm-up problems and games, often presented at the beginning of a lesson • Review problems intended to prepare students for the new content • Teacher lectures to remind students of previously learned content • Checking on the answers of completed homework problems • Quizzes and grading exercises Introducing new content: This category focused on introducing content that students had not worked on in an earlier lesson. Examples of segments of this type included: • Teacher expositions, demonstrations, and illustrations • Teacher and student explorations through solving problems that were different, at least in part, from problems students had worked previously • Class discussion of new content • Reading textbooks and working through new problems privately Practicing new content: This category focused on practicing or applying content introduced in the current lesson. These segments only occurred in lessons where new content was introduced. They typically took one of two forms: (i) the practice or app lication of a topic already introduced in the lesson, or the follow-up discussion of an idea, or (i i) follow-up discussion after the class engaged in some practice or application. Examples of segments included: • Working on problems to practice or apply ideas or procedures introduced in an earlier lesson • Class discussions of prob lem methods and solutions previously presented • Teacher lectures summarizing or drawing conclusions about the new content presented earlier Assessment: This category focused on students being measured on their understanding of the mathematics content in a forma l way that wouldn't be considered simply practicing. Examples of segments included: • Quizzes • Formal tests The most common use of time was introducing new content, but each purpose type had a great deal of variation. The average t ime used for review, new content, practice and assessment is shown in Table 4.3 below. The average t ime used to practice in class was 24.4 m inutes or 34% of classroom t ime. New content was 30.8 minutes (43%), review was 7.2 m inutes (1 0%) and assessment was 1.2 minutes (1.6%). Indonesia spent relatively less time in review and more time in practice than other countries. In comparing these numbers to ot her countries, it can be seen in Figure 4.11 that Indonesia's 10% for review was significantly lower than in other countries. The next lowest percentage was 24% for Hong Kong and Japan, wh ile the Czech Republic dedicated 58% of classroom time to review on average. In practice, on the other hand, Indonesia dedicated relatively more time, with 37%. Hong Kong was the highest at 37%, while Japan was the lowest at only WYo. For new content, Indonesia fell in the middle with 43%. Table4.3 Time (in minutes and seconds} used for review, new content, practice and assessment Mathematical Activities Std. Deviation Review 7.2 4.6 9.2 0.3 50.4 60 New content 30.8 28.7 20.5 4.7 85.4 64 Practice 24.4 25.2 18.5 1.2 75.9 61 Assessment 1.2 0 3.4 15.7 10 41  Figure 4.11 Duration for different activities in Indonesia and other countries 100 90 80 70 60 Practice so New Content 40 30 • Review 20 10 0 Indonesia Australia Czech Hong Kong Japan Netherlands Sw itzerland United Republic SAR States Source: Indonesia results combined with Hiebert, J. et. al., (2003), page 50 Note: "Practice" is made up of the "practice" and "assessment" categories. If the "assessment" portion is not included, then the breakdown would be 12% "review~ 48% "new content" and 40%"practice': 4.2.5. Classroom Interaction a. Public (full-class) and private (small group and individual) interaction An important distinction in the use of class time is in the type of interaction, with public (full-class) and private (individual or group) work. The TIMSS 1999 Video Study defined public and private interactions in the following way: • Public interaction: Public presentation is made by the teacher or one or more students intended for al l students. • Private interaction: All students work at their seats, either individually, in pairs or in sma ll groups, while the teacher often circulates around the room and interacts privately w ith individ ual students. 16 Indonesia was in the middle compared to other countries in terms of public vs. private interaction, with 57% of class time being public and 43% private. As shown in Figure 4.12 be low, Hong Kong SAR was at one extreme, w ith 80% of class time being public interaction, and the Netherlands was at the opposite ext reme, with on ly 44%. The public and private interaction can also be broken down by segment lengths, meaning the amount of uninterrupted time for a given interaction. Table 4.4 conta in s the segment length fo r t im e used in public interaction and private interaction. It shows that the average t ime used for pri va te interaction was 9.22 minutes, with an average of four segments in a lesson, and that for public interaction was 6.67 minutes, with an average of 3.6 segments. -- - - Table4.4 Segment length (in minutes) for public interaction and private interaction Interaction Std. Deviation Avg. Segments Private interaction 9.2 6.2 9.2 0.2 60.7 4.0 Public interaction 6.7 4.0 7.4 0.1 55.2 3.6 Note: lessons are made up of multiple segment s as the class switches from one type of interaction to another. 16 (Hiebert et al, 2003, pp. 53-54) INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 42 - A TTMSS video stud y of leaching practices and student achievement  Video Results and Cross- Country Comparisons Figure 4.12 Percentage of time for public interaction and private interaction ------------------~ Hong Kong SAR ; '• :: ·,,.,_, ~:t .· ·>.".:"'o.·r~~ ;. I 20 ', ·~::•.;..'ir4:.j<c,_,,.,_ ;~· ·Jf:~):'; 1~'~'\•'- I Czech Republic ··'' ' 22 United States ·~ . •· ''. '''· c,f 32 Japan ,; /:'' ''"'' "'"' ~·· :;;:~<;~ 34 Indonesia 43 I Switzerland . .:• '\.T' ss:··<T.':.~' '"'*"'' ; 45 Austra lia ,. : ':. 52)~>.';:'!1\f '1'?.{ 0'.'~1i;;iilfj 48 Netherlands 44 ''"'<•. :,~;<~· ··I 55 Oo/o 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% • Public Interaction Private Interaction Source: Indonesia results combined w ith Hiebert, Jet aL, (2003), page 54 Figure 4.13 Public interaction breakdown: b. Public Interaction: teacher and student teacher and student involvement participation Of public interaction time, the majority was teacher-only (lecture) interaction. Public interaction can further be broken down into teacher- only, teacher-student and student-on ly time. This breakdown is shown in Figure 4.13 below. As wou ld be expected, the teacher-only (lecture) was the most common form of public interaction, making up 59% of all public intera ction. Student-only time made up 19%, with the remaining 22% involving both the teache r and students. c. Private Interaction: group work and individual work Note: Percent is ca lculated by taking the average percentage of time in each individual lesson rather than taking a cumu lative time from all lessons. Private interaction was fairly evenly separated between group and individual time. As pointed out above, private interaction included time w hen students were either worki ng individually or in small groups. Here we define "group work" as activities where students wo rk or discuss in small groups, either w ith or w ithout the guidance of the teacher, and "individual work" as activities where individual students are working alone, either w ith or without the teacher assisting individual students. As shown in Figure 4.14 below, group work was more common, making up 55% of total individual interaction. The teacher was typically invo lved in this form of activity, visiting groups as they worked . 43  Figure 4.14 Private interaction breakdown: time used for group work and individual work 4.2.6. Pedagogical Features that Influence Lesson Clarity and Flow Techniques to improve the clarity and flow of lessons can assist students in learning. One way to enhance the clarity and the f1ow of the lesson is for the teacher to make goa l statements about the lesson (at the beginning of the lesson and at appropriate points in the lesson) and to summarize the lesson from time to time. On the other hand, outside interruptions disrupt the f1ow of the lesson. These factors will be exami ned in this section . a. Goal statements Indonesia used goal statements more frequently than most other countries. Goa l statements "can help students identify the key mathematical points of a lesson" (Brop hy, 1999, quoted in Hiebert, 2003, p.59) and thus shou ld be helpful in achieving the learning goal of the lesson. Figure 4.15 below shows the percentages of lessons whic h included at least one goal statement in Indonesia and other coun tries. In 84% of classes a goal statement was used, whic h was much higher than most countries; only the Czec h Repub lic had a higher percentage. Figu re 4.15 Percent of lessons which included at least one goal statement 100 91 90 84 80 75 71 70 59 60 53 50 43 40 30 21 20 10 0 Netherlands Switzerland Hong Ko ng United States Australi a Japa n Indonesia Czec h Republic Source: Indonesia results combined with Hiebert, J. et. al., (2003), page 60 INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 44 - A TIMSS video study of teaching practices and studen t achi evement  Video Results and Cross- Country Comparisons Figure 4.16 Percent of lessons which include at least one summary statement ------- 90 82 80 70 60 50 40 28 30 25 21 20 10 10 6 2 0 Switzerland United States Austra lia Hong Kong Czech Japan Indonesia SAR Republi c Source: Indonesia results combined with Hiebert, J. et. al., (2003), page 61 Note:The Netherlands was not included because reporting standards were not met; too few cases were reported. b. Summary statements Indonesia used summary statements much more frequently than all other countries. Summary statements highlight points that have just been covered in the lesson and are helpfu l for students to recognize the key ideas in a lesson (Hiebert, 2003, p.60). In Indonesia, it was found that some 82% of the mathematics lessons contained at least one summary statement (see Figure 4.16 below). This was much higher than in other countries, with Japan being the next closest at 28%. In teacher training, Indonesian teachers are taught to use a summary statement, and it appears that most teachers follow this tra inin g. c. Outside interruptions Indonesia's classrooms had many fewer interruptions than in other countries. Outside interruptions include announcements over the speaker or intercom system, people w ho want to meet the teacher or the students, teachers talking to students who come late and other outside disruptions that break the Aow of the classroom activity. These events, which often happen during a lesson, disrupt the lesson and distract the teacher and the students from concentrating on the teaching and learning. However, in this study it was found that only 7% of the Indonesian lessons had interruptions recorded. As can be seen in Figure 4.1 7 below, Indonesia had the lowest percent of classes w ith interruptions. Figure 4.17 Percent of lessons with at least one interruption from outside -------. 35 32 29 30 30 28 25 20 14 15 11 10 7 8 5 0 Indonesia Japan Switzerland Czech Hong Kong Un ited Austra lia Netherlands L Republic Source: Indonesia results combined with Hiebert. J. et. al., (2003), page 62 SAR States 45  4.3. Lesson Content Mathematics is a universal science that underlies the development of modern technology and has an important role in many disciplines. In particular, growth in the field of information and communication technology has matured through the development of mathematics. To construct and control future technologies, therefore, requires an early and strong grasp of mathematics. The subject of mathematics needs to be taught to all students from primary school in order to encourage logical thinking and enhance their analytical, systematic, critical, creative and co-operative abilities. Mathematics competence enables students to obtain, manage and utilize the information needed to survive in conditions that are always changing, uncertain and increasing ly competitive. Given current trends in development, especially in regard to new technologies, students are expected to have competencies in mathematics that are required to meet these demand. These competencies wi ll be obta ined if the students reach the goals of learning mathematics that are stated in the Appendix to the Regulation of the Minister of National Education (Permendiknas), No. 22, Year 2006, about Standards of Content. The subject of mathematics aims for students to have the ability to: • Understand the concepts of mathematics, explain the relevance of concepts and apply the concepts or algorithms in a flexible, accurate, efficient and precise way in problem-solving • Use reasoning patterns and nature to man ipulate mathematical generalizations to make, prepare evidence about, and explain ideas and statements in mathematics • Solve problems that include the ability to understand a problem, design and complete a mathematical model to solve it and interpret the solution obtained • Communicate ideas w ith symbols, tab les, diagrams or other media to clarify the situation or prob lem • Appreciate the purpose of mathematics in life and have both curiosity about and interest in learning mathematics, with an attitude of trust and confidence in problem-so lving. The Standards of Contents indicate that the applicable curriculum is Kurikulum Tingkat Satuan Pendidikan (KTSP) or the curriculum of the appropriate education level - namely, the curriculum developed by the individual school fo llowing specific MONE guidel ines. This ensures that the curriculum will be relevant to the students' needs in meeting the above-mentioned demands of a changing world. 4.3.1. Level of Mathematics Evident in the Lessons - Complexity of the Problems Students work on problems of various levels of complexity. Fo llowing the pract ice of the TIMSS 1999 Video Study, this study divided the complexity of each problem into three categories: low, moderate and high complexity. These categories were defined as follows: 1. Low complexity: problems wh ich require four steps or less to solve using the usual or conventional procedure (example: solve 2x+7=2) 2. Medium complexity: problems which need more than four steps to solve and which include one su b- problem (example: solve the system of equations 2y =3x- 4; 2x + y =5) 3. High complexity: prob lems which need more than four steps to solve and wh ich include two or more sub-prob lems (example: graph the following linear inequalities and find the area of intersection: y ~ x + 4; x ~ 2; y ?_ -I) INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 46 - A TIMSS video study of teaching practices and student achievement  Video Results and Cross- Country Compari ons Figure 4.18 Level of complexity of problems 100% 6 6 3 12 11 8 8 90% 80% 39 16 22 27 22 25 29 70% 60% 50% High Complexity 40% Medium Complexity 30% 20% • Low Complexity 10% 0% Source: Indo nesia results co mbined w ith Hiebert, J. et. al., (2003), page 71 The complexity of problems depends on the ability of the students as wel l as the ski ll of the teacher. Teachers will likely choose to give eas ier problems to students of lower abil ity. It may also be the case that teachers who have lower competency in mathematics w ill tend to avoid higher complexity problems. Indonesian classrooms in the study presented very few problems of high complexity but had a large proportion of problems with medium complexity. Figure 4.1 8 shows how the prob lems solved in the classrooms of Indones ia compared to other countries. Low comp lexity problems ranged from 17% to 77%, w ith average, medium comp lexity problems between 22% and 45%, and high comp lexity problems between 6% to 39%. The problems solved in Indonesian classrooms were generally of lower complexity, w ith the average percentage of low complexity level problems being 57% and of medium complexity 40%, w ith only 3% of the prob lems being of high level complexity. The fact of Indonesia having a lower percentage of high complexity problems may be expected sin ce the other countries are economically advanced relative to Indonesia. An unexpected resu lt is that Indones ia had on ly 57% of prob lems of low comp lexity with most countries having a larger percentage. On ly Japan had a lower percentage, at 17%. 4.3.2. Type of Mathematics Evident in the Lessons- Problems with Proofs and Applications Working on mathematical problems can take a variety of forms. As pointed out by Hiebert et al (2003), exercises are a simple approach w here students are taught a particular procedure and then asked to practice that procedure using simi lar prob lems. A more advanced approach is what are cal led applications, where students are asked to apply procedures they have learned in one context in order to solve problems presented in a different context. Indonesia had very few problems that involved applications. Applications often are presented using verbal descriptions, graphs or diagrams rather than j ust mathematica l symbo ls. Th ey are important beca use they requi re students to make decisions about how and w hen to use procedures they may have already learned and practiced. In th is sense, appl ications are, by definition, more conceptually demanding than routine exercises for the same topic. Figure 4.19 below shows that the percentage of problems wh ich included applications in Indonesia was 16% compared to 34% to 74% in other countries. 47  ,- Figure 4.19 1 80 70 60 so Average percentage of problems per Grade 8 mathematics lesson that were applicat ions 40 45 51 55 74 40 34 35 30 20 16 10 0 ~ o"' "'~"' ~,c.., 4:(~" ~~ ~,~ 00 <::-~ '?'-~ ,$> ~'? v'? ..:,~ <::-~" <::-~ 00 . .~o ~~« o" ~ '10~ . .(}~ "" ;:y0' 0' ~ ~~"" '?~' f < -<:-o" Source: Indonesia results combined w ith Hiebert, J. et. al., (2003), page 91 The National Research Council has noted that one feature that distinguishes mathematics from ot her school subjects is the special forms of reasoning that can be involved in solving problems (Nationa l Research Counci l, 2001 ). One kind of problem that requires specia l reasoning is a mathematica l proof. To prove that something is true in mathematics means more than inferring it is true by checking a few cases. Rather, it requires demonstrating, through logica l argument, that it must be true for al l cases. Use of proofs in mathematics teaching has been recommended as an import ant aspect of elementary and middle schoo l mathematics (National Council ofTeachers of Mathematics 2000; National Research Council 2001 ). The use of proofs was relatively more common in Indonesia than in other countries that participated in the video study. The results in Figure 4.20 indicate that 13o/o of problems included proofs, compared to the 1o/o to 26% in other co untries. Figure 4.20 Average percentage of problems per Grade 8 mathematics lesson that included proofs 30 26 25 20 15 13 10 5 3 2 0 Czech Republic Hong Kong SAR Sw itzerland Indonesia Japan L Source: Indonesia results combined w ith Hiebert, J. et. al., (2003), page 74 Note: Australia, the Netherlands, and the US were not included because reporting standards were not met; too few cases were reported. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 48- A TIMSS video study of teaching practices and student achievement  Video Results and Cross- Country Comparisons 4.4. Instructional Practices 4.4.1. Teaching Strategies Teaching strategies employed by teachers are an important insight into how students learn to solve problems. In analyzing the teaching strategies used in the samp led mathematics lessons, the following strategies were coded: • Exposition: The teacher lectures while students listen and answer closed questions (with no discussion). • Discussion: The teacher and student(s) discuss their own ideas about mathematics. • Problem-solving: The teacher provides a problem/ situation as a basis to discuss ideas in mathematics. • Practical work: Equipment or situations in the real world are used to explo re ideas in mathematics. • Investigation: Students explore the issues (prob lems) in va ri ous mathematical situations. By far the most common teaching strategy utilized was exposition. Figure 4.21 show s the percentages of mathematics time used for the various activities in mathematics, with 51 o/o on average bein g dedicated to exposition, 21 o/o to problem-so lving, 15o/o to discussion and 11 o/o to practical work. Only 2o/o of t he time was dedicated to investig ati on . 4.4.2. How Mathematical Problems Were Presented and Solved a. Mathematical processes suggested by problem statements A different perspective that can be applied to the presenting and solving of mathematical problems is to compare the nature ofthe problem statements with the way in which the problems are publicly solved. Heibert et al divided the statements of mathematica l problems into three types: using procedures, stating concepts, and making connections. This analysis was app lied to al l independent and concurrent problems for wh ich a solution was reached publ icly. The category defin itions for each are: • Using procedures -- problem statement s that suggest the problem is typically solved by applying a procedure or set of procedures. These in clude using arithmetic with w hole numbers, fractions and decima ls; man ipu lating algebraic symbols to simplify expressions and solve equations; finding areas and perimeters of simple plane figures, and so on . Problem statements such as "solve for x in the equation 2x + 5 = 6- x" were class ified as usi ng procedures. Figure 4.21 Time for different learning activities in mathematics lessons Disussion 15% Practical 10% Exposition 52% 49  Stating concepts -- problem statements that call for a mathematical convention or an example of a mathematical concept. Problem statements such as "p lot the point (3, 2) on a coordinate plane" or "draw an isosceles right triangle" were classified as stating concepts. Making connections -- problem statements that imply the problem wi ll focus on constructing relatio nships among mathematical ideas, facts or procedures. Often such a problem statement suggests that students w ill engage in special forms of mathematical reasoning such as conjecturing, generalizing, and verifying. Problem statements such as "graph the equations y = 2x + 3, 2y = x- 2, andy= -4x, and examine the role played by the numbers in determining the position and slope of the associated lines" were classified as making connections. Relative to other countries, Indonesian teachers stated concepts more for problems while they used procedures less frequently. When comparing Indonesia to other countries, it can be seen in Figure 4.22 that Indonesia stated concepts much more often than in other co untries, wi th 35o/o compared to between So/o and 24%. Indones ia made relatively less use of procedures with only 41 o/o and made connections relatively frequently with 24o/oP b. Problems related to the real world and using mathematical language and symbols only Indonesian teachers tended to use the problem context of real-life situations relatively less often than in other countries. The average percentage of problems involving a real world context was 12o/o w hile that in other countries was between 9o/o and 42%. On the other hand, 88o/o of the problems in the Indonesian classroom involved mathematical language and symbols only, compared to 40o/o to 89o/o in other countries (Figure 4.23 below). Figure 4.22 Average percentage of problems per mathematics lesson of each problem statement type 100 90 80 70 60 so Make Connections State Concepts 40 • Use Procedures 30 20 10 0 Australia Czech Hong Kong Japan Netherlands United Indonesia Republic SAR States Source: Indonesia results combined with Hiebert, J. et. al., (2003), page 99 17 Note: The number of problems differs significantly from the amount of time for each type. Make connections was only 3.6% of the total time while state concepts was 60% and use procedures was 35% INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 50- A TIMSS video study of teaching practices and student achievement  Video Results and Cross- Country Comparisons Figure 4.23 Problems related to the real world and using mathematical language and symbols only 100 90 80 70 72 40 60 71 88 81 83 69 89 so 40 30 20 10 0 Set-u p contained mathematics • Set-up contained rea l life co nnection lang ua ge or sy m bo ls o nly Source: Indonesia resu lts combined with Hiebert, J. et. al., (2003), page 85 c. Problems that include drawing a chart, table or graph. The use of diagrams in Indonesia was relatively common while graphics and charts were less common . The average number of problems with a diagram was 3.4, w hich were found in 34 schools (48%) . Problems w ith a table were found on ly in seven (1 Oo/o) schools, with an average of 2.7 problems, and problems w ith graphics and charts were also found in seven (1Oo/o) schools w ith an average of 2.0 problems. d. Problems using physical tools The use of physical tools in Indonesia was found in over half the classrooms. Physical tools in mathematics learn ing include tools for measuring (e.g., ru lers, protractors), compasses, visua l aids tools, tiles, three-dimensional and t wo-d imensional geometry models, etc. Essentially, physical tools are tools that can be manipulated by teache rs and students. Sixty three percent of the lessons in the sample used physical too ls. The average number of problems wh ich requi red the use of physical tools was 3.7. e. Problems with more than one solution method The demonstration of more than one solution method is rarely performed . Mathematics problems often can be solved by using more than one method. If the teacher asks students to find different ways to solve the same problem, the stud ents wil l become more creative; this w ill increase students' abilities in reasoning and prob lem-solvin g. Figu re 4.24 below shows that 87o/o of the Indonesian lessons did not include problems w ith more than one so lution method. 12o/o of the lessons contained one problem with more than one solution met hod, and 2o/o of the lessons con tained three problems w it h more than one solution method (there were none with two prob lem s). 51  Figure 4.24 Percent of classes with a given number of problems with more than one solution method 100 87 90 80 "' Ql 70 "' "' l1l u 60 .... 0 50 .... r::: Ql 40 v 30 Qj c.. 20 10 0 2 0 0 2 3 Number of problems with more than one solution during class f. The percentage of lessons with at least one problem where more than one solution is presented Indonesia's results show that the number of problems where more than one solution is presented was lower than in other countries. Figure 4.25 below compares Indonesia's results to other countries. The 10% of lessons with one prob lem where more than one solution is presented was less than other countri es which showed between 16% and 42%. g. Lessons with examining-methods problems Indonesia had significantly more lessons with at least one "examining-methods" problem than in other countries. Examining-methods problems can include one of these activities: (i) students may choose the solution method; (ii) alternatives of solution methods are presented publicly; (ii i) at least one solution method is presented by a student fo llowed by a discussion or critic ism of the method, or (iv) there is a compa ri son of the method with other methods. Results of ana lysis of the transcripts show that 33.8% of the lessons had exam ining-methods activities. In comparison to other countries (Figure 4.26), this figure was quite high, with the next closest country only having 24% of lessons w ith at lea st one exam ining-methods problem. Figure 4.25 Percent of lessons with at least one problem where more than one solution is presented 100 90 80 70 60 50 40 Class had no problems w ith more than one soluti on 30 • Class had at least one problem w ith 20 more than one solution 10 0 Source: Indonesia results combined w ith table 5.1 in Hiebert, J. et. al., (2003), page 94 INSIDE INDONESIA'S MATHEMATICS CLASS ROOMS: 52- A TIMSS video study of teaching practices and student achievement  Video Results and Cross- Country Comparisons Figure 4.26 Percent of lessons that contained at least one examining-methods problem 100 90 80 70 60 so 40 34 30 24 14 17 20 12 8 10 3 0 Czech Austra lia Hong Kong Switzerla nd United Japan Indonesia Republic SAR States Source: Indonesia results combined w ith table 5.3 in Hiebert, J. et. al., (2003), page 96 Figure 4.27 Percent of lessons that h. Lessons that include summaries included a lesson summary The use of lesson summaries occurred in most classes. Summarizing keys things learned in the lesson is an important teaching and learning activity. The classroom observation and transcript results show that on ly 65o/o of the Indones ian lessons contained a summary (Figure 4.27). 4.4.3. Opportunities to Talk a. Words spoken by teachers and students in each lesson The number of words spoken by both teachers and students in Indonesia's classrooms was significantly less than in other countries. The average number of words spoken by t he teacher in a lesson (standardized to 50 m inutes) was 2,633 . Th ere we re teac hers w ho spoke on ly 813 words, and there were teachers who spoke up to 5,687 wo rds. In contrast, the average number of words spoken by the students in a lesson was 197. There were lessons where students spoke on ly an average of eight words per lesson, and there were also lessons where students spoke up to 1,539 words. Although there were many students but on ly one teacher, the number of words spoken by the teacher was sti ll much more than the number of words spoken by the students. Figure 4.28 be low shows th e average number of words spoken by the teacher and the students during the lessons compa red to other countrie s. As can be seen, Indonesia's numbers were significantly lower than other countries. For teachers the next lowest country still had more than twice as many words spoken. For students the next lowest had more than three times the words spoken. 53  Figure 4.28 Average number of words spoken by the teacher and students during a lesson Average Teacher and Student Words Per Class 7.000 5.798 5.902 6.000 5.360 5.452 5.536 5.148 5.000 4.000 3.000 2.000 .018 1.000 i?''b ~q, o" ~'I>~ 0 ~'I>~ ~ ~,<- ~,'1> ,'I> {-0 ~¢o q,'> ,_,.,-if. q,'- § 00 q,<- ~q,<:/. '?-v ~¢o .,._q,o ,~ ~ q, .;;;. ~~ (J" ~0 ~~' • Average Teacher Words per Class " <.,'-q, Average Student Words per Class Source: Indonesia results combined with data from figure 5.14 in Hiebert, J. et. al., (2003), page 109 b. Ratio of teachers' words compared to students' words Indonesia's teachers spoke much more than students, particularly when compared to other countries. The average ratio of the number of words spoken by the teacher to those spoken by the students was 25:1. This means that on average, teachers spoke 25 words while the students spoke one word . The corresponding figures for other countries in the TIMSS 1999 Video Study were 8:1 to 16:1 ((Figure 4.29). Teacher dominance in the classroom was still very evident wo rld w ide, at least in Grade 8 mathematics lessons, but the comparative figures show that students in other countries were far more active than Indonesian students. c. Words per sentence spoken by the teachers Indonesian teachers rarely spoke in long sentences. Fifty six percent of all sentences spoken by teachers were in the range of 1-4 words, which was much higher than the corresponding figures of 18% to 29% in other countries. For 24 words per sentence or above, the percentage in Indonesia was very small, at 3%, while for other countries it was between 25% and 41% (Figure 4.30). This shows that in Indonesia, teachers rarely spoke in long sentences. Figure 4.29 Average number of teacher words to every one student word per lesson Indonesia 25 Hong Kong 16 Netherlands 13 Switzerland 10 Czech Republic 9 Australia 9 United States 8 0 5 10 15 20 25 30 Number of teacher words to every one student word Source: Indonesia results combined with data given in Hiebert, J. et. al., (2003), page 109 INSIDE INDONESIA'S MAT HEMATICS CLASSROOMS: 54- A TIMSS video study of teaching practices and student ac hi evement  Video Results and Cross- Country Compari sons Figure 4.30 Average words per sentence spoken by teachers 100% 90% 80% 70% 60% SO% 40% 10+ uttera nces 30% 20% 5-9 utterances 10% 0% • 1-4 utterances *-0 t;;-0, c., '-1> ,_'-> "' :$) ~'(., ,, ~,7> ~1>0 o" 1>0 'l~ · 1> e'->' t;;-0, ,_o ~,_<:( "?-"" '> ~ ,_<.. 000 '-.0 --<--0 .;::,~ '"' "' c_;'v iS' ..;:."' Source: Indonesia results combined with Figure 5.16 in Hiebert, J. et. al., (2003), page 111 Figure 4.31 Average words per sentence spoken by students 100% 90% 80% 70% 60% 50% 40% 10+ utterances 30% 20% 5-9 uttera nc es 10% • 1-4 utterances 0% Source: Indonesia results combined with Figure 5.16 in Hiebert, J. et. al., (2003), page 112 d. Words per sentence spoken by students Students also rarely spoke in long sentences. The number of words spoken in one sentence (or a series of words) by students is shown in Figure 4.3 1 be low. In Indonesia, 79% of t he student s' utterances were in the range of 1-4 words per sentence, while in other countries it varied from 66% to 77%. The number of utterances between 5-9 words in Indonesia was 12%, wh ile in other co untries it was between 23% to 34%; and the number of utterances of9 words or above in Indonesia was 9%, compared to 4% to 9% in other countries. So we can see that the Indonesian st udents, like thei r teachers, rare ly spoke in long sentences. 4.4.4. Resources Used During the Lesson a. Tools and resources used The tools used by teachers provide insights into how teaching practices are conveyed. The resources used during the lesson can include chalk and boa rds, overhead or liqu id crystal display (LCD) projectors, textbooks, tools, rea l-world objects, etc. The learn ing tool or resource wh ich was primarily used in Indonesia was the blackboard. As many as 97% of the classrooms used blackboa rds, and in other countries th e percent age was 55  Figure 4.32 Use of various resources (proportion of videotaped classes where the given resource was used) • Indonesia • Lowest of other countries Highest of other countries "'C 5( 100 :::0 ~ 90 "' 3: 80 " ::! :::0 70 ~ ~ 60 " £ 50 .s: v 40 :<: 3: 30 .!; ~ 20 " ~ - ~ 0 #- 10 lnd OCs lnd OCs lnd OCs lnd OCs lnd OCs lnd OCs textbook! special math chalkboard projector worksheet calcu lator real·objects materials Source: Indonesia results combined with data from table 5.6 in Hiebert, J. et. al., (2003), page 114 Note: OCs is an abbreviation for"other countries" and the lower number represents the country with the lowest proportion of the videotaped classes in which the resource was used, while the upper boundary represents the country with the highest proportion. between 71% and 100%. In addition, in Indonesia 9% of the classrooms used projectors w hile in other countries projector use was between 3% and 59%. Similarly, 93% of the schools used textbooks as a teaching and learning resource, and 85% used special mathematics tools such as protractors, compasses, and graph papers, compared to a usage of between 30% and 86% in other countries (Figure 4.32). Use of real objects was relatively more prevelant in Indonesia than in other countries. Approximately 28% of the lessons in the sample used real objects compared to 4% to 21% in other countries. b. The use of calculators The percentage of lessons in Indonesia that used a calculator was very small --only 13% schools in the sample (Figure 4.33). This was in great contrast to the practice in other countries, w here the usage could be up to 91%. This situation in Indonesia was due to the schools' or teachers' policy of not permitting calculator use in the mathematics learning process, the intention being to familiarize students with the national mathematics examination during w hich students are not allowed to use calculators. Figure4.33 Percentage of lessons which used calculators 100 91 90 80 70 56 56 60 48 so 39 40 31 30 20 13 10 0 Source: Indonesia resu lts combined with data from table 5.6 in Hiebert, J. et. al., (2003), page 115 Note: Japan is not included because too few cases were reported. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 56 - A TIMSS video study of teaching practices and student achievement Section 5 Classroom Patterns: Indonesia's "Lesson Signature" The video study data allows for analysis of what takes place in the classroom over time. The videos are coded in a way where segments are created to identify what happens every second in each classroom. While this is useful for examination of the individual classroom, it also allows for combining across classrooms to identify common patterns of the particular lesson features analyzed in the previous section. Such patterns were labeled by Hiebert et. a/ (2003) to be a country's "lesson signature': The analysis in the following section merges all classroom data and then looks for patterns. The lesson signature is constructed by looking at the activities and teaching practices taking place in the 72 classrooms across the different layers for each percentage of lesson time elapsed. 5.1. Method of Constructing the Lesson Signature The classes were analyzed based on percent of time that passed rather than absolute time. The length of each class varies, with the videos in Indonesia's sample having a range of 40 to 100 minutes. In order to combine the data from each class, the actual time was not used but rather the relative time from 0 to 100% of class time. For example, for the class of 40 minutes, the 50% mark (halfway through the class) was 20 minutes, while for the class of 100 minutes the mark was 50 minutes. For each class the study team determined what activity was taking place as each percentage of the total class time passed (1%, 2%, 3%, etc.). 5.2. Pattern of Mathematical, Non-mathematical and Mathematical Organization Time Non-mathematical time and mathematical organization time tended to take place at the beginning and end of the lesson. Indonesia stood out in contrast to the other countries in that a relatively significant amount of time was spent on non-mathematical and mathematical organization time, with 89% dedicated to 57  mathematical activities compared to 95% to 98% in other countries. As can be seen in Figure 5.1 below, the non- mathematical time took place almost exclusively in the first 5% and last 5% of class time. This often involved an introductory prayer or opening ceremony or a closing activity. The fact that Indonesia dedicated more time to non-mathematical activities might indicate that class time is seen as not only a time for learning but also a time for cultural ritua ls. The fact that so much time was dedicated to these activities might be du e to the fact that Indonesia's lessons were significantly longer than in other countries (70 minutes vs. between 41 and 51 minutes) and involved more transitions between activities (e.g., from lecture to group work), requiring that the teacher spend more time on mathematica l organization activities. Figure 5.1 Lesson Signature of layer 1: Mathematical, non-mathematical and mathematical organization time Mathematical Time 100 80 60 0 10 2 0 304 0 506 0 70 8 0 901 00 Non-mathematical Time 80% 1 100% 60% 2g~ Ill ... 40% ...1 0 10 2 0 304 0 506 0 708 0 901 00 Mathematical Organization Time 100% 80% 1 ~!! olllllllllolllolllo. """'''''''''''"""''•''"''''" •· 0 10 2 0 30 4 0 50 6 ·•·······-- . ........... .. ... ...•• 1111. 0 708 0 901 00 Note: the three layers are mutually exclusive and add up to the full class time. Graphs show what pe rcent of the 72 cl asses are conductin g t he given activity for each percentage of lesson time elapsed 5.3. Pattern of Purpose of the Lesson Segment In the time dedicated to review and the presentation of new content and practice, Indonesia stood out in contrast to other countries in that less time was dedicated to review and more time was dedicated to practice. Figure 5.2 below shows that review in Indonesia took place in the beginn ing of class; by the time that 25% of the class time had passed, almost all review had been completed. In the lesson signatures of other countries (show n in Figure 5.3), by the time 20% of the class had passed, most classrooms had also stopped review (with the exception of the Czech Republic and the United States w hich had over 50% of classroom time dedicated to review) . Indonesia's difference is in both the percentage of classes that undertake review and the length of the review. Exam ination of the data indicates that 14% of all classes had no review take place, 24% spent less than one minute and 34% spent less than 5 minutes on review. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 58 - A TIMSS video study of leaching practices and student achievement  Classroom Patterns: Indone- s ia's "Lesson Signatu re" The general pattern of first conducting review, followed by introducing new content, followed by practice was the same as in other countries although Indonesia tends to begin practice earlier in the lesson. The lesson signature indicates that as early as within 40o/o of lesson time many classes were conducting practice activities and that by the second half of class time, most classes were conducting practice activities. In contrast, Figure 5.3 shows that most other countries did not tend to begin practice activities until after 60o/o-70o/o of the class time had passed. Figure 5.2 Lesson Signature of the purpose of segments: Review, new content, practice and assessment Review 100%1 80% 60% ;g;; lllllllllllllllllllllllllloo.,,. .... -· 0 10 20 30 40 so 60 70 80 90 100 New Material 80 % 100%1 60 % 40 % 20 % 0% ..• lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 0 10 20 30 40 so 60 70 80 90 100 Practice 100%1 80 % ~~:: O % L-~~~~~uw~~~uw~~~~uwwu~~uwwu~~uw~~~uw~~~ 0 ····················•••lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll. 10 20 30 40 so 60 70 80 90 100 Assessment 100%1 80% 60% 40% 20% 0 %~----------------~------------------------------~~--~·~··~·~··~··~·~··L- 0 10 20 30 40 so 60 70 80 90 100 - Note: Review, New content, practice and assessment are mutually exclusive and add up to the full "mathematics time" as specified in Layer 1. The non-mathematical and mathematical organization times does not contain these activities. Graphs show what percent of the 72 classes are conducting the given activity for each percentage of lesson time elapsed 59  Figure 5.3 Lesson Signature of the purpose of segments for other countries i Percentage of l~son time elap$ed 0 20 '' : 40 60 80 10 : Australia Revlewing I Introducing new content I I Pra<tlclng new content Czech Republic Reviewing I I Introducing new content I I Pra<tlelna new content ! .Hong Kong ReVIewing I Introducing nev content I Pra<tlelna new content ' I Japan ; Revlewing I -- Introducing new content I I Pra<tlelng new content I Netherlands Revlewlng Introducing new content ' I I Pra<tklna new content Switzerland Reviewing I Introducing new content I Pra<t lelna new content I ~ nited States Reviewing I I I Introducing new content i I Pr~ctkl na nPW cont<>nt 5.4. Problem vs. Non-problem Non-problem work tended to take place earlier in the class, whereas problem work, while consistently visible throughout the lesson, tended to be slightly higher in the second half of the lesson. As discussed in Section 4, Indonesia spent relatively more time on non-problem segments than other countries. Much of this activity involved, for example, mathematical information such as presenting or discussing new materia l or material previously presented, perhaps through a brief lecture by the teacher. It also involved contextual information such as describing the goal for the lesson and presenting historical background. Such activities were more likely to take place at the beginning of the lesson. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: .~ 60 -- A TIMSS video study of teaching practices and stud ent achievement  Classroom Patterns: Indone- sia's "Lesson Signalure" Figure 5.4 lesson Signature of problem vs. non-problem mathematics time Problem 100% 0 102 0 304 0 S06 0 708 0 901 00 Non-problem 100%: 80 % 60 % 40 I II ~ ~ llllllllllllllllllllllllllllllllllll111111111111llllllllllllllllllllllolllllll11lllllllll111111 2 % 0 102 0 304 0 S06 0 708 0 901 00 Note: Problem and non-problem time are mutually exclusive and add up to the full "mathematics time" as specified in Layer 1. The non- mathematical and mathematical organization times does not contain these activities. Graphs show what percent of the 72 classes are conducting the given activity for each percentage of lesson time elapsed 5.5. Public vs. Private Interaction As noted in the previou s sect ion, the t ime between pub lic and private interaction was 57% to 43% respective ly. As can be seen in Figure 5.5 below, t he beg inning of class tended to involve pub lic interaction, but by the 40% mark, the t wo were evenly divided. A slight bulge in public interactio n occu rred near t he end of class, often w it h groups re-gathering to discuss w hat had been practiced that day and performing tasks such as assigning homework. Figure 5.5 Lesson Signature: Public (full class) vs. private (small group and individual) interaction Public/ Full -C la ss Interaction 100 % 80 % 60 % 40 % 20 % 0% AUJ....._ LA.I.JU.ai...... 0 10 20 30 40 so 60 70 80 90 100 Private/ Individual Interaction 100 %1 80 % 60 % ;: : ..m.. lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllilllllllllllllllllllllllh,. O OfoL-~dWwuwo~uwwu~uw~wu~~uwwu~uw~wu~uwwuwu~uwwuwuuwuw~ 0 10 20 30 40 so 60 70 80 90 100 Note: Public/full class interaction and Private/Individual interaction time are mutually exclusive and add up to the full "mathematics time" as specified in Layer 1. The non-mathematical and mathematical organization times do not contain these activities. Graphs show what percent of the 72 classes were conducting the given activity for each percentage of lesson time elapsed 61  5.5.1. Public Interaction Breakdown (Teacher, Teacher and Student, Student) Public interaction showed slight patterns, but all forms were fairly evenly distributed. The different types of public interaction are show n in Figure 5.6 below. The method of teacher lecturing w as by far the most common form of public interaction, w ith teacher and student interaction (e.g., a question and answer segment) and student interaction (e.g., student presenting) being much less common . The teacher lecture had a slightly larger bulge at the beginning of class. Student and teacher interaction tended to happen slightly more at t he beg inning and end of the class. This was often due to students discussing their homew ork results or presenting the results of their practice assignment. Figure 5.6 Lesson Signature of public interaction breakdown Public Interaction - Teacher 80 % 1 100% 60 % 40 % 20 % 0% 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 0 10 2 0 304 0 506 0 708 0 901 00 Public Interaction- Teacher and Student 100% 80 %1 60 % 40 % 20 % 0% .lllllllllllll•l•••••l•···························l••••·l···························•llolllllllllo 0 102 0 304 0 506 0 708 0 901 00 Public Interaction- Student 80 % 1 100% I 60 % 40 % 20 % I ••lll111llltllll•llllllo•·••••lllllll•••••••••lll••••••••llllllllllll••••••llllllll•ll11l1 111 11 L 0 o/o _ o 1o 2 o 30 4 o so 6 o 70 8 o 90 1 00 __J I 5.5.2. Private Interaction Breakdown The pattern for private interaction was, again, surprisingly evenly disbursed across the lesson. Individua l student work that did not involve the teacher tended to be a bit more common in the latter half of the class, w hile teacher-student group interaction was slightly more common in the middle portion of the class. Group w ork in general (combining group work w ith teacher interaction and group work w ithout teacher interaction) tended to take place earlier in the class than individual w ork (either w ith or w ithout teacher interaction.) INSIDE INDONESIA'S MATHEMATICS CLASS ROOM S: 62- A TIMSS video study of teaching practi ces and student achi evement  Classroom Patterns: Indone- s ia's "Lesson Signature" Figure 5.7 Lesson Signature of private interaction breakdown Private Interaction - Teacher with Individual Student 100%1 80% 60% 40% 20% 0 %'------~--~·--~~~·•u·~·~·~·~•lulwl•u•w••u•wll~lu•l~l•ul~l•ulwllulw••~•ull~llul~l•h•~••ulwllululo~lull~l,dl~••ulwllu•~·~·~ 0 10 20 30 40 so 60 70 80 90 100 Private Interaction- Teacher with Group 100%1 80% 60% 40% 2 g~ '-----~...,, . . . . . . le&~n , I... d.. .. II"""•I .. II... I.... .. II IC&III .. II...,II I... ..IC&III... I... .. II"""II U I....... .. II"""II I .. I... IILIJU .. II I... ULIJU ... I... ..I... II... I ... .. IIl&JII I... II... l.... h.. • .. ll... .. nLIJI• •......~ 0 10 20 30 40 so 60 70 80 90 100 Private Interaction - Individual Student Work Only (No Teacher) 100%~ 80% 60% 40% 2 g~ '---~·.._, ., ... _,. ..... nulwhoal.udLMJIIulwll,.lulll&JUul.ull... ,_..,l... llulwllul.uiiLIJIIul.l..lll,.lullwllul.uU,.IuUwllulullwllul...,ll ... lullwllul.ull.._ll ... .. l...,ll ll.,lwloL- lullwhul.._,d... 0 10 20 30 40 so 60 70 80 90 100 Private Interaction - Group Work Only (No Teacher) 100%1 80% 60% 40% 20% 0o ;0 ' - - - - ----''LI'.ILI''u.•... ''LI.'u'''-'-''a.oiUOJ••oao.ILIII ... t..,••u.l.,.ll'-'-l'a.olwii._.ILI.J•I ..,..,,,._.,.. ,I... Iwu•LIJ·• ,,..,..,.''-'-'•oa•w•• •...,.wul.l..lllul..,, ._. .. llul.l..lll,.lull'-'-ll._.lwl• l ... .. ,~ •'"'"-"•...,,,.,.... 0 10 20 30 40 so 60 70 80 90 100 ----------- 5.6. Teaching Strategy Exposition tended to take place more often earlier in the class while problem-solving tended to take place later in the class. As noted in 4.4. I Teaching Strategies, exposition was the most common form of teaching strat egy, making up 52% of the time. Figure 5.8 indicates that exposition had a bulge in the earlier portion of the class whi le problem-solving tended to take place more often in the latter portion of the class. Discussion was fairly evenly spread but tended to happen most often in the middle of the class. 63  Figure 5.8 Lesson Signature of discussion, exposition, investigation, practical work and problem- solving Discussion 80 % 1 100% 60% 40 % 20% Oo/o ........................... lllllllllllllllllllllllllllllllllllllllh.......................... . 0 10 20 30 40 so 60 70 80 90 100 Exposition ~~ 1 1.11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111 0 10 20 30 40 so 60 70 80 90 100 Investigation 80% 1 100% 60% 40 % 20% .. O o/o L-------~~~~~~~~··~·~""~""~""~""~·~~-~-·~·~----~~-~~~~~~-- 0 10 20 30 40 so 60 70 80 90 100 Practical Work 80% 100 %1 60% 40 % 20% ---·~--~·•wlluluunh~ O % L_------~ ..~--~---·u--~ ..~·•uluhnii~Uwllulullnh~nwlluoullnhwu~uuouu~uw••wuuod••~h~••wuuouu~ ..~--~·--- 0 10 20 30 40 so 60 70 80 90 100 Problem Solving 80 % 1 100 60 % ~ 2 o/o g~ ulllllllllll•l•ll•••••.. •••• ... nulllllllllnllh•dllllllllllllllhlllfllllllllllllllllu. Ill 0 10 20 30 40 so 60 70 80 90 100 5. 7. Description of the Typical Pattern by the Study Team Before going into the actual data patterns, it would be useful to look at insights from the study team on what a typical lesson signature of a class was and what interesting activities were seen in the videos. Because of their expertise as mathematics teachers and teacher trainers, their descriptions capture how teachers are actually trained and contain subtleties that cannot be captured through simple coding. Three distinct sections were described: (i) the introduction stage, (ii) the development stage, and (iii) the closing stage. Introduction Stage: The lesson begins with checking the readiness of students for the class and may include homew ork discussion. For example, questions related to the homew ork assignment include how many problems INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 64 - A TIMSS video study of teaching practices and student achievement  Classroom PaLterns: Indone- sia's "Lesson Signature" they completed, whether there were problems that the students considered difficult and which problems need to be further discussed. The ways in which teachers discuss homework vary. Some teachers only ask for the answer of each problem oral ly while others ask students the answers of problems that they considered did not need further discussion with the remaining problems being described by teachers on the blackboard. There are some schools that use almost all the lesson time for doing and/or discussing the homework problems so there is no time to discuss new material. This is because only some students or no students actual ly comp leted the homework. Development stage: This stage conta ins the introduction of new content and usually beg ins by build ing the motivation of students w ith an explanation of the importance of studying the lesson, followed by the teacher posing questions of prerequisite knowledge that will be used in the development of the new material. At this stage, teachers usually discuss the facts and concepts of the new lesson. Understanding concepts and facts is usually done by providing practice, with prel iminary discussion of some examples. For teachers who are more aware of the need for proper time management, the problems that have been solved by the students are discussed as a full class. The students can identify what parts of the new lesson were not well understood so they can be discussed again. But there are some teachers who provide practice w ithout good planning, so many unresolved problems are given as homework. Closing stage: In this section, some of the teachers, often w ith students, bui ld summaries of the new lesson and give students tasks to work on as homework problems. Many teachers do not make a summary but instead directly give the homework assignment. 65  Section 6 Regression Analysis to Identify Relationships between Teaching Practices and Student Mathematics Scores Regression analysis is employed to determine what relationships exist between teaching practices and student mathematics scores. The coding and analysis presented in Section 4 provides deta iled, quantifiable insights into what happens in Indonesia's classrooms. Using the same methodology as seven other countries that conducted video studies provided a reference point for where Indonesia falls relative to these countries. But what do these numbers mean in terms of student achievement/ The fact that Indonesian teachers spend comparatively less time in review (for example) does not necessarily mean that Indonesia should pursue a policy of encouraging teachers to spend more time on review. The fact that Indonesian teachers and students speak significantly less than in other countries does not necessarily mean that more verbal interaction is necessary. By using the TIMSS sample for the video study, a unique opportunity is provided to identify relationships between student achievement (examination scores) and teaching techniques. This section first lays out the methodology used in identifying relationships between student achievement and teaching technique, followed by the results obtained. Many teaching practices emerged as having a statistica lly significant relationship w ith student mathematics scores. It must be recognized upfront, however, thatthe data only provides a snapshot with a single examination score so it is not possible to create a "before and after" picture of student achievement. It is therefore also not possible to determine the cause and effect relationship between teaching practices and student achievement. Still, the snapshot provide insights into how teaching practices relate to student achievement and, combined with theory and existing knowledge of what would be expected to lead to student achievement, a picture of what works in Indonesia can be at least partially formed. 66  Regression Analysis to Identify Relationsh ips betwee n Teac hing Practices and Student Mathemati cs Scores 6.1. Methodology of Regression Analysis A key challenge in education is to determine what can be done to improve student outcomes. In general, cou ntries develop their education policy based on w hat is believed w ill lead to students lea rning w hat they wa nt to and need to learn both in terms of specifi c knowledge and more general skil ls, va lues, and attitudes. 18 For examp le, Indonesia has been promoting a more student-centered learning approach 19 w ith the idea that it is more effecti ve for student learning, but is it true that the teacher-centered "chalk and talk" approach to teaching is less effective than a more interactive, student-centered approach 7 While the data from the video study certainly cannot provide a definitive answer to such a question, it does at least permit in sights into w hich classroom instruction and teaching practices have positive or negative relationshi ps w ith student learn ing. Regressions reveal the relationship between various teaching practices and student mathematics scores while controlling for many other factors that are known to have an influence on student achievement. In its simplest form, the classroom in struction and teaching practices captured through the cod ing of each video can be linked to TIM SS exam in ation results through t wo-variable corre lations, but such resu lts are mi sleading becau se they do not capture the comp lexity of student learning or the multiple fa ctors influencing student achievement. In order to ga in a more accurate picture of how teach ing practices are related to student ach ievement, a model must be co nstructed to separate out (co ntrol for) these multiple factors and to isolate teaching practices. The fol lowing section defines the model co nstructed for the regression analysis. 6.1.1. Steps in Framework Development Figure 6.1 Estimated influence of key factors on student Step 1: Recognizing the multiple achievement influences on student achievement Teachers l A critical first step in developing a 30o/o methodology was to recognize the complexity of, and multiple influences on, student achievement. Whi le teachers certainly play an important role in how much students learn, they are only a piece in the overa ll puzzle. Professor John Hattie Student Schools from the University of Auckland performed Characteristics 7o/o a meta-an alys is of various stud ies that 49o/o attempted to quantify influ ences on Home student achievement. Wh ile placement 7o/o of a percentage of influence on student Peers achievement should be viewed with 7o/o extreme caution, Hattie's meta-ana lysis of Source: Professor John Hattie from the University of Auckland of 51 studies at least provides a basis for developing a model. The results indicate that the biggest influ ence on student 18 19 The term "student-centered learning" has different interpretations, but in the case of Indonesia it is similar to what Rogers (1985) describes as the shift in power from the expert teacher to the student learner, driven by a need for a change in t he traditional environment where in this 'so-ca lled educational atmosphere, students become passive, apathetic and bored' (Rogers, 1986, page 25). The teacher-focused/transmission of information formats, such as lecturing, have begun to be increasingly criticized, and this has paved th e way for th e widespread growth of'student-centered learning' as an alternative approach. 67  achievement is student characteristics (socio-economic status, inherent intel ligence, etc.) which account for approximately half of student achievement outcomes. Teachers are the next biggest influence, accounting for approximately 30%. School, home and peer factors account for approximately 7% each. For the purposes of developing the framework, student, school, home and peer characteristics should therefore be taken into account and controlled for when linking student achievement and teaching techniques. Step 2: Separating teacher background vs. classroom instruction and teaching practices The next step in developing the framework was to separate out who a teacher is from what a teacher does . A teacher's background includes characteristics such as educational attainment, years of experience, whether the teacher has majored in mathematics and the teacher's level of motivation. What a teacher does in the class includes how the teacher structures the lesson, what the teacher does to prepare the lesson plan, the content of the lesson and the choice of teaching techniques. Figure 6.2 be low shows the separation between teacher background and classroom instruction and teaching practices. The dotted line from teacher background to student learning is intended to capture the fact that a teacher's background (competency, motivation, etc.) directly influences student learning, but that much of the teacher's influence is captured through his/her classroom instruction and teaching practices. Figure 6.2 Illustration ofteacher background vs. classroom instruction and teaching practices I Who a teacher is What a teacher does Cla ssroom instruction Teacher Stud ent and background t eaching practices learnin g ------------- ------------------------- --- ---- ----- -- --- ---------------- ----- ------ - ~ Step 3: Establishing a framework for analysis By combining the two above steps, a model was then constructed to show the multiple factors influencing student achievement. For the purposes of this study, classroom instruction and teaching practices are the main area of interest. Wh il e student, school, home and peer characteristics are certa inly of importance, they are included here as control factors rather than as the main factors of interest. Teacher background is also of interest but mainly in relation to how it relates to classroom instruction and teaching practices. Figure 6.3 below depicts how classroom instruction and teaching practices are the centra l focus of the model. While the model is most directly related to teacher background, it is also influenced by the other factors influencing student learning. For example, for a large class the teacher may tend to have students work frequently in sma ll groups, whereas for a small class more public (full class) interaction may be used more often. The teacher's background is con sidered the factor most related to cho ice of classroom instruction and teaching practices and therefore has the closest link. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 68- A TIMSS video study of leaching practices and student achievement  Regression Analysis to Identi fy Relati onships between Teaching Practices and Student Mathemati cs Scores Figure6.3 Framework used in analyzing classroom instruction and teaching practices Student learni ng and achievement 6.1.2. Use of Multiple Data Sources The above framework requires data for the various dimensions of student, home, school, peer and teacher background, along with classroom instruction and teaching practices. The TIMSS results (as described above in 3.2.3.2), along with additional data collected for the study, provide a rich array of data that can be used in the analys is. Video study data was also used, including: • Video coding : breakdown by second of all classroom activities • Observation instrument to complement the video • Teacher questionnaire • Student questionnaire • Review and ranking of teacher's lesson plan (score 1-40) • Homework evaluation 6.1.3. Recognizing the Limitations While there is value in attempting to identify linkages between student achievement and teaching techniques, the challenges in attempting such analysis are vast. These challenges must be kept in mind both for developing the models and in interpret ing the resu lts. A few of the key cha llenges to keep in m ind for the analysis include: • The multiple factors that affect student learning outcome are extremely complex and cannot easily be captured through surveys. Many unobservable factors exist and indicators that attempt to measure factors such as student and teacher motivation are relative to t he learning envi ronment. 69  • Because of data limitations, the given model may contain confounding variables, making conclusions on the cause and effect of teaching techniques on student outcomes potent ially inaccurate. If an extraneous variable in the statistical models presented below correlates (positively or negatively) with both the dependent variable and the independ ent variable, then a type I error exists. It is possible that an erroneous 'false positive' conclus ion could be made in linking teaching techn iques with student outcomes. It is therefore im portant to on ly view the results in terms of relationships and not cause and effect. • Teacher factors cannot easily be isolated from other factors. Good schools are likely to attract better teachers. The school environment is li kely to influence teacher motivation, with positive environments most like ly providing greater teacher motivation. Such factors create issues of multi-collinearity in models. • Teachers will choose techniques based on their personal strengths. For example, a teacher who has a stronger mathematics background may choose to present problems using mathematics language and symbols more often than real-life scenarios. • A teacher's technique will be, in part, chosen to address specific student needs. For example, the same teacher may choose different teaching techniques in a high ab ility classroom vs. low abil ity classroom or large vs. small class size. • Two teachers using the exact same technique may have different outcomes based on their abilities. For every teach ing technique, the study team found examples of what they thought were good and bad uses of the technique. A teaching technique used in a given classroom may have a negative relati onship w ith student mathematics scores, but it may be that a teacher who is properly using the techn ique can have positive resu lts. 6.1.4. Model Development Identification and Grouping of Teaching Techniques and Classroom Instruction Variables Th e main focus of the regression ana lys is is to identify the relationship between student mathematics scores and the various teach ing techniques and classroom instruction approaches used. The coding of the videos was done following the methodology developed by Hiebert et al (2003). It is the results of this coding that are to be ana lyzed in terms of student mathematics scores on the TIMSS exam ination, using the rich set of survey data in order to contro l for various factors. The main grouping and cod ing are presented in Table 6. 1 below: Table 6.1 Grouping of teaching techniques and classroom practices Grouping Variables Structure of Time Mathematical time Non-mathematical time Mathematical organization Activity Purpose Review New content Practice Assessment Teaching Strategy Discussion Exposition Investigation Problem-solving Practical work Interaction Type Public (full class) Private (small group or individual) INSIDE INDONESIA'S MATH EMATICS CLASS ROOM S: 70 - A TIMSS video stud y of teaching practi ces and student achievement  Regression Analysis to Iden tify Relationshi ps betwee n Teac hing Practices a nd Student Mathematics Scores Grouping Variables Public time Teacher only Teacher and student Student only Private time Individual only Individual with teacher Group only Group with teacher Problem vs. non-problem Problem Non-problem Routine vs. non-routine Routine problem-solving Non-routine problem-solving Use of materials Use projector Use textbook Use mathematics materials Use calculator Use real-world objects The percentage oftime spent on various teaching techniques was determined to be the most appropriate measure. The above variables were analyzed based on the amount of time in two forms: (1) absolute terms and (2) as a percent of total time for the given category. This distinction is important because classes are of different lengths and teachers use different techniques, so analyzing certain categories would be misleading if on ly viewed in absolute terms. Analyzing in terms of percentages is considered to be the more accurate measure. The time spent is calculated as the portion of time for a given grouping. In most cases it is a percent of total mathematical time, but in some cases it is the percent of a different grouping. For example, the sub-group ings of public time (teacher only, teacher and student, student only) are a percent of tota l public time rather than total mathematical time. Initial Identification of Predictor (Control) Variables through Stepwise Regression Over 1,000 student, teacher, school, community and home variables were available from the various surveys, but these had to be narrowed down to the key variables influencing student achievement. A method was necessary to choose the variab les to include in the models. An approach of log ic and initia l ana lysis was used to identify the best candidate variables. The models should certainly be based on what is already known about influences on student achievement. The use of educational theory and the results of previous studies were used as for initial selection. For example, the education level of parents has been shown in many studies to have an influence on student achievement. It was therefore important to use variables that capture this dimension. But it is possible that good variables that have been captured and, in fact, influence student achievement could be left out. In order to not allow these variables to slip through the cracks, an initial correlation analysis was performed between student mathematics scores and all survey variab les. Variables that had high correlation were brought to the next stage of analysis. Model Structures General model structures were then developed in order to lookatthe influence of the various teaching techniques on student learning. Two waves of models were developed, with the first wave being kept to a minimum number of variables and the second wave including additional variables also seen to be of interest and statistical significance. The third version within each wave includes dummy variables for 16 of the 17 provinces included 71  in the sample 20 Models both w ith and without province variables were run because while the importance of reg ional d ifferences is recogn ized, the dummy variables for half of the provinces tended to be dropped in the regress ion. While the resu lts are sti ll valid, it w as decided that both w ith-province and w ithout-province models would be of value. - Table 6.2 Models used for regressions Small set of key variables Logic behind model - Modell .l : Inclusion of a small set of key home, student, school and To get a picture of the relationship classroom variables but not including teacher background of teaching techniques with student variables mathematics scores regardless of teacher background - Model1 .2: Same variables as in model 1.1 but also including a small set of To see how the results change when teacher background va riables teacher background is introduced - Model1 .3: Same variables as in model 1.2 but including provincial To control for regional factors* variables Wave2 Similar to Wave 1 but with a larger set of variables - Model2.1: Inclusion of a larger set of key home, student, school and Same as model 1.1 but introducing classroom variables but not including teacher background additional variables found to be variables statistically significant - Model2.2: Same variables as in model 2.1 but including teacher Bringing in a larger set of teacher background variables variables - Model2.3: Same variables as in model 2.2 but including provincial To control for regional factors* variables For each of the above models the following general formula was used: (1) ,JJ + PupifA Math1 = Technique '1 + Home.Ah 1Pp + CommunitykA t-'cm + SchoolkA A A+ Teachernt-' 1-'s+Class nt-'c t + u, 1 • Math denotes the mathematics score of a given student i • Technique denotes the specific teach in g technique being analyzed from the list} of all teaching tech niques (It is im portant to note that on ly one teaching technique is analyzed at a time.) • Home denotes a vector of observed home characteristics of pupi l i • Pupil denotes a vector of observed characteristics of pupil i • Community denotes a vector of observed characteristics where school k is located • School denotes a vector of observed characteristics of school k • Class denotes a vector of observed characteristics of classroom n • Teacher denotes a vector of observed characteristics of teacher n The teach ing techn ique variables were inserted into the models one at a t ime. Fo r examp le, t he percent of time spent on introducing new content was put into Technique and the regression was run to obtain results. The va riable was then replaced by a new one (e.g., percent of time spent on practicing), and the regression was run again. A total of 250 teach ing techniques was ana lyzed for each of the 6 models, so a total of 1,500 regressions were run. In a given classroom there are many students who are all being exposed to th e same school, cla ss and teache r background characteristics. This creates a t ype of correlation (between observations) w hich is called an intra - 20 One province was leh out in order to be the baseline of comparison with the other provinces. INSIDE INDONESIA'S MAT HEM ATICS CLASSROOMS: 72 - A TIMSS video stud y of leaching practices and student achievement  Regression Analysis to Identify Relationships between Teaching Practices and Student Mathematics Scores class correlation. If th is is not t aken into account, the standard errors of the esti mates w ill be off, rendering sign ificance tests invalid. In order to address this issue, the records were clustered at the cla ssroom level. 6 .2. Regression Results The t-statistic was used as the outcome of interest because it captures both the direction and the statistical significance of the variables on mathematics scores. Th e full results of the regressions can be found in Appendix 5: Regression Results, but because a total of 1,500 regress ions was run, a consol idated presentation is given here. For the purposes of this section, the focus is on: a) Determining w hether each variable's relationship is positively or negatively related with mathematics scores, and b) Determ ining w hether the relationsh ip is statistica lly significant. An admittedly important dimension that is not shown in the fol lowing summary tables is the coeffi cient, w hi ch represents the estimated rate of change of one va ri able (y) as a fun ction of changes in the other. For space constraints, however, thi s va lue is not presented in thi s section and can in stead be seen in the append ix. The t-statistics were labeled based on their level of statistical significance. Although the resu lts wil l va ry depending on the reg ression, in genera l, at-statistic of over 1.67 is statistica lly significa nt at the 1O o/o level (90o/o con fid ence interva l); above 2.00, statistica lly significant at the So/o level (95o/o confidence interval); and above 2.58, statistically sig nificant at the 1o/o level (99o/o confidence interval). T-statistics of -1.67,-2.00 and -2.58 are also signifi can t at the 1Oo/o, So/o and 1o/o levels, but indicate a negative relationship. Table6.3 Legend for presentation of statistical significance and direction of variables in the regression POSITIVE relationsh ip with mathematics scores and statistically significant at 1% level (99% confidence level) POSITIVE relationship with mathematics scores and statistically significant at 5% level (95% confidence level) POSITIVE relationship with mathematics scores and statistically significant at 10% level (90% confidence level) NEGATIVE relationship w ith mathematics scores and statistically significant at 10% level (90% confidence level) NEGATIVE relationship w ith mathematics scores and statistically significant at 5% level (95% confidence level) NEGATIVE rela tionship with mathematics scores and statistically sign ificant at 1% level (99% confidence level) Variable is not included in the regression 6.2.1. Identification of Control Variables Through the process of stepwise regression analysis, a total of 27 key home, student, school, classroom and teacher background variables were identified for inclusion in the various models. In regression 1.1, a total of 10 va riables were included; six additional teacher background variables were included in model 1.2 for a tota l of 16; and the same 16 plu s additiona l provincial variables 21 are included in Model 1.3. Models 2.1 - 2.2 foll ow the sa me pattern, sta rting w ith 20 variabl es in model 2.1, then adding seve n teacher background variables in 2.2 and fina lly adding the provincial va ria bles in 2.3. The results of the t-statistics, with co lor coding fo r stati stical signifi cance, are presented be low: 21 The province variables are coded as dummy variables, with 11 dummy variables included and South Sulawesi left out of the regression so that it represents the baseline province. 73  Table 6.4 T-statistic results for home, student, school, class, community and teacher background variables (with mathematics examination score as the independent variable) -1 .28 -0.83 -1.45 1.19 0.95 0.75 -0.23 -0.71 Level of job satisfaction 0.35 0.91 0.85 Province dummy variables included N N y N N y INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 74- A TIMSS video study of teaching practices and student achievement  Regression Analysis to Identify Relationsh ips between Teaching Practices and Student Mathematics Scores The focus of this study is not to attempt to measure all characteristics leading to student achievement, but the results of the control variables are still of interest. The above variables are intended to serve as controls in measuring the relationships between teach ing techniques and student mathematics scores. Still, some brief points can be made about the results: • HOME: The home variables of parents' education and dictionary showed a strong positive relationship with test scores, meaning the higher the education level of the parents, the higher the student's test score was. Having a dictionary in the house also had a strong positive relationship. While the dictionary wouldn't directly be used in math, it may be an indication of the importance of education in the household and, in some cases, an indication of the fam ily's wealth. • STUDENT: Many student characteristics had a strong statistically significant relationship w ith mathematics scores. Age had a negative relationship, meaning older students tended to score lower. Because all students were in 8'h grade, the older age tends to indicate that the student had to repeat previous grades or started schoo l late. Students who work also scored lower. An 8'h grade student who is working tends to come from a poorer famil y, so this result is likely catching socio-economic factors. Student time spent on homework had a strong positive relationship, likely capturing student effort and motivation. • COMMUNITY: A community's poverty rate generally had a negative and statistically significant relationship with mathematics scores, although it was not as strong as what might be expected w ith confidence interva ls of only 90% for some of the regressions. Community size, on the other hand, tended to have a positive relationship wi th mathematics scores, but it was only statistically significant in the final regre ssion. • SCHOOL: Private and religious schools had a negative relationship with test scores, although in the final two models the private variable was not statistica lly sign ificant. Perception of student effort of their peers at school was also negatively correlated and is statistically significant at the 99% confidence interval. This is counterintuiti ve in that we would expect sc hool s w ith students trying harder to score higher. It may be that the concept of"students try" is relative, and in higher achieving schoo ls the measure of effort may be seen as very different from a small rural school. • CLASS: The classroom va riables tended to not be statistically significant with only class size showing a positive, statistically significant relationship in one regression model. • TEACHER BACKGROUND: Surprisingly few teacher background variables were statistically sign ificant. Only gender was statistically significant across models, where female teachers had a positive and statistically significant relationship w ith student mathematics scores. Years of experience, w hich had a simple correlation being positive and statistically significant in modell.l, did not remain statistically significant in subsequent models. Wh ile the simple correla tion of civil servant teachers had a strong, positive relation ship with student mathematics scores, thi s did not hold true in the regressions. 6.2.2. Analysis of Relationship between Teaching Practices and Student Mathematics Scores 100% Structure of Time 90% 80% In looking at the structure of class 70% 60% Non-mathematics time, students in classes that were Mathematics Organization SO% longer and had more time dedicated 40% • Mathematics 30% to mathematics tended to score higher 20% on examinations while classes with a 10% lot of non-mathematics time tended to .. . "" score lower. While it may seem intuitive , &-' that more time spent in class would result in higher scores, it also goes against one 75  concern that arose out of Indonesia's lengthy classes (70 minutes on average) which may be too long for keeping the attention of 8'h graders. The results in Table 6.5 below, however, indicate that students in longer classes tended to have higher mathematics scores. As noted in the earlier sections, Indonesian classes spent relatively more time for non-mathematical activities (3%) compared to other countries (1o/o-2%). The results showed a statistically significant negative relationship between mathematics scores and non-mathematics time, both in term s of total time and as a percentage of class time. Table6.5 T-statistic results for structure of time 100 Purpose of Lesson Segment 90 20 16 80 41 28 37 27 " " 70 60 22 The relationship between percent of 23 Prac tice 33 60 34 40 50 40 time spent on review and assessment New Content 30 • Review with mathematics scores was positive 20 10 and statistically significant. Indonesia Indonesia Australia Czech Hong Kong Japan Netherlands Swi tzerland Uni ted spent significantly less time on review Republic SAR States compared to other countries, but the results indicate that students in cla sses that dedicated a larger percent of the class to review tended to score higher. Assessment was also not often seen in the videos, but students in classes that spent a larger portion of time on assessment tended to score higher. Practice and new content tended to be negative, but only showed up as statistically significant in two of the models. Table 6.6 T-statistic results for purpose of lesson segment Public (full class) and Private 20 Czech Republic 22 (individual and group) Interaction United States 32 - M Classes that spent more time overall (in Indonesia absolute terms) on public interaction Switzerland 45 tended to have higher test scores, but Australia 48 there was no relationship between Netherlands 55 scores and percent of class time spent 0% 10% 20% 30% 40% SO% 60% 70% 80% 90% 100% •Publlc lnteraction Prlvatelnteraction on public vs. private interaction. Public INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 76- A TIMSS video study of teaching practices and student achievement  Regress ion Analysis to Ide ntify Relationshi ps between Teaching Practices and Stud ent Mathematics Sco res interaction showed a strong positive relationship with test scores in all models, but the fact that this relationship disappeared when looking at percent of class time may indicate that the longer classes simply tended to have more public interaction and that the higher scores were related more to amount of time than to structured proportions of the classes. Table6.7 T-statistic results for public (full class) vs. private (individual and group) Interaction Private interaction 0.24 -0.09 -0.17 -0.23 0.33 -0.1 3 -0.23 Percent public interaction -0.82 103 1.27 1.23 0.80 143 1.36 Percent private interaction 0.82 -1.03 -1.27 -1.23 -0.80 -143 -1.36 Focus on public interaction only Classes that involved more teacher-only public interaction tended to have lower mathematics scores, while classes with more student-only and student-teacher public interaction had higher mathematics scores. When examining the breakdown of time for pub lic interaction (Table 6.8), a pattern emerged. When the time involved the tea cher only (lectures), t here was a statistically sign ificant negative re lationsh ip with mathematics scores. When it involved both teacher and student, there was a statistically signifi cant positive re lationship. When it involved students only, the relationship was also positive (but is only statistica lly significant in two of the models). Such results generally indicate that there is a positive relationship between mathematics scores and classes w ith more active student participation. Table 6.8 T-statistic results for types of public interaction Focus on private interaction only There is no consistent pattern when looking at the breakdown of time for private interaction. The percent of group time with no teacher assistance tended to be negative and, in some cases, statistical ly significant. Otherwise the results provided very few insights. 77  Table6.9 T-statistic results for types of private interaction (as percentages of total private interaction) Variable Covariance R 1.1 R 1.2 R 1.3 R 2.1 R 2.2 R2.3 Private: percent individual tim e w ith teacher -1.93 -0.1 7 0.43 0.73 -0.08 0.05 0.31 Private: percent group time wit h tea cher 0.60 0.03 0.15 0.37 -0.14 0.07 0.45 Private: percent individual time, no teacher 1.10 1.42 0.72 0.77 1.66 1.05 1.05 Pri va te: percent group time, no teacher -0.01 -1.13 -1.36 -1 .93 -1.33 -1.20 -1.70 Teaching Strategy Disussion There was very little that stood out in terms of the statistical 15% significance for different types of teaching strategies. As shown in 4.4.1Teaching Strategies, the most common strategy was exposition, w ith a proportion of 52% of all teaching strategy time. 10% Exposition 52% Exposition tended to be positive, but not statistical ly significant. Overall, no teaching strategy emerged as having a clear 3% relationsh ip with student mathematics scores. This result may be due to the fact that the analys is d idn't focus on teaching strategies for specific activities. For example, exposition may tend to be more effective in presenting new material but less effective w hen conducting review. It may also be an indication that teaching strategies are complex and that the effectiveness of techniques cannot be captured through regression analysis, but instead must be analyzed through how the teacher chooses the appropriate strategy as we ll how the strategy is applied. Finally, from the review of the videos, it is clear that for each strategy there are teachers w ho use it effectively and others w ho are less effective. For teac hing strategies in particular, effectiveness may not necessarily be based on what the teacher does but how the teacher does it. More in-depth ana lysis of this area w ill be performed in subsequent phases. Table 6.10 T-statistic results for teaching strategies Variable Covariance R 1.1 R 1.2 R 1.3 R 2.1 R2.2 R 2.3 Percent of time for discussion -0.50 0.08 -0.05 0.05 0.06 -0.18 0.06 Percent of time for exposition 0.25 0.93 0.90 0.06 0.59 0.80 -0.15 Percent of time for investigation 1.21 0.03 -0.16 -1.64 0.80 0.56 -0.45 Percent of time for practical wo rk 0.67 -0.1 3 -0.25 0.27 -0.13 -0.25 0.20 Percent of tim e for problem-solving -0.76 -1.03 -0.52 0.29 -1.05 -0.65 0.07 Problem vs. Non-Problem Time lndone>iaE~ Japan 18 Hong I<Ong SAR Students in classes where teachers dedicated a 15 Australia·----------- Czech Republic 15 larger proportion of time to problems tended to 15 have higher mathematics scores. Indonesia spent Swltzorl•od ==========-~13 24% of mathematics time on non-problem activities, United States 11 wh ich is significantly more than other countries w hich Netherlands spent between 4% and 18%. The average percentage of • Problem Non-problem mathematics lesson time that was devoted to problem segments tended to have a positive relationship w ith mathematics scores and showed up as statistica lly sign ificant in al l models w hile time devoted to non-problem segments had a negative relationship, although it was statistica lly significant in only half of the models. INSIDE INDONESIA'S MATHEMATICS CLASSROOM S: 78 - A TIMSS video study of teachin g practices and student achievement  Regression Analysis to Identify Relationships between Teaching Practices and Student Mathematics Scores Table 6.11 T-statistic results for problem and non-problem time Use of Applications and Proofs Students in classes where a larger number of proofs were introduced tended to have higher mathematics scores. These results were statistical ly significant for all except the models in cluding provincial variables. On the other hand, the results were not statistically significant regarding the number of problems with applications. Both techniques wou ld be considered more advanced forms of problem so lving, so the fact that there was not a statistically significant relationship between mathematics scores and the use of applications wou ld go against the expected resu lt. Table 6.12 T-statistic results for use of applications and proofs 100 Method for Setting up Problems 15 90 80 There was a positive and statistically l4 70 significant relationship between student test scores and classrooms that had a higher 50 Make Connections State Concepts percentage of problems that involved making 40 30 • Use Procedures a connection. This result is logical in that the 10 technique of making a connection is typ ica lly 10 a more complex and chal lenging method for setting up problems. In thi s case the causal Australia Czech ~ng Hong Japan Netherlands United Indonesia Republic SAR States direction is particularly ambiguous because it is li kely that teachers w ith more advanced classes may simply have more opportunities to use this technique more often, but that the technique may or may not contribute more to student s' ability to learn. Alternatively, it is also possible that higher abil ity teachers are better able to set up problems using connections. Ta ble 6.13 T-statistic results for methods for setting up problems Mathematics Language vs. Real World Context Cla sses where problems were often discussed using mathematics language and symbols tended to have higher mathematics scores than those tending to use real life contexts. This result goes counter to some theories which encourage the use of real life mathematics (e.g. Bottoms and Sharpe, 1996) as we ll as Indonesia's 79  100 program of Contextual Teaching and Learning (CTL). In 90 add ition, an Asian Development Bank study found that in 80 Indonesia, 60% of the learners are contextual learners (as 70 opposed to conceptual learners), w ith contextua l learners 72 40 60 88 81 83 69 71 needing extra clarification in order to understand concepts 89 50 taught by the teacher (AOB, 2001 ). One important point 40 about the TIMSS testing instrument is that it was designed 30 with more of a mathematics language, symbols and 20 procedures orientation. This can be contrasted with the 10 Program for Internationa l Student Assessment (PISA) which is designed from the perspective of applying mathematics to real life situations. The positive relationship found Set-up oontained mathematics • Set-up contained real life connection between mathematics scores and the use of prob lems language or symbolsonly with mathematics language and symbols may indicate that students in classes that tended to use this problem approach were better prepared for TIMSS questions, but the result may be quite different if the questions were simi lar to PISA. Table 6.14 T-statistic results for types of mathematics problem language Use of lnstrumentsffools Students in classes where a projector was used tended to have higher mathematics scores, while classes that used textbooks tended to have lower mathematics scores. The fol lowing data was gathered through observation so these variables capture whether the instruments were used at the time the class was observed. The students in classes that used a projector during the lesson tended to have higher mathematics scores, with the relationsh ip being statistica lly significant in the last three models. Students in classes that used textbooks or alternative books tended to have lower mathematics scores, and the relationship was statistically sign ificant in all models. The use of mathematics materials also tended to be negative and statistically significant. These results may not reflect on the quality or usefulness of the textbooks or materia ls themselves but could instead be due to less experienced or capable teachers having to rely more on them . More experienced or capable teachers may be able to develop and conduct lessons with less need for supporting materials. Table 6.15 T-statistic results for the use of instruments during class -Variable Use textbook Use alternative books Use mathematics materials Covariance -1 .80 0.72 '- -- R 1.1 -2.04 -2.45 -0.79 R 1.2 -2.12 -2.48 -2.28 R 1.3 -1 .85 -2.05 -1.93 R 2.1 -1.70 -2.31 -0.84 R 2.2 -1.83 -2.92 -2.35 R 2.3 -1.68 -2.48 -1.97 Use calculators -2.43 -0.89 -0.77 -0.34 -1.14 -1.12 -0.70 Use real world objects -0.01 -0.51 -0.55 -0.60 -0.69 -0.46 -0.58 INSIDE INDONES IA'S MAT HEMATICS CLASSROOMS: 80- A TIMSS video study of teaching practices and student achievement  Regression Analysis to Identify Relationships between Teaching Practices and Student Mathemati cs Scores Lesson Planning In lesson planning, the students in classes with teachers that specified they spent more time developing the lesson plan and/or had developed it with another teacher tended to have higher mathematics scores. Other ca tegories did not show a stro ng pattern. Table 6.16 T-statistic results for lesson planning Variable Covariance Time developing lesson plan beforehand 0.87 Lesson plan development with other person 1.85 Lesson plan development with a group 1.74 159 135 0.93 0.13 0.04 -1 .29 Time for developing lesson plan 0.82 -1.09 -063 -053 -0.06 -0.6 -154 Evaluation score of teacher's lesson plan 1.66 0.68 0.36 149 125 138 0.62 Teacher Influences Teachers who said their lessons were influenced by the curriculum and national test tended to have classes with higher mathematics scores. Teachers were asked, on a sca le of 1 to 3 (none, some, a lot), to w hat degree they were influenced by various items. There was a strong positive rel ationsh ip between mathematics scores and teachers w ho said they were influenced by the cu rri culum. There was also a positive relationship between mathematics scores and teachers w ho said they were influ enced by the national test. Table 6.17 T-statistic Results for Teacher Influences 6.2.3. Summary of Regression Results As was stressed earlier, the regression results presented here must be viewed as simply identifying patterns and relationships between student mathematics scores and teaching techniques. They should not be interpreted as showing any causal effects. It is also important to stress that, beca use the mathematics scores are for a single test and stu dents build up their mathematics abiliti es ove r the course of many yea rs w hile working with various teachers, attributing student test resu lts solely to th e teaching techniq ues of the 8'h grade mathematics tea chers wo uld be a presumptuous leap. In order to address these wea kn esses, the 2011 phase of the study w ill include additional before- and after- tests for participating students. At this stage, the relationships should be viewed only as providing initial insights into th e link between teaching practices and student mathematics scores. Even with keeping the analysis limitations in perspective, many interesting results emerged. Indonesia has been pursu ing a po licy of more student-centered learn ing, and va rious ind ica tors of classes w here students 81  are more involved and active emerged as having a positive relationship with student mathematics scores. Classroom management practices also emerged as positive, where classes that had more t ime dedicated to mathematics activities and problem time tended to have higher test scores. How mathematics is approached may also be related to mathematics scores, w ith having problems that make a connection, setting up problems with mathematics language and symbols, and having problems that involve proofs all tending to have positive relationships with test scores. How teachers prepare their lessons may also be related to student learning, with teachers using lesson plans prepared beforehand and lesson plans prepared with others being positive. Relationships also emerged re lated to what too ls are used in the classroom, with classes using projectors being positive wh ile classes using textbooks had a negative relationship. Teacher influences also play a role, with students in classes where teachers state that their lessons are influenced by the national curriculum and national examination tended to score higher. These insights are an initial step in understanding the linkages between teaching techniques and classroom practices with student mathematics scores. Many of the resulting relationsh ips are supported by theories of good practice in teaching. The results also help in understanding the relationships that may exist within the context of an Indonesian classroom rather than classrooms in general, since cultural factors make effective teaching and learning contextual rather than universa l. The insights are a foundation for additional analysis of rather than a final answer about, on effective teaching practices. Within each activity, examples of what are likely to be effective or ineffective teaching can be found. For example, just because review of previous material had a positive relationship with student test scores doesn't mean that simply increasing the time dedicated to review of previous material will lead to increased scores. It is much more important what the teacher does within the activity and how the teacher engages the students rather than the amount of time spent. The results do point to w hat activities appear to be important, though, and help in determining where to focus additional research. This study is continuing with the analysis of the 2007 data and will also include a 2011 phase. The results have provided a strong basis for additional qualitative analysis in the areas of teacher-student dialogue, how teachers pose questions, classroom management, student engagement and innovative use of tools. The resul ts have also been integral in enhancing the design for the 2011 phase, w hich w ill involve case studies and wi ll probe into teacher beliefs and orientation towards mathematics teaching as well as the role of how teacher subject knowledge and pedagogical skills shape what takes place in the classroom . It is hoped that the study results w ill spur further research that can help improve the effectiveness of what takes place in Indonesia's classrooms. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 82 - A TlMSS video study of teaching practices and student achievement Section 7 Summary and Implications The findings in the last section provide a rich picture of Grade 8 mathematics teaching in the Indonesian classroom. From the results, we can identify a number of pos itive aspects as well as potentia l areas for improvement of mathematics teaching in Indonesia when compared with other co untries in the TIMSS 1999 Video Study and through analysis of the relationships between teach ing practices and student outcomes. 7.1. Positive Aspects of Mathematics Teaching in Indonesia Many positive results emerged from the analysis, indicating that Indonesia is employing many good practices. In many cases, the results surpass those of other countries. Among t he results: • The classroom environment is often conducive to learning, and mathematics tea ch ing in Indonesia is mainly conducted w ith few outside interruptions. • Indonesia has relatively more lessons with goal statements and lesson summaries which should lead to improved clarity and flow of the lessons. • Although only a few problems we re categorized as high comp lexity, a large proportion of problems are of medium complexity, with re latively few of low complexity when compa red to other co untries. • Students are given ample opportunity of practicing w hat they have just learned in the lessons. • Compared to other countries, students have more time working in sma ll groups. • There is more use of real-life objects in the lessons than in comparator countries. • Relative to other countries, a larger propo rtion of teachers use the set-up approach of making a connection, and this approach was found to have a positive relationship with student mathematics scores. 7.2. Potential Areas for Improvements in Mathematics Teaching The findings of this video study also point to some areas for improvement in the mathematics classroom organization and instructional practices in Indonesia: • The duration of gra de 8 mathematics lessons is rather long compared to other countries. As a result, students may not be ab le to concentrate on the subject matter to be learned for the whole duration of the lesson. • Relative to other countries, much more lesson time is spent on non-mathematical and mathematical 83  organization work with a result that less lesson time is spent on teaching and learning mathematics. The amount of time spent on non-mathematical activities has a negative relationship w ith student mathematics scores. • Within mathematical time, Indonesian teachers spend comparatively less time on problems, and students from classes that have relatively more non-problem time tend to score lower. • Comparatively less time is spent on reviewing what has been learned in past lessons before going on to introducing now content although there is a positive relationship between classes that spend more time on review and student mathematics scores. • Assessment is relati vely rare, but students in classes that have more assessment time tend to have higher mathematics scores. • Relatively little homework is given, and much lesson time is consumed on practicing . • Both teachers and students speak relatively few words in the lessons, and the lengths of their utterances are short in general. • The ratio of student words to teacher words is very low compared to other countries. • Very few of the mathematics problems dealt with are of high complexity. • There are few problems involving appl ications. • The choice of different solution methods is not stressed, w ith most teachers only focusing on a single solution to problems. • Not many students have the chance to examine the methods of solution of problems . • Calculators are rarely used in the classrooms. 7.3. Additional Observation Notes from the Videos Beyond the coded videos, additional observations were made by the core team regarding teaching practices in Indonesia. Although the cod ing of the videos provides for objective data analysis, it cannot always capture what the observers of the videos could see. The study team (who are mathematics experts and practitioners themselves) noted interesting patterns and felt that certain activities were not being properly conducted. Recommendations included: • There is a need to apply better time management in the classroom and to use the time effectively to teach relevant content • More emphasis should be put on higher order thinking in the instructional delivery • There was often a mismatch in level of content coverage (the leve l and the amount of the con tent covered is equal to the level and the amount understood by a student) • There is a need to create a more stimulating environment that wi ll maintain student engagement and involvement 7.4. Implications for Educational Policies While any policy measures to be taken need to ensure that the many strengths of mathematics teaching in Indonesia as listed above are not lost, the various deficiencies above point to some specific improvement measures. While this upgrading of qualification exercise introduced by the Teacher Law is in the right direction and should be applauded, it is important to remember that mere upgrading of qualification is not sufficient for high quality teaching. In particular, the educational background of teachers should match the subjects that they are teaching . In the event that this is not the case, in-service professional development activities need to be provided to make sure that the teacher is able to build on his/her qualifications to develop expert knowledge in the field that he or she is teaching. INSIDE INDONESIA'S MATHEMATICS CLASS ROOMS: 84 - A TIMSS video study or teach ing practices and student achievement  Summary and Impli cations The relevant authority should review the organization of lesson time. Seventy minutes per lesson may be too long for children of Grade 8 (although the regress ion results indicate that longer classes actually have a positive re lationship with mathematics scores). More importantly, measures need to be taken to reduce the organization work of the teacher during the lesson so that more time can be devoted to the most important activity in the classroom- that of teaching. Possibly a pi loting of more but shorter classes cou ld be conducted and set up to determine w hich is more effective in terms of both student outcomes and teacher and student satisfaction. The policy for the use of calculators in mathematics examinations should be reviewed . The calculator is not merely a calculation device. When used properly, it is an extremely useful tool for learning (e.g., in exploring number patterns) (Fey and Hirsch, 1992). And if graphing ca lculators are uti lized, it contributes even more positively to mathematics teaching and learning (e.g., in linking algebra and geometry) (Ruthven, 1990; Embse, 1992; Shoaf-Grubbs, 1995; Penglase and Arnold, 1996; Doerr and Zangor, 2000). Examination policies have strong backwash effects on teaching, especially for a country such as Indonesia which puts a strong emphasis on publ ic examinations. So in order to enhance mathematics teaching and learning in the classroom through the capitalization of the strengths of the calculator, a review of the calculator policy in examinations is important. The policy of promoting student-centered learning appears to be a valid approach in the Indonesian context, with the more student-centered classes tending to have higher mathematics scores. The relat ively low number of both teacher and student words compared to other countries, as we ll as the relati vely high amount of teacher speaking time com pared to student time, indicates that the student-centered approach is not being implemented in many cla ss rooms. Methods to further promote student-centered learning in mathematics should be pursued. Tea cher training and supervision programs could leverage both the results of the video study and the videos themselves to enhance teacher training programs. Visually w itnessing effective and less effective practices can be a powerful teaching method. Other countries that conducted the TIMSS video study have incorporated the videos into training activities; Indonesia could do the same. The videos could also be leveraged to train head teachers and supervisors who are involved in assessing and providing feedback to teachers. 7 .5. Implications for Teachers As pointed out above, many of the problems dealt with in the Indonesian classroom were of low complexity. While the teacher should always pitch the level of difficulty and comp lexity of the subject matter towa rd s the level of the students, care needs to be taken not to repeated ly reduce the difficulty level of the content. This is an endless retreat and in the end not co ndu cive to enhancing student achievement. In particular, proof and applications are both important characteristics of mathematics and should occupy a proper pl ace in mathematics teaching and learning. Mere procedural problems are not enough to raise the achievement level of st udents. Developing flexibility in the approach to the solution of problems is an important aim of mathematics education. This can be en han ced by discussing more with students different ways of tackling problems (exam ining methods) and by encouraging different sol utions to the same mathematical problem. Communication in mathematics is another important aim of mathematics education. A noticeable finding of this study is the reticen ce of both teachers and students in the Indonesian classroom. While this may be rooted in the Indonesian culture itse lf, teachers need to realize the importance of communication in the learning of mathematics. Students need to be given the chance and the encouragement to express themselves verbal ly, in addition to in wri ting. They sho uld be encou raged to talk more and in longer phrases or sentences. 85  In this regard, the teachers themselves also need to talk more and in longer sentences to stimulate students and to act as role models for their students. Assessment and review activities are very rarely used, but both appear to have a strong positive relationship with student mathematics scores. The increased use of assessment may assist in increasing student learning. Review of previous material (homework, etc.) may also be important to stress continuity between lessons as well as to reinforce key concepts. Lesson planning activities are an important aspect of successful teaching. Teachers who spend more time on lesson planning tend to have students scoring higher. Working with another teacher on the lesson plan also has a positive relationship, possibly indicating that lesson planning activities conducted in teacher working groups are beneficial. More efficient and targeted classroom management could lead to improved outcomes. Th ere is a positive relationship with teachers who spend less time on non-mathematical activities and spend more of their mathematical time on problem activities and student test scores. Indonesian students have very little homework relative to other countries. At the same time, a large amount of class time is devoted to conducting practice activities. While practice in class can have the benefit of students being able to directly discuss problems with the teacher and other students, it appears that class time is often being used to conduct practice that could be done in the form of homework. 7 .6. Concluding Remarks In any analysis of student achievement in Indonesia, the complexities of teaching must be kept in mind. There is not a single, correct way to teach mathematics, and this report is not intended to define a magical combination of teaching techniques to be used in Indonesia. Each classroom is different. It is critical that teachers be able to understand their own classroom situation, including the level of ability and specific needs of their students and the context in which mathematics w ill be most useful and understandable to them, and then be able both to use teaching practices that will best fit w ithin that context and to adapt those practices dynamically as the needs of their students change over time. This requires training teachers to be, above all, "ref1ective practitioners" of their own work, able to see themselves and assess their own performance in the classroom- and to be able to help other teachers do the same. The abundant data generated from the video study has already provided rich information on Grade 8 mathematics teaching in Indonesia, and the comparison with results of the TIMSS 1999 Video Study and the regression analysis have pointed to important policy and classroom implications for the country. This is the first phase of the two-phase study, which wi ll be followed with a replication study in 2011. The full results w ill prove to be even more powerful in informing policy and practice. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 86 - A TIMSS video stud y of teac hing practices and studen t achi evement Appendix Appendix l: Indicators for the Preliminary Research Questions Research questions Indicators: teacher behavior or teacher actions to bring about the relevant student behavior 1. Does the teaching help 1. Giving alternative representations of a concept students understand 2. Classifying objects based on specific characteristics mathematics concepts? 3. Giving examples and non-examples of a concept 2. Does the teaching enhance 1. Presenting mathematics statements in written form student communication in 2. Presenting mathematics statements verbally mathematics? 3. Presenting mathematics statements using tables/ pictures/ diagrams, etc. 3. Does the teaching enhance 1. Making a reasonable hypothesis student ability in reasoning? 2. Testing a hypothesis 3. Drawing correct conclusions 4. Showing a proof or giving reasons for a mathematics argument 5. Checking the validity of an argument 6. Finding patterns or making generalizations in mathematics 4. Does the teaching help 1. Organizing and/or reorganizing data develop student ability in 2. Discussing the choice of relevant information in solving a problem problem-solving? 3. Applying alternative methods in solving the same problem 4. Discussing the choice of approaches or methods in solving a problem 5. Discussing general problem-solving strategies 6. Developing and/or interpreting a mathematics model of the problem 7. Solving non-routine problems 8. Checking the procedure and/or solution of a problem 9. Looking back and/or drawing lessons after solving a problem 5. Does the teaching enhance 1. Doing mathematical manipulations student competence in 2. Applying suitable algorithms to solve a given problem applying mathematics 3. Practising mathematics skills procedures? 87  Appendix 2: Indicators and Data Sources for the Research Questions Questions Indicators Sources of Data Code Planned Analysis 1. How do teachers Lesson planned Lesson plan LP (look at LP Use Excel to prepare before systematically rubrics for teacher encode and teaching? certification) analyze data Lesson based on the Curriculum guide LP Use Excel curriculum and lesson plan The t ime teachers need TQ2#10, 11 to prepare recorded lesson Time for teacher preparation of typical lesson 2. Whatis the mathematics Mathematics content Lesson plan (or LP,OR Use Excel and content? taught observation report) Studiocode 3. What are teachers' Accuracy (easier to code Vdeo Under non-problem Excel & Studio abilities/competencies in misconceptions) branches teaching mathematics? Encourage student's Video Under problem Studiocode reasoning skills (proof, branches connection) Encourage student Video Student and Studiocode communication teacher public skills (student public interaction, student interaction, group public interaction, individual interaction, group interaction, group and teacher teacher&group individual interaction, interaction ~udentandteacher public interaction) Encourage student Video Problem-solving Studiocode problem -solving skills strategy and investigation (strategi pemecahan soal! strategy (see also problem) how students work on problems) Encourage student skills Video group interaction, Studiocode to work cooperatively teacher&group interaction 4. How is time Time spent by class on Video Main tree: review, Stud iocode management during the events for (mathematics: new lesson, practice, lesson? review, new lesson, assessment, non- practice, assessment, mathematics non- mathematics and mathematics and mathematics organization organization) INSIDE INDONESIA'S MATHEMATICS CLASS ROOM S: 88- A TIMSS video study of leaching practi ces and stud ent achi evement  Appendix Questions Indicators Sources of Data Code Planned Analysis Sa. What types of Kind of problem (nature, Video nature, context, Excel & Studio mathematics problems context, qua lity, type of quality, type of do students solve? problem) problem Number of problems Lesson plan and Excel & Studio solved in class video Sb. How are mathematics How problems are Video under how Excel & Studio problems solved? worked on (trial and problems are error, make a pattern, try worked on a simpler one, working backwards, using graphs, tables, or diagrams) 6a. What teaching Strategies used Video exposition, strategies are used by (exposition, discussion, discussion, the teachers? investigation, problem- investigation, solving, practical work) problem-solving, practical work Teaching aids/resources List List used 6b. What types of Types of questions (open, Video Types of questions, questions do teachers closed, routine, non- under exposition ask? routine, Y/N, rhetoric) non problem, quality under kind of problem 7. How do teachers assess Type of assessment tasks/ Lesson plan LP (code based on Excel & Studio student learning? questions using procedure, stating concepts, making connection, proof, real world, mathematics language, open, closed, routine and non-routine) Assessment results Refer to TIMSS refer to TIMSS result Excel & Studio resul ts 8.What are the indicators Attended seminars, Questionnaire TQ1#16,17c Excel to show teachers' workshops, MGMP motivation to improve meetings, trainings etc. her/his teaching skills? Read books, articles, Questionnaire TQ2 #3f,g,h, i,j Excel journa ls, etc., and TQl #28* other med ia materials (including on li ne resources) pertaining to mathematics and in related areas Tried out what has been Questionnaire TQ2 #3j Excel learnt from reading, training, etc. 89  Questions Indicators Sources of Data Code Planned Analysis 9. What learning resources Text books, computers, Questionnaire TO 1 # 21 a,b,c d,e,f,g are used for supporting calculators, teaching aids, teaching and learning? VCD, LCD, OHP 10. What are the profiles of The educational Questionnaire TO 1 # 10,11 teachers? background of teachers (highest level of education, subject matter) Teaching experience in Questionnaire TO 1 # 12 math Age Teacher status, Gender 11. What are student Attitude toward Questionnaire so# 16 attitudes toward mathematics mathematics? Perception about Questionnaire so# 17 mathematics Teacher perceptions Tl3a INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 90- A TlMSS video stud y of teaching practices and student achievement  Appendix Appendix 3: Summary of Definitions Note: Most of the definitions come directly from the 1999 TI MSS v ideo study in order to have common cod ing that allows for comparison with the other countries that participated in that study. All descriptions that come from the 1999 study are marked with an asterisk(*). Description Definition [ Structure ofnme ----------~~------------------------------------------------------~ Mathematical time* Time spent on mathematica l content presented either through a mathematical problem or outside the context of a problem . Examples: talking or reading about mathematical ideas, solving mathematical problems, practicing mathematical procedures or memorizing mathematical definitions and rules). Mathematical organization At least 30 continuous seconds devoted to preparing materials or discussing time* information related to mathematics but not qualifying as mathematical work. Examples: distributing materials used to solve problems, discussing the grading scheme to be used on a test or distri buting a homework assignment. Non-mathematical time* At least 30 continuous seconds devoted to non-mathematical content. Examples: talking about a social function, beginning or ending a class with a prayer, ca lli ng the roll, disciplining a student while other students wait, or listening to school announcements on a public-address system. ILesson Segment Purpose Review* Th is category, more technically ca lled "addressing content introduced in previous lessons;' focused on the review or reinforcement of content presented. These segments typically involved the practice or appl ication of a topic learned in a prior lesson or the review of an idea or procedure learned previously. Examples include: 1. Warm-up problems and games, often presented at th e beginning of a lesson; 2. Review problems intended to prepare students for the new content; 3. Teacher lectures to remind students of previously learned content; 4. Checking the answers for previously completed homework problems; and 5. Quizzes and grading exercises. Introducing new content This category focused on introducing content that students had not worked on in an (New content)* earlier lesson. Examples of segments of this type included: 1. Teacher expositions, demonstrations, and illustrations; 2. Teacher and student explorations through solving problems that were different at least in part, from problems students had worked previously; 3. Class discussions of new conten t; and 4. Reading textbooks and working through new problems privately. Practice* This category focused on practicing or applying content introduced in the current lesson. These segments only occurred in lessons where new content was introduced. They typically took one of two forms: the practice or application of a topic already introduced in the lesson or the follow-up discussion of an idea or formula after the class engaged in some practice or application. Examples of segments include: 1. Working on problems to practice or apply ideas or procedures introduced in an earl ier lesson; 2. Class discussions of problem methods and solutions previously presented; and 3. Teacher lectures summarizing or drawing conclusions about the new content presented earlier. Evaluation/ Assessment Exams or quizzes that are given to students in order to eva luate their knowledge. 91  Description Definition Public and Private Interaction Full class I Public interaction* Public presentation by the teacher or one or more students that is intended for all students 1. Teacher interaction The teacher lectures to all students. 2. Teacher and student Presentation made by both teachers and students (in intervals), for all students. interaction 3. Student interaction Presentation made by students, aimed at the teacher and all students Small group or individual/ All students wo rk at their seats, either individually in pairs, or in small groups, while Private interaction* the teacher often circulates around the room and interacts privately with individual students 1. Teacher and group Students work in groups or have discussions with the teacher going from group to interaction group to provide guidance. 2. Group interaction Students work in groups or have discussions without teacher's guidance. 3. Teacher and individual Teachers provide individual counseling to students. interaction 4. Student interaction Each student works alone with no interaction with the teacher. Student presents information* A student presents information publicly in written form, sometimes accompanied by verbal interaction between the student and the teacher or other students about the written work; other students may attend to this information or work on an assignment privately. Problem Solving Strategy Exposition The teacher lectures while students listen and answer closed questions (with no discussion). Discussion The teacher and student or students discuss their own ideas about mathematics. Problem solving The teacher provides a problem I situation as a basis to discuss ideas in mathematics. Practical Equipment or situations in the real world are used to understand ideas in mathematics. Investigation Students explore the issues (problems) in various mathematical situations. Problem vs. Non-Problem Problem* Events that contained a statement asking for some unknown information that could be determined by applying a mathematical operation. Simple questions asking for im mediately accessible information were not coun ted as problems. Examples of mathematical problems included: 1. Adding, subtracting, multiplying and dividing whole numbers, decimals, fractions, percents and algebraic expressions; 2. Solving equations; 3. Measuring lines, areas, volumes and angles; 4. Plotting or reading graphs; and 5. Applying formulas to solve real-life problems. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 92- A TIMSS video study of teaching practices and student achievement  Appendix Description Definition Non-problem* Mathematical work outside the context of a problem. Without presenting a problem statement, teachers (or students) sometimes engaged in: 1. Presenting mathematical definitions or concepts and describing their mathematical origins; 2. Giving an historical account of a mathematical idea or object; 3. Relating mathematics to situations in the real world; 4. Pointing out relationships among ideas in this lesson and previous lessons; 5. Providing an overview or a summary of the major points of the lesson; and 6. Playing mathematical games that did not involve solving mathematica l problems (e.g., a word search for mathematical terms). IMethod of Problem-Solving Using Procedures* Problem statements that suggested the problem was typically solved by applying a procedu re or set of procedures. These included arithmetic with whole numbers, fractions and decimals; man ipulating algebra ic symbols to simpl ify expressions and solve equations; finding areas and perimeters of simple plane figures; and so on. Example: "Solve for in the equation 2x + 5 = 6- x"was classified as using procedures. Stating Concepts* Problem statements that ca lled for a mathematical convention or an example of a mathematical concept. Examples: "Plot the point (3, 2) on a coordinate plane" or "Draw an isosceles right triangle" was classified as stating concepts. Make a connection* Problem statements that implied the problem would focus on constructing relationships among mathematical ideas, facts or procedures. Often, the problem statement suggested that students would engage in special forms of mathematical reasoning such as conjecturing, generalizing and verifying. Example: "Graph the equations y = 2x + 3, 2y = x- 2, andy= -4x, and examine the role played by the numbers in determin ing the position and slope of the associated lines" was classified as making connections. IProblem Context Real world* Mathematics problems presented w ithin a real-life context. Examples: "Estimate the su rface area of the frame in the picture below,"and"Samantha is collecting data on the time it takes her to walk to school. A table shows her travel times over a two-week period; find the mean:' Math language* Problems presented only with mathematical language, Examples:"Graph the equation: y = 3x + 7" and "Find the volu me of a cube whose side measures 3.5 em:' IProblem Type Closed A form of question which can normally be answered using a simple"yes"or"no'; or with a specific simple piece of information. Open Questions that solicit additional information from the students. They are broad and require more than one- or two-word responses. IProblem Solution Method Routine Problem that could be solved directly using a formula, definition or proposition. Non-routine Problem that could not be solved with a routine procedure (see above), but instead had to be solved using a non-routine strategy. In a non-routine problem, the student did not initially have a specific method for solving the problem. 93  Description Definition Problem Complexity Low complexity sions Solving the problem, using conventional procedures, required four or fewer dec1 by the students (decisions could be considered small steps). The problem contained no sub-problems, or tasks embedded in larger problems that could themselves be coded as problems. Example: Solve the equation: 2x + 7 = 2. Moderate complexity Solving the problem, using conventional procedures, required more than four decisions by the students and could contain one sub-problem. Example: Solve the set of equations for x and y: 2y = 3x- 4; 2x + y = 5. High complexity Solving the problem, using conventional procedures, required more than four decisions by the students and contained two or more sub-problems. Example: Graph the following linear inequalities and find the area of intersection: y ~ X+ 4; X~ 2; y 2 -1. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 94 - A TIMSS video stud y of teaching practices and student achievement  Appendix Appendix 4: Comparison of Full Sample Results with Subset As specified in section 3.6 Achieved Sample and Problems Encountered, 28 schools were found to have different students and/or teachers from those that eventually participated in the TIMSS exam. These schools were automatically excluded from the analysis linking the results of activities in the classroom to the TIMSS results since it wou ld have invalidated the analysis. For the task of identifying w hat happens in Indonesia's classrooms and comparing it to other countries, though, the 28 schools still have value in providing insights into w hat happens in Indonesia's classrooms. The analysis was done both wi th the subsample of 72 classrooms as well as with the full samp le of 100 22 Below is a table summarizing the results using both the full sample and the subset of 72 schools. As can be seen, most results are simi lar, but some differences emerge. In particular for the 28 classes that were removed: • Activity Purpose: Much more t ime was dedicated to review and less on new content for the removed sample. • Public vs. Private Interaction: More public time and less private time were used o Public Interaction: More teacher-on ly public time (lecture) was used o Private Interaction: Much more individual only and group-only time was used (ind icating less teacher involvement) • Teaching strategy: much more exposition and problem-solving work were used • Problem vs. non-problem time: more non-problem time • Problem Set-up Type: much more use of procedure and less use of a concept • Mathematics language vs. real world: More real world language and less mathematics language • Routine vs. non-routine: more routine and less non-routine Interestingly, the regression results indicate that the 28 dropped classes tend to use the techniques negatively related with student mathematics scores more frequently than the 72 kept classes. In the case where teachers were substituted for the video, it wou ld have been expected that higher quality teachers were replacing lower qual ity teachers, but the regression results tell the opposite story. These results below do not control for student, home, school, or classroom characteristics, so it is possible that these substitutions took place in, say, lower performing schools. Sti ll, the general results are surprising, and it may be of interest to do further analysis on the types of schools/classrooms that were el iminated from the sample and the background of the teachers themselves. 22 One additional classroom was videotaped, but the students took an examination for the full classroom period. It was decided by the team to not include this class in the analysis. 95 \0 0\ >:z AVERAGES -'lw §:;6 28 wM w _ < z 0.:0 "' 0 0z ~ M " (f) a.- '< ~ 0 (f) STRUCTURED TIME ::':s; ~ ~ TOTAL TIME ::::c: " M Mathematics time "' :s: ] ~ Non-mathematics time i 2.6% 2.3% 11 0:01 :29 l 2.5% 0:01:45 " (") =· ("> (f) - ~ (") Mathematics organization "' r time L 8.3% - l 7.9% 1~ 0:05:07] 8.2% 0:05:42 " > "- w "' (f) [§5 100.0% 1:11:18 100.0% 1:04:53 100.0% 1:09:30 100% 100% 100% 99% 100% 100% "' 0 ;:?. :s: Review I 11.3% '- 1 17.4% ILQj:Q} !iJ 12.9% 0:08:02 16% 1 12.9% 12% "' ("> (f) .. 17.7% 11 :r- o;· New content - 48.5% 1 1 42.0% 0:24:501 46.8% 0:29:09 - -[ 41 .8% [ 38:2il 46.3% 41 o/o ~ ;:?. " Practice l - 38.4% 1 1 39.8% 11 r 0:23:3 11 38.8% 0:24:08 --L - -1 40.0% 11 36 %1 39.4% 35% Assesment r 1.8% ,- 1 0.7% 11 0:00:24 1 1.5% 0:00:57 0.5% 11 o% 1 1.4% 1% 100.0% 0:59:03 100.0% 1:02:16 100% --I 89% 100% 90% 100% 89% Individual/private 37.0% !1 0:21:37 41 .4% 0:25:43 35.8% !1 32% 1 40.5% 36% Class/public 63.0% 11 0:36:47 ,: 58.6% 0:36:20 __ --[ , 64.2%] C 57% 1 59.5% 53% 100.0% 0:58:24 100.0% 1:02:03 100% 89% 100% 89% 100% 89% Public: teacher 64.4% I 0:24:02 ~1 60.6% 0:22:12 62.4% 11 36% 1 60.3% 32% Public: teacher/student 21.3% ICo_:g7:iiJ 21.1 o/o 0:07:44 22.7% ll 13% 1 21.9% 12% - -1 Public: student L 14.3% ICo"Ds:ill: 18.3% 0:06:41 --[ 14.9%]1 9%] 17.7% 9% 100.0% 0:37:20 100.0% 0:36:37 100% 52% 100% 58% 100% 53% Private: teacher-ind ividua l f 22.4% 11 0:04:51] 19.7% 0:05:03 --[ 22.4% 11 7% 1 19.3% 7% Private: teacher-g roup 16.8% I 0:03:39 1 31.8% 0:08:10 17.5%] 6%1 30.8% 11 % Private: individual only 39.1 o/o II 0:08:311 29.2% 0:07:30 37.8%J~ 12!) 29.6% 11% Private: group only 21.7% ;; 0:04:44 r 19.3% 0:04:58 22.3%]1 7% I 20.2% 7% 100.0% 0:27:12 100.0% 0:21:45 100.0% 0:25:40 100% 32% 100% 36% Discussion 14.4% \~~ • • lr~n 10.0% 0:05:42 " 13.3% 0:08:08 9.8% II 9o/o l 13.2% 12%  PERCENTAGES (Avg.% of Each Class) Exposition o:32:5o r 52.5% 0:32:1 1 60.7% ~l 54.5% 48% Investigati on 0:00:00 I 2.8% 0:01:42 o.oo/o tl Oo/o t 2.3% 2% Practi cal work 0:05:09 ! 10.1 % 0:06:1 3 8.4%l J 7% ]: 9.3% 8% Problem-solving 0:13:21 1 21.3% 0:1 3:04 21.1% jj 18%] 20.6% 18% 100.0% 1:02:57 100.0% 0:57:03 100.0% 1:01:18 100% 88% 100% 87% 100% 88% Problem nA%~1111111 n ;J[ 0:42:04 1 76.0% 0:47:03 ~~,~-f:c'~;~~~~[ 71.1% 11 63% 1 75.3% 67% Non-problem 22.6% l;~l~BJI 28.1% 1~ 0:16:25 1 24.0% 0:14:53 I· *1111~1tt'K~~20iU 28.9% 11 26% 1 24.7% 22% 100.0% 1:03:17 100.0% 0:58:29 100.0% 1:01:56 100% 89% 100% 89% 100% 89% Make connec tion L 4.3% l./- LjJ% J ]1 0:02:2~: 4.7% 0:02:1 3 4.4% 1[ 3% 1 4.9% 3% Concept [_59.5% 1 lf~l 39.4%]i 0:16:23 54.5% 0:25:29 38.7% 11 24% 11 53.2% 35% Procedure L 35.1 % iP- I 54.0% 11 0:22:28 1: 39.8% 0:1 8:37 56.2% 1l 35% 11 40.9% 27% Showing [ 1.1% 1~[ 0.7% 1[ O:OO: W t 1.0% 0:00:28 0.7% 11 0% 11 0.9% 1% 100% 0:48:48 100% 0:41:35 100% 0:46:47 36% 68% 57% 63% 42% 66% Mathematics language [ 85.6% 11 0:34:34 11 89.0% 0:39:59 3.9% 11 0:34:34 11 88.5% 13:42:40 1 Real world r 14.4% il 0:05:5o ll 11.0% 0:04:57 0.7% 11 0:05:50 1 10.2% 1:34:44 100.0% 0:46:42 100.0% 0:40:24 100.0% 0:44:56 5% 61% 99% 64% 0 Routin e [ j6.1% - l 94.0% 11 o:38:o8 1i 88.0% 0:40:03 4 .3% 11 o:38:o8 1 89.0% 13:47:09 Non-routine ~' Ll3 .9% lf:: 6.0% 11 0:02:25 1 12.0% 0:05:26 o.3o/o II 0:02:251 11.0% 1:42:04 > "0 "0 <> 0.. " ;:;· \0 ---1  Appendix 5: Regression Results Note: the regression results were determ ined to be too large to include within this report and are instead available in a separate file. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 98 - A TIMSS video study of leaching practices and student ach ievement  Appendix Appendix 6: Study Costs As mentioned in section 3.7 Justification of a Video Study, the use of video provides many unique advantages for understanding what takes place in the classroom, but it also tends to cost significantly more than other methods of gathering data such as interviews, questionnaires or classroom observation. This appendix is intended to be of use for those who may be considering whether to undertake a video study. Indones ia's expenditures provide insights into w hat design decisions and country context factors may play an influential role in determining the overall costs in a video study. Table A6.1 below categorizes the expenses in curred for the Indonesia video study by phase. The overall cost amounted to just over $400,000. Fifty seven percent of the costs came in the data collection phase, which involved visits to 101 schools, typically lasted three (?) days, and involved teams made up of three individuals (two technical people and one mathematics education expert). The coding, data analysis and reporting phase made up approximately 13% of the total cost, w ith most of the expenditures resulting from the lengthy process of coding the multiple layers of video. Consultant fees over the course of the study also made up a significant proportion of the cost, mainly through international consultants. • Table A6.1 Cost Categories (by Phase) Expenditure (USD) A. Study Preparation (design; instrument development, 31,617 8% training for coding of videos; sampling; piloting) c. Data Collection (1 01 classrooms in 30 provinces; filming of 232,216 57% two classroom sessions per teacher; administering student, teacher and schoo questionnaires) D. Coding, Data Analysis and Reporting (coding of videos for 51,979 13% multiple layers; data analysis; report writing; peer review) E. Dissemination 21,739 5% F. Consultant Fees (mainly involved in design, training, analysis) International 50,852 13% Domestic 16,729 4% TOTAL 405,134 Each video study has its own unique characteristics wh ich shape the overall study cost. Key design and country- specific factors for the Indonesia video study included : • Sample size - with 101 classrooms, the Indonesia TIMSS study involved a relatively large sample. This design approach was chosen in order to get a representative sample of Indonesia's 8'h grade classrooms. Indonesia is large and highly diverse, requiring a larger sample size than in smaller, more homogeneous co untries. Most video studies have smaller sample sizes and many use a case study approach without the goal of constructing generalities. • Number of classroom sessions videotaped - in the case of the Indonesia study, two classroom sessions were taped, both of which were done in a single visit. An alternative approach used in many studies is to film multiple (e.g. 10+) sessions with the goal of capturing full patterns of individual teachers over t ime. The increased number of sessions could increase the costs significantly, particularly if multiple visits are required. 99  • Labor costs - the Indonesia video study team included many civil servant employees from the Ministry of National Education who did not receive a salary for their participation. Some of the actual labor costs are therefore not captured in the overall expenditures. • Travel costs - Indonesia is a large country and visiting the schools for the data collection typica lly required f1ights. Airfare to get the video teams out to the schools made up approximately 9% of the overall costs, ground transportation 4% and accommodation made up another 9%. Trave l costs made up nearly% of the overall expenditure. • Equipment rental costs - video equipment made up approximately 1Oo/o of the overall costs. When computer editing and raw materials such as DVDs are included, the costs are 14%. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: lOG- A TIMSS video study of leach ing prac ti ces and student ach ievement References Asrijanty eta/, Presentation of Pi lot for the Video Study given to the Ministry of National Education, (August 2006) Barber, M. and Mona, M. (2007). How The World's Best-Performing School Systems Come Out On Top. London: McKinsey & Company. Clarke, D., Keitel, C. and Shimizu, Y (Eds.) (2006). Mathematics Classrooms in Twelve Countries: the insider's perspective. Rotterdam: Sense Publishers. Darling-Hammond, L. (1999). Teacher Quality and Student Achievement: A Review of State Policy Evidence. Center for the Study ofTeaching and Policy, Un iversity of Washington. Dela mont,S. and Galton,M. (1986)./nside the Secondary Classroom. London: Routledge & Keg an Paul. Doerr, H.M. and Zangor, R. (2000). Creating Meaning for and with the Graphing Calculator. Educational Studies in Mathematics, 41, 143-163. Embse, C.V. (1992). Concept Development and Problem-solving Using Graph in g Calcu lators in the Mid dle School. In J.T. Fey and C.R. Hirsch (Eds.), Calculators in Mathematics Education: 1992 Yearbook. Reston, VA: National Council ofTeachers of Mathematics, 65-78. Fey, J.T. and C.R. Hirsch, C.R. (Eds.) (1992), Calculators in Mathematics Education: 1992 Yearbook, 65-78. Reston, VA: National Council ofTeachers of Mathematics. Filmer, D. (2006). A Mil/enium Learning Goal. Measuring Real Progress in Education. CGD Working Paper 97. Flanders,N.A. (1970). Analyzing Teacher Behavior Reading, Mass.: Addison-Wesley. Hattie, J. (2003). Teachers Make a Difference: What is the research evidence? Talk given at the Austral ian Counc il of Educational Research, October. Hiebert, J., Gallimore, R., Garnier, H., Givvin, K.B., Hollingsworth, H., Jacobs, J., Chui, A.M.Y., Wearne, D., Smith, M., Kersting, N., Manaster, A., Tseng, E., Etterbeek, W., Manaster, C., Gonzales, P and Stigler, J. (2003). Teaching Mathematics in Seven Countries. Results From the TIMSS 1999 Video Study, Nationa l Center for Education Statistics, Wash ington DC. Hiebert, J.S. and Grouws, D.A. (2007). The Effects of Classroom Mathematics Teaching on Students' Learning, in Lester, F.K. (ed.) Second Handbook of Research on Mathematics Teaching and Learning. Charlotte, NC: Information Age Pub Inc. Hill H.C., Rowan, B amd Ball, D.L. (2005). Effects ofTeachers' Mathematical Knowledge for Teaching on Student Ach ievement. American Educational research Journal, 42, 2, 371 -406. 101  Ma, L. (1999). Knowing and Teaching Elementary Mathematics. Mahwah, N.J.: Lawrence Erlbaum Associates, Publishers. Martin M.O. and Mullis, I.V.S. (2006). Indonesia in TIMSS 2003: A comparison with Malaysia and Singapore of Achievement in B'h grade mathematics. A presentation at the World Bank, May 12. Martin, M.O., Mu lli s, I.V.S. and Foy, P. (2008). TIMSS 20071nternational Science Report, Boston: International Study Center, Lynch School of Education, Boston Co llege. Mullis, I.V.S., Martin, M.O., Kennedy, A.M. and Foy, P. (2007). lEA's Progress in International Reading Literacy Study in Primary Schools in 40 Countries, Boston: International Study Center, Lynch School of Education, Boston College. Mullis, I.V.S., Martin, M.O. and Foy, P. (2008). TIMSS 20071nternational Mathematics Report, Boston: International Study Center, Lynch School of Education, Boston College. National Council ofTeachers of Mathematics (NCTM) Commission on Standards for School Mathematics. (1989). Curriculum and eva lu ation standards for school mathematics. Reston, VA: NCTM. National Council ofTeachers of Mathematics. (2000) Principles and Standards for School Mathematics. Reston, VA: National Council ofTeachers of Mathematics. National Research Council. (2001 ). Adding it up: Helping Children Learn Mathematics. J. Kilpatrick, J. Swafford, and B. Findell (Eds.). Mathematics Learning Study Committee, Center for Education, Division of Behavioral and Socia l Sciences and Education.Washington, DC: National Academy Press. Organisation for Economic Co-operation and Development (2005). Attracting, Developing and Retaining Effective Teachers- Final Report: Teachers Matter. Paris: OECD Publications. Organisation for Economic Co-operation and Development (2007). Science Competencies for Tomorrow's World. Paris: OECD Publications. Penglase, M. and Arnold, S. (1996). The Graphics Calculator in Mathematics Education: A critical review of recent research. Mathematics Education Research Journal, 8(1 ), 58-90. Rogers, C. R. (1983). As a teacher, can I be myself? In Freedom to Learn for the 80's. Ohio: Charles E. Merrill Publish ing Company. Ruthven, K. (1990). The Influence of Graphing Calculator Use on Translation from Graphic to Symbolic Forms. Educational Studies in Mathematics, 21 (5), 431-450. Shoaf-Grubbs, M.M. (1995). Research Results on the Effect of the Graphic Calculator on Female Students'Cognitive Levels and Visual Thinking. In L. Burton and B. Jaworski (Eds.), Technology in Mathematics Teaching, 213-227. Chartweii-Bratt. Wright, S.P., Horn, S.P. and Sanders W.L. (1997). Teacher and Classroom Context Effects on Student Ach ievement Impl ications for Teacher Evaluation, Journal of Personnel Evaluation in Education 11: 57-67. INSIDE INDONESIA'S MATHEMATICS CLASSROOMS: 102- A TIMSS video study of L eaching practices and student achievement