Philippines - Educational Radio Technical Assistance Project

Document of The World Bank FOR OFFICIAL USE ONLY FIEPOY Report No. 5004 PROJECT COMPLETION REPORT PHILIPPINES MASS MEDIA PILOT PROJECT (LOAN S8-PH) March 23, 1984 Projects Department East Asia and Pacific Regional Office This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.   PC99373/ED-91/D2111/61 CURRENCY EQUIVALENT Currency Unit = Peso (P) US$1.00 = e 11.0 (1983) ACRONYMS CET - In-Service Teacher Training (commonly referred to as Continuing Education of Teachers) DEC - Department of Education and Culture ECO - Education Communications Office EDPITAF - Education Development Projects Implementing Task Force MECS - Ministry of Education, Culture and Sports (successor of DEC) PTC-RD - Philippine Training Center for Rural Development RATES - Radio-assisted Teaching in Elementary Schools SOUTELE - Survey of the Outcomes of Elementary Education UPIMC - University of the Philippines Institute of Mass Communications SCHOOL YEAR July-April   FOR OFFICIAL USE ONLY PHILIPPINES MASS MEDIA PILOT PROJECT (LOAN S8-PH) PROJECT COMPLETION REPORT Table of Contents Page No. PREFACE . . . . . . .. . . . . . . . . . . . . . . .. . . i BASIC DATA SHEET . . . . . . . . . . . . . . . . . * * * * ii HIGHLIGHTS . . . . . . . . . ... . . . . . . . . . . . . . . . . iv I. -PROJECT BACKGROUND . . . . . . . . . . . . . . . . . ... 1 Sector Setting . . . . . . . . . . . . . . . . . . . . . 1 Issues in Elementary Education. . . . . . . . . . . . . . 1 Project Generation . . . . . . . . . . . . . . . . . . . 2 Project Objectives and Strategies . . . . . . . . . . . . 2 Project Description . . . . . . . . . . . . . . . . . . 3 II. PROJECT IMPLEMENTATION . . . . . . . . . ... . . . . . 4 Overview ... 0 0...... 4 Project Management . . . . . . . . . 0 . * *.*.. 5 In-service Teacher Training (CET) . . . . . . . . . . 5 Radio-assisted Teaching in Elementary Schools (RATES) . 6 Evaluation . . . . . . . 6 Studies. ... . . . . . . 6 Engineering Designs .; ... . . . . . . . 7 Procurement . . . . . . . . . . . . . . 7 Technical Assistance .... . . . . . . . . * ... * * . *. 7 Bank Performance. . . . . . . . . .. . . . 8 Covenants . . . . . . . . . . . . . . . . .. . * 9 Implementation Period . . . *. . . . . . . * . g 9 III. PROJECT COSTS . . . . . . . . . 9 . . . . . . . . . . . . . 9 Project Costs by Type of Expenditure . . . . . . . . . . 9 Disbursements . . . . . 0.. . . . . . . . . . . . . . . . 10 Recurrent Costs per Student . . ... . . . . . . . . . . . 10 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.  Page No. IV. OPERATING OUTCOMES . . . . . . . . . . . . . . . . . . . . 10 Overview . . . . . . . ... . . . . . . . . . . . . . . .10 Attainment of Objectives . . . . . . . . . . . . . . . . 11 In-service Teacher Training (GET). . . . . . . . . ; . . 12 Radio-assisted Teaching in Elementary Schools (RATES). . 12 Evaluation of CET and RATES. . . . . a . . . . . . . . . 14 Study on Communications Technology for Rural Education . 15 Future of the Training Materials . . . . . . . . . . . . 16 ANNEXES 1. Organization of the Education Communications Office (ECO). 17 2. Local Educational Administration . . . . . . . . . . . . . . 18 ATTACHMENT I Comments from the Borrower ... . . . . . . . . . . . . . . 19  PHILIPPINES MASS MEDIA PILOT PROJECT (S8-PH) PROJECT COMPLETION REPORT Preface This is a Project Completion Report (PCR) for the Mass Media Pilot Project in the Philippines, for which Loan S8-PH was approved in March 1978. The loan amount was for US$2.0 million; of this amount US$1.15 was disbursed and US$0.85 million cancelled. The Closing Date was December 31, 1981. The PCR and the Highlights were prepared by the Bank Group's East Asia and Pacific Regional Office. The PCR is based on: (a) a draft completion report prepared by the Education Communications Office (ECO) of the Borrower; (b) information obtained by a Bank consultant who visited the Philippines in June/July 1982; (c) an evaluation report prepared by the University of the Philippines Institute of Mass Communications (UPIMC); (d) records in the Bank and ECO, including the President's Report, P-2250-PH of March 8, 1978 and the Loan Agreement dated April 21, 1978; and (e) discussions with Bank and ECO staff. The Operations Evaluation Department decided not to audit the project and sent the PCR to the Government for comment in November 1983. Comments received from the Government have been taken into account in finalizing the report and are reproduced as Attachment I to the PCR.  a  - ii - PROJECT COMPLETION REPORT BASIC DATA SHEET Philippines Fifth Education Project (Loan S8-PH) KEY PROJECT DATA Appraisal Actual or Item Estimate Current Estimate Total Project Cost (US$ million) 3.86 2.86 Cost Underrun (X) - -26% Loan/Amount (US$ million) 2.0 1.15 Disbursed ) April 14, 1982 1.15 Cancelled ) May 3, 1982 0.85 Repaid to } December 31, 1982 - Date Physical Components Completed 12/01/81 12/01/81 Proportion Completed by Above Date (X) 100X Proportion of Time Overrun (%) nil Economic Rate of Return (Z) n.a. n.a. Financial Performance n.a. n.a. Institutional Performance Good Satisfactory OTHER PROJECT DATA Original Actual or Item Plan Revisions Est. Actual First Mention in Files 03/01/76 Government's Application 12/23/77 Negotiations 02/22-23/78 Board Approval 03/21/78 Loan Agreement Date 04/21/78 Effectiveness Date 07/21/78 08/22/78 Closing Date 12/31/81 05/03/82 Borrower Government of the Philippines Executing Agency Education Communications Office of EDPITAF Projects Office Fiscal Year of Borrower January-December Follow-on Project Name Elementary Education Sector Loan Loan Number Loan 2030-PH Amount (US$ million) 100.0 Loan Agreement Date 08/25/81 MISSION DATA Sent No. of No. of Date of Item by Month, Year days Persons Staff-weeks Report Appraisal Bank 09/77 26 2 10.40 11/17/77 Total 10.40 Supervision I Bank 05/78 4 1 0.80 06/09/78 Supervision II Bank 01-02/79 10 2 4.00 03/29/79 Supervision III Bank 11/79 6 1 1.20 11/12/79 Supervision IV Bank 06/80 6 1 1.20 06/30/80 Supervision V Bank 02/81 5 1 1.00 02/20/81 Supervision VI Bank 11/81 1 3 0.60 01/07/82 Completion Bank 06-07/82 9 1 1.80 Total 10.60 COUNTRY EXCHANGE RATES Name of Currency (Abbreviation) Philippine Peso (P) Exchange Rates: Appraisal Year Average US$1 - P 7.4 Intervening Years Average US$1 - P 7.5 Completion year US$1 - P 8.2   - iii - PHILIPPINES MASS MEDIA PILOT PROJECT (S8-PH) ALLOCATION OF LOAN PROCEEDS (US$) Original Actual Category allocation disbursements Equipment and materials 1,060,000 } 954,460.64 Technical assistance 540,000 } 199,530.83 Unallocated 400,000 } Subtotal 2,000,000 } 1,153,991.47 Cancelled - } 846,008.53 Total 2,000,000 } 2,000,000.00   - iv - PHILIPPINES MASS MEDIA PILOT PROJECT (S8-PH) PROJECT COMPLETION REPORT Highlights A loan of US$2.0 million to assist in financing a Mass Media Pilot Project in the Philippines, was approved in March 1978 and signed in April 1978. The original closing date of the loan was December 31, 1981 and this was not formally changed. The loan account was closed on May 3, 1982, when US$1.15 million had been disbursed and US$0.85 million cancelled. The final project cost was US$2.86 million, 26% lower than the appraisal estimate. The lower cost was largely due to a reduction in the amount of technical assistance. The project was designed to develop and evaluate the cost effective- ness of using educational radio: (a) for in-service teacher training; and (b) as a direct medium for classroom teaching. In broad terms the project has met these objectives. Evaluation results show that in-service teacher training through radio lessons combined with provisions of textbooks to children at a 1:2 ratio (i.e. one textbook per subject and grade for every two pupils) is more cost effective than direct transmission to pupils (para. 4.10). The per pupil cost of transmitting programs directly to students was found to be more than ten times higher than that of the cost of in-service teacher training (para. 4.10). Materials developed under the project are being used on a national scale to retrain teachers to use the new curriculum. In this respect the project has had a major impact on national education (para. 4.13). Mass media, however, remains a supplemental activity and does not play a major role in education. One lesson learned from the project is that highly innovative projects need close supervision and monitoring. The evaluation design should have been reviewed at various stages of implementation to ensure the necess- ary adjustments were made. The pilot project was a success in the sense that it tested the feasibility and effectiveness of using mass media as a means of transmitting instruction. Evaluation results show that the instructional medium and materials were effective but problems were encountered with radio transmission, which made wide-scale implementation difficult.  Another lesson learned concerns procurement, where low priced tape-recorders were chosen rather than higher priced machines which used less amounts of consumable materials (batteries). Life cost evaluation methodology might have been more appropriate in this instance (para. 2.11). Experience gained under this project suggests that the Bank is not the ideal agency to carry out small-scale, innovative projects. Other insti- tutions which have greater technical assistance resources probably have an advantage in this kind of activity.  I. PROJECT BACKGROUND Sector Setting 1.01 At the time of project generation (1976/77) the Philippine education system was characterized by high enrollment ratios at all levels and the formal school system had accomplished universal primary enrollments more than a decade earlier. Secondary enrollment ratios were above 60% and tertiary ratios around 20% of the relevant age group. More than half of the secondary students and 90% of tertiary students were in private institutions. Enroll- ments of females equalled or surpassed those of males at all levels. Student:teacher ratios were generally good. Nonformal education and training had also reached impressive proportions, with around two million persons receiving instruction annually. Issues in Elementary Education 1.02 Despite the above-mentioned high participation rate, much needed to be done in the elementary sector to improve the quality and coverage of education. A basic problem in primary education was the low quality of instruction, particularly in rural areas. Although the majority of teachers were college graduates, the quality of preservice training had been weak. Some in-service teacher training programs existed, but they were costly, limited to the summer term and the program content tended to become diffused as it passed from the national to local level. Only 40% of primary teachers surveyed in 1976 had received any in-service training. Student achievement averaged only 50% of the expected standard items; the poorest overall performances were in reading, languages and mathematics. Substantial variations in achievement were found between urban and rural areas, with test scores ranging from an average of 30% in rural areas to 70% in Manila. Ways needed to be found to improve educational quality and remove the disparities between urban and rural areas. 1.03 The Role of Mass Media in Education. Media coverage was more limited than might be expected. Despite high literacy levels, the 60 Philip- pine newspapers and periodicals reached only about a quarter of the adult population while circulation of printed educational material was but a fraction of this. Formal educational broadcasting was undertaken by only one television station and 23 radio stations in contrast to nonformal education broadcasts that were made by more than 200 stations. Geographic coverage was limited and many rural areas had no radio coverage at all, much less television. Evaluation of direct primary school instruction via radio in the early 1970s showed that the programs were educationally successful. However, due to limited coverage, unsystematic use, and inadequate programming capacity, the Department of Education and Culture discontinued formal educational radio broadcasts in 1975, pending comprehensive reassessment of the role of educational radio. The limited coverage of educational broad-  casting meant that five million out-of-school youths were out of reach of the Government and the quality of both formal and nonformal education in rural areas was low. Project Generation 1.04 The Bank-financed Textbook Project (Loan 1224T-PH) which became effective early in 1976, included resources for a pre-investment study on communication technology for education. During the first year of implemen- tation of this loan, EDPITAF organized a study team to do the preinvestment study. This team studied areas in which communication technology could provide solutions to educational problems. In November 1976 the team presented the range of possibilities to the Office of the President of the Philippines in order to obtain directives for focussing the study towards specific project proposals. The Government instructed the team to focus on the use of communication technology for two purposes: (a) quality improve- ments in basic education, particularly through in-service teacher training; and (b) educational support for rural development. The study team's proposal for a National Educational Communication System reflected these emphases, including provision for development of an educational communications infrastructure and for phased implementation of its recommendations. 1.05 The study team's conclusions led to an understanding in June 1977 between EDPITAF and the World Bank that the Bank would support a mass media pilot project and by August 1977, the EDPITAF study team had prepared a pro- posal for such a project. A Bank mission appraised the proposal in September 1977 and reached a general understanding with EDPITAF concerning the magni- tude, composition, and timing of the pilot project. The appraisal mission recommended that the project be included as an educational radio component of a proposed Rural Infrastructure Development Project. However, in December 1977, Bank management decided to process the mass media pilot project as a technical assistance loan of US$2.0 million, rather than as a component of the proposed Rural Infrastructure Project because it was considered to be more difficult to manage if it were integrated into this project. The Philippine Government endorsed this decision on December 23, 1977. Project Objectives and Strategies 1.06 The main objective of the pilot project was to establish a basis for possible nationwide implementation of educational radio as a means to improve educational quality in the Philippines. Two different approaches to improving primary eduation were to be developed and their cost effectiveness evaluated. The first approach was indirect use of radio for in-service teacher training, with the expectation that additional teacher training would improve the quality of instruction received by primary school students. The second approach was to broadcast radio lessons directly to students as a component of their classroom activities. Improvement in language training was one of the central objectives of the mass media program. The proposed project therefore focussed on two predominantly rural provinces where Pilipino was not the major  - 3 - language. These provinces were Pangasinan on the island of Luzon, and Leyte in the Visayas (only 55% of the population in Pangasinan and about 28% in Leyte speak Pilipino). A third province, Maguindanao, in the third major region of the Philippines (Mindanao) was later added to the project at the Government's request. 1.07 The pilot project had two other objectives: (a) exploring the use of radio for rural education as a complement to other activities - the rural and farmer training centers financed under Loan 1374-PH; and (b) evaluation of communications technologies appropriate to the Philippines. Project Description 1.08 The project consisted of three parts: (a) In-service Teacher Training [also referred to as Continued Education of Teachers (CET)]; (b) Radio assisted Teaching in Elementary Schools (RATES); and (c) Evaluation, Studies and Future Project Preparation. 1.09 In-service Training (CET). This component of the project stemmed from a need to provide in-service training to teachers within a short period of time. Instruction was expected to be given via radio broadcasts supple- mented by printed materials and periodic meetings. The effect of this training on student performance would then be assessed. The proposed training program was expected to provide semester-length courses for primary school teachers in five subjects: Pilipino, English, social studies, mathematics and science. The courses were designed to upgrade both teacher mastery of subject matter and teaching skills, and to create a favorable attitude towards proposed educational changes. These distance study courses were to include daily radio lessons (80 programs, 30 minutes each). The first course (Pilipino) was to be broadcast in the second semester of the 1978/79 school year (July-March). Evaluated and revised broadcasts were then expected to reach a larger audience over expanded geographical areas; concomitantly, courses were to be developed in the other subject areas. The target was to reach 12,600 primary school teachers during implementation, about two-thirds in Pangasinan province and the remainder in Leyte province. Teachers would receive either in-service or academic credit for the courses. 1.10 Radio-assisted Teaching in Elementary Schools (RATES). This compo- nent of the project was to employ radio for direct classroom instruction. It was expected to minimize urban/rural differences in the quality of instruc- tion. Primary school pupils were to receive daily 30-minute radio lessons in Pilipino. Pilipino was chosen as the subject area because of its importance to achievement in over half the school subjects and a pervasive low level of achievement in communication skills in Pilipino. During the first school year of the project (1978/79) the radio-assisted teaching series was to be devel- oped for Grade 4 students and field-tested in a small number of classrooms. In the second year (1979/80), Grade 4 Pilipino was to be revised, broadcast more widely and evaluated; simultaneously, Grade 5 Pilipino was to be developed. A similar pattern was expected to continue into the third year  - 4 - (1980/81). During the pilot period, this component was expected to reach about 2,500 classes with about 75,000 primary pupils. 1.11 Evaluation, Studies and Project Preparation. The principal purpose of the project was to provide information on the cost-effectiveness of in-service teacher training and radio-assisted teaching in primary schools so that informed decisions could be made on the use of media (para. 1.09). The project, therefore, included a strong evaluation component. The basic evaluation was designed to determine how in-service training through radio and textbooks, separately or in a variety of combinations, affected student performance. Tests were to be administered to students before and after the project to measure its impact on student achievement. 1.12 A separate activity, included under the "studies" category, was designed to explore and evaluate the use of media-based programs to complement the activities of the Philippine Training Center for Rural Development (PTC-RD), a component of the Fourth Education Project (Loan 1374-PH). The study was intended to explore the potential of radio to reinforce, complement and supplement effort of the PTC-RD. While categorized as a study during appraisal, this component was action research on the use of radio in a distance learning system for small farmers. It is referred to below as the "Communication Technology for Rural Education Study (CTRE)". 1.13 Several smaller studies and experiments were planned on the techni- cal aspects of the communication support system. These can be categorized under four headings: methods of production of mass media material, distribu- tion/transmission options, arrangements for reception of mass media communi- cations, and two-way administrative communication for implementation of nationwide education programs. The purpose of these studies was to provide educational and technical planners with an up-to-date assessment of the cost and performance characteristics of available technological options (including satellite communications). 1.14 An educational materials project (including mass media and text- books) was expected to be prepared on the basis of the findings of the above- mentioned studies. The loan was signed on April 21, 1978 and became effective on August 22, 1978. II. PROJECT IMPLEMENTATION Overview 2.01 With a couple of exceptions, the project was generally implemented as envisaged at appraisal. The proposed management team was set up on time; one province was added to the project; the various studies and evaluations were carried out; and the project was completed on schedule. It was a pilot project based on radio transmission and printed material, and in some areas outside the regions where evaluation was conducted, tape recorders were used. Cost parameters of the project also underwent a major change. There was a  - 5 - large under-utilization of technical assistance, which led to a major cost saving but also negatively affected the quality of the summative evaluation. Project Management 2.02 A separate Education Communications Office (ECO) was established and financed within EDPITAF to implement this project. At appraisal ECO was designed with a small staff of ten, but with access to staff resources in EDPITAF and the Textbook Board Secretariat (which is responsible for primary and secondary textbook development and production). A sizeable amount of specialist services (9 man-years) was also envisaged to further strengthen ECO's staff (para. 2.13). During implementation ECO grew temporarily into a strong unit of education specialists, media specialists, research and develop- ment staff planners, service staff and field staff. At its peak ECO had 65 staff and was much less dependent on outside staff resources than origin- ally envisaged. The lifespan of this organization was, however, limited (only three staff remained in July 1982). Most of ECO's staff have been absorbed into other parts of the Ministry of Education, Culture and Sports (MECS) which, together with EDPITAF, carry out tasks initiated under the aegis of ECO. A large number of the project staff were absorbed by the Development Academy of the Philippines which was contracted by the MECS to implement the Teacher Reorientation program under the Elementary Education Sector Loan (Loan 2030-PH). Thus the organization was temporary but its expertise is in use in other parts of the education system. In-service Teacher Training (CET) 2.03 The CET program was implemented over a three-year period: 1978/79- 1980/81. The first cycle, broadcast in 1978/79 in Pangasinan, was formative. During this cycle, information gathered about the lessons was used to make program improvements. The second cycle transmitted the tested and modified material in 1979/80 to a larger audience of about 1,000 teachers. This was originally planned for Pangasinan and Leyte only, but teachers in a province in the third major region of the Philippines (Maguindanao) learned about the test phase of the project through the Regional Education Officer, asked to participate and were included. The third cycle was disseminative and aimed at about 2,000 teachers. Initially, all lessons were broadcast, but during implementation ECO encountered numerous transmission quality and scheduling problems at the contracted radio stations. To overcome these problems in areas which were not part of the experimental comparison, ECO resorted to the use of tape recordings of the lessons that had been prepared for broadcasting. 2.04 The CET program consisted of a series of 45 radio lessons, transmitted to teachers in each of five subject areas, namely Pilipino, Educational Trends, Social Studies, English and Math/Science. Supplementing the transmissions were a series of three teacher's manuals in each subject covering the 45 lessons. The scripts for the broadcasts were written by ECO staff assisted oy foreign experts and local teachers and supervisors. The  - 6 - lessons were broadcast so that in the three selected geographical areas, teachers in schools that were issued with radio receivers could participate in the CET program as well as use the receivers for RATES. To encourage them to do so teachers were awarded certificates after each completed semester and credit was given towards promotion. With some additional work, the teachers could also be given three credit units towards an M.A. degree. Teachers participating in the MA program had to attend monthly sessions of three hours duration throughout each semester of study and to pass an examination at the accrediting institution. Radio-assisted Teaching in Elementary Schools (RATES) 2.05 This component explored the effective use of radio in school as a primary support medium to assist teachers in classroom instruction. It covered the same three geographical areas as those for the CET programs (para. 2.03). During the first year of implementation (1978/79), formative evaluation was conducted. The subsequent two years saw a general dissemina- tion to all classes that could be supplied with receivers. The instructional program covered grades 4, 5, and 6. For each grade, 140 lessons were scripted and broadcast, giving a total of 420 lessons. For each lesson a pupil's worksheet was prepared and distributed. Achievement tests were designed to evaluate the impact of the broadcast lesson and worksheets. The total number of classes covered during the project period rose from the initial 30 to 2,075, consistent with the number of receivers distributed in the three areas. Evaluation 2.06 Summative evaluation of both CET and RATES was conducted as a single study carried out by the Institute of Mass Communications of the University of the Philippines, Diliman, under contract to EDPITAF. Formative evaluation activities, for both CET and RATES, were conducted by ECO, as part of the development of the instructional programs. Systematic plans for formative evaluation were developed with the assistance of local and foreign consultants and covered both materials development and implementation activities. Similarly, formative evaluation of CTRE, the distance learning system for small farmers, was conducted in-house. In addition, CTRE undertook special process evaluation studies to assess the progress being made in implementing the teaching programs and to obtain preliminary assessments of effectiveness. These studies were conducted with the help of consultants. Studies 2.07 Communication Technology for Rural Education was a two-year action research component designed to explore the use of radio as a support medium: (a) to the training given to personnel being trained at the Philippines Training Center for Rural Development (PTC-RD); and (b) in person-to-person contacts in expanding the knowledge and improving the attitudes of agricul- tural extension workers and local leaders. This study was implemented by the University of the Philippines at Los Banos. The programs were broadcast to  - 7 - two centers (Bulacan and Cavite) each having both local leader and farmer family groups. The activities covered the writing of scripts, their broad- casting, and university staff contact with the groups in weekly dialogues. The segment aimed at extension workers was never executed because the construction of the PTC-RD (aided by the Fourth Education Project) was not completed by the time the study was to have occurred. 2.08 The study on options in communication technology evaluated the quality and reliability of existing and proposed communications services for the MECS. Engineering Designs 2.09 There were only two minor items of construction in the project, namely that of an earth satellite receiving station at the MECS Regional Office in Cotabato and a recording studio at headquarters. The Cotabato station receives and transmits signals from and to the Indonesian earth satellite Palapa II and links it with an existing installation near Manila and thence by radio to a terminal in MECS. This station became operational in December 1980. The recording studio was intended for recording teaching programs on tape for the duration of the project and to be of use to the MECS thereafter. A portion of the leased office space of EDPITAF/ECO headquarters was converted for this purpose at low cost in December 1978 but was neither large enough to hold all the recording personnel and equipment, nor suffi- ciently soundproof. It could therefore be satisfactorily used only for reproduction and tape dubbing. Procurement 2.10 Procurement for the project was carried out in accordance with the procedures specified in the Loan Agreement. In addition to the earth satel- lite receiving station, expenditures covered about 2,400 radio-cassette recorders, 171,000 batteries, 26,000 taped lessons, 16.7 million worksheets, 21,000 manuals and one set of studio recording and tape duplicating machinery. 2.11 Procurement problems included lack of local funds which caused some procurement delays. End-users also complained about the rationale for select- ing a low-priced radio cassette receiver model, which consumes more batteries than the second lowest-priced model. A life-span costing might have been more appropriate in this instance to determine the most advantageous bid. Technical Assistance 2.12 The project envisaged technical assistance and fellowship training and made provision for both. The purpose of each was to help in: (a) preparation of teaching materials; and (b) evaluation of the various project elements. Recruitment of the technical assistance took two forms. One was the direct employment of experts by ECO. The other was the hiring by contract of the University of the Philippines Institute of Mass Communications (UPIMC),  - 8 - and the University of the Philippines at Los Banos, each of which provided the staff who had the required technical knowledge. 2.13 About nine man-years of foreign technical assistance was envisaged and two man-years for fellowships. Only 16 man-months of short-term foreign technical assistance was financed under this project because project manage- ment considered that most of the required expertise could be hired locally./1 This was reflected in the staffing pattern of ECO, which had a much larger staff than that envisaged at appraisal. Of the two man-years planned for fellowships, 18 man-months was used by 21 participants. The planned and actual utilization was as below: Table 2.1: TECHNICAL ASSISTANCE - PLANNED & ACTUAL Man-months Participants (Number) Planned Actual Actual Foreign technical 108 16 14 assistance Fellowships 24 18 21 Of the 14 foreign experts for technical assistance, five were concerned with assisting the production and use of the teaching media and the other nine were specialists in management or evaluation. Five were selected directly by ECO and the rest via UNESCO or other agencies. The two institutions of the University of the Philippines provided the services of their own specialist staff. Bank Performance 2.14 The project originated as a study in the Bank-financed Textbook Project (Loan 1224T-PH) which became effective early in 1976. Appraisal was done during a 26 man-day appraisal mission in September 1977. This two-man mission reviewed the facets of Loan 1224T-PH that became the underpinnings of Loan S8-PH. As this was considered a technical assistance project a full appraisal report was not required. A President's Report and legal document were prepared. There have since been six supervision missions all undertaken by the Bank and involving 53 man-days of mission time. Because of the specialized nature of the project, practically all these missions were undertaken by mass-media and evaluation specialists. The heavy reliance on education specialists from other parts of the Bank to supervise the project /1 A similar amount of foreign specialist services for this project was financed from the proceeds of the Textbook Project (Loan 1224T-PH).  led to a less than desirable monitoring of the overall project design and objectives by the education division (AEPED). Covenants 2.15 There were two special conditions regarding implementation, both of which were complied with. These were: (a) that ECO would arrange with specified local agencies to carry out project studies; and (b) that the Government would maintain ECO during implementation with a satisfactory organization and staffing. Implementation Period 2.16 The project was implemented substantially in accordance with the originally envisioned time frame. The originally established Closing Date was December 31, 1981 and the loan account was closed on May 3, 1982. III. PROJECT COSTS Project Costs by Type of Expenditure 3.01 Estimates at appraisal were that the project would cost US$3.86 mil- lion with a foreign exchange component of US2.0 million or 52% of the total. Actual total cost of the project at loan closing amounted to only US$2.86 mil- lion. Table 3.1 presents a comparison between appraisal estimates and actual costs of the project by type of outlay: Table 3.1: ESTIMATED AND ACTUAL PROJECT COSTS BY OUTLAY (US$ million) Actual costs as % of Appraisal appraisal estimates Actual costs estimate Local Foreign Total Local Foreign Total (+ or -) Equipment 0.10 0.69 0.79 0.04 0.48 0.52 -34% Program materials 0.11 0.37 0.48 0.26 0.47 0.73 +52% Technical assistance 0.52 0.54 1.06 0.75 0.20 0.95 -10% Administration 0.76 - 0.76 0.66 - 0.66 -13% Subtotal 1.49 1.60 3.09 1.71 1.15 2.86 -11% Contingencies 0.37 0.40 0.77 - - - - Total 1.86 2.00 3.86 1.71 1.15 2.86 -26%  - 10 - 3.02 Reasons for the cost under-run included: (a) reduction in the amount of foreign technical assistance estimated at about US$500,000; (b) decision to use the radio recorders from the RATES program on a double shift basis for the CET program at a saving of 2,900 recorders (US$200,000); and (c) reduction in the procurement of vehicles at a saving of US$50,000. Disbursements 3.03 The rate of disbursements compared with that planned was: Table 3.2: ESTIMATED AND ACTUAL CUMULATIVE DISBURSEMENTS Disbursements % (actual Year Estimated Actual compared with --- US$ million --- estimate) 1979 0.6 0.1 17 1980 1.3 0.6 46 1981 1.8 1.0 56 1982 2.0 1.1 55 The undisbursed amount of the loan account - US$0.9 million - was cancelled when the loan account was closed in May 1982. Recurrent Costs Per Student 3.04 Annual unit costs were computed by the UPIMC as shown in Table 3.3. Recurrent cost per primary school pupil in the Philippines was about US$42 in 1980./l Table 3.3 shows that using any of the combinations that involved direct radio transmissions to the pupils (RATES) added about US$10-11 to this sum - i.e., a 25% increase in recurrent costs. Such an increase on a large scale would obviously not be feasible. /1 Source: Basic Data, IBRD SAR Sector Program for Elementary Education Report No. 3423a-PH, May 29, 1981.  - 11 - Table 3.3: ANNUAL COST PER PRIMARY LEVEL STUDENT BY PROGRAM AND TREATMENT Range of unit costs Treatment Peso US$ combination/area Low High Low High A. Pangasinan With RATES, without CET, 2:1 textbook ratio 76 81 10.13 10.80 With RATES, with CET, without textbooks 78 83 10.40 11.06 With RATES, without CET, without textbooks 74 79 9.86 10.53 Without RATES, with CET 2:1 textbook ratio 5 6 0.74 0.82 Without RATES, with CET 4:1 textbook ratio 5 5 0.64 0.71 Without RATES, without CET without textbooks - - - - B. Leyte With RATES, without CET without textbooks 74 79 9.86 10.53 Without RATES, without CET without textbooks - - - - Source: UPIMC Final Report on Summative Evaluation. June 1982. Table 15, page 131. IV. OPERATING OUTCOMES Overview 4.01 The project was completed on time and at a lower total investment cost than originally estimated. It reached the targetted audiences of elemen tary teachers and pupils and it introduced an earth satellite communication system which the MECS is now operating. A considerable amount of data on the CET/RATES elements has also been accumulated. Project results indicate that the use of mass media improves pupil performance in varying degrees depending on the audience and the concurrent use of other materials. There are indications that the combination of: (a) providing one textbook to a pair  - 12 - of pupils; and (b) training teachers through mass media was the most cost effective delivery method. Although the project organization was disbanded, the core of the project staff continues to work on development of education materials in other agencies. Attainment of Objectives 4.02 The four-year pilot project sought to develop the use of communica- tion technology to improve educational quality. The project succeeded in achieving this objective. However, the use of radio has not been widely adopted by the Government because of transmission problems. 4.03 The original design of the project called for using radio as the main medium of instruction. Problems with this delivery method included: (a) poor quality transmissions from the contracted broadcasting station; (b) excessive amount of power cuts; and (c) lack of advance information for teachers and group leaders on the content of the sessions. Project management decided therefore to produce tape recordings for most project items which they used outside the experimental areas rather than rely on direct transmission. Present use of the programs are through tapes which are sent to project schools. These had obvious advantages over using radio: (a) the quality of sound was superior; (b) the program was fully reliable; (c) the audio material could be used at any convenient time; (d) the tapes could be played over in advance by the leader or teachers who then knew what was in each session; and (e) tapes could be stopped at any point for repetition and/or clarification. In-service Teacher Training (CET) 4.04 This project component was designed to: (a) develop a system capable of writing, broadcasting and modifying course materials for the five subject areas; and (b) use the materials developed, together with a learning manual and face-to-face contacts in regular group meetings in an instruction program for as many teachers as possible in three selected areas. 4.05 Table 4.1 gives an overall picture of the training operation by semesters. By the end of the project, some 7,700 sessions in Pilipino had been provided but only 100 in Science/Mathematics. A full year's course was available in the other subject areas. Over 17,000 teachers participated, or 92% of the target. All the course material has been recorded on cassette tape for further use and much is in use already under Loan 2030-PH.  P - PLANNED A - ACTUAL Table 4.1: IN-SERVICE TEACHER TRAINING PLANNED AND ACTUAL TEACHER AUDIENCE REACHED PANGASINAN | LEYTE | MAGUINDANAO Subject Year 1 Year 2 Year 3 | Year 2 Year 3 | Year 2 Year 3 l Area School 1st 2nd lst 2nd x lst 2nd Ist 2nd i 1st 2nd 1st 2nd | TOTAL year 78 79 79-80 79-80 80-81 80-81 | 79-80 79-80 80-81 80-81 79-80 79-80 80-81 80-81 Educational Trends P-O P-O P-O P-O P-0 P-0 I P-O P-O P-O P-0 IP- P-0 P-0 P-0 P-O A-162 A-250 A-671 A-789 A-100 A-1032 | A-384 A-472 A-659 A-598 A-150 A-300 A-300 A-300 | A-6167 Pilipino P-500 P-500 P-500 P-1000 P-1000 P-1000 P-500 P-1000 P-1000 P-1000 P-0 P-0 P-O P-O 0 P-8000 A-222 A-504 A-749 A-846 A-979 A-1122 I A-413 A-504 A-706 A-636 A-150 A-300 A-300 A-300 | A-7731 Social Studies - - P-500 P-500 P-1000 P-1000 - P-500 P-1000 P-1000 P-c P-cl P-3500 A-0 A-100 A-500 A-834 0- A- A-273 A-495 _ - A-200 A-200 | A-2602 English - - - P-500 P-500 P-1000 I - - P-500 P-1000 I - - - P-0 I P-3500 A-0 A-100 A-490 - A-c A-307 A-200 A-1097 Science/Math - - - P-500 P-500 | - _ - P-500 | - _ _ - | P-1500 A-c A-100 I _ A-0 | | A-100 Planned 10,500 | Planned 8,000 | Planned - Nil | 18,500 Actual 9,550 | Actual 5,447 | Actual 2,700 | 17,697 Grand Total Planned - 18,500 Grand Total Actual - 17,679  - 14 - Radio-assisted Teaching in Elementary Schools (RATES) 4.06 During its first year of operation this project component developed radio-assisted teaching of Grade 4 Pilipino in Pangasinan, which combined radio lessons with the use of pupils' worksheets. After formative evaluation, the program was continued for two years in subsequent grades 5 and 6 following the same pupils. After the initial testing year, other pupils were added, as tested teaching material became available. The radio lessons were supple- mented by the use of pupils' worksheets throughout the three years. The number of pupils who participated was governed by the number of available operational radio-cassette players. The pattern is shown in Table 4.2. Table 4.2: RATES IMPLEMENTATION SCHEDULE BY CLASSES School Year Grade 1978-79 1979-80 1980-81 Target Actual Target Actual Target Actual {Classes 30 30 100 120 n.a. 1,718 4 {Pupils 1,200 1,200 4,000 4,800 n.a. 68,720 {Classes 30 30 100 330 5 {Pupils 1,100 1,110 3,700 12,210 {Classes 300 27 6 {Pupils 1,050 945 Total Classes 30 30 130 150 n.a. 2,075 Total Pupils 1,200 1,200 5,100 5,910 n.a. 81,875 Evaluation of CET and RATES 4.07 The evaluation of the effects of radio on pupils' performance was a major objective of the project. To test the impact at the end of the project period, nine treatment combinations were established. These were: (1) RATES alone; (2) CET assisted teachers, without other inputs; (3) RATES and CET combined; (4) RATES and one textbook to two pupils; (5) CET and one textbook to two pupils; (6) A 1:2 textbook/pupil ratio without other inputs; (7) A 1:1 textbook/pupil ratio without other inputs; (8) RATES, CET and 1:2 textbook/pupil ratio; and (9) CET and a 1:1 textbook/pupil ratio.  - 15 - Students were given pre-tests and post-tests in Pilipino, covering reading and listening skills. Although other outcomes of CET were assessed, the major attempt at measuring the impact of lessons for teachers was made in the experimental design, which assessed student achievement in Pilipino. Analyses of the test data were conducted in 1980 by the University of the Philippines, Institute of Mass Communications (UPIMC) which issued its final report in June 1982. Unit costs of the various treatments rate are shown in Table 4.3. 4.08 The findings indicate that in the two main testing regions, Pangasinan and Leyte, the strongest improvements in pupil performance resulted from the treatment combinations shown in Table 4.3. Table 4.3: EFFECTIVE MEANS OF IMPROVING PUPIL PERFORMANCE Cost range Treatment per pupil Treatment number US$ /a RATES with CET trained teachers 10.40-11.06 (3) RATES with a 1:2 textbook/pupil ratio, 10.10-10.80 (4) RATES alone 9.86-10.53 (1) CET and a 1:2 textbook/pupil ratio 0.74-0.82 (5) /a The costs are UPIMC's figures. Caution should be used in applying these costs in absolute terms because of reasons stated in paras. 4.02-4.03. 4.09 While all these methods produced significantly increased pupil performance, it is interesting to note the cost differentials. The combination of providing one textbook to a pair of students and training the teachers through mass media (treatment #(5)) produced a result which was less than 1/10th as costly on a per pupil basis as any of the other combinations. Study on Communications Technology for Rural Education 4.10 Quantitative analysis of this study is difficult because there is no information on how many persons listened to the broadcasts. Some people listened but did not attend group meetings, others attended but did not listen often, and continuity was sporadic. The UPIMC findings were that:  - 16 - (a) radio (in spite of its limitations), is a stimulus which can interest farmers who are not very literate both to listen and learn and can also promote group discussion; (b) radio is a good supplement to existing rural community and agricultural services but it is difficult to form new groups to use it; and (c) timing of radio broadcasts must be flexible and discontinued in certain seasons as they will not be listened to when farmers are in their fields. 4.11 The radio programs did have another impact. The Ministry of Agriculture asked to have copies of the scripts and in 1981 the Ministry of Public Instruction broadcast them in toto to the whole of the Philippines on its national 50 kW station. Future of the Training Materials 4.12 The pilot project accumulated a considerable amount of knowledge and the means of extending it to more pupils and teachers. This consists of: (a) the pedagogic material in the pupils' worksheets (RATES) and the teachers' manuals (CET); and (b) the equipment to make use of the above and to replicate more material. Project results are being used in the Elementary Education Sector Program (Loan 2030 of FY81) where teachers are being trained in new curricula and teaching methodology with the aid of taped material. Textbooks are being distributed to schools at the rate of one textbook per two pupils. This is in accordance with earlier practices, but the results of Loan S8-PH show that it is an effective practice in terms of learning achievements.  PHILIPP INES MASS MEDIA PILOT PROJECT ORGANIZATION OF THE EDUCATION COMMUNICATIONS OFFICE (ECO) Pit'tate-r. ft Diectora, MC Ex-acutive Director. cOIar eiamsLXaOfft eas (1, VE&I & UE) |Internationa 1 oo"oratinsL ' gte : |e IPeticipattg Natfoaat AgencieN ECO pro rc Director 1 m tCO istent Project DLr eto _ry | Suvnwtie Ev;l.ztten Curricu.No Development And NEC C.rrt cuIu rou l _ tE~~~~~~~~~Aanl trAv too Currculw DOw esoP-at Media ftanning ftoductivo Rsaearch and Dw lopsent Project Planomog Untl Progra * rvie, Unit Field Dpratle n 1ett Field 'tilieatioa __ .tirit untt Unit | r- Curricutunp D"lopmest e l dis PlananmeoJroductton Research and D.vlop.eot Project FlnontV Head Lay-Out Deeignotl Tield tratltas ?Paaestaroc Offi!er Officer Officer Copy Editor Officer a) I Rrgioaai Coot tAtIS Cucrriaum 1p hedta Vesgg Officer Field Reuearch Officer Assistaut projeet Lay-Out Assist eot/eIt 14 lull-time field lejon I Specislist lAns ScriptWitetr PATES Evlusttioe Officer ?leantni Officer StothL Officer staff ti pa"astem bI I Di.. Ftetd Cl I part-tbm l_ 1aa print Editor Ciz Formative svluatteen tAIS fto5rae fl?a yrint Tachnticn))ls.- 2 full-time Felid CtilikatLon '' writers, Production AeeiutAnt Offtcer cl Officer ograph Operator Staff to Leyte Coeisatoct. (I 'cld ypfat Typits, " T t Planenrng Ivstrialian Officer fie fellies Field i. a tatu rsi ocfiet toffnlerr Staff io Modulea e) a tge Sra%i" Audi* Terhtcinelo/Wr,- deijo. lAoders (factch ducefon Traffict.trc fr- officear P ng..lnac. Scat a) I Ragaonal Cool A.Flon Ill b) I flivitito Vtei Vtflitatioo C.~,d lmarstr fhiuperintend*nq c) 7 Fejn= Seg id L.eader% (FAcMal s4iojor of Dcvi, Word lnii. & flistrict Sipen son La Leyte) 1 1 legiocal Coci lef ion %II lb) 2 t'ivisicv Ft.e (Ed. S~PswvlseE c) 3 For... Stsait Leaders lid. supervisors.  - 18 - ANNEX 2 PHILIPPINES MASS MEDIA PILOT PROJECT LOCAL EDUCATIONAL ADMINISTRATION (Ministry of Education (General and Culture Administration) Director Region Region Superintendent ; I Assistant Superintendent Educational Province General Education Division Supervisors Subject Supervisor District Municipality Schools District (Barangay)  COMMENTS FROM THE BORROWER - 19 - ATTACHMENT I (p. 1 of 2) REPUBLIKA NG PILIPINAS (Republic of the Philippines) MINISTRI NG EDUKASYON AT KULTURA (MINISTRY OF EDUCATION AND CULTURE) M A Y N I L A (MANILA) rANGGAPAN NG MIHISTRI OFFICE OF THE MINISTER 15 January 1984 Shiv S. Kapur Director Operations Evaluation Department World Bank 1818 Street NoW. Washington D.C. 20433 U. S. A. Dear Mr. Kapur: Thank you very much for your letter asking for comments on the S008-Ph Project Completion Report prepared by the Operations Evaluation Department. I would like to draw your attention to the need for revision of the Highlights Section, paragraph 3; 2.01; 2.03;4.01; and 4.02 of the report. In particular, there is need to correct said sections: which describe the transmission of radio lessons in the project sites. The actual transmission activities were: 1. For Year One (Schoolyear 1978-79) which was the shakedown phase, we piloted the Project in 30 Grade IV Pilipino classes in Pangasinan and 162 teacher participants of the Educational Trends Course of CET. In accordance with the loan covenant, we used taped lessons as it would be rather expensive to use radio air time for a very limited number of project participants. 2. For Year Two (Schoolyear 1979-80), we used radio broadcast lessons to reach 120 Grade IV Pilipino classes and 5,582 teacher participants in Educational Trends and Pilipino Courses of CET, in the two (2) pilot sites of Pangasinan and Leyte. These were the main project sites included in the summative evaluation. Again, as per loan agreement, tapes were used in 30 classes of Grade V Pilipino and the cET courses (Social Studies and English) which were pre-tested to a limited number of teacher participants (p.4-20 Project Completion Report).  - 20 - ATTACHMENT I (p. 2 of 2) In Maguindanao, radio transmission was used by RATES and taped lessons were used by CET. Take note, however, that Maguindanao was not Dart of the testing sites of UP-IMC. We extended the project to this area primarily because of the strong request from the MECS Regional Office to extend our servicea -to wr 8-ao. 3. For Year Three (Schoolyear 1980-81), which was the expansion phase of the projeat, radio lessons were transmitted to RATES Grade IV and V Pilipino and the CET courses in the 2 project sites (Pangasinan end Leyte). Taped lessons Wrere used for the pre-testing of RATES Grade VI Pilipino and OFT courses (science/Math). In addition, we conducted two experiments using tape-assisted instruction (pp. 6-46 up to 6-54 of the Project Completion Report). These studies were in addition to, not a switch from, the radio lessons aired in Pangasinan and Leyte. In sum, although taped lessons were used all throughout the life span of the Project, they were just pre-test activities conducted in accordance with the loan covenant. Special Studies on taped instructions were carried out also in accordance with Schedule 2 B 2 of the loan covenant. Radio transmission was adhered to in the two major project sites from 1978 to 1981. We are sending you a telex version of these comments. We are also sending, through WB Education Consultant John Middleton, a copy of the Project Completion Report which we prepared for your reference. Best regards. Very truly yours, ZENAIDA T. DOMINGO