Experiences with the monitoring and evaluation of training and visit extension in India

FIlE 001 SWP595 Experiences wiiie the- l toringi anorEvaluation of Training and Visit Extension in India Gershon Feder Roger H. Slade WORLD BANK STAFF WORKING PAPERS Number 595 FILE COPY i   WORLD BANK STAFF WORKING PAPERS C49 7 Number 595 Cv3 Experiences with the Monitoring and Evaluation of Training and Visit Extension in India Gershon Feder Roger H. Slade INTERNATIONAL MONETARlY FM JOINT LIBRARY [AAR 6 1984 INTERNATIONAL BANK FOR RECONSTRUCTION ANS DEVELOPMENT WYASHINGTON, D.C. 20431 The World Bank Washington, D.C., U.S.A.  Copyright ( 1983 The International Bank for Reconstruction and Development / THE WORLD BANK 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. First printing May 1983 All rights reserved Manufactured in the United States of America This is a working document published informally by the World Bank. To present the results of research with the least possible delay, the typescript has not been prepared in accordance with the procedures appropriate to formal printed texts, and the World Bank accepts no responsibility for errors. The publication is supplied at a token charge to defray part of the cost of manufacture and distribution. The views and interpretations in this document are those of the author(s) and should not be attributed to the World Bank, to its affiliated organizations, or to any individual acting on their behalf. Any maps used have been prepared solely for the convenience of the readers; the denominations used and the boundaries shown do not imply, on the part of the World Bank and its affiliates, any judgment on the legal status of any territory or any endorsement or acceptance of such boundaries. The full range of World Bank publications is described in the Catalog of World Bank Publications; the continuing research program of the Bank is outlined in World Bank Research Program: Abstracts of Current Studies. Both booklets are updated annually; the most recent edition of each is available without charge from the Publications Distribution Unit of the Bank in Washington or from the European Office of the Bank, 66, avenue d'Iena, 75116 Paris, France. Gershon Feder is a senior economist in the Agriculture and Rural Development Department, and Roger H. Slade an economist in the Projects Department of the South Asia Regional Office, of the World Bank. Library of Congress Cataloging in Publication Data Feder, Gershon, 1947- Experiences with the monitoring and evaluation of training and visit extension in India. (World Bank staff working papers ; no. 595) Bibliography: p.' 1. Agricultural extension work--India--Evaluation. 2. Agricultural development projects--India--Evaluation. I. Slade, Roger, 1941- . II. Title. III. Series. S544.5.15F4 1983 630'.7'15054 83-6840 ISBN 0-8213-0192-6  ABSTRACT The paper describes the recent experiences of implementing monitoring and evaluation systems in T&V extension projects in India. Operational problems relating both to administrative and methodological aspects are covered, and partial preliminary results from a detailed case study of extension operations and farmers' practices are reviewed. In the course of this review, the authors demonstrate how the results can be presented in a manner which is useful for monitoring and evaluation purposes. The paper concludes with some suggested lessons from experience gained so far.   Table of Contents Page No. I. Introduction ..... .00 0 0..................... II. Background to a,nd Outline of the Current M&E System...9 . ........... ... ... . 2 III. Some Operational Problems. ........................ 7 IV. An In-Depth Study of the Impact of T&V Extension......................................... 19 V. Possible Improvements to the M&E System.................... 34 References.   I. Introduction The proper conduct of monitoring and evaluation is an important element contributing to the successful implementation of extension projects. The highly structured format of extension operations under the Training and Visit (T&V) system, offers opportunities for effective monitoring of several extension activities, and with a well defined set of impact points (improved practices), some aspects of evaluation can be addressed in a relatively straightforward manner. But the overall complexity of assessing eventual extension impact on farmers' yields and incomes poses a problem with no easy or simple solutions. In the present paper we describe recent experiences with implementing monitoring and evaluation systems in T&V extension projects in India. Opera- tional problems relating both to administrative and methodological aspects are covered, and partial preliminary results from a detailed case study (being undertaken by the authors) of extension operations and farmers' prac- tices are reviewed. In the course of this review, we endeavour to demon- strate how the results can be presented in a manner which is useful for monitoring and evaluation purposes. The ultimate purpose of the data gathered within the framework of this case study is to serve as a basis for an econometric analysis of extension impact. Thus, the study will need to grapple with complicated methodological issues, but we expect to derive  -2- insights with operational relevance both for extension management and M&E units. The structure of the paper is as follows: the next section provides background on the implementation of monitoring and evaluation systems of extension projects in India. It is followed by a discussion of operational and methodological problems affecting the establishment of a functioning M&E system in the context of Indian extension projects. An in-depth study of extension impact is described in a subsequent section, and preliminary results are presented. The paper concludes with some suggested lessons from experience gained so far. II. Background to and Outline of the Current M&E System The T&V system is a hierarchically organized and time bound structure designed to deliver selected, timely and feasible technology to farmers with strict regularity. The technology is encapsulated in carefully formulated recommendations often termed impact points which extension agents convey to contact farmers and such other farmers who attend the fortnightly visits or who approach the agent on their own initiative at other times. Full details of the structure and operating procedures of T&V extension are set out in Benor and Harrison (1977). The system is designed not only to deliver messages to the farming population but to identify and feed back to senior extension officers, and ultimately applied researchers, farm level problems. This feedback mechanism  -3- is designed to ensure that the system remains responsive to farmers' needs and continues to deliver relevant technology. This reverse flow of informa- tion, however, is not sufficient for managers to monitor the system's efficiency in delivering information nor to form judgments about the extent to which impact points are adopted by farmers and the consequent effects on crop yields and productivity. Hence, at an early stage in the introduction of T&V extension in India, the need for monitoring and evaluation was iden- tified and resources for such work were built into the projects. Sub- sequently, it was realized that the monitoring and evaluation function itself needed to be defined, structured and systematized. The first attempt at this, for Indian T&V projects, was made by Cernea and Tepping (1977) who, in World Bank Staff Working Paper 272, defined three zones of concentration for monitoring and evaluation: (i) the visits of the VEW to the contact farmers; (ii) the extent to which the recommended practices (impact points) conveyed by the VEW are adopted or not adopted; and (iii) the yields obtained by farmers in the project area. The implementation of an effective M&E system in various projects, as recommended by the Bank, was unfortunately slow, for the simple reason that adequate M&E units, with proper staffing and resources were not established as planned. Project managers were often reluctant to establish M&E and accorded higher priority to staffing other operational activities. In the case of those projects where the commitment to M&E was stronger, the recom- mended systems started to be implemented and produced results, but this was not the general situation. In an effort to overcome this implementation  -4- bottleneck, the procedures laid down by Cernea and Tepping were summarized by Harrison (1979) and again recommended by the Bank for application in all extension projects, but staffing resources for M&E were still inadequate. Given the growing demands for information derived from monitoring and evalua- tion, steps were taken to provide direct assistance to projects for estab- lishing M&E systems, and resulted in the preparation in the World Bank of a manual for survey-based monitoring and evaluation (Slade and Feder, 1981), as well as a research project designed to produce, inter alia, improvements in the M&E of extension projects. This manual also recommends that the surveys be complemented by special in-depth and rapid case studies, the methodologY for which is formulated in detail in the 1977 study by Cernea and Tepping. In 1981, in conjunction with the development of the manual, the Government of,India took the initiative to increase the priority accorded to M&E by the states and to promote uniform methods on a national scale. Accordingly, the Directorate of Extension organized two 'All India' workshops for senior state level M&E staff. The first of these workshops was held in Rajasthan in July 1981 and the second in Tamil Nadu in February 1982. 1/ Much of what follows is related to the experience of implementing the recom- mendations contained in this manual which, in combination with the special studies described in SWP 272, were suggested as a standard approach to monitoring and evaluation in T&V systems in India. Extension in India, being 1/ M&E practitioners from Rajasthan, Haryana, Karnataka, Madhya Pradesh and Gujarat were present at the first workshop and from Tamil Nadu, Kerala Maharashtra, West Bengal, Bihar and Andhra Pradesh at the second.  -5- a part of agriculture, is constitutionally a state prerogative and hence the Union Government's role with regard to extension is essentially advisory. Consequently, some diversity in the approach to monitoring and evaluation is inevitable for this reason alone, apart from the more obvious differences resulting from adaptations made at the initiative of individual projects, or differences in local experiences, culture, climate and agricultural prac- tices. In many ways, the monitoring and evaluation system now under widespread implementation in India focusses on monitoring rather than evalua- tion. Apart from the routine and increasingly standardized reporting of physical, financial and administrative progress of extension implementation in each state, the thrust of the M&E system is to objectively measure and report the extent and quality of agents contact with farmers and the degree to which impact points are adopted. These activities are complemented by yield measurements which act as an important, albeit crude, method of evaluating system effects. This simple approach is the direct consequence of the lack, at the project level, of computer technology for the processing and analysis of field survey data. It also reflects the difficult methodological problems that must be overcome in any attempt to attribute productivity effects to extension. Typically in India, the extension system is part (albeit a very large one) of the State Department of Agriculture. At the head of this organiza- tion is the Secretary for Agriculture (or frequently, the Agricultural Production Commissioner) supported by the Director of Agriculture who, in  -6- turn, has a number of Additional Directors at his command, one of whom nor- mally heads the extension system. These senior officers collectively com- prise the management of the system and are thus the primary audience for information flowing from the monitoring and evaluation system. Because of their decision making functions, they are the officers who must initiate actions to modify and improve extension operations on the basis of the avail- able evidence. For these actions to be taken in the best interests of the extension organization and its clients they should be based on information which is as timely, objective and accurate as possible. In pursuit of these ends, the monitoring and evaluation cell should be located close to all of these senior officers but be directly responsible to an officer senior to the Additional Director responsible for extension. This will enhance the inde- pendence of operation and judgement which are crucial if the M&E cell is to successfully perform its duties. These principles are well understood in India but established rules of bureaucratic precedence and seniority sometimes operate to make its implementation difficult. In some states for example, the organization is nominally as above but owing to the lower rank of the head of M&E cell (usually a Joint Director), the Additional Director for Extension is able to comment on (and, therefore, influence) M&E reports before they reach the Secretary for Agriculture. Elsewhere the Additional Director (Extension) in the face of weaker or more disinterested superiors has succeeded in formally establishing his control over monitoring and evaluation. In yet another case, a sensitive extension director has reacted adversely to M&E reports not  -7- .because they contained damaging findings but because they were first, and quite correctly, considered and cleared by the Secretary for Agriculture. These cases highlight the difficulties of guaranteeing objectivity whilst simultaneously ensuring harmonious working relationships. In the last analysis, it seems that formal structures are less important than sound judgement, integrity and a will to make the system work well on the part of all involved. III. Some Operational Problems Monitoring and evaluation is a technical activity requiring highly developed and specialized skills in the social sciences and statistics, combined with substantial knowledge of the T&V system and the constraints surrounding its development. It is a task which requires the full time attention of those engaged in it. This principle is well observed in many states but in some the M&E unit has inherited or been subsequently charged with other duties (usually, it should be added, of a monitoring or evaluative nature). These other duties may extend from assistance in preparing the departmental work plan and budget to conducting M&E studies of other depart- mental activities or simply undertaking urgent administrative tasks handed down by busy superiors. Such deviations are counterproductive and have led, where they occur, to lower quality work or to long delays in producing reports. This results in loss of topicality and hence credibility in the eyes of both the M&E unit's masters and the extension wing at large. Specialization and the exclusion of extraneous activity are as important for M&E as they are the T&V system proper.  Of equal importance to successful M&E is a sufficient number of staff. A suggested staffing structure for a cell in a state of fifteen districts is set out in Fig I. This 'model', however, has found little expression in reality. In part this is the result, in some early state projects, of staff allocations being made before the M&E system had been designed in detail, in others it is a result of staffing structures inherited from pre-T&V days. Elsewhere, it reflects the exigencies of recruitment. The most serious and frequently encountered weaknesses in staffing lie in the senior ranks. In some cases, insufficient professional and supervisory posts have been provided to ensure reasonable spans of command whilst in others posts have been created but remain unfilled owing to a lack of qualified staff or grading difficulties. This can have severe consequences on both data quality and work output. In one large state, extensive and apparently acceptable data have been collected and satisfactorily tabulated but remains unreported owing to a shortage (unfilled positions) of economists to inter- pret and summarize these results. In the aggregate, however, the position in early 1982 was not unmanageable as Table 1 makes clear: Table 1: STAFF POSITIONS IN M&E UNITS OF FIVE STATES /a (March 1982) Number of Sanctioned Vacant Positions Posts Number Percent Senior posts 23 8 35 Middle and supervisory posts 67 6 9 Field and clerical posts 275 57 /b 21 /a Haryana, Gujarat, Rajasthan, Madhya Pradesh and Karnataka. 7b- Twenty-seven vacant posts in one state.  -9- Rgure 1 Structure and Size of a Representative Monitodng and Evaluation Cell for a State or Project With Rfteen Dlstilcts Total Staff Head I 1 Sr. Ag. Ecan. I I Ag.Econ. I1I3 Statistician Field Supels 2 2 2 6 Clerks 1 12 Investigators 101 030 54 World Bank-24898  -10- Numerical sufficiency is only one method of determining staff ade- quacy. Quality, means of control and turnover are also of importance. In general, staff turnover is not a serious problem except in one or two states where entry qualifications, especially for field investigators have been set high (degrees in statistics), but the salaries are not commen- surate with positions requiring similar qualifications elsewhere in govern- ment. Under these circumstances, highly qualified staff, often with mainly urban backgrounds, become disgruntled, and unwilling to work in rural areas and hence seek other positions. At more senior levels, and partly as a result of promotion being largely dependent on seniority, there is a tendency for inappropriately qualified staff to be appointed to senior and specialized M&E positions. The consequent dilution of professionalism has deleterious effects on the overall quality of M&E work. Work quality is also much influenced by the efficiency of supervision and the directness of lines of command, especially for field staff who are commonly stationed throughout the state. We note two common problems. First, administrative convenience requires that M&E field inves- tigators at the district and sub-district level be attached to the district agricultural office, controlled by the District Agricultural Officer (DAO). This creates the possibility that the M&E unit losses some control over its  field staff and in practice, depends crucially on the good will of the DAO's in maintaining speedy lines of communication and disinterested supervision. Second, and of greater importance, is the ability of senior M&E staff stationed at headquarters to regularly visit their far flung charges in the field partly to check on their work and partly to keep themselves abreast of current farming conditions. This ability hinges crucially on the availability of suitable transport facilities. Such facilities are not everywhere available (despite adequate provision at the time of project design) either because vehicles have not been procured or have been procured and allocated to other uses. In some cases, vehicles are available only from a departmental pool whilst elsewhere government restrictions on mileage and number of occupants materially hinder effective field supervision. The need for M&E information to be efficiently collected and speedily tabulated and reported requires that the M&E unit use simple but effective methods of work planning. Each season requires a detailed plan of campaign to be drawn up and diligently executed. One season passes inexorably to another and early slippage becomes cumulative to the point where control and order is lost. Figure 2 sets out, for hypothetical rabi (dry) and kharif (rainy) seasons, the primary steps involved in executing monitoring and evaluation surveys.  -12- Such timetables are now in use in most states in India but several have failed to allow enough time 1/ for critical activities and serious slippage has ensued. Experience will rectify this weakness but in the shorter run it gives rise to backlogs of work that are not easily removed. In the light of these difficulties, it is clear that M&E survey samples should not only be statistically efficient, but should also minimize resources necessary for their execution. Additionally, experience in India shows that it is rarely feasible to conduct more than two sample surveys in a season. Hence the manual (Slade and Feder, 1981) recommends two sample surveys each season. The first of these (the monitoring survey) is conducted during the growing period and the second (the monitoring cum evaluation survey) during the immediate post-harvest period. The monitoring survey is conducted on a specially designed random sample of farmers whilst the monitoring cum evaluation survey is conducted using the normal crop-cutting samples in the state concerned. Whilst sample sizes for the monitoring cum evaluation survey vary somewhat, mainly in relation to the particular state (and, by definition, district), the manual offers detailed guidance on the sample size for the monitoring survey. In general, the emphasis is on mini- mizing the sample size while maintaining acceptable levels of precision at the district level. 2/ 1/ Often because of a failure to allow for the effects of existing staff shortages. 2/ Whilst the number of farm families in a district is variable, districts containing 200,000 farm families are not uncommon.  ii , tDc X E 1YrI XX Ipp I-. anYr2 t _ J>neA lst Yr2 3 X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Jt l e d Yr 2