China - Watershed Management Project : development of a monitoring and evaluation system - final report

56644 China Watershed Management Project (CWMP) Development of a Monitoring & Evaluation System Final Report ITAD Ltd 12 English Business Park English Close, Hove June 2006 1.1.1.1.1.1.1 BN3 7ET U.K. Team Leader: Tel: 01273 765 250 Johanna Pennarz Email: mail@itad.com Web: www.itad.com  China Watershed Management Project Final Report (June 2006) 2 Table of Contents: Executive Summary........................................................................................................ 7 1 Background .......................................................................................................... 12 1.1 Project Background ...................................................................................... 12 1.2 Objectives and scope of this assignment (ToR) ........................................... 13 1.3 Steps in Developing the M&E system ........................................................... 14 2 Review of Loess Plateau Project M&E system...................................................... 16 2.1 M&E system administration .......................................................................... 16 2.2 Indicators monitored in Loess Project .......................................................... 18 2.3 Physical progress and outputs quality monitoring ....................................... 19 2.4 Monitoring project impact............................................................................. 20 2.5 Findings on soil and water monitoring ......................................................... 21 2.6 Findings on poverty monitoring ................................................................... 23 2.7 Conclusions on M&E system design ............................................................ 25 3 Developing an M&E System for CWMP................................................................. 27 3.1 Stakeholder Analysis.................................................................................... 27 3.2 Stakeholder agreement on focus of M&E system ......................................... 29 3.3 Objectives of M&E for CWMP ....................................................................... 30 3.4 M&E framework development....................................................................... 31 3.5 Learning and capacity building..................................................................... 32 3.6 From Loess Project to CWMP: A major shift in M&E .................................... 34 4 Towards an Approach to Watershed M&E ............................................................ 36 4.1 How to define a watershed?.......................................................................... 36 4.2 Objectives of Watershed Management ......................................................... 37 4.3 Environment-Poverty Linkages..................................................................... 38 4.4 Focus of watershed M&E.............................................................................. 40 4.5 Piloting M&E in CWMP.................................................................................. 41 5 M&E Framework for CWMP................................................................................... 43 5.1 Conceptual Framework for M&E ................................................................... 43 5.2 Indicators...................................................................................................... 45 5.3 Data collection methods ............................................................................... 49  China Watershed Management Project Final Report (June 2006) 5.4 Sampling strategies...................................................................................... 50 5.5 Reporting ...................................................................................................... 54 5.6 Data analysis................................................................................................ 57 5.7 Integrating and presenting M&E information ................................................ 60 6 Stakeholder participation in M&E ......................................................................... 64 6.1 Institutional framework................................................................................. 64 6.2 Rationale for stakeholder participation in M&E ............................................ 65 6.3 Approach to participatory monitoring in CWMP ........................................... 68 6.4 Indicators for participatory monitoring ......................................................... 70 6.5 Participatory evaluation of project activities ................................................ 73 6.6 Monitoring stakeholder participation in CWMP ............................................ 74 7 Implementing baseline surveys............................................................................ 76 7.1 What is a "baseline"? ................................................................................... 76 7.2 Basic steps in implementing a baseline........................................................ 77 7.3 Using baseline information........................................................................... 80 8 Further steps ........................................................................................................ 82 8.1 Building capacities for participation ............................................................. 82 8.2 Building capacities for M&E.......................................................................... 83 8.3 Accountability and transparency in M&E...................................................... 87 8.4 Criteria for assessing the quality of M&E...................................................... 88 9 Conclusions and Outlook ..................................................................................... 90 Annexes........................................................................................................................ 92 Loess II Log-Frame and Key Performance Indicators............................................... 92 Key performance indicators for CWMP..................................................................... 93 National System for Monitoring Soil and Water Conservation .................................. 96 Terms of Reference for Baseline Survey on Key Poverty Indicators .......................100 Terms of Reference: Baseline survey on Soil and Water Indicators .......................108 Terms of Reference: Baseline on Biodiversity Indicators .......................................116 4  China Watershed Management Project Final Report (June 2006) List of Boxes: Box 1: From the Project memorandum: ............................................................................ 13 Box 2: Indicators used in the existing M&E for Loess-I and Loess-II ................................... 18 Box 3: Progress M&E steps for Loess-Project.............................................................. 20 Box 4 Monitoring project impact in Loess ProjectI ....................................................... 21 Box 5: Stakeholders in Loess Project M&E................................................................... 27 Box 6: From implementation monitoring to results monitoring ............................................ 34 Box 7: Criteria to assess quality of indicators .................................................................... 49 Box 8: Criteria to a assess quality of data collection .......................................................... 49 Box 9: Wealth Ranking of Rural Households in Mahuang Village, Kongtong County............ 51 Box 10: Wealth Ranking of 24 Villages in Dazhai Township of Kongtong County ................ 52 Box 11: Sampling of villages within a watershed ............................................................... 52 Box 12: Quantitative & Qualitative Assessment of Poverty compared................................. 59 Box 13: Scoring key performance indicators on poverty (example) ..................................... 61 Box 14: Methods for participatory project evaluation.......................................................... 73 Box 15: Marginal Groups in Watershed Management Projects........................................... 74 Box 16: Indicators to monitor stakeholder participation ...................................................... 75 Box 17: Basic steps in implementing a baseline................................................................ 77 Box 18: Small watersheds selected for CWMP (Gansu Province)....................................... 78 Box 19: M&E skills for CWMP project staff........................................................................ 84 Box 20: Quality criteria and key features of M&E............................................................... 88 List of Figures: Figure 1: Institutional structure of Loess M&E ................................................................... 17 Figure 2: Typical Hierarchy of Watersheds at Farm and Village Levels ............................... 36 Figure 3: Levels of Monitoring.......................................................................................... 43 Figure 4: Levels for monitoring impacts in CWMP ............................................................. 44 Figure 5: Information pyramid and information flows in M&E .............................................. 55 Figure 6: Information triangle in M&E ............................................................................... 58 Figure 7: Monitoring platforms and stakeholder groups in CWMP ....................................... 64 Figure 8: National soil and water monitoring network ......................................................... 98 5  China Watershed Management Project Final Report (June 2006) List of Tables: Table 1: Steps for setting up the M&E system in CWMP .................................................... 14 Table 2: Objectives for improved M&E in CWMP, changes and institutional implications...... 31 Table 3: Overview - Building capacities in the process of M&E development ....................... 33 Table 4: Structure of indicators on environment and povert y .............................................. 46 Table 5: Key performance indicators to monitor long-term changes at the watershed level .. 47 Table 6: Indicators to monitor environment-poverty linkages within a watershed ................. 47 Table 7: Indicators to monitor direct project outcomes ....................................................... 48 Table 8: Indicators to monitor access to project services by poor and marginal groups ........ 48 Table 9: Sampling strategies for M&E .............................................................................. 54 Table 10: Reporting system on M&E ................................................................................ 56 Table 11: Responsibilities for data collection and analysis ................................................. 60 Table 12: Scoring indicators for different watersheds (example) ......................................... 61 Table 13: Integrated data analysis at different management levels ..................................... 63 Table 14: Comparing questionnaire survey and participatory data collection ....................... 68 Table 15: Indicators that can be monitored through participatory assessments (selection) ... 71 Table 16: Collecting information on indicators through participatory assessment (example) . 71 Table 17: Indicators and methods for participatory impact assessments ............................. 72 Table 18: Household poverty indictors and data (2005 baseline) ........................................ 76 Table 19: Basic data of 4 experimental small watersheds............................................ 79 Table 20: Institutional change and capacity building for M&E ............................................. 85 Acronyms used: CWMP China Watershed Management Project DFID Department for International Development EV Extension Volunteer LGU Local Government Unit Loess Loess Plateau Project (funded by World Bank) M&E Monitoring and Evaluation MWR Ministry of Water Resources MoU Memorandum of Understanding PADO Poverty Alleviation and Development Office PAB Participatory Assessment of Biodiversity PIA Participatory Impact Assessment PIO Project Implementation Office (Xi'an) PLG Project Leading Groups (provincial, city and county levels) PMO Project Management Office PTD Participatory Technology Development RSO Rural Survey Organisation WB World Bank WCB Water Conservation Bureau WCMN Water Conservation Monitoring Network YRCC Yellow River Conservancy Commission 6  China Watershed Management Project Final Report (June 2006) Executive Summary Background The objective of this assignment was to develop an M&E system for watershed management in the Loess Plateau area. The M&E system for the China Watershed Management Project (CWMP) has been developed on the base of the M&E system that has been implemented during the previous two phases of the World Bank Loess Plateau Project. The China Watershed Management Project (CWMP) is located in Gansu Province which is part of the Second Loess Plateau Watershed Rehabilitation Project. The Loess project area shows high levels of environmentally-related poverty. Without sufficient vegetation covers, soil erosion is severe in the Loess area, causing loss of fertile land in the upper reaches of the watershed, and problems of sedimentation in the lower reaches. There is a critical need to be able to reliably assess the effectiveness and impact of watershed management in the Yellow River Basin. The China Watershed Management Project (CWMP) is contributing to the improved management of the Yellow River, and other basins, by developing best practice models for oess project M&E system has been successfully watershed management. The previous L implemented during two project phases. It had its strength in monitoring progress of project interventions against planned targets. It has faced a number of problems on monitoring impacts on soil erosion, sedimentation and water quality, mostly due to capacity constraints and lack of relevant data. It had less emphasis on monitoring environmental impacts, and it had obvious shortcomings in monitoring poverty impacts. A key problem for the Loess project was attribution: how could project results possibly be linked to wider environmental and socio- economic changes within the loess area? The challenge for this assignment was to develop an approach which would allow for better monitoring of the different dimensions of change within a watershed context and at the same time address the problem of attribution. Approach A framework for M&E in CWMP has been developed through the following steps: The M&E systems review has been looking at the existing M&E in Loess Project. It has assessed existing of practices for monitoring environmental and socio-economic indicators and identified a number of practical problems and issues. The M&E systems review has also been looking at the overall structure and functioning of the Loess M&E system. It found that 7  China Watershed Management Project Final Report (June 2006) the Loess Plateau Projects' M&E system was strongly focused on checking physical implementation progress, with a less structured approach to monitoring outputs or evaluating outcomes and impact. It suffers from limitations due to design (small sample size, poor poverty targeting, etc), use and flow of data (lower levels collect and pass data upwards, where decisions are made, and passed back down), and lack of farmer participation in the system (which would improve accountability and cost effectiveness During the inception phase, a stakeholder analysis has been done and the project has agreed that stakeholders in the project area, especially poor farmers, need to participate more in future M&E. The project has also agreed on the main objectives for the revised M&E system. The new M&E system should