Crisis monitoring and response system

 70639 Crisis Monitoring and Response System (CMRS) Detailed Report September 2010  Table of Contents Abbreviations, Acronyms and Terminology .................................................................... 4 1 Purpose and Content of Document ........................................................................... 6 PART A: CMRS DESIGN, DEVELOPMENT AND IMPLEMENTATION ................. 7 2 CMRS Initiation and Organisational Structure ......................................................... 7 2.1 Origin and General Objectives ...................................................................... 7 2.2 Terminology .................................................................................................. 7 2.3 Scope.............................................................................................................. 8 2.4 Funding and Budget ....................................................................................... 8 2.5 Data Requirements: Content, Frequency and Timing ................................... 8 2.6 Existing Sources of Data ............................................................................... 9 2.7 CMRS Components ....................................................................................... 9 2.8 Complementary Qualitative Crisis Monitoring and Analysis ....................... 9 3 Management and Roles ........................................................................................... 11 3.1 Introductory Remark .................................................................................... 11 3.2 Client............................................................................................................ 11 3.3 Development and Analysis Team ................................................................ 11 3.4 Data Collection and Capture Contractor...................................................... 11 3.5 Consultant .................................................................................................... 12 3.6 Project Manager ........................................................................................... 13 3.7 Data Analyst and IT Specialist .................................................................... 14 3.8 Funding Agency........................................................................................... 14 3.9 Core Analysis Team and Analysis Review Group ...................................... 15 3.10 Other Stakeholders....................................................................................... 15 3.11 Summary of Roles ....................................................................................... 15 4 CMRSS General Design Considerations................................................................. 17 4.1 Specific Objectives ...................................................................................... 17 4.2 Coverage Considerations ............................................................................. 17 4.3 Content Considerations................................................................................ 18 4.4 Frequency and Timing Considerations ........................................................ 18 4.5 Relationship to Existing BPS Surveys ......................................................... 19 4.6 Choice of Panel Design ............................................................................... 20 4.7 Survey Collection Methods ......................................................................... 20 4.8 Obtaining Measures of Change ................................................................... 20 4.9 Approach Modelled on Lot Quality Assurance Sampling (LQAS)............. 21 5 CMRSS Sample Design .......................................................................................... 23 5.1 Sakernas Sample Design.............................................................................. 23 5.2 CMRSS Sample Design ............................................................................... 25 6 CMRSS Questionnaire and Indicators .................................................................... 27 6.1 Sakernas Questionnaire ............................................................................... 27 6.2 CMRSS Questionnaire ................................................................................ 27 6.3 Definition and Derivation of Indicators ....................................................... 28 CMRS Detailed Report Page 1 of 91 7 Data Collection ........................................................................................................ 32 7.1 Three Rounds of Data Covering Four Quarters ........................................... 32 7.2 Interviewer Assignment and Training ......................................................... 32 7.3 Handling Non-Response .............................................................................. 32 7.4 Transmission of Completed Questionnaires to Head Office ....................... 33 8 Data Capture, Storage and Preparation for Analysis............................................... 34 8.1 Data Capture, Storage and Transmission..................................................... 34 8.2 Data Preparation .......................................................................................... 34 9 Aggregation and Estimation .................................................................................... 36 9.1 Introductory Remarks .................................................................................. 36 9.2 Basic Probability Theory Assuming Simple Random Sampling ................. 36 9.3 Basic Probability Theory with Complex Survey Design Effects ................ 37 9.4 Sakernas Estimation Formulae .................................................................... 39 9.5 CMRSS District Level Estimation............................................................... 40 9.6 CMRSS District Change Estimation ........................................................... 41 9.7 CMRSS Provincial Level Estimation Formulae .......................................... 44 9.8 CMRSS National Estimation Formulae ...................................................... 44 9.9 Computation of Other Statistics Relating to Changes at District Level ...... 45 10 Analytical Framework ............................................................................................. 47 10.1 Aims of Analysis ......................................................................................... 47 10.2 CMRSS Data ............................................................................................... 48 10.3 Grouping of Indicators ................................................................................. 49 10.4 Analysis Procedures for National and Provincial Data ............................... 49 10.5 Analysis Procedures for District Data ......................................................... 51 10.6 Distinguishing Crisis Effects from Other Effects ........................................ 56 10.7 Data Quality Considerations ........................................................................ 57 11 Collection and Analysis of Health Data .................................................................. 57 11.1 Data Sources and Collection........................................................................ 57 11.2 Data Capture and Preparation ...................................................................... 57 11.3 Data Preparation .......................................................................................... 57 11.4 Data Analysis ............................................................................................... 58 12 Dissemination and Use of Results........................................................................... 59 12.1 Transfer of Outputs to Clients ..................................................................... 59 12.2 Response Identification ............................................................................... 59 12.3 Dissemination .............................................................................................. 59 PART B: RESULTS OF ANALYSIS ........................................................................... 61 13 General Effects of Crisis: Context for Analysis ...................................................... 61 14 Impacts of Crisis at National Level ......................................................................... 62 14.1 Labour Impacts ............................................................................................ 62 14.2 Household Economics ................................................................................. 65 14.3 Coping Strategies ......................................................................................... 67 15 Impact of Crisis at Provincial and District Levels .................................................. 71 15.1 Impact at Provincial Level ........................................................................... 71 15.2 Impact of Crisis at District Level................................................................. 72 CMRS Detailed Report Page 2 of 91 PART C: EVALUATION AND INSTITUTIONALIZATION ..................................... 73 16 CMRS Evaluation ................................................................................................... 73 16.1 Evaluation of Processes ............................................................................... 73 16.2 Quality Considerations ................................................................................ 74 16.3 Summary of Strengths and Weaknesses ...................................................... 76 17 Future Crisis Monitoring and Response System ..................................................... 77 17.1 Introduction.................................................................................................. 77 17.2 Objectives, Clients and Concepts ................................................................ 77 17.3 Previous Crisis Monitoring Experiences ..................................................... 80 17.4 System Design Considerations .................................................................... 82 17.5 Sakernas and Susenas as Potential Data Sources ........................................ 83 17.6 Data Analysis and Dissemination ................................................................ 85 17.7 Crisis Monitoring by Phase ......................................................................... 85 Annex I: Reference Documents ...................................................................................... 88 CMRS Detailed Report Page 3 of 91 Abbreviations, Acronyms and Terminology As applied to an indicator and district; a movement that is in the Adverse direction of a district being more at risk, for example, reduction movement in average hours worked by head of household. (Also referred to as a negative change.) As applied to a district, as evidenced by an adverse movement in At risk an indicator due to the effects of a crisis, caused by, for example, a global financial crisis or a tsunami. Australian Government agency responsible for managing AusAID Australia's overseas aid program Badan Perencanaan Pembangunan Nasional - Bappenas National Development Planning Agency. Baseline Same as benchmark. The value of an indicators obtained from a more reliable source Benchmark than the CMRSS, for example Sakernas, to which a value obtained from the CMRSS can be compared. As applied to an indicator, meaning having two possible values; Binary particular case of a categorical indicator. BPS Badan Pusat Statistik – Central Bureau of Statistics. As applied to an indicator, meaning having two or more possible Categorical values but not quantitative; aggregation over a population leads to counts and proportions. Change As applied to an indicator, a measure of difference over time. CMRS Crisis Monitoring and Response System. CMRSS Crisis Monitoring and Response System Survey. Effect of shock or more gradually deteriorating situation. Can be caused by a natural event such as disease or earthquake, or be human-related, for example, financial or political turmoil, or Crisis conflict. It can develop suddenly, following a shock, or over a longer period as conditions gradually worsen, for example, as result of a prolonged drought. Data item Same as indicator. Dichotomous (As applied to an indicator) same as binary. Dinas District health centre. Kabupaten - in this document the term will also be used to refer District to the kota, the urban equivalent of a district. GEC Global economic crisis beginning in 2008. GOI Government of Indonesia. CMRS Detailed Report Page 4 of 91  A district in crisis is defined as one adversely affected by the In-crisis GEC or other crisis, as determined by adverse changes in a number of indicators Characteristic of interest belonging to members of a population Indicator of interest. As applied to an indicator, the value of the indicator for given Level time reference period or point. Lot quality assurance sampling – procedure for taking small samples from subpopulations of a large population, with the aim of measuring a binary characteristic of interest and deducing from the sample results whether or not the proportion in each LQAS subset falls short or meets a specified target. For example: the population could be children within country; the subpopulations the children within each province: the characteristic, vaccination against smallpox: and the target 80%. Negative As applied to an indicator and district; the same as an adverse change movement. Group of entities of interest; in CMRSS context, populations of Population interest are persons, households, census blocks, districts, and provinces. PPS Probability proportional to size (sample) Puskesmas Pusat Kesehatan Masyarakat - community health centre. As applied to an indicator, meaning having value set comprising Quantitative a range of integers or real numbers. For an indicator and a district, indicating whether or not district is considered to be at risk based on the value of the indicator. Risk Flag For a district, indicating whether or not the district is considered to be in crisis. Survei Angkatan Kerja Nasional – national labor force survey Sakernas conducted by BPS. Sudden, unanticipated event, such as current financial crisis or Shock tsunami, that is not considered a part of a business trend or cycle, or seasonal, trading day or random effect. SMERU Independent institution for research and public policy. Survei Sosial Ekonomi Nasional - National Socio-Economic Susenas Survey conducted by BPS. Variable Same as indicator. Crisis and Vulnerability Monitoring and Response System VSMRS (possible successor to CMRS) World Bank staff and consultants involved in CMRS WB Team development. CMRS Detailed Report Page 5 of 91 1 Purpose and Content of Document This document contains a detailed description of the design, development, and operations of the Crisis Monitoring and Response System (CMRS). It covers the period from January 2009, when the CMRS was initiated, until September 2010 when the final CMRS reports (including this one) were completed. In addition to providing a definitive record of CMRS design and operational decisions, the document describes the results of the analyses based on the data collected. The document is intended to be the definitive source of reference material. It should prove particularly useful if and when consideration is given to the design, development and implementation of crisis monitoring system in the future. Part A: CMRS Design, Development and Implementation Chapter 2 describes the origin of the CMRS, its basic objectives and parameters Chapter 3 details how and by whom it was developed and managed. Chapter 4 covers the general aspects of the CMRS design. Chapter 5 describes the sample design. Chapter 6 details the questionnaire design. Chapter 7 documents the field operations - data collection and follow-up Chapter 8 describes capture and processing operations. Chapter 9 describes estimation procedures. Chapter 10 details the framework for analysis of the data collected by the CMRS at national, provincial and district levels. Chapter 11 describes the acquisition and analysis of health data. Chapter 12 refers to dissemination and use of the data for crisis monitoring. Part B: Results of Analyses Chapter 13 summarises the general effects of the crisis as reflected in data from the CMRS and other sources. Chapter 14 summarises the effects of the crisis as reflected in CMRS national level data. Chapter 15 summarises the effects of the crisis as reflected in CMRS provincial and district level data. Part C: Evaluation and Institutionalisation Chapter 16 summarises the key aspects of the design, development and results, and contains an evaluation. Chapter 17 completes the main body of the report, summarising the possibilities for institutionalising a crisis monitoring system. The appendices include additional references. CMRS Detailed Report Page 6 of 91 PART A: CMRS DESIGN, DEVELOPMENT AND IMPLEMENTATION 2 CMRS Initiation and Organisational Structure 2.1 Origin and General Objectives In January 2009, the Government of Indonesia (GOI), through its National Development planning Agency (Bappenas), decided it would develop a Crisis Monitoring and Response System (CMRS) to determine the impact of the global economic crisis (GEC) on Indonesia over the course of the following year. The CMRS was to generate data to assess the impact of the crisis and to identify the policy responses appropriate to alleviating the effects of the crisis on the poor and vulnerable. The ultimate objective was to ensure timely and appropriate policy responses in those districts identified as adversely affected as result of the global economic crisis (or any other shock). The output was to be an operational CMRS, comprising a data collection system and analyses that facilitated identification of appropriate responses. 2.2 Terminology In general terms, a crisis is a time of severe difficulty or danger. It is the effect of shock or more gradually deteriorating situation. It can be caused by a natural event such as disease or earthquake, or be human-related, for example caused by financial or political turmoil, or conflict. It can develop quickly, following a shock, as in the case of a tsunami, or over a longer period as conditions gradually worsen, for example, as result of a prolonged drought. The particular crisis that initiated the CMRS was the GEC. The effects of a crisis can be confined to a certain region, for example affected by flooding, landslide, or earthquake, or felt country-wide, such as food or fuel price increases. (Individual households may experience a crisis, for example resulting from death of the main income earner, but this sort of crisis is not being considered here.) For the CMRS a district in crisis was defined as one adversely affected by the crisis as determined by changes in a number of relevant indicators. When a district was identified as in crisis it was assigned a red risk flag. The households in such a district were referred to as stressed or struggling. An adverse movement (or negative change) in an indicator in a district was defined as one that was in the direction of the district being more at risk. Examples of adverse changes are reduction in rice consumption, reduction in average hours worked, and increase in unemployment. If an adverse movement in an indicator in a district was sufficiently large it led to the assignment of a red risk flag for that indicator for that district. A slightly smaller but still significant adverse movement resulted in assignment of an orange risk flag for that indicator for that district. A sufficient number of risk flags resulted in the district being declared in crisis. CMRS Detailed Report Page 7 of 91 2.3 Scope Although the CMRSS collected levels (i.e., current values) of indicators, movements (i.e., changes in level) of indicators were the main goal. For some indicators, for example employment, benchmark levels were available for earlier periods from more reliable sources. They were used as a basis for computing changes relative to a benchmark. However, there were no benchmark estimates of quarterly change for any indicator. Thus, the level of change was considered sufficiently large to result in the assignment of a red or orange risk flag was determined empirically. A distinction was drawn between crisis and vulnerability. Vulnerability refers to the propensity of a household or group of households to sink below the poverty level or to enter a period of extreme hardship. It is a state in which a household’s current situation is fine, but the household is on the borderline of hardship and can easily be pushed there. Vulnerability is a structural/chronic condition, typically measured in terms of low average levels of indicators such as household income, household assets, and household employment. A vulnerable district is more likely to become in crisis following a shock, but this is not inevitably the case as a district that is not vulnerable can become in crisis, and conversely. In summary, the term in crisis was used to describe a district that experienced negative changes in a number of relevant indicators over a recent period whereas a vulnerable district was defined as one that had generally low levels of indicators. Vulnerability is the subject of annual and semi-annual surveys, It was not the subject of the CMRS. 2.4 Funding and Budget Funding for the CMRS was provided by AusAID. The level of funding determined the scale of the CMRS. The budget was sufficient to support three rounds of a survey collecting data via a relatively short questionnaire from about 15,000 households. 2.5 Data Requirements: Content, Frequency and Timing Early in the design phase, a group of stakeholders came together for at a two day workshop (March 31– April 1, 2009). The general role of the CMRS and the specific data requirements were discussed. A program was proposed that comprised three rounds of quarterly data collection with initial results being available within a month. Subsequently this target period was extended to six weeks. One of the main workshop outputs was a comprehensive list of indicators that could be used for crisis monitoring purposes. As the list was far too long for practical implementation, it was reviewed and refined by the WB Team over the following months. The result was a shorter and operationally more feasible list (copy embedded below) which formed the basis for decided what data were required in addition to those already available from surveys and administrative sources List of Crisis Indicators (as of April 30, 2009) Crisis Indicators April 2009 CMRS Detailed Report Page 8 of 91 2.6 Existing Sources of Data Two existing, regular sources of data were identified as potentially useful for crisis monitoring:  data produced by BPS, in particular data from the labour force survey (Sakernas) and the national socio-economic survey (Susenas); and  administrative data available from other government departments and agencies, in particular, data related to the provision of health related government services provided by community (Puskesmas) and district (Dinas Kesehatan) health centres. Sakernas and Susenas data were (and are) semi-annual and thus not of sufficient frequency to meet CMRS data needs but useful in providing benchmarks. Health data were (and are) collected monthly and hence were seen as a very useful potential source. 2.7 CMRS Components The indicators required for monitoring that were not available from one or other of these sources became the target of a new quarterly household survey referred to as the CMRS Survey (CMRSS). The distinguishing features of the CMRSS were that it was quarterly, that it collected indicators not otherwise available, and that was conducted by the BPS under contract in conjunction with the existing BPS semi-annual labour and employment survey (Sakernas) but with much faster processing. The CMRSS was the first, and major, component of the CMRS. The second component of the CMRS was the collection of health data from the community (Puskesmas) and district (Dinas Kesehatan) health centres. 2.8 Complementary Qualitative Crisis Monitoring and Analysis Complementary to the CMRS was a qualitative analysis by conducted by SMERU. The intention was that, via regularly conducted rapid assessments, qualitative crisis monitoring would provide a deeper understanding of the phenomena causing the changes. In particular, the assessments would confirm (or not) the CMRS results in specific areas, would provide the possible causes and trends for observed changes in the quantitative data, and would help in determining the potential effectiveness of various policy responses. The main objective of combining the quantitative and qualitative crisis monitoring is to increase the trustworthiness of the results from both types of assessments. Such an increase may occur in one or more of four possible ways.  Internal validity or credibility. This is linked to the “truth