Measuring the quality of education and health services : the use of perception data from Indonesia

WPS5033 Policy Research Working Paper 5033 Measuring the Quality of Education and Health Services The Use of Perception Data from Indonesia Basab Dasgupta Ambar Narayan Emmanuel Skoufias The World Bank Poverty Reduction and Economic Management Network Poverty Reduction Group August 2009 Policy Research Working Paper 5033 Abstract Satisfaction surveys offer a potentially convenient facility and the household's expected standard for quality, and cost-effective means for measuring the quality of which is influenced by its socioeconomic characteristics. services. However, concerns about subjectivity and The findings show that, after taking into account the selection bias impede greater use of satisfaction data. expectations of households, reported satisfaction does This paper analyzes satisfaction data about health and vary significantly with objective indicators of quality. The educational services from the 2006 second round analysis also checks for possible selection bias affecting of the Governance and Decentralization Survey in the results by using a two-stage selection model. The Indonesia to assess whether satisfaction data can serve model yields policy-relevant insights into the aspects of as reliable indicators of quality, despite dubiously service delivery that most affect satisfaction, highlights high levels of reported satisfaction. The authors use differences across rich and poor districts, and shows that an expectation disconfirmation model that posits that once the role of expectations has been factored in, the a user's satisfaction with a facility improves with the variation in user satisfaction can be highly informative for (positive) difference between the actual quality of the policymakers and researchers alike. This paper--a product of the Poverty Reduction Group, Poverty Reduction and Economic Management Network --is part of a larger effort in the group to analyze poverty and monitor and evaluate the effectiveness of poverty reduction programs. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at eskoufias@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team  Measuring the Quality of Education and Health Services: The Use of Perception Data from Indonesia Basab Dasgupta Ambar Narayan Emmanuel Skoufias The World Bank Keywords: Satisfaction Surveys; Perception Measures; Citizen Report Cards; Community Score Cards; Decentralization; Service Delivery; GDS-2; and Governance. Corresponding Author: Emmanuel Skoufias, The World Bank, Mail Stop MC4-415, 1818 H Street NW, Washington, DC 20433, USA. tel: (202)458-7539. fax: (202)522-3134. e-mail: eskoufias@worldbank.org. Acknowledgments: The authors are grateful to Elizabeth Bryan for her excellent research assistance and to Susan Wong, Claudia Rokx, and Javier Luque for comments in earlier drafts of this paper. Kai Kaiser, Blane Lewis, Emmanuel Jimenez, and Daan Pattinasarany also provided valuable suggestions. The findings, interpretations, and conclusions in this paper are entirely those of the authors and do not reflect the views of the World Bank. I. Introduction Can satisfaction-related questions be valuable in measuring the quality of public services, specifically in health and education? In examining this question, the paper develops a model for identifying the range of factors that influence satisfaction with services among users and examines how these determinants of satisfaction may vary by the type of service and economic status of regions within a country.1 Measuring the quality of public services (or changes in their quality over time) has become increasingly important, particularly to monitor or evaluate the impact of fundamental reforms in service delivery, such as decentralization. This has resulted in a range of initiatives attempting to measure "performance" indicators in service delivery. Many of such indicators are conceptually complex and expensive to collect information on. Satisfaction surveys can offer a cheaper and more convenient alternative, but only if they can be shown to have information content that is meaningful to measure performance in service delivery. In recent years, significant progress has been made in the realm of public service delivery measurement (see Amin, Das and Goldstein, 2007 for an overview of these instruments). An increasing number of the tools developed include subjective instruments that gauge citizen perceptions. These encompass citizen report cards, community scorecards, facility exit polls, and citizen satisfaction surveys. We focus on one type of subjective tool for gauging citizen perceptions, namely citizen satisfaction with services after Indonesia's "Big Bang" decentralization in 2001, which 1This paper draws significantly from a companion paper by Amin, Dasgupta, and Skoufias (2008) that presents a detailed analysis of satisfaction with health services in Indonesia, using the same dataset. It is also complementary to the study of Lewis and Pattinasarany (2009) focusing on satisfaction with educational services but abstracting from the issue selection bias. 2 transformed a highly centralized government to one that gave broad autonomy to the regions in most domains. We use nationally representative data on health and education services gathered in 2006 from the second round of the Governance and Decentralization Survey (GDS-2), which contains detailed questions related to governance and service delivery. The survey collected data on satisfaction from households and information on objective quality directly from facilities. In addition to the methodological question of whether satisfaction surveys yield useful results on facility quality, the findings of this paper also have more direct implications for policymakers in Indonesia. Our analysis of the determinants of satisfaction among service users suggests that users value certain dimensions of "quality" in public services (health and education) more than others, and that these determinants of satisfaction vary between rich and poor districts. This information is likely to be useful to inform policies engaged to improve basic services, well beyond what generic and aggregated measures of satisfaction typically provide. Section II explains the motivation for this study, examining the limitations of satisfaction surveys and outlining some of the concerns this paper seeks to address. Section III describes the dataset used for our analysis and the model used. In Section IV, we describe the models and the results on the determinants of satisfaction with both health and education facilities. Section V discusses the implications of our results for policy and the design of satisfaction surveys. 3 II. Why do satisfaction surveys merit study? Satisfaction surveys merit study for a number of reasons.2 Even though perceptions of citizens are imperfect indicators of quality of services, satisfaction surveys have considerable appeal as a practical way of measuring the impact of governance reform and decentralization. While other tools have been developed to collect more objective information on service quality (e.g. facility surveys, public expenditure tracking surveys),3 they are typically more time and labor intensive than satisfaction surveys. Thus the latter can be a quick and easy way for policymakers to measure the impact of governance reforms on government performance, particularly for sectors where measurement of service quality is not easy, provided citizen satisfaction is closely correlated with the actual quality of services. Even as satisfaction surveys are increasingly being used to measure the impact of governance reforms,4 there is little consensus on whether citizens' satisfaction reflects the actual quality of services satisfaction surveys. More research is therefore merited on the question of (i) whether data from perception surveys are useful in measuring quality of services, and (ii) if there is information content, how should such data be used and interpreted to measure quality of services. Moreover, understanding what factors influence citizen satisfaction is crucial in order to evaluate the impact of decentralization. In the typical model linking decentralization to 2 See Amin et al (2008) for a more detailed discussion 3 Facility surveys are used to directly measure the quality of infrastructure and resources. An excellent example is found in Banerjee et al (2004)