Indonesia's PNPM Generasi program : interim impact evaluation report

59567 January 2010 Indonesia's PNPM Generasi Program: Interim Impact Evaluation Report Benjamin A. Olken, M.I.T. Department of Economics Junko Onishi, The World Bank Susan Wong, The World Bank THE WORLD BANK Jakarta Stock Exchange Building Tower II/12th Fl. Jl. Jend. Sudirman Kav. 52-53 Jakarta 12910 Tel: (6221) 5299-3000 Fax: (6221) 5299-3111 Printed in January 2010. This volume is a product of staff of the World Bank. The findings, interpretations, and conclusions expressed herein do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement of acceptance of such boundaries. Indonesia's PNPM Generasi Program: Interim Impact Evaluation Report Benjamin A. Olken, M.I.T. Department of Economics Junko Onishi, The World Bank Susan Wong, The World Bank  Table of Contents Acknowledgments v Glossary vi Executive Summary 7 1. Introduction 11 1.1 Background 11 1.2 The Generasi project 12 1.3 Experimental Design 17 1.4 Survey Design and Implementation 19 2. Evaluation Methodology 23 2.1 Regression Specifications 23 2.2 Balance Tests 25 3. Main Results 27 3.1 Impact on the Twelve Main Health and Education Indicators 27 3.1.1 Overall effects 27 3.1.2 Robustness Tests 30 3.1.3 Regional Differences 32 3.2 Impact on Long-Term Final Outcomes 34 3.2.1 All Provinces 34 3.2.2 Regional Breakdown 36 3.3 Impact on Non-Targeted Indicators 37 3.3.1 All Provinces 37 3.3.2 Regional Breakdown 38 4. Where Were Generasi Effects Largest? 41 4.1 Areas 41 4.1.1 All Provinces 41 4.1.2 Regional Breakdown 42 4.2 Individuals 43 4.2.1 All Provinces 43 4.2.2 Regional Breakdown 44 4.3 Direct Benefits of Generasi Funds 45 4.3.1 All Provinces 45 4.3.2 Regional Breakdown 46 5. How and Why did the Generasi Project Work? 47 5.1 Changes in Provider Quantities 48 5.1.1 All Provinces 48 5.1.2 Regional Breakdown 48 5.2 Changes in Provider Inputs 48 5.2.1 All Provinces 49 5.2.2 Regional Breakdown 49 5.3 Changes in Provider Effort 49 5.3.1 All Provinces 49 5.3.2 Regional Breakdown 50 5.4 Changes in Community Effort 51 Indonesia's PNPM Generasi Program: ii Interim Impact Evaluation Report  5.4.1 All Provinces 52 5.4.2 Regional Breakdown 52 5.5 Quantities from Provider Data 53 5.5.1 All Provinces 53 5.5.2 Regional Breakdown 54 6. Discussion 55 7. Policy Implications and Conclusion 59 References 61 Annex 63 Annex I: Randomization and Implementation of Generasi in 2007 63 List of Tables Table 1. Performance metrics and weights 16 Table 2. Questionnaire modules and sample size 20 Table 3. Generasi implementation and randomization results 64 Table 4. Baseline regressions, 12 main indicators * 65 Table 5. Baseline regressions, long-term final outcomes 66 Table 6. Results for main indicators, all provinces (baseline as control variable) 67 Table 7. Results for main indicators, all provinces (robustness to alternative specifications) 69 Table 8. Results for main indicators, Java (baseline as control variable) 70 Table 9. Results for main indicators, East Nusa Tenggara (baseline as control variable) 72 Table 10. Results for main indicators, Sulawesi /Gorontalo (baseline as control variable) 74 Table 11. Results for final outcomes, all provinces (baseline as control variable) 76 Table 12. Results for final outcomes, Java (baseline as control variable) 77 Table 13. Results for final outcomes, East Nusa Tenggara (baseline as control variable) 78 Table 14. Results for final outcomes, Sulawesi /Gorontalo (baseline as control variable) 79 Table 15. Results for non-targeted outcomes, all provinces (baseline as control variable) 80 Table 16. Results for non-targeted outcomes, Java (baseline as control variable) 82 Table 17. Results for non-targeted outcomes, East Nusa Tenggara (baseline as control variable) 84 Table 18. Results for non-targeted outcomes, Sulawesi /Gorontalo (baseline as control variable) 86 Table 19. Results for main indicators, all provinces (interaction with pre-period level of outcome variable, model 1) 88 Table 20. Results for main indicators, all provinces (interaction with pre-period level of outcome variable, model 2) 89 Table 21. Results for main indicators, Java (interaction with pre-period level of outcome variable, model 1) 90 Table 22. Results for main indicators, Java (interaction with pre-period level of outcome variable, model 2) 91 Table 23. Results for main indicators, East Nusa Tenggara (interaction with pre-period level of outcome variable, model 1) 92 Table 24. Results for main indicators, East Nusa Tenggara (interaction with pre-period level of outcome variable, model 2) 93 Table 25. Results for main indicators, Sulawesi (interaction with pre-period level of outcome variable, model 1) 94 Table 26. Results for main indicators, Sulawesi (interaction with pre-period level of outcome variable, model 2) 95 Table 27. Results for main indicators, all provinces (split by per-capita consumption quintile, model 1) 96 Table 28. Results for main indicators, all provinces (split by per-capita consumption quintile, model 2) 97 Table 29. Results for main indicators, Java (split by per-capita consumption quintile, model 1) 98 Indonesia's PNPM Generasi Program: Interim Impact Evaluation Report iii  Table 30. Results for main indicators, Java (split by per-capita consumption quintile, model 2) 99 Table 31. Results for main indicators, East Nusa Tenggara (split by per-capita consumption quintile, model 1) 100 Table 32. Results for main indicators, East Nusa Tenggara (split by per-capita consumption quintile, model 2) 101 Table 33. Results for main indicators, Sulawesi (split by per-capita consumption quintile, model 1) 102 Table 34. Results for main indicators, Sulawesi (split by per-capita consumption quintile, model 2) 103 Table 35. Direct benefits, all provinces (baseline as control variable) 104 Table 36. Direct benefits, Java (baseline as control variable) 105 Table 37. Direct benefits, East Nusa Tenggara (baseline as control variable) 106 Table 38. Direct benefits, Sulawesi /Gorontalo (baseline as control variable) 107 Table 39. Results for service provider quantities, all provinces (baseline as control variable) 108 Table 40. Results for service provider quantities, Java (baseline as control variable) 109 Table 41. Results for service provider quantities, East Nusa Tenggara (baseline as control variable) 110 Table 42. Results for service provider quantities, Sulawesi /Gorontalo (baseline as control variable) 111 Table 43. Results for service provider inputs, all provinces (baseline as control variable) 112 Table 44. Results for service provider inputs, Java (baseline as control variable) 113 Table 45. Results for service provider inputs, East Nusa Tenggara (baseline as control variable) 114 Table 46. Results for service provider inputs, Sulawesi /Gorontalo (baseline as control variable) 115 Table 47. Results for service provider efforts, all provinces (baseline as control variable) 116 Table 48. Results for service provider efforts, Java (baseline as control variable) 117 Table 49. Results for service provider efforts, East Nusa Tenggara (baseline as control variable) 118 Table 50. Results for service provider efforts, Sulawesi /Gorontalo (baseline as control variable) 119 Table 51. Community effort, all provinces (baseline as control variable) 120 Table 52. Community effort, Java (baseline as control variable) 122 Table 53. Community effort, East Nusa Tenggara (baseline as control variable) 124 Table 54. Community effort, North Sulawesi/Gorontalo (baseline as control variable) 126 Table 55. Results for service-provider-based quantities, all provinces (baseline as control variable) 128 Table 56. Results for service-provider-based quantities, Java (baseline as control variable) 133 Table 57. Results for service-provider-based quantities, East Nusa Tenggara (baseline as control variable) 139 Table 58. Results for service-provider-based quantities, Sulawesi /Gorontalo (baseline as control variable) 144 Table 59. Detail of mortality vis-