Report No. 15300-CO Colombia Paving the Way for a Results-Oriented Public Sector January 28, 1997 Country Operations Division I Country Department III Latin America and Caribbean Region Document of the World Bank Acting Vice President: Gobind T. Nankani Director : Paul Isenman Division Chief : Dan Morrow Task Manager : Ernesto May Table of Contents EXECUTIVE SUMMARY .................................................................. vii Design of SYNERGY .................................................................. vii Linkage with Civil Society ................................................................... viii Development of Institutional Capacity .................................................................. ix Strategic Policy Formulation and Incentive Framework ............................................x 1. EVALUATION - A CATALYST FOR PUBLIC MANAGEMENT REFORM ............1 INTRODUCTION ..................................................................1I DEFINING THE STRATEGY: FRAMEWORK, OPPORTUNITIES AND RISKS .....2 Results-Oriented Public Sector Management (ROPM): The Framework ...................................................................2 Opportunities ...................................................................2 Risks ...................................................................4 THE PROPOSED STRATEGY ...................................................................5 2. SYNERGY - A REAL MANAGEMENT TOOL ............................................................6 THE PROPOSED EVALUATION SYSTEM ................................................................6 INDICATIVE PLANS ...................................................................8 Coverage: Going Beyond the National Development Plan (NDP) ............................ 8 Type of Performance Indicators ...................................................................8 Coverage of Performance Indicators .................................................................. 10 Data Collection Procedures .................................................................. 10 Setting Performance Targets .................................................................. 11 Use of Performance Information .................................................................. 12 PROGRAM PERFORMANCE UNITS .................................................................. 13 STRATEGIC EVALUATIONS .................................................................. 14 POLICY/PROGRAM ANALYSIS (EX-ANTE EVALUATION) ................................. 15 EVALUATING THE NATIONAL DEVELOPMENT PLAN ...................................... 15 LIMITATIONS OF EVALUATION .................................................................. 17 OTHER ISSUES RELATED TO SYNERGY .............................................................. 17 3. LINKAGE WiTM CIVIL SOCIETY - RESULTS FOR PEOPLE .............................. 19 INTRODUCTION .................................................................. 19 THE CENTER OF GRAVITY OF EVALUATION: FROM INSTITUTIONS TO COMMUNITIES .................................................................. 20 CIVIL SOCIETY: A PARTNER IN EVALUATION .................................................. 22 INITIAL STEPS TOWARDS IMPLEMENTATION ................................................... 23 4. BUILDING CAPACrTY - DEFINING A CONSTRUCTIVE LEARNING PROCESS FOR IMPLEMENTATION .................................................................. 24 RAISING INSTITUTIONAL CAPACITY - A TWO PRONG STRATEGY ................ 24 BUILDING CAPACITY TO EVALUATE AND USE THE RESULTS TO IMPROVE SERVICES ............................................................... 24 LEARNING BY DOING AND MOTIVATING BY SHOWING RESULTS ............... 25 Capacity Building in Central Government Ministries through Performance Measurement .............................................................. 26 A Senior Leadership Network in Central Government ............................................ 26 Local Government Best Practice Workshops .......................................................... 27 Dissemination of Experiences and Information across Local Governments .............................................................. 28 Rewarding Best Practices ............................................................... 29 5. PUBLIC MANAGEMENT FRAMEWORK - PROVIDING AN ENABLING ENVIRONMENT FOR A RESULTS-ORIENTED PUBLIC SECTOR ..................... 30 INTRODUCTION ................................................................... 30 STRATEGIC POLICY FORMULATION .............................3..................1........ . 31 Unbiased Choice of Policy Instruments ............................. 31 Macroeconomic Control and Microeconomic Performance ..................................... 32 MANAGEMENT INCENTIVES .............................................................. 35 Clarity in the Assignment of Responsibilities and Accountabilities ........................... 35 Budget Incentives ............................................................... 37 Personnel Management .............................................................. 38 BIBLIOGRAPHY ................................................................... 40 ANNEXES ANNEX 1 - DEFINITION OF KEY PERFORMANCE MEASUREMENT TERMS ..44 ANNEX 2 - IMPLEMENTATION OF PERFORMANCE MEASUREMENT: MINISTRY OF ENVIRONMENT - POSSIBLE PILOT PROJECT ..................... 47 1: Client identification and data use .................................................................. 47 2: Framing of issues to be addressed ................................................................. 47 3: Identification and analysis of currently available service delivery data ..................... 48 4: Identification of pilot municipalities .................................................................. 48 5: Preliminary design of baseline survey .................................................................. 48 6: Pilot testing of baseline instruments .................................................................. 48 7: Field work ................................................................. 49 8: Data entry, preliminary analysis and local discussions ............................................. 49 9: Definite analysis, development of performance targets, and publication of baseline information .................................................................. 49 10: General application of surveys and capacity building .............................................. 49 Summary of Deliverables .................................................................. 50 ANNEX 3 - SELECTED ISSUES IN RESULTS-ORIENTED PUBLIC SECTOR MANAGEMENT ............................................................... 51 INTRODUCTION ............................................................... 51 MANAGING FOR RESULTS IN THE PUBLIC SECTOR ....................................... 51 STRATEGIC POLICY INTEGRATION ............................................................... 52 Clarifying the Role of Government ............................................................... 55 MANAGEMENT SYSTEMS, STRUCTURES AND INCENTIVES ......................... 55 Alignment of the Management System with Key Objectives of Government ............. 55 Clarifying the Different Interests of Government in its Various Activities ................. 56 Structuring the Functions of Government to Promote Performance through Allocating Functions, Roles and Responsibilities ...................................................... 56 Specification of the Performance Objectives of Government Institutions ......... ......... 57 Management Systems and the Softer Aspects of Management ................................. 58 Financial Management ............................................................... 59 Information Systems ............................................................... 61 Institutional Capacity ............................................................... 61 Personnel Management ............................................................... 62 BOXES 1: Service Delivery Survey ................................................................. 20 2: Best Practice from Medellin - Contracting Out Education ................................................. 28 DIAGRAMS 1: Results-Oriented Public Sector Management Framework ....................................................3 2: Governance and Public Sector Management ................................................................. 53 3: Managing the Results in the Public Sector ................................................................ 53 4: The Financial Management Process ................................................................ 59 FIGURES 1: The National Evaluation System .................................................................6 2: Performance Indicator Sequence Chart .................................................................9 ACRONYMS AND ABBREVIATIONS CONPES - National Council for Economic and Social Policies (Consejo Nacional de Politica Econ6mica y Social) DANE - National Statistics Department (Departamento Administrativo Nacional de Estadisticas) DNP - Department of National Planning (Departamento Nacional de Planeaci6n) DRI - Integrated Rural Development Program (Desarrollo Rural de Inversiones) ECOFONDO - Environment Participation Fund (Fondo de Participaci6n en Asuntos Ecol6gicos) GOC - Government of Colombia ICBF - Family Welfare Institute (Instituto Colombiano de Bienestar Familiar) MTEF - Medium-term Expenditure Framework NDP - National Development Plan NGO - Non-Governmental Organization PPU - Program Performance Unit (Unidad de Gesti6n) ROPM - Results-oriented Public Sector Management SDE - Special Division for Evaluation SDS - Service Delivery Surveys SINPRO - Information System on Standards and Processes SYNERGY - National Evaluation System of Public Sector Performance (Sistema Nacional de Evaluaci6n de Resultados de la Gesti6n Publica) During the past three years, the Special Division for Evaluation (SDE) in the Department of National Planning (DNP) has been working in the definition and implemention of a National Evaluation System of Public Sector Performance (SYNERGY) in Colombia. In mid-1995, DNP requested the Bank to review the system and define proposals to strengthen SDE's strategy to pave the way for a results-oriented public sector in Colombia. The report, product of the collaborative effort between DNP and the World Bank, presents the findings of the Bank team. The World Bank team was led by Ernesto May (LA3C1) and integrated by Malcolm Holmes (PSP), Petter Langseth (EDIRP), Graham Scott (consultant), Harry P. Hatry (consultant) and Neil Andersson (consultant). The counterpart team in the DNP was the Special Division for Evaluation (SDE) headed by Sylvia Escovar G6mez and more recently by Miguel Gandour, and integrated by Pablo Abitbol Piiieiro (who traveled to Washington the last week in August 1995 to present to the Bank team the proposed National Evaluation System of Public Sector Performance), Gloria G6mez, Maria Cristina Obreg6n, Juan Fernando Robayo, Rafael Lopez, Adriana Sanjuan Galvis, Monique Mekler, Julian Marifio and Fernando Pieschac6n. The Bank team visited Colombia during the second week of October 1995. The report is based on the findings of the mission and background material prepared by SDE. Initial conclusions of the mission were presented in Colombia on the First Seminar on Public Sector Performance Evaluation held in Santafe de Bogota 4-5 December, 1995. The draft report was discussed in a one-day workshop held in Bogota on December 9, 1996, with the participation of representatives from DNP, Ministry of Finance, Office of the President, Administrative Department of Public Office, Ministry of Agriculture, Ministry of Defense, National Police, FIS and INVIAS. The Bank team benefited greatly from preparatory work done by Tony Dale (consultant - collaborating with Graham Scott) and from the support of the Resident Mission in Colombia provided by Natalia G6mez de Pizano. The peer reviewers are Desmond F. McCarthy (DECVP) and Sanjay Pradhan (PRDPE). Report processing was handled by Neil Wirth and Kathy Scalzulli. EXECUTIVE SUMMARY The National Evaluation System of Public Sector Performance (SYNERGY) now being put in place in Colombia--by the Special Division for Evaluation (SDE) at the Department of National Planning (DNP)--has the potential to revolutionize the public sector, turning it from a more inward looking, process oriented body that in ocassions restrains economic and social progress, to being a more dynamic, results-oriented partner of the community and private sector in development. Critical to turning this potential into reality are the following four factors: * Design of SYNERGY - Making it a real management tool. * Linkage with Civil Society - Focusing on results for people and leveraging on their energy and resources. * Development of Institutional Capacity - Defining a constructive learning process for implementation. * Strategic Policy Formulation and Incentive Framework - Providing an enabling environment for a results-oriented public sector. The report analyzes each of these components and puts forward proposals to strengthen SDE's strategy to pave the way for a results-oriented public sector in Colombia. It identifies both opportunities and risks. Critical to the success in this endeavour, in moving SYNERGY out of the conceptualization phase to development and implementation, is the management of change: * Cultivating strong support from senior management in central government to deal with the expected resistance to change. * Managing the implementation process, looking for early successes through the use of pilots. * Identifying change-champions for the pilot programs, clearly defining areas of responsibility (who is responsible for what results), and resources needed to ensure the required successes. D Developing a learning process where the different pilots will allow decision-makers to adjust and to fine-tune the proposed national evaluation system. * Managing expectations about the impact of SYNERGY both within civil society and government. Design of SYNERGY For success, SYNERGY needs to prove itself as an evaluation system of public entities for public entities. Public managers need to find it a useful management tool. For this to happen, the design of the system needs to be strengthened by addressing, among others, the following issues: * extending the coverage of indicative plans to all entities in the public sector, not just those involved in executing the national development plans; * encouraging entities to develop the full range of indicators they need to manage their programs, including intermediate outcomes, even though DNP will only require a smaller range of indicators for its own purposes; * ensuring a focus on outcomes as the reason for government interventions; Colombia: Paving the Way for A Results-Oriented Public Sector viii * including the indicators developed as part of the related "It's a Deal" program, which are expected to focus on the quality of services delivered, in the indicative plans; * developing new or revised data collection procedures to address the specific needs of the system--data quality control procedures should be implemented; * coordinating target setting with all relevant entities and, through partnership agreements, between all levels of government; * treating initial targets as experimental, given the inherent difficulties in setting them, until some experience is attained; * training managers in the use of performance information, with best practice being shared; * encouraging managers to provide explanatory information along with performance reports, which should be published; * defining selection criteria to identify the programs, projects or entities that should be subject to strategic evaluations each year; * restricting program performance units (PPUs), intended to coordinate entities where more than one contributes to a program, to an advisory and coordination role; - excluding DNP from the project evaluation process--leaving it to the line ministries-- focusing its efforts instead on strategic policy analysis; - assigning sector specialists at DNP to work with the PPUs--as one of their major responsibilites rather than a side duty; and - establishing an Evaluation Advisory Committee to discuss common issues and to resolve common evaluation problems among entities. Linkage with Civil Society As stated by SDE, SYNERGY can only address the government's development objectives by taking into account its results for civil society. At the same time it is essential that the vitality and extensive resources of civil society be harnessed and freed in the cause of development for the people. The proposed involvement of civil society in SYNERGY is mainly through the National Planning Council in the definition of the National Development Plans (NDPs) and the participation of diverse stakeholers--identified by the national and territorial planning councils--in the PPUs. This, however, could easily decay in tokenism. The recommendations of the report are aimed at promoting civil society participation in two ways: * identifying what the results are for individuals and communities; and * being a partner in the evaluation process. Institution or provider based data (e.g. hospital and school records) need to be complemented by data which is community or user based. The report highlights the potential for Service Delivery Surveys (SDS) to be an important component of the evaluation system and to contribute to improving public sector management and service delivery (ensuring client needs are more adequately met). But only part of the challenge in an evaluation system is to obtain the data. Providing feedback to households and clients is a critical link that has often been forgotten. This lack of feedback neglects the potential multiplicative effects of communication with civil society and other stakeholders. The full range of communication techniques should be exploited. Colombia: Paving the Way for A Results-Oriented Public Sector ix While there is a considerable reserve of energy and skills in civil society, this should not be taken as meaning that it has much evaluation capacity. The vision of SYNERGY needs to be extended, and both public and private sectors need to consolidate their efforts. Coordination and working arrangements must be strengthened and streamlined at national, departmental and local levels. The view among government officials that evaluation and links with civil society will be against their own interests must be tackled head on. As an illustrative case of what this working relation with civil society might look like, the government should consider having a central ministry which provides services to the public, and involves all levels of government as well as NGOs, to test the use of Service Delivery Surveys as part of a performance measurement pilot project. Development of Institutional Capacity By promoting a change in culture (results orientation), innovative and responsible leadership and community participation, SYNERGY can generate effective demand for capacity- building thereby creating the missing link to institutional development. In this context, the efforts to enhance institutional capacity for the successful implementation of SYNERGY and results- oriented public management (ROPM) in Colombia should be viewed as part of a broad capacity building process in which the government allows itself, through experience, to: * build capacities to evaluate and use the results to improve policies and services; and l learn by doing while motivating by showing results. As much attention needs to be given to the supply side of technical assistance as the demand side. On the supply side, the GOC will need to ensure training and technical assistance is provided to those responsible for performance measurement and evaluation processes in their entities, as well as those using the evaluation system for managerial purposes. Some ways to provide such training would be to: (i) contract universities or private organizations; (ii) develop VCR tapes, computer-based training software, and even TV programs; and (iii) use internal personnel. There is the need, on the other hand, to ensure there is effective demand to build capacity by central government, local administrations and communities. One way to create such demand is through demonstrations: showing authorities, managers, and personnel that they can reap the benefits of using the proposed evaluation system. As an example of learning by doing the SDS is a potentially powerful tool as are a whole range of evaluation techniques. For central government, as mentioned before, a performance measurement pilot project is proposed. A senior leadership network in central government would be another mechanism for sharing learning as well as providing a focus for the training of senior managers in current approaches to change management and reforming government. For local governments access to information about innovations and best practice in other municipalities and departments may well be the most powerful tool available for capacity strengthening. The report proposes that a series of two-day workshops bringing together top departmental and local government officials be initiated. More generally, a data base and information service on best practice is proposed. Finally, the idea being contemplated by SDE to establish a Best Practice Committee which would recognize entities on the basis of excellent service is endorsed. Colombia: Paving the Way for A Results-Oriented Public Sector x Strategic Policy Formulation and Incentive Framework For broad based success SYNERGY must feed its information into coherent decision- making processes at all levels and into management systems embedded in an overall incentive framework which will encourage and require appropriate actions to attain desired results--a system that demands performance information. To create this enabling environment special attention should be given to the areas of: * strategic policy formulation; and * management incentives. It is essential that the total policy framework retain strategic coherence. The framework must allow for unbiased choices between policy instruments, whereas, in reality, there appears to be a risk of bias towards public expenditure, especially investment. It is also important that there be contestability in policy advice. To support this strategic focus, DNP should use SYNERGY as a vehicle to emphasize its role in strategic policy analysis and integration of advice to government. For government's fiscal decision-making processes, the challenge is to strengthen macroeconomic control while ensuring that resources are allocated according to strategic priorities and resource use is improved in executing agencies. There are pressures on fiscal policy, including from the National Development Plan (NDP). If these are not managed carefully, SYNERGY will be undermined through the uncertainty associated with macroeconomic instability. Over the medium-term, greater coherence and stability in planning and budgeting would be achieved by putting in place (i) a medium-term expenditure framework, (ii) capped funding in the annual budget, (iii) a unified budget, and (iv) a results-oriented financial management information system. The on-going effort to put in place an integrated financial information system needs to be redefined and ensure it adequately responds to results-oriented management. Performance at the entity level is a function not only of the strategic environment but of a host of factors which impact at that level every day. Self-evaluation, the focus of SYNERGY, and performance improvement will only occur if there are incentives for entities, and the people who constitute them, to deliver. This requires that entities have clear objectives and tasks, that they be given the flexibility to achieve them, and that they be held accountable for the use of their authority. SYNERGY can make an enormous contribution to defining objectives, tasks and accountability, and the promise of increased flexibility can be used as a lever to aid implementation of SYNERGY. An important aspect of flexibility concerns budgetary resources. The report proposes that government undertake a review of budgetary procedures with the objective of minimizing the constraints they impose on entity (and local government) performance. The bias should be towards maximizing the freedoms available over the mix of inputs. Similar thinking should apply to the approach to personnel management. Generally speaking, the approach should be to set standards from the center but delegate to local managers the responsibilities for obtaining and managing staff. At the same time, staff appraisal and promotion should be more closely linked to performance. 1. EVALUATION - A CATALYST FOR PUBLIC MANAGEMENT REFORM INTRODUCTION 1.1 In accordance with the 1991 Constitution (Article 343), the National Department of Planning (DNP) is responsible for organizing and coordinating the monitoring and evaluation of public policies, programs and projects as defined in the National Development Plans (NDPs). To fulfill this legal obligation, DNP is developing a national evaluation system. Within DNP, the unit responsible for defining and implementing the system is the Special Division for Evaluation (SDE). 1.2 Putting in place a well-functioning national evaluation system is a major undertaking and presents an enormous challenge for Colombia. Critical to success is to understand that the evaluation system is but one component of a broader management framework to improve performance of the public sector in Colombia. The existence of performance information does not guarantee a performance oriented public sector. It is critical that the management system as a whole functions to achieve improved performance over time, rather than any one of its components--such as the evaluation system--be a benchmark of best practice in some static sense. Without an enabling environment, the evaluation system--no matter how sophisticated and refined it is--is likely to be a waste of resources. From this perspective, the question of how much program evaluation can add to the effectiveness of public administration depends on how it is set up. If it is something done by outsiders over the top of skeptical managers who control vital flows of information it will not be effective. An alternative framework, focusing evaluation on topics of strategic importance, creating strong incentives for high quality analysis and full information disclosure, and integrating it into the ongoing process of policy formulation and resource allocation, could produce significant benefits. 1.3 SDE, having identified the dimensions of the challenge, has made the decision to use evaluation as a tool to promote the desired change in culture towards a results-oriented public sector. It has identified the National Evaluation System of Public Sector Performance (SYNERGY) as the entry point for public management reform. This is based on a conviction that the evaluation process and its results can themselves draw attention to the need for change in the environment. It emphasizes the bottom-up dimension of evaluation--one that ensures both support and demand for improved public sector performance. 1.4 The purpose of this study is to review the proposed evaluation system and the way in which the government intends to put it in place to attain its ultimate objective: improved public sector performance. The study analyzes the design of the system, both at a conceptual and practical level, and identifies existing deficiencies and gaps that need to be corrected. Particular attention is paid to the role of civil society and capacity development. Drawing on the experience in other countries--including developed countries where broad public management reforms have been undertaken and evaluation systems implemented (e.g., Australia, New Zealand, United Kingdom, Canada and United States)--the study supports SDE's efforts to define the broader policy agenda Colombia needs to cover in its pursuit of a performance oriented public sector. It Colombia: Paving the Way for A Results-Oriented Public Sector 2 focuses on the critical areas and steps that need to be taken to put in place a performance management system. In all, the study puts forward proposals to strengthen SDE's strategy to pave the wayfor a results-orientedpublic sector in Colombia. DEFINING THE STRATEGY: FRAMEWORK, OPPORTUNITIES AND RISKS 1.5 As the different actors in the economy are now adjusting to the new realities of an open and more competitive environment, the public sector is facing the challenge of transforming itself into a results-oriented agent. SYNERGY is being developed as an instrument of this transformation. Its success or failure will ultimately be measured by its ability to deliver improved performance. The challenge, therefore, is to ensure that SYNERGY meets this ultimate objective. For this to happen, policy makers will need to follow a strategy that will lead the evaluation system to be: * Designed properly--internally consistent and with the necessary linkages to a results- oriented decision-making process. * Supportive of other government initiatives, particularly in the related areas of connecting with civil society and decentralization. * Implemented as a learning process which will build institutional capacity and pay particular attention to the management of change. * Set within an institutional framework which is results-oriented, including strategic decision-making, budgeting and personnel administration. * Used as a lever to shift as quickly as possible to a comprehensive results-oriented management system. Results-Oriented Public Sector Management (ROPM): The Framework 1.6 The results-oriented public sector management framework being used by SDE can be schematized in Diagram 1. Evaluation, capacity-building, institutional feedback and incentives, improved policy formulation and service delivery, are all part of a virtuous circle in which many society actors--public and private, national, regional and local--become agents of change. By promoting a change in culture (results orientation), innovative and responsible leadership and community participation, SYNERGY can generate effective demand for capacity-building thereby creating the missing link to institutional development and setting the basis for a performance- oriented public sector in Colombia. Opportunities 1.7 The 1991 Constitution seeks to redirect the actions of the state to serve the community promptly and efficiently. It sets out the basis to give the community greater freedom to choose the provider of services--whether it be the state, the private sector or community organizations. It reinforces the rights of the population to demand results in public administration and thereby attempts to create an environment where managers are to take responsibility for the services offered, and be held accountable for their performance. Thus, the Constitution promotes a new Colombia: Paving the Way for A Results-Oriented Public Sector 3 Diagam 1 RESUTLS-ORIENTED PUBLIC SECI'OR MANAGEMNT FRAMEWORK DeveloynSent - lmnpact - Oute- Oco / ~~~~SYNECRGY\ CY lrsdwional | MWt Servi FaTrsli ~~~Based Evaluation Bae Evaluation\ Govenunentg ng I iimO \t - NMuind-se - Sldl - N. NG. O'sll Dqr= ed* ols - Pivyate Se / \ ~~~Institutional Framnewor \ ~~- Decision - Making Processes/ * _ - Budget Mbnagmet - Personnel Managemnit Colombia: Paving the Way for A Results-Oriented Public Sector 4 culture of results-oriented public management (ROPM), in which performance is centered on the general interests of the community and a permanent effort to make better use of resources. 1.8 In accordance with these mandates, the Government of Colombia (GOC) is currently undertaking a number of initiatives which should complement SYNERGY. The Presidential Advisory Council for Public Administration is promoting, among others, several projects aimed at improving administrative practices: (i) rationalization of bureaucratic procedures; (ii) information system on standards and processes (SINPRO); (iii) technical support to public entities provided by an Efficiency Unit with the objective of improving performance; and (iv) a program entitled "It's a Deal", by which national entities are invited to publicly commit themselves to improve service delivery and systematically consult users on whether targets are being met. The Administrative Department of Public Office is advancing a plan to appraise and reform the civil service and personnel administration in order to promote public administration based on results. A new law has recently been enacted, Law 190 of 1995, to fight public service corruption. Finally, the Ministry of Finance--in conjunction with the Department of National Planning and the Central Bank--is developing an Integrated Financial Information System which should enhance macroeconomic and fiscal policy analysis. 1.9 All these initiatives indicate the enormous effort currently being undertaken in Colombia to develop a results-oriented public management framework. SYNERGY can and should take advantage of these efforts. There is the need, however, to properly coordinate and join forces with the different entities advancing these initiatives. These efforts need to be more interrelated, feed from each others' successes, and be implemented in a way that leads to a gradual, but radical, change in the culture of the public sector. Risks 1.10 Several risks need to be recognized and addressed if the transformation now being pursued is to be successful. In particular: o Institutional arrangements currently exhibit significant imbalances between responsibilities, freedom of action and accountability. * The decentralization process is creating significant tensions between central and local governments. Central government agencies may oppose reallocation of responsibilities. * Performance-improvement actions may involve retrenchment in several areas of government, creating resistance from labor unions. GOC may fail to introduce enough flexibility in civil service reform. * Some government officials hold the view that evaluation and links with civil society may be against their own interests. * Lack of clarity between policy instruments and objectives reduces the scope for effective government action. * Macro and micro economic tensions are being exacerbated by current fiscal imbalances. GOC may fail to revise the existing budget law which is creating significant distortions in resource allocation. Colombia: Paving the Way for A Results-Oriented Public Sector 5 Institutional capacity is very limited in many public entities, and management information systems are weak or non-existent. Implementation requirements of the proposed framework may exceed existing capacity, leading the GOC to abort the effort prematurely. 1.11 In implementing SYNERGY, it is critical to identify ways to deal with, and manage, these risks. Indeed, the implementation process of SYNERGY will by itself be critical to ensure its long-term viability and success. In moving SYNERGY out of the conceptualization phase to development and implementation, attention needs to be given to a number of key areas: * Cultivating strong support from senior management in central government to deal with the expected resistance to change. * Managing the implementation process, looking for early successes through the use of pilots. * Identifying change-champions for the pilot programs, clearly defining areas of responsibility (who is responsible for what results), and resources needed to ensure the required successes. * Developing a learning process where the different pilots will allow decision-makers to adjust and to fine-tune the proposed national evaluation system. - Managing expectations about the impact of SYNERGY both within civil society and government. THE PROPOSED STRATEGY 1.12 In defining proposals to strengthen SDE's strategy to pave the wayfor a results-oriented public sector in Colombia, the study focuses on four critical components, which encompass: * Design of SYNERGY - Making it a real management tool. 3 Linkage with Civil Society - Focusing on results for people and leveraging on their energy and resources. * Development of Institutional Capacity - Defining a constructive learning process for implementation. * Strategic Policy Formulation and Incentive Framework - Providing an enabling environment for a results-oriented public sector. 1.13 The following chapters analyze each of these components and provide recommendations on ways to deal with the identified risks and to put in place a performance management system in Colombia. 2. SYNERGY - A REAL MANAGEMENT TOOL THE PROPOSED EVALUATION SYSTEM 2.1 SYNERGY has been conceived as a management tool to improve public administration-- bringing the results of government action in line with set objectives. ROPM is a continuous interaction between planning, execution and evaluation. Evaluation, in turn, constitutes an on- going learning process which provides feedback for planning by analyzing and measuring the results of actions. Essentially, SYNERGY is an evaluation system of public entities for public entities. Self-evaluation stands as the core of the proposed system. SYNERGY has been designed as an instrument to orient public management to the achievement of the specific objectives set by the national development plans and, in that light, to assess public sector performance. Figure 1: The National Evaluation System Programs of the National Government Agencies Development Plan Indicative Plan Program Performance Methodology Units Self - Evaluations Policy NATIONL \Formulation ~~~NATIONAL > SYNERGY DEVELOPMENT Meta Evaluation >Feedback Institutional PLAN Development External Evaluations Budget Allocations Strategic Evaluations Methodology 2.2 The main features of SYNERGY are described in CONPES Document 2790 of 1995 and schematically presented in Figure 1. These features are briefly summarized in the following paragraphs. 2.3 Indicative Plans by Entity. Public entities responsible for executing public expenditure programs identified in the four-year development plan are to define their so-called indicative plans. These plans should include the policies and programs defined in the national development Colombia: Paving the Way for A Results Oriented Public Sector 7 plan, specify the expected outputs and outcomes and the way in which these would be attained, and identify specific indicators of performance to be used for the evaluation process. These four- year plans would be updated annually, defining in turn annual plans with corresponding objectives. The annual plans are then to be used as the basis for yearly on-going and ex-post evaluations of public sector performance. The updated indicative plan of each entity is to be submitted to the related line ministry or administrative department. Line ministries and administrative departments, in turn, need to submit their plans to DNP by December 15. In all, the indicative plan by entity is the principal tool for the self-evaluation system being proposed in Colombia. 2.4 Ex-post Evaluation of Yearly Results by Entity. Planning offices in each entity would be responsible for an annual ex-post evaluation of activities. Based on the annual indicative plans, the self-evaluations would quantify the performance of each entity in attaining stated goals for the year. These evaluations need to be presented by each entity, line ministry and administrative department to DNP by February 15 each year as pre-requisite for their budget allocation request. 2.5 Ex-post Evaluation of Strategic Areas or Programs. On a selective basis, DNP is to conduct in depth ex-post evaluations of performance in selected sectors, programs, projects or entities. Five or six studies are to be conducted each year. 2.6 Annual CONPESDocument and Progress Report to Congress. DNP is to review all evaluations and present a progress report on the National Development Plan. Together with the indicative plans for the coming year, this report would provide the basis to define next year's expenditure program and corresponding budget resource allocations, identify institutional development needs and correct strategic policy formulation. 2.7 Program Performance Units (PPUs). A PPU is to be established for each program in the National Development Plan where interagency coordination is required (though the number and scope of these is not yet clear). These units are expected to provide coordination, cross-entity planning, and evaluation oversight for the programs. They are not expected to manage activities, but rather to establish agreements aimed to attain particular results. The units are to be comprised of senior representatives of entities that have significant roles in the delivery of the program's services (e.g., ministries, executing agencies, departments and municipalities, civil sector). Participation of civil society in the PPUs is proposed to be through representatives selected by the national and territorial planning councils. 2.8 Overall, although improvements can be made, the basic concepts of SYNERGY are excellent. Successful implementation, however, will clearly require careful planning, much work, time and resources. Following are recommendations on actions that should help correct identified deficiencies in the design of SYNERGY and suggestions for their implementation. These recommendations are presented in the following sections: Indicative Plans; Program Management and Evaluation Units; Strategic Evaluations: Policy/Program Analysis (Ex-Ante Evaluation); Evaluating the National Development Plan; Limitations of Evaluation; and Other Concerns about SYNERGY. Colombia: Paving the Way for A Results Oriented Public Sector 8 INDICATIVE PLANS Coverage: Going Beyond the National Development Plan (NDP) 2.9 Not all entities are involved in executing programs of the NDP, and not all the programs of each entity are thereby included. However, all entities should be encouraged, even if they are not required. to prepare indicative plans which include all their 'tignificant" programs--even if not embedded in the NDP. Related action plans should also be comprehensive. This will better contribute to the Government's objective to introduce results-based management throughout the government. On the contrary, managers of activities not covered by indicative plans, and their personnel would likely feel neglected and be given a reduced sense of importance (as has occurred on occasion in the United States when activities were excluded from strategic plans). To ensure compliance with DNP requirements, however, a list of programs and entities for each program that would be expected to prepare indicative plans needs to be formulated. This would provide a clearer picture of what needs to be done. Type of Performance Indicators 2.10 Entities should be asked to include in their indicative plans indicators of inputs, outputs, outcomes, and efficiency (see Annex I for a glossary of terms used in SYNERGY). Indeed, this is the intent of DNP. The major emphasis of entities in the GOC over the long run, however. should be placed on tracking outcomes, not outputs. One should therefore not settle for only output indicators. (Note that included in the definition of "outcomes" are the various service quality characteristics, such as timeliness and accessibility). The citizens of Colombia and the customers of particular public services want successful outcomes and usually are much less, if at all, interested in "outputs." The producers of services and those who provide them with public resources will be interested, of course, in outputs for important management and accountability functions. All public employees should recognize, however, that the reason they are employed is to help the public agency attain successful outcomes. The production of outputs is important, but unless these outputs help lead to improved outcomes, the production of outputs will be meaningless. The program advisory units (PPUs), discussed in the next section, should play a major role by keeping a focus on outcomes. 2.11 Indeed, the GOC would be sending the wrong message to its employees if projects are not evaluated on outcome/quality characteristics but only on outputs, such as the completion of "X" kilometers of roads. Inevitably, however, public agency personnel at all levels will have less control over outcomes than outputs. It is critical that this is explicitly acknowledged. It is important for all those using performance measurement information, to recognize that most data on outcomes do not tell what caused the outcomes. Therefore, the extent to which public officials can be held filly accountable for outcomes is more limited. The program has an important role in producing successful outcomes, but inevitably external factors not under the control of the program may also have affected the outcomes. Just as with managers of a sports team (such as soccer, baseball, basketball, or whatever), knowing the score, and whether the team is winning or losing, is vital information. The scores do not tell why the team is winning or losing; the manager needs to examine other information to help determine why. This is the same for managers of a Colombia: Paving the Way for A Results Oriented Public Sector 9 private business who monitor their bottom line "earnings." And this is the same for public sector managers. Only with information from completed in-depth program evaluations can one provide an assessment of program impacts. If an entity has done an in-depth (e.g., strategic) evaluation of the activity, the entity would be able and should be encouraged to provide impact indicators (defined as indicators of the extent to which the program has caused an outcome). 2.12 In this context, many entities are likely to find it quite useful to distinguish between indicators of "intermediate" outcomes. even if they are unable to provide, at least initially. indicators of "end" outcomes. Intermediate outcomes are events that occur outside the program and are expected to lead to "end" outcomes. For example, the number of persons that complete programs to help them stop smoking is an intermediate outcome indicator, while the number that actually stop smoking is an indicator of an end outcome. Figure 2 provides another example of the relation of the various categories of indicators. The entities may find it useful to prepare this type of "performance indicator sequence chart" for each of their own activities. Figure 2: Performance Indicator Sequence Chart 1' PROGRAM: Industrial Technical Assistance to Reduce Water Pollution PROCESS OUTPUT INTERMEDIATE OUTCOMES Prepare l: Number of Number of firms Percentage of *Amount of advertisement flyers sent that requested firms that adopted pollution campaign offering assistance production changes discharged assistance as a result of the by firms program END OUTCOMES Improved Improved health water quality of the population Developed with the assistance of Juan Fernando Robayo, DNP-SDE. Colombia: Paving the Way for A Results Oriented Public Sector 1 0 Coverage of Performance Indicators 2.13 DNP should be careful not to discourage entities from including as many performance indicators as they believe necessary to manage their programs. DNP may want to control the number of objectives on which indicators are submitted to it. However, DNP should make clear to entities that this is not meant to inhibit their identification of other objectives and associated indicators. Limiting the number of objectives (or indicators) provided to DNP can mean that it does not have a full perspective on the program's performance. Public programs can be complex and involve many different objectives and outcomes. 2.14 Aggregate data on any performance indicator can be misleading and hide much information that is likely to be useful to managers for improving their programs. Managers need to know where their program is being successful and where not successful. Public officials need to know whether differences in performance of some work units, or differences between time periods, is due to differences in the difficulty of the incoming work load. DNP should encourage all entities to break out data on their outcome indicators by important characteristics, such as by demographic characteristics of their customers, by location, by office that served the workload, and by any other characteristics of the workload that make certain workload more difficult to serve than other types of workload. Such informnation should make the performance measurement data much more useful and acceptable to managers. Breakouts by workload difficulty will allay managerial fears that they will be unfairly compared to other units doing similar work but whose workload is significantly less difficult. 2.15 The indicators likely to be included in "It's a Deal" program should be considered for inclusion in the program's indicative plan. If they are important enough to be included in "It's a Deal," and thus are presumably important to customers, they should be important enough to include in the entity's performance measurement system. 2.16 Finally, entities should also be asked to include indicators of potentially important unintended effects of each program--when such effects can be anticipated and should be monitored. For example, certain environmental effects of energy programs should be monitored, such as amounts of pollutants released. Data Collection Procedures 2.17 A major problem for many, and probably most, programs will be the availability of data for indicators that are required and the need for new, or revised, data collection procedures. This is likely to be particularly a problem with data for outcome indicators. DNP and the other entities should not rely solely on readily available data, tempting as this may be. Many of the most potentially useful indicators are likely to require new data collection procedures (or major modifications of existing procedures) and the development of staff capacity to oversee these procedures to assure that reliable and accurate data are obtained. 2.18 For example, systematic surveys of customers are likely to be a major way to obtain reasonably reliable data on various aspects of customer satisfaction with the services they have received and factual information on customer activities and behavior. Few Colombian entities are Colombia: Paving the Way for A Results Oriented Public Sector 11 currently conducting such surveys on a regular basis (so managers can identify progress and whether actions taken are producing desired results). As another example, to collect data on timeliness of services to their customers (a major indicator of service quality for many public services), many entities will probably need to revise their current data systems to record the time requests for services are received, the time the requests have been satisfied, and then calculate indicators of response times. 2.19 The key problem is that these new procedures. can cost non-trivial amounts of money. For example, conducting customer surveys on a regular basis as part of measuring results will likely require outlays of money. Good information costs money. If entities skimp, however, and rely primarily on existing data, the resulting information may not be very useful. 2.20 Many programs already collect some outcome data on their programs that will be useful for their performance measurement systems. For example, considerable data has been available on health such as on illness and mortality--and available on various demographic groups. Some data are available on education, such as on attendance and drop-out rates and from national tests taken voluntarily in the twelfth year of school (but tests are not conducted at other grade levels). Data are available on the condition of national roads and on numbers of accidents, on amount of reported crime, and DANE annually conducts large surveys of households obtaining data on employment and earnings. DANE also has conduced quality-of-life surveys of the Colombian households but on an irregular basis (last in 1992). DANE is currently completing an inventory of data being collected in many sectors which should prove useful for many entities. 2.21 Some of these data are likely to have a timeliness problem. Performance data useful to program managers, a major purpose of SYNERGY, needs to be available frequently and within a reasonably short time after the reporting period. Data that do not become available to the program manager until many months, or years, after the reporting period will be much less useful for program management. Setting Performance Targets 2.22 Target setting for each performance indicator will be difficult, particularly for outcome indicators on which the program managers have had little or no experience (such as indicators from customer surveys). For such indicators, the targets selected should be considered as experimental. Targets should be established by entities after reviewing the resources expected to be available to the program, past experience, and possible external events that are likely to affect the program's ability to produce desired outcomes. For example, expected changes in economic conditions will affect the ability of employment programs to create new jobs. 2.23 When more than one public entity has a role in service delivery, some targets, especially for outcomes, inevitably will be the shared responsibility of these entities. Often, producing desired results will require a "partnership" among more than one entity. The program advisory units (PPUs) will attain agreements on the targets for the entities represented and assure that the targets are properly coordinated. The entity targets should be consistent with targets for cross- cutting indicators to which each entity contributes. Colombia: Paving the Way for A Results Oriented Public Sector 12 2.24 For programs in which service delivery is done by departments and municipalities, these other levels of government should have an important role in selecting the targets for those indicators for which they have a significant responsibility. Together with the corresponding central government entities those levels of government should establish 'Performance Partnerships" (for further discussion of this idea see Chapter 5) and jointly select the performance indicators and targets. Mere representation of these other levels of government on the PPUs does not appear sufficient. These levels of government need to be actively involved with the initial selection of performance indicators and in the annual selection of targets on the indicators. For municipalities this might, for example, be accomplished by seeking formal participation of the Federation of Colombian Municipalities. 2.25 The ministries and their entities should be encouraged to include in their internal performance measurement processes provision for program managers to provide explanatory information along with their performance reports. Such information is likely to be particularly needed when the reported value for a performance indicator falls significantly short, or exceeds significantly, its targets. This will serve two major purposes. This will help users of the information better interpret the data and identify possible reasons for not meeting, or exceeding, targets. The process will also alleviate the fears of managers that the performance data will be misinterpreted (which is quite possible if explanations are not provided). While not all the explanations will be helpful, the opportunity to explain will likely be appreciated by managers. Use of Performance Information 2.26 A major constraint on the usefulness of performance measurement in many countries has been the lack of understanding by managers as to how to use the performance information and what they should expect from the system to make it useful to them. As will be discussed further in Chapter 4, it is vital to provide training to managers on such important uses as: how to select performance indicators and targets for them; and how the data on the indicators can be used to help them improve their programs and develop and justify budgets. 2.27 DNP, with the help of the entities, should identify "good" indicative plans and data collection procedures and disseminate them to other agencies to help these other agencies to improve their own plans. Share experiences. For example, the performance measurement procedures being planned by ICBF for its child care/child development program appear to be excellent. The information expected from that measurement process should be very useful to management for making program improvements and should provide highly relevant information on program success over time. However, these procedures are quite sophisticated and require substantial resources each year. Simpler versions of these procedures are likely to be useful to many other programs in many ministries and entities. 2.28 Performance reports should be shared with all the program's employees. The employees are "partners" in the provision of the services the program is providing. This will help encourage employees to work to improve services. Colombia: Paving the Way for A Results Oriented Public Sector 13 2.29 Each ministry should consider in its annual public reporting a "Report to the Nation" on its accomplishments (see Chapter 3 for further elaboration). This report would include highlights from the indicative plan performance reports, particularly information relating to outcomes. PROGRAM PERFORMANCE UNITS 2.30 The idea of forming a group that brings together entities that have considerable involvement in the delivery of a service in order to provide joint planning, coordination, cooperation, and evaluation is an excellent one. These units, however, should not "manage". It should be understood that these PPUs are intended only as advisory groups and not as new management units. Indeed, new formal institutional units are not desirable, and will likely confuse and be disliked by the existing management structure. SYNERGY should be kept as simple as possible and avoid blurring accountability. DNP should be careful that the language used for these new units does not suggest that they have any managerial responsibility. The units might be given a revised name, such as "program advisory units" or "program coordination units" to avoid possible misunderstandings, and tensions, on their role. 2.31 These new advisory units probably need to be established only for those programs where more than one ministry have substantial roles in delivering the programs' services. Sometimes such coordination groups will merely be formalizing existing relationships, such as it is understood has occurred in the water program. For programs in which the major players are entities within a single ministry, the Minister should be encouraged to establish such a coordinating committee if not already established. 2.32 The specific roles of these planning and coordination units should be made clear to the ministries and entities. These roles probably should include the following (all of these are advisory roles): * Examining entity indicative plans for adequacy and consistency across entities, suggesting modifications where appropriate. This includes examining both the performance indicators and the targets for each indicator. The focus of these units should be on outcomes and cost-effectiveness (that is, increasing the effectiveness of the program in relation to the amount of funds used by the program), not outputs. * Preparing, when necessary, a consolidated indicative plan. This may not be needed in many programs. * Examining entity action plans for consistency. * Monitoring and evaluating, periodically, the progress that the entities and overall program are making, as indicated by the latest performance reports of the entities and any consolidated indicative plan reports. * Considering budget implications of the indicative plans and making recommendations about budget allocations. * Recommending to the ministries and entities any coordinated actions that appear desirable to improve progress and when specific problems arise. Colombia: Paving the Way for A Results Oriented Public Sector 14 2.33 These planning and coordination units will need staff support. Presumably, they can be staffed by personnel from the entities represented on the units. DNP sector specialists working with the advisory units should be able to consider this participation as part of their major work, rather than a side duty. 2.34 The inclusion on these units of representatives from other levels of government and from the civil society can be very positive. However, it is likely to be difficult to obtain real and full participation from these members, and not merely nominal participation (because of travel distances, belief by those members that they have little influence over the findings of the units, and lack of time for them to obtain guidance from the groups they represent, etc.). Special effort, however, to obtain active participation should be made for those programs in which major services are delivered to the public and are being provided by other levels of government or NGOs. The Federation of Colombian Municipalities, for example, could provide its positions through representatives on the advisory unit. National, department, and municipal planning councils, could also play an important role and, where appropriate, should be consulted (for more discussion on this topic refer to Chapter 3). Meetings in these cases should alternate between Bogota and other cities. STRATEGIC EVALUATIONS 2.35 Strategic evaluations are used to examine past performance of programs, identify reasons for problems, and provide recommendations for improving programs' performance. These evaluations should, in general, not be used solely to assess program results. The evaluators should be able to provide insights into how the program can be improved (assuming that the program is salvageable). It is good practice to involve the program managers in certain parts of the evaluation process, especially to suggest (but not determine) the criteria to be used to evaluate the program, and later to review drafts of the findings before the evaluation report is finalized. This is likely to significantly increase the likelihood that the evaluation findings will be used. 2.36 Ministries and entities should be encouraged to undertake such evaluations, both for programs in the NDP and important programs not included. To the extent that entities do their own evaluations of NDP programs, this will help DNP in its responsibilities for evaluation of the NDP. These evaluations are also likely to be particularly helpful to individual programs and their managers. 2.37 Strategic evaluations, however, are costly and only a few can be done each year. DNP, ministries and entities that do evaluations should establish their own process for selecting the programs that each entity can afford to evaluate each year. Criteria for choosing among candidates include: the importance of the program (and its inclusion, or not, in the NDP); extent to which the program is a "hot issue" facing the Government; the likelihood that the evaluation findings will be completed at the time the information is needed; whether some other government unit (such as the comptroller or a ministry) is already doing, or has done, a recent evaluation of the program; the cost of the evaluation (particularly in relation to the size and importance of the program); and feasibility of the evaluation (e.g., are the data needed likely to be obtainable). Colombia: Paving the Way for A Results Oriented Public Sector 15 2.38 Ministries and/or their entities should be encouraged to maintain their own evaluation offices. These offices should probably be responsible for overseeing the indicative plan performance measurement process as well as oversee the preparation of sound program evaluations. A number of agencies already have evaluation units, in many instances located in the planning offices. 2.39 Identify "good" evaluations (and when available the uses of their findings) and disseminate them to other entities to help these other entities to improve their own evaluations. For example, the evaluation plan of ICBF for its child care/child development program is a good illustration of what can be attempted. Though the evaluation will be costly, many of its features can be used to illustrate what might be done in smaller, less expensive evaluations. (As noted earlier, since the ICBF evaluation is also planned to be used to provide annual performance information, it can also serve as an example of good practice for indicative plans). POLICY/PROGRAM ANALYSIS (EX-ANTE EVALUATION) 2.40 Analyzing future policy or program options in order to help officials make sounder choices is an essential activity but also is very difficult. It is inherently more difficult than evaluations that only examine past actions. Its primary difficulty is that it requires estimates of what will occur in the future, including both estimates of the expected future costs and of future outcomes of each policy or program option. Estimating the future is a treacherous business. 2.41 DNP has developed some capability in this area through its project evaluation work. This work, however, focuses on project analysis and not program analysis, a considerably more difficult task. At this time, and consistent with a results-oriented public management framework, DNP should move away from project evaluation--leaving it to the line ministries--and focus instead its efforts on strategic policy analysis. This analysis is intended to determine the best approach for program intervention and assessing what additional, or reduced, benefits would be likely if funding for the program significantly changed (see Chapter 5, section on Strategic Policy Formulation for further discussion of this issue). Evaluation results should increasingly contribute to the information base for policy analysis. EVALUATING THE NATIONAL DEVELOPMENT PLAN 2.42 DNP appears to be faced with the enormously difficult job of evaluating the results of the four-year development plan. The plan, however, is so wide ranging and involves so many different types of programs that it will, in effect, be impossible to do a comprehensive, complete evaluation. Probably the most practical approach for its annual and four-year evaluations of the NDP is the following: * Evaluate thoroughly some of the programs to assess the effects of these programs. These evaluations would be the strategic evaluations done to date by DNP and those program evaluations done by the entities--as discussed earlier. (DNP can not do this all by itself It needs the entities to provide part of the strategic evaluation information, probably a substantial part). Colombia: Paving the Way for A Results Oriented Public Sector 16 * Analyze the annual performance data reported by the various entities as part of the indicative plan reporting process. The emphasis probably should be on comparing the latest available performance data to the earliest available relevant data (the baseline) for each indicator. As noted earlier, this information will indicate how much progress is being made but is not likely to provide much information as to the extent to which the government's programs caused the outcomes. * Analyze the trends in relevant national statistics, such as those available from DANE and the private sector. 2.43 A major issue for the evaluation at the end of the four-year NDP will be the timing of the strategic evaluations. Since many of these are expected to be done in the early years, these evaluations would not necessarily pick up the full progress made by those programs at the end of the four-year NDP period. In any event, an evaluation of the full four-year development plan probably can not be completed until at least a year after the end of the corresponding period and the data become available. 2.44 The evaluations for the 1995-98 period will in most cases lack baseline data (such as on the years before the Plan) for the various programs in the Plan. Later NDPs should, hopefully, be able to use more complete performance data collected for the annual indicative plans. 2.45 For programs that involve service delivery by departments and municipalities, DNP will face difficulties in aggregating information obtained from each of these entities across the country. There might be an inclination for central government to mandate what performance indicators and data collection procedures are to be used, so as to avoid the information collected by each entity being different across entities. Preferably, the Government should use the partnerships discussed earlier, such as the program advisory units, to decide jointly on performance indicators and data collection procedures. This would also greatly increase the likelihood that future development plans of the central government, departments, and municipalities would be at least reasonably compatible. To address this issue, the ministries responsible for overall evaluation of these programs, and DNP, will have options such as the following: * Undertake new data collection by sampling the work done across the country and collecting the data needed on each performance indicator. Such data collection, however, will be considerably more informative if baseline data collection is done as soon as possible. Otherwise fully comparable information on the extent of improvement from an early baseline year will not be available. * Rely on the data collected from each local entity. Assuming that most local entities did not change their indicators or data collection procedures significantly over the time period, the central government entity could report the number, and percentage of localities for which the reported outcome data showed various amounts of change. * Combine, on a case by case basis, both of the above. Colombia: Paving the Way for A Results Oriented Public Sector 17 LIMITATIONS OF EVALUATION 2.46 Evaluation and performance measurement have a number of limitations. These should be recognized by those doing and using evaluations. Creating excessive expectations for evaluation will likely backfire and lead to reduced credibility for evaluators. Following are listed key limitations of which users should be aware. * As described earlier, the regularly collected performance measurement information seldom is sufficient by itself to determine what caused the outcomes. The in-depth evaluations are usually intended to provide causal information. * Even well designed and implemented evaluations often do not give clear-cut information as to how successful the program has been or to what extent the program has cause the observed outcomes. * In-depth evaluations usually require considerable skilled personnel and time and costs to complete. Thus, not many are likely to be feasible during any year, and the findings may not be available for a long time. * Neither the findings from strategic evaluations nor those from the regular performance measurement process for the indicative plans will tell how to apportion funds among sectors (such as among health, education, energy, etc.). The evaluation information should indicate the outcomes of each program, but judgments are needed by public officials as to the budgetary implications for each sector. - The findings from the performance measurement and strategic evaluation processes can provide some information as to how the programs can be improved. However, policy and program analysis are needed to provide estimates of the costs and benefits of extending the programs in the future and of other, competing options. * While the findings from performance measurement and in-depth evaluations can help inform decisions, other considerations, such as political considerations, can affect decisions and override the information from the evaluations. * The findings from SYNERGY will not tell what the government should be doing, they can only indicate how well the government has been doing on its programs. OTHER ISSUES RELATED To SYNERGY 2.47 The GOC will need to establish procedures for assessing the quality of the data that is reported. This will become increasingly important as the data are used for accountability purposes (e.g., establishing budgets or completing individual performance appraisals). Agencies, such as entity internal control units and the Comptroller General, might be used to check periodically on data collection procedures and on samples of the data. Such quality control units should provide constructive suggestions for improving the quality of data collection procedures. 2.48 DNP and the various ministries and entities face many common issues. In addition, joint cooperation and planning on evaluation and performance measurement processes can sometimes achieve many economies of scale, and save entities time in all having to work out each issue by themselves. Therefore, an Evaluation Advisory Committee should be established to discuss common issues and to resolve common evaluation problems. The Committee should include representatives from DNP and evaluation offices throughout the government. Issues such as the Colombia: Paving the Way for A Results Oriented Public Sector 18 following should be addressed by this Committee: procedures for undertaking customer surveys; clarification of definitions and forms; training and technical assistance needs; procedures for coordinating and working with departments and municipalities; computer/MIS issues (such as the availability and testing of new software for tabulating the responses from customer surveys or from "trained observer" ratings); identification and dissemination of evaluation success stories; and recommendations to DNP as to evaluation policy. Involving the various evaluation units across the government should lead to higher quality evaluation as well as more support for the evaluation effort across the government (see a related recommendation on Chapter 4 on the establishment of a Senior Leadership Network in central government). 2.49 Many programs in the NDP involve departments and municipalities in the delivery of public services. The degree of success of such programs will be affected by all three levels of government (and, often, by the civil sector). Considerable performance-related information is likely to be needed from these governments. This need will increase as decentralization is fully implemented. As discussed in Chapter 4, the central government will need to provide assistance to these other governments to help them in their evaluation efforts and avoid overloading them with requirements. As already noted, it is recommended that for programs that involve department and municipal service delivery, central government and its entities establish true partnerships with these other levels of government relating to evaluation, as well as service delivery issues (see Chapter 5, section on Clarity in the Assignment of Responsibilities and Accountability for further reference to the partnership concept). 2.50 SYNERGY, as currently defined, by itself does not include many elements that are important to the successful implementation of a process of evaluation and use of the evaluation information. For example, use of performance measurement and special evaluation findings in developing and justifying budgets each year is important to the success of SYNERGY. Similarly, the inclusion of performance information in performance appraisals of managerial employees, in performance contracting, in performance agreements between managers and higher level managers, is not explicitly part of SYNERGY--but is part of an overall results-based management framework. These topics are discussed further in Chapter 5. 2.51 Finally, one needs to point out that this very ambitious program will take many years before most of it is in place and working fully. This is likely to take about ten years, assuming that it does not go through major cutbacks over the period. The early years will be learning years. Inevitably, many changes will occur, and many improvements will need to be made, in the initial indicators and data collection procedures. To maintain the effort, DNP should develop a long- term (perhaps 10-year) schedule, and annual plans, for performance measurement implementation. Importantly, the performance measurement and strategic evaluation processes will require a continued commitment of resources from the government. Without this, SYNERGY will be bound to flounder and lose its momentum and usefulness. In this context, efforts need to be made to inform Congress and the community at large as to the importance and findings of the evaluation work. Congress' long term support is important to sustain the evaluation effort. 2.52 Despite this, the Government of Colombia should expect that much useful information on many of its programs should be available for use in the near future. SYNERGY should be more than paying for itself in program improvements in the not too distant future. 3. LINKAGE WITH CIVIL SOCIETY - RESULTS FOR PEOPLE INTRODUCTION 3.1 Within SYNERGY's conceptual framework, a participatory civil society plays a central role in the definition of development objectives and the evaluation of corresponding public management results. The enormous resource found in Colombian civil society' provides a unique starting point for a process that aims at bringing together citizens and public managers in the definition, evaluation and accomplishment of the nation's development objectives. In Colombia, non-governmental and private sector organizations provide a substantial proportion of services for the public, sometimes under contract to government and others competing or complementing the public sector. This is an increasing trend. With the modernization of the health sector, for example, the government will all but cease to be a direct provider of services. Within civil society, in addition, national coordination structures have long been established for NGOs and for civic associations (Juntas de Acci6n Comunao. A large number of institutions and participatory instances in civil society (e.g., Veedurias, Consultas Publicas, the Plan Nacional de Rehabilitaci6n, and Red de Solidaridad) are currently available at the community level. There is also an increasing trend among NGOs to join forces with each other and with governmental agencies on specific development initiatives. 3.2 Thel991 Constitution, its requirements for evaluation of the NDP, and the laws enabling this process, explicitly recognize civil society's fundamental responsibility in the definition of development objectives, and thereby in the evaluation of their achievement, as well as its importance as recipients or users of public services. Indeed, the constitutional origin of SYNERGY is this more public-minded approach to public service. Its stated objectives (better service, results orientation) represent a corresponding focus of evaluation on results for people. The proposed involvement of civil society in SYNERGY is through the representation of civil actors defined by the national and territorial planning councils in the PPUs. Although impressive examples exist of how some of the planning councils (at the national, departmental and municipality level) have gone beyond their mandate to review the development plans at each level, this could easily decay into tokenism. Militating against effective inclusion of civil society in the national evaluation system are: (i) the voluntary nature of the participation in these councils, (ii) their limited formal role to review plans, (iii) the limited technical skills in evaluation of members, and (iv) the delinking of these institutions from each other and from SYNERGY. 3.3 The national evaluation system cannot address the public sector's development objectives without considering its results for civil society. By the same token, it is inappropriate (and wasteful) to ignore the enormous energy and resources of civil society that can be harnessed in this effort. Following are recommendations aimed at ensuring civil society participation in SYNERGY in two critical ways: (i) identifying what the results are for individuals and Civil society is a distinct public realm located between the family and the state, where individuals join together to pursue collective goals. Colombia: Paving the Way for A Results-Oriented Public Sector 20 communities; and (ii) being a partner in the evaluation process. Suggested initial steps towards implementation of the recommended strategy are then presented. THE CENTER OF GRAvITY OF EVALUATION: FROM INSTITUTIONS TO COMMUNITIES 3.4 Gathering Community-basedInformation. Most of the data for evaluation now contemplated by SYNERGY are service-based (hospital records, school records, attendance registers, etc.). Although convenient--the data are readily available--and within the long tradition of dependence on service-based data, these institutional data are not sufficient for the evaluation of public services. A shift is required to complement institution-based data with those from the communities. This shift can be made by introducing a periodic survey process as an integral component of SYNERGY, to obtain the population-based data necessary for transparency and accountability in the terms envisaged in the Constitution and supporting laws. As summarized in Box 1 below, Service Delivery Surveys (SDS) can be effective instruments of evaluation and Box 1: Service Delivery Survey Missing Ingredients. The success of public sector management has been limited by governments' lack of: * an accurate baseline of service delivery, to design and to benchmark appropriate changes; * the mechanism to measure community-based impact through improved service delivery, * the means to instill a customer orientation to the services in question; * a dialogue between citizens and government on expectations and preferences regarding service delivery, and * an accountability mechanism for efficient, effective, and equitable public spending. Focus on the customer. An underlying assumption of SDS is that citizens' opinions matter, and that the citizen should be treated as a customer might be in a commercial transaction. In order to reorient management toward satisfying the customer, the customer's expectations, preferences, and experiences must be known. SDS accomplishes this through collecting the views of a representative sample of ordinary citizens and other recipients of government-provided services (that is, businesses, local governments, and civic associations). Using the physical presence in the communities to do the survey, additional monitoring techniques can also be included (institutional reviews, key informants and focus groups are the usual tools). Qualitative Dimensions of Performance. The methods of community-based data collection should include qualitative dimensions like definition of expectations, satisfaction, identification of problems, suggestions for improvements in addition to the quantitative dimensions of who benefited. SDS can be used to investigate both attitudes and behavior (the major limitation is time and resources). Among others, here are a few examples of what can be investigated more closely: * Public perception - how well-known is the entity and its services to the public? * Public access - how accessible is the entity? (handling times, telephone service reception times, confirmation of business, reply) * Courtesy - to what extent are citizens treated politely and attentively? * Availability - can citizens count on service being regular and as publicized? * Accuracv - to what extent are all or most transactions completed correctly and are errors rectified expeditiously? * Meeting service standards - to what extent are service standards generally met by those who serve the public? When they are not, what procedures exist for dissatisfied citizens to seek redress, and to what extent is appropriate action taken by the service provider? * Convenient hours - to what extent do the hours when services are provided allow reasonable access for citizens with different work and personal schedules? * Proximiti - to what extent are services provided so that citizens are not required to travel unreasonable distances? * Coordination - to what extent are services that respond to simnilar needs provided in ways that minimize the number of contacts a citizen must make? * Oualitv of actual services delivered - for example in education, health, water, transport, energy, roads, agriculture extension, redistribution of land, food production and employment programs. Instrument for Reform. SDS can also improve the government's capacity to design and to measure progress of reform programs. SDS builds capacity to manage the process of service delivery and produce services that better satisfy the public's needs. It also creates a consultative medium between government and the public to enhance the participatory process. Colombia: Paving the Way for A Results-Oriented Public Sector 21 public sector management. SDS can be used together with data from services on results, coverage and costs. They aggregate opinions from both users and service workers about possible solutions to problems and reasons for successes. 3.5 Precisely what constitutes good performance of a public sector can be defined from two points of view, that of the provider and that of the client. The provider has a technical basis for definition of functions. Clients sometimes, however, are in the best position to assess performance. Indeed, good service from public administration and public authorities should start from the requirements, wishes and demands of those being served. Although frequently there are restrictions--regulations, case conflicts, equity considerations, financial limitations--that mean that all requirements cannot be met, well-grounded knowledge of expectations and demands is always an invaluable asset for public authorities and administrative bodies for improving performance. Through the use of surveys, SYNERGY has the opportunity to reinforce and to improve financial management through transparency on the full costs of public policies. This goes beyond documentation and analysis of the costs of programs to the service providers. It implies documentation of "costs" to the users, the communities, in terms of cash and opportunity. SDS has been used in Zimbabwe and Nicaragua to document the transfer of "costs" from the state to households, and the "costs" to the communities of ineffective or inefficient services (for example, vaccines that might have lost potency, education programs where teachers fail to appear for work). Overall, SDS provides an innovative approach to help evaluate the National Development Plan from both a top-down (priorities of a ministry) and a bottom-up perspective (what people think is the best/worst service, what they think needs improvement). 3.6 Feedback - Back to the Community. Only part of the challenge in an evaluation system is to obtain the data. A perhaps greater challenge is how to put these data to work to improve performance. At present, four uses of data are envisaged in the SYNERGY scheme: (i) information for managing and operating; (ii) budgeting; (iii) information for planning, specifically related to the NDP; and (iv) microprocesses (identifying hot spots for improvement). The attention is focused on internal government decision-making processes (see Chapter 5 for a full discussion on these issues). Importantly, however, these efforts have often in the past neglected a critical link: providing feedback to households and clients. This lack of feedback neglects the potential multiplicative effects of communication with civil society and other stakeholders--and is also inconsistent with other government programs such as Anti-corruption and "It's a Deal." A concerted effort is required to ensure SYNERGY includes systematic feedback from and to stakeholders: * At central and departmental level, agencies should be encouraged to produce an annual "Report to the Nation" (reported through TV, newspapers, etc.). * At municipal level, authorities should take advantage of the wide variety of communication techniques available for local feedback (local radio, pamphlets, posters, child-to-family, community meetings). * At all levels, the interface with the NGOs, Juntas de Acci6n Comunal and other civil society institutions should be used to provide feedback to community; each of these existing instances has internal communication mechanisms that can be put to work to socialize evaluation-generated information at marginal cost. Colombia: Paving the Way for A Results-Oriented Public Sector 22 CIVIL SOCIETY: A PARTNER IN EVALUATION 3.7 Parallel to government structures, several important civil structures have organizational links that run through local level, to departments and national level, through federations and confederations. If the development weight of these organizations is to be pulled behind the NDP and, by extension, the evaluation of that Plan, coordination and working arrangements must be strengthened and streamlined at national, departmental and local levels. 3.8 Coordination. Specific tactics are required to achieve a more productive and unified interface between all the existing civil participatory instances and SYNERGY. The consultative role and voluntary participation of civil society in the territorial development councils, if this is to go beyond their ceremonial value, should be given life through (i) a formal and systematic role in ongoing evaluation and, in order to do this, (ii) access to evaluation skills necessary for an informed opinion. There is already (in Bogota) an NGO initiative to develop a support network for the Veedurias. Ways should be sought to tie the Veedurias, the Consultas and the territorial planning councils (preferably together) into SYNERGY. A consultative process should be mounted with these institutions to explore possibilities. This effort could receive financial support from the Fondopara la Participaci6n Comunal and co-financing for technical assistance from DRI. SYNERGY should ensure that civil society participation in evaluation and planning becomes a reality. 3.9 Institutional Capacity. Implementing SYNERGY implies considerable technical capacity, currently not available in many central government entities or at department or municipal level. Some of the shortfall could be provided by NGOs and the private sector. Local and national partnerships with NGOs could provide one way of building up this capacity. This could follow existing arrangements for government contracting of NGOs to provide services or to participate in collaborative projects. An example is ECOFONDO, which joins some 300 NGOs and over 20 government agencies contractually in joint projects. Early development of collaborative learning exercises that, while producing hard data for planning, also build capacities could prove critical in this effort. This might involve identification or, if needed, development of an NGO or consortium of NGOs with technical capacities in evaluation and an informed commitment to results-oriented management. (See Chapter 4 for further discussion of this issue). 3.10 NGOs and Private Sector - Focus on Results. Although there is undoubtedly a huge reserve of energy and skills in civil society that could be tapped to support SYNERGY, it is important to understand that NGOs, universities, trades unions and Comunales alike have no track record in results-oriented evaluation. Performance evaluation should also be applied in these entities. As a large and increasing part of services previously provided by government is passed to the private and NGO sector, and contracts between governmental and non- governmental agencies become more common place, evaluation for results should be expected to become normal practice in both public and private sectors alike. The vision of SYNERGY should be extended. Only in this way can a culture of service performance (including integrity) be attained. Colombia: Paving the Way for A Results-Oriented Public Sector 23 INTLAL STEPS TowARDs IMPLEMENTATION 3.11 Whatever the Constitution and NDP specifies, there will be individuals who feel that evaluation and links with civil society are against their own interests. An early demonstration of how both evaluation and civil society can increase efficiency and effectiveness, and how some of these benefits accrue to or reflect positively on central government senior managers, will go some way to reduce obstacles at central levels. An early partnership between NGOs, a flagship ministry, departments and municipalities to consolidate capacities of evaluation while generating hard data to demonstrate the functioning of SYNERGY and to examine some of the dynamics involved in its implementation nationally should be put in motion within the next few months. 3.12 Indeed, the coordinating and contracting mechanisms will not achieve their full potential as a link between SYNERGY and civil society, unless based on a new evidential base--evidence of the results of services for civil society. As an illustrative case of what this working relation with civil society might look like, the government should consider having a central ministry which provides services to the public and involves all levels of government as well as NGOs to test the use of service delivery surveys. This exercise would simultaneously establish a process for learning through doing with the NGOs and municipalities; it would allow for rapid turnaround of data while local capacities are being improved; and it would produce evidence for planning at municipal, departmental and national levels (see Chapter 4 for further discussion of this recommendation as well as Annex 2 for the steps that can be taken). 4. BUILDING CAPACITY - DEFINING A CONSTRUCTIVE LEARNING PROCESS FOR IMPLEMENTATION RAISiNG INSTITUTIONAL CAPACITY - A TwO PRONG STRATEGY 4.1 The efforts to enhance institutional capacity for the successful implementation of SYNERGY and ROPM in Colombia could be viewed as part of a broad capacity building process in which the government allows itself, through experience, to: (i) build capacities to evaluate and use the results to improve services; and (ii) learn by doing while motivating by showing results. BUILDING CAPACITY TO EVALUATE AND USE THE RESULTS TO IMPROVE SERVICES 4.2 The techniques of performance measurement and evaluation are not likely to be familiar to most Colombian planning and other personnel. The GOC will need to ensure training and technical assistance is provided to those responsible for performance measurement and evaluation processes in their entities. This includes planning personnel in ministries and their agencies, the internal control personnel (who are expected to play an increasing role in examining entity progress on the national evaluation system), the sector specialists and evaluation personnel in DNP, and other interested central and local government personnel. Such assistance will likely be needed on topics such as: * selection of performance indicators--what are the various categories of indicators and how to select them; * data collection techniques such as customer surveys and "trained observers" procedures; * procedures for making various types of comparisons, including various powerful evaluation techniques (such as quasi-experimental, comparison group designs, and randomized controlled experiments); and * procedures for establishing targets and benchmarks. 4.3 Various entities in the Government have already begun to undertake some staff training, including the Comptroller General and DNP. In addition, various entities such as ICBF have personnel already trained in many of the needed skills. The universities can be an important source of help. Existing training efforts, however, need considerable strengthening and expansion to meet the need. One existing resource that could provide training and technical assistance on some of the procedures, like customer surveys, is DANE. 4.4 Critically, the techniques of performance measurement and evaluation--and the use of this information--are not familiar to most Colombian managers. It is vital that managers be provided training in the use of SYNERGY. Such training should include information on: * the basics of the process; * the various categories of performance indicators and how to select them; * how to identify benchmarks and establish targets; Colombia: Paving the Way for A Results-Oriented Public Sector 25 * how to interpret the information coming from the indicative plans and special evaluations; and * how the data can help them manage for improved performance. 4.5 This last point is indeed critical. Managers need to understand that the evaluation system is a useful tool for them. It is important for them to understand how the system can help them make improvements in their programs and manage them more effectively by, for example: * identifying where the program is having problems in meeting targets--for which locations/offices or for what categories of customers; * obtaining clues as to what is wrong and how the program might be improved (such as the information obtained from customers who have been surveyed and asked to identify what they did not like about the service and how the service might be improved); * helping motivate program personnel to work towards improving their services; * identifying whether new procedures have led to improved outcomes; * including performance indicators and performance targets in contracts with private organizations or other public agencies--perhaps combined with monetary rewards and penalties in the contract (performance contracting); and linking services and outcomes to costs, thus helping to formulate and to justify budgets. 4.6 No systematic effort is currently underway to provide this training. Some ways to provide it include: * Contracting universities or private organizations. The issue here is to find organizations that can do a good job in different instances. * Developing VCR tapes, computer-based training software, and even TV programs, on the various topics--for use by individual entities and their personnel (in their own training efforts). Such programs/tapes probably should be targeted, such as by sector and level of official. Separate materials should be prepared for management and for technical personnel, as well as for department and municipality personnel. These same items can also be adapted for use in providing training to different stakeholders such as members of Congress, NGOs and citizens. * Using internal personnel to provide this service. This procedure is likely to be particularly successful if internal ministry/entity managers who have made good use of performance information have a major role in the training. LEARNING BY DOING AND MOTIVATING BY SHOWING RESULTS 4.7 There is need to ensure there is effective demand to build capacity by central government, local administrations and communities. One way to create such demand is through demonstrations: showing authorities, managers and personnel that they can reap the benefits of using the proposed evaluation system. Following are recommendations geared to generate this demonstration effect. Colombia: Paving the Way for A Results-Oriented Public Sector 26 Capacity Building in Central Government Ministries through Performance Measurement 4.8 The most compelling evidence that the national evaluation system has improved the efficiency and effectiveness of the service delivery to the public, is to document improvement using measurable indicators. To show the relationship between an intervention and improved service delivery to the public, the key is to document the service delivery before and after the intervention. Implicit in SYNERGY is that baseline indicators are to be established in order to measure progress of implementation. The before/after dynamics can be illustrated by focusing on performance measurements (such as customer surveys) in selected areas where early success can help gain confidence and credibility for the evaluation process. Identifying potential areas for early success, and applying performance measurement in this initial stage of implementation of SYNERGY, can also serve to motivate entities to participate by showing early positive results. 4.9 It is recommended that performance measurement be pilot-tested in two central ministries as early as possible. The Ministry of Environment could be a suitable candidate--given that it is already an identified pilot for the initial stages of implementation of SYNERGY and the characteristics of the Ministry. Performance measurement can be conducted at relatively low cost, beginning from existing data, framing them in a results-oriented way. Additional data, like measurements of water and air quality would be obtained using different instruments by teams that, in Colombia, might include DNP and DANE staff, selected NGOs and representatives from the pilot Ministry. Implementation costs can be kept down by using local enumerators supervised by the team. The pilot project would help build capacity to design and implement indicators and surveys and to administer the surveys on a periodic basis. While at the same time, it should generate data that would feed the decision-making process, hopefully leading to actions that will produce the desired results. A step by step presentation of the suggested implementation of a performance measurement pilot project is summarized in Annex 2. A Senior Leadership Network in Central Government 4.10 GOC is facing some major challenges dealing with key issues such as redefining the role of government, making it more results oriented, cost-effective, less corrupt, and closer to the public it is there to serve. To facilitate these major changes will require change in mind-sets, awareness and skills. The establishment of a Senior Leadership Network is recommended. The purpose of this Network is to generate a critical mass of skilled central government leaders committed to good governance and to seeking and championing result orientation, integrity and sustainable development. 4.11 The Network will be open to all senior central government officials (ministers, vice- ministers and directors). Through regular meetings and seminars, participants are to review and to present best practice case studies in Colombia and abroad. They should be exposed to the latest in the management of change, strategies for reforming government and general leadership practice. A tailor-made senior management program by a leading education institution could also be developed for the members of the Network. This effort could be headed by the Administrative Department of Public Office. Colombia: Paving the Way for A Results-Oriented Public Sector 27 Local Government Best Practice Workshops 4.12 Access to information about innovations and best practice in other departments or municipalities and available sources of local support is a powerful tool for local governments to strengthen their capacity. To facilitate this learning process, a series of two-day departmental workshops (perhaps based on a stratification of departments and municipalities) bringing together governors, mayors and senior local government officials are recommended. Probably the Federation of Colombian Municipalities would be in the best position to organize such workshops. Among others, the purpose of these best practice workshops could be to: * Provide local government authorities and selected NGOs information about SYNERGY and the principles of evaluation, including the concepts of evaluation planning in the context of local needs and capacities. It would be important that any presentation and/or hand-outs would be tailor-made to the audience (e.g., practical and easy to understand). * Discuss planning objectives, priorities and programs of a diverse number of local governments including their experience with planning bottlenecks and how these are being tackled. 3 Share best practice of involving the civil society, the use of service delivery surveys and other performance measurement techniques (e.g., trained observer ratings), cutting-red tape to improve service delivery, cost cutting measures, use of NGOs in capacity building, the involvement of private sector and NGOs in delivering services and exchange of concrete management experiences, tools and procedures. * Give some concrete tools for analysis of available data in the planning context, to be used by governors, mayors and their staff upon their return. * Share best practice from other successful countries in the region and possibly beyond (for example Australia, New Zealand, Singapore, Malaysia and United States). 4.13 Participants to these workshops should include senior government officials from each department and municipality (governor, mayor, senior administrative officer, chairman of the planning council), selected representatives from NGOs and/or private sector. With four representatives from each municipality and one hundred people at each workshop, national coverage would mean conducting around forty workshops. Should there be a need for involving external experts in the first pilot, they should work closely with the Federation of Colombian Municipalities and local NGOs and build the capacity for them to take over the process after one event. 4.14 In preparation for the workshops all participating departments and municipalities should bring their development plans stating their key priorities, objectives and action programs. These will be collected by DNP and used to increase their awareness on how to guide, plan and evaluate the implementation of the national plan in the context of local priorities. The workshop would have plenary presentations identifying best practice in results-oriented public sector management including areas such as: evaluation practices, community involvement, local interventions leading to improved service delivery (contracting out, contract management), reduction in cost and concrete measures to improve internal management. Following the plenary presentations the Colombia: Paving the Way for A Results-Oriented Public Sector 28 participants would be divided into small working groups to discuss best practice, identify key bottlenecks and agree on future collaboration and cooperation activities. Dissemination of Experiences and Information across Local Governments 4.15 The dissemination of experiences and information on capacity building will require a concerted effort by NGOs and the Federation of Colombian Municipalities, among others. The returns, however, can be substantial. Examples of potential programs include: * Data-base and library on best practices and local government innovations. The general proceedings from the forty workshops would be available for all local governments on request. * Contracting out services, process and procedures for bidding and management of the contracts. * Telephone consultations for mayors, council members and citizens. * Service Delivery Survey questionnaires and guidelines for how to implement, possibly using local consultants--providing names of qualified candidates. * Advisory services on how to recruit and sign contracts with NGOs, universities, and co-operative agencies. Box 2: Best Practice from Medellin - Contracting Out Education In the city of Medellin the cost of primary public schools has been estimated to be 45,000 pesos per student per month. Tendering to the private sector, the cost to the city has been reduced to 22,000 pesos per student per month. Municipal authorities, supported by the DNP, should now evaluate the outcome of this decision by monitoring the performance of private schools being used against standard exams and to compare this with public schools. Importantly, however, this provides an example of how local governments can learn to work with new partners (private sector and/or NGOs) to significantly expand their capacity to perform their responsibilities more cost-effectively and provide better quality services to the public. Practically all municipalities have private contractors executing part of their construction projects and, based on discussions with local authorities, this seems increasingly to be the case in the education sector. As a result, contract management seems to be an important capacity to be developed across to municipalities. Colombia: Paving the Way for A Results-Oriented Public Sector 29 Rewarding Best Practices 4.16 SDE has indicated that as part of the implementation strategy for SYNERGY, the government would give best practices awards. In moving forward this initiative, it is recommended that DNP in close collaboration with the Ministry of Interior, the Federation of Colombian Municipalities, relevant NGO and private sector representatives, establish a process by which a Best Practice Committee invites line ministries, departments and municipalities to report on their interface with civil society and observed improvements in service provision. The Best Practice Committee with civil society representatives would identify and reward the entities with the best results. Criteria for giving the awards have to be established and disseminated in advance. An appropriate dissemination of the selection of the best practice cases to all ministries, departments and municipalities could potentially speed up the process of improving the national interface between the service providers and the public. Rewarding and disseminating these best practices could possibly also increase the incentives for public entities to improve their community involvement and thereby facilitate the management of expectations. 5. PUBLIC MANAGEMENT FRAMEWORK - PROVIDING AN ENABLING ENVIRONMENT FOR A RESULTS-ORIENTED PUBLIC SECTOR INTRODUCTION 5.1 The National Evaluation System is designed to evaluate the National Development Plan, provide information about performance and the cost-effectiveness of programs and be a catalyst for reforming public management. This management reform is aimed at greater effectiveness and efficiency of the public sector through a culture of results-oriented management, continuous improvement and involvement of civil society. 5.2 The conditions for success in achieving these objectives entail issues both within SYNERGY itself and also in the environment defined by wider public management systems beyond the immediate reach of SYNERGY. The issues concerning the design of the evaluation system, involvement of civil society, and institutional capacity have already been presented. Discussed here are issues in the wider management environment not under SYNERGY but which have major effects on what SYNERGY can achieve. 5.3 It is not intended to suggest that SYNERGY should be expanded into a comprehensive review of public management. However, setting realistic expectations for it, especially in terms of its contribution to successful implementation of the National Development Plan and changing the management culture of the civil service, requires consideration of the constraints arising from other aspects of public management. A major challenge will be to ensure that this wider enviromnent both supports and requires a performnance orientation. 5.4 SYNERGY could succeed in producing evaluation information and, like water dripping on a stone, it can eventually, by improving transparency, induce desirable policy changes. In view of the objective to have SYNERGY stimulate the development of results-oriented management it could still, however, be seen as a failure if the evaluation information shows the Plan has not been implemented or has not met its objectives. For success in broader terms, SYNERGY should feed its information into coherent decision-making processes at all levels and into management systems embedded in an overall incentive framework which will encourage and require managers to take necessary actions to attain desired results. If this is not happening then the evaluation effort will be set back. 5.5 To enhance the prospects of success in relation to SYNERGY's broader objectives regarding the NDP and ROPM, policy makers should undertake a concerted effort to create an enabling environment. Special attention should be given to the areas of: * strategic policy formulation; and * management incentives. Colombia: Paving the Way for A Results-Oriented Public Sector 31 The following paragraphs present specific recommendations in these areas.' STRATEGIC POLICY FORMULATION 5.6 SYNERGY and the wider results-oriented management framework need to succeed in terms of achieving the objectives of the NDP as a whole and not just a selection of successes in some areas. The whole thrust of the Plan is to set in motion a web of mutually reinforcing policies that is more than the sum of its parts and accelerates the country's economic and social development. For this to happen the total policy framework should retain strategic coherence without which the policies will pull against each other, and destroy the essential synergy so that success in some areas comes at the expense of failure in others. 5.7 The most difficult political and policy problems for the government are likely to arise not in the design of new programs but in resolving the hard trade-offs between conflicting goals and searching for ways to bring them into harmony. Deciding between protecting public service jobs and seeking efficiency from contracting out services is a good example. To deal with these trade- offs the government needs to have a strategic policy-making framework that will allow: (i) policy choices based on efficiency and effectiveness criteria; and (ii) maintenance of macroeconomic control without jeopardizing performance in the executing agencies. SYNERGY can play a useful role in promoting such framework, not only focusing attention on this dimension of public sector management but as a vehicle to identify the nature of the corresponding necessary changes. Unbiased Choice of Policy Instruments 5.8 Policy analysis at all levels should go beyond entrenched intellectual commitments, institutional walls or sectional interests in society. This is necessary to ensure the wider public's welfare is the only objective sought. There are, however, certain features of the way public entities are to be involved in implementing the NDP and its evaluation that might bias policy choices. 5.9 There is a strong potential for bias towards public expenditure, especially investment, over other policy instruments in the way the NDP is to be implemented. The planning framework has a bias towards the government taking action and doing so through public expenditure which wili inevitably create upward pressure on aggregate expenditure. The achievement of policy objectives may often be pursued more cost-effectively through alternative policy instruments such as taxes, regulations, modifying property rights, structural reform, privatization, etc. Policy analysis should be motivated by effectiveness in pursuit of the wider public interest and not be biased towards the choice of particular policy interventions over others. For example, two local government authorities commented that the key issue in making efficiencies and quality improvements in education and health was the freedom to contract with non-government providers. As shown in Box 2 before, in one example the cost saving was 50 percent (with the authorities stating that this had been accompanied with quality improvements). 1 For a detailed discussion on issues related to the broader framework of Results-Oriented Public Sector Management see Annex 3. Colombia: Paving the Way for A Results-Oriented Public Sector 32 5.10 But sometimes these considerations will be difficult, particularly when the approach to program design involves participation by the providers of services who have something to lose from a new approach being taken. While there are many advantages to getting all the affected parties involved in planning the work within a program there is danger of capture of the policy by existing providers. It is important that there be contestability in the area of policy advice. The DNP should see its role as including being a counterweight to any excessive sectoral interests that might advance policy options that are not superior in terms of the wider public interest. The in- built bias it should seek to circumvent is its own responsibility for public investment. Indeed, SYNERGY can help offset this bias by focusing its strategic program evaluations on the major expenditure pressure points and considering options for goal achievement by means other than public expenditure and assessing the consequences of different levels of expenditures. Funding the demand side rather than the supply side of health illustrates the importance of clear unbiased strategic thinking in the choice of policy to achieve program objectives. 5.11 The Minister of Finance and the Director of DNP should satisfy themselves that the structures, processes and resources for top level strategic decision-making in the government are adequate. The roles and functions of the various units in the DNP should be reviewed to ensure there is a clear focus on strategic policy integration in a medium-term framework. This must obviously involve an emphasis on evaluation throughout DNP--including its sectoral units--as a means to attain the 'big picture' about the countly's development. DNP should move away from reviewing detailed projects from the project bank (see below) and emphasize its tactical role in strategic policy analysis and integration of advice to government. SYNERGY provides a valuable tool within DNP to encourage this shift. But more will be required. The proposed reforms to budgeting and management incentives set out below are also important. Macroeconomic Control and Microeconomic Performance 5.12 The challenge for the central government's fiscal decision-making processes and the budgeting and financial management systems is to strengthen macroeconomic control while ensuring resources are allocated according to strategic priorities and performance is improved in the executing agencies. This is very difficult to do, and few countries have succeeded. The most common pattem is for the central financial authorities to retain the most flexibility they can by avoiding medium term commitments and periodically confront the executing agencies with unplanned budget cuts that take little account of the effects on performance--at both the strategic and operational level--and impose central directions on resource use. This is not consistent with efficient and effective performance in the agencies, even though usually justified in terms of fiscal policy contingencies. 5.13 The harmonization between macro fiscal control and efficient management entails a number of requirements: (i) a medium-term expenditure framework; (ii) capped funding; (iii) a unified budget; and (iv) a results-oriented financial management information system. 5.14 Medium-term expenditure framework. Fiscal policy should be managed to medium-term targets consistent with a comfortable low-risk debt situation, stable tax rates, and a monetary policy which is targeted on inflation. Inconsistency here will damage those sectors of the economy exposed to international competition. It will also undermine the effort being put into Colombia: Paving the Way for A Results-Oriented Public Sector 33 improving the efficiency and effectiveness with which the development objectives of the country are pursued. Mounting evidence exists that a more medium-term approach to expenditure management can reduce instability at both the macro and micro levels. 5.15 The key features of a medium-term expenditure framework (MTEF) are: (i) medium-term fiscal targets; (ii) forward estimates of the costs of current government policies; and (iii) high correlation between forward estimates and final budget allocations. It is essential for it not to be seen as simply a numbers exercise. The whole rationale for such a system is that it forces policy making (planning) to be undertaken within the framework of budget realities. The trade-offs at the macro and micro levels are to be constantly addressed. Such a system will only deliver if all policy matters having financial implications are forced into this framework (which is not the case with earmarked funds, for example). In terms of performance at the micro level the contribution of the MTEF is through the greater certainty it can provide. If there is no obvious relationship between the estimates for the forward years and final budget allocations, then the benefits of certainty are lost. 5.16 Cappedfunding. The transition to the annual budget consists of two phases, both of which involve a tight top-down budget constraint. The first phase concerns formulation and the second appropriation and execution. The first phase begins with a firming up of an aggregate expenditure ceiling derived from the fiscal targets. Within this aggregate ceiling, an approach to resource allocation could involve the following stages: 1. Government collectively (CONPES) determines any necessary changes in key strategic policies (which may be made necessary by a significant gap between the aggregate of forward estimates coming into the budget year and the aggregate ceiling or by a significant change in strategic direction). 2. Individual ministers are then given a constrained resource envelope within which they have to formulate a proposed allocation of resources to the remaining policies and programs for which they are responsible. The proposed allocation is then reviewed by the center to ensure that it is consistent with the government's strategic policy framework and that there are no hidden out year costs. 3. Administrative expenses are determined on a transparent formula basis (a component of which could be some form of efficiency dividend to force efficiency gains). The proposed budget which flows from the above three stages would be submitted to Congress. The funding allocations appropriated by Congress would be capped and entities required to live within these limits once they have been given the management freedom to do so. 5.17 A number of important issues must be noted in relation to this proposal. The first is that to be fully effective this capping arrangement must be set within a medium-term expenditure framework. The second is that the approach to administrative expenditures is no substitute for thoroughly reviewing staffing levels and for moving towards an arrangement whereby there is a tighter link between administrative expenses and the outputs being produced by these resources. Thirdly, the proposed budgetary system is consistent and supports ongoing efforts to push decisions down to the level where all relevant information is available (this applies equally to the design of the decentralization policy and to central government's own activities). Finally, as Colombia: Paving the Way for A Results-Oriented Public Sector 34 mentioned before, DNP should get out of reviewing detailed projects for the project bank. This database could be dispersed to ministries after the hard budget constraint on the total funding available to each ministry is put in place. It would then be the responsibility of each ministry to determine the best way of allocating resources. Any changes in policy beyond the authority of the individual ministry that require Cabinet consideration should be presented as a formal submission within the framework of budget decision-making. As discussed below, these submissions could become a powerful tool for imbedding SYNERGY in the government's decision-making process. 5.18 A Unified Budget. This involves bringing all expenditures together in the budget process, making it easier to force the difficult trade-offs. It has the potential to remove the biases inherent in a dual budget, where control and the relative ease of obtaining funding will inevitably vary between the current and investment budgets. At the same time, current distortions to the input mix--as exemplified by the lack of operating and maintenance expenditures in some programs-- will be minimized. Since the outcome and output information being developed under SYNERGY will often be only relevant for decision makers when linked to full costs, the case for unifying the budget will only strengthen in the years ahead. 5.19 The development of a unified budget system would raise issues about the allocation of responsibilities between the Ministry of Finance and the DNP. Cooperation between the two agencies appears to be satisfactory but the integration of development strategy, planning, macro and fiscal policy together with results-oriented public sector management would call for a review of responsibilities. In the future, a possible merger should be considered as one of the options--as happened in Korea recently. In the meantime, the Budget Law should be modified to move in the direction of a unified budget (see section on Budgeting Incentives below for further discussion). 5.20 Results-Oriented Financial Management Information System. The changes proposed above, together with those being driven by SYNERGY, will have significant implications for the financial management information system. As things stand, the lack of information on outcomes and outputs which is being addressed under SYNERGY is more than matched by the lack of readily available data on the costs of policies and programs. Unless this is remedied SYNERGY will be undermined. 5.21 The financial management information system needs to be able to record expenditure on the basis of services provided or things accomplished. Priority setting requires reliable information on the costs of policies and programs2. The GOC's budgeting and financial systems are undergoing modernization with the focus being on enhancing macro control and information. This is highly desirable. There is not, however, an emphasis on integrating macro control with the efficiency and effectiveness requirements or management information needs for results-oriented 2 Information on the full resource cost of programs will require that the lack of precision about current and capital expenditure definition is addressed. When the management information systems and management skills develop in the agencies it would be timely to consider introducing accrual accounting. This will be a necessary step in removing the bias in resource use stemming from the use of cash-based accounting which does not spread asset consumption over the life of assets and provides little incentive to managers to manage assets efficiently. Colombia: Paving the Way for A Results-Oriented Public Sector 35 management. Early attention will need to be given to reformulating the modermization effort so as to support results-oriented management. MANAGEMENT INCENTIVES 5.22 The effectiveness of SYNERGY in triggering a cycle of self-improvement in public sector management depends considerably on its place within a larger framework of incentives surrounding managers in the public sector. Incentives are basically about the effects on the behaviors of managers that are induced by the key features of the total management system. These are concerned with: * setting both strategic and operational objectives in assigning organizational and personal accountabilities; * allocating resources to achieve those objectives; * giving managers authority--freedom and decision rights--to manage; * using performance information; and * following the consequences for those who are accountable of success or failure in their endeavors. 5.23 Whether the evaluation process itself is done well, and what effects it has in generating improved outcomes depends on how these management system features are represented in the details of the management framework within the Colombian Government. Three areas which warrant specific comment in this regards are the assignment of responsibilities and accountabilities, budget incentives and personnel management. 5.24 In considering these issues and how they might be tackled there appears to be considerable scope to exploit SYNERGY to lever the necessary changes in management incentives. Indeed it, of itself, provides a framework for designing and implementing important features of the total management system. Clarity in the Assignment of Responsibilities and Accountabilities 5.25 The principle for setting objectives should be that they are set at each level in a way that accurately reflects what the people at that level have responsibility for and the freedoms they have to manage so that they can fairly be held accountable for related results. While each level in the hierarchy must be motivated by the bigger picture surrounding their own activities, there must be a fair matching of accountability and freedom and this should be reflected in the way their objectives are described. Line managers can rarely be held solely accountable for the outcomes associated with strategic objectives. By the same token, those responsible for strategic objectives should not be held solely accountable for the micro decisions of managers pursuing their responsibilities within those objectives. 5.26 Needless overhead expense and lost efficiency and effectiveness can result from lack of clarity about the roles and accountabilities of the organizations responsible for programs or projects. Where this clarity is lacking there can be implicit incentives for managers to avoid risk Colombia: Paving the Way for A Results-Oriented Public Sector 36 by seeking approvals for everything because they are not clear what they are responsible for. Unclear assignment of objectives makes it impossible to establish accountability for results and undermines the whole thrust of results-oriented management. There are many examples of blurred responsibilities in the public sector. The roles of central government ministries vis-a-vis the executing agencies is not as clear as it should be in terms of the details of implementation. There is a fine line between setting strategic objectives and monitoring their achievement, on the one hand, and getting involved in the decision-making about how things will be done, on the other. But nowhere is this problem clearer than in the decentralization process. Taking this as an example one can identify the guiding principles that should govem a results-oriented public sector management framework: (i) clarity of purpose and tasks; (ii) authority to perform the tasks; (iii) clear linkage between instruments and objectives; and (iv) commensurate accountability mechanisms. 5.27 If one analyzes the decentralization process in the education sector, for example, all four of the mentioned guiding principles are disregarded. First, no clear assignment of responsibilities. The existing legal framework for the provision of education services--defined by the Decentralization Law 60 of 1993, the General Education Law of 1994, the Teaching Statute of 1979 and different reglamentary decrees--has led to a fragmentation of responsibilities among different levels of government and a lack of clarity resulting in a situation in which no one is fully responsible. Second, no authority to perform the tasks. Municipalities and schools which are ultimately delivering education services have limited control over the administration of personnel-- teachers--which is the responsibility of Departments and, as discussed above, lack of freedom to contract with non-government providers. Third, overlap and confusion between instruments and objectives. Given the existing ambivalence between a principal agent and a local choice model for the delivery of public services in Colombia, education is being financed through earmarked transfers, co-financing funds and own resources. Signals are mixed, and although the amount of resources being transferred is quite significant there is still the sense that the newly assigned responsibilities are not being properly funded. And fourth, no accountability mechanisms. With blurred responsibilities for the provision of education services, no one is then really accountable. 5.28 To obtain the full benefits of the decentralization process it is critical to attain greater clarity in the responsibilities of the different levels of government. It also requires the design of new systems of transparent formula-based funding (e.g., capitation transfers for education and health where a principal agent model could be applied, and general purpose grants together with co-financing--to provide a relative price signal with central government priorities--for public services where the local choice model should prevail) and systems of control that strengthen the incentives and capacities of executing agencies while retaining strategic control at the center. With both the principal agent and local choice models there is the need to emphasize the concept of agreement between levels of government. It would be advisable to more forcefully pursue a partnership concept (referred to previously in this report) in which local governments enter agreements with the central government over objectives and responsibilities rather than the latter directing local governments and seeking to intervene in detail in their management of programs. This should be done separately for each service as the responsibilities for different levels of government and the means of delivery vary widely. Colombia: Paving the Way for A Results-Oriented Public Sector 37 5.29 In pursuit of the above changes, SYNERGY can play a catalytic role by providing a transparent framework to identify responsibilities, resources and accountabilities. The decentralization process itself would then generate demand for the information resulting from the evaluation system. There would be a push for SYNERGY from the bottom as local governments would have a stronger need for information to improve their performance in making limited funds go further. Faster and higher quality information would then be available to the center as local governments develop their own evaluation capacity. Experience has shown that where there is a shift to a results orientation, considerable gains are attained simply by eliminating many unnecessary traditional input and workload indicators demanded by higher levels of government. Budget Incentives 5.30 In the previous section on Macroeconomic Control and Microeconomic Performance, specific proposals for a budgeting framework were suggested to provide a more strategic approach to resource allocation decisions. A unified budget, developed within a medium-term framework and hard budget constraints on ministries and agencies was recommended. But the budgeting process needs to go further to support fully a results orientation. It should provide incentives to continually enhance public sector performance, and to ensure resources are most effectively and efficiently used in attaining targeted results. 5.31 To identify ways to enhance the links between budgeting and performance, it is recommended that the government undertake a review of budgetary procedures with the objective of minimizing the constraints they impose on agency (and local government) performance. This night follow the same lines as the review of bureaucratic processes by the Presidential Advisory Council for Public Administration. Particular consideration should be given to the following measures: * providing agencies with increased flexibility within the budget year in the use of administrative resources, and between staff and non-staff expenditures; - encouraging greater use of charges on the users of services; * defining mechanisms for agencies to share in efficiency gains--such gains might be imposed as a small required efficiency dividend each year, but could be refined later when the links between inputs and outputs are better known; and requiring budget submissions to include performance information, to identify the link between programs and their costs, and to indicate what evaluation information underpins them, what will be the measures on which success or otherwise will be judged and a related evaluation strategy. 5.32 The results of this review should set the basis for a comprehensive reform of the Budget Law, which should be proposed to Congress next year. 5.33 Putting in place a fully results oriented planning and budgeting system will require a long- term effort. To reap the full benefits of SYNERGY it is important that the reform effort begins now. The approach being recommended here is based fundamentally on changing the budget process so that it both encourages and requires a results orientation. This will require the center Colombia: Paving the Way for A Results-Oriented Public Sector 38 to shift away from involvement in detailed line item decision-making and elimination of detailed line item appropriations in order to provide the necessary flexibility to line managers. Over time, the budget system will link more closely with the changes at the organizational level under SYNERGY which will clarify outcomes, outputs and their costs. As information on performance increasingly influences budget decision-making, consideration will need to be given to whether this should have any further impact on the structure of appropriations. Personnel Management 5.34 Promotion of human talent, and encouragement to focus on results, in the service of the state are necessary conditions of a results-oriented public sector management. At the end, it is people defining policy strategies, implementing projects and providing the public services. If they are not properly trained and motivated, the scope for improving public sector performance will always be limited. Putting in place the changes proposed above will go a long way towards providing this motivation. It will not, however, be adequate on its own. 5.35 At this time, the Administrative Department of Public Office is developing a program to reform the civil service and personnel administration in order to promote public administration based on results. The Advanced School of Public Administration is being restructured and training networks are being developed to educate public employees in results-oriented management. A system for individual performance appraisal, motivation and incentives is also being developed. 5.36 The current emphasis in personnel management on the development of a professional civil service is strongly supported, but the approach being taken to the many issues involved appears too centralized. For a results-oriented public sector it is necessary that managers are given sufficient flexibility to administer their personnel effectively. Balancing these freedoms with the central controls needed for other objectives is required but the general approach should be to set standards from the center and delegate to local managers the responsibilities for obtaining and managing staff--so they can be held responsible for results. 5.37 In support of the results-oriented management approach, individual performance appraisal should move to include more explicit performance specifications. Appraisal results need to be linked to achievement of objectives. Promotion and remuneration, in turn, should be linked to personnel appraisals. Although remuneration policies are a complex topic, one element of importance to results-oriented management is the need to offer pay scales at a level that enables the public sector to recruit and retain the people needed to perform effectively. 5.38 Given the experience in other countries it is important to recognize that once greater flexibility is given to ministries and agencies within the budget year in the use of administrative resources and between staff and non-staff expenditures, there may be a tendency to reduce the payroll. Furthermore, it could be expected that the transfer of responsibilities to lower levels of government under the decentralization process will create downward pressure on personnel numbers in central government. While such an eventuality would be consistent with improving efficiency and effectiveness it will generate opposition of labor unions and can create an opposing force to SYNERGY and ROPM. This has to be tackled within a coordinated strategy for public Colombia: Paving the Way for A Results-Oriented Public Sector 39 sector reform, in which civil society plays an increasing role in creating a positive environment for the reform process. BIBLIOGRAPHY Aucoin, Peter (1990). "Administrative Reform in Public Management: Paradigms, Principles, Paradoxes and Pendulums", Governance: An International Journal of Policy andAdministration, Vol. 3, No. 2, pp. 115-137, April 1990. Boston, Jonathan, et. al. (1991). Reshaping the State: New Zealand's Bureaucratic Revolution, Oxford University Press, Auckland. CONPES, DNP (1994). "Appraisal of Results in the Public Sector Nationwide", CONPES Document 2688-DNP, DECER, Approved Version, Santafe de Bogota, Colombia, January 21, 1994. CONPES, DNP (1995). "Public Administration Directed at Results", CONPES Document 2790-Vice President of the Republic-DNP, DEE-Presidential Advisory Board for Institutional Development, Approved Version, Santafe de Bogota, Colombia, June 21, 1995. Cook, Thomas, Jerry Vansant, Leslie Stewart and Jamie Adrian (1995). "Performance Measurement: Lessons Learned for Development Management", World Development, Vol. 23, No. 8, pp. 1303-1315. Department of Justice, USA Government (1995). "Government Performance and Results Act: Department of Justice Manager's Handbook on Developing Useful Performance Indicators-Managing for Results", Justice Management Division, Management and Planning Staff, Washington, D.C., April 1995. DNP (1992). Controly Evaluaci6n de la Gesti6n Puiblica: Los Desafios de la Nueva Constituci6n, Servigraphic Ltda., Santafe de Bogota, Colombia. DNP (1995). Informe al Congreso de la Repuiblica (Agosto de 1994 - Julio de 1995), Gente Nueva Editorial, Santafe de Bogota, Colombia. Doyle, Peter (1994). "Setting Business Objectives and Monitoring Performance". European Management Journal, Vol. 12, No. 2, pp. 123-132. Efficiency Unit (U.K.) (1991), Making the Most of Next Steps: The Management of Ministers' Departments and Their Executive Agencies, Report to the Prime Minister, HMSO, London, May 1991. Hatry, Harry P., James R. Fountain, Jr., Lorraine Kremer, and Jonathan M. Sullivan, eds., (1990). Service Efforts andAccomplishments Reporting: Its Time has Come, Government Accounting Standards Board, Norwalk, Connecticut. Colombia: Paving the Way for A Results-Oriented Public Sector 41 Holmes, Malcolm and David Shand (1995). "Management Reform: Some Practitioner Perspectives on the Past Ten Years", draft paper, The World Bank, Washington, D.C., May 1995. Jordan, Grant (1992). "Next Steps Agencies: From Managing by Command to Managing by Contract?", Aberdeen Papers in Accountancy and Finance and Management, W6, University of Aberdeen, Aberdeen. Keating, Michael and Malcolm Holmes (1990). "Australia's Budgetary and Financial Management Reforms", Governance: An International Journal of Policy and Administration, Vol. 3, No. 2, pp. 168-185, April 1990. Kemp, Peter (1990). "Can the Civil Service Adapt to Managing by Contract?" Public Money Management, Vol. 10, No. 3, Autumn 1990, pp. 25-31, Basil Blackwell, Oxford. Kettl, Donald F. (1994). "Reinventing Government? Appraising the National Performance Review. A Report of the Brookings Institution's Center for Public Management" (CPM Report 94-2), Brookings Institution, Washington, D.C., August 1994. Langan, Patricia E. (1994). "Notes on U.S. National Performance Review Customer Service Standards--Developing and Using Performance Indicators", mimeo, The World Bank, Washington, D.C., November 2, 1994. Miller, Michelle and Thomas Miller. Citizen Surveys: How to Do Them, How to Use Them, and What TheyMean. ICMA, Boulder, Colorado. National Academy of Public Administration (US) (1993). A Guide to Designing Performance Indicators (A Report by a Panel of NAPA for the US Department of Education), Washington, D.C., November 1993. National Academy of Public Administration (US) (1994). "Toward Useful Performance Measurement: Lessons Learned from Initial Pilot Performance Plans Prepared under the Government Performance Results Act," Washington, D.C., October 1994. National Performance Review Office, Office of the Vice President of the United States (1994). Creating a Government that Works Better and Costs Less: Status Report (Report of the National Performance Review), Washington, D.C., September 1994. National Performance Review Office, Office of the Vice President of the United States (1994). Putting Customers First: Standards for Serving the American People (Report of the National Performance Review), Washington, D.C., September 1994. Prahhan, Sanjay (1995). "Evaluating Broad Allocations of Public Spending: A Methodological and Data Framework for Public Expenditure Reviews", draft paper Public Colombia: Paving the Way for A Results-Oriented Public Sector 42 Economics Division, Policy Research Department, The World Bank, Washington, D.C., February 21, 1995. Presidencia de la Repuiblica, DNP (1995). El Salto Social: Plan Nacional de Desarrollo, Ley de Inversiones, 1994-1998, Santafe de Bogota, Colombia. Scott, Graham C. (1995). "Government Reform in New Zealand", draft paper prepared as Visiting Scholar, [MF, Fiscal Affairs Department, Washington, D.C. Scott, Graham, Peter Bushnell and Nikitin Sallee (1990). "Reform of the Core Public Sector: New Zealand Experience", Governance: An International Journal of Policy and Administration, Vol. 3, No. 2, pp. 138-167, April 1990. SDPE-DNP (1994). "The Indicative Plan: A Management Tool", Santaf6 de Bogota, Colombia, mimeo, November 1994. SDPE-DNP (1995). "Methodology for Program Management in the Colombian Public Sector--Synthesis Document for Revision", Santafe de Bogota, Colombia, mimeo, August 1995. SDPE-DNP (1995). "SYNERGY: The Proposed System--Background Paper", Santafe de Bogota, Colombia, mimeo, August 1995. Task Force on Management Improvement (Australia) (1992). he Australian Public Service Reformed: An evaluation of a decade of management reform, Australian Government Publishing Service, Canberra, December 1992. Wholey, J. and Harry Hatry (1992). "The Case for Performance Monitoring", Public Administration Review, November/December 1992, Vol. 52, No. 6. Wiesner D., Eduardo. (1993). From Macroeconomic Correction to Public Sector Reform: The Critical Role of Evaluation". (World Bank Discussion Paper, No. 214). The World Bank, Washington, D.C. World Bank (1991). Colombia: Public Sector Expenditure Review, Report No. 7891- CO, The World Bank, Washington, D.C. World Bank (1993). Colombia: Public Financial Management Project, Staff Appraisal Report No. 12167-CO, The World Bank, Washington, D.C. World Bank (1994). "Evaluation Capacity Development: Report of the Task Force". Washington, D.C., July 5, 1994. Colombia: Paving the Way for A Results-Oriented Public Sector 43 World Bank (1995). "Review of Public Expenditure Work", draft paper Office of the Vice President Development Economics, The World Bank, Washington, D.C., January 1995. Zeithaml, Valarie, A. Pasasuraman, Leonard Berry (1990). Delivering Quality Service: Balancing Customer Perceptions and Expectations, Free Press, New York. ANNEXES ANNEX 1 DEFINITION OF KEY PERFORMANCE MEASUREMENT TERMS Below are suggested definitions for a number of key terms relating to performance measurement and the National Evaluation System: 1. Mission is the reason that an organization or a program exists, a broad statement of purpose. 2. Program is an organized endeavor of an agency that has measurable objectives that have a direct, primary relationship to the agency's mission; has a defined budget and staffing; and has an identifiable target population. 3. Activities are the "means to the end;" actions, with specific objectives, that make up a program and lead to the attainment of that program's goals and objectives. 4. Performance. This covers the work an organization performs in converting inputs, through a process, to outputs, to outcomes. Inputs, process outputs and outcomes can be considered to fall along a "performance continuum." 5. Performance Measurement. This refers to measuring the performance of a program, a service, or fimction. The various categories of what can be measured are listed below under "Performance Indicators." 6. Performance Indicators are signs to help a manager answer the question, "How will we know when we have been successful?" They refer to what specifically is to be measured for each aspect of performance, that is, the specific numerical measurement that are to be made, such as the "number of customer complaints" or "percentage of customers that report being satisfied with the service they received." The various categories of performance indicators are as follows: a. Inputs. The resources used to produce outputs or outcomes. Inputs are usually expressed as amount of expenditures or amount of staff time (such as number of employee hours or days). b. Outputs. The products and services produced by a program or activity. Outputs results from internal activity or effort. Outputs are important for measuring internal work performance, but do not in themselves indicate the extent to which progress has occurred toward achieving the program's purpose. c. Outcomes. Events, occurrences, or conditions that indicate progress toward achievement of the purposes of the program. Outcomes can be measured in terms of the extent to which these occur. Outcome indicators can also reflect the quality of the service delivered or customer satisfaction with different aspects of service delivery. It is useful to distinguish "intermediate outcomes" from "end outcomes." Programs usually have both types of outcomes. Colombia: Paving the Way for A Results-Oriented Public Sector 45 c-I.End Outcomes. The desired end or ultimate results that are hoped to be achieved by the program's activities. These results are directly related to the agency's mission. Examples: clients whose condition improved, reduced crime, clean air or water. c-2.Intermediate Outcomes. Events, behavior, conditions, etc. that are effects of government agencies and are expected to lead to the ends desired, but are not themselves "ends." Intermediate outcomes have several advantages for program managers. These outcomes often (but not always) occur earlier in time than end outcomes, and thus, may provide more timely feedback. Using intermediate outcome indicators can ensure that programs (and managers) receive credit for early events, especially it if takes a long time before the end outcomes of program activities can be measured. End outcomes, however, should not be neglected. Intermediate outcomes are usually more under the control of programs than are end outcomes. End outcomes are more likely to be influences by factors that are outside program control. Program managers should be encouraged to identify the sequence of outcomes sought, as illustrated in Figure 2 in Chapter 2, and distinguish intermediate from end outcomes. c-3.Quality. Refers to characteristics of how a product or service is delivered, not the results of the service. Quality characteristics that might be measured include timeliness, accessibility (of customers to a service), accuracy, and conformance to requirements. Quality, as defined here, is an intermediate outcome. Examples: * Percentage of cases in which the time from request for service to when it was provided was within two days. * Percentage of laboratory tests that correctly identified the water pollutants. c-4.Customer Satisfaction. Refers to the level of satisfaction expressed by customers on some aspect of the provision of a service to them, such as its timeliness, accessibility, and pleasantness with which the service is delivered. Customer satisfaction is usually an intermediate outcome but sometimes is an end outcome, such as customer overall satisfaction with their experiences in national parks. d. Impacts. These indicators are those that show the extent to which a program actually produced effects on client populations (usually based on more in-depth program evaluations). e. Efficiency and Productivity. These terms describe ratios of inputs to outputs or to outcomes. Efficiency and productivity are essentially equivalent, differing only in the way in which the ratio is expressed. Efficiency is defined as the ratio of inputs to outputs (or outcomes), and is usually expressed as the number of employees or amount of employees time per unit of output (or outcome). It is sometimes referred to as "unit cost." Productivity is the ratio of the amount of output or outcome to inputs, traditionally expressed as the amount of output (or outcome) per unit of input. Colombia: Paving the Way for A Results-Oriented Public Sector 46 Output Examples: * Number of persons or days expended per repair made. (Efficiency) * Number of prisoners transported divided by the cost of transportation. (Productivity) Outcome Examples: * Repair cost per kilometer of roads that were repaired to satisfactory condition. (Efficiency) * Number of criminals convicted per number of prosecutor-days. (Productivity) 7. Target Population. This term refers to the specific categories of the population that a program affects. (Such population groups might be categorized by geographical location of residence, by age group, by ethnicity, gender, by income group, by type of handicap, etc.). 8. Performance Targets. These are numerical target levels of performance against which actual achievements can be compared. Targets probably should be set by each program for each indicator for each future reporting period. For example, the statement to achieve an 85% success rate in helping clients in 1997 includes the target of 85%. Program Evaluations. This terms refers to in-depth examinations of particular programs to estimate the effects of those programs on outcomes. These evaluations usually involve various technical procedures such as statistical analyses. These are usually expensive and, thus, an agency is only likely to be able to sponsor a few of these each year. ANNEX 2 IMPLEMENTATION OF PERFORMANCE MEASUREMENT: MINISTRY OF ENVIRONMENT - POSSIBLE PILOT PROJECT As an example of how one would implement a performance measurement pilot project, following are the steps that would need to be taken to strengthen performance measurement in the Ministry of Environment in Colombia. 1: Client identification and data use The principal customers are the citizens of Colombia. Before implementation, it is necessary to establish working relationships, in particular, the identification of Ministry of Environment counterparts in the Department-level Corporations, NGOs and focal points in related sectoral ministries. NGOs like ECOFONDO, with their links to over 200 organizations throughout the country, would be an appropriate counterpart. There are also three NGO networks (recycling, urban environment and national reserves) that might be appropriate collaborators. Representatives of departments and municipalities should probably also be included. These should be constituted as a work group that will follow the process through all of its steps to communication of the results; this is part of the training component implicit in performance-based management. The immediate objective is to supply appropriate data to the Ministry, as a baseline to monitor progress in implementing the Indicative Plan. The same data will be useful to the partner NGOs and Corporations. Part of the function of the data is to provide a substrate for dialogue with the civil society and government agencies. 2: Framing of issues to be addressed Data needs of the Ministry for environmental protection have to be defined. Some indicators have been identified (listed in the software of the Ministry Management Information System). A first concern is to establish baselines for as many as possible of the areas of concern in the Ministry. A second concern is to identify an important issue requiring early resolution, where the flagship can demonstrate the functioning of SYNERGY. This choice would be addressed by the work group identified in step 1. Additional issues, or approaches to the issues implicit in the Management Information System, might be discussed with the Departmental Corporations, the NGOs, and the other collaborating in the initiative. A focus group format has proved valuable for this task in other countries. The idea is to chose the Indicative Plan components to be addressed. Ideally, a T-shaped design is preferred: a cross-cutting board sweep of issues and services accompanied by a single in-depth, sub-sectoral evaluation. Thus, baseline data from the available data (especially data identifying where conditions are worse) should help indicate the importance of each Plan component. It should be possible to identify issues of national importance, regional and municipal importance. Colombia: Paving the Way for A Results-Oriented Public Sector 48 The focus probably should cover different types of polluters and pollutants. As has been done in Nicaragua, representatives of the Ministry of Environment, peer ministries, businesses, NGOs and communities might be interviewed to get a fuller view of the issues. 3: Identification and analysis of currently available service delivery data Performance measurements should begin with the critical analysis of existing data. Data available on environmental services, consumer perceptions and/or observations of the environment provide the backdrop to the design. A 1-2 day analysis seminar could follow a critical analysis of existing data, permitting discussions of implications with stakeholders. This will help to identify the areas where additional data are required. 4: Identification of pilot municipalities Criteria for selection include: extent of the pollution problem, likely ability of the municipality and its citizens to benefit from the project, cost, timeliness, quality, and commitment of management. Ideally, the selection should permit representation of departments to have an idea of what happens in the country at large. An appropriate number of municipalities to balance issues of credibility and feasibility might be 10 municipalities (each with 2-3 sites), in two or three departments that represent a spread of conditions in the country. The selection of Department Corporations should be done as early as possible, so that appropriate focal points can be drawn into the work group. 5: Preliminary design of baseline survey A diagnostic survey would be designed to identify the types of industrial, commercial and/or household behavior contributing to environmentally dangerous conditions. Data collection instruments, indicators and the survey process will be developed. The role in the overall National Evaluation System, and Indicative Plans will be identified as well as the data not available from existing sources. The baseline work might include: (1) business, household, and government facility surveys; (ii) an institutional review of local facilities in the communities reviewed; (iii) focus group discussions with community members; and (iv) key informant (e.g., key industry leaders) interviews. Data entry and analysis formats will be developed. Preliminary formats for the presentation of data for ultimate use by decision-makers will be developed. The size and representation of samples, degree of reliability, and cost of implementing surveys will be established. Logistics of field work, with due attention to security, will include timing of the overall initiative and, in each Department, the schedule of making contacts, data gathering, entry, preliminary analysis, feedback and systematizing qualitative data for aggregation. This can be done by a small group of experts. 6: Pilot testing of baseline instruments Surveys and observation studies using representative samples will be pilot-tested. Pilot testing would be combined with training of data gatherers. It would include: Colombia: Paving the Way for A Results-Oriented Public Sector 49 * supervisors * data collection (such as undertaking water or air quality tests, collecting samples for laboratory study, and undertaking interviews with businesses or households) * data entry * data analysis * institutional review * mock focus group with supervisors Refinement and finalization of instruments, as in their design, should be a collaborative task between national level and participating local planning entities. 7: Field work The data collection and focus group discussions, will be carried out in a number of communities in each of the participating municipalities. Data entry will be done using a public-domain data entry package. Government personnel might be the primary data collection personnel. However, NGO's and citizen volunteers might also be used, if properly trained, for some of the data collection efforts. This would both reduce costs and encourage non-government participation in public issues. 8: Data entry, preliminary analysis and local discussions Preliminary analysis will focus on those data of immediate use at local level. Local targets could be developed and agreed upon with clients representatives in the territorial planning councils. It is also possible to hold a Consulta Pzuhlica to discuss the preliminary implications. Other formats used for interpretation of specific results are focus groups. Additional discussions could be held at Department level. A major purpose of all these meetings is to identify actions to reduce environmental dangers. The group should identify who should do what and by when. Subsequent data collection work should be planned that is aimed at tracking the extent to which the actions succeeded--and making modifications to the action steps where needed. 9: Definitive analysis, development of performance targets, and publication of baseline information The fuller aggregated analysis, appropriately weighted to represent the sampled domain, would take into account both the quantitative and qualitative data. Baselines and targets emerging from this project might be published in line with the "It's a Dearl program. 10: General application of surveys and capacity building To improve environmental quality, a capacity building plan to continue data collection locally should be developed. The plan would also instruct local, regional, and national officials in how to use the data to improve the efficiency and effectiveness of the public sector. Colombia: Paving the Way for A Results-Oriented Public Sector 50 Summary of Deliverables 1. Data for the Ministry Management Information System. 2. Identification of specific problems addressed and corrective recommendations developed. 3. People trained, people sensitized. 4. Links with departments, municipalities, and NGOs consolidated. 5. Demonstration case of how SYNERGY supports communication between national, epartmental and municipal levels. 6. Identification of participatory modalities; experience with territorial planning councils. ANNEX 3 SELECTED ISSUES IN RESULTS-ORIENTED PUBLIC SECTOR MANAGEMENT NTRODUCTION The intention of this annex is to draw upon some of the ideas and experiences underlying results- oriented public sector management in other countries in a way that provides a useful perspective from which to comment in the main report on the proposed Colombian reforms. The topics are chosen on the basis of their apparent relevance to aspects of these reforms. MANAGING FOR RESULTS IN THE PUBLIC SECTOR Governments in many countries have sought to orient their systems of public administration towards the rational allocation of resources in the pursuit of the objectives of government and done so in lots of different ways over the last 30 years. The most widespread systems have been based on program planning and budgeting originating in the USA in the 1960s. There have been developments over the last 10 to 15 years, particularly in certain English speaking countries which have broken new ground. Starting in the United Kingdom, then Australia and New Zealand, more recently the United States, there have been new designs of public administration which have endeavored to bring to government a new philosophy of management which have been described as managerialism or the new public management. There are also relevant experiences in countries outside this group where the same basic ideas have been applied in different political contexts. Malaysia, for example, has adapted the British Citizens' Charter and has announced that it will adopt an output-based budgeting system similar to New Zealand's. Singapore is also moving to an output-based budgeting system. It also appears to be gearing its public administration in at least some departments towards holding public service managers responsible for the achievement of policy outcomes. While there are fundamental differences in the philosophy and detail of the management regimes installed in these countries, they have certain features in common which distinguish them from their previous systems and those of most other countries: * Clearer definition of the performance expectations of government organizations and the policies and programs they administer. * Greater delegation of authority to the managers of organizations to control the resources used in meeting these objectives. * Stronger personal accountability of managers for the performance of the organizations they run. * Tighter budgetary constraints on resources available to government institutions. * Improved reporting of the performance of organizations and a general increase in transparency of their operations. * Widespread structural reform in the allocation of functions to organizations. * A more commercial approach to the management of any business activities. Colombia: Paving the Way for A Results-Oriented Public Sector 52 * Greater focus on privatization of government business activities in a variety of ways-- in the United Kingdom and New Zealand there has been an extensive program of asset sales, and in all these countries, and in quite different ways, there has been a growth in the purchase of public services from private providers and the contracting out of various government functions (in some of the States and provinces of the United States and Canada particularly, there have been extensive and quite radical use of these approaches). * More attention is paid by government organizations to be responsive to the needs and preferences of citizens, clients and customers. These new approaches to management are concerned about the forces shaping the performance of organizations. The various actors in the government sector are seen as having motivations similar to those of people in the rest of the society, rather than obediently carrying out instructions in the pursuit of the public interest. From this viewpoint an organization is more likely to be successful in meeting the performance goals set for it if there is careful attention to the clarification of the rights and roles of the parties in the governance and management structures. Performance is improved by creating incentives backed by information flows which align the interests of the people in control of an organization with the interests of the stakeholders and, in particular, those responsible for setting the organization's goals. STRATEGIC POLICY INTEGRATION Effective governance can be said to involve the avoidance of political and managerial structures and processes which either distort policies away from the wider public interest, or undermine the successful implementation of policy through poor incentives and poor resource allocation within the management systems of government. In considering this it is conceptually useful to think about government as involving two broad relationships. One between the elected representatives of the people and the citizens themselves through the democratic process. The other is the relationship between those elected representatives and the public administrators whose task it is to implement government policy. With respect to the first relationship, politicians gain political power by proposing various policies in the pursuit of the ultimate objectives they believe the citizens want. The citizens make their choice of politicians who then must develop a strategy for achieving the results that people are looking for. This relationship can be seen graphically in Diagram 2. This diagram shows the feedback loop through which Executive Government is constantly seeking to maintain its mandate on the Legislature and ultimately the people through policies which generate the results or "outcomes" that are sought. Colombia: Paving the Way for A Results-Oriented Public Sector 53 Diagram 2 Governance and Public Sector Management I FEEDBACK S THE ' ONSTITUTIONAL % I EXECUTIV PEOPLE FRAMEWORKNENT Conventions tion C binet Elections 1 Referenda iny Advisors Media Development Strateaies Information Policies ~~~~~~~~~~Public Seco The second governance relationship is shown in Diagram 3 which shows the key elements of the feedback loop in which the Government's development strategy is translated into policy choices which are then implemented either through private or public sector service delivery in the form of "outputs" which in turn lead to results or outcomes. The assessment of the effectiveness of these activities is then fed back into the management processes, policy choices and the development strategy itself in a recurring cycle. Diagram 3 Managing for Results In the Public Sector |FEEDBACK | GOVERNMENT POIC o UTTSOCMEi DEVELOPMENT CHOICES b'PUBLIC SECIT R _ STRATEGY ..MANAGEMEN 4 Investnets Sst5Dliverables, fet /CulureNlues Purchase l* tAto Environ. Scan Transfers XJi Resources Information Informatlon :ommunicatlon Pln Colombia: Paving the Way for A Results-Oriented Public Sector 54 In a high performing government these feedback loops in the two diagrams above produce a spiral of constant improvement in policy design, efficiency in management and hence greater effectiveness in the achievement of results and in citizen satisfaction. In unsuccessful governments these feedback loops are distorted and do not lead to constant improvement, but to stagnation and declining performance. This failure of government institutions and processes is thought of in the economic literature as stemming from two main problems. In the feedback loop (Diagram 2) involving people, the Legislature and the Executive, policies can be distorted through the political pressures of special interests who succeed in getting the Government to design the policy in a way that serves their interest, but at the expense of some wider concept of the public welfare of the citizens as a whole. The main source of failure in the second feedback loop (Diagram 3) involving Executive Government and public administrators is distortions in the incentive structures for public institutions which divert their resources and efforts to some extent away from serving the policy goals that were set for them and towards other objectives such as expanding their own resources, maintaining their position of influence, etc. Unethical behavior in public institutions falls into this category of distortion. Viewed in this way effective governance can be seen to involve the avoidance of political and managerial structures and processes which either distort policies away from the wider public interest, or undermine the successful implementation of policy through poor incentives and poor resource allocation within the management systems of government. Achieving this requires very effective political leadership and cultures and values within public administration which, together with strong incentives systems to align the behavior of administrators with the wider strategic goals, lead to efficiency and effectiveness in management. Perhaps the biggest challenge facing any government which wishes to design and implement an ambitious development strategy is to maintain the necessary strategic coherence in order to fulfill these plans, which are disruptive to a lot of existing habits and institutions and fundamentally changing the way in which government business is done. The forces of resistance to change are powerful in government institutions everywhere and if the strategy is to succeed then powerful forces favoring that must be put in place. This involves aligning the policies, the politics, the communications with the people and the management incentives and systems throughout public administration at the various levels of government. This is a rare formula which few countries manage to maintain over the long periods of time needed. From a technical standpoint, the most important aspect of strategic coherence is to ensure that the policies themselves are in harmony, both at the micro-economic level and in the sense of adding up to a coherent medium-term macro-economic and social policy. While pragmatism is favored by the politicians in most countries, it can, if taken to an excessive degree, lead to ad hoc compromises in specific policy areas which remove the necessary strategic coherence of the policy framework as a whole. For example, poor monetary and fiscal policy can overwhelm the effects of specific industrial development or employment policies. Lack of integration between economic and social policy has led in many countries to perverse incentives to the citizens that can trap them into dependency or poverty. Colombia: Paving the Way for A Results-Oriented Public Sector 55 Establishing clear policy frameworks which are easily understood and can be applied comprehensively across large areas of government activity is perhaps the best way to ensure that resistance to change is overcome. This can be done by establishing a set of principles upon which a new policy regime will rest and setting clear goals and objectives and allocating resources with a management program to see that goals are met. This promotes understanding of what the policy framework is about and also shifts the focus of policy debate and public sector management to questions of implementation, rather than constant reconsideration of the original policy objectives. Clarifying the Role of Government Perhaps the most powerful organizing principle to give coherence to a government development strategy is to establish in clear, even ideological terms, what the Government sees its fundamental role in the society as being. Even though the practical details of any given policy will ultimately produce a lot of variation around any underlying principles, the design, management and communication of the development strategy will all be assisted by people in their roles in the society understanding how the Government intends to position itself with regards to its responsibilities in the economy and society. Clarity in this regard enables the other actors in society to be able to move forward on the basis of some confidence about what their role is in the scheme of things. The obvious example where this applies is with regard to the issues of privatization and government ownership of business activities and also the Government's attitudes with regard to the presence of foreign interests in the economy. MANAGEMENT SYSTEMS, STRUCTURES AND INCENTIVES This section considers some issues in the management systems, structures and incentives which are significant in relation to establishing the objectives for public sector institutions, motivating managers to achieve them, generating effective information on performance and building capacity for better service delivery. Alignment of the Management System with Key Objectives of Government How is the government to ensure that its broad strategic goals are reflected right across the many public institutions that are involved in detailed service delivery. The pressures of tradition and local circumstances can easily lead a public organization to head in directions which are not motivated by the strategic thrust from the center of government in any real sense. Strategic goals are often expressed in a very general way and it is common for previous programs and projects to be repackaged as consistent with the new strategic direction without really making the necessary changes. There are a variety of different approaches which can be taken, but at a general level success is likely to require the following: * there should be an agreement struck between the center of the government and the subsidiary institutions to ensure that the particular contributions which are required to ensure that the broadest strategic goal is advanced are delivered, * the strategic priorities should be clearly reflected in the allocation of resources; * there should be strong incentives to promote performance in terms of organizational efficiency and program effectiveness; Colombia: Paving the Way for A Results-Oriented Public Sectoi 56 * there should be monitoring of achievements in key strategic areas by the center to evaluate of the results. Clarifying the Different Interests of Government in its Various Activities It is useful in practical terms to define a Government's interests in the economy and society because different interests call for different policy and management frameworks. A Government's presence in any sphere of the society can be through establishing rights, regulating the behavior of others, taxing or subsidizing activities, people and institutions, owning businesses providing either public or private services and purchasing service from either public or private providers for itself or on behalf of others. Each of these interventions should be chosen, on the basis of the comparative advantage of the intervention over others in the pursuit of particular policy objectives. There should be general principles which establish and guide the use of each of these classes of intervention in any particular circumstance to ensure that there is coherence and harmony across the society as a whole in the way the policies are applied. Failure to do this can lead to contradictions between policy areas and loss of performance over the strategy as a whole. Of particular interest in terms of the Government's relationship with public organizations generally, is the distinction between the purchase and ownership interest. This distinction is fundamental to the performance management system in New Zealand, for example, where it is proved useful in setting and evaluating performance objectives for institutions. Basically the ownership interest is concerned about the rate of return on the resources used by the organization while the purchase interest concerns the price and quality of the service which is emerging. Structuring the Functions of Government to Promote Performance through Allocating Functions, Roles and Responsibilities The allocation of functions to government organizations and the relationships between them can have powerful effects on the two fundamental feedback loops in the diagrams above by either creating or removing distortions in those loops of the kind noted. Many of the problems of inefficiency and ineffectiveness in traditional bureaucratic organizations is contributed to because of the incentives in the information flows created by those structures. Poorly designed structures can lead to the capture of policy advice by providing agencies, problems associated with managing incoherent conglomerate organizations, concealment of information flows inside organizations which should be available externally. The following is a list of structural reforms in core government functions that would be relevant to most governments concerned with these issues: * Separation of ownership and purchase responsibilities. * Separation of policy from operations. * Separation of funding, purchasing and provision of services. * Competition between service providers. * Reallocation of functions for focus, synergy and information. Colombia: Paving the Way for A Results-Oriented Public Sector 57 Specification of the Performance Objectives of Government Institutions The manner in which performance objectives are specified will have a powerful effect on the organization's activities assuming that the incentives systems are properly aligned with the Government's objectives. There are some useful concepts which can be used to help specify performance in a way that promotes its achievements. It is commonly useful to distinguish between the performance of an institution which is charged with making some contribution to the achievement of a policy objective from the effectiveness of the policy itself. While for simpler policies this might not apply it deserves serious concern wherever there are a lot of influences on a policy outcome that lie beyond the control of the managers of the relevant service delivery organization. This raises the question of distinguishing between "outputs" and "outcomes". Outcomes are the effects of government services and decisions on citizens. For example, higher levels of education or reduced incidence of disease. Outputs, on the other hand, are the services supplied by the Government to citizens, e.g., tuition and water purification. These outputs require inputs for their production, such as teachers' salaries and water purification equipment. While there is no argument that public sector management should be directed towards the achievement of outcomes there are important practical issues which would lead to the conclusion that outputs are a better concept for specifying institutional performance than outcomes in certain circumstances. In particular, if the management philosophy is generally based on concepts of performance management and accountability then outputs have an advantage over outcomes in terms of their practical use in performance specification and assessment. For most public service activities the ultimate outcome is not controllable by the managers of the organization providing relevant outputs. A management system based on personal accountability cannot operate in an environment where this applies. In New Zealand the output concept is used as the center of the performance setting for heads of government agencies in part because deep delegations of authority for managing inputs were to be balanced by a clear definition of outputs. The conventional criticism of this approach is that it motivates managers to produce outputs which may not in reality be promoting the achievement of the outcome as much as might otherwise have been possible. In the environment surrounding the performance contract, however, there are strong incentives for the manager to seek to pursue the outcomes that the politicians had in mind. It would be a relatively short time before a department producing outputs that were seen to be ineffective in contributing to outcomes was regarded as malfunctioning. A Government concerned to install a system of clear accountabilities for individual managers will need to establish an output-based performance specification in order for this to work, and give deep delegations to managers for the management of resources. Where assessment of personal performance is not an important part of the management system then the looser specifications of performance objectives which are generally involved if outcomes are used may be acceptable. The choice also has implications for budgetary systems and cost-accounting systems to assess the cost of achieving objectives. Colombia: Paving the Way for A Results-Oriented Public Sector 58 In this context, the concept of accountability is taken to be the relationship between two parties when: * one party has performance expectations of the other party; * managerial authority is delegated to the party of whom performance is expected; * consequences result from the success or failure of performance; * performance information is available. Where an accountability-based management system is being used, the potential distortion from using an output-based definition of performance should be overcome by adding or substituting outcome-based objectives where the system can be structured so that they are expressed in a way that the accountable managers can have reasonable control over them. For example, in health policy it may be feasible to hold the providers of health services accountable for ensuring that certain diseases which can be contained through immunization programs are eliminated. This would be a superior objective to rewarding them for the number of inoculations given as it would motivate them to focus the immunization program on those who are most at risk of actually catching the disease. Management Systems and the Softer Aspects of Management It would be accepted by most management experts that the following items are key elements of effective management in most organizations: * strategic vision; * effective leadership; * clear objectives and business plans; * resources allocated to meet objectives; * incentives to produce results; * freedom to manage; * systems that support and empower; * effective communications; and * values congruent with organizational mission. It is apparent from this list of management system characteristics that they are a mixture of systematic rational management techniques and items which depend on the human culture and capabilities of the organization. These can be labeled the hard and soft sides of management respectively as noted in Diagram 3. Management experts in both public and private sectors see the hard side of management as establishing a set of necessary conditions for effective management. These systems set afloor under performance and enable the people to whom an organization is accountable to assess whether the organization is achieving the objectives that were specified. The softer aspects of management are generally thought to provide the basis for higher levels of performance where the achievements are beyond the minimal requirements established in the formal system, and instead show high levels of innovation, motivation and strategic awareness in searching constantly for improvements. Colombia: Paving the Way for A Results-Oriented Public Sector 59 Seeking to increase organizational and institutional capacity is, therefore, not just about management information systems, training and development strategies, it is a much more subtle process of building on top of those necessary elements the characteristics of leadership and organizational culture which high performing organizations display. Financial Management Fundamentally the purpose of financial management systems in the public sector is to ensure that the resources entrusted to the Government by the citizens are acquired and used lawfully, efficiently and effectively. A typical description of the topic describes the scope of financial management as encompassing the functions and processes of: * planning and programming; * budgeting; * budget execution and accounting; and * audit and evaluation. Diagram 4 The Financial Management Process PLANNING & PROGRAMMING AUDIT BUDGETING EVALUATION BUDGET EXECUTION & ACCOUNTING In 1985 the General Accounting Office of the United States set out some principles which organizations saw as being a guide to financial management reform and these are as follows: 1. Use a structured planning and programming process for evaluating and choosing alternatives for achieving desired objectives. 2. Make resource allocation decisions within a unified budget. Colombia: Paving the Way for A Results-Oriented Public Sector 60 3. Integrate budgeting and accounting. 4. Use accounting principles which match the delivery of services with the cost of those services. 5. Encourage financial accountability. 6. Measure outputs as well as inputs. 7. Prepare consolidated reports. Sound financial management has to rely on an integrated system of budget formulation, budget execution and accounting which focuses on the cost of past and future decisions. Similar descriptions of the field of government financial management from an international perspective are found in World Bank and International Monetary Fund documents. Broader definitions of the scope of public financial management include fiscal policy, cash management, debt management, value for money auditing, efficiency in the provision of public services and financial management systems to support results-oriented government management systems. As a component of a results-oriented management system, financial management needs to achieve the list of objectives set out above, while in addition dealing with some specific issues. In a results-oriented system it is critical that the financial management systems allocate resources on the basis of the objectives which the government is seeking to achieve. A traditional input- oriented financial control system does not do this so that the government does not have processes for ensuring that scarce resources are going to its important priorities. Also in the pursuit of efficiency and effectiveness the financial management system should produce accurate information on the full costs of pursuing policy objectives and account separately for the resources consumed in the administering agencies. It is useful in this regard to distinguish and account separately for the purchase and ownership interests it has in a public institution as noted earlier. Doing so will provide the government with information about the assets and liabilities and resources consumed in these organizations. In a decentralized management environment, the financial management incentives and disciplines that are imposed on service providers are a powerful influence on efficiency and effectiveness. The key principle in designing a financial incentive framework is to ensure that there is a balance between freedom and accountability so that financial freedoms are not misused on the one hand, or that managers do not have sufficient freedom to be able to use resources effectively on the other. What is the appropriate degree of decentralization of financial management depends on the total management context and policy objectives that a country has. While the countries that have introduced the greatest degree of financial delegation to managers report improvements in efficiency, those countries have had high levels of institutional capacity and a reasonably ethical civil service. The freedoms were introduced incrementally, starting from a position of effective central control of line item financial management. A very important issue to be considered in any country seeking to install a devolved financial management system, but beginning from a different point, is whether it is sensible to build a well- functioning, but relatively centralized system of financial management first, and then introduce greater freedoms at a later stage. Colombia: Paving the Way for A Results-Oriented Public Sector 61 The design of financial management and budget systems can greatly assist in solving the fundamental problem of harmonizing the need for central accountability for resource use and sound fiscal policy with the requirements for decentralization of management authority to enhance efficiency and innovation. This harmonization can be achieved by installing a financial management system which includes: * a medium-term financial strategy * capped funding * a unified budget * an integrated financial management system These are discussed in some detail in Chapter 5 of the main body of this report. Together they can create a financial management system that is both 'tight' and 'loose' at the same time. While freedoms can be granted within it to managers to escape central input controls there are medium term fiscal objectives set for the aggregate expenditures. Capped budgets enable these to be met through managing fixed allocations when the financial information systems are comprehensive. Information Systems Financial management information systems are one component of a total management information system that a well-run government requires. There should be integration between the financial system and the wider system to ensure that there are common bases for collecting and reporting on the different aspects of planning and performance across policies and programs. This means that informational requirements for management should be designed to reflect the fundamental accountability structures of the management system. In highly centralized systems financial and other management information will all flow to the center which, retains detailed controls. The Brazilian system of financial management is a centralized system of this kind. In a decentralized system the key principle is that information should be passed to those people with the authority and the incentives to take action on it. In such a system the information flows would fit the following pattern: * information about the monitoring of critical success factors is passed to the people who have authority to respond quickly in the work place, a lot of this information is very decentralized, although information about the achievement of the strategic goals is passed back to the center; * management information needed to prepare standard accounting reports, annual accounts, etc. flows routinely to the center for external accountability purposes; * central management has an internal or management assurance function that routinely checks the key operations of organizations and has the capacity to respond to identified failures. Institutional Capacity For a successful results-oriented public sector management system, particularly one that involves substantial decentralization, it is fundamental that organizations have the capacity to carry the responsibilities they are assigned. The politicians and central organizations are still ultimately accountable for the performance of decentralized managers who they do not have under their Colombia: Paving the Way for A Results-Oriented Public Sector 62 direct day-to-day control. Control is through the indirect techniques of establishing and monitoring objectives while giving substantial freedoms to managers as to how they achieve their goals. It was noted above what the key elements of effective management are and it can be seen by reference to that list that for most government organizations in most countries there is a gap between what exists today and what is necessary for the kind of management capacity, in particular, that results-oriented systems require. Compared with the traditional bureaucracy there is a complete change in mind-set required in order to imbed the accountability of the organizations for the achievement of its goals. The traditional government organization has a culture of compliance with central controls and is motivated to avoid risk taking. In emphasizing these points it is not intended that the provision of resources, information technology, training and development, strategies, etc. are not fundamentally important. They are, however, not sufficient to ensure a high performing organization. This requires a particular kind of leadership and a transformation in culture. Personnel Management Historically in developed countries the emergence of more sophisticated and higher quality public services over a comprehensive range of activities was preceded by reform in the civil service. For example, the early years of the twentieth century saw major public service reform movements in the United States, Australia, New Zealand and elsewhere. The French, Japanese and German traditions also historically had civil service reforms which established effective modern bureaucracies which took on unprecedented new tasks in the delivery of public services. In many of these countries today, there are movements to ease the central controls over the freedom of public servants to manage effectively. To date, this is not seen to have led to significant costs as the price of the efficiency gains which decentralized management has been proven to produce. While it remains a largely unresearched question as to whether a country without a strong tradition of effective and honest public administration can move straight to decentralized approaches, it is likely to be quite difficult because systems and approaches for inculcating values of efficiency, public service, honesty and innovation would need to be implemented at the same time as greater freedom and authority was being granted. In principle this may be possible but it seems rather more likely that centralized personnel management systems may be necessary at least for a while to strengthen ethics, culture, training and development in a country without a highly effective civil service. This is certainly not to argue that programs of privatization and the development of local government capacity need be held back so that the central government bureaucracy can be reformed as a precondition. The traditional way to strengthen civil service capacity is through the establishment of a merit based non-political civil service which typically provides employment protections for insiders against competition from non-government workers. Wages are usually well below equivalent market rates, though not always, and pay increases are awarded either on seniority or through rigidly controlled central wage adjustment mechanisms. Also typically promotions are on a Colombia: Paving the Way for A Results-Oriented Public Sector 63 competitive basis and reward the academic qualifications and seniority of the individuals with some measure of influence of the views of management. This rigid regime of management is to be found all over the world and has endured over a very long period of time. Clearly it has many advantages or this would not be the case. However, it does have a number of problems which can be seen in many countries. It is common for the rigid system to be evaded by bringing in casual workers paid on a daily or weekly rate and also contractors of various kinds to get around the system. Also in some countries the pay levels paid to civil servants are so low that the controls are evaded through ghost workers. Corruption has many complex causes but low wages and highly-interventionist systems of administration which give arbitrary powers to public servants obviously encourage it. Results-oriented public sector management raises significant challenges to traditional systems of personnel management in the public sector. Typically countries moving in this direction make steps toward freeing up personnel constraints that are administered from the center of government and give somewhat more authority to line managers in agencies. Many countries, however, find this extremely difficult however and are very concerned about these delegations. There are even countries which have been prepared to introduce delegations and decentralization in financial management while maintaining central control over personnel management. The essence of results-oriented management is establishing accountability for results and strengthening incentives for their achievement. It follows logically from this that the managers of delivery agencies must be able to delegate their accountabilities to their staff. They will then discriminate among staff on the basis of their performance in terms of meeting their objectives in a framework of personnel management which is much more like the private sector. There may be rewards and sanctions for meeting specified goals. Promotion may depend on performance rather than qualifications and seniority. In some circumstances, it may be efficient to introduce monetary bonuses for performance though there are many complications about this. Money is only one aspect of the motivation of workers in the public sector and has been called a "hygiene factor" to reflect the idea that the pay levels must be sufficient to ensure staff are contented without necessarily being subjected to complex performance bonuses. Bonuses have, however, been seen to be accepted and efficient in some agencies in some countries. The path of wisdom would appear to be to leave the decisions about bonus regimes at a decentralized level. It is almost certain that a government trying to define a single bonus regime that applies across the whole of the civil service will fail. The most decentralized system of personnel management existing in the world today is the one operating in the New Zealand government. This provides for the heads of government agencies to be the employing authority under New Zealand labor law and therefore able to negotiate with their staff independently from any central administration and control. Agency heads are on five year fixed-term contracts, other managers tend to be on individual contracts while other staff are typically represented by a union in an enterprise-based wage bargain. There is broad banding of pay scales with increments paid on the basis of proven performance with bonus systems in operation in some departments. There are no protections in the civil service regime for insiders against competition from outsiders, although New Zealand labor laws do give general preferences Colombia: Paving the Way for A Results-Oriented Public Sector 64 to insiders who are performing satisfactorily. This applies as much in the private sector as the public sector and has been established by Labor Court decisions rather than statute. In conclusion, it can be observed that a results-oriented management program in almost any national circumstances is likely to require a system of personnel management which enables the top management in agencies held accountable for delivering results to institute new management systems which enable them to acquire the human resources necessary to meet the task and to discriminate amongst individual staff members on the basis of their performance in delivering the results. It is unreasonable to expect managers to carry accountability for results if they have no control over the key resource in their agencies and cannot discriminate on the basis of the personal contributions of staff to organizational performance. IMAGING Report No: 15300 CO Type: ER
Группа Всемирного банка · Pre-2003 Economic or Sector Report
Colombia - Paving the way for a results-oriented public sector
Открыть оригинал документа
Полный текст размещён на сайте публикующей организации. lawenc.com индексирует метаданные и ведёт на официальный источник.
Полный текст
Основные сведения
Организация
Группа Всемирного банка
Тип документа
Pre-2003 Economic or Sector Report
Страна
Колумбия
Источник
Всемирный банк