Information systems for health sector monitoring in Papua New Guinea.

Information systems for health sector monitoring in Papua New Guinea R.E. Cibulskis1 & G. Hiawalyer2 Abstract This paper describes (i) how a national health information System was designed, tested and implemented in Papua New Guinea, (ii) how the system was integrated with other management information systems, and (iii) how information has been used to support decision-making. It concludes that central coordination of systems design is essential to make sure that information systems are aligned with government priorities and can deliver the information required by managers. While there is often scope for improving the performance of existing information systems, too much emphasis can be placed on revising data collection procedures and creating the perfect information system. Data analysis, even from imperfect systems, can stimulate greater interest in information, which can improve the quality and completeness of reporting and encourage a more methodical approach to planning and monitoring services. Our experience suggests that senior decision-makers and political leaders can play an important role in creating a culture of information use. By demanding health information, using it to formulate policy, and disseminating it through the channels open to them, they can exert greater influence in negotiations with donors and other government departments, encourage a more rational approach to decision-making that will improve the operation of health services, and stimulate greater use of information at lower levels of the health system. The ability of information systems to deliver these benefits is critical to their sustainability. Keywords Information systems/organization and administration; Management information systems/utilization; National health programs; Information management/organization and administration; Papua New Guinea (source: MeSH, NLM ). Mots cle´s Syste`me information/organisation et administration; Syste`me information gestion/utilisation; Programme national sante´; Gestion information/organisation et administration; Papouasie-Nouvelle-Guine´e (source: MeSH, INSERM ). Palabras clave Sistemas de informacio´n/organizacio´n y administracio´n; Sistemas de informacio´n administrativa/utilizacio´n; Programas nacionales de salud; Gerencia de la informacio´n/organizacio´n y administracio´n; Papua Nueva Guinea (fuente: DeCS, BIREME ). Bulletin of the World Health Organization 2002;80:752-758. Voir page 757 le re´sume´ en franc¸ais. En la pa´gina 757 figura un resumen en espan˜ol. Introduction Information on health needs, the delivery of services, and the availability and use of resources is important to all health service organizations. Such information can help an organiza- tion to increase its efficiency, effectiveness and responsiveness in several ways. First, it can help managers to align health system resources with client needs (a planning or prospective role) and determine whether their plans are progressing satisfactorily or whether there is a need for corrective action (a monitoring or retrospective role) (1). Second, information can be used to increase accountability within an organization and allow the public, their elected representatives, or donors to determine whether they are obtaining value for money (2, 3). Third, information can be used to market health programmes, secure appropriate levels of funding or engage public support (4). Fourth, information amassed over time can help an organization to learn what works and does not work and thereby provide valuable know-how, which can lead to greater efficiency in the production process (3). Experience suggests that health service organizations have difficulty acquiring information for use in any of these roles (5, 6). Successful information systems have been developed on a small scale, such as for an individual district or programme, but the sustainability of these initiatives is often questionable and their relation to national information systems development is not clear (4). This paper considers how a national health information system (NHIS) was established, how it was integrated with other information systems, and how the information collected has been used to plan and monitor services. It then considers some of the factors necessary for the development of successful and sustainable information systems. Background Information systems received much attention in Papua New Guinea after health service administration was decentralized to the provincial level in 1983. A management strengthening project sought to improve the use of data from existing systems by defining a core set of indicators and introducing computers for data processing at provincial level (7–10). While broadly successful in what it sought to achieve, the project did not attempt to revise data collection systems or consider how systems should be managed in a decentralized environment. These issues became increasingly important since, after 1 Takemi Program, Department of Population and International Health, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115, USA (email: cibulskis@aol.com). Correspondence should be addressed to this author. 2 Department of Health, Waigani, Papua New Guinea. Ref. No. 01-1177 Policy and Practice 752 Bulletin of the World Health Organization 2002, 80 (9) decentralization, national level managers ceased to have control over information systems; they could advise provinces on what data should be collected, but lacked the authority, capability or resources to ensure that their recommendations were heeded. Provinces could develop their own systems if they wished and differences gradually emerged between them, induced partly by donor projects that introduced programme- specific information systems in selected areas. Any uniformity in systems appeared to be due to several provinces ordering their stationery from the same printing company, which provided a limited range of forms in its catalogue. The haphazard state of the information systems made it increas- ingly difficult to aggregate data on a national scale, compare results between provinces, or maintain systems. This prompted the Department of Health to embark on a major programme to standardize its health management information systems. Introducing a national health information system System design A task force to review information systems was established in 1994, consisting of programmemanagers from the PapuaNew Guinea Department of Health and an international consultant. Nine out of the 20 provinces were visited and discussions held with health workers, managers, training schools, other government departments, nongovernmental organizations, and donors. One district was visited on a monthly basis for a year to understand information needs at health facility level. The review identified several problems in data collection, reporting and analysis, but indicated that there would be little benefit in collecting radically different data, changing reporting procedures or establishing differentmethods of analysis, as this would require major investments in training and support that could not be delivered using available resources. The major change proposed was to limit monthly reporting to a single form, which consisted of a folded sheet of A3 paper, andwhich combined the essential features of the seven or more forms in use. This would reduce the reporting burden on health facility staff and make it easier for provincial and national level staff to compile and analyse data. The form would contain data elements for calculating key indicators already in use and a few additional indicators for monitoring the latest national health plan. Instruments were also designed to support data capture, such as tally sheets, daily summary books, andmonthly analysis books. These represented the minimum amount of stationery required for data gathering. Health facilities would still need registers, patient record cards, and other materials but this stationery could not be afforded at the time the work was undertaken. Testing The proposed system was tested in only one province of the country owing to financial and time constraints. Stationery and training materials were prepared and computer software for processing data at provincial level was rewritten. The software was deliberately similar to the applications already in use but was upgraded to accommodate new data items and new possibilities for analysis, and made year 2000 compliant. A 3-day training workshop was held for staff from every health facility. The system was allowed to run for 6 months. Testing suggested that the number of disease categories should be increased to prevent health workers misclassifying certain conditions. Otherwise the new system was found to be more useful and easier to operate than previous systems. Implementation The need to train staff from all health facilities prevented the simultaneous implementation of the revised information system in all health facilities. Instead the systemwas introduced province by province over 12 months. This enabled visits to be made to the first provinces to check that the system was functioning as intended, that supplies of stationery were adequate, and that training was appropriate. Thereafter implementation was intensified and follow-up visits reduced. Most provinces started using the new system within 2 months of training, although workshops were repeated in three provinces where implementation was incomplete. Provinces were provided with a 2-year supply of stationery, but it soon became obvious that larger quantities were required if occasional shortages were to be avoided. Integration of systems Department of Health systems Health information systems are part of a wider management information system that provides data on population, resource availability and use (Table 1). While the NHIS was being developed, other components of themanagement information system were strengthened. A national inventory of health facilities was undertaken to obtain data on the number of facilities, staff, buildings and equipment. Such data are useful in their own right but are more valuable if they can be linked to health information. A major impetus to link information systems came from a reform of provincial and local-level government in 1995, in which district boundaries were revised to coincide with political electorates. This required health facility codes to be updated on all computer systems (the middle two digits of a health facility code represent a district) and allowed databases to be merged. Inputs could then be related to outputs and the efficiency of resource use explored. For example, performance in family health programmes was found to be related to staff:population ratios. Other sectors The health management information systemwas also linked to other sectors using the national census as a common base. Each health facility was traced to its nearest census unit. Because the approximate location of census units was known, maps could be drawn showing the distribution of health facilities, health conditions or any other feature that uses the common coding systems of the Department of Health. The maps are a convenient way of summarizing data and are useful for influencing decision-makers who are not overly interested in numbers (Fig. 1). Because of its intrinsic link to population, the system can also be used to determine how many people live within a certain distance of a health facility with specific attributes, e.g. the nearest facility with a doctor or radio. Such information is important because geographical distance influences use of services and is critical in deciding where to open, close, or upgrade facilities and locate staff (11–13). Other government 753Bulletin of the World Health Organization 2002, 80 (9) Health information systems in Papua New Guinea departments are using a similar scheme to identify the location of schools and other infrastructure. When such data are linked with health information they provide a tremendous resource for intersectoral planning and the targeting of poverty- reduction programmes (14). Using information Attempts were made to increase the analysis of information at all levels of the health system. Health workers were provided with tools and training in data analysis including a ‘‘Health centre record’’— a booklet designed to assess trends over time and provide a permanent record of a health facility’s activities. At provincial level, strategies for analysing indicators were reinforced by revising computer software and training provincial managers. At national level, analysis and dissemination were initially limited to an annual booklet on family health indicators. By limiting the analysis to a small area of activity inwhich datawere readily available, and considered important, the publication could be prepared in advance of an annual meeting of provincial health managers where the performance of individual provinces was publicly disclosed. Such feedback helped to increase reporting rates from 73% in 1994 to 85% in 1995 and to 93% in 2000. Improvements in information systems eventually al- lowed the government to undertake more extensive analysis and use information for assessing priorities in the national health plan. Targets were set on the basis of previous performance statistics and they helped managers to assess which health programmes were proceeding as planned and how each province was performing. Information was also used to market the plan to other government sectors and parliamentary leaders. Each Member of Parliament was presented with a graphical summary of health indicators, which showed the status of their constitu- encies in comparison to others (Fig. 2). The aim was to inform national leaders about the health situation and bring health issues to the forefront of the political agenda. More specifically it was hoped it would secure health budgets and persuade Members of Parliament to pay attention to health development in their own constituencies. Discussion System design The definition of management functions at each level of the health system and the identification of information needs and indicators are crucial steps in revising information systems (15, 16). It is customary to seek the involvement of health service managers in this task to ensure that systems meet their needs and to facilitate eventual implementation (16, 17). In practice, however, managers are not always sure of the information they Table 1. Components of a health management information system Type of information Information systema 1 Population National census Demographic and health surveys 2 Morbidity and mortality Health information system Demographic and health surveys 3 Health service activities Health information system Demographic and health surveys 4 Facilities and equipment Asset management systems Survey of health facilities 5 Human resources Workforce registration Employment and training records Survey of health facilities 6 Medical supplies Procurement and distribution system Health information system 7 Financial information Budgeting and accounting systems 8 Other sectors Government-wide systems Geographical information systems a Surveys and censuses are considered as part of a routine information system if they gather similar information at periodic intervals (as opposed to surveys and rapid assessments conducted on a one-off basis to answer specific questions). 754 Bulletin of the World Health Organization 2002, 80 (9) Policy and Practice require and the process can result in long lists of indicators, many of which cannot be measured (17, 18). In Papua New Guinea the definition of information needs was assisted by four factors. First, managers were familiar with concepts of indicators and monitoring as a result of previousmanagement strengthening efforts (7–10). Second, existing information was analysed and disseminated before system revision commenced — decisions on what indicators should be generated by an information system are easier if managers have data in front of them and can see how various indicators compare with targets, between geographical loca- tions and over time. Third, revision of systems coincided with the development of a national health plan that provided guidance on what should be measured. Fourth, the decision to use a single form for monthly data collection — which was taken following extensive consultation with health workers — placed limitations on the amount of data that could be collected. These factors resulted in an information system that was not very different from previous systems, but was more clearly defined, making it easier for health workers to adapt to it, reducing the need for major retraining and ensuring consistency in time-series. Implementation Good information systems may fail to take hold if they are poorly implemented. In Papua New Guinea several strategies were used to reduce the risk of failure (18–20). Widespread consultation ensured that staff were aware of the changes and had contributed to them, and that the system’s design was realistic. Testing on a limited scale helped to confirm the system’s appropriateness. Improved analysis and dissemina- tion of information before implementation created a more favourable climate in which to introduce new systems. It stimulated interest in information, generated support from senior levels in the Department of Health, and gave credibility to programme managers leading the change. Attention was given to how the system would be introduced to each province, with the organization of work- shops, printing and distribution of stationery, revision of computer software, and discontinuation of existing systems. Private printing companies and training schools were kept aware of the changes. At national level, procedures were established for follow-up of missing reports, data quality control, updating coding systems, data summary and provision of feedback. Provisions were included in the National Health Administration Act, 1997, to compel all health facilities to report using the NHIS. Certificates were awarded to health facilities and provinces that provided the best reports. Financial support was secured for stationery, training, freight/postage, commu- nications, periodic upgrading of computers, and software. Management of systems Our experience suggests that tasks required for the smooth functioning of information systems are best managed and 755Bulletin of the World Health Organization 2002, 80 (9) Health information systems in Papua New Guinea financed by a single unit at national level rather than being devolved to provinces or split among separate programmes at national level. Such an arrangement provides more stable funding, greater consistency in approach, and avoids duplication. There is no standard model, however, and in other situations a different strategy may be preferred, particularly if themonitoring capacity of individual programmes is well developed and programme staff are motivated to seek and use the information. Nevertheless it will still be necessary to ensure that opportunities exist to undertake consolidated analysis of information if management of systems is distributed over several units. Coordination of systems development Mechanisms are needed to coordinate system developments across programmes (21). In Papua New Guinea, a committee was established to review all information system proposals and develop a coordinated strategy for improving systems. The committee included representation from several programme managers and donors but there were still some agencies that preferred to bypass government procedures and install parallel systems, sometimes employing consultants who use non- standard software and fail to work with local systems developers (19). This approach is partly explained by the perception that governments are not capable of managing information systems and it is easier for a donor to bypass dysfunctional structures than to strengthen them. However, such short cuts do not help to develop the management capacity of health systems. Moreover, experience suggests that independently developed information systems rarely outperform government systems and rapidly collapse once donor support is withdrawn. They put undue pressure on government staff, disrupt normal operations, and are ultimately detrimental to information system perfor- mance and sustained development. What should donors do? In most cases it is preferable to channel donor support into existingmonitoring units, with funds for stationery, equipment and technical assistance where necessary. If several donors are operating in a country, it may be advantageous to employ common indicators of performance and common data collection methods. Clearly, there are difficulties in developing a coordinated monitoring strategy where donors do not share common objectives. However, they can be assisted if government objectives are clear. In Papua New Guinea, the national health plan defines the goals and objectives of the health sector and howprogress will bemonitored. The plan has helped to ensure donor support for national programmes and how they are monitored. Human resources A critical factor governing the sustainability of information systems is the availability of qualified and experienced personnel. Many systems are developed with external assistance because necessary skills are not available locally. It is unrealistic to expect these skills to be developed rapidly. Certainly, it would help to strengthen the capacity of local institutions to offer training in statistics, epidemiology, and computing to diploma or degree level. But it is difficult to recruit or retain staff without significantly improving pay or working conditions. Many countries are therefore faced with the stark choice of neglecting information systems or relying on external assistance whose goals may be at variance with those of the government. Creating a culture of information use It is hoped that lower levels of management, including health care providers, will use information for planning and monitoring health services, and it is possibly at this level that information use can have the greatest impact on the efficiency and effectiveness of health services (22). This can be encouraged through training and the provision of tools for data analysis. While necessary, such initiatives are seldom sufficient to change the way staff use information. Our experience suggests that an important way of developing the periphery in this respect is by setting a good example at the top. If senior management seeks information and uses it openly then the importance of information is reinforced throughout the health system. A promising initiative has been to summarize health statistics according to parliamentary con- stituencies and ensure their widespread circulation. This is not only of interest to health managers but also to a wider audience who wish to see accountability among public servants and politicians (23). It has given a readily observable purpose for the health management information system and helps to legitimize requests for data from health workers and mid-level management. Information has also been used to defend the position of the health sector in negotiations with other government departments and donors. Indeed, the ability to set priorities and monitor progress was a key factor in attracting a health sector programme loan in which national health plan targets are used as benchmarks for assessing progress, and ultimately to the adoption of a sector-wide approach for the coordination of donor inputs. These benefits will ultimately be critical to ensuring the sustainability of information systems. According to UNICEF (24), sustainability is the ability of a system to produce benefits valued sufficiently by users and stakeholders to ensure enough resources to continue activities. Hence, information systems can be sustainable if they deliver benefits to senior decision-makers and donors who control resources, and if there is sufficient motivation for health workers to support them. n Acknowledgements The work described in this paper was undertaken while Dr Cibulskis was Adviser in Health Information Systems for the Asian Development Bank Third Rural Health Services Project and Dr Hiawalyer was Deputy Director of Monitoring and Research in the Department of Health, Papua New Guinea. We recognize the efforts of staff from the Monitoring and Research Section and provincial health information officers who have been responsible for building the country’s information systems. Special credit should be given to Ahouta Badu, Ila Rouka (discharge information); Gahusi Gahusi, Kheila Tawai (national health information system); Mikes Arere, Tware Teka (health facilities); Steve Mellor, Jojie Urbiztondo, John Mondo, George Tuges (computer systems); and Kwalu Ora (publications). We also acknowledge the efforts of all health workers who routinely submit their monthly activity reports through the NHIS. Dr Tonya Villafana and three anonymous referees reviewed and made helpful comments on the manuscript. A preliminary draft of this paper was presented at the Twelfth Meeting of Commonwealth Health Ministers, Barbados, 1998. Conflicts of interest: none declared. 756 Bulletin of the World Health Organization 2002, 80 (9) Policy and Practice Re´sume´ Syste`mes d’information pour la surveillance du secteur de la sante´ en Papouasie-Nouvelle-Guine´e Le pre´sent article de´crit comment un syste`me d’information sanitaire national a e´te´ conc¸u, teste´ et mis en œuvre en Papouasie-Nouvelle-Guine´e, comment ce syste`me a e´te´ inte´gre´ avec les autres syste`mes de gestion de l’information et comment l’information a e´te´ utilise´e dans la prise de de´cision. S’il est vrai que la performance des syste`mes existants d’information peut souvent eˆtre ame´liore´e, il arrive qu’une trop grande place soit accorde´e a` la modification des me´thodes de collecte des donne´es et a` la conception d’un syste`me d’information parfait. L’analyse des donne´es, meˆme quand elles proviennent d’un syste`me imparfait, permet de de´velopper l’inte´reˆt pour l’information, ce qui peut ame´liorer la qualite´ et la comple´tude des signalements et favoriser une approche plus me´thodique de la planification et de la surveillance des services. Les strate´gies possibles sont nombreuses pour ame´liorer l’utilisation de l’information dans le secteur de la sante´ mais l’adoption de nombreuses initiatives est variable. D’apre`s notre expe´rience, les responsables des de´cisions et les responsables politiques peuvent jouer un roˆle important en suscitant une culture de l’utilisation de l’information, en sollicitant l’information sanitaire, en l’utilisant pour formuler les politiques et en la disse´minant par les moyens a` leur disposition. On peut s’attendre a` en retirer plusieurs avantages, et notamment la capacite´ d’exercer une plus grande influence dans les ne´gocia- tions avec d’autres services gouvernementaux et les donateurs, une plus grande rationalite´ des de´cisions qui ame´liorera le fonctionnement des services de sante´ et une incitation a` utiliser l’information aux niveaux infe´rieurs du syste`me sanitaire. La faculte´ des syste`mes d’information a` ge´ne´rer des avantages pour les divers inte´resse´s est en fin de compte capitale dans leur maintien. Resumen Sistemas de informacio´n para la vigilancia del sector de la salud en Papua Nueva Guinea En este artı´culo se describe co´mo se disen˜o´, ensayo´ e implanto´ en Papua Nueva Guinea un sistema nacional de informacio´n sanitaria, co´mo se integro´ dicho sistema con otros sistemas de informacio´n para la gestio´n, y co´mo se ha utilizado la informacio´n para apoyar la adopcio´n de decisiones. Si bien pueden identificarse a menudo diversas posibilidades para mejorar el funcionamiento de los sistemas de informacio´n existentes, se corre el riesgo de hacer demasiado hincapie´ en la revisio´n de los procedimientos de acopio de datos y en el perfeccionamiento incesante de esos sistemas. El ana´lisis de los datos, aun de sistemas imperfectos, puede suscitar ma´s intere´s por la informacio´n, lo que puede traducirse en una mejora de la calidad e integridad de los datos notificados y propiciar un enfoque ma´s meto´dico de la planificacio´n y vigilancia de los servicios. Pueden utilizarse diversas estrategias para mejorar el uso de la informacio´n en una organizacio´n de salud, pero el intere´s de las iniciativas es desigual. Nuestra experiencia parece indicar que las altas instancias decisorias y los dirigentes polı´ticos pueden contribuir considerablemente a promover una cultura del uso de la informacio´n solicitando informacio´n sanitaria, utiliza´ndola para formular polı´ticas y difundie´ndola por los circuitos a su alcance. Cabe prever como resultado diversos efectos positivos, como la capacidad de ejercer mayor influencia en las negociaciones con otros ministerios y donantes, la adopcio´n de decisiones ma´s racionales que mejorara´n el funcionamiento de los servicios de salud, y el fomento del uso de la informacio´n en los niveles inferiores del sistema sanitario. La capacidad de los sistemas de informacio´n para beneficiar a los interesados directos es en u´ltimo te´rmino decisiva para su sostenibilidad. References 1. Smith P. Information systems and the white paper proposals. In: Culyer AJ, Maynard A, Posnett J, eds. Competition in health care: reforming the NHS. Basingstoke: Macmillan; 1990. 2. Smith P. The use of performance indicators in the public sector. Journal of the Royal Statistical Society Series A 1990;153:53-72. 3. Mackay K. Public sector performance — the critical role of evaluation. In: Mackay K, editor Public sector performance — the critical role of evaluation: selected proceedings of a World Bank Seminar. Washington (DC): World Bank;1998. p.ix-xvi. 4. Cibulskis RE, Izard J. Monitoring systems. In: Janovsky K, editor. Health policy and systems development: an agenda for research. Geneva: World Health Organization; 1996. p.191-206. 5. Keller A. Management information systems in maternal and child health/family planning programs: a multi-country analysis. Studies in Family Planning 1991;22:19-30. 6. Finau SA. National health information systems in the Pacific Islands: in search of a future. Health Policy and Planning 1994;9:161-70. 7. Newbrander WC, Thomason JA, Kolehmainen-Aitken RL, Campos-Outcalt D, Afo G, Davidson W. Management support for provinces: a programme for developing provincial health management capabilities in Papua New Guinea. International Journal of Health Planning and Management 1988;3:45-55. 8. Newbrander WC, Thomason JA. Computerising a national health system in Papua New Guinea. Health Policy and Planning 1988;3:255-9. 9. Campos-Outcalt D, Vickers P. For fuller utilization of health data. World Health Forum 1988;9:405-8. 10. Campos-Outcalt D. Microcomputers and health information in Papua New Guinea: a two year follow-up evaluation. Health Policy and Planning 1991;6:348-53. 11. Muller I, Smith T, Mellor S, Rare L, Genton B. The effect of distance from home on attendance at a small rural health centre in Papua New Guinea. International Journal of Epidemiology 1998;27:878-84. 12. Rose M, Abderrahim N, Stanton C, Helsen D. Maternity care: a comparative report on the availability and use of maternity health services. Chapel Hill (NC), Measure Evaluation, 2001. MEASURE Evaluation Technical Report Series, No. 9. 13. Oppong JR, Hodgson MJ. Spatial accessibility to health care facilities in Suhum District, Ghana. Professional Geographer 1994;64:199-209. 14. Bigman D, Dercon S, Guillaume D, Lambotte M. Community targeting for poverty reduction in Burkina Faso. The World Bank Economic Review 2000;14:167-93. 15. Lippeveld T, Sauerborn R. A framework for designing health information systems. In: Lippeveld T, Sauerborn R, Bodart C, editors. Design and implementation of health information systems. Geneva: World Health Organization; 2000. p. 15-32. 16. Bodart C, Shrestha L. Identifying information needs and indicators. In: Lippeveld T, Sauerborn R, Bodart C, editors. Design and implementation of health information systems. Geneva: World Health Organization; 2000. p. 49-72. 757Bulletin of the World Health Organization 2002, 80 (9) Health information systems in Papua New Guinea 17. Lippeveld T. Approaches to strengthening health information systems. In: Lippeveld T, Sauerborn R, Bodart C, editors. Design and implementation of health information systems. Geneva: World Health Organization; 2000. p. 243-52. 18. Bunge C. Health management information system for provincial health administrations — Cameroon. In: The practice of health information systems. London: Commonwealth Secretariat; 2001. p. 19-31. 19. Jayasuriya R. Managing information systems for health services in a developing country: a case study using a contextualist framework. International Journal of Information Management 1999;19:335-49. 20. Heeks R, Mundy M, Salazar A. Why health care information systems succeed or fail. Manchester: Institute for Development Policy and Management; 1999. Working Paper Series, No. 9. 21. Van Konkelenberg R, Ring I. Guidelines for the development of health management information systems. Manila: WHO Regional Office for the Western Pacific; 1993. 22. Sandiford P, Annett H, Cibulskis RE. What can information systems do for primary health care? An international perspective. Social Science and Medicine 1992;34:1077-87. 23. Marcel M. Lessons from Chile. In: Mackay K, editor. Public sector performance — the critical role of evaluation: selected proceedings of a World Bank Seminar. Washington (DC): World Bank; 1998. p. 67-71. 24. Health policies and strategies. Sustainability, integration and national capacity-building. New York (NY): UNICEF: 1992. Unpublished document E/ICEF/1992/L.2. 758 Bulletin of the World Health Organization 2002, 80 (9) Policy and Practice