Evaluation of occupational health and industrial hygiene services: report on a WHO working group, Stockholm, 15–17 April, 1980

EURO Reports and Studies 56 \--\ L \ ~ Evaluation of Occupational Health and Industrial Hygiene Services ~ ,.,. t ·•1·••1 .,-.• - : ··--7 . ~ . - ,. . " .- RE l -~-G. A;:. 1 q 82 -1 ~.~wr•1i~•"')l.l•.J Report on a WHO Working Group INFORMATION COPY ORt"N.Al/.DlJPUCATE Ir FU E •• .. O.z./,u.,Jc.'tl_~ REGIONAL OFFICE FOR EUROPE World Health Organization COPENHAGEN W fl-t> .J .11 C• I Rt: . EURO Reports and Studies 56 Evaluation of Occupational Health and Industrial Hygiene Services Report on a WHO Working Group Stockholm 15-17 April 1980 I \\\Ill \\Ill \\111 \\II\ II\ II lll\111111111 11 111111 * O O O 2 8 l 3 8 * REGIONAL OFFICE FOR EUROPE World Health Organization COPENHAGEN 1982 ICP/WKH 006 ISBN 92 890 1222 6 © World Health Organization 1982 Publications of the World Health Organization enjoy copyright pro- tection in accordance with the provisions of Protocol 2 of the Universal Copyright Convention. For rights of reproduction or translation, in part or in toto, of publications issued by the WHO Regional Office for Europe application should be made to the Regional Office for Europe, Scherfigsvej 8, DK-2100 Copenhagen </), Denmark. The Regional Office welcomes such applications. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the Secretariat of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific companies or of certain manufacturers' products does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not men- tioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. The views expressed in this publication are those of the participants in the Working Group and do not necessarily represent the decisions or the stated policy of the World Health Organization. PRINTED IN DENMARK ISSN 0250-8710 CONTENTS Introduction . . .. .. .. ... . . . . . .. . Development of health programme evaluation Current state of occupational health services in Europe Bulgaria .... Czechoslovakia Finland . .. . France . . .. . German Democratic Republic Germany, Federal Republic of Hungary Italy .. Poland . Sweden Yugoslavia . Commission of the European Communities . Objectives and goals . . . . Methodology of evaluation Basic concepts . . . . Indicators and criteria Steps in evaluation .. Levels of evaluation . Components and specific aspects to be evaluated Variables to be evaluated . ... International comparisons . Evaluation at national level Variables at national level . Page 2 3 4 4 5 6 6 7 7 8 8 9 10 10 11 13 13 13 14 15 17 21 21 21 21 Cost-benefit and cost-effectiveness Cost-benefit analysis .. . Cost-effectiveness analysis . Conclusions . . .. Recommendations References .. . . Annex List of participants . 26 26 27 27 30 31 33 INTRODUCTION A Working Group on Evaluation of Occupational Health and Industrial Hygiene Services (OHS) was convened in Stockholm, Sweden from 15 to 17 April 1980 by the WHO Regional Office for Europe in collaboration with the Government of Sweden. The meeting was held under the sponsorship of the Swedish Work Environment Fund and with the active collaboration of the International Labour Organisation (ILO). The purpose of the Working Group was to identify the objectives of the evaluation of OHS , to define the components of OHS to be evaluated, to elaborate relevant criteria for evaluation and to identify needs and priorities for further development of the methodology of OHS evaluation . The Working Group was composed of 14 temporary advisers from 14 countries. In addition, two representatives of international organizations attended , one from the Commission of the European Communities (CEC) and one from ILO . Eight observers from various authorities or organizations were present during the meeting (see Annex fo r list of participants). Pro- fessor J . lndulski was elected Chairman and Dr R. Edstrom Vice-Chairman. Dr J . Rantanen was appointed Rapporteur. The meeting was opened by Mr J.I. Waddington, Director , Promotion of Environmental Health , WHO Regional Office for Europe. In his opening ad- dress Mr Waddington emphasized the importance of OHS for industry and agriculture in meeting the problems caused by health risks arising both from traditional industries and from new technologies . The objectives of OHS are common to Member States, though organiz- ational patterns may vary. The Regional Office has long recognized the need to develop an adequate methodology for the evaluation of OHS. Several studies in this field have been undertaken and a recent comparative survey of occupa- tional health services in eight European countries (R. Murray , unpublished data, 1979) provides useful guidelines. In addition to the development of evaluation of OHS, other steps in the promotion of occupational health in the European Region have also been undertaken. The Regional Office has started an intensive programme on many different aspects of the evaluation , surveillance and control of toxic chemi- cals, which is closely linked with the International Programme on Chemical Safety. Priorities include toxicological training, contingency planning for chemical accidents, management of hazardous wastes, and technical cooper- ation with Member States. Particular emphasis is being placed on legislative and administrative structures. The programme embraces problems affecting human health in relation to air, water, food and the working environment. Mr Waddington recommended a pragmatic approach to the evaluation of occupational health services. The Working Group should seek to stimulate governments to carry out evaluation at the national level , to organize the col- lection of relevant data needed for evaluation at the national level, and to lay down relevant criteria for the evaluation of OHS. ln her welcoming address Dr Barbra Westerholm , Director-General of the National Board of Health and Welfare , Sweden , stressed the importance of critical and continuous evaluation of all types of health services, difficult though that task might be. There had to be effective allocation of limited resources in the field . The Working Group would hopefully provide guide- lines for OHS evaluation for Member States. Guidelines of this sort were of great interest to Sweden because a Government committee was at present planning the future content and organization of OHS in Sweden . Dr Wester- holm emphasized also the great value of epidemiological approaches to health services evaluation, including OHS. She hoped that the Working Group , through its deliberations , would make a contribution towards the common goal in the development of health services: health for all by the year 2000. The scope and purpose of the Working Group were described by Dr M.I. Mikheev, Regional Officer for Workers' Health. The Working Group was to study the possibilities for evaluation , to identify the points of OHS that can be evaluated, and to identify criteria that can be used in evaluation . Evaluation should be carried out at all relevant levels - plant , regional and national. DEVELOPMENT OF HEALTH PROGRAMME EVALUATION The scarcity of resources and negative development of national econ- omies during the 1970s made the Member States increasingly interested in the effective use of health service resources and in the relevance of health programmes to the most important social and health needs of their populations . A document entitled Development of evaluation in WHO was presented to the fifty-seventh session of the Executive Board in 1976 (J), and from this provisional guidelines were prepared by WHO headquarter5 in 1978 (2). The Thirty-first World Health Assembly in 1978, in resolution WHA3 l.l l , gave its support to the principles and methods of health programme evaluation and 2 stressed the important role of Regional Committees in promoting health pro- gramme evaluation. The Health Assembly adopted resolutions which: (a) requested the Director-General to continue to develop health pro- gramme evaluation as an integral part of health development, including the establishment of positive indicators of health; (b) requested the Executive Board to review periodically the develop- ment of health programme evaluation; and (c) urged Member States to introduce progressively procedures for the evaluation of national health programmes and services by national health personnel, and to collaborate with WHO in evaluating the impact of the Organization's programmes in their countries. The development of OHS has been a subject of great interest in the WHO European Region since the establishment of the Regional Office in 1949. A number of reports on the development, planning, organiz- ation, effects and evaluation of OHS have been issued by the Office, in addition to numerous internal documents on the issue . For the meeting, two background documents had been prepared, one on the evaluation of occupational health services and another on the evaluation of industrial hygiene services. The Working Group felt that standardization of methodology and guide- lines for the practice of OHS evaluation were needed . A systems-analysis approach was seen as the appropriate orientation for OHS evaluation . Models for the evaluation of other types of health service have been developed, for example by the International Institute for Applied Systems Analysis (3), and evaluations have been carried out by WHO for a number of other programmes of major importance, such as those in public health (4) and environmental health (5). General guidelines for evaluation are currently being developed by WHO. CURRENT STATE OF OCCUPATIONAL HEALTH SERVICES IN EUROPE Reviews of the current situation as regards occupational health services were received from Bulgaria, Czechoslovakia, Finland, France , the German Democratic Republic, the Federal Republic of Germany , Hun- gary, Italy, Poland, Sweden, Yugoslavia , and the Commission of the Euro- pean Communities. 3 Bulgaria The Bulgarian report reviewed the organizational and administrative structure of OHS at all relevant levels (national , regional and local) and highlighted the relationships among the various organizations. The organizational model is a variant of the centralized OHS system typical of the socialist countries of eastern Europe, OHS being under the control of the occupational health and medical sections of the Ministry of Public Health . Trade unions participate in OHS activities as a matter of course. OHS is based on statutes included in the Labour Code and in the Public Health Act. A national programme for the development of OHS has been elaborated, consisting of two parts. 1. Increasing the effectiveness of preliminary and permanent sanitary control. A classification of working conditions has been introduced with categories ranging from O (hazardous) to 4 (optimal). Several properties of the working environment have been surveyed according to this classification, including noise , dust, chemical substances, microclimate, ionizing and non- ionizing radiation, and the degree of static and monotonous work. This kind of ranking of working conditions provides guidelines for the setting of prior- ities for OHS. 2. Improvement of methods for the evaluation of occupational risks. This programme includes the development of methodology for biological monitoring, development of mathematical modelling for be- haviour of toxic chemicals in the human body, assessment of the effects of multiple exposures to hazardous substances or radiations, and the ef- fects of combinations of hazardous exposures. The programme is intended to advance epidemiological studies and to provide the scientific basis for new legislation. The effects of the Bulgarian effort in OHS can be seen in the decreasing number of occupational diseases and industrial accidents. Czechoslovakia OHS in Czechoslovakia is statutory, and is carried out under the super- vision of the Ministry of Health; it is thus strongly integrated with the general health services. The overall goals and content of OHS are clearly de- fined by legislation. To statutory OHS belong preventive activities, start- ing with primary prevention of risks at the planning stage of work places and industrial processes, and continuing with secondary prevention of risks in existing plants. Curative treatment of occupational and other diseases also belongs to the province of OHS. 4 In addition to the traditional physical and chemical risk factors , psycho- logical aspects of work and the work environment are taken into consideration in national legislation. An interesting feature is that emphasis is given to non- specific as well as specific occupational diseases. Similarly, the importance of multiple factors in the etiology of chronic disease is taken into account in OHS planning. A national programme exists for the development of OHS, with well- defined priorities but without strict time-limits. The programme consists of a national survey and categorization of workplaces on the basis of risk factors found in the work or the working environment. Categories from A (virtually safe) to D (unsatisfactory) are used. For an evaluation of the effectiveness of OHS several criteria are em ployed, including data that allow ranking of working places into different categories, data on the health status of workers (e.g. statistics on occupational diseases and work disabilities), data on absenteeism, and qualitative and quantitative statistics on available manpower resources. The health indicators demonstrate decreasing risks and a declining number of illnesses during the last few years. Finland In Finland a national programme for the development of OHS has been elab- orated partly in the form of an Act on Occupational Health Services with a number of lower level regulations (1978) , and partly in the form of a five-year plan for the development of health services, including OHS. Supervision of OHS practice is deputed to the labour inspectorate for questions concerning the statutory obligations of employers, to health author- ities for questions concerning the medical content of OHS and the work of medical personnel , and to plant safety committees for questions concerning applications of the Act at plant level. The Finnish report presented an evaluation of OHS at the national level and an analysis of how the new Act meets the nationwide problems of OHS provision. Several conclusions were drawn from the evaluation: there are gaps in the coverage of OHS, especially in small enterprises and in various branches of the economy such as the building industry ; there is a shortage of well-trained manpower resources ; the inputs to OHS in plants and branches where the services have been organized are comparable to in- puts in other branches of the health services and to the inputs to OHS in some other countries ; and the outputs are quantitatively of the same order as in other parts of the health services. Analysis of the effectiveness of OHS yielded a surprising result : the number of occupational diseases initially showed an increasing trend in situations where OHS was intensively developed . This is attributable to the improved organization of services, which led to a more complete registration of workers and the identification of a higher proportion of occupational 5 diseases. Later , during the second half of the 1970s, the number of occupa- tional diseases and work accidents began to decline. Evaluation of the impact of OHS is complicated by the fact that the general health of the population improved in Finland during the 1970s, for reasons that are not fully under- stood. The impact of OHS is difficult to extract from this general improve- ment in health. Data on environmental conditions at places of work could form the basis of a comparatively rapidly responding indicator of the effectiveness of OHS. For example , the level of exposure to hazardous chemicals, as indicated by the number of measurements exceeding threshold limit values (TLV), declined during the 1970s, despite the fact that consumption of chemicals increased. France Since 1946, OHS has been statutory in France , though the activity as such has a tradition extending back more than 100 years. New legislation in 1979 gave national control to the Ministry of Labour Safety Inspectorate and to medical inspection authorities . Though French OHS is exclusively preventive , the general state of health of workers must also be taken into consideration. The stipulated content of OHS accords with the principles laid down by CEC, ILO and WHO for pre- ventive activities. The content of OHS is designed to take into account not only the traditional hazards at work and in the working environment but also the psychological and psychosocial aspects of work. Several constraints affecting the evaluation of OHS were dealt with in the French report , including the confounding effects of non-occupational factors on workers' health , and the difficulties of measuring certain effects of OHS (for example , effects on mental wellbeing, and the difficulties of evaluating the effectiveness of OHS in solving problems that require management decisions outside the OHS system). Nevertheless the national statistics on occupational diseases show a positive development, with a declining trend in morbidity. German Democratic Republic OHS is an integral part of state health services stipulated by legislation. All the physicians and other members of medical staff in OHS are employees of state public health services and are thus independent of management. Legisla- tion stipulates that the basic tasks of OHS include both preventive and cura- tive activities. The form and extent of OHS and the number of OHS personnel are regulated by the number of workers and the extent of risks in a plant. High-risk activities, such as construction , the chemical industry , public transport and agriculture are intensively studied and served by special centres of industrial hygiene, which are statutory plant-based organizations for the local development of OHS. 6 An ambitious programme to study working conditions and their health consequences in the whole labour force has been started. The programme is at present in a pilot phase in which the health and working conditions of selected groups of workers are being monitored. In the pilot study, 400 000 workers have been tested for health and work capacity, and the workplaces of about 60Q 000 workers have been monitored . Preliminary results have indicated the most important risk factors in work and the most relevant health effects among workers. The results of the project will be used to set priorities for the further development of OHS and the targeting of future scientific efforts in the field . Federal Republic of Germany OHS has been statutory in the Federal Republic of Germany since 1974. The relevant Act stipulates the content of occupational hygiene and medical activities and defines the basic obligations and rights of employers, workers and occupational health and safety personnel. The content of OHS is purely preventive, OHS personnel having no responsibility for curative services. The organization of OHS is somewhat unusual in giving great responsi- bility for practical implementation of OHS legislation to the Central As- sociation of Industrial Employment Accident Insurance Funds, which can also issue accident prevention (including occupational disease) regulations and supervise accident prevention measures. The delegation of this authority is the result of an agreement between governmental bodies and insurance funds. Funds are organized on a regional basis according to the branch of the economy affected. OHS practice is to a great extent guided by occupational medical check- ups issued by the Central Association of Industrial Employment Accident Insurance Funds. The check-up instructions can be problem-oriented or oriented towards prevention of health risks in a defined industry or occupation. Other aspects regulated by the Act are competence criteria for OHS phys- icians, forms of service provision, and quantification of OHS manpower resources with respect to the number of served workers. Evaluation of OHS is carried out by Insurance Funds. Aspects of OHS which have been evaluated include time input by OHS expert personnel , the qualifications of industrial medical officers, and the level and adequacy of material equipment for OHS. The report also describes an extensive list of criteria that can be used for the characterization and evaluation of OHS. Hungary OHS is arranged on a statutory basis in Hungary and the administrative model is at central , regional and local levels, much the same as in other socialist countries, with trade unions having the authority to control practical aspects 7 of OHS activities jointly with the state health authorities . As in other eastern European countries, OHS and the preventive and curative branches of the health services are administratively separated at central and regional levels but are integrated at the top level in the Ministry of Health and at the lowest level in plant OHS units . The National Institute of Occupational Health offers an advisory service and supports research into OHS. The content of OHS is defined by an order of the Ministry of Health made in 1979. Both preventive and curative activities are included in the tasks of plant-level OHS. The number of OHS personnel with respect to the number of served employees is regulated by the Act. A preliminary evaluation of OHS has been carried out , but the results have not been reported so far. Further developments will include the strengthening of the hygienic and medical aspects of OHS at regional and local levels. Italy Since the second decade of this century there have been several Acts or regu- lations concerning specific aspects of occupational health. At the beginning of 1980 a comprehensive new OHS Act came into force. In bigger plants OHS is well organized, but there are problems with small and medium-sized plants. In order to meet the problems of small industries a comprehensive new health service system, including OHS, has been organized by local authorities, and the Institute for Prevention of Occupational Injuries (ENPI) has organized group services for small and medium-sized industries. At present almost 70% of industry has a contract with ENPI for the provision of OHS. In the report three important aspects were emphasized : first, the importance of training doctors in OHS, and the need for competence criteria for OHS doctors ; second, the improvement of career prospects for doctors in OHS to match those in other fields of medicine ; and third , the importance of integrating OHS with other branches of the health services to prevent the danger of OHS becoming detached from its medical base. Poland The Polish report reviewed the philosophy and systems analysis methodology of OHS evaluation, using the national Polish OHS system as a model. Thorough analysis of OHS organization at all relevant levels is given in the report. The effectiveness of vertical and horizontal connections between various organizational bodies is mentioned as an important target for evalu- ation. The organizational model parallels those described in various other socialist countries and is not described here in detail . Full integration with the health service system, a statutory background, and joint control of OHS by health authorities and by trade unions are essential elements of the system. The Institute of Occupational Health and branch-oriented institutes play a 8 substantial role in the OHS system. Though prevention is given priority, cura- tive activities are also included in OHS. The report on evaluation of OHS starts with a systematic discussion of general theory, concepts and definitions in evaluation. Two different approaches to the national management of OHS are given: the principle of maximal effectiveness on the one hand and the principle of minimal costs on the other. Because the resources available for the health services are usually defined in advance of the goals to be achieved, the principle of maximal effectiveness is more relevant in this connection. The report sep- arates the internal effectiveness from the external effectiveness of the programme; both must be analysed, starting with external effectiveness. In the evaluation exercise , the authors go through an analysis of the in- tended effects of OHS programmes, and continue with an analysis of incidental effects, and analyses of several other aspects ; for example, criteria are given for the evaluation of relevance , adequacy , progress and efficiency. The report finally emphasizes the continuous character of OHS evaluation. Sweden In Sweden, OHS is integrated into a comprehensive system for workers' health and safety which is regulated by the Work Environment Act. Specific legislation concerning the organization of OHS does not exist , and there is currently a governmental committee planning the development of OHS in the country. The Swedish report emphasized the importance of evaluation of OHS at plant level. According to present legislation, OHS is a subject for joint regula- tion at workplaces, and this enables workers to participate actively in the development and evaluation of OHS. Because most of the development of OHS has taken place on the basis of collective agreements between the social partners at local level , there is no comprehensive national programme for the development of OHS. The agree- ments are relatively general in character and do not contain long-term pro- grammes for the development of OHS. This makes detailed analysis of national OHS somewhat difficult. The content of OHS is primarily preventive , but a substantial part of current resources is used for the curative care of workers . The organ- ization of service provision is arranged using one of three options: an in- ternal OHS unit, group services, or branch-oriented services. In addition, local authorities are obliged to provide OHS facilities for small industries. Manpower resources are actively trained in the Swedish OHS but no formal competence criteria for specialist education in OHS exist. There are, however, recommendations by a national OHS committee which can be con- sidered as semi-official competence criteria. 9 There are several priorities for the further development of OHS, for ex- ample, strengthened integration of OHS with general health care , the provision of services in small industries, the development of technical facilities for OHS, the building up of competence criteria for OHS personnel, and the strengthen- ing of regional activities for both prevention and cure of occupational disease. The report gives several relevant comments concerning evaluation of OHS in general, and describes the national registers of workers exposed to specific risk factors such as silica and asbestos. The registers give data on both the ex- posure of workers and the effect of exposure on health. Because the person- register system in Sweden allows computer-based handling of data from all citizens, there are plans for developing even more comprehensive register systems related to worker exposure and risks and their health effects. This kind of system will allow detailed evaluation of several aspects of OHS, as well as the establishment of an early-warning system for work-derived risks. Yugoslavia The OHS programme in Yugoslavia is an integral part of health service pro- grammes and has a statutory basis. The provision of OHS is based on the activity of local health centres, although there is also a restricted number of independent OHS units in the big industries. The content of OHS includes both preventive and curative activities. There are specific industrial hygiene and safety organizations responsible for the control and elimination of risk factors in working environments and for the technical prevention of hazards. The evaluation of OHS has been discussed at all relevant levels. As input indicators, manpower resources and costs of OHS are used, while the outputs and effectiveness are assessed by an analysis of the coverage of OHS and by studying the trends in occupational diseases, general diseases, sickness, absen- teeism and accidents. A new pattern of morbidity among workers has appeared since the decline of the traditional causes of occupational disease, such as heavy intoxication. The present profile in workers' morbidity is dominated by chronic, nonspecific diseases of the respiratory and locomotor systems. Commission of the European Communities In 1962 the Commission of the European Communities issued recommenda- tions on OHS and in 1966 produced a revised list of occupational diseases. Since then the Commission has produced or has under preparation several recommendations and resolutions concerning occupational health and safety. At present the most important documents are : the resolution of 28 June 1978 on an action programme of the European Communities on safety and health at work (6) and a proposal for a directive on the protection of workers from harmful exposure to chemical , physical and biological agents at work (7). These documents were produced as a guide to the preparation of national legis- lation and are based on the philosophy of keeping exposures to harmful agents as low as possible. The action programme is broadly based; it emphasizes the humanization of work , and includes proposals for improvements in the psy- chological aspects of working conditions, as well as dealing with traditional physical and chemical hazards. There are also more specific documents in preparation concerning control of exposures to selected heavy metals such as lead and cadmium (8). A direc- tive on new chemical substances is also in preparation . The Council of Ministers follows up the implementation of direc- tives in member countries, gives advice , and works for harmonization of national legislation and practices. There have been some difficulties in harmonization for technical and cultural reasons. One of the difficulties has been the variation of concepts and definitions among countries. Nation- al traditions are also somewhat variable , and harmonization therefore proceeds slowly. Evaluation of the implementation of directives and resolutions is carried out by the Commission by continuous follow-up. There are prac- tical difficulties in evaluation because of a lack of harmonized criteria and standardized indicator systems. Also , evaluation at national level can be difficult if routine procedures for evaluation are lacking. The main goal is progressively to harmonize national legislation, and to define con- cepts and definitions. Harmonization and follow-up will take into consider- ation the need for flexibility in the implementation of directives , to accord with local needs. On the basis of the national reports it can be stated that no country has so far developed an evaluation mechanism for all levels of the health service system. There are, however, numerous examples of restricted evaluation pro- jects at plant or branch level. Furthermore , evaluation of the effectiveness of OHS programmes in the solution of specific problems, such as noise abate- ment or the development of health examinations, has been carried out in many countries, e.g. Finland (9) , Sweden (10) , the United Kingdom (11) and the United States (12). OBJECTIVES AND GOALS In order to implement OHS and to evaluate its effects , an analysis of the general objectives of OHS is needed. This analysis must provide answers to the following questions: (a) why are specific OHS programmes needed 11 in addition to the public health services and other types of primary care, and {b) what are the ultimate goals of OHS? An ILO committee defined in 1959 the aims of occupational health as follows: "Occupational health should aim at : the promotion and main- tenance of the highest degree of physical, mental and social well-being of workers in all occupations ; the prevention among workers of depar- tures from health caused by their working conditions; the protection of workers in their employment from risks resulting from factors adverse to health; the placing and maintenance of the worker in an occupational environment adapted to his physiological and psychological equipment and, to summarize: the adaptation of work to man and of each man to his job" (13) . In relation to the workers' health programme, WHO has defined the gen- eral objective of occupational health as follows: "to control occupational health risks, to protect and promote the health of working populations, and to promote the humanization of work" (14). At national level, the definition of goals of OHS varies with national needs and practices, with the structure of the health service and occupational safety systems, and even with the level of industrialization. The Working Group defined the ultimate goal of OHS as promoting conditions at work that guarantee the highest degree of quality of work- ing life by: - protecting workers' health; - enhancing their physical , mental and social wellbeing; and - preventing ill health and accidents. To achieve this goal, medical, industrial hygienic , and related expertise is required. This operational definition of the goals of OHS is in good accord with the earlier definitions given by ILO and WHO. The general goal of OHS may be documented in national legislation, in lower-level statutes, or in collective agreements , or it can be understood to be implicitly included in traditional practice . The Working Group agreed that authoritative documentation of the definition of goals is necessary for effec- tive practice . The extent to which goals are achieved varies in different countries, and also within countries, between different branches of the economy. Thus, evaluation of goals must be related to national needs and problems. The most important component in the evaluation of goals is an analysis of the compati- bility of goals with national priorities in solving the most important health problems of the working population. 12 METHODOLOGY OF EV ALVA TION Basic concepts The WHO Symposium on Methods of Evaluating Public Health Programmes, held in Kiel in 1967, defined evaluation as "the process of assessing the achieve- ment of the stated objectives of a programme, its adequacy, its efficiency and its acceptance by all parties involved" (J 5). With certain modifications this is also a valid definition for evaluation of OHS. In the WHO provisional guidelines for health programme evaluation (2) the purpose of evaluation in the health development process is taken to be the im- provement of health programmes and the provision of guidance for the allocation of human and financial resources in current and future programmes. The Working Group emphasized the role of evaluation as a tool for the decision-maker. This tool must be perceived as an integral part of all health pro- grammes rather than as a sporadic instrument for use in exceptional situations. The evaluation process also has an important role in the training of decision- makers and personnel who are concerned with implementing the programme. Evaluation cannot be carried out as an isolated process but must be seen as an integral part of an entity which involves policy-making, planning, pro- gramming, operations, actions and follow-up. Thus, a systems approach is necessary for successful evaluation. This issue has been discussed by WHO in another context (J 6). Evaluation can be divided into two basic categories (4): (a) pre-decision evaluation, in which options for the attainment of the desired goals are assessed and compared; and (b) post-decision evaluation, in which the results of current or past pro- grammes are compared with the intended goals. Evaluation can also be categorized on the basis of the system to be evalu- ated. Thus, evaluation can be targeted to assess the success of a general policy- programme of a country, or it can be focused on well-defined projects or pro- grammes at national, regional, local or plant levels. Indicators and criteria Evaluation employs indicators and criteria as tools to characterize the phe- nomena that form part of the programme under scrutiny. Indicators are vari- ables used to measure changes in a defined system, and criteria are standards towards which the actions or changes can be referred and compared. It is frequently emphasized that the reliability, validity, objectivity, sensitivity and 13 specificity of indicators must be ascertained before their use in research, planning or evaluation is justified. Criteria are expressions of desired optimal states in the system which, for example, guarantee the desired levels of health or safety of a population. The provision of indicators and criteria for evaluation requires well- developed information-gathering facilities at various administrative levels. Un- availability of data often causes severe problems for the evaluator. Relevant indicators and criteria in the evaluation of OHS are listed below in connection with a detailed description of evaluation processes. Where insufficient information and an incomplete supply of indicators and criteria are available, evaluation must proceed by using the available data. At the same time the development of a data base should be started. Steps in evaluation Evaluation should proceed in a defined, logical order, which will be partly dependent on whether the evaluation is of the pre-decision or post-decision type. The minimal requirements for evaluation, regardless of its type, are the following: - the objectives and quantified targets of the programme must be defined; - all the essential programme elements must be documented (inputs, outputs, time schedules); - an appropriate information supply must be established; - a general strategy for evaluation must be chosen. The individual steps of the evaluative process are described in the WHO draft guidelines (2) and are fully relevant for the evaluation of OHS. Further steps in evaluation are an analysis of the relevance of the goals, evaluation of the adequacy of the programme (a step which mainly concerns analysis of the inputs) and an analysis of the efficiency, effectiveness and impact of the programme (a step which is related to a comparative analysis of inputs and outputs). The Group stressed that external evaluation must be separated from in- ternal evaluation of OHS. Evaluation of external effectiveness aims at an analy- sis of the relevance of the goals and sub-objectives with respect to the overall objective of improvement of workers' health. Internal evaluation is directed towards an analysis of the technical effectiveness of the programme in pro- ducing outputs that guarantee the achievement of goals and sub-objectives to the greatest possible extent and with the highest possible efficiency. It was emphasized that all competent bodies, including social partners, must be able to participate in the external evaluation. The internal evaluation 14 is mainly carried out by the competent authorities and OHS experts. In addition to other methods (mainly systems analysis) surveys must be used to collect data for evaluation from groups served by OHS facilities. It was clear from the country reports, and from the survey of national OHS activity in eight selected countries in Europe, that one of the most critical questions for evaluation of OHS activities is the nonexistence of national OHS programmes in several countries. Even in those countries where programmes have been established the definition of goals is some- times so general that only a qualitative analysis can be made. In spite of this, most of the evaluative efforts aim, at least in principle, towards both quantitative and qualitative analyses of OHS programmes. Levels of evaluation The level of evaluation can be chosen according to the character of the programme to be evaluated. Several levels of evaluation of OHS can be rec- ognized. In terms of the administrative hierarchy, the level of evaluation can be at national, district/regional , local or plant levels. The prerequisite for effective evaluation at any level is the existence of adequate infor- mation relevant to the respective level. Evaluation of OHS can also be problem-oriented ; for example, has a noise abatement programme been successful, or what is the economical cost-benefit relationship of the OHS programme? Evaluation of OHS may be in the interest of national authorities , regional or local authorities, social partners or health personnel involved in the implementation of the programme. Affirming that evaluation is an integral part of the decision-making process and that it also plays an important part in the process of learning from experience, the Working Group stressed that all interested authorities , as well as social partners , should be aware of the importance of external evaluation. Internal evalu- ation would continue to be the responsibility of OHS personnel. Indicators and criteria can vary from one level to another and from one country to another. Comparisons must therefore be made with great care. For example , the socialist countries have a policy of protecting female workers by guiding them to work tasks with low risk factors for female organs or the unborn child. On the other hand, the Scandinavian countries have adopted a policy of guaranteed sexual equality of job opportunities. Thus, different criteria apply in socialist countries and Scandinavian countries. Furthermore , analysis on the basis of indicator figures is likely to lead to different conclusions in different socioeconomic systems. In Finland the number of workers with registered and compensated occupational dis- eases was comparatively low in the 1960s, although experts claimed the real number of cases was about 20-fold that of registered cases. During 15 the 1970s intensive efforts were made to identify all cases of occupational disease by strengthening diagnostic activities , developing OHS, promoting research and developing more effective registration procedures . As a re- sult the number of registered cases increased about 20- to 30-fold com- pared to the number of cases in the 1960s. The increase in this case was interpreted as a successful outcome for the programme; more usually , the indication of a successful OHS programme is a declining trend in occupational diseases . These two examples demonstrate the importance of careful analy- sis of national programmes and their goal-setting before comparative evalu- ations are made at international level. The purpose of evaluation may vary according to who does the evalu- ation. National authorities may be interested both in the effective use of resources and in the positive impact of a programme on the health of the working population , whereas , for example, some of the social part- ners may be primarily interested in the health effects. The Working Group was of the opinion that the external effectiveness of the programme (see below) should be assessed jointly by the competent authorities and social partners , whereas the internal effectiveness should be evaluated by the responsible authorities and OHS personnel. At the global level , evaluation will be carried out by international organizations. This requires well-harmonized indicator and criteria systems for comparative studies. Such systems are not widely available at present and this restricts possibilities for evaluation. It might be thought that criteria for evaluation of OHS at the inter- national level could be provided by recommendations of international organizations concerned with the development of OHS. These recom- mendations are , however , made only after wide discussions among a great number of countries at various levels of development and with different socioeconomic systems. For example , curative services in connection with OHS can be essential to meet the health problems of working popu- lations in developing countries. OHS can even form the backbone for the whole primary care system in the society . On the other hand, in coun- tries where primary care is effectively organized in local health centres (e.g. in Finland and the United Kingdom) OHS can be directed primarily to preventive activities. Thus, a given international recommendation can- not always provide relevant criteria for evaluation in countries at differ- ent stages of development , with different socioeconomic systems and with deviating philosophies in health policy. Nevertheless the Working Group recognized the desirability of international comparison, provided the problems and pitfalls discussed above are taken into consideration before conclusions are drawn. The evaluators in the case of international comparisons are primarily the international organizations. 16 Components and specific aspects to be evaluated Adequacy An OHS programme should be analysed for adequacy of available information, adequacy of problem definition and adequacy of programme or project for- mulation. Adequacy of information can be checked by monitoring the data systems for input indicators such as costs and manpower resources, and output indicators such as the health status of the working population , which are available in most countries from the records of responsible authorities. In testing the adequacy of problem definition, the main problems of OHS can usually be defined on the basis of multifactorial analysis and of morbidity statistics of the working population . The adequacy of the programme or project formulation can be tested by checking that the objectives, and the resources necessary to attain them, are defined , and that time schedules and outputs are formulated. For example, a national OHS development programme might include , as the definition of the problem, the insufficient coverage of OHS, especially in sparsely populated areas and in small industries. The ad- equacy definition of the programme would include goals for desired coverage figures, and time schedules according to which the programme would pro- gressively expand the coverage to the required level. Finally, the resources necessary to achieve the defined goals would be identified. Relevance By the relevance of an OHS programme is meant the extent to which the actions set out in the programme are likely to meet the needs or prob- lems which led to the setting up of the programme. For example , the programme outlined above for the development of OHS is irrelevant if an over-ambitious goal is set for providing service units with sophisticated analytical instruments when the field organization for taking samples or doing crude primary analysis with direct-reading instruments at plant level is lacking. Again , an irrelevant programme might train OHS personnel in developing countries to learn the highly sophisticated systems of indus- trialized countries when the OHS needs of the target country are clearly directed towards the development of manpower resources for solving classical problems of prevention and cure . An assessment of the relevance of a programme can be made by analys- ing the relevant outputs , effects and impacts of the programme and by com- paring the results with the policy goals and needs derived from a study of the indicators which characterize the main health problems of the working population . In Scandinavian countries , criteria for identifying the most important problems are volume , intensity , vulnerability , and trends in de- velopment of the OHS problem (17). 17 An aspect of the analysis of relevance is to ask whether the problem in question can be influenced by the methods and means of OHS at all. An ex- ample of an irrelevant programme by this criterion would be a programme for the intensive health examination of workers where hygienic and technical preventive measures for elimination of the causative exposure had been omitted from the programme. Progress Progress is analysed by comparing the actual phase of implementation of an OHS programme with the planned time schedule. Examples providing the basis for an evaluation of progress are supplied, at national level, by the time schedule for the progressive implementation of OHS in different branches of the economy by 1983 in Finland, and, at international level , by the rate of implementation of the CEC recommendations on the development of OHS in member countries. Input indicators for the analysis of progress include allocated financial, material or human resources, and achieved coverage of organized services when compared with planned intermediate goals or criteria. If deviations from planned schedules are found, their causes must be analysed. In certain cases, delays may be caused by factors external to the programme, which will thus not necessarily be invalidated. Efficiency Analysis of the efficiency of an OHS programme involves a comparative analy- sis of the inputs and outputs of the programme with other health service programmes. For example, the number of specialist man-years expended on various activities in OHS can be compared with the corresponding figures in another region or branch of the health services. In Finland this type of analysis has shown that the efficiency of OHS compares favourably with the general primary care services. Reliable analysis requires standardization of output units. The indicators of efficiency are the figures obtained when output results are related to the inputs of costs or other resources. Again , a deep knowledge of OHS practice is needed for relevant evaluation. Simplistic calculations may easily give misleading results; for example, plant visits of OHS physicians to highly automated production plants cannot be directly compared with plant visits in poorly mechanized small industries. Commonly used input indicators are the number of working hours of OHS personnel or the costs of OHS; the most usual output indicators are improvements in the health status of workers, expressed in terms of morbidity statistics, and improvements in working con- ditions, expressed in terms of reductions in the levels of toxic chemicals, noise , and other physical factors. 18 At regional , national and branch levels the number of specialist man- years allocated to OHS for certain defined populations can be used for comparative evaluation. At national level, the number of OHS personnel can be related to the size of the served population and compared with corres- ponding data from other countries or with international recommendations. It is worth emphasizing again, however , the risks involved in making over- simplified calculations without taking into consideration the variations in national systems outside the OHS which can substantively influence the efficiency of OHS. Effectiveness Effectiveness measures the degree to which the goals of the programme have been attained, i.e. the extent to which the programme has been able to respond specifically to the problem in question . It is possible for a pro- gramme to have been efficient in providing a great number of defined outputs, while these outputs have had a minimal effect on the problem to be solved . To take an example, an OHS programme may have carried out a great number of periodic health examinations with high efficiency, but have failed to take adequate measures to decrease the level of expo- sure which gave rise to the need for health examination: the programme has been efficient but not effective. To take another example, the expo- sure level to a specific chemical may have been decreased, at considerable expense, by detailed attention to ventilation and encapsulation of the work process, while an alternative chemical, equally suitable for the purpose and with virtually no health risk , has been overlooked . In this case the pro- gramme has been effective but not efficient. Relevant indicators for evaluating the effectiveness of OHS are stat- istics on occupational diseases, work disabilities and work accidents. These statistics are all indicators of the delayed effects of a risk factor. Examples of more immediate indicators are results of hygienic measurements, data on acute poisonings and behavioural effects of the working environment. When effectiveness is analysed the time lag between the implementation of a programme and its taking effect must be taken into consideration. For example, a decrease in cancer morbidity among workers who have had organ- ized OHS for less than five years is, provided turnover of workers has been low, almost certainly not an effect of the OHS programme, but a reflection of changes in exposure levels 20 years before. The problems of cost-effectiveness are discussed below. Further development The Working Group considered that future development should concen- trate on extending and improving evaluation at national level. This will 19 require a number of actions by countries in order to achieve a thorough level of OHS evaluation: (a) national data bases for inputs, outputs, and effects must be es- tablished; (b) aspects of OHS cover which need to be strengthened to meet the main problems of workers' health must be identified; (c) goals and programmes for the development of OHS must be formalized at national level; (d) methodology, concepts and definitions of evaluation must be standardized in order to enable intercountry comparisons to be made, at least on a restricted scale; and (e) in order to guarantee integration of evaluation with everyday decision-making processes, training in the methodology of evaluation must be organized for responsible authorities and specialists. Evaluation of impact Impact indicates the extent to which the programme has been able to improve the overall health of the working population when the goals of the programme have been achieved. The programme may have had high effectiveness, i.e. the goals of the programme may have been effec- tively attained , but with only a minor effect on the general health of the working population. Consider, for example, a country in which work accidents are a much more extensive problem than occupational diseases. A programme targeted towards the elimination of acute chemi- cal poisonings may well be effective as such, but fail to have any great impact on the health of the working population, whereas a programme on accident prevention could substantially improve the average level of health of workers. Relevant indicators for an analysis of impact are statistical indices of general morbidity and overall morbidity of workers, differences in mor- bidity patterns of workers served by OHS when compared with groups without OHS. Changes in the trend of overall morbidity within groups may also serve as an indicator. Parallel indicators, such as statistics on sickness absenteeism, and delayed indicators such as changes in the statistics of premature pensions and mortality among workers, are also useful. The analysis of impact is made the more difficult by a number of con- founding factors which can have negative or positive effects on health. 20 VARIABLES TO BE EVALUATED International comparisons General methods of evaluation of OHS programmes at international level have been discussed above. At present, however, the available information sources do not give enough data for comprehensive evaluation of OHS at all relevant levels. Comparative evaluation of national OHS programmes is a difficult task for several reasons : - there are differences in national traditions, legislation and practices - the basic philosophy of OHS provision varies from one country to another - data systems are poorly developed in several countries - organizational patterns may vary greatly. The Working Group decided to avoid analysis of organizational patterns of OHS provision, and to concentrate instead on the evaluation of variables that are to a great extent common to all Member States. The Group was unanimous that the organizational patterns of OHS must be chosen on the basis of national needs, and that in this respect technical efficiency is not the only criterion. Evaluation at national level At present there are several constraints on the evaluation of OHS at national level, the most important being the non-existence in some countries of nation- al OHS programmes and inadequate data systems for OHS. This makes com- prehensive analysis of OHS a difficult task. Nevertheless , the Working Group considered it important to carry out a partial evaluation, even if a complete analysis was impossible. At the same time, the Group recommended that governments encourage the development of sources that promote and facilitate evaluative activities. Variables at national level The Working Group identified a number of common denominators which could be evaluated at national level and which could form the basis for comparisons between countries, namely , OHS coverage , manpower resources , and coordination and co-determination of social partners. 21 Content of OHS The relevant content of OHS comprises those services which address effectively and specifically the most important work-derived health problems of the working population. Although the relevant content of OHS will vary accord- ing to local needs and conditions, the following components at least must be included: (a) identification and assessment of risk factors hazardous to workers' health and wellbeing at work and in the working environment; (b) collection and communication of information on risk identification and assessment, including health and accident records and records on hygienic measurements , and provision of advice and information to management and workers on risks and their prevention; (c) health examinations, including pre-employment, pre-placement and periodic health examinations as well as examinations of workers exposed to specific health hazards; (d) measures to adapt work and working conditions to the needs of the disabled, handicapped, chronically ill and other vulnerable and underserved groups, as well as investigations into specific requirements for special types of work; (e) measures to encourage workers to use rehabilitation services at as early a stage as possible after illness or injury in order to minimize the dur- ation and extent of disability; (f) health education of workers and management, directed specially at the prevention of specific risks at workplaces; general health education given in connection with OHS could be planned in collaboration with the authorities responsible for general health education within the framework of the public health services ; (g) preparation of facilities for first aid at the place of work; (h) guidance by health personnel in the selection, use , and assessment of personal protective devices; and (i) curative activities, when appropriate, in order to provide rapid out- patient services for workers and to provide clinical feedback on the efficiency of preventive measures. 22 In most European countries OHS content covers the majority of the above- mentioned components, although gaps might exist in certain areas. It was emphasized again by the Working Group that content must be related to needs as identified on the basis of risk-assessment, and also to the overall needs of health service provision for the working population. The Group was unanimous in recognizing the need to evaluate all components of the content of OHS together. The principal question to be asked in the evaluation of content is whether content is adequately related to the actual problems encountered in the working environment. For example, industrial hygiene is important in the case of manufacturing industry, but in office work other aspects of OHS con- tent, such as ergonomic and psychosocial factors, should attract more atten- tion. Thus, evaluation of the content of OHS forms part of the analysis of relevance of OHS programmes. Coverage of OHS provision In most countries the problem of coverage has proved difficult to solve. Workers in large, well-organized industries are usually sufficiently well served by OHS, but there have been difficulties in the provision of services in other areas such as: - small industries - the self-employed - transport and other mobile industries - the construction industry - workplaces in sparsely populated areas - agriculture. There are two main aspects to the evaluation of coverage of OHS. I . What is the proportion of the served population with respect to the total working population? 2. How well does OHS cover those places of work and populations that are most at risk? In most countries there is a coverage of only about 50-60%. Furthermore, OHS does not always cover work situations with the highest risks. On the other hand, in some countries a coverage of 100% is the official national goal. As a minimum interim objective, coverage must be sufficient to guarantee relevant services at least for all workers who are exposed to health risks in their work or working environment, including physical , chemical, physiological, 23 ergonomic, psychological, psychosocial and accident risks. The ultimate goal of OHS remains the organization of OHS for all workers, regardless of the size of plant, location of workplace , or type of work. Manpower resources and technical facilities The evaluation of manpower resources is an evaluation of an input factor and includes two separate components. 1. Quantitative analysis of manpower resources. Are the resources ad- equate with reference to the goals? 2. Qualitative analysis of manpower resources. What categories of speci- alist are needed? What are the criteria for evaluating the quality of resources? The Working Group made the point that it is not possible to define precise specifications relating, for example, the number of OHS specialists to the number of served workers. The need for specialist resources depends on several factors which vary from one country to another, and from one organizational pattern and branch of industry to another. Thus, the needs for specialist re- sources in big industry are different from the needs of small industry, the needs of in-plant services are different from those of inter-plant services, and the needs of densely populated areas are different from those of sparsely popu- lated areas. The need for resources is also highly dependent on the contents of OHS, and also on the extent to which curative medical care is carried out in parallel with preventive activities. In most European countries there is a severe shortage of specialists , which inhibits the effective development of OHS. The qualitative analysis of manpower resources is directed to two main questions. 1. What is the appropriate level of training and competence for special- ists in OHS? 2. What is the relevant composition of the OHS team? The Working Group recognized problems relating to both questions. There are only a few countries, for example the Federal Republic of Germany and the Netherlands, in which definitive competence criteria have been estab- lished for doctors working in OHS. Even in those countries where criteria ex- ist , their appropriateness in relation to the needs of OHS has not been evalu- ated . Furthermore, no international standardization of competence criteria exists . The Working Group agreed that manpower resources for OHS must be based on the training of specialists in well-defined skills of various kinds which can be presented in the form of competence criteria to allow their recognition by competent authorities. 24 Modem OHS requires employment of multidisciplinary expertise, which includes not only medical personnel but also industrial hygiene specialists , psychiatrists, safety engineers, psychologists, industrial toxicologists and others. This kind of multidisciplinary team can be organized only in big in- dustries. Medium-sized and small industries, and other under-served areas and groups can be provided with multidisciplinary services by organizing the pro- vision of services into units large enough to encompass the whole spectrum of OHS expertise . The need for various types of specialist service varies from one branch of industry to another, and the provision of services and composition of teams must be adapted to local needs. The Working Group emphasized the need to strengthen training pro- grammes for all the above-mentioned specialist groups working in OHS. Train- ing curricula must be designed to give students qualifications that can be re - cognized by the competent authorities. In the planning of curricula for medi- cal personnel, both preventive and curative activities of OHS must be taken into consideration. The successful practice of OHS requires instrumentation for measure- ments of risk factors at places of work, for sampling of various substances· to be analysed in laboratories , and laboratory facilities for the analysis of samples taken from the working environment and toxicological analysis of samples of biological fluids or tissues from exposed workers (biological monitoring). The quality and quantity of instrumentation depends on the content of OHS, on the variety of risk factors in the work or workplaces in question , and on the availability of analytical and hygienic services from external sources. Instru- mentation requirements can be easily assessed by an experienced OHS phys- ician or industrialist hygienist. Coordination of OHS with other relevant activities OHS plays an essential part in both health service and labour safety activities. In many cases OHS requires the active cooperation of workers and employers. The possibility of OHS activities overlapping with primary health care and public health services implies the risk of an inefficient use of resources. In addition, OHS activities may be poorly coordinated with labour safety func- tions, which can interfere with the achievement of OHS goals and reduce the efficiency of the use of resources in both types of activity. To avoid problems caused by poor coordination, effective communication between responsible bodies in the fields of general health care, OHS, and labour safety is needed. It is important that workers' health be seen as an entity, requiring a holistic approach in which both the preventive and curative health services, together with labour safety activities , are planned as a whole at plant level. This approach guarantees the most effective coordination of activities, and best corresponds to the interests of plant-level joint safety committees, which 25 in most countries participate in the planning, decision-making and steering of health and safety programmes. Co-participation of social partners in OHS Active participation of the social partners in the planning, programming, organization and evaluation of OHS at all levels is necessary if the goals of OHS are to be achieved. In some countries, for example Sweden and Finland, this participation is stipulated by legislation. In eastern European countries, participation of workers is a central principle of the organization of all activities concerning occupational safety and health. Member countries of the EEC have also been recommended to include the principle of co-participation in their OHS legislation. Participation must be adapted to national needs and policies. Estab- lishment of new administrative organizations for joint regulation of OHS is not recommended. COST - BENEFIT AND COST-EFFECTIVENESS OHS is a tool for helping to achieve the general goals of health policy among the working population. This goal has a value which cannot be measured solely in monetary terms. Thus, cost-benefit analysis is not used to decide whether or not OHS should be organized, but rather to establish the most effective OHS programme within the resources available. In fact, in the case of OHS, cost-effectiveness analysis is more relevant than cost-benefit analysis because of a number of intangible benefits which cannot be quantified in monetary terms. Cost-benefit analysis A restriction of cost-benefit (C-B) analysis is the fact that in most cases the quantity and quality of the resources available have been established before the goals of the programme are defined. In this kind of situation, analysis is targeted to determine the highest possible effectiveness which can be achieved with the use of resources of fixed volume. As with any type of evaluation the requirements for C-B analysis are defi- nitions of goals, inputs and outputs, and measurements of the effects of the programme. The process of C-B analysis is as follows: 26 (a) analysis of the situation in which a decision has to be made; (b) identification of options; (c) analysis of the essential variables (indicators and criteria) and their adequacy for C-B analysis; (d) identification and definition of the principles used in recognizing and measuring costs and benefits; (e) analysis of external limiting factors; (f) analysis of costs and benefits (large errors can occur at this point); (g) calculations of costs and benefits, transformed into monetary units for comparison; and (h) interpretation of results and conclusions. Cost-effectiveness analysis Cost-effectiveness (C-E) analysis includes the following steps : (a) analysis of the situation in which a decision has to be made; (b) identification of the principles which enable an assessment of the advantages and disadvantages of the programme to be made; (c) analysis using relevant indicators; and (d) interpretation of results and conclusions. A comparison of C-B and C-E analyses is presented in Table I . Problems which arise with C-E analysis include the selection of in- dicators and criteria, measurements of intangible costs, benefits and ef- fects, and compatibility of the various types of criteria with the mathe- matical model employed. CONCLUSIONS After consideration of the documents prepared for it by some of its members and by the Regional Office for Europe, the Working Group reached the following unanimous conclusions. l. The ultimate goal of occupational health services is to promote con- ditions at work which maximize the quality of working life by : - protecting workers' health - enhancing physical, mental and social wellbeing - preventing ill health and accidents. To achieve this goal medical , industrial hygienic, and related expertise is required. 27 Table 1. Comparison of application areas of cost- benefit and cost-effectiveness analysis Type of concern Purpose of analysis Level of ambition Time horizon Units of calculation Discounting Comparison of differ- ent programmes and setting of priorities Maximand "Benefits" Cost-benefit analysis Often more general; an instrument for setting rough priorities among major disease pro- grammes Very high Often longer Monetary units Yes, both costs and benefits Yes, implicit in analysis (investment criteria) National product Ideally everything of concern Source: Kamper-J4rgensen ( 18) . 28 Cost-effectiveness analysis Often specific; often technical analysis following a political decision in favour of a specific disease pro- gramme More modest Often shorter Cost side in monetary units; effectiveness side in health or performance units Yes, costs only Usually no, since costs and benefits (effec- tiveness) are not com- mensurate Depends on criteria selected Selected variables only 2. Evaluation is an essential part of the planning process and should be carried out as an integral part of the development of occupational health service programmes at all organizational levels. The main purposes of evaluation are to analyse the present status of occupational health service programmes in relation to their goals, and to provide data upon which justification for the use of resources can be based. More generally , evaluation promotes enhanced efficiency and stimulates the further de- velopment of OHS. 3. In order to permit such an evaluation the goals , intermediate objec- tives, policies and procedures should be clearly defined and established by legislation. 4. The methods of systems analysis can profitably be used in the evaluation of occupational health service activities. The analysis can be divided into an evaluation of external and internal effectiveness of the programme. Analysis of internal effectiveness should include an evaluation of relevance, adequacy, progress and efficiency of the programme, while in the case of external effectiveness, planned effects and impact should be examined. 5. In addition to the evaluation of occupational health service pro- grammes in terms of their success in promoting the goals listed in paragraph 1 above, an essential tool of decision-making is the economic approach con- cerned with the comparison and choice between various alternative options. 6. Economic analysis should be multidisciplinary and its main use- fulness is the active search for alternatives in terms of resources, services and programmes. There are various methodologies for comparing the advan- tages and disadvantages of different programmes, and of alternative projects and activities within a given programme: - systems approach and systems analysis - decision-making methods - organization and management studies - epidemiological and behavioural methods - economic methods. 7. In cases where insufficient information prevents a complete analysis of all the aspects mentioned in points 4-6 , an evaluation should nevertheless be made on the basis of available data . 29 8. There are several components of occupational health services which could be considered as common to all countries and which should be evalu- ated using a common methodology, for example : - content of services - coverage of service provision - manpower and technical resources - extent of coordination with other related activities - participation of the social partners in the development of occupational health services. 9. All the competent bodies, including authorities and social partners, must be given an opportunity to participate in the definition of goals and the design of occupational health service programmes, and to take part in the pro- cess of external evaluation. Internal evaluation should be the responsibility of the relevant authorities and specialists in the occupational health services. 10. A critical factor in the development of occupational health services is the shortage of adequately trained manpower resources in occupational medicine, industrial hygiene, occupational health service nursing, toxicology, psychiatry, occupational psychology and safety engineering. RECOMMENDATIONS 1. Where the goals of the occupational health services have not been defined, every effort should be made by governments to develop such goals and to encourage the design of occupational health service programmes at all levels. 2. The evaluation of occupational health services should be a regular activity, fully integrated into the planning and implementation of occupa- tional health and safety programmes. The relevance of programmes, and the effective use of resources, is facilitated by appropriate and regular evaluation. 3. All the responsible authorities, including the social partners, should be involved in the evaluation of occupational health services. 4. In the evaluation of occupational health services, health, social, technological and economic factors should be taken into account. Economic analysis should be applied as part of the evaluation process at all levels. 30 5. Sufficient data are needed at national, regional and local levels for successful evaluation, and responsible authorities are encouraged to establish appropriate information systems. 6. It is recommended that the development of adequately trained man- power resources should be strengthened . This will involve specialists from various disciplines such as occupational medicine, industrial hygiene, occupa- tional health service nursing, toxicology, psychiatry, occupational psychology and safety engineering. Training should be sufficiently formalized to ensure recognition by competent authorities. 7. The World Health Organization , in collaboration with the Interna- tional Labour Office and with other interested organizations, should keep the question of evaluation of occupational health services under review. REFERENCES 1. WHO Official Records, No. 231, 1976 , p. 225. 2. WHO provisional guidelines for health programme evaluation (unpub- lished document HPC/DPE 78.1). 3. Shigan, E.N . & Gibbs, R., ed. Modeling health care systems. Proceedings of a IIASA workshop, 28-29 March 197Z Laxenburg, Austria , Inter- national Institute for Applied Systems Analysis , 1977. 4 . The evaluation of public health programmes: report on a Working Group. Copenhagen, WHO Regional Office for Europe, 1973 (unpublished document EURO 4004). 5. WHO Technical Report Series , No. 528 , 1973 (Evaluation of environ- mental health programmes: report of a WHO Scientific Group). 6. Council resolution of 28 June 1978 on an action programme of the European Communities on safety and health at work. Official journal of the European Communities, 21: Cl 65 (1978). 7. Proposal for a Council Directive on the protection of workers from harm- ful exposure to chemical, physical and biological agents at work. Official journal of the European Communities, 22: C89 (1979). 8. Proposal for a Council Directive on the protection of workers from harm- ful exposure to metallic lead and its ionic compounds at work. Official journal of the European Communities, 22 : C 324 (I 979). 9. Tolonen, M. & Hassi , J., ed. (Organizing occupational health services for small industries in a community health centre.] Tyi/terveyslaitoksen tutkimuksia, 143: I -182 (l 978). 31 10. Ulfvarsson, U. [The physical working environment in small indus- try.] Stockholm, Arbetarskyddsstyrelsen, 1976 (Undersokningsrap- port AMT 106/76). 11 . Atherley, C.R.C. et al. An approach to the financial evaluation of occupa- tional health services. Journal of the Society of Occupational Medicine, 26 : 2 1 -30 (1976) . 12. Social section: cost effectiveness of occupational health programs. Journal of occupational medicine, 16: 153-186 (1974). 13. Recommendation concerning occupational health services in places of employment. Geneva, International Labour Office, 1959 . 14 . Workers ' health programme: progress report by the Director-General. Gen- eva, World Health Organiza tion , I 980 (unpublished document A33/1 2). 15 . Methods of evaluating public health programmes: report on a Sympo- sium. Copenhagen, WHO Regional Office for Europe, 1968 ( unpub- lished document EURO 0375). 16. WHO Technical Report Series , No . 596 , 1976 (Application of systems analysis to health management: report of a WHO Expert Committee). 17 . Rantanen , J. New developments and policies in the optimisation of the working environment. In : Proceedings of the International Sy mposium on New Trends in the Optimisation of the Working Environment, Istan- bul, 16-1 9 May 1979. Geneva, International Labour Office, 1979 . 18 . Kamper-J~rgensen, F. Scien tific methods in long-term outcome evalua- tion of preventive health programmes . A critique of the randomized controlled trial . Scandinavian journal of social medicine, Suppl. 13, pp. 8 1-9 I (1978). 32 Annex LIST OF PARTICIPANTS Temporary advisers Dr C. Chambet , Directorate-General of Labour and Employment, Minis- try of Labour, Paris, France Dr R. Edstrom, Medical Director, National Occupational Safety and Health Administration, Solna, Sweden (Vice-CJzairman) Dr J.G.H. Franz, Federation of Professional and Trade Associations in Industry , Bonn , Federal Republic of Germany Dr T. Guthe , Corporate Medical Director, Elkem-Spigerverket a/s , Oslo, Norway Professor J. Indulski , Director, Institute of Occupational Medicine , Lodz , Poland ( CJzairman) Dr V. Kodat, Ministry of Health of the Czech Socialist Republic , Prague , Czechoslovakia Dr R. Murray , Quality House , London, United Kingdom Dr J. Rantanen, General Director , Institute of Occupational Health , Helsinki , Finland (Rapporteur) Dr K. Ruppe, Central Institute of Occupational Medicine , Berlin- Lichtenberg, German Democratic Republic Professor M. Saric, Institute fo r Medical Research and Occupational Health , Academy of Sciences and Arts, Zagreb , Yugoslavia Dr M. Spasovski, Institute of Hygiene and Occupational Health, Sofia, Bulga ria Professor E.C . Vigliani, President, Permanent Commission and In te r- national Association on Occupational Health, Carlo Erba Found- ation, Milan, Italy 33 Dr V. de Vicente Martin, Chief, Department of Occupational Heal th, Ministry of Public Health and Social Security, Madrid , Spain Dr M. Ylikoski, Ministry of Finance, Helsinki, Finland Representatives of other organizations Commission of the European Communities Dr W J . Hunter , Health and Safety Directorate , Brussels, Belgium International Labour Office Dr M. Stilon de Piro, Occupational Safety and Health Branch, Geneva , Switzerland Observers Ms E. Allgulander, Ministry of Health and Social Affairs, Stockholm, Sweden Dr E. Bolinder , Swedish Confederation of Trade Unions, Stockholm, Sweden Mr P. Granicki , Institute of Occupational Medicine, Lodz , Poland Ms E. Kvarfordt, First Secretary, Ministry of Labour, Stockholm, Sweden Dr N. Masreliez, Head , Industrial Medicine Section , Swedish Employers' Confederation , Stockholm, Sweden Dr U. Nicolausson , Head of Section, National Board of Health and Wel- fare, Stockholm, Sweden Dr L. Rinder , Head of Division, National Board of Health and Welfare, Stockholm , Sweden Ms Y. Wessman , Head of Division and Secretary to the Swedish Occupa- tional Health Services Commission , National Board of Occupational Safety and Health , Solna, Sweden WHO Regional Office for Europe 34 Dr M.I. Mikheev, Regional Officer for Workers' Health (Secretary) Mr J .I. Waddington, Director, Promotion of Environmental Heal th Dr H .F .K. Zo1lner, Regional Officer for Health Economics WHO publlcatlona may be obtained. direct or through boob1l1r1, from: ALGERIA ARGENTINA AUSTRALIA AUSTRIA BANGLADESH BELGIUM BRAZIL IURMA CANADA CHINA COLOMBIA CYPRUS CZECHO- SLOVAKIA DENMARK ECUAOOII EGYPT EL SALVAOOII FIJI FINLAND FRANCE GERMAN DEMOCRATIC REPUIUC GERMANY. FEDERAL REPUIUC OF GHANA GREECE HAm HONG KONO HUNGARY ICELAND INOIA INOONESIA IRAN IRAQ IRELAND ISRAEL ITALY JAPAN KOREA u,ua,co, ltUWAIT LAO PIOfLn -TIC UIANON Sociae Nalionale crEdilioll et de Diffusioa. J bd z;,- Youml. Al.Gas Cartoo Hindi SRL Florida 165. Oalerial Gaea.. milorio 453/465. llulNol Aaa ftltlil 0,., s.Jn: Aullraliu ao-n- l'llblillli .. Serville. P.O. loa 14. C•- A.C.T. 2600; or - tlw .-wr f-: Aullralian Oowem- l'llblilllill( Senice 8ootshopl •t: 70 A1inp SlNII. c-Cm A.C.T. 2600: 294 Adelaide Sa-..._.., QINeMlalld -,.,: 347 S-.- Sliwa. Ma• _,.,... VIC 3000: 309 Pin SlNII. SvDNIY. N.S.W. :ZOOO; Mt ,..._ H- 200 St. 0-.'• T- Plmt. WA eGOO: IINlullry ~ 12 Pirie 51,.._ Alm-. SA 5000: 156-162 Mac,quarie 51,.._ HoeAn, TAS 7000 - Hu- l'llblicaionl. 51A Gippa 51,.._ COWNOwooo, VIC 3066 - R. Hill & Son Lid., 60I Sl. Kilda Road. M._....... VIC 3004; La-~ IG-12 Cllrl< SlNII. C:-'1 Nm, NSW 2065 Gerold & Co. Graben 31, 1011 v- I The WHO.......,.._ C--.li-. G.P.O. loll 2'0. l>Mx• S-The Aaociatioa ol Vohuury A,1acia. P.O. loa 5045. l>Mx• 5 Office illlCmllionlll de Liblairie. 30 -• Marnia. 1050 ~ -~ ,o Worl,I H-'111 .,,,,_,._. Jeu de .._.,,,,, 202 ·-.. Roi, 1060 ea_. liblioleca Reaiollal de Medici• 0MS,'OPS. Unidade de Vt811a de ~ Caixa Palla! 20.311, Vila 0.-ino. NW~ P-.o, S.P. - India. WHO ....... Off'a ~-6ollt <o,ia af illtlttiaMI ,-Ho:rltw ,_ -.6la$11Nu): C■-liu Public Heallll Aalcialion, 1335 Carli .. A-. Suile 210. OffAWA, 0nL KIZ INII Wn ,.,.,_, ~ ..... -•..-1iN' h.1·dw.,. ... _ ,o,. Royal ... ol~ 0Uwa. Aalaolm World H..iih 0...--. MI/IW/lk _,IO. World Haldi °'lllaiaaiaft. P.O. loll IIOO. l"aal 5ulioa B. OffAWA, 0.. KIP SIU. C- ..,.,.,,,., ~ ....,,,.,_ MI/IWII h ..__. 1/w World Haldi~ DiarilMioll and Sales, 1211 o..v. 27, s.ilarluld China ,...._. Publicaianl lfflllOII Corp,ntioa. P.O. loll a. .._ Cl'lluNo) Dislrililllal LMI. Pio Alf- Oalcla, Carma 4a. NIii ~I 19, CAIT- l'llblishen' Dillribulon c,,.... 30 0.-.ia Ave Ayiol oi-iaul. P.O. loll 4165, .._,. Ania, Ve s...cbdl JO. 111 27 l'IIMlul I Mun...-,.i Eapon and Sulllcription Serva....,....,._ JS, mo c:-- K Libmta Cienllfic■ 5.A. P.O. loa .162. Luque 223, OuAYAQI& Oliril Off'a ror laou and Re¥iewl. SO Ka. El Nil Sa-. C- 1..ibnrta Eaudianlil. Edificio C-.:ial I No 3. A....- Ubenad. SAN 5-1.vAIIIJa The WHO ~-c~. P.O. loa 113. 51,v,. ~i- KiljabuppL "-ltlllUIII 2.00101 H-- 10 Libnirie A-. 2 .,. Calimir-O■lavipa. 75006 p- luchhaul uiim1, Pm1rac11 ,...,_ ,01 ~ Oovi-Verl■a GmbH. Ginnhei-- :ID. l'mlfadl SleO, 6236 ElaaoaN - W. E. 5Mrlladl. Paaradl 101610. Fcilleni- 2. 5000 COLCJOHa I - Alla. Hora. Sp ' 1111aM11 9. Paaradl 3340. 6300 W-- f"ldes Elllerpl'ia. P.O. loll 1621. Aocu G.C. Elertheroudwl S.A. Ubnirie ;__,...._ ,_ Niu 4, A- er. 126) Mu ~ Ubrairie "A la eam.111", .......... 111-B. Pon...w-PIINc■ Hont ~ Oowemmen1 lnr-ion S.V-. a.-n.rield ~ 6th Floor, QuNa'a Road. CeMnl. VICftlalA Kullu,a. P.0.1. 149, "--62-Abd6miai ~ V•-22. .__v Snaebj9rn '-& Co.. P.O. loll 1131. IW-...i 9, RmuA- WHO Ralional Off'a ror SoUlll,l!aa Aaa. World Haldi~,.....,..._ EataM. Rilla Road. Nlw Dlull 114002 - Odonl llloll & 5laliolllry Co., !cindia ~ Nlw Olull 110001: lf"l'lrll 5lNII. CALCVffA '916 C.W..,0,1 M/1 Kai- llloll Service Lid.. K ..... Ila,- No. 11, P.O. loll 3105/Jb. JMMTA ,.....,, .\-•TJWI llillribulion ,.._,, ISi DiaNIIS-,., TWAN Miniltry ol lnror-ion. ,...._. Hcue ror Nllilllilll, Diluilluli• ud Alhfflilil'I, ■-­ TIii 5utioeer7 Off'a. 01.-... Heiu,er&Co..3Nllllu~a-.,- Edizioni MiMrvl Medici. Cono .._...13-15, 10126 n-: V'■ ~ J, 20100 MuN Man&laft Co. I.Ml., P.O. loll 5050, TOKYO 1--...I. 100-31 TIii WHO........_ c:cionu-, Cealnl P.O. loll 540. 5alL 1'11...ii ........ CaLlll.,....AMJt -f.0. .. aG.luwMr 1'1WIIO ....... ,C S ,P.Cl .. JG,YwR.- n.a_. Dillal 1111 Cal.AJLI-. .. 1111......_...._ .._ _ -.r A/1/11 WHO publications may be obtained, direct or through booksellers, from : LUXEMBOURG MALAWI MALAYSIA MEXICO MONGOLIA MOROCCO MOZAMBIQUE NEPAL NETHERLANDS NEW ZEALAND NIGERIA NORWAY PAKISTAN PAPUA NEW GUINEA PHILIPPINES POLAND PORTUGAL SIERRA LEONE SINGAPORE SOlJTH AflllCA SPAIN SRI LANKA SWEDEN SWITZERLAND SYRIAN ARAB REPUBLIC THAILAND TUNISIA TURKEY UNITED KINGDOM UNITED STATES OF AMERICA USSR VENEZUELA YUGOSLAVIA ZAIRE L1bn11rie du Centre, 49 bd Royal. luXEMIOUl0 Malawi Book Servoa,, P.O. Box 30044, Choch1t1, lluP<TYU 3 The WHO Programme Coordinator, Room 1004, Fitzpetrock Buikhng. Jalan RAJ• Chulan, KuAU luMPUl 0S-02 - Jubtlee (Book) Store Lid, 'YI Jalan Tuanku Abdul Rahman, P 0 . Box 629, KUAU luMl'Ua 01-08 - Parry's Book Center, K. L Hilton Hotel, Jin. Trcacher, P.O. Box 960, KuAU luMPUl La Prensa Medoca Mexocana, Ediciona Cien11rocu. Pueo de las Faculu.des 26. Apt. Postal 20-413, MEXICO Cm 20, D.F. sn India, WHO Rq,onal Offoa: Ediuons La Pone, 281 avenue Mohammed V, lluAT INLD, Caixa Postal 4030. MAl'VTO sn India, WHO Rq,onal Off oa, Medical Boolu Europe BV, Noorderwal 38, 7241 BL l..ocHEM Government Pnn11n1 Offoa,, Publicatoons Sectoon. Mulpave St,-. Pnvate Bag. WEUJNOlON I; Walter Street. Wo..uNOTON ; World Trade Building. Cuboalde, Cuba St,-. WEUJNOlON. Go_, IJoo4.sJtops at· Hannaford Bu non Building. Rutland Street, Pnvate Bag. AUCltUND; I 59 Hereford Street. Pnvate Bag. CHllSTCHUlCH; Alexandra Street. P.O. Box 857, H.uat.lON ; TA G Bu11di111, Pnnces Street, P.O. Box 1104, DuNEDIN - R. Hill A Son Lid, Ideal Houae, Cnr Gillies Avenue A Eden St,-. Newmarket. AUCltUNO I University Bookshop NiteN Lid, University of Ibadan. IAAIMN J. G. Tanum A/S, P.O. Box I In Sen!Nm, OsLO I Mirza Book Aa,ency, 65 Shahral>-E,..Quaod-E.-Azam, P.O. Box 729, LAHou 3 The WHO Programme Coonl111111<>r, P.O . Box 5896, lloaol<o World Health OrpnlDloon, Reaional Offoa, for the Western Paaric, P.O. Box 2932, MANllA - The Modem Book Company Inc., P.O . Box 632, 922 Rizal Avenue, MANllA 2800 Skladnica Ksiqarska, ul Mazowiecu 9, 00052 W"-""w (ucrpt pn,od,cals) - BKWZ Ruch, ul Wronia 23, 00840 W "-"" w (pn,odia,b o,o/y) LivraN Rodrip,es. 116 Rua do Ouro, 1....- 2 Njala Un1~ty Collqo Boouhop (University of Sierra Leone), Private Mail Bag. Fum>WN The WHO "'--mrne Coonl1na10r, 14-4 Moulme1n Road, G.P.O. Box 3457, SINGAl'Olf I -Select Bpolu (Pie) Lid. 215 fansJin Shoppinc Centre, 2/F, 19 TansJin Rood. SINOAIOAE 10 Van Schaik's Boolutorc (Ply) Lid. P.O. Box 724, 268 Church Street. PlnoaiA 0001 Corncmal Athcncum S.A., ConscJO de Cicnto 1~136. 11.utCll.ONA IS ; General Moocard6 29, MAOII0 20 - Libreria Diaz de San1oo, laplca 95 y Maldonado 6, MADllD 6; Balmes 417 y 419, 11.utcnoNA 22 sn India, WHO Rq,onal Offoa, Aktiebolq,t C.E. Fritza Kuna!. Hovbokhandcl. Rqcrinpp1an 12, 10327 STOCIUtOUI Mediz1nwcher Verlq Hans Huber, Ungasa SlruK 76, 3012 IIEllNE 9 M. Farras Kekhia. P.O. Box No. 5221. A1DfO Stt India, WHO Rqoonal Offoa, Societe Tunisienne de D,lfusoon, s avenue de Canhqc, TUNIS Hasct Kitapcv1, 469 lstoklal Caddai, Bc)-oB)u, lsTANIUL H.M. Slauoncry Office : 49 Hiah Holborn, l..oNOON WCIV 6HB; 13• Casile Street, ECMNIUlOH EH2 JAR ; 41 The Hayes. CAICMF"f CFI IJW; 80 Chichester Street, llnfur BTI 4JY ; Braunnose Slrce1, MANOIDTD M60 SAS; 258 Broad Street, 81lMINOHAM Bl 2HE; Southey House, Wine Street, BalSTOL BSI 2BQ. All ma,/ ordns sliould ~ s,nt to P.O. Bo• 569, l..oNOON SEI 9NH Sing/, and bulk cop1u of indmdliOI publoca110,u (nor subscnp11o,u): WHO Pubhcat1ons Centre USA. 49 Shcndan Avenue, ALaANY, N.Y 12210. Subscnp11ons · Subscnp11on ortkn, acrompanr,d b_v ch«k nuuh out to th, Chemical Bank, New York. Account World Health Orpnizatoon, should~ s,nt to th, World Health Orpnwat10n, P 0 . Box 5284. Church Street Slat10n. NEW YoaK. N.Y. 10249; Cof?Pspond,nc, cort- a,m1ng n,l,,cnp11o,u should~ add,us,d 10 rh, World Health Orpnozauon. Distrobutoon and Sales. 1211 GENEVA 27, Switzerland. l'llblicallOns arr also ava,labl, from th, United Na110ns Bookshop, NEW YOlK. N.Y. 10017 (rrtarl onlv) For r,ad,r, •• r/t, USSR ITtfU'""f Rws,an ,d1110,u Komsomolsk1j proopckt 18, MedlClnskaja Knip. Moscow - For muJ,r, ouu,d, rh, USSR ITtfU'""f Ruman tdillo,u . Kuzrieck1j moot 18, Mddunarod- naJa Knip. Moscow G-200 Editonal lnteramerocana de Venezuela C.A., Apenado S0.785, CAllACAS 105 - Libreria del Este, Aper- lado 60.337, CAllACAS 106 - Libreria Modica Pans. Apenado 60.681. CARACAS 106 Jugoolovenska KnJ1p. TentzlJC 27/11. 11000 lla.GllADE Librairie unoven11a1re, avenue de la Paox N• 167, B.P 1682, KINSHASA I Special terms for developing co411tries are obtainable on application to the WHO Programme Coordinators or WHO Regional Offices listed above or to the World Health Organization, Distribution and Sales Service, 1211 Geneva 27, Switzerland. Orders from ccuntries where sales agents have not yet been appointed may also be sent to the Geneva address, but must be paid for in pounds sterling, US dollars, or Swiss francs. Price: Sw. fr ◄ .- Prices are subject to change without notice.