Technology Appraisal Programme of the National Institute for Clinical Excellence A review by WHO Suzanne Hill Silvio Garattini Jos van Loenhout Bernie J. O’Brien Kees de Joncheere June-July 2003
Ref. no.: 5045738
ABSTRACT The National Institute of Clinical Excellence (NICE) is responsible for providing guidance to the National Health Service in England and Wales on the clinical- and cost-effectiveness of medicines and medical technologies. After a review of the Institute by the House of Commons Health Select Committee in 2002, NICE commissioned a series of internal reviews on their work, and requested the WHO Regional Office for Europe to carry out an external review on their methods and processes as well as their scientific robustness. The WHO team of experts based their report on the review of a series of technology appraisals, including confidential material, and extensive discussions with NICE staff, Appraisal Committee members, members of the Technical Assessment Groups and other stakeholders during the course of two visits to NICE in summer 2003. The report contains a series of recommendations on how NICE could further develop the technology appraisal process. As NICE is internationally a leading agency in technology assessment, the important recommendations and observations contained in the report will be useful to other countries that are embarking on similar endeavours.
KEYWORDS TECHNOLOGY ASSESSMENT, BIOMEDICAL PHARMACEUTICAL PREPARATIONS – STANDARDS COST-BENEFIT ANALYSIS ECONOMICS, PHARMACEUTICAL EVALUATION STUDIES EVIDENCE-BASED MEDICINE UNITED KINGDOM EUROPE
Address requests about publications of the WHO Regional Office to: • by e-mail publicationrequests@euro.who.int (for copies of publications) permissions@euro.who.int (for permission to reproduce them) pubrights@euro.who.int (for permission to translate them) • by post Publications WHO Regional Office for Europe Scherfigsvej 8 DK-2100 Copenhagen Ø, Denmark © World Health Organization 2003 All rights reserved. The Regional Office for Europe of the World Health Organization welcomes requests for permission to reproduce or translate its publications, in part or in full. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Where the designation “country or area” appears in the headings of tables, it covers countries, territories, cities, or areas. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. The World Health Organization does not warrant that the information contained in this publication is complete and correct and shall not be liable for any damages incurred as a result of its use. The views expressed by authors or editors do not necessarily represent the views of their institutions or the decisions or stated policy of the World Health Organization.
CONTENTS Page List of abbreviations and definitions ..................................................................1 Acknowledgements ...........................................................................................2 Executive summary ...........................................................................................3 Recommendations ............................................................................................5 1. Introduction..................................................................................................10 1.1 Background to the review ......................................................................10 1.2 The health policy context .......................................................................12 1.3 What is NICE? .......................................................................................13 2. Methods and approach of the review ..........................................................15 3. Findings and recommendations ..................................................................17 3.1 Principles that determine the NICE approach to technology assessment. ..........................................................................17 3.2 The process of Guidance development .................................................19 3.3 The quality of the science of the assessment and appraisal process..................................................................................34 4. Conclusions.................................................................................................39 References ......................................................................................................41 Annex 1: Persons interviewed (in person or by teleconference) .................42 Annex 2: Documents reviewed ....................................................................45 Annex 3: NICE processes ............................................................................46 Annex 4: Process of topic selection .............................................................48 Annex 5: Comparison of recommendations in NICE documents .................49 Annex 6: Comparison of international recommendations.............................50
List of abbreviations and definitions AC ACD Consultee Appraisal Committee Appraisal Consultation Document Organizations from the following areas that accept an invitation to participate in the appraisal: the manufacturer(s) or sponsor(s) of the technology; national professional organizations; national patient organizations; Department of Health and Welsh Assembly Government ; Primary care Trusts in England and Local Health Boards in Wales Canadian Coordinating Office for Health Technology Assessment Chronic myeloid leukaemia Committee on Proprietary and Medicinal Products Health Insurances Board of The Netherlands Executive Lead (on each technology appraisal) Final appraisal determination Published document issued by NICE describing appraisal of technology (from NICE) published document describing the appropriate treatment and care of people with specific diseases and conditions within the National Health Service in England and Wales Health technology assessment National Coordinating Centre for Health Technology Assessment National Health Service National Institute for Clinical Excellence Provisional appraisal determination (now the Appraisal Consultation document Pharmaceutical Benefits Advisory Committee (Australia) Project Manager Quality adjusted life year Research assessment exercise Initial document describing the question to be addressed in a NICE technology appraisal Technical Assessment Group Technical assessment report Technical Lead World Health Organization United Kingdom
CCOHTA CML CPMP CVZ EL FAD Guidance Guideline
HTA NCCHTA NHS NICE PAD PBAC PM QALY RAE Scope TAG TAR TL WHO UK
1
Acknowledgements For the Review Team, undertaking this review was an extraordinary opportunity and privilege. The chance to assess such a high profile and innovative agency is not one that occurs frequently and the Review Team hopes to have been able to provide an assessment and recommendations that will assist The National Institute for Clinical Excellence (NICE) (and others) in undertaking the vital task of health technology assessment (HTA) and guidance for the improvement of the quality of health care. The Review Team would like to acknowledge the generous assistance and co-operation provided by everyone they met during the course of the review. In particular, the Team would like to thank Professor Sir Michael Rawlins, Chairman of the Board of NICE Mr Andrew Dillon, Chief Executive Professor David Barnett, Chair of the Appraisal Committee Dr Carol Longson, Appraisal Programme Director for their hospitality and enthusiasm which was manifest in the time they spent in discussion with the Team. The Team would also like to thank all the members of the Appraisal Committee, the appraisal teams, and the additional experts and interviewees (Annex 1) who very kindly agreed to be interviewed at very short notice. Again, the Team’s job was made substantially easier by the openness of these people and their enthusiasm about contributing to the to the review. The review would not have been possible at all without the assistance of the NICE administrative personnel and especially Kathleen Dalby who dealt with all the logistics of the visit, including the arrangement of 50 interviews at two weeks’ notice and the provision of endless numbers of documents.
2
Executive summary NICE was established as a Special Health Authority to address a number of problems with the introduction and use of new technologies and medicines within the United Kingdom National Health Service (NHS). The review of NICE carried out by the House of Commons Health Select Committee in 2002 identified a number of issues and concerns and recommended an external review of the Institute, its functions and processes. NICE requested the World Health Organization (WHO) Regional Office for Europe to carry out this review. The terms of reference for the review focused on the methods and processes used in the NICE technology appraisal programme. The review was undertaken by a team of international experts in June-July 2003. The conclusion of the Review Team was that, in only four years, NICE has developed a well-deserved reputation for innovation and methodological developments that represent an important model for technology appraisals internationally. In the environment in which NICE functions, substantial changes are taking place in the way in which the NHS operates and additional resources are being invested in the health services. This facilitates the achievement of the NICE objectives of improving quality of care throughout the health services and the uptake of new cost-effective medicines and technologies. Achievements that are particularly valuable include: the transparency surrounding the process of technology assessment; the intensive participation of different stakeholders and the inclusiveness of the approaches taken; the commitment to using the best available evidence for decision-making; the commitment of the technical and management staff; and the dedication of the appraisal team and the Appraisal Committee members. All of these form the cornerstones of an organization that continues to invest in quality development. Overall, the Review Team was impressed by the commitment to using of rigorous methodology throughout the process of technology assessment, starting with the use of academic centres of excellence for independent technology appraisal, which is supplemented by the additional evidence handled by the technical teams in NICE. Published technology appraisals are already being used as international benchmarks - an obvious recognition of their credibility. As part of the commitment to further develop and improve methods and processes, NICE is currently carrying out internal reviews of these areas. In undertaking the review and preparing the report, rather than reconfirming the well-established achievements of NICE, the Review Team focussed on the areas where there is controversy or potential for improvement. A number of key recommendations are worthy of specific mention here, namely:
3
•
To reconcile the inherent contradiction between the Institute’s key principle of transparency and the acceptance of material that is designated as confidential. The Review Team welcomes the steps that NICE has already taken to date, and encourages NICE to use its leverage to advance transparency in this area. This would be of benefit internationally. To revise the contractual arrangements for the development and production of assessment reports so that they fully meet the needs of NICE and its Appraisal Committee. To consider ways to reduce unnecessary duplication of effort in the assessment phase. Efforts should be made to ensure that the Appraisal Committee is presented with a single set of analyses produced by the Technical Assessment Group that incorporates consultation with and input from the manufacturer(s). To ensure the collection of relevant information from all stakeholders as early as possible in the assessment process. To increase the exchange of information and interaction – at all stages of the process - between the Technical Assessment Group, the Appraisal Committee and the NICE Appraisal Team. To develop a handbook on the preparation of assessment reports; this could be a product of the current review on methods and process. To make clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than representation of particular interests. Although there is a need to ensure that the manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. To continue to work on the difficult but important task of developing clear criteria to obtain optimal interaction between the ethical and social values and the scientific evidence for use by the Appraisal Committee and NICE in decision-making.
•
•
• •
• •
•
The recommendations are to be seen as an effort to further enhance the operations of NICE, and assist organizations with similar responsibilities in other countries to deal with their difficulties and meet their expectations.
4
Recommendations On the principles 1. NICE should continue to develop operational procedures that are consistent with its core principles of transparency, consultation and inclusiveness with respect to stakeholders’ involvement in evidencegathering and decision-making. The NICE model of partnership in the scientific endeavour of health technology appraisal offers valuable international leadership. 2. NICE should reconcile the inherent contradiction between its principles of transparency and its acceptance of evidence for the decision-making process that a stakeholder deems to be confidential. NICE should continue the work already started on this issue to ensure that all material submitted for consideration can be made available to the public. 3. The principle of transparency requires that NICE codifies and justifies the specific criteria used in decision-making. Difficult but important elements of this task are articulation of the ethical and social value judgements, and definition of the interaction of these judgements with the appraisal of the scientific evidence used by the Appraisal Committee in reaching its decisions. On the process – general issues 4. For drug and device technologies where a sponsor exists, the current hybrid process of HTA used by NICE may give rise to unnecessary duplication of effort. NICE may wish to consider using different approaches depending on the subject of the appraisal. The Review Team suggests that an effort be made to ensure that the Appraisal Committee is presented with a single set of analyses produced by the Technical Assessment Group that incorporates consultation with and input from the manufacturer(s). On the start of the process and topic selection 5. The consultees’ meeting should become a formal ‘Preliminary Exchange of Evidence’, at which time all stakeholders should be asked to provide details of what they propose to submit. Only in extraordinary circumstances should the Technical Assessment Group accept to include, as part of the review, information over and above that declared for submission by the stakeholders at this meeting. 6. More attention needs to be paid to the important task of setting the Scope of the appraisal. The general goal is to give timely and comprehensive guidance to the Technical Assessment Groups on the question(s) to be addressed by the Appraisal Committee, thereby reducing the risk of mismatch between the technical assessment report and the needs of the Appraisal Committee. 5
7. Stakeholder submissions should be required to be lodged as soon as possible after the start of the assessment process. On the assessment and appraisal process 8. The timeframes for the assessment and appraisal process should be reviewed, so that the current time pressures at the end of the assessment report period and early appraisal period are reduced and more time allowed for critical evaluation of the consultees’ comments. This does not necessarily mean that the overall timeline should be increased. 9. The procedures for document management need to be carefully considered. This may include keeping some hard copy files of key documents for each Guidance, as well as the electronic files. 10. NICE should improve interaction between Technical Assessment Groups, the NICE technical staff and the Appraisal Committee throughout the assessment and appraisal process. This could include, for example, having the continued involvement of the Technical Assessment Group as technical experts after the initial review of the appraisal consultation document. On the functions of the Appraisal Committee 11. Given that a third Appraisal Committee is being introduced, NICE should take the opportunity to consider how to sustain the high quality performance of the Appraisal Committee. An issue that needs particular consideration is the question of consistency in decisionmaking across the three committees. NICE is already giving this question careful attention. Two possibilities to consider are: 1) having three individual Chairs and one common Vice-Chair; and 2) having the three Committees assess different types of technologies (e.g. diagnostic procedures for one and specific classes of pharmaceuticals for the other two). 12. It should be made clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than on the representation of particular interests. Although that there is a need to ensure that manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. 13. The role of the Chair may need to be further refined. While it may be necessary for the Chair to continue to take an active role in leading the discussion on many items, the possible risks associated with this need to be carefully considered. Asking members to play a greater role in the ongoing review of a technology would allow the Chair to facilitate rather than lead the discussion, although the increased workload that would result for the Committee members would need to be assessed.
6
14. NICE should assess whether the overall sustainability of the process and the functioning of the Appraisal Committee could be improved by the introduction of some type of reimbursement for members’ time. 15. The induction and training process for Appraisal Committee members needs to be further developed to include, where appropriate, enhancing skills in the critical appraisal of clinical evidence and economic evaluation. 16. The Appraisal Committee procedures should be modified to ensure that a clear statement of what the Committee approves is recorded on the same day. On decision-making 17. NICE and the Appraisal Committee should continue their process to develop a system of decision-making that encourages articulation of the grounds for a particular recommendation, including specification of the weight that is placed on clinical evidence, economic evidence and other factors, such as equity and social values. 18. As part of the process of articulating the criteria for decision-making, NICE must resolve the confusion related to the use of a value-formoney threshold. If a threshold is to be used as a basis for recommendations, it should be specified and justified for reasons of transparency. 19. The Appraisal Committee may wish to consider having a legal advisor present during meetings, particularly those involving matters referred to the Committee by the Appeal Board. On the Appeals 20. In view of the considerable number of appeals lodged and their time and resource implications, thought needs to be given to how to reduce the number of appeals and the length of the appeal process. The further development of the appeal process will help enhance the quality and the transparency of the appraisal programme.
7
On budget impact and research 21. Although budget impact is not a consideration in making recommendations on the use of a technology within the NHS, it is important to develop methods for budget impact modelling that would enable NICE to provide more detailed information on the implementation costs to the local authorities. This could be a task for the technical analysts in NICE. The advantages of doing so include not only the provision of useful advice to the Trusts, but also the avoidance of a duplication of effort by the Trusts in making their budget analyses for implementing the new technology. 22. NICE should further develop the Research Required sectionof the Guidance and link it specifically with the review of Guidance review process. This would help to obtain the additional clinical evidence needed to ensure full understanding of the benefit of a technology. The Guidance review process would be an ideal way to stimulate the generation of such evidence. 23. NICE should adopt more flexible timeframes for reviewing existing Guidance not only to ensure that the review answers a specific question but also to assist in managing workload. One approach might be for the set review dates to be dependent on the emergence of new evidence or significant changes in existing evidence. On the technical assessment report 24. The contractual arrangement between the Technical Assessment Groups and the NCCHTA should be revised to recognize NICE as the primary client for the assessment reports. 25. A handbook of standard methods for the assessment reports should be developed, after consultation between NICE staff, the Appraisal Committee and the Technical Assessment Groups. The handbook, which should be regularly updated, should also be used as the basis for training new reviewers and new NICE staff. 26. A detailed template for the reports should be developed, including standardized data presentation and summary material. This would facilitate review of the information by the Appraisal Committee.
8
On input from the consultees 27. On the basis of experience to date, NICE should consider what aspects of patient and professional submissions are most useful and develop standards for the content of these types of submissions. 28. NICE should review the process for assessing comments from consultees and others (including the general public) that are submitted after the appraisal consultation document is drafted. NICE should determine the most appropriate way of responding to these inputs and assess their value in the decision-making process.
9
1. Introduction 1.1 Background to the review In 1999, NICE was established as a Special Health Authority to address problems related to variation in the quality of care and the introduction and use of new technologies and medicines within the NHS in England and Wales. As described by the founding and current Chair of the Board1, the initial three broad functions of the Institute were: • • • appraisal of new and existing health technologies; development of clinical guidelines; promotion of clinical audit and confidential enquiries.
There were perceptions that availability of technologies, as well as quality of care, varied by ‘post-code’. ‘Post code prescribing’ described the situation where patients’ access to treatments was determined by decisions made by local health authorities that have budgetary control over purchasing new technologies for a given geographic area. There were concerns that there should be faster access to new medicines and that the NHS needed to achieve greater value for money. NICE was designed as an independent and national authority to address all of these needs. In 2002, there was a House of Commons Health Select Committee review of the performance of NICE. In its report 2, the Committee identified a number of issues and concerns and recommended an external review of the organization, its functions and processes. The key matters raised by the Committee were that: • • • there was a perception of problems and issues in relation to the consultation process and methodology used in assessments and appraisals of technology; there was a need for an assessment of the scientific validity of processes used particularly regarding the robustness of the approach; there was uncertainty about the practical value of Guidance and its implementation by Health Authorities.
Following the House of Commons Health Select Committee report, in December 2002, NICE requested the WHO Regional Office for Europe to carry out a review of the Technology Appraisal Programme. The particular aspects identified for the WHO review were the methods and scientific robustness of the process. NICE had also started to carry out separate reviews on issues related to the consultation processes and the implementation of Guidance, as well as on methodological issues related to Guidance development. The results of this external review will contribute to the separate reviews on methods and process. From information provided to the Review Team, it appears that these reviews are essentially designed to address two areas: the technology appraisal process and the methodologies used to develop technology appraisal guidance. The outputs are planned as documents on the appraisal process and an update and expansion of the current Guidance for manufacturers. The Review Team was provided with copies of the draft documents from these reviews. 10
The agreed terms of reference for the WHO review are listed in Box 1. WHO identified a team of international experts with skills and experience in scientific methodology relevant to HTA as well as in decision-making and policy (Box 2). Box 1. Terms of reference 1. With reference to established and credible processes and methodologies, review criteria should be identified for the evidence assessment and appraisal components of NICE technology appraisals. These criteria should test: the scientific and methodological rigor of the health technology assessment reports used by the Appraisal Committee; the utility of the other components of the evidence base, including submissions from manufacturers, patient organizations and professional groups; the suitability of the processes used to assemble and interpret evidence and to engage with stakeholders during the appraisal; the consistency of the outcome of the appraisal with conclusions reached on the same topics by reputable scientific groups outside the UK. 2. A minimum of 3 and a maximum of 5 health technology appraisals, published by the Institute between March 2000 and December 2002, should be selected for review. The appraisals selected should reflect, as far as possible, the range of technologies, which the Institute has so far considered and should be topics, which have been considered by reputable scientific groups outside the UK. 3. A report should be prepared for publication by WHO and for consideration by the Board of the Institute, setting out the results of the review, the conclusions reached by the reviewers and their recommendations to the Institute for any changes to the current appraisal process and methodology, which will have the effect of enhancing the scientific credibility or clinical utility of the Guidance.
11
Box 2. External Review Team members Suzanne Hill B.Med PhD FAFPHM Senior Lecturer, Clinical Pharmacology Faculty of Health, University of Newcastle Australia Director, Newcastle Evaluation Group for the Australian Pharmaceutical Benefits Advisory Committee Silvio Garattini MD Istituto di Ricerche Farmacologiche 'Mario Negri' Milan, Italy Member of Committee for Proprietary Medicines Products of European Agency for the Evaluation of Medicines Jos van Loenhout MD PhD Senior Medical Advisor to the Reimbursement Committee of the Dutch Healthcare Insurance Board Diemen, The Netherlands Bernie J O’Brien, PhD Professor, Dept Clinical Epidemiology and Biostatistics McMaster University, Canada Member, Economics Sub-Committee of the Drug Quality and Therapeutics Committee for Ontario Kees de Joncheere Regional Adviser for Health Technology and Pharmaceuticals WHO Regional office for Europe
1.2 The health policy context Before considering the technology appraisal process in detail, it is important to define the context in which NICE is operating and making its Guidance, as the context is particular to England and Wales. The UK National Health Service has provided universal access to health care for British citizens since 1948. Features of the NHS that are pertinent to this review are that policies and standards of care are set centrally (by the Department of Health), and the implementation and delivery of the health services are managed locally and funded by NHS Trusts (hospital and primary care) that employ the staff involved in health care delivery and provide the services. Further details on the NHS can be found at http://www.nhs.uk. The current management approach for resource allocation decisions is decentralised. Budget responsibility lies with the Primary Care Trusts that commission services from general practitioners and Hospital Trusts. Health technologies and health services are selected, provided and paid for by Trusts. Each Trust is free to allocate its funds at its own discretion to best provide for the health care needs of its population within a geographical area. In the UK, there are two notable policy concerns about the quality of health care provided in the NHS. These are that the UK does not spend as much on health care as do comparable countries (see OECD Health Data 2003 for recent comparisons, http://www.oecd.org) and that the services and technologies available vary from one Trust to the next1. Health professionals have partly attributed this variability to the slow uptake of new technologies. 12
These two issues mean that it is the current overall policy direction and current government commitment to increase expenditure on healthcare and to promote the adoption and uptake of health technology that is established as cost-effective. A further aspect of the policy environment that should be noted is the commitment within the health sector to innovation. New technologies are seen to have a value because of their novelty. This policy environment is different from other national environments, where the major need is to limit or control expenditure on health care and to be more selective about the uptake of health technologies, while continuing to guarantee high quality healthcare. As Primary Care Trusts make the funding and purchasing decisions, the issues of the local opportunity cost of new health care interventions are likely to be a primary consideration in the adoption of new technologies and services at the local level. There is, as a result, a tension between the national policy to increase health expenditure and the decentralised process of purchasing. This tension is one of the most frequently mentioned issues in the recently published commentary about NICE3. From an international perspective, it is important that other organizations that may be considering emulating some of the functions of NICE recognize this specific policy environment. NICE is actively establishing international links.
1.3 What is NICE? NICE has been in operation for only four years. Within this short period of time, NICE has already established an enviable national and international reputation for innovation and developments in HTA. NICE is evolving and the following description of the organization, its structure and functions is based on information available in June 2003. NICE has at present four functions: • • • • To produce Guidance in relation to the use of new and established medicines and treatments available within the NHS (technology appraisals). To produce clinical guidelines about the appropriate treatment and care of people with specific diseases and conditions within the NHS. To produce Guidance on whether interventional procedures used for diagnosis or treatment are safe enough and work well enough for routine use. To identify ways of improving the quality of care (confidential enquiries).
NICE describes itself as a virtual organization and has a relatively small staff (approximately 60) based in London. As in other NHS Special Health Authorities, a Board of Executive and Non-Executive Directors manage the Institute. It has a number of committees that are required for administrative functions (the Audit Committee, the Risk Management Committee, the Research and Development Committee, the Remuneration and Terms of Service Committee) as well as the technical expert independent advisory committees that assist in its technical and scientific activities. The Technology 13
Appraisal Committee is the relevant technical advisory committee for this review. The Directions that established NICE also include the requirement for a ‘Partners Council’, membership of which is drawn from organizations (patient, professional and health care industries) that have a special interest in the work of NICE. The in-house team is divided into four functional groups: the Communications Team, the Corporate Team, the Guidelines Team and the Technology Appraisal Team. The Technology Appraisal Team comprises a Director, two Technical Team Leaders each responsible for a group of health technology analysts and information specialists, and two project management teams responsible for overseeing the administrative management of the technology appraisals. The increasing workload of the Institute has led to a rapid expansion of the staff. The outputs from the NICE programme of work over the four years of its existence are a tribute to the commitment and hard work of all involved. The products to date are 62 technology Guidances (covering topics ranging from the extraction of wisdom teeth to treatments for managing diabetes), 14 clinical guidelines, including some inherited from the previous guidelines development programme in the UK, commencement of the interventional procedures work programme, as well as the carrying out of a number of confidential enquiries. In addition, NICE has developed documents that provide descriptions of the processes and methods it uses for each aspect of its work. For this review, the Team specifically considered NICE Guidance (as distinguished from its Guideline programme). Further details about NICE can be found at http://www.nice.org.uk.
14
2. Methods and approach of the review The Review Team visited the NICE offices in London twice: on 5-12 June 2003 and on 17-21 July 2003. As the starting-point for its investigations, the Review Team consulted publicly available information about NICE, including its web site, and four technology appraisals for which Guidance had been issued in the 12 months up to the end of 2002. The technology appraisals were used as illustrative case studies and as a means of understanding the process and evaluating the quality of the assessments. Given that the process of technology assessment and appraisal by NICE is changing rapidly, these technology appraisals clearly do not constitute a representative sample of the 62 appraisals published to date. The technology appraisals selected included one non-drug or medical device technology, one that had been subject to appeal one oncology technology, and one drug technology. This selection ensured that the work of a number of different assessment groups was represented. The four technology appraisals are listed in Table 1. Table 1: Technologies assessed Technology Etanercept and infliximab for rheumatoid arthritis Atypical antipsychotics for schizophrenia Ultrasound device for central venous catherisation Imatinib for chronic myeloid leukaemia Guidance #36 Assessment report authors Birmingham Appeals None Date Guidance issued March 2002
#43 #49 #50
York Sheffield Exeter
Appeal None None
June 2002 Sept. 2002 Oct. 2002
The Review Team interviewed NICE staff, members of the assessment groups, Appraisal Committee members and external experts. Those interviewed (by phone and in person) are listed in Annex 1. All the documents, including stakeholder submissions, that had been considered by the Appraisal Committee in respect of each technology were provided to the Review Team. The Team reviewed these documents not only in the context of the report of the House of Commons Health Select Committee (also made available to the Review Team, including the submissions of the British National Formulary and the Drugs and Therapeutics Bulletin to the Health Select Committee) and the terms of reference for the review, but also in the light of recently published commentaries on NICE3-15. A full list of the documents considered is given in Annex 2. The members of the
15
Review Team signed confidentiality agreements to enable them to review confidential material. The Review Team attended an Appraisal Committee meeting during the first visit. The Review Team used the technologies selected to develop an understanding of the NICE appraisal and guidance development process and studied the four technology appraisals specifically. For each technology, the Review Team considered: • • • • • the quality and rigor of the assessment report the quality and relevance of the other documents provided to the committee views on the usefulness and quality of the reports from those involved in the process of decision-making; views from the authors of the reports on the outcomes of the process in relation to the evidence the final Guidance and the recommendations in the light of the evidence assessed in the evaluation reports.
These discussions and assessments were used as a basis to: • • • identify any aspects of the review process and reports that could be modified to improve the reliability and scientific robustness of the Guidance identify any areas where improvements in methodology could improve the validity of the Guidance make other recommendations about the process.
On completion of the first visit, the Review Team drafted a report, which was provided to NICE during the second visit for correction of any factual errors. The draft report and recommendations were discussed at this time with NICE staff and the Chair of the Board. In undertaking the review and preparing the report, rather than reconfirming the well-established achievements of NICE, the Review Team focussed on areas of controversy or where there is potential for improvement. The report represents the consensus view of the Review Team.
16
3. Findings and recommendations 3.1 Principles that determine the NICE approach to technology assessment. The key principles that underpin the NICE approach to decision-making and Guidance development are: • • • • • use of the best available evidence in decision-making transparency consultation inclusion all key stakeholders responsiveness to change.15
From the stage of considering which topics are to be selected for appraisal to completion of the final Guidance, there is ample opportunity for all stakeholders to provide input into the process: from formal submissions by stakeholders to comments by experts and the general public. The information available on the NICE web site (from the selection of topics for assessment to completion of the final Guidance, including information on the appeals) provides interested parties and the general public with a clear view of the assessment and guidance development process. There are numerous consultative bodies that can feed into the processes, the principles - on the basis of which NICE works - and the ongoing appraisals. “Use of best available evidence” implies the use of data on final clinical outcomes rather than on surrogate endpoints, a preference for data from controlled studies and the transparent and complete presentation of the data. Equity considerations play an important role in the appraisal process, as the health implications for the wider group of patients in the NHS should be considered alongside the effects for those patients directly benefiting from the technology under consideration. NICE takes the following factors into consideration in technology appraisals: • • • • • The broad clinical priorities of the NHS The degree of clinical need of patients with the condition under consideration The broad balance of benefits and costs Guidance already available within the NHS and guidance from the Secretary of State The effective use of available resources.
Within the assessment process - through the TAGs - a systematic and independent evaluation of the relevant available evidence takes place. As NICE has to take decisions on the basis of the “best available” evidence, obviously problems occur where there is limited or incomplete evidence, and where NICE has to make judgements in formulating the Guidance. Currently NICE is refining the scientific and social value judgements that are guiding its 17
decisions. These discussions also benefit from the deliberations of the Citizens Council that attempts to capture views from society. The Review Team noted the following. • In all of these areas, it is clear that NICE is setting a new, international benchmark, for which it can and should be congratulated. In particular, commitment to the consideration of stakeholder inputs in the decisionmaking process and to the provision of documents and information publicly, via the NICE web site, are recognized by the Review Team as setting new standards. There is, therefore, an apparent contradiction in the current policy that allows material that is designated confidential to be used as a basis for decision-making. The majority of the confidential material is submitted by the pharmaceutical industry, although there have been examples of other material, such as that from academic centres, being designated confidential. Such material is removed from assessment reports before their circulation and publication and is not included in any other public document. Allowing confidential material as part of the decision-making process obscures the transparency of decisions and possibly jeopardises the quality of the NICE assessment process. As described by Drummond10, using ‘commercial-in-confidence’ data can inhibit the opportunity for there to be potential checks and balances on an agency’s work. There is also the question of conflict in philosophy and values given the public commitment of NICE to transparency. Although the problem of ‘commercial-in-confidence’ data is present in all European countries and although the Review Team realizes that under the current legal system the submission of ‘commercial-in-confidence’ material is clearly admitted, the commitment of NICE to transparency and public accountability should override the confidentiality issues, and make it possible for decision-making to become even more transparent and open to scrutiny.
•
•
NICE has already taken steps in discussions with the industry to limit the inclusion of confidential information in the assessment process. The Review Team encourages further steps to be taken in this direction.
18
Recommendations 1. NICE should continue to develop operational procedures that are consistent with its core principles of transparency, consultation and inclusiveness with respect to stakeholders’ involvement in evidencegathering and decision-making. The NICE model of partnership in the scientific endeavour of health technology appraisal offers valuable international leadership. 2. NICE should reconcile the inherent contradiction between its principles of transparency and its acceptance of evidence for the decision-making process that a stakeholder deems to be confidential. Although NICE has already started on this issue, further steps should be taken to ensure that all material submitted for consideration can be made available to the public. 3. The principle of transparency requires that NICE codifies and justifies the specific criteria used in decision-making. Difficult but important elements of this task are the articulation of the ethical and social value judgements, and definition of the interaction of these judgements with the appraisal of the scientific evidence used by the Appraisal Committee in reaching its decisions.
3.2 The process of Guidance development 3.2.1 General issues Prior to the visit in June, the Review Team was provided with a scheme of the Guidance development process (Annex 3). As this process has clearly evolved, the Review Team developed a description of the current process of Guidance development (Annex 3) that was used as a basis for discussions and recommendations. The timeframe for the development of NICE Guidance is stated as 62 weeks from the start of the process to the close of a possible appeal and issue of the final Guidance. A process of ‘topic selection’ (see section 3.2.2) determines whether a particular drug or technology will be assessed. If a product is not assessed, it is up to the local Trusts to determine whether it will be paid for. NICE issues each technology Guidance with a planned date for review. Functionally, there are two key components of the system: the assessment component, defined as the collation and synthesis of the clinical and economic evidence to address the policy question, and the appraisal component that is the consideration of the evidence, including stakeholder submissions, by the Appraisal Committee. The task of the Appraisal Committee is to weigh the evidence and judge whether it would be appropriate to make the technology available within the NHS. For sponsored technologies, such as pharmaceuticals and medical devices (that have been the main focus of the NICE’s work to date), there are two possible approaches to the assessment process. One is to request the 19
manufacturer of the technology (the sponsor) to conduct the required collection and analysis of the evidence and make a submission to the health care payer (e.g. the Government). This approach has been used in countries such as Australia and Canada when assessing new pharmaceuticals in respect of value-for-money, with a view to inclusion in government formularies. A second approach would be for the health care payer to commission an academic group to conduct the assessment. NICE has adopted a hybrid of these two approaches in that it commissions an academic centre to carry out an independent review of the technology and also invites submissions from stakeholders, such as the pharmaceutical industry. Furthermore, the academic group contracted to carry out an assessment is invited to comment on the submission of the manufacturer. There are advantages and disadvantages to the hybrid approach to HTA. On the plus side the Appraisal Committee will see two analyses of the evidence, each conducted independently. Some of the Appraisal Committee members felt that the disagreements which inevitably arise in connection with the results and conclusions - particularly those related to cost-effectiveness models could be very instructive in that, before agreement is reached, clarification is required on whether it is the differences in model structure, parameters values or other assumptions that produce the different results. To the credit of NICE and the appraisal process, it is evident that when there is conflicting evidence, the goal of the Appraisal Committee is to better understand why the differences exist and not to dismiss either the technical assessment report or the stakeholder submissions. A natural consequence of this is a further round of analyses, sometimes conducted in-house by NICE technical staff, with the aim of integrating and/or adapting one of the models presented. The main disadvantage of the current hybrid process of HTA at NICE is an unnecessary duplication of effort. Because the manufacturer and the academic group work in apparent isolation, difficulties arising from conflicting views are resolved only late in the appraisal process. A more efficient approach might be to provide a forum for an early, on-going exchange of information and ideas with respect, for example, to issues of relevant comparators, clinical pathways for an economic model and parameter values. In this integrated approach the goal would be to produce one analysis from the Technical Assessment Group, which includes the relevant evidence and the manufacturer’s input. Unresolved disagreements on the key aspects of a model or its parameters could then be identified as candidates for a sensitivity analysis to be presented to the Appraisal Committee. Recommendation 4. For drug and device technologies where a sponsor exists, the current hybrid process of HTA used by NICE may give rise to unnecessary duplication of effort. NICE may wish to consider using different approaches depending on the subject of the appraisal. The Review Team suggests that an effort be made to ensure that the Appraisal Committee is presented with a single set of analyses produced by the Technical Assessment Group that incorporates consultation with and input from the manufacturer(s). 20
3.2.2 Topic selection and start of appraisal process The first stage of the process is the selection of topics for NICE Guidance by the Department of Health. A flow chart outlining the current approach (as of January 2003) to topic selection is illustrated in Annex 4. Ministers select technologies for appraisal based on advice from the Advisory Committee on Topic Selection . The criteria for topic selection are given in Box 3. NICE combines reviews of individual new medicines with reviews of classes of medicines. This is in contrast to most other countries that review and assess individual medicines coming onto the market before making a reimbursement decision. Not every new medicine that is licensed in the UK (and which can therefore be prescribed on the NHS unless put on the negative/’black’ list) is appraised by NICE.
21
Box 3: Criteria for topic selection 1. Would guidance promote the best possible improvement in patient care given available resources? In particular, are one or more of the following satisfied? a. Does the proposed guidance relate to one of the NHS clinical priority areas, or to other government health-related priorities such as reducing health inequalities? b. Does the proposed guidance address a condition, which is associated with significant disability, morbidity or mortality in the population as a whole or in particular subgroups? and/or c. Does the proposed guidance relate to one or more interventions which could significantly improve patients’ or carers’ quality of life and/or reduce avoidable morbidity or avoidable premature mortality, relative to current standard practice, or if used more extensively or more appropriately would do so? d. Does the proposed guidance relate to one or more interventions which if more extensively used would impact significantly on NHS or other societal resources (financial and other)? e. Does the proposed guidance relate to one or more interventions, which could without detriment to patient care be used more selectively, thus freeing up resources for, use elsewhere in the NHS? 2. Will NICE be able to add value by issuing guidance, taking into account the following factors?: a.Is the evidence base sufficient to develop robust guidance across most or all of the interventions to be covered by the proposed guidance? b.Is there evidence and/or reason to believe that there is or will be, inappropriate practice and/or significant variation in clinical practice and/or variation in access to treatment (between geographical areas or social groups) in the absence of guidance? 3. Would the most appropriate form of guidance consist of an appraisal, a clinical guideline, or a combination of the two, taking into account: a. the availability of an existing clinical guideline from NICE or from another authoritative source for the condition in question? b. the degree of urgency for guidance on any specific intervention for the condition in question? c. the possible complexity of the proposed guidance if formulated as an appraisal? 4. For new interventions, does the balance of advantage for patient care lie with appraisal at time of launch or at some specified future date, taking account of the following factors and the attached checklist? a. The possible impact on uptake or equity of access in the absence of guidance at time of launch. b. The likely robustness of the evidence base at time of launch. c. The prospect of relevant additional data becoming available in the period immediately after launch. d. For surgical and related interventions, whether safety and efficacy have already been assessed (or will be assessed in the near future) by the Interventional Procedures Advisory Committee. Source: Http://www.doh.gov.uk/nice/consultation2002, accessed 30 June 2003
As a next step, a subgroup of the Advisory Committee on Topic Selection makes a further assessment of the possible technology topics that meet the selection criteria and prioritises them. The selected topics need the clearance of the Minister before the programme of work can be decided and the topics are posted on the web.The plan was to announce a new ‘wave’ of topics every six months but this has not occurred consistently. Once a wave of technology topics is selected and announced, project management begins. Each topic is seen as a specific project for which a timeframe is scheduled and managed by a project manager within NICE. As a first step in the project management, the National Coordinating Centre for Health Technology Assessment assigns the topics to a review group and NICE schedules a meeting of the Appraisal Committee to review the Appraisal
22
Consultation Document (effectively the first draft of the Guidance. The assessment and appraisal process is planned within the scheduled timeframe. Following the announcement of the topic and the appointment of the review group, the project manager within NICE identifies consultees. All consultees are stakeholders in the technology being assessed but not all stakeholders are invited to become consultees. A consultee enjoys the privileges of being invited to make a formal submission to NICE on the technology in question, receiving appraisal documentation that is not on the NICE web site, and being able to lodge an appeal against the final NICE guidance. A meeting between consultees, the Technical Assessment Group and the technical lead from NICE takes place prior to the commencement of the assessment review, to further discuss the Scope of the project. The “scoping” exercise is crucial to the success of the appraisal in general and specifically to guiding the Technical Assessment Group on the precise questions which need to be addressed. Feedback to the Review Team from some members of the Technical Assessment Groups indicated that the scoping exercise was not always adequate in that the evidence required and questions to be addressed were not clearly specified. From a review of the NICE web site, it appears that there may be up to an 18month time lag between the announcement of a topic and the commencement of the assessment process, depending on the Technical Assessment Group’s schedule of work1. As the topics are publicly listed, this offers an opportunity for stakeholder groups to begin to develop submissions. However, it is often not clear before the Scope for a topic is decided what particular questions are going to be considered. NICE technical staff develops the first draft of the Scope for each topic. The Technical Lead in NICE who develops the Scope has continuing responsibility for that technology or project and, as a result, significant ownership of that particular Guidance. The Review Team identified the following issues: • Given the time gaps at the beginning of the assessment process (i.e. between announcement of the topic and commencement of the review), the late deadline for stakeholder submissions puts unreasonable time pressures on the Technical Assessment Groups at the end of the assessment process. If the scope and protocol could be defined effectively in a relatively early phase, there is no reason to have stakeholder submissions delayed to ten weeks before the end of the assessment period. The current system of relatively late stakeholder submissions is also prone to strategic ‘gaming’ by virtue of a stakeholder being able to produce new evidence late in the process. An alternative approach would be to adopt a legal analogy of ‘disclosure of evidence’ at the stage of finalising the Scope for a technology appraisal. The purpose of this would be to declare currently
•
For example, statins are currently [as of 1 July 2003] listed on the website as a new topic, with the Scope to be finalised in January 2004 and the Guidance due to be issued in May 2005. Similarly, donepezil is currently listed as a technology to be assessed, with the Appraisal Committee meeting listed as due in October 2004.
1
23
available information and importantly, to identify any anticipated information that is likely to become available later in the assessment and appraisal process. This would reduce the opportunity for strategically ‘gaming’ the process by late provision of data. Given the number of jurisdictions that now require compilation of clinical and economic data to support reimbursement decisions, pharmaceutical manufacturers in particular should be well placed to meet such a disclosure request. Recommendations 5. The consultees’ meeting should become a formal ‘Preliminary Exchange of Evidence’, at which time all stakeholders should be asked to provide details of what they propose to submit. Only in extraordinary circumstances should the Technical Assessment Group, accept to include as part of the review information over and above that declared for submission by the stakeholders at this meeting. 6. More attention needs to be paid to the important task of setting the Scope of the appraisal. The general goal is to give timely and comprehensive guidance to the Technical Assessment Groups on the question(s) to be addressed by the Appraisal Committee, thereby reducing the risk of mismatch between the assessment report and the needs of the Appraisal Committee. 7. Stakeholder submissions should be required to be lodged as soon as possible after the start of the assessment process. 3.2.3 Assessment and appraisal process The key document in the assessment process is the technical assessment report prepared by the Technology Assessment Groups based in the academic centres. A Technical assessment report is a systematic review of clinical and economic evidence relevant to the Scope for each topic, and it may also include an economic evaluation prepared by the Technical Assessment Group, as well as a critical appraisal of stakeholder submissions. The Technical Assessment Group starts the assessment process by defining the protocol for the assessment report. The report, which is to be completed within 30 weeks, includes: a systematic review of the clinical evidence; an assessment of the economic evidence with or without the development of an economic model and, in most cases, an assessment of the stakeholder submissions. Further aspects of the technical assessment report are discussed in section 3.3.1. Once the technical assessment report is completed, two versions are provided to NICE (see below for further details). The process then becomes one of appraisal and consultation. The key steps are listed in Box 4.
24
Box 4. Summary of appraisal process The technical assessment report is circulated for comment to consultees. The NICE Technical Lead prepares an overview document for the Appraisal Committee; this document summarises the evidence, the comments of the consultees and the key issues for the Appraisal Committee. The technical assessment report is used as the basis for drafting the body of the Appraisal Consultation Document (by the NICE Technical Lead for each technology, in consultation with the Appraisal Committee Chair). The Appraisal Committee meets to consider the evidence and then finalises the Appraisal Consultation Document. The appraisal consultation document is circulated and published for comments. The appraisal consultation document is redrafted by the NICE Technical Lead in consultation with the Appraisal Committee Chair to take any comments into account. It is then considered by the Appraisal Committee as a Final Appraisal Determination. The final appraisal determination is published for a 15-day period, during which consultees can appeal. Any appeals are heard and resolved, including (where necessary) referral back to the Appraisal Committee. The final appraisal determination becomes draft Guidance and is submitted for approval to the Guidance Executive2 within NICE.
The time from completion of the technical assessment report to approval of the Guidance is approximately 5 months. The Review Team identified the following matters in relation to the assessment and appraisal processes: • NICE has been very effective in taking on the challenges of developing a system that meets multiple needs: timely production of Guidance that addresses important questions related to improving clinical care with the NHS. In doing so, points of stress have inevitably been introduced into the system, the most obvious of which is the timeframe. For the last part of the appraisal and consultation process in particular (from completion of the technical assessment report issuance of the Guidance) the timeframe is structured to allow incorporation of new information without necessarily allowing time for it to be reviewed and critically appraised. Relatively little emphasis seemed to be placed by the Technical Assessment Groups on protocol development. It was not clear what strategies they had adopted to ensure that there was appropriate clinical input into the development of the protocol, although the most recent version of the Annex to the contracts requires that theTechnical Assessment Groups ensure adequate clinical input. Difficulties at the protocol development stage may be contributing to problems related to the appropriateness and usefulness of assessment reports. Interaction between the Technical Assessment Group and the Appraisal Committee appears to be somewhat limited. Although the authors of the Technical assessment report attend the first Appraisal
• •
•
•
The Guidance Executive is the group of staff to whom the Board of NICE has delegated the authority to approve and issue Guidance on behalf of the Institute.
2
25
Committee meeting, they are not expected to present their reports, nor are they expected to attend the subsequent meetings. Furthermore, under the current contractual structures they are not necessarily expected to review additional data that is submitted unless this is stated explicitly in the contract. • The effect of this delineation of responsibility is that the Technical Lead for each technology within NICE becomes the main repository of the technical aspects of the evidence. They may also have to take on the task of reviewing and assessing any new evidence that is submitted. It is not clear to the Review Team whether this is feasible in terms of the resources available, notwithstanding the newly established Decision Support Unit whose role it may be to assist in this process.3 Effective document management is a challenge for NICE that has received special attention over the last 12 months. NICE has adopted a ‘paperless’ office approach, and thus all documents are kept electronically. Significant effort is being put into making sure this meets the administrative demands of the Guidance development process. One of the challenges of the rapidly evolving system in this particular area is to ensure that not only current material is managed appropriately, but also that archiving of previous work is optimal. This is particularly important in relation to possible legal challenges that might arise.
•
Recommendations 8. The timeframe for the assessment and appraisal process should be reviewed so that the current time pressures at the end of the assessment report period and early appraisal period are reduced, and more time is allowed for critical evaluation of the consultees’ comments. This does not necessarily mean that the overall timeline should be increased. 9. The procedures for document management need to be carefully considered. This may include keeping some hard copy files of key documents for each Guidance, as well as the electronic files. 10. NICE should improve interaction between the Technical Assessment Groups, the NICE technical staff and the Appraisal Committee throughout the assessment and appraisal process. This could include, for example, the continued involvement of the Technical Assessment Group as technical experts after the initial review of the appraisal consultation document.
The Decision Support Unit is a newly developed academic consortia, contracted directly to NICE. Its role is to provide additional technical support and advice to the Appraisal Committee and technical analysts, particularly in relation to the appraisal of economic models.
3
26
3.2.4 The functions of the Appraisal Committee The Appraisal Committee plays a key role in the process of Guidance development and advises the Board of NICE on: • the use, within the NHS, of any new or established health technology in relation to its clinical- and cost-effectiveness, taking into account the interests of the NHS as a whole; • other matters, on which the Board may, from time to time, seek guidance. The Appraisal Committee currently comprises 51 members. This number will increase to approximately 75 in January 2004. It includes lay representatives, NHS clinicians and managers, academic experts and industry representatives (as described in the most recent draft of the revised Guide to the Technology Appraisal Process). The Board appoints members for three-year terms, normally following a process of public advertisement. The membership currently includes 2 lay representatives and 2 industry representatives who are physicians employed by pharmaceutical companies. In the process of expanding the Appraisal Committee, it is planned to include representatives of other parts of the health care industry. The responsibility of members is described in a recent document used in the recruitment process for new members: “To consider and interpret evidence on the clinical and costeffectiveness of health technologies (pharmaceuticals, medical devices, surgical procedures, diagnostic procedures and health promotion interventions) prepared by the Institute. They take collective responsibility for the formulation of recommendations to the Institute on the use of technology in the National Health Service in England and Wales.” In order to complete its workload, the original Appraisal Committee has been divided into two to form Appraisal Committees A and B. Currently both have the same Chair who is now almost full-time employed in this function. It has been decided to recruit new Appraisal Committee members and expand to a third committee, commencing in January 2004. One person will chair two of the three committees; one of the current Vice-Chairs will chair the third Committee. The Chairs will have cross-membership of all three Committees. Members are reimbursed for travel, accommodation and out-of-pocket expenses but receive no fees or payment for attending meetings of the Committees. In addition, members may apply for coverage of the costs of employing a locum (e.g. in the case of general practitioners), and the costs for child care. The Committee currently meets every two weeks. At each Committee meeting, the agenda usually comprises a mixture of first reviews of a technology (appraisal consultation documents), final reviews (final appraisal determinations) and matters arising from Appeals that have been referred to the Committee by the Appeal Panel of the Board. In order to complete each 27
agenda, a tightly structured ‘ideal’ time allocation for discussion has been devised. For review of an appraisal consultation document, two members of the Committee are nominated to lead the discussions (Appraisal Team Leads), one from the clinical/disease perspective and one from the technical/evidence perspective. Each Appraisal Team Lead is expected to prepare a presentation summarizing the key issues and evidence to be considered by the Committee. For review of a final appraisal determination, however, Committee members are not expected to lead the discussion; this is done by the Chair. When considering an appraisal consultation document, the Committee hears evidence from stakeholders and technical and clinical experts. These include professional and patient representatives. The consultees for each technology nominate experts. The approach taken for this input is a formal hearing of evidence: experts make presentations in connection with which Committee members may ask questions; the experts then leave the meeting to allow the Committee to consider the matter in camera. During an appraisal consultation document review, the Technical Assessment Group that prepared the Technical assessment report is also available to answer questions, but the Technical Assessment Group does not present its view of the data. Decisions are made primarily on the basis of a consensus of views of the Committee members, although matters may be put to the vote if required. In order to complete its workload, the Appraisal Committee has adopted a process of decision-making that includes post-meeting electronic review and approval of documents. The Review Team noted the following matters in relation to Appraisal Committee processes. • NICE has expended consistent effort in developing a carefully constructed Committee to meet the requirements of technology appraisal and Guidance development. The work undertaken by the Appraisal Committee is substantial and the success is attributable to the commitment and enthusiasm of all Appraisal Committee members and the NICE appraisal team. In relation to membership of the Committee, the principles governing selection of the members representing the industry would seem to be ambiguous. Given the roles and functions of the Appraisal Committee and NICE, the presence of NHS clinicians and managers, academic experts and lay representatives meets the needs of the Appraisal Committee for forming judgements, making decisions and developing Guidance. Views have been expressed that its members who are employees of the pharmaceutical industry provide significant expertise to the Appraisal Committee. The Review Team considers that this input could be more appropriately obtained through the stakeholder submission process. Furthermore, the nature of the evidence considered by the Appraisal Committee at present includes commercial-in-confidence data. Apart from potential problems arising from commercial and personal conflicts of interest, functional problems 28
•
are bound to arise in connection with reviewing the confidential data that is required during the process of developing guidance. • Over the last four years, during the development of the Appraisal Committee, the role of the Chair has been pivotal. The Chair has had the usual task of facilitating the discussions and completing the agenda. In addition, the Chair has carried the major responsibility of being thoroughly informed about the technical issues related to each appraisal in order to guide the Committee in its deliberations. Although Committee members act as team leads in the initial discussions that result in the appraisal consultation document, the Chair has generally led the discussions at subsequent stages of the Guidance development for each technology. While this leadership has been of great value during the establishment and development phase of the Appraisal Committee, the Review Team has some concerns about its long-term sustainability. These concerns include the fact that the corporate memory for decision-making rests primarily with the Chair so that the process is highly dependent on the continuing involvement of a single individual. An alternative approach of asking other Committee members to take on a more intensive role would have obvious time and workload implications for the members that would need to be carefully considered. Variable attendance by members and the professional composition of the quorum may be an issue (based on a review of attendance in the Minutes of the Committee). Workload outside the Committee was mentioned as a major contributing factor to this matter.
•
•
•
The Review Team was made aware of a proposal to open the Committee meetings to the public with the aim of improving transparency. The Review Team is of the opinion that such a step would be linked to significant risks and could potentially lead to inefficient and inappropriate decisions. To ensure adequately critical review and discussion of the evidence, the Committee needs to retain the option to conduct in private at least part of the discussions leading to the formation of judgements and decisions about each technology appraisal. Recommendations 11. Given that a third Appraisal Committee is being introduced, NICE should take the opportunity to consider how to sustain the high quality performance of the Appraisal Committee. An issue that needs particular consideration is the question of consistency in decisionmaking across the three committees. NICE is already giving this question careful attention. Two possibilities to consider are: 1) having three individual Chairs and one common Vice-Chair; and 2) having the three Committees assess different types of technologies (e.g. diagnostic procedures for one and specific classes of pharmaceuticals for the other two). 29
12. It should be made clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than on the representation of particular interests. Although there is a need to ensure that manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. 13. The role of the Chair may need to be further refined. While it may be necessary for the Chair to continue to take an active role in leading the discussion on many items, the possible risks associated with this need to be carefully considered. Asking members to play a greater role in the ongoing review of a technology would allow the Chair to facilitate rather than lead the discussion, although the increased workload that would result for the Committee members would need to be assessed. 14. NICE should assess whether the overall sustainability of the process and the functioning of the Appraisal Committee could be improved by the introduction of some type of reimbursement for members’ time. 4 15. The induction and training process for Appraisal Committee members needs to be further developed to include, where appropriate, enhancing skills in the critical appraisal of clinical evidence and economic evaluation. 16. The Appraisal Committee procedures should be modified to ensure that a clear statement of what the Committee approves is recorded on the same day. 3.2.5 The Appraisal Committee approach to decision-making In order to assess the way in which NICE decisions are made, the Review Team considered the various documents available for each of the technologies, and identified the key recommendations in each. These recommendations are summarized in Annex 5. In addition, the Review Team compared the recommendations in the final Guidance with recommendations made about the same technologies by other authorities, including the Pharmaceutical Benefits Advisory Committee, Australia, the Canadian Coordinating Office of Health Technology Assessment, the Netherlands Reimbursement Authority and the Italian Health Authority (Annex 6). Not all decisions could be compared to all authorities. The Review Team noted the following issues with regard to the decisionmaking process of NICE and the Appraisal Committee. • NICE is in the process of developing explicit criteria for use in decisionmaking that are consistent with the broad principles that were defined with the establishment of NICE. This laudable objective represents a significant development internationally and may lead to important advances in knowledge within the field of HTA, particularly as the There is precedent for this already; both the Chair and Vice-Chair currently have their time costs reimbursed to their institutions (at 90% and 50% respectively). 4
30
criteria include consideration of methods to define community values (via the Citizens’ Council) as well as methodological research into equity-efficiency trade-offs. • To date the criteria have not been explicit. Consideration has been given to the clinical and economic evidence and it appears that Committee members also pay significant attention to patient and professional group opinions. It is not yet clear how heavily additional considerations, such as equity of access to technologies, the rule of rescue, or total budget impact weigh with the Committee. One particular criterion about which there is confusion is the use of a threshold or benchmark for value for money. On the one hand, it appears that there may be a threshold (£30000/QALY) being used informally when considerating technologies and this figure was mentioned by almost everyone the Review Team consulted. On the other hand, there have been statements by NICE that the Committee is not using a threshold, and some of the Committee members publicly take exception to the idea that a threshold is an appropriate approach to decision-making. The sample of technology appraisals reviewed suggest that there may be a shift from restrictive or negative recommendations about the use of the technology in the appraisal consultation document to less conservative recommendations in the final Guidance. Given the current structure of the assessment process, where an objective independent case is the first prepared by the Technical Assessment Group, this shift, if confirmed as a consistent finding in other appraisals, would be a logical consequence of the current sequence of input of evidence into the decision-making process. Benchmarking the recommendations of NICE with those of other countries is not straightforward, owing to differences in reimbursement systems from country to country. For the three appraisals that the Team reviewed and compared, the NICE recommendations seemed less conservative than those made by other authorities but this obviously cannot be generalised. Currently, the Committee does not appear to seek legal advice in the course of the meetings. In other comparable decision-making bodies, for example, it is customary to have a lawyer present to provide advice on the interpretation of decisions and legal structures..
•
•
•
•
Recommendations 17. NICE and the Appraisal Committee should continue their process to develop a decision-making system that encourages articulation of the grounds for a particular recommendation, including specification of the weight that is placed on clinical evidence, economic evidence and other factors, such as equity and social values.
31
18. As part of the process of articulating the criteria for decision-making, NICE must resolve the confusion related to the use of a value-formoney threshold. If a threshold is to be used as the basis for recommendations, it needs to be specified and justified for reasons of transparency. 19. The Appraisal Committee may wish to consider having a legal advisor present during meetings, particularly those involving matters referred to the Committee by the Appeal Board. 3.2.6 Appeals The process for appealing a final appraisal determination is well established. There is a 15-day period after the publication of the final appraisal determination in which consultees may appeal. Only consultees can appeal. Appeals are heard by the Appeal Board, which is chaired by the Chair of the Board of NICE. There are three grounds on which a final appraisal determination may be appealed: • • • that the Institute has failed to act fairly and in accordance with its published procedures, that the final appraisal determination is perverse in light of the evidence submitted, that the Institute has exceeded its powers.
If an appeal is upheld, it may be referred back to the Committee for review of the Guidance. From documentation from the Appeal Board and from the NICE web site, it appears that since the start of the Appeal programme until March 2003, NICE has received 26 appeals, 23 of which were considered. This means that 38% of the FADs have been appealed. 20 of the 23 appeals were lodged by the industry. Some appeals have more than one appellant. The appeal process can add significantly to the time taken to issue final Guidance, which can lead to problems with the availability of a technology. The Review Team noted the following matters in relation to the appeals. • The Appraisal Committee, when agreeing on the final appraisal determination, clearly addresses the possibility of an appeal being lodged. Concerns had been raised by the Health Select Committee about the independence of the Appeal process. However, in many jurisdictions, the first stage of any appeal process is the submission of the appeal to the organization that made the decision. The current structure for appeals used by NICE is an example of this model. Adding the possibility of a second stage of appeal (prior to formal judicial review) would carry risks and benefits. The main risk relates to resource and
•
32
time implications of an increase in the number of appeals and the length of the appeal process. Recommendation 20. In view of the considerable number of appeals lodged and their time and resource implications, thought needs to be given to how to reduce the number of appeals and the length of the appeal process. The further development of the appeal process would help to enhance the quality and the transparency of the appraisal programme. 3.2.7 Budget impact and research recommendations The final Guidance for any technology contains sections on budget impact, recommendations for auditing compliance with the Guidance and planned dates for the review of the Guidance. Budget impact is not a decision criterion in the development of NICE Guidance. The estimates of budget impact are initially provided from the technical assessment report, and may be refined during the appraisal process. The recommendations for audit are written by a clinical audit specialist unit, after the literature on audit of the specific topic has been reviewed and after consultation with appropriate health professionals. The final Guidance also contains recommendations for further research. These recommendations appear to be developed by the Appraisal Committee during the review of the appraisal consultation document and the final appraisal determination, with significant input from the Committee Chair and the NICE Technical Leads. Review dates for each Guidance are currently standardized as either three years after issue, or one year in the case of a rapidly evolving technology. The Review Team noted the following matters in relation to these sections of Guidance. • Much of the recently published criticism of NICE Guidance relates to inadequate consideration of budget impact at local level, particularly in regard to the local opportunity cost of implementing new Guidance. Budget impact sections of Guidance tend to be fairly limited, often providing only an estimate of the national cost of a technology. The Technical Assessment Groups provide an estimate of budget impact as part of the Technical assessment report. However, these estimates are of national budget impact rather than local budget impact, which is the information needed by Trust managers in connection with the implementation of Guidance. The Research Required section can assist in generating new information on the effectiveness of the medicines under appraisal, as well as in targeting the medicine to those patients most likely to benefit. 33
•
•
•
The first rounds of review to update published Guidance are about to commence. From discussions with NICE staff, it appears that NICE is already developing a process for handling the workload connected to reviewing and updating Guidance, by determining the types of review needed and tailoring them to the availability of new evidence.
Recommendations 21. Although budget impact is not a consideration in making recommendations, it is important to develop methods for budget impact modelling that would enable NICE to provide more detailed information on the implementation costs to the local authorities. This could be a task for the technical analysts in NICE. The advantages of doing so include not only the provision of useful advice to the Trusts but also the avoidance of duplication of effort by the Trusts in making their budget analyses for implementing the new technology. 22. NICE should further develop the Research Required section of the Guidance and link it specifically with the Guidance review process. This would help to obtain the additional clinical evidence needed to ensure full understanding of the benefit of a technology. The Guidance review process would be an ideal way to stimulate the generation of such evidence. 23. NICE should adopt more flexible timeframes for reviewing existing Guidance not only to ensure that the review answers a specific question, but also to assist in magaging workload. One approach might be for the set review dates to be dependent dependent on the emergence of new evidence or significant changes in existing evidence.
3.3 The quality of the science of the assessment and appraisal process 3.3.1 The technical assessment reports The technical assessment reports are prepared by six independent Technical Assessment Groups contracted by the National Coordinating Centre on Health Technology Assessment from academic centres. As noted above, the assessment reports submitted to NICE comprise a systematic review of the clinical evidence and a review of the economic evidence about the technology. The assessment reports are eventually published as monographs in the NCCHTA monograph series. The Technical Assessment Groups are required to provide two versions of the technical assessment report to NICE: an ‘unstripped’ version that contains all the evidence and is the basis for the Appraisal Committee’s consideration, and a ‘stripped’ version from which all confidential data are removed. The ‘stripped’ version is circulated to consultees for comments prior to the
34
Committee’s considering the appraisal consultation document and is also put on the NICE web site for general comments as soon as it is finalised. The Review Team considered the four selected technical assessment reports in detail and, since the system is in rapid evolution, also briefly considered other,, more recent, technical assessment reports available on the NICE web site. The Review Team noted the following issues. • NICE, through its process of working with leading academic centres, is in the process of developing standards for HTA that are leading to methodological innovations and developments that are likely to have a major influence on how HTA is carried out internationally. The pace of development is extremely rapid. Although the four TARs reviewed by the Team were published within the 12 months prior to the review, it is clear that these cannot be seen as a representative sample since the work was carried out in 2001 and 2002, and major changes occurred in the approach taken to assessment reports subsequent to the publication of the four selected TARs. The Appraisal Committee members generally recognize that the TARs are of high quality and provide the essential information required by the Committee. Given the rapidly changing system, it is not surprising that the sample of reports reviewed by the Team varied in method and approach. For example, not all of the reports contained a detailed critique of the stakeholder submissions; the systematic reviews in the reports were carried out in different ways; methods used for data analysis varied leading to differences in the way results were presented. NICE (and NCCHTA) are currently developing standard templates and specifications to reduce the amount of variability where appropriate, to ensure that the reports fit the purposes of the Appraisal Committee. This will include more detailed specifications of the types of evaluations and analyses to be included in a technical assessment report, as well as specifications for the presentation of the information to the Appraisal Committee. There have been some difficulties in ensuring that all academic centres have the appropriate combination of expertise to provide reports that fully integrate the clinical and economic evidence in a way that meets the needs of the Appraisal Committee. Recent modifications to the contracts with the academic centres have been instituted to address this issue. In some instances, the technical assessment reports have not necessarily addressed policy questions of relevance to NICE. As noted in Section 3.2.2, aspects of the assessment process that may have contributed to this problem were uncertainties in the development of the Scope for the technology and difficulties in communication between the academic centre and NICE because of the contractual arrangements between academic centres and NCCHTA.
•
•
•
•
•
35
•
Although NICE and the Appraisal Committee attach importance to the submissions from stakeholders, the Technical Assessment Groups were inconsistent in their approach to the inclusion of stakeholders’ data in the TARs. It is clear to the Review Team that the contractual arrangements contributed to this difficulty. There is also the question of the need for academic groups to be able to use the NICE technology assessment process to generate independent academic products. A more appropriate approach might be for academic centres to regard NICE work as contract research that allows them to carry out other projects through the expertise they acquire in the process. As mentioned in Section 3.2.2, the quality of reports may be compromised by the late arrival of stakeholder submissions, particularly manufacturers’ submissions. The preparation of stripped versions of the technical assessment reports increases workload unnecessarily at a critical time in the process and may lead to errors. The requirement for stripped reports would be removed if the option to submit commercial-in-confidence data were discontinued.
•
•
In addition to the matters noted above, the Review Team identified a number of possible reasons for these difficulties. The first issue relates to the structure of the current contractual arrangement. The contract is between the Technical Assessment Group and the NCCHTA rather than directly with NICE. The reasons for this arrangement are historical. The NCCHTA structure was well-established prior to NICE and was seen as the appropriate body to undertake the work needed by NICE in its Guidance programme. However, there are significant problems with this arrangement. The right of quality control rests with NCCHTA; it is not clear whether the specified personnel included senior staff as well as the research fellows who often do the majority of work on the reviews (based on the authorship and responsibilities described in each technical assessment report); the contract fee per assessment did not (until a recent contract review) cover the full cost of preparing an assessment; there are still dual outputs required – the technical assessment report and a HTA monograph, which serve significantly different purposes. There is a need for this three-way arrangement to be clarified so that the primary client, NICE, is defined and the needs of this client recognized in the terms of the contract. The second issue relates to the siting of the assessment process in academic centres. This arrangement is used in many countries and may be the best way to obtain an independent report. However, it leads to the need for the academics involved to ensure that their work on assessments is recognized for academic purposes. This recognition includes the publication of peerreviewed articles. There is obviously tension between the need to produce TARs that contain methodological advances that are independently publishable and, at the same time, sufficiently robust to meet the needs of the Appraisal Committee. This tension also compromises the transparency of the process. For reasons of intellectual property, there have been problems in making available full copies of the economic models developed by the assessment groups. 36
Thirdly, although the systems for communication and interaction among the various groups involved in the assessment process are being revised and improved, there have been some difficulties in the interaction between the NICE staff responsible for the project (i.e. an individual Guidance), the Technical Assessment Group, the Appraisal Committee and the external experts. Recommendations 24. The contractual arrangement between the Technical Assessment Groups and the NCCHTA should be revised to recognize NICE as the primary client for the assessment reports. 25. A handbook of standard methods for the assessment reports should be developed, after consultation between NICE staff, the Appraisal Committee and the Technical Assessment Groups. This handbook, which should be regularly updated, should also be used as the basis for training new reviewers and new NICE staff. 26. A detailed template for the reports should be developed, including standardized data presentation and summary material. This would facilitate review of the information by the Appraisal Committee. 3.3.2 The consultees submissions and comments There are three groups of submissions that feed into the appraisal consultation document: submissions from manufacturers, submissions from patient groups and submissions from professional groups. Consultees’ submissions can take a variety of forms, and can be submitted by consultees designated for a given technology. The submissions from manufacturers vary, depending on the technology. In the case of pharmaceuticals, although there is technically a 50-page limit on the submission, large volumes of clinical trial data and economic models are often provided. In the case of devices and other medical technologies, the submissions are usually less complex. Submissions from health professionals range from complete literature reviews to short comments on the proposed technologies. Similarly, patients’ submissions also seem to vary; from letters that are simple testimonials, to complex survey data describing the impact of a particular disease on quality of life. Consultees and the general public also have the right to provide additional input after the appraisal consultation document is published. This input can be substantial and seems to vary from handwritten notes to re-analyses of economic evaluations. In the current system, the post-appraisal consultation document comments are dealt with in the short time period between the publication of the document and the Appraisal Committee’s consideration of the final appraisal determination. In general, this time period is approximately eight weeks, although NICE has the option to extend the time if critical new 37
information is submitted that needs detailed evaluation. If they do so, however, it may mean that the timeframe for the overall Appraisal is substantially prolonged. The team noted the following matters. • It is clear that NICE and its Appraisal Committee have an strong commitment to ensuring that their process for obtaining input from stakeholders into the Guidance development process is inclusive and as flexible as possible. In particular, the team noted the weight and value given to patient group submissions, including the financial support available to these groups to assist them to present their view to the Appraisal Committee. The consultation process is important, but there does not seem to be a systematic approach to responding to comments received between the publication of the appraisal consultation document and the final appraisal determination. In general, it seems that the comments are tabulated and provided to the Committee without interpretation. Lack of a systematic response system could lead to the consultation process being perceived as not receiving the attention it deserves, which is clearly not the intention of NICE. The potential provision of substantial evidence after the appraisal consultation document is published places additional pressures on the system, particularly on the technical staff of NICE and the Chair of the Committee, who have a major role in ensuring that the additional evidence is adequately presented to and considered by the Appraisal Committee as a whole.
•
•
Recommendations 27. On the basis of the experience to date, NICE should consider what aspects of the patient and professional submissions are most useful, and develop standards for the content of these types of submissions. 28. NICE should review the process for assessing comments from consultees and others (including the general public) that are submitted after the appraisal consultation document is drafted. NICE should determine the most appropriate way of responding to these inputs and assess their value in the decision-making process.
38
4. Conclusions The conclusion of the Review Team is that, in only four years, NICE has developed a well-deserved reputation for innovation and methodological development that represent an important model for technology appraisals internationally. It is important to emphasize that NICE functions in an environment where substantial changes are taking place in the way in which the NHS operates, and where additional resources are being invested in the health services. This facilitates the achievement of the NICE objectives of improving the quality of care throughout the health service and the uptake of new, cost-effective medicines and technologies. Achievements that are particularly valuable include: the transparency surrounding the process of technology assessment; the intensive participation of different stakeholders and the inclusiveness of the approaches taken; the commitment to using the best available evidence for decision-making; the commitment of the technical and management staff; and the dedication of the Appraisal Team and the Appraisal Committee members. All of these form the cornerstones of an organization that continues to invest in quality development. The major task that the Review Team was asked to carry out was to assess the processes and methodologies used in the technology appraisal with regard to their scientific rigor and credibility. The Review Team noted that, with respect to this task, NICE has embarked on a series of internal reviews of processes and methodologies and it is likely that a number of the issues discussed in this report are also being considered in these internal reviews. Overall, the Review Team was impressed by the commitment to the use of rigorous methodology throughout the process, starting with the use of academic centres of excellence for providing independent appraisals that are supplemented by additional evidence that is handled by the technical teams in NICE. The published technology appraisals are already being used as international benchmarks, which is an obvious recognition of their credibility. In order to consolidate the operation of the technology appraisal programme the Review Team identified a number of issues for further development and improvement. A number of key recommendations are worthy of specific mention here, namely: • To reconcile the inherent contradiction between the Institute’s key principle of transparency and the acceptance of material that is designated as confidential. The Review Team welcomes the steps that NICE has already taken to date, and encourages NICE to use its leverage to advance transparency in this area. This would be of benefit internationally. To revise the contractual arrangements for the development and production of assessment reports so that they fully meet the needs of NICE and its Appraisal Committee..
•
39
•
To consider ways to reduce unnecessary duplication of effort in the assessment phase. Efforts should be made to ensure that the Appraisal Committee is presented with a single set of analyses, produced by the Technical Assessment Group, that incorporates consultation with and input from the manufacturer(s). To ensure the collection of relevant information from all stakeholders as early as possible in the assessment process.. To increase the exchange of information and interaction – at all stages of the process - between the Technical Assessment Group, the Appraisal Committee and the NICE Appraisal Team. To developing a handbook on the preparation of assessment reports. This could be a product of the current review of methods and process. To make clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than on representation of particular interests. Although there is a need to ensure that the manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. To continue to work on the difficult but important task of developing clear criteria in order to obtain optimal interaction between the ethical and social values and the scientific evidence, for use by the Appraisal Committee and NICE in decision-making.
• •
• •
•
In providing these recommendations, the Review Team notes that there are some broader issues that NICE (and other organizations seeking to carry out a similar task) may wish to consider. Transparency of process and decisionmaking comes with a price, and the price includes time, resources and perhaps challenges to the way decisions can be made. The trade-off between transparency with its credibility, and efficiency in decision-making is a challenge that all authorities such as NICE will need to continue to consider carefully. In addition, although the Review Team is aware that budget impact is not part of the remit of NICE in its Guidance development, fundamentally, the questions of local opportunity cost are integrally related to decision-making and help to determine the value-for-money benchmarks that a decisionmaking body considers. The main outcome that will be the measure of the ultimate success of an organization such as NICE will be the implementation of the Guidance and its impact on the heath outcomes of the community it is seeking to serve. These recommendations are to be seen as an effort to further enhance the operations of NICE and to assist organizations with similar responsibilities in other countries to deal with their difficulties and meet their expectations..
40
References 1. Rawlins M. In pursuit of quality: the National Institute for Clinical Excellence. Lancet; 353:1079-1082 (1999). 2. House of Commons Health Committee Review of the National Institute of Clinical Excellence (NICE). London, June 2002. 3. Davies E, Littlejohns P. Views of Directors of Public Health about NICE Appraisal Guidance: results of a postal survey. National Institute for Clinical Excellence. J Public Health Med; 24:319-325 (2002). 4. Birch S, Gafni A. On being NICE in the UK: guidelines for technology appraisal for the NHS in England and Wales. Health Econ; 11:185-191 (2002). 5. Cookson R, McDaid D, Maynard A. Wrong SIGN, NICE mess: is national guidance distorting allocation of resources? BMJ,:323; 743-5 (2001). 6. Poole-Wilson PA. NICE: guessing is not guidance. Heart, 87:510512 (2002). 7. Freemantle N, Bloor K, Eastaugh Pharmacoeconomics; 20:389-391 (2002). J. A fair innings for NICE?
8. Raftery J. NICE: faster access to modern treatments? Analysis of guidance on health technologies. BMJ, 323:1300-3 (2001). 9. Madhok R, Taylor A, Soltani H, Ireland K. NICE and LICE: lessons from a health district. National Institute for Clinical Excellence. Local institutes of clinical excellence. J Public Health Med, 24:2-5 (2002). 10. Drummond M. Should commercial-in-confidence data be used by decision makers when making assessment of cost-effectiveness? Applied Health Economics and Health Policy, 2:3-4 (2002). 11. Dowie J, Claxton K, Sculpher M, Drummond M. NICE (and the NHS) – quo vadis? In: Health Care Priority Setting. Implications for health inequalities. Proceedings of a meeting of the Health Equity Network. Ed: Adam Oliver. The Nuffield Trust. UK. 2003. 12. Gafni A, Birch S. NICE methodological guidelines and decision making in the National Health Service in England and Wales. Pharmacoeconomics; 21:149-157 (2003). 13. Another NICE mess. Editorial. Lancet Oncology; 3:385 (2002). 14. Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute for Clinical Excellence. Lancet; 360:711-15 (2002). 15. Littlejohns P, Barnett D, Longson C. The cancer technology appraisal programme of the UK’s National Institute for Clinical Excellence. Lancet Oncology; 4: 242-50 (2003).
41
Annex 1 Persons interviewed (in person or by teleconference) Dr Jane Adam, NICE Appraisal Committee and Radiologist, St George's Hospital, London Professor Ron Akehurst, NICE Appraisal Committee and Dean of School of Health and Related Research, University of Sheffield Dr Anne-Marie Bagnall, Research Fellow, NHS Centre for Reviews and Dissemination, University of York Professor David Barnett, Chair, NICE Appraisal Committee Professor Michael Barnett, Professor of Transplantation Oncology, Department of Medical Oncology, St Bartholomew's Hospital, London Dr Pelham Barton, Lecturer in Mathematical Modelling, West Midlands Health Technology Assessment Collaboration, Department of Public Health and Epidemiology, University of Birmingham Mr Neil Betteridge, Head of Public Policy and Campaigning, Arthritis Care Professor Carol Black, President, Royal College of Physicians and Consultant Physician, Royal Free and University College Medical School, London Professor Stirling Bryan, Professor of Health Economics, Health Services Management Centre, University of Birmingham Professor Martin Buxton, NICE Appraisal Committee, Director of Health Economics, Health Economics Research Group, Brunel University Dr Neil Calvert, Research Fellow Sheffield Health Economics Group, School of Health and Related Research, University of Sheffield Dr Karl Claxton, NICE Appraisal Committee and Lecturer in Economics, Department of Economics and Related Research, University of York Mr Andrew Dillon, Chief Executive, NICE Ms Nancy Dixon, Audit Specialist, NICE Professor Jack Dowie, NICE Appraisal Committee and Health Economist, Public Health and Policy Department, London School of Hygiene Ms Sophie Edwards, Chief Executive, Arthritis and Musculoskeletal Alliance, London Dr Dogan Fidan, Health Technology Analyst, NICE Dr Sarah Garner, Health Technology Analyst, NICE Ms Ruth Garside, Research Fellow, Peninsula Technology Assessment Group, University of Exeter
42
Ms Laura Ginnelly, Research Fellow, Health Economics, NHS Centre for Reviews and Dissemination, University of York Professor Tricia Greenhalgh, NICE Appraisal Committee and Professor of Primary Health Care, Primary Care and Population Sciences, University College London Dr Danny Hind, Research Associate, School of Health and Related Research, University of Sheffield Ms Marian Hodges, Editorial Manager, NICE Dr Paresh Jobanputra, Consultant Rheumatologist, West Midlands Health Technology Assessment Collaboration, Department of Public Health and Epidemiology, University of Birmingham Ms Lynn Kerridge, Acting Executive Director, National Coordinating Centre for Health Technology Assessment Dr Carole Longson, Appraisals Programme Director, NICE Dr Alec Miners, Health Technology Analyst, NICE Professor Miranda Mugford, NICE Appraisal Committee and Professor of Health Economics, University of East Anglia Mr David Murray, Technical Team Leader, NICE Ms Judith Paget, NICE Appraisal Committee and Chief Executive, Caerphilly Local Health Board, Wales Ms Suzy Paisley, Managing Director, School of Health and Related Research, University of Sheffield Ms Seren Phillips, Technical Team Leader, NICE Ms Nina Pinwill, Technology Appraisal Project Manager, NICE Professor Sir Michael Rawlins, Chairman, NICE Dr Alison Round, Consultant in Public Health Medicine, Peninsula Technology Assessment Group, University of Exeter & Wessex Institute for Health Research and Development, University of Southampton Mr Miles Scott, NICE Appraisal Committee and Chief Executive, Harrogate Health Care NHS Trust Professor Mark Sculpher, Professor of Health Economics, Centre for Health Economics, University of York Professor Andrew Stevens, Vice-Chair, NICE Appraisal Committee and Professor of Public Health, Department of Public Health and Epidemiology, University of Birmingham Ms Andrea Sutcliffe, Planning and Resources Director, NICE Dr David Taylor, Chief Pharmacist, South London & the Maudsley NHS Trust
43
Mr Adrian Towse, Director, Office of Health Economics, London Professor Norman Waugh, NICE Appraisal Committee, Department of Public Health, University of Aberdeen
44
Annex 2 Documents reviewed 1. NICE documents for each of the four technology appraisals: Scope Protocol Stakeholder submissions Stripped TAR (on the web site) Unstripped TAR (where applicable) Overview [or equivalent] draft Appraisal Consultation Document Slide presentations to Appraisal Committee - Appraisal Committee Lead, NICE TL Appraisal Appraisal Consultation Document (previously PAD) Committee minutes Stakeholder [consultees] responses General public responses draft Final Appraisal Determination Final Appraisal Determination Committee minutes Draft Guidance Final Guidance 2. Other NICE documents Guidance to the technology Appraisal Process Submissions of BNF, DTB to House of Commons Health Select Committee review of NICE NICE responses to BNF and DTB submissions NICE List of personnel and job specifications NICE: A summary of the review of the appraisal process and review of methodology NICE Technology Assessment Report Feedback Form (TARQA) Appraisal Committee: Committee members (description of terms of reference and requirements), June 2002 Declaration of interest on appointment and at meetings: a code of practice for nonexecutive directors, members of advisory committees and employees 3. Draft documents Scientific and social value judgements, first draft v2, 2003 Draft annual report 2002-2003, July 16, 2003 Guide to technology appraisal process, draft 2003 Guide to the Methods of Technology Appraisal, 3rd draft July 2003 4. Internal NICE documents Review of appeals (July 2003) Technology Assessment Report Feedback form, May 27,2003 Project plans for development of the appraisal consultation document and the final appraisal determination Notes for taking and writing the minutes of an Appraisal Committee meeting, January 2003 Technology appraisal programme: procedure for the sign-off of Appraisal Committee 5. Official documents Statutory Instrument 1999 No.260: the National Institute for Clinical Excellence Regulations 1999; http://www.hmso.gov.uk NICE Corporate plan 2002-2005, September 2002, ISBN 1-84257-221-0 NICE Business plan 2002-2003, September 2002, ISBN 1-84257-220-2 NHS NICE Annual report 2001-2002, September 2002, ISBN 1-84257-211-3
45
Annex 3 NICE processes Original scheme of NICE Guidance development process provided to Review Team Technology selected Ð Technology announced Ð Submissions to NICE (manufacturers, patient and professional groups) Ð Submission provided to academic centres (review commissioned by NHS) Ð
Assessment report Ð Response to report, including peer review Ð NICE technology assessment team review and compilation of material Ð
Evaluation report Ð Appraisal committee review (includes consideration of professional groups’ and patients’ groups’ reports) Ð Draft recommendation – development of Appraisal Consultation Document Ð Further comments Ð Interim Guidance (final Appraisal determination)
Appeals
Ð
Final guidance Ð Health authority implementation Ð Feedback and review of Guidance
46
Revised description of process What Topic selection Who (primary) DOH, Welsh government, committee with NICE member (ACTS) NICE staff: technical lead, project manager NCCHTA, TAG NICE, TAG, Consultees TAG TAG (30 weeks out) NICE TL Company submissions developed. Stakeholder submissions developed Who (secondary) Horizon scanning unit (Bham) In-parallel activities
Definition of consultees; development of Scope; call for submissions Contract for specific topic Consultees meeting TAR starts Protocol development
Systematic review commences Industry submissions received and sent to TAGs Unstripped TAR, stripped TAR completed, submitted Stripped TAR Circulates to consultees – responses Stripped TAR (corrected) on web Î responses Overview prepared AC meeting ÎACD finalized ACD circulated, published, comments Redraft of ACD Î draft FAD Second AC– FAD approved FAD posted on web for appeal Appeals if submitted FAD becomes draft guidance Guidance approved Guidance implementation within two months Guidance review
TAG TAG (10 weeks out) TAG NICE, consultees TAG, NICE, general public TL AC {AC lead} Consultees, general public TL, EL AC NICE NICE appeals Board NICE staff Guidance executive, Board Regional authorities, Trusts NICE TAGs? Appeals upheld – go back to AC for review TL, EL TL, EL, expert witnesses TL, EL, PM TL, audit specialist ACD drafted, including audit and expanded budget impact Draft ACD begins (TL-NICE)
47
Annex 4 Process of topic selection Topics suggested by policy (National Service Framework), Horizon Scanning unit, others Ð NCCTHA prepares briefing notes Ð Review by Advisory Committee on Topic Selection against criteria Ð No/yes/maybe/not now decision Ð ‘Yes’ referred to Joint Planning subgroup of ACTS Ð Revised list based on priority and capacity Ð Minister of Health review Ð Suggested list Ð Department of Health, Welsh Government for comments Ð Final list Ð Work program – ‘wave’ announced every 6 months
48
Annex 5 Comparison of recommendations in NICE documents Technology Rheumatoid arthritis: etanercept and infliximab
Assessment from TAR Not clearly stated: ‘ both improve outcomes compared to placebo…no convincing treatment difference between etanercept and a methotrexate’
Recommendation from ACD ‘Neither etanercept nor infliximab is recommended for adults with RA except in the context of a controlled long term b clinical study’
Recommendation from FAD Etanerecept and infliximab (infliximab only in combination with methotrexate) are recommended options for the treatment of adults who have continuing clinically active RA that has not responded adequately to at least two disease modifying drugs including c methotrexate (unless contraindicated) Imatinib is recommended as a treatment option for the management of Philadelphia chromosome positive CML in chronic phase in adults who are intolerant of IFN-α therapy. Imatinib is recommended as an option.in accelerated phase or blast crisis provided they have not received imatinib treatment at an g earlier stage.
Final guidance Etanercept and infliximab (infliximab only in combination with methotrexate) are recommended options for the treatment of adults who have continuing clinically active RA that has not responded adequately to at least two disease modifying drugs including methotrexate (unless fd contraindicated) As for FAD with the addition of ‘failed IFN-α h treatment’
Chronic Myeloid Leukaemia: Imatinib
‘Based on the limited evidence available, imatinib appears to offer an alternative treatment for CML in accelerated and blast phases. As yet, in the authors’ opinion, there is not enough information about imatinib in chronic phase to draw firm conclusions. Cost –utility estimates for imatinib are particularly sensitive to assumptions about long term e survival and may be extremely high.’
Imatinib is recommended for the treatment of patients with accelerated phase CML. There is insufficient evidence on which to recommend the use of imatinib for the routine treatment f of chronic or blast phase CML.
Schizophrenia: Atypical Antipsychotics
‘Evidence for the effectiveness of the atypical antipsychotics compared to older drugs is in general of poor quality…available evidence suggests that [atypicals] are as effective or i more effective.. than typical antipsychotics. ‘Real time2-D US is significantly better than m landmark’
It is recommended that the atypical antipsychotics are considered as the first-line treatment of choice for people j with newly diagnosed schizophrenia… It is recommended that 2-D imaging US guidance should routinely be used when a central venous line is being inserted in an elective or emergency n situation.
As for ACD
k
As for FAD
l
Ultrasound device for central venous catherisation
2-D imaging US guidance is recommended as the preferred method for insertion of central venous catheters into the internal jugular vein in adults and children in elective situations. The use of US…either electively or in an emergency o situation.
As for FAD
p
a
TAR Executive summary page 12 (unstripped version); b PAD, page 1; c FAD, page 1; d Section 1.1, Guidance; e Assessment report, executive summary, page 7; f ACD, Sections 1.1 and 1.2; g FAD Sections 1.1 and 1.3; h Guidance, Section 1.1 and 1.3. I Assessment report, executive summary, pages 8, 9; j PAD Section 1.1; k FAD, Section 1.2; l Guidance, Section 1.2; m Assessment report, Executive summary page 2;n ACD Section 1.1; o FAD Section 1.1 and 1.2; p Guidance Sections 1.1, 1.2
49
Annex 6 Comparison of international recommendations Technology Rheumatoid arthritis; etanercept, infliximab
Final guidance E and I (I only in combination with methotrexate) are recommended options for the treatment of adults who have continuing clinically active RA that has not responded adequately to at least two disease modifying drugs including methotrexate (unless contraindicated)
CCOHTA Infliximab not assessed for RA. Etanercept: Canadian clinical and economic data comparing methotrexate and etanercept alone versus etanercept in combination with methotrexate would be necessary before informed predictions could be made on its cost-effectiveness in our health care system. While the direct costs of this therapy are high, if etanercept controls the disease more effectively than current treatments, or earlier in the progression of the disease, it may have incremental cost-effectiveness. Infliximab
PBAC
Netherlands Infliximab Adults and children with RA who have failed standard treatment with at least methotrexate at optimal dose. Use is in the context of a Phase 4 study. Etanercept
Italy (Ministry of Health, CUF) Infliximab Adults and children with RA who have failed standard treatment with at least methotrexate at optimal dose. Only for hospital use (can be given to outpatient after started in hospital) in the context of the observational ANTARES study. Not available through community pharmacies. Etanercept Patients with RA who have failed standard treatment with at least methotrexate at optimal dose. Only for hospital use (can be given to outpatient after started in hospital) in the context of the observational ANTARES study. Not available through community pharmacies
Section 100 listing for the treatment of adult patients with severe active rheumatoid arthritis who have a record of rheumatoid factor positive status, who have failed other treatment regimens. Etanercept Section 100 listing for the treatment of adult patients with severe active rheumatoid arthritis, who have a record of rheumatoid factor positive status and who have failed other treatment options; For the treatment of juvenile patients aged 4 to 17 years with active polyarticularcourse juvenile chronic arthritis, who have failed other treatment regimens.
Adult Patients with RA who have failed standard treatment with at least methotrexate at optimal dose. Use is in the context of a Phase 4 study.
50
Comparison of international recommendations (continued) Technology Chronic Myeloid Leukaemia: imatinib
Final guidance Imatinib is recommended as a treatment option for the management of Philadelphia chromosome positive CML in chronic phase in adults who are intolerant of, or who have failed IFN-α therapy. Imatinib is recommended as an option .in accelerated phase or blast crisis provided they have not received imatinib treatment at an earlier stage.
CCOHTA The advent of imatinib is considered an advance in the treatment of CML, allowing for targeted therapy against inappropriate tyrosine kinase activity. Available preliminary data, although not published in full, appears promising.
PBAC Section 100 listing for treatment of patients with chronic myeloid leukaemia expressing the Philadelphia chromosome or the transcript, bcr-abl tyrosine kinase. Patients must be in the chronic phase where the use of interferon alfa has failed. Recommended for listing as requested with additional requirement that patients must achieve a major cytogenetic response within 18 months and demonstrate that the response is maintained by testing every 12 months. The PBAC will review new survival data after 1 year, 2 and 5 years. PBAC – currently lists most atypicals as ‘Authority required for the treatment of schizophrenia’
Netherlands Imatinib may be considered for use in the treatment of patients with chronic myeloid leukaemia in case of failure, intolerance or contraindications to interferonalpha. Definite data on effectiveness of imatinib in terms of survival data are lacking. This advice that it may be considered for use is only based on 2 Phase 2 studies and indirect comparison with standard therapy comprising hydroxyurea and combination chemotherapy.
Italy (Ministry of Health, CUF) Imatinib is for the treatment of patients with chronic myeloid leukaemia in case of failure, intolerance or contraindications to interferon-alpha. Only for hospital use.
Schizophrenia: Atypical antipsychotics
It is recommended that the atypical antipsychotics are considered as the first-line treatment of choice for people with newly diagnosed schizophrenia
CCOHTA: The cost-utility analysis demonstrated that risperidone was the dominant strategy compared to haloperidol, haloperidol decanoate or fluphenazine decanoate in hospitalized patients with chronic schizophrenia with moderate symptoms. The estimated cost savings (versus haloperidol) was approximately $6,500 per patient per year while producing 0.04 more QALYs per year.
Olanzapine: Patients who failed or are intolerant to classical antipsychotics, and patients with negative signs and symptoms of schizophrenia, or patients who responded with an acute dystonia to other antipsychotic drugs. Atypicals Patients who failed or are intolerant to classical antipsychotics, and patients with negative signs and symptoms of schizophrenia.
Risperidone and olanzapine: Restricted to an initial prescription from a specialist together with a therapeutic program.
Sources: CCOHTA: etanercept: CCOHTA Issues in emerging health technologies, Issue 8, 1999; imatininb: CCOHTA Emerging drug list 2001; Glennie JL. Pharmacoeconomic evaluations of clozapine in treatment-resistant schizophrenia and risperidone in chronic schizophrenia. Ottawa: Canadian Coordinating Office for Health Technology Assessment (CCOHTA); 1997 Italy – Gazzetta Ufficiale della Republica Italiana Netherlands: www.cvz.nl, accessed 19 July 2003. PBAC web site (http://www.health.gov.au/pbs/listing/pbacrec/index.htm). Note that Price –volume agreement has been agreed as part of listing process for etanercept’ with the total cost to Government not to exceed more that $AUD100million per year (see Ministerial press release dated 30 June on the same web site.
51
WHO REVIEW OF THE TECHNOLOGY APPRAISAL PROGRAMME, NATIONAL INSTITUTE FOR CLINICAL RESEARCH
Technology Appraisal Programme of the National Institute for Clinical Excellence A review by WHO Suzanne Hill Silvio Garattini Jos van Loenhout Bernie J. O’Brien Kees de Joncheere June-July 2003 Ref. no.: 5045738 ABSTRACT The National Institute of Clinical Excellence (NICE) is responsible for providing guidance to the National Health Service in England and Wales on the clinical- and cost-effectiveness of medicines and medical technologies. After a review of the Institute by the House of Commons Health Select Committee in 2002, NICE commissioned a series of internal reviews on their work, and requested the WHO Regional Office for Europe to carry out an external review on their methods and processes as well as their scientific robustness. The WHO team of experts based their report on the review of a series of technology appraisals, including confidential material, and extensive discussions with NICE staff, Appraisal Committee members, members of the Technical Assessment Groups and other stakeholders during the course of two visits to NICE in summer 2003. The report contains a series of recommendations on how NICE could further develop the technology appraisal process. As NICE is internationally a leading agency in technology assessment, the important recommendations and observations contained in the report will be useful to other countries that are embarking on similar endeavours. KEYWORDS TECHNOLOGY ASSESSMENT, BIOMEDICAL PHARMACEUTICAL PREPARATIONS – STANDARDS COST-BENEFIT ANALYSIS ECONOMICS, PHARMACEUTICAL EVALUATION STUDIES EVIDENCE-BASED MEDICINE UNITED KINGDOM EUROPE Address requests about publications of the WHO Regional Office to: • by e-mail publicationrequests@euro.who.int (for copies of publications) permissions@euro.who.int (for permission to reproduce them) pubrights@euro.who.int (for permission to translate them) • by post Publications WHO Regional Office for Europe Scherfigsvej 8 DK-2100 Copenhagen Ø, Denmark © World Health Organization 2003 All rights reserved. The Regional Office for Europe of the World Health Organization welcomes requests for permission to reproduce or translate its publications, in part or in full. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Where the designation “country or area” appears in the headings of tables, it covers countries, territories, cities, or areas. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. The World Health Organization does not warrant that the information contained in this publication is complete and correct and shall not be liable for any damages incurred as a result of its use. The views expressed by authors or editors do not necessarily represent the views of their institutions or the decisions or stated policy of the World Health Organization. CONTENTS Page List of abbreviations and definitions ..................................................................1 Acknowledgements ...........................................................................................2 Executive summary ...........................................................................................3 Recommendations ............................................................................................5 1. Introduction..................................................................................................10 1.1 Background to the review ......................................................................10 1.2 The health policy context .......................................................................12 1.3 What is NICE? .......................................................................................13 2. Methods and approach of the review ..........................................................15 3. Findings and recommendations ..................................................................17 3.1 Principles that determine the NICE approach to technology assessment. ..........................................................................17 3.2 The process of Guidance development .................................................19 3.3 The quality of the science of the assessment and appraisal process..................................................................................34 4. Conclusions.................................................................................................39 References......................................................................................................41 Annex 1: Persons interviewed (in person or by teleconference) .................42 Annex 2: Documents reviewed ....................................................................45 Annex 3: NICE processes............................................................................46 Annex 4: Process of topic selection .............................................................48 Annex 5: Comparison of recommendations in NICE documents .................49 Annex 6: Comparison of international recommendations.............................50 1 List of abbreviations and definitions AC Appraisal Committee ACD Appraisal Consultation Document Consultee Organizations from the following areas that accept an invitation to participate in the appraisal: the manufacturer(s) or sponsor(s) of the technology; national professional organizations; national patient organizations; Department of Health and Welsh Assembly Government ; Primary care Trusts in England and Local Health Boards in Wales CCOHTA Canadian Coordinating Office for Health Technology Assessment CML Chronic myeloid leukaemia CPMP Committee on Proprietary and Medicinal Products CVZ Health Insurances Board of The Netherlands EL Executive Lead (on each technology appraisal) FAD Final appraisal determination Guidance Published document issued by NICE describing appraisal of technology Guideline (from NICE) published document describing the appropriate treatment and care of people with specific diseases and conditions within the National Health Service in England and Wales HTA Health technology assessment NCCHTA National Coordinating Centre for Health Technology Assessment NHS National Health Service NICE National Institute for Clinical Excellence PAD Provisional appraisal determination (now the Appraisal Consultation document PBAC Pharmaceutical Benefits Advisory Committee (Australia) PM Project Manager QALY Quality adjusted life year RAE Research assessment exercise Scope Initial document describing the question to be addressed in a NICE technology appraisal TAG Technical Assessment Group TAR Technical assessment report TL Technical Lead WHO World Health Organization UK United Kingdom 2 Acknowledgements For the Review Team, undertaking this review was an extraordinary opportunity and privilege. The chance to assess such a high profile and innovative agency is not one that occurs frequently and the Review Team hopes to have been able to provide an assessment and recommendations that will assist The National Institute for Clinical Excellence (NICE) (and others) in undertaking the vital task of health technology assessment (HTA) and guidance for the improvement of the quality of health care. The Review Team would like to acknowledge the generous assistance and co-operation provided by everyone they met during the course of the review. In particular, the Team would like to thank Professor Sir Michael Rawlins, Chairman of the Board of NICE Mr Andrew Dillon, Chief Executive Professor David Barnett, Chair of the Appraisal Committee Dr Carol Longson, Appraisal Programme Director for their hospitality and enthusiasm which was manifest in the time they spent in discussion with the Team. The Team would also like to thank all the members of the Appraisal Committee, the appraisal teams, and the additional experts and interviewees (Annex 1) who very kindly agreed to be interviewed at very short notice. Again, the Team’s job was made substantially easier by the openness of these people and their enthusiasm about contributing to the to the review. The review would not have been possible at all without the assistance of the NICE administrative personnel and especially Kathleen Dalby who dealt with all the logistics of the visit, including the arrangement of 50 interviews at two weeks’ notice and the provision of endless numbers of documents. 3 Executive summary NICE was established as a Special Health Authority to address a number of problems with the introduction and use of new technologies and medicines within the United Kingdom National Health Service (NHS). The review of NICE carried out by the House of Commons Health Select Committee in 2002 identified a number of issues and concerns and recommended an external review of the Institute, its functions and processes. NICE requested the World Health Organization (WHO) Regional Office for Europe to carry out this review. The terms of reference for the review focused on the methods and processes used in the NICE technology appraisal programme. The review was undertaken by a team of international experts in June-July 2003. The conclusion of the Review Team was that, in only four years, NICE has developed a well-deserved reputation for innovation and methodological developments that represent an important model for technology appraisals internationally. In the environment in which NICE functions, substantial changes are taking place in the way in which the NHS operates and additional resources are being invested in the health services. This facilitates the achievement of the NICE objectives of improving quality of care throughout the health services and the uptake of new cost-effective medicines and technologies. Achievements that are particularly valuable include: the transparency surrounding the process of technology assessment; the intensive participation of different stakeholders and the inclusiveness of the approaches taken; the commitment to using the best available evidence for decision-making; the commitment of the technical and management staff; and the dedication of the appraisal team and the Appraisal Committee members. All of these form the cornerstones of an organization that continues to invest in quality development. Overall, the Review Team was impressed by the commitment to using of rigorous methodology throughout the process of technology assessment, starting with the use of academic centres of excellence for independent technology appraisal, which is supplemented by the additional evidence handled by the technical teams in NICE. Published technology appraisals are already being used as international benchmarks - an obvious recognition of their credibility. As part of the commitment to further develop and improve methods and processes, NICE is currently carrying out internal reviews of these areas. In undertaking the review and preparing the report, rather than reconfirming the well-established achievements of NICE, the Review Team focussed on the areas where there is controversy or potential for improvement. A number of key recommendations are worthy of specific mention here, namely: 4 • To reconcile the inherent contradiction between the Institute’s key principle of transparency and the acceptance of material that is designated as confidential. The Review Team welcomes the steps that NICE has already taken to date, and encourages NICE to use its leverage to advance transparency in this area. This would be of benefit internationally. • To revise the contractual arrangements for the development and production of assessment reports so that they fully meet the needs of NICE and its Appraisal Committee. • To consider ways to reduce unnecessary duplication of effort in the assessment phase. Efforts should be made to ensure that the Appraisal Committee is presented with a single set of analyses produced by the Technical Assessment Group that incorporates consultation with and input from the manufacturer(s). • To ensure the collection of relevant information from all stakeholders as early as possible in the assessment process. • To increase the exchange of information and interaction – at all stages of the process - between the Technical Assessment Group, the Appraisal Committee and the NICE Appraisal Team. • To develop a handbook on the preparation of assessment reports; this could be a product of the current review on methods and process. • To make clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than representation of particular interests. Although there is a need to ensure that the manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. • To continue to work on the difficult but important task of developing clear criteria to obtain optimal interaction between the ethical and social values and the scientific evidence for use by the Appraisal Committee and NICE in decision-making. The recommendations are to be seen as an effort to further enhance the operations of NICE, and assist organizations with similar responsibilities in other countries to deal with their difficulties and meet their expectations. 5 Recommendations On the principles 1. NICE should continue to develop operational procedures that are consistent with its core principles of transparency, consultation and inclusiveness with respect to stakeholders’ involvement in evidence- gathering and decision-making. The NICE model of partnership in the scientific endeavour of health technology appraisal offers valuable international leadership. 2. NICE should reconcile the inherent contradiction between its principles of transparency and its acceptance of evidence for the decision-making process that a stakeholder deems to be confidential. NICE should continue the work already started on this issue to ensure that all material submitted for consideration can be made available to the public. 3. The principle of transparency requires that NICE codifies and justifies the specific criteria used in decision-making. Difficult but important elements of this task are articulation of the ethical and social value judgements, and definition of the interaction of these judgements with the appraisal of the scientific evidence used by the Appraisal Committee in reaching its decisions. On the process – general issues 4. For drug and device technologies where a sponsor exists, the current hybrid process of HTA used by NICE may give rise to unnecessary duplication of effort. NICE may wish to consider using different approaches depending on the subject of the appraisal. The Review Team suggests that an effort be made to ensure that the Appraisal Committee is presented with a single set of analyses produced by the Technical Assessment Group that incorporates consultation with and input from the manufacturer(s). On the start of the process and topic selection 5. The consultees’ meeting should become a formal ‘Preliminary Exchange of Evidence’, at which time all stakeholders should be asked to provide details of what they propose to submit. Only in extraordinary circumstances should the Technical Assessment Group accept to include, as part of the review, information over and above that declared for submission by the stakeholders at this meeting. 6. More attention needs to be paid to the important task of setting the Scope of the appraisal. The general goal is to give timely and comprehensive guidance to the Technical Assessment Groups on the question(s) to be addressed by the Appraisal Committee, thereby reducing the risk of mismatch between the technical assessment report and the needs of the Appraisal Committee. 6 7. Stakeholder submissions should be required to be lodged as soon as possible after the start of the assessment process. On the assessment and appraisal process 8. The timeframes for the assessment and appraisal process should be reviewed, so that the current time pressures at the end of the assessment report period and early appraisal period are reduced and more time allowed for critical evaluation of the consultees’ comments. This does not necessarily mean that the overall timeline should be increased. 9. The procedures for document management need to be carefully considered. This may include keeping some hard copy files of key documents for each Guidance, as well as the electronic files. 10. NICE should improve interaction between Technical Assessment Groups, the NICE technical staff and the Appraisal Committee throughout the assessment and appraisal process. This could include, for example, having the continued involvement of the Technical Assessment Group as technical experts after the initial review of the appraisal consultation document. On the functions of the Appraisal Committee 11. Given that a third Appraisal Committee is being introduced, NICE should take the opportunity to consider how to sustain the high quality performance of the Appraisal Committee. An issue that needs particular consideration is the question of consistency in decision- making across the three committees. NICE is already giving this question careful attention. Two possibilities to consider are: 1) having three individual Chairs and one common Vice-Chair; and 2) having the three Committees assess different types of technologies (e.g. diagnostic procedures for one and specific classes of pharmaceuticals for the other two). 12. It should be made clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than on the representation of particular interests. Although that there is a need to ensure that manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. 13. The role of the Chair may need to be further refined. While it may be necessary for the Chair to continue to take an active role in leading the discussion on many items, the possible risks associated with this need to be carefully considered. Asking members to play a greater role in the ongoing review of a technology would allow the Chair to facilitate rather than lead the discussion, although the increased workload that would result for the Committee members would need to be assessed. 7 14. NICE should assess whether the overall sustainability of the process and the functioning of the Appraisal Committee could be improved by the introduction of some type of reimbursement for members’ time. 15. The induction and training process for Appraisal Committee members needs to be further developed to include, where appropriate, enhancing skills in the critical appraisal of clinical evidence and economic evaluation. 16. The Appraisal Committee procedures should be modified to ensure that a clear statement of what the Committee approves is recorded on the same day. On decision-making 17. NICE and the Appraisal Committee should continue their process to develop a system of decision-making that encourages articulation of the grounds for a particular recommendation, including specification of the weight that is placed on clinical evidence, economic evidence and other factors, such as equity and social values. 18. As part of the process of articulating the criteria for decision-making, NICE must resolve the confusion related to the use of a value-for- money threshold. If a threshold is to be used as a basis for recommendations, it should be specified and justified for reasons of transparency. 19. The Appraisal Committee may wish to consider having a legal advisor present during meetings, particularly those involving matters referred to the Committee by the Appeal Board. On the Appeals 20. In view of the considerable number of appeals lodged and their time and resource implications, thought needs to be given to how to reduce the number of appeals and the length of the appeal process. The further development of the appeal process will help enhance the quality and the transparency of the appraisal programme. 8 On budget impact and research 21. Although budget impact is not a consideration in making recommendations on the use of a technology within the NHS, it is important to develop methods for budget impact modelling that would enable NICE to provide more detailed information on the implementation costs to the local authorities. This could be a task for the technical analysts in NICE. The advantages of doing so include not only the provision of useful advice to the Trusts, but also the avoidance of a duplication of effort by the Trusts in making their budget analyses for implementing the new technology. 22. NICE should further develop the Research Required sectionof the Guidance and link it specifically with the review of Guidance review process. This would help to obtain the additional clinical evidence needed to ensure full understanding of the benefit of a technology. The Guidance review process would be an ideal way to stimulate the generation of such evidence. 23. NICE should adopt more flexible timeframes for reviewing existing Guidance not only to ensure that the review answers a specific question but also to assist in managing workload. One approach might be for the set review dates to be dependent on the emergence of new evidence or significant changes in existing evidence. On the technical assessment report 24. The contractual arrangement between the Technical Assessment Groups and the NCCHTA should be revised to recognize NICE as the primary client for the assessment reports. 25. A handbook of standard methods for the assessment reports should be developed, after consultation between NICE staff, the Appraisal Committee and the Technical Assessment Groups. The handbook, which should be regularly updated, should also be used as the basis for training new reviewers and new NICE staff. 26. A detailed template for the reports should be developed, including standardized data presentation and summary material. This would facilitate review of the information by the Appraisal Committee. 9 On input from the consultees 27. On the basis of experience to date, NICE should consider what aspects of patient and professional submissions are most useful and develop standards for the content of these types of submissions. 28. NICE should review the process for assessing comments from consultees and others (including the general public) that are submitted after the appraisal consultation document is drafted. NICE should determine the most appropriate way of responding to these inputs and assess their value in the decision-making process. 10 1. Introduction 1.1 Background to the review In 1999, NICE was established as a Special Health Authority to address problems related to variation in the quality of care and the introduction and use of new technologies and medicines within the NHS in England and Wales. As described by the founding and current Chair of the Board1, the initial three broad functions of the Institute were: • appraisal of new and existing health technologies; • development of clinical guidelines; • promotion of clinical audit and confidential enquiries. There were perceptions that availability of technologies, as well as quality of care, varied by ‘post-code’. ‘Post code prescribing’ described the situation where patients’ access to treatments was determined by decisions made by local health authorities that have budgetary control over purchasing new technologies for a given geographic area. There were concerns that there should be faster access to new medicines and that the NHS needed to achieve greater value for money. NICE was designed as an independent and national authority to address all of these needs. In 2002, there was a House of Commons Health Select Committee review of the performance of NICE. In its report 2, the Committee identified a number of issues and concerns and recommended an external review of the organization, its functions and processes. The key matters raised by the Committee were that: • there was a perception of problems and issues in relation to the consultation process and methodology used in assessments and appraisals of technology; • there was a need for an assessment of the scientific validity of processes used particularly regarding the robustness of the approach; • there was uncertainty about the practical value of Guidance and its implementation by Health Authorities. Following the House of Commons Health Select Committee report, in December 2002, NICE requested the WHO Regional Office for Europe to carry out a review of the Technology Appraisal Programme. The particular aspects identified for the WHO review were the methods and scientific robustness of the process. NICE had also started to carry out separate reviews on issues related to the consultation processes and the implementation of Guidance, as well as on methodological issues related to Guidance development. The results of this external review will contribute to the separate reviews on methods and process. From information provided to the Review Team, it appears that these reviews are essentially designed to address two areas: the technology appraisal process and the methodologies used to develop technology appraisal guidance. The outputs are planned as documents on the appraisal process and an update and expansion of the current Guidance for manufacturers. The Review Team was provided with copies of the draft documents from these reviews. 11 The agreed terms of reference for the WHO review are listed in Box 1. WHO identified a team of international experts with skills and experience in scientific methodology relevant to HTA as well as in decision-making and policy (Box 2). Box 1. Terms of reference 1. With reference to established and credible processes and methodologies, review criteria should be identified for the evidence assessment and appraisal components of NICE technology appraisals. These criteria should test: the scientific and methodological rigor of the health technology assessment reports used by the Appraisal Committee; the utility of the other components of the evidence base, including submissions from manufacturers, patient organizations and professional groups; the suitability of the processes used to assemble and interpret evidence and to engage with stakeholders during the appraisal; the consistency of the outcome of the appraisal with conclusions reached on the same topics by reputable scientific groups outside the UK. 2. A minimum of 3 and a maximum of 5 health technology appraisals, published by the Institute between March 2000 and December 2002, should be selected for review. The appraisals selected should reflect, as far as possible, the range of technologies, which the Institute has so far considered and should be topics, which have been considered by reputable scientific groups outside the UK. 3. A report should be prepared for publication by WHO and for consideration by the Board of the Institute, setting out the results of the review, the conclusions reached by the reviewers and their recommendations to the Institute for any changes to the current appraisal process and methodology, which will have the effect of enhancing the scientific credibility or clinical utility of the Guidance. 12 Box 2. External Review Team members Suzanne Hill B.Med PhD FAFPHM Senior Lecturer, Clinical Pharmacology Faculty of Health, University of Newcastle Australia Director, Newcastle Evaluation Group for the Australian Pharmaceutical Benefits Advisory Committee Silvio Garattini MD Istituto di Ricerche Farmacologiche 'Mario Negri' Milan, Italy Member of Committee for Proprietary Medicines Products of European Agency for the Evaluation of Medicines Jos van Loenhout MD PhD Senior Medical Advisor to the Reimbursement Committee of the Dutch Healthcare Insurance Board Diemen, The Netherlands Bernie J O’Brien, PhD Professor, Dept Clinical Epidemiology and Biostatistics McMaster University, Canada Member, Economics Sub-Committee of the Drug Quality and Therapeutics Committee for Ontario Kees de Joncheere Regional Adviser for Health Technology and Pharmaceuticals WHO Regional office for Europe 1.2 The health policy context Before considering the technology appraisal process in detail, it is important to define the context in which NICE is operating and making its Guidance, as the context is particular to England and Wales. The UK National Health Service has provided universal access to health care for British citizens since 1948. Features of the NHS that are pertinent to this review are that policies and standards of care are set centrally (by the Department of Health), and the implementation and delivery of the health services are managed locally and funded by NHS Trusts (hospital and primary care) that employ the staff involved in health care delivery and provide the services. Further details on the NHS can be found at http://www.nhs.uk. The current management approach for resource allocation decisions is decentralised. Budget responsibility lies with the Primary Care Trusts that commission services from general practitioners and Hospital Trusts. Health technologies and health services are selected, provided and paid for by Trusts. Each Trust is free to allocate its funds at its own discretion to best provide for the health care needs of its population within a geographical area. In the UK, there are two notable policy concerns about the quality of health care provided in the NHS. These are that the UK does not spend as much on health care as do comparable countries (see OECD Health Data 2003 for recent comparisons, http://www.oecd.org) and that the services and technologies available vary from one Trust to the next1. Health professionals have partly attributed this variability to the slow uptake of new technologies. 13 These two issues mean that it is the current overall policy direction and current government commitment to increase expenditure on healthcare and to promote the adoption and uptake of health technology that is established as cost-effective. A further aspect of the policy environment that should be noted is the commitment within the health sector to innovation. New technologies are seen to have a value because of their novelty. This policy environment is different from other national environments, where the major need is to limit or control expenditure on health care and to be more selective about the uptake of health technologies, while continuing to guarantee high quality healthcare. As Primary Care Trusts make the funding and purchasing decisions, the issues of the local opportunity cost of new health care interventions are likely to be a primary consideration in the adoption of new technologies and services at the local level. There is, as a result, a tension between the national policy to increase health expenditure and the decentralised process of purchasing. This tension is one of the most frequently mentioned issues in the recently published commentary about NICE3. From an international perspective, it is important that other organizations that may be considering emulating some of the functions of NICE recognize this specific policy environment. NICE is actively establishing international links. 1.3 What is NICE? NICE has been in operation for only four years. Within this short period of time, NICE has already established an enviable national and international reputation for innovation and developments in HTA. NICE is evolving and the following description of the organization, its structure and functions is based on information available in June 2003. NICE has at present four functions: • To produce Guidance in relation to the use of new and established medicines and treatments available within the NHS (technology appraisals). • To produce clinical guidelines about the appropriate treatment and care of people with specific diseases and conditions within the NHS. • To produce Guidance on whether interventional procedures used for diagnosis or treatment are safe enough and work well enough for routine use. • To identify ways of improving the quality of care (confidential enquiries). NICE describes itself as a virtual organization and has a relatively small staff (approximately 60) based in London. As in other NHS Special Health Authorities, a Board of Executive and Non-Executive Directors manage the Institute. It has a number of committees that are required for administrative functions (the Audit Committee, the Risk Management Committee, the Research and Development Committee, the Remuneration and Terms of Service Committee) as well as the technical expert independent advisory committees that assist in its technical and scientific activities. The Technology 14 Appraisal Committee is the relevant technical advisory committee for this review. The Directions that established NICE also include the requirement for a ‘Partners Council’, membership of which is drawn from organizations (patient, professional and health care industries) that have a special interest in the work of NICE. The in-house team is divided into four functional groups: the Communications Team, the Corporate Team, the Guidelines Team and the Technology Appraisal Team. The Technology Appraisal Team comprises a Director, two Technical Team Leaders each responsible for a group of health technology analysts and information specialists, and two project management teams responsible for overseeing the administrative management of the technology appraisals. The increasing workload of the Institute has led to a rapid expansion of the staff. The outputs from the NICE programme of work over the four years of its existence are a tribute to the commitment and hard work of all involved. The products to date are 62 technology Guidances (covering topics ranging from the extraction of wisdom teeth to treatments for managing diabetes), 14 clinical guidelines, including some inherited from the previous guidelines development programme in the UK, commencement of the interventional procedures work programme, as well as the carrying out of a number of confidential enquiries. In addition, NICE has developed documents that provide descriptions of the processes and methods it uses for each aspect of its work. For this review, the Team specifically considered NICE Guidance (as distinguished from its Guideline programme). Further details about NICE can be found at http://www.nice.org.uk. 15 2. Methods and approach of the review The Review Team visited the NICE offices in London twice: on 5-12 June 2003 and on 17-21 July 2003. As the starting-point for its investigations, the Review Team consulted publicly available information about NICE, including its web site, and four technology appraisals for which Guidance had been issued in the 12 months up to the end of 2002. The technology appraisals were used as illustrative case studies and as a means of understanding the process and evaluating the quality of the assessments. Given that the process of technology assessment and appraisal by NICE is changing rapidly, these technology appraisals clearly do not constitute a representative sample of the 62 appraisals published to date. The technology appraisals selected included one non-drug or medical device technology, one that had been subject to appeal one oncology technology, and one drug technology. This selection ensured that the work of a number of different assessment groups was represented. The four technology appraisals are listed in Table 1. Table 1: Technologies assessed Technology Guidance Assessment report authors Appeals Date Guidance issued Etanercept and infliximab for rheumatoid arthritis #36 Birmingham None March 2002 Atypical antipsychotics for schizophrenia #43 York Appeal June 2002 Ultrasound device for central venous catherisation #49 Sheffield None Sept. 2002 Imatinib for chronic myeloid leukaemia #50 Exeter None Oct. 2002 The Review Team interviewed NICE staff, members of the assessment groups, Appraisal Committee members and external experts. Those interviewed (by phone and in person) are listed in Annex 1. All the documents, including stakeholder submissions, that had been considered by the Appraisal Committee in respect of each technology were provided to the Review Team. The Team reviewed these documents not only in the context of the report of the House of Commons Health Select Committee (also made available to the Review Team, including the submissions of the British National Formulary and the Drugs and Therapeutics Bulletin to the Health Select Committee) and the terms of reference for the review, but also in the light of recently published commentaries on NICE3-15. A full list of the documents considered is given in Annex 2. The members of the 16 Review Team signed confidentiality agreements to enable them to review confidential material. The Review Team attended an Appraisal Committee meeting during the first visit. The Review Team used the technologies selected to develop an understanding of the NICE appraisal and guidance development process and studied the four technology appraisals specifically. For each technology, the Review Team considered: • the quality and rigor of the assessment report • the quality and relevance of the other documents provided to the committee • views on the usefulness and quality of the reports from those involved in the process of decision-making; • views from the authors of the reports on the outcomes of the process in relation to the evidence • the final Guidance and the recommendations in the light of the evidence assessed in the evaluation reports. These discussions and assessments were used as a basis to: • identify any aspects of the review process and reports that could be modified to improve the reliability and scientific robustness of the Guidance • identify any areas where improvements in methodology could improve the validity of the Guidance • make other recommendations about the process. On completion of the first visit, the Review Team drafted a report, which was provided to NICE during the second visit for correction of any factual errors. The draft report and recommendations were discussed at this time with NICE staff and the Chair of the Board. In undertaking the review and preparing the report, rather than reconfirming the well-established achievements of NICE, the Review Team focussed on areas of controversy or where there is potential for improvement. The report represents the consensus view of the Review Team. 17 3. Findings and recommendations 3.1 Principles that determine the NICE approach to technology assessment. The key principles that underpin the NICE approach to decision-making and Guidance development are: • use of the best available evidence in decision-making • transparency • consultation • inclusion all key stakeholders • responsiveness to change.15 From the stage of considering which topics are to be selected for appraisal to completion of the final Guidance, there is ample opportunity for all stakeholders to provide input into the process: from formal submissions by stakeholders to comments by experts and the general public. The information available on the NICE web site (from the selection of topics for assessment to completion of the final Guidance, including information on the appeals) provides interested parties and the general public with a clear view of the assessment and guidance development process. There are numerous consultative bodies that can feed into the processes, the principles - on the basis of which NICE works - and the ongoing appraisals. “Use of best available evidence” implies the use of data on final clinical outcomes rather than on surrogate endpoints, a preference for data from controlled studies and the transparent and complete presentation of the data. Equity considerations play an important role in the appraisal process, as the health implications for the wider group of patients in the NHS should be considered alongside the effects for those patients directly benefiting from the technology under consideration. NICE takes the following factors into consideration in technology appraisals: • The broad clinical priorities of the NHS • The degree of clinical need of patients with the condition under consideration • The broad balance of benefits and costs • Guidance already available within the NHS and guidance from the Secretary of State • The effective use of available resources. Within the assessment process - through the TAGs - a systematic and independent evaluation of the relevant available evidence takes place. As NICE has to take decisions on the basis of the “best available” evidence, obviously problems occur where there is limited or incomplete evidence, and where NICE has to make judgements in formulating the Guidance. Currently NICE is refining the scientific and social value judgements that are guiding its 18 decisions. These discussions also benefit from the deliberations of the Citizens Council that attempts to capture views from society. The Review Team noted the following. • In all of these areas, it is clear that NICE is setting a new, international benchmark, for which it can and should be congratulated. In particular, commitment to the consideration of stakeholder inputs in the decision- making process and to the provision of documents and information publicly, via the NICE web site, are recognized by the Review Team as setting new standards. • There is, therefore, an apparent contradiction in the current policy that allows material that is designated confidential to be used as a basis for decision-making. The majority of the confidential material is submitted by the pharmaceutical industry, although there have been examples of other material, such as that from academic centres, being designated confidential. Such material is removed from assessment reports before their circulation and publication and is not included in any other public document. • Allowing confidential material as part of the decision-making process obscures the transparency of decisions and possibly jeopardises the quality of the NICE assessment process. As described by Drummond10, using ‘commercial-in-confidence’ data can inhibit the opportunity for there to be potential checks and balances on an agency’s work. There is also the question of conflict in philosophy and values given the public commitment of NICE to transparency. Although the problem of ‘commercial-in-confidence’ data is present in all European countries and although the Review Team realizes that under the current legal system the submission of ‘commercial-in-confidence’ material is clearly admitted, the commitment of NICE to transparency and public accountability should override the confidentiality issues, and make it possible for decision-making to become even more transparent and open to scrutiny. NICE has already taken steps in discussions with the industry to limit the inclusion of confidential information in the assessment process. The Review Team encourages further steps to be taken in this direction. 19 Recommendations 1. NICE should continue to develop operational procedures that are consistent with its core principles of transparency, consultation and inclusiveness with respect to stakeholders’ involvement in evidence- gathering and decision-making. The NICE model of partnership in the scientific endeavour of health technology appraisal offers valuable international leadership. 2. NICE should reconcile the inherent contradiction between its principles of transparency and its acceptance of evidence for the decision-making process that a stakeholder deems to be confidential. Although NICE has already started on this issue, further steps should be taken to ensure that all material submitted for consideration can be made available to the public. 3. The principle of transparency requires that NICE codifies and justifies the specific criteria used in decision-making. Difficult but important elements of this task are the articulation of the ethical and social value judgements, and definition of the interaction of these judgements with the appraisal of the scientific evidence used by the Appraisal Committee in reaching its decisions. 3.2 The process of Guidance development 3.2.1 General issues Prior to the visit in June, the Review Team was provided with a scheme of the Guidance development process (Annex 3). As this process has clearly evolved, the Review Team developed a description of the current process of Guidance development (Annex 3) that was used as a basis for discussions and recommendations. The timeframe for the development of NICE Guidance is stated as 62 weeks from the start of the process to the close of a possible appeal and issue of the final Guidance. A process of ‘topic selection’ (see section 3.2.2) determines whether a particular drug or technology will be assessed. If a product is not assessed, it is up to the local Trusts to determine whether it will be paid for. NICE issues each technology Guidance with a planned date for review. Functionally, there are two key components of the system: the assessment component, defined as the collation and synthesis of the clinical and economic evidence to address the policy question, and the appraisal component that is the consideration of the evidence, including stakeholder submissions, by the Appraisal Committee. The task of the Appraisal Committee is to weigh the evidence and judge whether it would be appropriate to make the technology available within the NHS. For sponsored technologies, such as pharmaceuticals and medical devices (that have been the main focus of the NICE’s work to date), there are two possible approaches to the assessment process. One is to request the 20 manufacturer of the technology (the sponsor) to conduct the required collection and analysis of the evidence and make a submission to the health care payer (e.g. the Government). This approach has been used in countries such as Australia and Canada when assessing new pharmaceuticals in respect of value-for-money, with a view to inclusion in government formularies. A second approach would be for the health care payer to commission an academic group to conduct the assessment. NICE has adopted a hybrid of these two approaches in that it commissions an academic centre to carry out an independent review of the technology and also invites submissions from stakeholders, such as the pharmaceutical industry. Furthermore, the academic group contracted to carry out an assessment is invited to comment on the submission of the manufacturer. There are advantages and disadvantages to the hybrid approach to HTA. On the plus side the Appraisal Committee will see two analyses of the evidence, each conducted independently. Some of the Appraisal Committee members felt that the disagreements which inevitably arise in connection with the results and conclusions - particularly those related to cost-effectiveness models - could be very instructive in that, before agreement is reached, clarification is required on whether it is the differences in model structure, parameters values or other assumptions that produce the different results. To the credit of NICE and the appraisal process, it is evident that when there is conflicting evidence, the goal of the Appraisal Committee is to better understand why the differences exist and not to dismiss either the technical assessment report or the stakeholder submissions. A natural consequence of this is a further round of analyses, sometimes conducted in-house by NICE technical staff, with the aim of integrating and/or adapting one of the models presented. The main disadvantage of the current hybrid process of HTA at NICE is an unnecessary duplication of effort. Because the manufacturer and the academic group work in apparent isolation, difficulties arising from conflicting views are resolved only late in the appraisal process. A more efficient approach might be to provide a forum for an early, on-going exchange of information and ideas with respect, for example, to issues of relevant comparators, clinical pathways for an economic model and parameter values. In this integrated approach the goal would be to produce one analysis from the Technical Assessment Group, which includes the relevant evidence and the manufacturer’s input. Unresolved disagreements on the key aspects of a model or its parameters could then be identified as candidates for a sensitivity analysis to be presented to the Appraisal Committee. Recommendation 4. For drug and device technologies where a sponsor exists, the current hybrid process of HTA used by NICE may give rise to unnecessary duplication of effort. NICE may wish to consider using different approaches depending on the subject of the appraisal. The Review Team suggests that an effort be made to ensure that the Appraisal Committee is presented with a single set of analyses produced by the Technical Assessment Group that incorporates consultation with and input from the manufacturer(s). 21 3.2.2 Topic selection and start of appraisal process The first stage of the process is the selection of topics for NICE Guidance by the Department of Health. A flow chart outlining the current approach (as of January 2003) to topic selection is illustrated in Annex 4. Ministers select technologies for appraisal based on advice from the Advisory Committee on Topic Selection . The criteria for topic selection are given in Box 3. NICE combines reviews of individual new medicines with reviews of classes of medicines. This is in contrast to most other countries that review and assess individual medicines coming onto the market before making a reimbursement decision. Not every new medicine that is licensed in the UK (and which can therefore be prescribed on the NHS unless put on the negative/’black’ list) is appraised by NICE. 22 Box 3: Criteria for topic selection 1. Would guidance promote the best possible improvement in patient care given available resources? In particular, are one or more of the following satisfied? a. Does the proposed guidance relate to one of the NHS clinical priority areas, or to other government health-related priorities such as reducing health inequalities? b. Does the proposed guidance address a condition, which is associated with significant disability, morbidity or mortality in the population as a whole or in particular subgroups? and/or c. Does the proposed guidance relate to one or more interventions which could significantly improve patients’ or carers’ quality of life and/or reduce avoidable morbidity or avoidable premature mortality, relative to current standard practice, or if used more extensively or more appropriately would do so? d. Does the proposed guidance relate to one or more interventions which if more extensively used would impact significantly on NHS or other societal resources (financial and other)? e. Does the proposed guidance relate to one or more interventions, which could without detriment to patient care be used more selectively, thus freeing up resources for, use elsewhere in the NHS? 2. Will NICE be able to add value by issuing guidance, taking into account the following factors?: a.Is the evidence base sufficient to develop robust guidance across most or all of the interventions to be covered by the proposed guidance? b.Is there evidence and/or reason to believe that there is or will be, inappropriate practice and/or significant variation in clinical practice and/or variation in access to treatment (between geographical areas or social groups) in the absence of guidance? 3. Would the most appropriate form of guidance consist of an appraisal, a clinical guideline, or a combination of the two, taking into account: a. the availability of an existing clinical guideline from NICE or from another authoritative source for the condition in question? b. the degree of urgency for guidance on any specific intervention for the condition in question? c. the possible complexity of the proposed guidance if formulated as an appraisal? 4. For new interventions, does the balance of advantage for patient care lie with appraisal at time of launch or at some specified future date, taking account of the following factors and the attached checklist? a. The possible impact on uptake or equity of access in the absence of guidance at time of launch. b. The likely robustness of the evidence base at time of launch. c. The prospect of relevant additional data becoming available in the period immediately after launch. d. For surgical and related interventions, whether safety and efficacy have already been assessed (or will be assessed in the near future) by the Interventional Procedures Advisory Committee. Source: Http://www.doh.gov.uk/nice/consultation2002, accessed 30 June 2003 As a next step, a subgroup of the Advisory Committee on Topic Selection makes a further assessment of the possible technology topics that meet the selection criteria and prioritises them. The selected topics need the clearance of the Minister before the programme of work can be decided and the topics are posted on the web.The plan was to announce a new ‘wave’ of topics every six months but this has not occurred consistently. Once a wave of technology topics is selected and announced, project management begins. Each topic is seen as a specific project for which a timeframe is scheduled and managed by a project manager within NICE. As a first step in the project management, the National Coordinating Centre for Health Technology Assessment assigns the topics to a review group and NICE schedules a meeting of the Appraisal Committee to review the Appraisal 23 Consultation Document (effectively the first draft of the Guidance. The assessment and appraisal process is planned within the scheduled timeframe. Following the announcement of the topic and the appointment of the review group, the project manager within NICE identifies consultees. All consultees are stakeholders in the technology being assessed but not all stakeholders are invited to become consultees. A consultee enjoys the privileges of being invited to make a formal submission to NICE on the technology in question, receiving appraisal documentation that is not on the NICE web site, and being able to lodge an appeal against the final NICE guidance. A meeting between consultees, the Technical Assessment Group and the technical lead from NICE takes place prior to the commencement of the assessment review, to further discuss the Scope of the project. The “scoping” exercise is crucial to the success of the appraisal in general and specifically to guiding the Technical Assessment Group on the precise questions which need to be addressed. Feedback to the Review Team from some members of the Technical Assessment Groups indicated that the scoping exercise was not always adequate in that the evidence required and questions to be addressed were not clearly specified. From a review of the NICE web site, it appears that there may be up to an 18- month time lag between the announcement of a topic and the commencement of the assessment process, depending on the Technical Assessment Group’s schedule of work1. As the topics are publicly listed, this offers an opportunity for stakeholder groups to begin to develop submissions. However, it is often not clear before the Scope for a topic is decided what particular questions are going to be considered. NICE technical staff develops the first draft of the Scope for each topic. The Technical Lead in NICE who develops the Scope has continuing responsibility for that technology or project and, as a result, significant ownership of that particular Guidance. The Review Team identified the following issues: • Given the time gaps at the beginning of the assessment process (i.e. between announcement of the topic and commencement of the review), the late deadline for stakeholder submissions puts unreasonable time pressures on the Technical Assessment Groups at the end of the assessment process. If the scope and protocol could be defined effectively in a relatively early phase, there is no reason to have stakeholder submissions delayed to ten weeks before the end of the assessment period. The current system of relatively late stakeholder submissions is also prone to strategic ‘gaming’ by virtue of a stakeholder being able to produce new evidence late in the process. • An alternative approach would be to adopt a legal analogy of ‘disclosure of evidence’ at the stage of finalising the Scope for a technology appraisal. The purpose of this would be to declare currently 1 For example, statins are currently [as of 1 July 2003] listed on the website as a new topic, with the Scope to be finalised in January 2004 and the Guidance due to be issued in May 2005. Similarly, donepezil is currently listed as a technology to be assessed, with the Appraisal Committee meeting listed as due in October 2004. 24 available information and importantly, to identify any anticipated information that is likely to become available later in the assessment and appraisal process. This would reduce the opportunity for strategically ‘gaming’ the process by late provision of data. Given the number of jurisdictions that now require compilation of clinical and economic data to support reimbursement decisions, pharmaceutical manufacturers in particular should be well placed to meet such a disclosure request. Recommendations 5. The consultees’ meeting should become a formal ‘Preliminary Exchange of Evidence’, at which time all stakeholders should be asked to provide details of what they propose to submit. Only in extraordinary circumstances should the Technical Assessment Group, accept to include as part of the review information over and above that declared for submission by the stakeholders at this meeting. 6. More attention needs to be paid to the important task of setting the Scope of the appraisal. The general goal is to give timely and comprehensive guidance to the Technical Assessment Groups on the question(s) to be addressed by the Appraisal Committee, thereby reducing the risk of mismatch between the assessment report and the needs of the Appraisal Committee. 7. Stakeholder submissions should be required to be lodged as soon as possible after the start of the assessment process. 3.2.3 Assessment and appraisal process The key document in the assessment process is the technical assessment report prepared by the Technology Assessment Groups based in the academic centres. A Technical assessment report is a systematic review of clinical and economic evidence relevant to the Scope for each topic, and it may also include an economic evaluation prepared by the Technical Assessment Group, as well as a critical appraisal of stakeholder submissions. The Technical Assessment Group starts the assessment process by defining the protocol for the assessment report. The report, which is to be completed within 30 weeks, includes: a systematic review of the clinical evidence; an assessment of the economic evidence with or without the development of an economic model and, in most cases, an assessment of the stakeholder submissions. Further aspects of the technical assessment report are discussed in section 3.3.1. Once the technical assessment report is completed, two versions are provided to NICE (see below for further details). The process then becomes one of appraisal and consultation. The key steps are listed in Box 4. 25 Box 4. Summary of appraisal process The technical assessment report is circulated for comment to consultees. The NICE Technical Lead prepares an overview document for the Appraisal Committee; this document summarises the evidence, the comments of the consultees and the key issues for the Appraisal Committee. The technical assessment report is used as the basis for drafting the body of the Appraisal Consultation Document (by the NICE Technical Lead for each technology, in consultation with the Appraisal Committee Chair). The Appraisal Committee meets to consider the evidence and then finalises the Appraisal Consultation Document. The appraisal consultation document is circulated and published for comments. The appraisal consultation document is redrafted by the NICE Technical Lead in consultation with the Appraisal Committee Chair to take any comments into account. It is then considered by the Appraisal Committee as a Final Appraisal Determination. The final appraisal determination is published for a 15-day period, during which consultees can appeal. Any appeals are heard and resolved, including (where necessary) referral back to the Appraisal Committee. The final appraisal determination becomes draft Guidance and is submitted for approval to the Guidance Executive2 within NICE. The time from completion of the technical assessment report to approval of the Guidance is approximately 5 months. The Review Team identified the following matters in relation to the assessment and appraisal processes: • NICE has been very effective in taking on the challenges of developing a system that meets multiple needs: timely production of Guidance that addresses important questions related to improving clinical care with the NHS. • In doing so, points of stress have inevitably been introduced into the system, the most obvious of which is the timeframe. • For the last part of the appraisal and consultation process in particular (from completion of the technical assessment report issuance of the Guidance) the timeframe is structured to allow incorporation of new information without necessarily allowing time for it to be reviewed and critically appraised. • Relatively little emphasis seemed to be placed by the Technical Assessment Groups on protocol development. It was not clear what strategies they had adopted to ensure that there was appropriate clinical input into the development of the protocol, although the most recent version of the Annex to the contracts requires that theTechnical Assessment Groups ensure adequate clinical input. Difficulties at the protocol development stage may be contributing to problems related to the appropriateness and usefulness of assessment reports. • Interaction between the Technical Assessment Group and the Appraisal Committee appears to be somewhat limited. Although the authors of the Technical assessment report attend the first Appraisal 2 The Guidance Executive is the group of staff to whom the Board of NICE has delegated the authority to approve and issue Guidance on behalf of the Institute. 26 Committee meeting, they are not expected to present their reports, nor are they expected to attend the subsequent meetings. Furthermore, under the current contractual structures they are not necessarily expected to review additional data that is submitted unless this is stated explicitly in the contract. • The effect of this delineation of responsibility is that the Technical Lead for each technology within NICE becomes the main repository of the technical aspects of the evidence. They may also have to take on the task of reviewing and assessing any new evidence that is submitted. It is not clear to the Review Team whether this is feasible in terms of the resources available, notwithstanding the newly established Decision Support Unit whose role it may be to assist in this process.3 • Effective document management is a challenge for NICE that has received special attention over the last 12 months. NICE has adopted a ‘paperless’ office approach, and thus all documents are kept electronically. Significant effort is being put into making sure this meets the administrative demands of the Guidance development process. One of the challenges of the rapidly evolving system in this particular area is to ensure that not only current material is managed appropriately, but also that archiving of previous work is optimal. This is particularly important in relation to possible legal challenges that might arise. Recommendations 8. The timeframe for the assessment and appraisal process should be reviewed so that the current time pressures at the end of the assessment report period and early appraisal period are reduced, and more time is allowed for critical evaluation of the consultees’ comments. This does not necessarily mean that the overall timeline should be increased. 9. The procedures for document management need to be carefully considered. This may include keeping some hard copy files of key documents for each Guidance, as well as the electronic files. 10. NICE should improve interaction between the Technical Assessment Groups, the NICE technical staff and the Appraisal Committee throughout the assessment and appraisal process. This could include, for example, the continued involvement of the Technical Assessment Group as technical experts after the initial review of the appraisal consultation document. 3 The Decision Support Unit is a newly developed academic consortia, contracted directly to NICE. Its role is to provide additional technical support and advice to the Appraisal Committee and technical analysts, particularly in relation to the appraisal of economic models. 27 3.2.4 The functions of the Appraisal Committee The Appraisal Committee plays a key role in the process of Guidance development and advises the Board of NICE on: • the use, within the NHS, of any new or established health technology in relation to its clinical- and cost-effectiveness, taking into account the interests of the NHS as a whole; • other matters, on which the Board may, from time to time, seek guidance. The Appraisal Committee currently comprises 51 members. This number will increase to approximately 75 in January 2004. It includes lay representatives, NHS clinicians and managers, academic experts and industry representatives (as described in the most recent draft of the revised Guide to the Technology Appraisal Process). The Board appoints members for three-year terms, normally following a process of public advertisement. The membership currently includes 2 lay representatives and 2 industry representatives who are physicians employed by pharmaceutical companies. In the process of expanding the Appraisal Committee, it is planned to include representatives of other parts of the health care industry. The responsibility of members is described in a recent document used in the recruitment process for new members: “To consider and interpret evidence on the clinical and cost- effectiveness of health technologies (pharmaceuticals, medical devices, surgical procedures, diagnostic procedures and health promotion interventions) prepared by the Institute. They take collective responsibility for the formulation of recommendations to the Institute on the use of technology in the National Health Service in England and Wales.” In order to complete its workload, the original Appraisal Committee has been divided into two to form Appraisal Committees A and B. Currently both have the same Chair who is now almost full-time employed in this function. It has been decided to recruit new Appraisal Committee members and expand to a third committee, commencing in January 2004. One person will chair two of the three committees; one of the current Vice-Chairs will chair the third Committee. The Chairs will have cross-membership of all three Committees. Members are reimbursed for travel, accommodation and out-of-pocket expenses but receive no fees or payment for attending meetings of the Committees. In addition, members may apply for coverage of the costs of employing a locum (e.g. in the case of general practitioners), and the costs for child care. The Committee currently meets every two weeks. At each Committee meeting, the agenda usually comprises a mixture of first reviews of a technology (appraisal consultation documents), final reviews (final appraisal determinations) and matters arising from Appeals that have been referred to the Committee by the Appeal Panel of the Board. In order to complete each 28 agenda, a tightly structured ‘ideal’ time allocation for discussion has been devised. For review of an appraisal consultation document, two members of the Committee are nominated to lead the discussions (Appraisal Team Leads), one from the clinical/disease perspective and one from the technical/evidence perspective. Each Appraisal Team Lead is expected to prepare a presentation summarizing the key issues and evidence to be considered by the Committee. For review of a final appraisal determination, however, Committee members are not expected to lead the discussion; this is done by the Chair. When considering an appraisal consultation document, the Committee hears evidence from stakeholders and technical and clinical experts. These include professional and patient representatives. The consultees for each technology nominate experts. The approach taken for this input is a formal hearing of evidence: experts make presentations in connection with which Committee members may ask questions; the experts then leave the meeting to allow the Committee to consider the matter in camera. During an appraisal consultation document review, the Technical Assessment Group that prepared the Technical assessment report is also available to answer questions, but the Technical Assessment Group does not present its view of the data. Decisions are made primarily on the basis of a consensus of views of the Committee members, although matters may be put to the vote if required. In order to complete its workload, the Appraisal Committee has adopted a process of decision-making that includes post-meeting electronic review and approval of documents. The Review Team noted the following matters in relation to Appraisal Committee processes. • NICE has expended consistent effort in developing a carefully constructed Committee to meet the requirements of technology appraisal and Guidance development. The work undertaken by the Appraisal Committee is substantial and the success is attributable to the commitment and enthusiasm of all Appraisal Committee members and the NICE appraisal team. • In relation to membership of the Committee, the principles governing selection of the members representing the industry would seem to be ambiguous. Given the roles and functions of the Appraisal Committee and NICE, the presence of NHS clinicians and managers, academic experts and lay representatives meets the needs of the Appraisal Committee for forming judgements, making decisions and developing Guidance. Views have been expressed that its members who are employees of the pharmaceutical industry provide significant expertise to the Appraisal Committee. The Review Team considers that this input could be more appropriately obtained through the stakeholder submission process. Furthermore, the nature of the evidence considered by the Appraisal Committee at present includes commercial-in-confidence data. Apart from potential problems arising from commercial and personal conflicts of interest, functional problems 29 are bound to arise in connection with reviewing the confidential data that is required during the process of developing guidance. • Over the last four years, during the development of the Appraisal Committee, the role of the Chair has been pivotal. The Chair has had the usual task of facilitating the discussions and completing the agenda. In addition, the Chair has carried the major responsibility of being thoroughly informed about the technical issues related to each appraisal in order to guide the Committee in its deliberations. Although Committee members act as team leads in the initial discussions that result in the appraisal consultation document, the Chair has generally led the discussions at subsequent stages of the Guidance development for each technology. • While this leadership has been of great value during the establishment and development phase of the Appraisal Committee, the Review Team has some concerns about its long-term sustainability. These concerns include the fact that the corporate memory for decision-making rests primarily with the Chair so that the process is highly dependent on the continuing involvement of a single individual. • An alternative approach of asking other Committee members to take on a more intensive role would have obvious time and workload implications for the members that would need to be carefully considered. • Variable attendance by members and the professional composition of the quorum may be an issue (based on a review of attendance in the Minutes of the Committee). Workload outside the Committee was mentioned as a major contributing factor to this matter. The Review Team was made aware of a proposal to open the Committee meetings to the public with the aim of improving transparency. The Review Team is of the opinion that such a step would be linked to significant risks and could potentially lead to inefficient and inappropriate decisions. To ensure adequately critical review and discussion of the evidence, the Committee needs to retain the option to conduct in private at least part of the discussions leading to the formation of judgements and decisions about each technology appraisal. Recommendations 11. Given that a third Appraisal Committee is being introduced, NICE should take the opportunity to consider how to sustain the high quality performance of the Appraisal Committee. An issue that needs particular consideration is the question of consistency in decision- making across the three committees. NICE is already giving this question careful attention. Two possibilities to consider are: 1) having three individual Chairs and one common Vice-Chair; and 2) having the three Committees assess different types of technologies (e.g. diagnostic procedures for one and specific classes of pharmaceuticals for the other two). 30 12. It should be made clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than on the representation of particular interests. Although there is a need to ensure that manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. 13. The role of the Chair may need to be further refined. While it may be necessary for the Chair to continue to take an active role in leading the discussion on many items, the possible risks associated with this need to be carefully considered. Asking members to play a greater role in the ongoing review of a technology would allow the Chair to facilitate rather than lead the discussion, although the increased workload that would result for the Committee members would need to be assessed. 14. NICE should assess whether the overall sustainability of the process and the functioning of the Appraisal Committee could be improved by the introduction of some type of reimbursement for members’ time. 4 15. The induction and training process for Appraisal Committee members needs to be further developed to include, where appropriate, enhancing skills in the critical appraisal of clinical evidence and economic evaluation. 16. The Appraisal Committee procedures should be modified to ensure that a clear statement of what the Committee approves is recorded on the same day. 3.2.5 The Appraisal Committee approach to decision-making In order to assess the way in which NICE decisions are made, the Review Team considered the various documents available for each of the technologies, and identified the key recommendations in each. These recommendations are summarized in Annex 5. In addition, the Review Team compared the recommendations in the final Guidance with recommendations made about the same technologies by other authorities, including the Pharmaceutical Benefits Advisory Committee, Australia, the Canadian Coordinating Office of Health Technology Assessment, the Netherlands Reimbursement Authority and the Italian Health Authority (Annex 6). Not all decisions could be compared to all authorities. The Review Team noted the following issues with regard to the decision- making process of NICE and the Appraisal Committee. • NICE is in the process of developing explicit criteria for use in decision- making that are consistent with the broad principles that were defined with the establishment of NICE. This laudable objective represents a significant development internationally and may lead to important advances in knowledge within the field of HTA, particularly as the 4 There is precedent for this already; both the Chair and Vice-Chair currently have their time costs reimbursed to their institutions (at 90% and 50% respectively). 31 criteria include consideration of methods to define community values (via the Citizens’ Council) as well as methodological research into equity-efficiency trade-offs. • To date the criteria have not been explicit. Consideration has been given to the clinical and economic evidence and it appears that Committee members also pay significant attention to patient and professional group opinions. It is not yet clear how heavily additional considerations, such as equity of access to technologies, the rule of rescue, or total budget impact weigh with the Committee. • One particular criterion about which there is confusion is the use of a threshold or benchmark for value for money. On the one hand, it appears that there may be a threshold (£30000/QALY) being used informally when considerating technologies and this figure was mentioned by almost everyone the Review Team consulted. On the other hand, there have been statements by NICE that the Committee is not using a threshold, and some of the Committee members publicly take exception to the idea that a threshold is an appropriate approach to decision-making. • The sample of technology appraisals reviewed suggest that there may be a shift from restrictive or negative recommendations about the use of the technology in the appraisal consultation document to less conservative recommendations in the final Guidance. Given the current structure of the assessment process, where an objective independent case is the first prepared by the Technical Assessment Group, this shift, if confirmed as a consistent finding in other appraisals, would be a logical consequence of the current sequence of input of evidence into the decision-making process. • Benchmarking the recommendations of NICE with those of other countries is not straightforward, owing to differences in reimbursement systems from country to country. For the three appraisals that the Team reviewed and compared, the NICE recommendations seemed less conservative than those made by other authorities but this obviously cannot be generalised. • Currently, the Committee does not appear to seek legal advice in the course of the meetings. In other comparable decision-making bodies, for example, it is customary to have a lawyer present to provide advice on the interpretation of decisions and legal structures.. Recommendations 17. NICE and the Appraisal Committee should continue their process to develop a decision-making system that encourages articulation of the grounds for a particular recommendation, including specification of the weight that is placed on clinical evidence, economic evidence and other factors, such as equity and social values. 32 18. As part of the process of articulating the criteria for decision-making, NICE must resolve the confusion related to the use of a value-for- money threshold. If a threshold is to be used as the basis for recommendations, it needs to be specified and justified for reasons of transparency. 19. The Appraisal Committee may wish to consider having a legal advisor present during meetings, particularly those involving matters referred to the Committee by the Appeal Board. 3.2.6 Appeals The process for appealing a final appraisal determination is well established. There is a 15-day period after the publication of the final appraisal determination in which consultees may appeal. Only consultees can appeal. Appeals are heard by the Appeal Board, which is chaired by the Chair of the Board of NICE. There are three grounds on which a final appraisal determination may be appealed: • that the Institute has failed to act fairly and in accordance with its published procedures, • that the final appraisal determination is perverse in light of the evidence submitted, • that the Institute has exceeded its powers. If an appeal is upheld, it may be referred back to the Committee for review of the Guidance. From documentation from the Appeal Board and from the NICE web site, it appears that since the start of the Appeal programme until March 2003, NICE has received 26 appeals, 23 of which were considered. This means that 38% of the FADs have been appealed. 20 of the 23 appeals were lodged by the industry. Some appeals have more than one appellant. The appeal process can add significantly to the time taken to issue final Guidance, which can lead to problems with the availability of a technology. The Review Team noted the following matters in relation to the appeals. • The Appraisal Committee, when agreeing on the final appraisal determination, clearly addresses the possibility of an appeal being lodged. • Concerns had been raised by the Health Select Committee about the independence of the Appeal process. However, in many jurisdictions, the first stage of any appeal process is the submission of the appeal to the organization that made the decision. The current structure for appeals used by NICE is an example of this model. Adding the possibility of a second stage of appeal (prior to formal judicial review) would carry risks and benefits. The main risk relates to resource and 33 time implications of an increase in the number of appeals and the length of the appeal process. Recommendation 20. In view of the considerable number of appeals lodged and their time and resource implications, thought needs to be given to how to reduce the number of appeals and the length of the appeal process. The further development of the appeal process would help to enhance the quality and the transparency of the appraisal programme. 3.2.7 Budget impact and research recommendations The final Guidance for any technology contains sections on budget impact, recommendations for auditing compliance with the Guidance and planned dates for the review of the Guidance. Budget impact is not a decision criterion in the development of NICE Guidance. The estimates of budget impact are initially provided from the technical assessment report, and may be refined during the appraisal process. The recommendations for audit are written by a clinical audit specialist unit, after the literature on audit of the specific topic has been reviewed and after consultation with appropriate health professionals. The final Guidance also contains recommendations for further research. These recommendations appear to be developed by the Appraisal Committee during the review of the appraisal consultation document and the final appraisal determination, with significant input from the Committee Chair and the NICE Technical Leads. Review dates for each Guidance are currently standardized as either three years after issue, or one year in the case of a rapidly evolving technology. The Review Team noted the following matters in relation to these sections of Guidance. • Much of the recently published criticism of NICE Guidance relates to inadequate consideration of budget impact at local level, particularly in regard to the local opportunity cost of implementing new Guidance. Budget impact sections of Guidance tend to be fairly limited, often providing only an estimate of the national cost of a technology. • The Technical Assessment Groups provide an estimate of budget impact as part of the Technical assessment report. However, these estimates are of national budget impact rather than local budget impact, which is the information needed by Trust managers in connection with the implementation of Guidance. • The Research Required section can assist in generating new information on the effectiveness of the medicines under appraisal, as well as in targeting the medicine to those patients most likely to benefit. 34 • The first rounds of review to update published Guidance are about to commence. From discussions with NICE staff, it appears that NICE is already developing a process for handling the workload connected to reviewing and updating Guidance, by determining the types of review needed and tailoring them to the availability of new evidence. Recommendations 21. Although budget impact is not a consideration in making recommendations, it is important to develop methods for budget impact modelling that would enable NICE to provide more detailed information on the implementation costs to the local authorities. This could be a task for the technical analysts in NICE. The advantages of doing so include not only the provision of useful advice to the Trusts but also the avoidance of duplication of effort by the Trusts in making their budget analyses for implementing the new technology. 22. NICE should further develop the Research Required section of the Guidance and link it specifically with the Guidance review process. This would help to obtain the additional clinical evidence needed to ensure full understanding of the benefit of a technology. The Guidance review process would be an ideal way to stimulate the generation of such evidence. 23. NICE should adopt more flexible timeframes for reviewing existing Guidance not only to ensure that the review answers a specific question, but also to assist in magaging workload. One approach might be for the set review dates to be dependent dependent on the emergence of new evidence or significant changes in existing evidence. 3.3 The quality of the science of the assessment and appraisal process 3.3.1 The technical assessment reports The technical assessment reports are prepared by six independent Technical Assessment Groups contracted by the National Coordinating Centre on Health Technology Assessment from academic centres. As noted above, the assessment reports submitted to NICE comprise a systematic review of the clinical evidence and a review of the economic evidence about the technology. The assessment reports are eventually published as monographs in the NCCHTA monograph series. The Technical Assessment Groups are required to provide two versions of the technical assessment report to NICE: an ‘unstripped’ version that contains all the evidence and is the basis for the Appraisal Committee’s consideration, and a ‘stripped’ version from which all confidential data are removed. The ‘stripped’ version is circulated to consultees for comments prior to the 35 Committee’s considering the appraisal consultation document and is also put on the NICE web site for general comments as soon as it is finalised. The Review Team considered the four selected technical assessment reports in detail and, since the system is in rapid evolution, also briefly considered other,, more recent, technical assessment reports available on the NICE web site. The Review Team noted the following issues. • NICE, through its process of working with leading academic centres, is in the process of developing standards for HTA that are leading to methodological innovations and developments that are likely to have a major influence on how HTA is carried out internationally. • The pace of development is extremely rapid. Although the four TARs reviewed by the Team were published within the 12 months prior to the review, it is clear that these cannot be seen as a representative sample since the work was carried out in 2001 and 2002, and major changes occurred in the approach taken to assessment reports subsequent to the publication of the four selected TARs. • The Appraisal Committee members generally recognize that the TARs are of high quality and provide the essential information required by the Committee. • Given the rapidly changing system, it is not surprising that the sample of reports reviewed by the Team varied in method and approach. For example, not all of the reports contained a detailed critique of the stakeholder submissions; the systematic reviews in the reports were carried out in different ways; methods used for data analysis varied leading to differences in the way results were presented. NICE (and NCCHTA) are currently developing standard templates and specifications to reduce the amount of variability where appropriate, to ensure that the reports fit the purposes of the Appraisal Committee. This will include more detailed specifications of the types of evaluations and analyses to be included in a technical assessment report, as well as specifications for the presentation of the information to the Appraisal Committee. • There have been some difficulties in ensuring that all academic centres have the appropriate combination of expertise to provide reports that fully integrate the clinical and economic evidence in a way that meets the needs of the Appraisal Committee. Recent modifications to the contracts with the academic centres have been instituted to address this issue. • In some instances, the technical assessment reports have not necessarily addressed policy questions of relevance to NICE. As noted in Section 3.2.2, aspects of the assessment process that may have contributed to this problem were uncertainties in the development of the Scope for the technology and difficulties in communication between the academic centre and NICE because of the contractual arrangements between academic centres and NCCHTA. 36 • Although NICE and the Appraisal Committee attach importance to the submissions from stakeholders, the Technical Assessment Groups were inconsistent in their approach to the inclusion of stakeholders’ data in the TARs. It is clear to the Review Team that the contractual arrangements contributed to this difficulty. There is also the question of the need for academic groups to be able to use the NICE technology assessment process to generate independent academic products. A more appropriate approach might be for academic centres to regard NICE work as contract research that allows them to carry out other projects through the expertise they acquire in the process. • As mentioned in Section 3.2.2, the quality of reports may be compromised by the late arrival of stakeholder submissions, particularly manufacturers’ submissions. • The preparation of stripped versions of the technical assessment reports increases workload unnecessarily at a critical time in the process and may lead to errors. The requirement for stripped reports would be removed if the option to submit commercial-in-confidence data were discontinued. In addition to the matters noted above, the Review Team identified a number of possible reasons for these difficulties. The first issue relates to the structure of the current contractual arrangement. The contract is between the Technical Assessment Group and the NCCHTA rather than directly with NICE. The reasons for this arrangement are historical. The NCCHTA structure was well-established prior to NICE and was seen as the appropriate body to undertake the work needed by NICE in its Guidance programme. However, there are significant problems with this arrangement. The right of quality control rests with NCCHTA; it is not clear whether the specified personnel included senior staff as well as the research fellows who often do the majority of work on the reviews (based on the authorship and responsibilities described in each technical assessment report); the contract fee per assessment did not (until a recent contract review) cover the full cost of preparing an assessment; there are still dual outputs required – the technical assessment report and a HTA monograph, which serve significantly different purposes. There is a need for this three-way arrangement to be clarified so that the primary client, NICE, is defined and the needs of this client recognized in the terms of the contract. The second issue relates to the siting of the assessment process in academic centres. This arrangement is used in many countries and may be the best way to obtain an independent report. However, it leads to the need for the academics involved to ensure that their work on assessments is recognized for academic purposes. This recognition includes the publication of peer- reviewed articles. There is obviously tension between the need to produce TARs that contain methodological advances that are independently publishable and, at the same time, sufficiently robust to meet the needs of the Appraisal Committee. This tension also compromises the transparency of the process. For reasons of intellectual property, there have been problems in making available full copies of the economic models developed by the assessment groups. 37 Thirdly, although the systems for communication and interaction among the various groups involved in the assessment process are being revised and improved, there have been some difficulties in the interaction between the NICE staff responsible for the project (i.e. an individual Guidance), the Technical Assessment Group, the Appraisal Committee and the external experts. Recommendations 24. The contractual arrangement between the Technical Assessment Groups and the NCCHTA should be revised to recognize NICE as the primary client for the assessment reports. 25. A handbook of standard methods for the assessment reports should be developed, after consultation between NICE staff, the Appraisal Committee and the Technical Assessment Groups. This handbook, which should be regularly updated, should also be used as the basis for training new reviewers and new NICE staff. 26. A detailed template for the reports should be developed, including standardized data presentation and summary material. This would facilitate review of the information by the Appraisal Committee. 3.3.2 The consultees submissions and comments There are three groups of submissions that feed into the appraisal consultation document: submissions from manufacturers, submissions from patient groups and submissions from professional groups. Consultees’ submissions can take a variety of forms, and can be submitted by consultees designated for a given technology. The submissions from manufacturers vary, depending on the technology. In the case of pharmaceuticals, although there is technically a 50-page limit on the submission, large volumes of clinical trial data and economic models are often provided. In the case of devices and other medical technologies, the submissions are usually less complex. Submissions from health professionals range from complete literature reviews to short comments on the proposed technologies. Similarly, patients’ submissions also seem to vary; from letters that are simple testimonials, to complex survey data describing the impact of a particular disease on quality of life. Consultees and the general public also have the right to provide additional input after the appraisal consultation document is published. This input can be substantial and seems to vary from handwritten notes to re-analyses of economic evaluations. In the current system, the post-appraisal consultation document comments are dealt with in the short time period between the publication of the document and the Appraisal Committee’s consideration of the final appraisal determination. In general, this time period is approximately eight weeks, although NICE has the option to extend the time if critical new 38 information is submitted that needs detailed evaluation. If they do so, however, it may mean that the timeframe for the overall Appraisal is substantially prolonged. The team noted the following matters. • It is clear that NICE and its Appraisal Committee have an strong commitment to ensuring that their process for obtaining input from stakeholders into the Guidance development process is inclusive and as flexible as possible. In particular, the team noted the weight and value given to patient group submissions, including the financial support available to these groups to assist them to present their view to the Appraisal Committee. • The consultation process is important, but there does not seem to be a systematic approach to responding to comments received between the publication of the appraisal consultation document and the final appraisal determination. In general, it seems that the comments are tabulated and provided to the Committee without interpretation. Lack of a systematic response system could lead to the consultation process being perceived as not receiving the attention it deserves, which is clearly not the intention of NICE. • The potential provision of substantial evidence after the appraisal consultation document is published places additional pressures on the system, particularly on the technical staff of NICE and the Chair of the Committee, who have a major role in ensuring that the additional evidence is adequately presented to and considered by the Appraisal Committee as a whole. Recommendations 27. On the basis of the experience to date, NICE should consider what aspects of the patient and professional submissions are most useful, and develop standards for the content of these types of submissions. 28. NICE should review the process for assessing comments from consultees and others (including the general public) that are submitted after the appraisal consultation document is drafted. NICE should determine the most appropriate way of responding to these inputs and assess their value in the decision-making process. 39 4. Conclusions The conclusion of the Review Team is that, in only four years, NICE has developed a well-deserved reputation for innovation and methodological development that represent an important model for technology appraisals internationally. It is important to emphasize that NICE functions in an environment where substantial changes are taking place in the way in which the NHS operates, and where additional resources are being invested in the health services. This facilitates the achievement of the NICE objectives of improving the quality of care throughout the health service and the uptake of new, cost-effective medicines and technologies. Achievements that are particularly valuable include: the transparency surrounding the process of technology assessment; the intensive participation of different stakeholders and the inclusiveness of the approaches taken; the commitment to using the best available evidence for decision-making; the commitment of the technical and management staff; and the dedication of the Appraisal Team and the Appraisal Committee members. All of these form the cornerstones of an organization that continues to invest in quality development. The major task that the Review Team was asked to carry out was to assess the processes and methodologies used in the technology appraisal with regard to their scientific rigor and credibility. The Review Team noted that, with respect to this task, NICE has embarked on a series of internal reviews of processes and methodologies and it is likely that a number of the issues discussed in this report are also being considered in these internal reviews. Overall, the Review Team was impressed by the commitment to the use of rigorous methodology throughout the process, starting with the use of academic centres of excellence for providing independent appraisals that are supplemented by additional evidence that is handled by the technical teams in NICE. The published technology appraisals are already being used as international benchmarks, which is an obvious recognition of their credibility. In order to consolidate the operation of the technology appraisal programme the Review Team identified a number of issues for further development and improvement. A number of key recommendations are worthy of specific mention here, namely: • To reconcile the inherent contradiction between the Institute’s key principle of transparency and the acceptance of material that is designated as confidential. The Review Team welcomes the steps that NICE has already taken to date, and encourages NICE to use its leverage to advance transparency in this area. This would be of benefit internationally. • To revise the contractual arrangements for the development and production of assessment reports so that they fully meet the needs of NICE and its Appraisal Committee.. 40 • To consider ways to reduce unnecessary duplication of effort in the assessment phase. Efforts should be made to ensure that the Appraisal Committee is presented with a single set of analyses, produced by the Technical Assessment Group, that incorporates consultation with and input from the manufacturer(s). • To ensure the collection of relevant information from all stakeholders as early as possible in the assessment process.. • To increase the exchange of information and interaction – at all stages of the process - between the Technical Assessment Group, the Appraisal Committee and the NICE Appraisal Team. • To developing a handbook on the preparation of assessment reports. This could be a product of the current review of methods and process. • To make clear that membership of the Appraisal Committee is based on skills in and knowledge about evidence appraisal and judgement rather than on representation of particular interests. Although there is a need to ensure that the manufacturers’ views are taken into consideration, this should not be through membership of the Appraisal Committee but through the consultation process. • To continue to work on the difficult but important task of developing clear criteria in order to obtain optimal interaction between the ethical and social values and the scientific evidence, for use by the Appraisal Committee and NICE in decision-making. In providing these recommendations, the Review Team notes that there are some broader issues that NICE (and other organizations seeking to carry out a similar task) may wish to consider. Transparency of process and decision- making comes with a price, and the price includes time, resources and perhaps challenges to the way decisions can be made. The trade-off between transparency with its credibility, and efficiency in decision-making is a challenge that all authorities such as NICE will need to continue to consider carefully. In addition, although the Review Team is aware that budget impact is not part of the remit of NICE in its Guidance development, fundamentally, the questions of local opportunity cost are integrally related to decision-making and help to determine the value-for-money benchmarks that a decision- making body considers. The main outcome that will be the measure of the ultimate success of an organization such as NICE will be the implementation of the Guidance and its impact on the heath outcomes of the community it is seeking to serve. These recommendations are to be seen as an effort to further enhance the operations of NICE and to assist organizations with similar responsibilities in other countries to deal with their difficulties and meet their expectations.. 41 References 1. Rawlins M. In pursuit of quality: the National Institute for Clinical Excellence. Lancet; 353:1079-1082 (1999). 2. House of Commons Health Committee Review of the National Institute of Clinical Excellence (NICE). London, June 2002. 3. Davies E, Littlejohns P. Views of Directors of Public Health about NICE Appraisal Guidance: results of a postal survey. National Institute for Clinical Excellence. J Public Health Med; 24:319-325 (2002). 4. Birch S, Gafni A. On being NICE in the UK: guidelines for technology appraisal for the NHS in England and Wales. Health Econ; 11:185-191 (2002). 5. Cookson R, McDaid D, Maynard A. Wrong SIGN, NICE mess: is national guidance distorting allocation of resources? BMJ,:323; 743-5 (2001). 6. Poole-Wilson PA. NICE: guessing is not guidance. Heart, 87:510512 (2002). 7. Freemantle N, Bloor K, Eastaugh J. A fair innings for NICE? Pharmacoeconomics; 20:389-391 (2002). 8. Raftery J. NICE: faster access to modern treatments? Analysis of guidance on health technologies. BMJ, 323:1300-3 (2001). 9. Madhok R, Taylor A, Soltani H, Ireland K. NICE and LICE: lessons from a health district. National Institute for Clinical Excellence. Local institutes of clinical excellence. J Public Health Med, 24:2-5 (2002). 10. Drummond M. Should commercial-in-confidence data be used by decision makers when making assessment of cost-effectiveness? Applied Health Economics and Health Policy, 2:3-4 (2002). 11. Dowie J, Claxton K, Sculpher M, Drummond M. NICE (and the NHS) – quo vadis? In: Health Care Priority Setting. Implications for health inequalities. Proceedings of a meeting of the Health Equity Network. Ed: Adam Oliver. The Nuffield Trust. UK. 2003. 12. Gafni A, Birch S. NICE methodological guidelines and decision making in the National Health Service in England and Wales. Pharmacoeconomics; 21:149-157 (2003). 13. Another NICE mess. Editorial. Lancet Oncology; 3:385 (2002). 14. Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute for Clinical Excellence. Lancet; 360:711-15 (2002). 15. Littlejohns P, Barnett D, Longson C. The cancer technology appraisal programme of the UK’s National Institute for Clinical Excellence. Lancet Oncology; 4: 242-50 (2003). 42 Annex 1 Persons interviewed (in person or by teleconference) Dr Jane Adam, NICE Appraisal Committee and Radiologist, St George's Hospital, London Professor Ron Akehurst, NICE Appraisal Committee and Dean of School of Health and Related Research, University of Sheffield Dr Anne-Marie Bagnall, Research Fellow, NHS Centre for Reviews and Dissemination, University of York Professor David Barnett, Chair, NICE Appraisal Committee Professor Michael Barnett, Professor of Transplantation Oncology, Department of Medical Oncology, St Bartholomew's Hospital, London Dr Pelham Barton, Lecturer in Mathematical Modelling, West Midlands Health Technology Assessment Collaboration, Department of Public Health and Epidemiology, University of Birmingham Mr Neil Betteridge, Head of Public Policy and Campaigning, Arthritis Care Professor Carol Black, President, Royal College of Physicians and Consultant Physician, Royal Free and University College Medical School, London Professor Stirling Bryan, Professor of Health Economics, Health Services Management Centre, University of Birmingham Professor Martin Buxton, NICE Appraisal Committee, Director of Health Economics, Health Economics Research Group, Brunel University Dr Neil Calvert, Research Fellow Sheffield Health Economics Group, School of Health and Related Research, University of Sheffield Dr Karl Claxton, NICE Appraisal Committee and Lecturer in Economics, Department of Economics and Related Research, University of York Mr Andrew Dillon, Chief Executive, NICE Ms Nancy Dixon, Audit Specialist, NICE Professor Jack Dowie, NICE Appraisal Committee and Health Economist, Public Health and Policy Department, London School of Hygiene Ms Sophie Edwards, Chief Executive, Arthritis and Musculoskeletal Alliance, London Dr Dogan Fidan, Health Technology Analyst, NICE Dr Sarah Garner, Health Technology Analyst, NICE Ms Ruth Garside, Research Fellow, Peninsula Technology Assessment Group, University of Exeter 43 Ms Laura Ginnelly, Research Fellow, Health Economics, NHS Centre for Reviews and Dissemination, University of York Professor Tricia Greenhalgh, NICE Appraisal Committee and Professor of Primary Health Care, Primary Care and Population Sciences, University College London Dr Danny Hind, Research Associate, School of Health and Related Research, University of Sheffield Ms Marian Hodges, Editorial Manager, NICE Dr Paresh Jobanputra, Consultant Rheumatologist, West Midlands Health Technology Assessment Collaboration, Department of Public Health and Epidemiology, University of Birmingham Ms Lynn Kerridge, Acting Executive Director, National Coordinating Centre for Health Technology Assessment Dr Carole Longson, Appraisals Programme Director, NICE Dr Alec Miners, Health Technology Analyst, NICE Professor Miranda Mugford, NICE Appraisal Committee and Professor of Health Economics, University of East Anglia Mr David Murray, Technical Team Leader, NICE Ms Judith Paget, NICE Appraisal Committee and Chief Executive, Caerphilly Local Health Board, Wales Ms Suzy Paisley, Managing Director, School of Health and Related Research, University of Sheffield Ms Seren Phillips, Technical Team Leader, NICE Ms Nina Pinwill, Technology Appraisal Project Manager, NICE Professor Sir Michael Rawlins, Chairman, NICE Dr Alison Round, Consultant in Public Health Medicine, Peninsula Technology Assessment Group, University of Exeter & Wessex Institute for Health Research and Development, University of Southampton Mr Miles Scott, NICE Appraisal Committee and Chief Executive, Harrogate Health Care NHS Trust Professor Mark Sculpher, Professor of Health Economics, Centre for Health Economics, University of York Professor Andrew Stevens, Vice-Chair, NICE Appraisal Committee and Professor of Public Health, Department of Public Health and Epidemiology, University of Birmingham Ms Andrea Sutcliffe, Planning and Resources Director, NICE Dr David Taylor, Chief Pharmacist, South London & the Maudsley NHS Trust 44 Mr Adrian Towse, Director, Office of Health Economics, London Professor Norman Waugh, NICE Appraisal Committee, Department of Public Health, University of Aberdeen 45 Annex 2 Documents reviewed 1. NICE documents for each of the four technology appraisals: Scope Protocol Stakeholder submissions Stripped TAR (on the web site) Unstripped TAR (where applicable) Overview [or equivalent] draft Appraisal Consultation Document Slide presentations to Appraisal Committee - Appraisal Committee Lead, NICE TL Appraisal Appraisal Consultation Document (previously PAD) Committee minutes Stakeholder [consultees] responses General public responses draft Final Appraisal Determination Final Appraisal Determination Committee minutes Draft Guidance Final Guidance 2. Other NICE documents Guidance to the technology Appraisal Process Submissions of BNF, DTB to House of Commons Health Select Committee review of NICE NICE responses to BNF and DTB submissions NICE List of personnel and job specifications NICE: A summary of the review of the appraisal process and review of methodology NICE Technology Assessment Report Feedback Form (TARQA) Appraisal Committee: Committee members (description of terms of reference and requirements), June 2002 Declaration of interest on appointment and at meetings: a code of practice for non- executive directors, members of advisory committees and employees 3. Draft documents Scientific and social value judgements, first draft v2, 2003 Draft annual report 2002-2003, July 16, 2003 Guide to technology appraisal process, draft 2003 Guide to the Methods of Technology Appraisal, 3rd draft July 2003 4. Internal NICE documents Review of appeals (July 2003) Technology Assessment Report Feedback form, May 27,2003 Project plans for development of the appraisal consultation document and the final appraisal determination Notes for taking and writing the minutes of an Appraisal Committee meeting, January 2003 Technology appraisal programme: procedure for the sign-off of Appraisal Committee 5. Official documents Statutory Instrument 1999 No.260: the National Institute for Clinical Excellence Regulations 1999; http://www.hmso.gov.uk NICE Corporate plan 2002-2005, September 2002, ISBN 1-84257-221-0 NICE Business plan 2002-2003, September 2002, ISBN 1-84257-220-2 NHS NICE Annual report 2001-2002, September 2002, ISBN 1-84257-211-3 46 Annex 3 NICE processes Original scheme of NICE Guidance development process provided to Review Team Technology selected Ð Technology announced Ð Submissions to NICE (manufacturers, patient and professional groups) Ð Submission provided to academic centres (review commissioned by NHS) Ð Ð Response to report, including peer review Ð NICE technology assessment team review and compilation of material Ð Ð Appraisal committee review (includes consideration of professional groups’ and patients’ groups’ reports) Ð Draft recommendation – development of Appraisal Consultation Document Ð Further comments Ð Interim Guidance (final Appraisal determination) Ð Ð Health authority implementation Ð Feedback and review of Guidance Assessment report Final guidance Evaluation report Appeals 47 Revised description of process What Who (primary) Who (secondary) In-parallel activities Topic selection DOH, Welsh government, committee with NICE member (ACTS) Horizon scanning unit (Bham) Definition of consultees; development of Scope; call for submissions NICE staff: technical lead, project manager Contract for specific topic NCCHTA, TAG Consultees meeting NICE, TAG, Consultees TAR starts TAG Protocol development TAG (30 weeks out) NICE TL Company submissions developed. Stakeholder submissions developed Systematic review commences TAG Industry submissions received and sent to TAGs TAG (10 weeks out) Unstripped TAR, stripped TAR completed, submitted TAG Draft ACD begins (TL-NICE) Stripped TAR Circulates to consultees – responses NICE, consultees Stripped TAR (corrected) on web Î responses TAG, NICE, general public TL, audit specialist ACD drafted, including audit and expanded budget impact Overview prepared TL AC meeting ÎACD finalized AC {AC lead} TL, EL, expert witnesses ACD circulated, published, comments Consultees, general public TL, EL, PM Redraft of ACD Î draft FAD TL, EL Second AC– FAD approved AC TL, EL FAD posted on web for appeal NICE Appeals if submitted NICE appeals Board Appeals upheld – go back to AC for review FAD becomes draft guidance NICE staff Guidance approved Guidance executive, Board Guidance implementation within two months Regional authorities, Trusts Guidance review NICE TAGs? 48 Annex 4 Process of topic selection Topics suggested by policy (National Service Framework), Horizon Scanning unit, others Ð NCCTHA prepares briefing notes Ð Review by Advisory Committee on Topic Selection against criteria Ð No/yes/maybe/not now decision Ð ‘Yes’ referred to Joint Planning subgroup of ACTS Ð Revised list based on priority and capacity Ð Minister of Health review Ð Suggested list Ð Department of Health, Welsh Government for comments Ð Final list Ð Work program – ‘wave’ announced every 6 months 49 Annex 5 Comparison of recommendations in NICE documents Technology Assessment from TAR Recommendation from ACD Recommendation from FAD Final guidance Rheumatoid arthritis: etanercept and infliximab Not clearly stated: ‘ both improve outcomes compared to placebo…no convincing treatment difference between etanercept and methotrexate’a ‘Neither etanercept nor infliximab is recommended for adults with RA except in the context of a controlled long term clinical study’b Etanerecept and infliximab (infliximab only in combination with methotrexate) are recommended options for the treatment of adults who have continuing clinically active RA that has not responded adequately to at least two disease modifying drugs including methotrexate (unless contraindicated)c Etanercept and infliximab (infliximab only in combination with methotrexate) are recommended options for the treatment of adults who have continuing clinically active RA that has not responded adequately to at least two disease modifying drugs including methotrexate (unless contraindicated)fd Chronic Myeloid Leukaemia: Imatinib ‘Based on the limited evidence available, imatinib appears to offer an alternative treatment for CML in accelerated and blast phases. As yet, in the authors’ opinion, there is not enough information about imatinib in chronic phase to draw firm conclusions. Cost –utility estimates for imatinib are particularly sensitive to assumptions about long term survival and may be extremely high.’e Imatinib is recommended for the treatment of patients with accelerated phase CML. There is insufficient evidence on which to recommend the use of imatinib for the routine treatment of chronic or blast phase CML.f Imatinib is recommended as a treatment option for the management of Philadelphia chromosome positive CML in chronic phase in adults who are intolerant of IFN-α therapy. Imatinib is recommended as an option.in accelerated phase or blast crisis provided they have not received imatinib treatment at an earlier stage.g As for FAD with the addition of ‘failed IFN-α treatment’h Schizophrenia: Atypical Antipsychotics ‘Evidence for the effectiveness of the atypical antipsychotics compared to older drugs is in general of poor quality…available evidence suggests that [atypicals] are as effective or more effective.. than typical antipsychotics.i It is recommended that the atypical antipsychotics are considered as the first-line treatment of choice for people with newly diagnosed schizophrenia…j As for ACDk As for FADl Ultrasound device for central venous catherisation ‘Real time2-D US is significantly better than landmark’ m It is recommended that 2-D imaging US guidance should routinely be used when a central venous line is being inserted in an elective or emergency situation.n 2-D imaging US guidance is recommended as the preferred method for insertion of central venous catheters into the internal jugular vein in adults and children in elective situations. The use of US…either electively or in an emergency situation.o As for FADp a TAR Executive summary page 12 (unstripped version); b PAD, page 1; c FAD, page 1; d Section 1.1, Guidance; e Assessment report, executive summary, page 7; f ACD, Sections 1.1 and 1.2; g FAD Sections 1.1 and 1.3; h Guidance, Section 1.1 and 1.3. I Assessment report, executive summary, pages 8, 9; j PAD Section 1.1; k FAD, Section 1.2; l Guidance, Section 1.2; m Assessment report, Executive summary page 2;n ACD Section 1.1; o FAD Section 1.1 and 1.2; p Guidance Sections 1.1, 1.2 50 Annex 6 Comparison of international recommendations Technology Final guidance CCOHTA PBAC Netherlands Italy (Ministry of Health, CUF) Rheumatoid arthritis; etanercept, infliximab E and I (I only in combination with methotrexate) are recommended options for the treatment of adults who have continuing clinically active RA that has not responded adequately to at least two disease modifying drugs including methotrexate (unless contraindicated) Infliximab not assessed for RA. Etanercept: Canadian clinical and economic data comparing methotrexate and etanercept alone versus etanercept in combination with methotrexate would be necessary before informed predictions could be made on its cost-effectiveness in our health care system. While the direct costs of this therapy are high, if etanercept controls the disease more effectively than current treatments, or earlier in the progression of the disease, it may have incremental cost-effectiveness. Infliximab Section 100 listing for the treatment of adult patients with severe active rheumatoid arthritis who have a record of rheumatoid factor positive status, who have failed other treatment regimens. Etanercept Section 100 listing for the treatment of adult patients with severe active rheumatoid arthritis, who have a record of rheumatoid factor positive status and who have failed other treatment options; For the treatment of juvenile patients aged 4 to 17 years with active polyarticular- course juvenile chronic arthritis, who have failed other treatment regimens. Infliximab Adults and children with RA who have failed standard treatment with at least methotrexate at optimal dose. Use is in the context of a Phase 4 study. Etanercept Adult Patients with RA who have failed standard treatment with at least methotrexate at optimal dose. Use is in the context of a Phase 4 study. Infliximab Adults and children with RA who have failed standard treatment with at least methotrexate at optimal dose. Only for hospital use (can be given to outpatient after started in hospital) in the context of the observational ANTARES study. Not available through community pharmacies. Etanercept Patients with RA who have failed standard treatment with at least methotrexate at optimal dose. Only for hospital use (can be given to outpatient after started in hospital) in the context of the observational ANTARES study. Not available through community pharmacies 51 Comparison of international recommendations (continued) Technology Final guidance CCOHTA PBAC Netherlands Italy (Ministry of Health, CUF) Chronic Myeloid Leukaemia: imatinib Imatinib is recommended as a treatment option for the management of Philadelphia chromosome positive CML in chronic phase in adults who are intolerant of, or who have failed IFN-α therapy. Imatinib is recommended as an option .in accelerated phase or blast crisis provided they have not received imatinib treatment at an earlier stage. The advent of imatinib is considered an advance in the treatment of CML, allowing for targeted therapy against inappropriate tyrosine kinase activity. Available preliminary data, although not published in full, appears promising. Section 100 listing for treatment of patients with chronic myeloid leukaemia expressing the Philadelphia chromosome or the transcript, bcr-abl tyrosine kinase. Patients must be in the chronic phase where the use of interferon alfa has failed. Recommended for listing as requested with additional requirement that patients must achieve a major cytogenetic response within 18 months and demonstrate that the response is maintained by testing every 12 months. The PBAC will review new survival data after 1 year, 2 and 5 years. Imatinib may be considered for use in the treatment of patients with chronic myeloid leukaemia in case of failure, intolerance or contraindications to interferon- alpha. Definite data on effectiveness of imatinib in terms of survival data are lacking. This advice that it may be considered for use is only based on 2 Phase 2 studies and indirect comparison with standard therapy comprising hydroxyurea and combination chemotherapy. Imatinib is for the treatment of patients with chronic myeloid leukaemia in case of failure, intolerance or contraindications to interferon-alpha. Only for hospital use. Schizophrenia: Atypical antipsychotics It is recommended that the atypical antipsychotics are considered as the first-line treatment of choice for people with newly diagnosed schizophrenia CCOHTA: The cost-utility analysis demonstrated that risperidone was the dominant strategy compared to haloperidol, haloperidol decanoate or fluphenazine decanoate in hospitalized patients with chronic schizophrenia with moderate symptoms. The estimated cost savings (versus haloperidol) was approximately $6,500 per patient per year while producing 0.04 more QALYs per year. PBAC – currently lists most atypicals as ‘Authority required for the treatment of schizophrenia’ Olanzapine: Patients who failed or are intolerant to classical antipsychotics, and patients with negative signs and symptoms of schizophrenia, or patients who responded with an acute dystonia to other antipsychotic drugs. Atypicals Patients who failed or are intolerant to classical antipsychotics, and patients with negative signs and symptoms of schizophrenia. Risperidone and olanzapine: Restricted to an initial prescription from a specialist together with a therapeutic program. Sources: CCOHTA: etanercept: CCOHTA Issues in emerging health technologies, Issue 8, 1999; imatininb: CCOHTA Emerging drug list 2001; Glennie JL. Pharmacoeconomic evaluations of clozapine in treatment-resistant schizophrenia and risperidone in chronic schizophrenia. Ottawa: Canadian Coordinating Office for Health Technology Assessment (CCOHTA); 1997 Italy – Gazzetta Ufficiale della Republica Italiana Netherlands: www.cvz.nl, accessed 19 July 2003. PBAC web site (http://www.health.gov.au/pbs/listing/pbacrec/index.htm). Note that Price –volume agreement has been agreed as part of listing process for etanercept’ with the total cost to Government not to exceed more that $AUD100million per year (see Ministerial press release dated 30 June on the same web site. WHO REVIEW OF THE TECHNOLOGY APPRAISAL PROGRAMME, NATIONAL INSTITUTE FOR CLINICAL RESEARCH