Manual for the performance evaluation of regulatory authorities seeking the designation as WHO-listed authorities

Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities

Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Manual for the performance evaluation of regulatory authorities seeking the designation as WHO-listed authorities ISBN 978-92-4-007696-9 (electronic version) ISBN 978-92-4-007697-6 (print version) © World Health Organization 2023 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization (http://www.wipo.int/amc/en/mediation/rules/). Suggested citation. Manual for the performance evaluation of regulatory authorities seeking the designation as WHO listed authorities, version 2.0. Geneva: World Health Organization; 2023. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see https://www.who.int/publications/book-orders. To submit requests for commercial use and queries on rights and licensing, see https://www.who.int/copyright. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Design and layout: Harri Aittasalo iii Contents Acknowledgements ................................................................................................................................................................................................................. v Abbreviations .............................................................................................................................................................................................................................vii Code of conduct for WHO experts .............................................................................................................................................................................viii 1. Introduction ............................................................................................................................................................................................................................ 1 2. Preparing for, conducting and reporting on PE activities .......................................................................................................................4 3. Regulatory system.............................................................................................................................................................................................................6 4. Registration and marketing authorization ....................................................................................................................................................... 13 5. Vigilance ................................................................................................................................................................................................................................ 18 6. Market surveillance and control .............................................................................................................................................................................27 7. Licensing establishments .......................................................................................................................................................................................... 32 8. Regulatory inspection ..................................................................................................................................................................................................33 9. Laboratory testing ..........................................................................................................................................................................................................36 10. Clinical trial oversight .................................................................................................................................................................................................... 41 11. RA lot release .................................................................................................................................................................................................................... 47 References..................................................................................................................................................................................................................................48 Annex 1. PE indicators scorecard ...............................................................................................................................................................................51 Annex 2. Expert review of assessments of marketing authorization applications and clinical trial applications ..55 Annex 3. Vigilance field visit for assessing the performance of the vigilance function ......................................................... 101 Annex 4. GxP observed audit for assessing the performance of the regulatory inspection function.......................... 161 Annex 5. Expert review of laboratory testing activities ............................................................................................................................ 189 Annex 6. The abridged pathway .............................................................................................................................................................................. 229 Tables Table 1. Number of PE standard assessment components for the RS and each regulatory function .......................... 2 Table 2. PE indicators for RS: PE.RS.01 fact sheet ...........................................................................................................................................8 Table 3. PE indicators for RS: PE.RS.02 fact sheet .........................................................................................................................................8 Table 4. PE indicators for RS: PE.RS.03 fact sheet ....................................................................................................................................... 10 Table 5. PE indicators for RS: PE.RS.04 fact sheet ........................................................................................................................................ 11 Table 6 PE indicators for registration and marketing authorization: PE.MA.01 fact sheet ................................................ 15 Table 7. PE indicators for registration and marketing authorization: PE.MA.02 fact sheet ............................................... 15 Table 8. PE indicators for registration and marketing authorization: PE.MA.03 fact sheet ............................................... 16 Table 9. Number of sub-indicators per evaluation criteria in the PE tool for registration and marketing authorization (PE.MA.04) ............................................................................................................................................................................ 17 Table 10. PE indicators for vigilance: PE.VL.01 fact sheet ......................................................................................................................... 20 Table 11. PE indicators for vigilance: PE.VL.02 fact sheet ......................................................................................................................... 21 Table 12. PE indicators for vigilance: PE.VL.03 fact sheet ........................................................................................................................ 22 Table 13. PE indicators for vigilance: PE.VL.04 fact sheet ........................................................................................................................ 23 Table 14. PE indicators for vigilance: PE.VL.05 fact sheet ........................................................................................................................ 23 Table 15. PE indicators for vigilance: PE.VL.06 fact sheet ........................................................................................................................ 24 Table 16. PE indicators for vigilance: PE.VL.07 fact sheet ......................................................................................................................... 25 Table 17. Areas and number of indicators covered in the Vigilance field visit assessment questionnaire (PE.VL.08) ............................................................................................................................................................................................................. 26 Table 18. PE indicators for market surveillance and control: PE.MC.01 fact sheet ................................................................... 29 Table 19. PE indicators for market surveillance and control: PE.MC.02 fact sheet ...................................................................30 Table 20. Main areas covered by the Observed Audit Inspectors’ Evaluation Form: PE.RI.01 ...........................................35 Table 21. PE indicators for laboratory testing: PE.LT.01 fact sheet ......................................................................................................38 Table 22. Criteria and number of items covered in the PE tool for laboratory testing: PE.LT.02 .......................................40 Table 23. PE indicators for clinical trials oversight: PE.CT.01 fact sheet ..........................................................................................43 iv Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Table 24. PE indicators for clinical trials oversight: PE.CT.02 fact sheet .....................................................................................43 Table 25. PE indicators for clinical trials oversight: PE.CT.03 fact sheet .....................................................................................44 Table 26. Number of sub-indicators per evaluation criteria in the PE tool for clinical trials oversight (PE.CT.04) ........................................................................................................................................................................................................46 Table A6.1 Number of sub-indicators included in the abridged pathway, by maturity level and function (abridged/standard), compared to total number of GBT sub-indicators ......................................................... 240 Table A6.2 Total number of indicators in the abridged pathway tool: GBT + PE ......................................................................241 Figures Figure 1. GBT and PE components that contribute to designation of WLA status. ............................................................... 1 Figure 3. Flowchart for PE of registration and marketing authorization ....................................................................................... 13 Figure 4. Flowchart for PE of vigilance ............................................................................................................................................................... 18 Figure 5 Flowchart for PE of market surveillance and control ..........................................................................................................27 Figure 6. Flowchart for PE of licensing establishments ......................................................................................................................... 32 Figure 7. Flowchart for PE of regulatory inspection .................................................................................................................................34 Figure 8. Flowchart for PE of laboratory testing ..........................................................................................................................................36 Figure 9. Flowchart for PE of clinical trials oversight ............................................................................................................................... 42 Figure 10. Flowchart for PE of lot release ........................................................................................................................................................... 47 Acknowledgements v Acknowledgements The World Health Organization (WHO) would like to thank the Member States, technical partners, stakeholders, entities, individual experts and WHO staff who have  contributed to the development of this document and the accompanying operational guidance, as well as the WHO-listed authority (WLA) framework. A list of contributors to this document and the operational guidance is included below. Manuel Rodolfo Limeres and Mónica Bobbi, Administración Nacional de Medicamentos, Alimentos y Tecnología Médica (National Administration of Drugs, Food and Medical Devices), Argentina; Lilit Ghazaryan, Ministry of Health and Naira Romanova, Scientific Centre of Drug and Medical Technology Expertise, Armenia; Paul Huleatt and Richard Hill, Therapeutic Goods Administration, Australia; Geneviève Waeterloos, Sciensano, Belgium; Sakhile Dube-Mwedzi, South American Development Conference, Botswana; Bernardo Luiz Moraes Moreira, Brazilian Health Regulatory Agency and Fernanda Maciel Rebelo, Fernanda Simioni Gasparotto, Patrícia Kott Tomazett, Patricia Serpa, Yane de Carvalho Guedes and Varley Dias Sousa, Agência Nacional de Vigilância Sanitária (National Health Surveillance Agency), Brazil; Sea Thol, Ministry of Health, Cambodia; Gloria Giraldo, Amanda Jane Diniz, Maria Baca-Estrada and Richard Siggers, Health Canada, Canada; Andrea del Pilar Pando Seisdedos, José Miguel Crisóstomo Landeros, María Francisca Aldunate González, Nahum Vergara, Patricia Alexandra Carmona Sepúlveda and Verónica Vergara Galván, Instituto de Salud Pública de Chile (Public Health Institute of Chile), Chile; Blouie Po Yiu Tse, Christine Cheung, Michael Yim Tsz Kok, Raphael Yeung and Vincent Chow Wai-yan, Department of Health, Hong Kong Special Administrative Region, China; Carlos Rodriguez, Diana Mesa, Diego Alejandro Gutiérrez Triana, Jeannette Daza Castillo, Liliana Silvia Patricia Álvarez Espejo, Luz Andrea Gordillo Alarcón, María Alejandra Erazo Rebolledo, Ney Callas, Yenny Marcela Suarez González and Ximena Andrea Barbosa Aguillon, Instituto Nacional de Vigilancia de Medicamentos y Alimentos (National Institute of Food and Drug Monitoring), Colombia; Frank Diafouka and Hrvoje Tumir, Agency for Medicinal Products and Medical Devices of Croatia, Croatia; Biorkys Yáñez Chamizo, Celeste Aurora Sánchez González, Danay Mora Pascual, Giset Jiménez López, Liana Figueras and Mario Alberto Diaz Garcia, Centro Para el Control Estatal de la Calidad de los Medicamentos (Cuba Center for State Control of Drugs), Cuba; Amani Hegazi, Amira Amin, Asmaa Fouad, Dalia Abouhussein, Doaa Abd El Rady, Engy El Hosary, Heba Khalil, Ireen Habeeb, Irene Gamal Shehata and Shereen Mohamed, Egyptian Drug Authority, Egypt; Katrin Lambing and Marje Zernant, Estonian State Agency of Medicines, Estonia; Alain Alsalhani, Médecins Sans Frontières, Cristina Baccarelli, Catherine Milne, Michael Wierer and Richard Wanko, European Directorate for the Quality of Medicines and François Cano and Sylvie Morgeaux, Agence Française de Sécurité Sanitaire des Produits de Santé (National Agency for Medicines and Health Products Safety), France; Brigitte Mauel-Walbrol, Bundesinstitut für Arzneimittel und Medizinprodukte (Federal Institute for Drugs and Medical Devices/ Benchmarking of European Medicines Agencies) and Olga Rassokhina, Paul Ehrlich Institute, Germany; Abdul-Samad Issah, Abena Asamoa-Amoakohene, Eric Karikari- Boateng, Nathaniel N. K Nkrumah and Seth Kweku Seaneke, Food and Drugs Authority, Ghana; Ika Prawwahju, Elizabeth Ika P and Lucky Slamet, Badan Pengawas Obat dan Makanan (Indonesian Food and Drug Authority), Indonesia; Hamidreza Hozouri, Pasteur Institute, Iran (Islamic Republic of); Anna Laura Salvati and Sandra Petraglia, Agenzia Italiana del Farmaco (Italian Medicines Agency), Italy; Eriko Fukuda, Jun Kitahara, Teruyoshi Ehara, Pharmaceuticals and Medical Devices Agency; Hirooki Tanabe and Naoyuki Yasuda, Ministry of Health, Labour and Welfare; Koji Ishii, Masaki Ochiai, National Institute of Infectious Diseases, Japan; Zaril Zakaria, National Pharmaceutical Regulatory Agency, Malaysia; Adriana Padilla Mendoza, Bertha Araceli Rodríguez Arvizu, Daniela López, Elvira Espinosa Gutiérrez, Felipe Ángel de la Sancha Mondragón, Graciela Aguilar Gil Samaniego, Imelda Rocío Guzmán Cervantes, Iris Lorena Arreola Dominguez, Karla Maricela Félix Rodríguez, María de Lourdes Rodriguez Aveleyra, Miriam Jackeline Loera Rosales, Rocio Guzman and Virginia Ramirez, Comisión Federal para la Protección contra Riesgos Sanitarios (Federal Commission for the Protection against Sanitary Risk), Mexico; Houda Sefiani, National Pharmacovigilance Centre, Morocco; Martijn Bruysters, National Institute for Public Health and the Environment and Martin Harvey Allchurch, European Medicines Agency, Netherlands (Kingdom of the); Abiola Victor Olayiwola, Anthony Joseph Hotton, vi Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Ijeoma Ukachi Nwankwo, Monica Hemben Eimunjeze and Victor Abiola Olayiwola, National Agency for Food and Drug Administration and Control, Nigeria; Caren Mangorangca, Pia Angelique Priagola and Jade Mae D. Biyo, Food and Drug Administration, Philippines; Eugenia Lopes, Infarmed/Benchmarking of European Medicines Agencies and Helena Baiao, Autoridade Nacional do Medicamento e Produtos de Saúde I.P. (National Authority of Medicines and Health Products, I.P.) Portugal; Eun Hye Parks, Hee Young Park, In Yeong Hwang, Jung-Ah Byun and Yu Jeong Seo, Ministry of Food and Drug Safety, Republic of Korea; Alexandru Sirbu, Independent Consultant, Romania; Abrar Al Musharraf, Alya Aldahash, Khalid Mohammed Al Mowaizri, Mubarak Saeed Alshahrani, Turki Abdulaziz Al Rafie, Saudi Food and Drug Authority, Saudi Arabia; Dragana Šmidling Koruga, Medicines and Medical Devices Agency, Serbia; Sia Chong Hock, Health Sciences Authority, Singapore; Deon Poovan, Portia Nkambule, Quinton Meyer, Silverani Padayachee and Tohlang Sehloho, South African Health Products Regulatory Authority (SAHPRA), South Africa; Johan Ellenius and Monica Plöen, Uppsala Monitoring Centre and Sten Olsson, Independent consultant, Sweden; Andreas Pfenninger, Gabriela Zenhäusern and Michael Gilgen, Swissmedic, Fabienne Benoist, Drugs for Neglected Diseases initiative, Daniel Brunner, Pharmaceutical Inspection Co-operation Scheme, and J Bernat, International Federation of Pharmaceutical Manufacturers Associations, Switzerland; Achiraya Praisuwan, Suchart Chongprasert and Supatra Phongsri, Food and Drug Administration and Supaporn Phumiamorn and Wereyarmarst Jaroenkunathum, Institute of Biological Products, Thailand; Nihan Burul Bozkurt, Türkiye İlaç ve Tıbbi Cihaz Kurumu (Pharmaceuticals and Medical Devices Agency of Türkiye), Türkiye; Helen Byomire Ndagije and Helen Ndagije, National Drug Authority, Uganda; Lawrence Liberti and Prisha Patel, Centre for Innovation in Regulatory Science, United Kingdom of Great Britain and Northern Ireland; Sunday Kisoma, Emmanuel Alphonce Nkiligi and Felchism Apolnary, Tanzania Medicines and Medical Devices Authority, United Republic of Tanzania; Gopa Raychaudhuri and Theresa Mullin, US Food and Drug Administration; Jude Nwokike, United States Pharmacopeia, United States of America; Caroline Getrude Samatanga, Libert Chirinda and William Wekwete, Medicines Control Authority of Zimbabwe, Zimbabwe. Thanks are also due to the following for their financial support towards the development and production of this document and the accompanying performance evaluation manual: • Administración Nacional de Medicamentos, Alimentos y Tecnología Médica (National Administration of Drugs, Food and Medical Devices [ANMAT]), Argentina • Agence Française de Sécurité Sanitaire des Produits de Santé (National Agency for Medicines and Health Products Safety [ANSM]), France • Agência Nacional de Vigilância Sanitária (National Health Surveillance Agency [ANVISA]), Brazil • Agency for Medicinal Products and Medical Devices (HALMED), Croatia • Badan Pengawas Obat dan Makanan (Indonesian Food and Drug Authority [PADAN POM]), Indonesia • Bill & Melinda Gates Foundation (BMGF) • Centro Para el Control Estatalde la Calidad de los Medicamentos (Cuba Center for State Control of Drugs [CECMED]), Cuba • Comisión Federal para la Protección contra Riesgos Sanitarios (Federal Commission for the Protection against Sanitary Risk [COFEPRIS]), Mexico • Drugs for Neglected Diseases Initiative (DNDi) • European Medicines Agency (EMA) • GAVI, the Vaccine Alliance • Health Sciences Authority (HSA), Singapore • Indian Pharmacopoeia Commission, India • Instituto de Salud Pública de Chile (Public Health Institute of Chile [ISP]), Chile • Instituto Nacional de Vigilancia de Medicamentos y Alimentos (National Institute of Food and Drug Monitoring [INVIMA]), Colombia • International Federation of Pharmaceutical Manufacturers & Associations (IFPMA) • Management Sciences for Health (MSH) • Médecins Sans Frontières (Doctors Without Borders [MSF]) • Ministry of Food and Drug Safety (MFDS), Republic of Korea • National Agency for Food and Drug Administration and Control (NAFDAC), Nigeria • Pharmaceuticals and Medical Devices Agency (PMDA), Japan • Saudi Food and Drug Authority (SFDA), Saudi Arabia • SWISSMEDIC, Switzerland • Therapeutic Goods Administration (TGA), Australia • U.S. Food and Drug Administration (FDA), United Stated of America • United States Pharmacopeia (USP), United Stated of America Abbreviations vii Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Abbreviations AEFI adverse event(s) following immunization CT clinical trial (oversight) CTA clinical trial application CTD common technical document GBT Global Benchmarking Tool GxP good practices ICH International Council for Harmonisation of technical requirements for pharmaceuticals for human use LI licensing establishments LR (RA) lot release LT laboratory testing MA (registration and) marketing authorization MAA marketing authorization application MC market (surveillance and) control ML maturity level NRA national regulatory authority PBRER periodic benefit-risk evaluation reports PIC/S Pharmaceutical Inspection Convention/ Pharmaceutical Inspection Cooperation Scheme PE performance evaluation PMS post-market surveillance QMS quality management system RA regulatory authority (NRA or RRS) RI regulatory inspection RRS regional regulatory system RS regulatory system SOP standard operating procedure SRA stringent regulatory authority VL vigilance viii Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Code of conduct for WHO experts WHO values and relies upon the normative and technical advice provided by leading subject matter experts in the context of its advisory/technical committees, meetings, and other similar processes. Such advice contributes to the formulation of the public health policies and norms that are promulgated by WHO for the benefit of its Member States. In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice: a. fully and honestly disclose all relevant interests and biases on the declaration of interests (DOI) Form that may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting, or group members at the outset, that is unless this is done by the chairperson or WHO Secretariat b. spontaneously report any material changes to their disclosed interest on an ongoing basis during the period in which the expert serves the Organization respect the confidential nature of committee or meeting deliberations or of the advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO c. undertake not to engage in activities that may bring reputational harm to the WHO process that they are involved in d. undertake to represent their views in a personal and individual capacity with the best interest of WHO in mind as opposed to representing the views of their employers, other institutions, or governments e. actively and fully participate in discussions and deliberations within the relevant advisory group, committee, or meeting. WHO has zero tolerance towards sexual exploitation and abuse, sexual harassment (SEAH) and other types of abusive conduct (that is, discrimination, abuse of authority and harassment). Sexual exploitation, sexual abuse and sexual harassment are considered to be acts of serious misconduct and constitute a basis on which staff members, whether internationally or locally recruited, and contractors can be summarily dismissed. 1. Introduction 1 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 1. Introduction This document provides operational and technical details for the performance evaluation (PE) exercise that must be conducted for a regulatory authority (RA) to achieve listing as a WHO-listed authority (WLA) in relation to each regulatory function. This document should be read in conjunction with the Operational guidance for evaluating and publicly designating regulatory authorities as WHO-listed authorities (“operational guidance”, 1). For the purposes of this document the term regulatory authority (RA), unless otherwise stated, may refer to either a national regulatory authority (NRA) or a regional regulatory system (RRS). The basis for designation as a WLA is provided by the Global Benchmarking Tool (GBT, 2), which is complemented by a series of PE activities designed to establish a detailed picture of how the regulatory system performs on relevant regulatory processes, including how consistently it adheres to quality procedures and how well it delivers the desired regulatory outputs in accordance with good regulatory practices (3). 1.1 Criteria for designation as a WLA A pre-requisite for becoming listed as a WLA, according to the eligibility criteria set out in the operational guidance, is that an applicant must either be included in the transitional WLA (tWLA) list, or have attained at least overall maturity level (ML) 3, as determined through a formal benchmarking assessment using the latest version of the GBT. The RA applying for WLA status must also meet the following requirements in the regulatory system (RS) and in the regulatory function(s) relevant for the listing: ƒ fully implement all GBT sub-indicators of ML3, thus achieving ML3 with the “normal” cGBT algorithm According to the “flexible” algorithm, overall ML3 can be in fact deemed to be achieved if 100% of ML1 and ML2, and 90% of ML3 sub-indicators have been met and a plan is in place to comply with the remaining 10% of ML3 sub-indicators. ƒ meet the requirements of the PE framework This may include one or more of the following components: – a set of mandatory ML4 GBT sub-indicators (note: candidate WLAs may never score mandatory ML4 sub-indicators as “not applicable”) – a set of PE indicators, and/or – a set of PE tools. Fig. 1 presents the GBT and PE components that contribute to achieving WLA status; Table 1 shows the number of PE standard assessment components for the regulatory system and each regulatory function. For a list of elements assessed when applying for the listing of product categories, see the Operational guidance for evaluating and publicly designating regulatory authorities as WHO-listed authorities. GBT PE WLA Meet requirements for ML1, ML2 and ML3 GBT sub-indicators in all functions (overall ML3) Meet all GBT ML3 sub-indicators and mandatory GBT ML4 sub-indicators in the relevant function(s) Meet performance evaluation indicators in the relevant function(s) WLA status for relevant function(s) PE indicators GBT ML3 and ML4 sub- indicators + Meet performance evaluation tools in the relevant function(s) + PE tools Fig. 1. GBT and PE components that contribute to designation of WLA status 2 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 1. Number of PE standard assessment components for the RS and each regulatory function Assessment category Number of: mandatory ML4 GBT sub-indicators PE indicators PE tools Regulatory system (RS)a 14 4 - Registration and marketing authorization (MA) 3 3 1 Vigilance (VL) 4 7 1 Market surveillance and control (MC)b 2 2 - Licensing establishments (LI)c 2 - - Regulatory inspection (RI)d 5 - 1 Laboratory testing (LT) 3 1 1 Clinical trials oversight (CT) 2 3 1 Lot release (vaccines) (LR)e 1 - - a The RS is a qualifying requirement for listing in other categories, it is not possible for an RA to be listed for RS alone. b In order to become a WLA for market surveillance and control, RAs must also meet all the requirements for registration and marketing authorization, vigilance, regulatory inspection and laboratory testing. c In order to become a WLA for licensing establishments, RAs must also meet all the requirements for regulatory inspection. d Where applicable, a WLA for regulatory inspection must also meet all the requirements for licensing establishments. e In order to become a WLA for lot release, RAs must also meet all the requirements for laboratory testing. 1.2 Selection of PE components In order to ensure optimum use of information and evidence that is already available, PE components will be selected using a risk-based approach. Ideally, the components should be conducted in sequential order, so that first it is determined that all the mandatory ML4 sub-indicators have been met, next the PE indicators are evaluated and only then are the PE activities undertaken, driven by the PE tools. The specific requirements for the RS and each regulatory function are set out in the chapters below. PE indicators and tools are designed to be meaningful, measurable, and reasonable. In order to give assessors all the information they need to conduct an evaluation, they generally follow a similar structure to the GBT factsheets, including a description and objectives, a list of evidence to review and an assessment matrix with rating scale. The list of evidence to be reviewed is not intended to be either exhaustive or exclusive: assessors will need to rely on their own expertise to evaluate whether other evidence can be considered sufficient and adequate to demonstrate implementation of each requirement. 1.3 Self-assessment Whenever PE indicators have been developed, the RA should complete a self-assessment against the indicators and submit the self-assessment to WHO for review and evaluation. If possible, the RA is also expected to conduct a self-assessment for PE tools. 1.4 Timeframe for the evaluation The overall PE exercise is time bound. Depending on the requested scope of WLA listing and the size of evaluation team, it is expected to take no longer than six months to complete the PE for any given regulatory function. WHO and the RA may agree to a clock stop of up to six months to address outstanding issues, if deemed necessary. 1.5 The WHO assessment team PE will be carried out by a team of multi-national experts appointed by WHO. The number of evaluators assigned will vary depending on the requested scope of WLA listing, and on evaluator availability. All assessors will submit confidentiality undertakings and declaration of interest forms prior to any of the assessment processes described below. All members of the WHO team involved in the PE exercise in any way shall familiarize themselves, respect and follow WHO’s code of conduct (a short version of which can be found on page v). 1.6 Form and scope of the evaluation Review and evaluation may be conducted on- site or remotely, with information shared through secure platforms and virtual meetings, as needed. In coordination and agreement with the RA, the WHO Secretariat may request translation of key documents necessary for the evaluation. 1. Introduction 3 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual All of the activities conducted in the context of the WLA evaluation framework are only intended to evaluate the performance of the RA in relation to specific regulatory processes; it is beyond the scope of the PE exercise to evaluate the level of implementation and/or compliance to regulatory requirements of manufacturers, MA-holders, sponsors, or any other stakeholder involved in the PE process in any way. 1.7 Outsourced activities If a candidate WLA outsources technical regulatory activities – that is, it has agreements with third parties for the performance of tasks assigned to it – the PE exercise may extend to the outsourced organization(s), depending on the criticality of outsourced activities. Written contracts are expected to be in place with the outsourced organizations approved for the type of regulatory activity being conducted. The contracts should clearly define: the duties and responsibilities of each party; the decision-making process for issuing, renewing, or rescinding contracts for outsourced services; established and implemented procedures for managing outsourced activities; and procedures for handling communication exchanges. The candidate WLA is responsible for periodically assessing – through continuous monitoring of any outsourced activities – the technical, material, human and financial capacity, as well as the facilities, equipment, and technology of the contracted organization(s). The contracted organization should not pass along to a third party any work entrusted to it under the contract without prior evaluation and approval of the arrangements by the RA. 4 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2. Preparing for, conducting and reporting on PE activities This chapter outlines the elements involved in preparing for, conducting and reporting on PE activities. 2.1 Declaration of Interest and confidentiality undertaking Before any PE-related activity is undertaken, all members of the WHO team taking part in the PE exercise for a particular RA must sign confidentiality undertakings and declaration of interest forms – except for WHO staff, for whom other confidentiality and declaration of interest arrangements are already in place (that is, those applied by the human resources department during staff recruitment process). Completed and signed confidentiality undertakings and declaration of interest forms should be assessed and archived by the WHO Secretariat prior to the PE activity, following the relevant WHO procedures. The signed forms will be available at WHO and may be shared with the relevant authorities, if required. Individuals nominated to be members of the WHO team should be excluded if they are found to have a conflict of interest in relation to the PE of a given RA. In addition, if deemed necessary, WHO may engage in discussions with the relevant RA towards the signature of a specific confidentiality disclosure agreement to resolve any further legal impediments. In such cases, the guidance and procedure given in the Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans should be applied (4). 2.2 Terms of reference Prior to any PE activity, whether onsite or offsite, the WHO Secretariat prepares the terms of reference, following the applicable WHO procedures and in collaboration and agreement with counterparts at the relevant RA. The terms of reference should specify objectives, proposed dates, a tentative agenda, expected outcome and deliverables, the composition of the WHO team of assessors, along with the documents or information needed during the evaluation. It should be noted that the WHO team may request additional documents or information during the assessment. The terms of reference, along with a finalized workplan for both onsite and offsite assessments, should be shared with the relevant RA through official communication channels for agreement and concurrence. The terms of reference should also be distributed to all participants and made available for consultation and archival purposes via a WHO secure information-sharing platform. 2.3 Secure information sharing Once a PE activity has been agreed and confirmed between WHO and the RA as part of the roadmap to WLA, a specific webpage should be created under the relevant site of the WHO secure information- sharing platform. All related documents should then be uploaded to the platform for access and archival purposes, including but not limited to the terms of reference, programme, background documents, documented evidence submitted by the RA (such as procedures), WHO team information, presentations and reports. By default, access to the WHO information-sharing platform is restricted to authorized users. Once the confidentiality agreement and declaration of interest forms have been signed, access to the platform should be granted to the WHO team as well as selected officials nominated by the relevant RA, in order to enable them to communicate securely with one another and upload or download relevant information. 2.4 Interpretation and translation PE activities should be performed in a language that is understood well by the members of the WHO team and the participants. Preferably, the PE should be conducted in the official language of the target country if this is understood well by the WHO team. If needed, a simultaneous interpretation service may be provided for the WHO team. In addition to simultaneous interpretation, the WHO Secretariat may, in coordination and agreement with the RA, request translation of documentation related to the PE activities (such as applications, procedures, guidelines). Interpreters and translators should preferably have technical expertise with the concerned PE activities. 2. Preparing for, conducting and reporting on PE activities 5 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2.5 Team members’ attributes and competencies A roster of qualified experts who can conduct PE activities on behalf of WHO should be accessible to the WHO Secretariat. WHO team members (experts/ assessors) should be qualified and competent according to well-established criteria for conducting the requested activity. The criteria for designation of WHO team members are: ƒ Background and education: WHO team members should be staff members of a RA or WHO staff members or experts (such as a consultants) who are familiar with the relevant WHO and other internationally recognized standards and guidelines. ƒ Experience: WHO team members should be experienced in the field of medical products and should have at least seven years’ experience in the area specifically targeted by the PE activity. ƒ Training: WHO team members should be well trained in the processes and methodologies related to regulatory activities. WHO team members should also be familiar with WHO benchmarking and WLA concepts and methodologies. ƒ Skills: WHO team members should have advanced skills in assessment activities, as well as in questioning and listening, in case an interview is considered necessary as part of the assessment (for example an observed audit or field visit). ƒ Evaluation: WHO team members are subject to formal evaluation by WHO staff (for example WHO team leader) against pre-set criteria and in accordance with the relevant procedures. PE activities are performed by a group of experts whose number is commensurate to the assigned roles and responsibilities, with specific expertise on the type of product (chemical or biological), and regulatory function(s) in the scope of the WLA. 2.6 Conducting desk-based and onsite evaluations For any PE activity, the WHO Secretariat or WHO team leader will arrange remote briefing sessions so that details and methodology of the activity to be conducted can be shared with all WHO team members. The PE exercise may include desk-based activities, either as part of the preparation for onsite assessments or as formal remote PE assessments of evidence provided by the RA through the WHO secure platform. If additional clarifications and confirmations are needed, remote meetings can be arranged between the WHO team of assessors and representatives of the RA. For onsite evaluations (such as a Good practices (GxP) observed audit, a vigilance field visit, or an expert review of laboratory testing activities), a WHO team leader will be identified, and visits will be arranged with the agreement and concurrence of the relevant RA. Details of the roles and responsibilities for offsite and onsite assessments can be found in the relevant sections below. 2.7 Reporting on assessment activities At the end of each assessment an independent report will be issued, following WHO procedures, summarizing the activities conducted and the conclusions. Within the context of WLA evaluations, assessors may also make recommendations for improvement of the regulatory system. However, provided the RA is meeting the WLA criteria, any such recommendations will not be considered mandatory. At the end of the process, a comprehensive PE report that compiles the findings of all PE activities is issued. This will inform the opinion of the Technical Advisory Group on WLA, following the steps described in the operational guidance (1). 6 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Fig. 2 Flowchart for PE of regulatory system 3. Regulatory system 3.1 Methodology for PE of regulatory system PE of the regulatory system (RS) is designed to assess the effectiveness of the NRA or RRS as the overall framework for ensuring a sustainable, well- functioning RS that can ensure independent and competent oversight of medical products. It is not possible to achieve WLA listing solely for the RS, as this does not correspond to any specific regulatory activity. However, meeting to an acceptable standard the RS requirements listed below is a qualifying requirement for designation as a WLA in relation to any other regulatory function and/or product category. As shown by the flowchart in Fig. 2, in order to fulfil the PE requirements for the RS, the RA must, in addition to meeting the eligibility criteria, demonstrate that it: a) fully implements all ML3 sub-indicators for RS b) acceptably implements the 14 mandatory ML4 GBT sub-indicators for RS (see section 3.2) c) acceptably meets four PE indicators for RS (see section 3.3). Are eligibility criteria met? Are all ML3 and mandatory ML4 sub- indicators met? Assess regulatory system against PE indicators Are all regulatory system PE indicators acceptably met? PE of regulatory system is fulfilled Report negative conclusion and outcome Performance evaluation for other function(s) Start No Yes No No Yes Yes 3. Regulatory system 7 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 3.2 The mandatory ML4 GBT sub-indicators for RS To pass the PE for RS, candidate WLAs must fully implement the following ML4 sub-indicators, as defined in the GBT (2): ƒ RS03.05: The RA is promoting good regulatory practices. ƒ RS05.05: The RA establishes mechanisms to continually improve the quality management system (QMS). ƒ RS05.06: The RA has identified its regulatory processes, determined their interactions, and defined the methods needed to control these processes. ƒ RS05.08: External and internal issues, including relevant potential risks, are defined and assessed periodically for proper risk mitigation. ƒ RS05.10: A mechanism to evaluate the satisfaction of internal and external customers and other interested parties is in place for system improvement. ƒ RS05.12: Corrections, corrective actions, and other actions for risk mitigation and overall improvement, are implemented and documented and their effectiveness is verified. ƒ RS05.13: Top management reviews and documents the organization’s QMS at planned intervals (that is, management review). ƒ RS06.01: The RA has the power to select and recruit its own staff following documented procedures based on its own written criteria (that is, education, training, skills, and experience). ƒ RS07.04: The RA has authority to manage the funds allocated and/or generated internally. ƒ RS09.01: The RA participates in regional and/ or global networks to promote convergence and harmonization efforts and expand its collaboration in the regulatory field. ƒ RS09.03: Information on decisions related to regulatory activities is available to the public. ƒ RS09.05: All publicly available information is periodically reviewed and maintained. ƒ RS10.01: Requirements established to monitor, supervise, and review the performance of the RA and affiliated institutions using key performance indicators. ƒ RS10.02: Reports on the regulatory activities and on the progression and status of resources are available at regular intervals. 3.3 PE indicators for RS The RS of a candidate WLA must be assessed against four PE indicators and meet the acceptability criteria as set out in the rating scale (see the PE indicator fact sheets in Tables 2–5). Once each indicator has been assessed, its score should be recorded in the PE indicators scorecard (see Annex 1). 8 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 2. PE indicators for RS: PE.RS.01 fact sheet PE.RS.01 The RA participates in the WHO certification scheme on the quality of pharmaceutical products moving in international commerce and issue certificate of pharmaceutical product Description The assessor should verify that the RA is an active member of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce; and is listed as a certifying member on the WHO website. Note: this indicator should be evaluated according to the terms and conditions explained and covered by the WHO certificate of pharmaceutical product guideline. Any bilateral agreement or additional request by importing RAs or commercially interested parties is out of scope. Objective To ensure that the RA controls medical products exported from its jurisdiction; and that it follows the latest WHO guidelines and requirements for issuing certificates of pharmaceutical product, including having a reliable system for authorizing medical products, and licensing and inspecting manufacturing facilities. Although this indicator does not cover the export only products, similar regulatory oversight on these group of products is encouraged. Evidence to review The assessor should review the list of contacts for competent authorities of countries participating in the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce (5). References Guidelines on the implementation of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce (6). Resolution WHA50.3. WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce (7). WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce: questions and answers (Q&A) (5). Rating scale Not implemented (NI): The RA is not a member of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce. Implemented (I): The RA is a member of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce and is included in the WHO list of contacts published online. For an authority to be given WLA status, this indicator must be scored as implemented. Limitations and remarks This sub-indicator cannot be scored as "not applicable ". That is, this PE indicator always applies when assessing candidate WLAs.  Table 3. PE indicators for RS: PE.RS.02 fact sheet PE.RS.02 The RA has established an effective competency framework Description A competency framework defines the knowledge, skills, attitudes, and behaviours needed for people within an organization that are developed through education, training, and experience. A competency framework may go by a different name; but should still fulfil the principles and components mentioned here. As a minimum, it comprises six outputs: 1. A fit-for-purpose organizational competency manual or similar document. 2. Documentation of competency requirements for each position. 3. Competency assessment methods and tools. 4. Records of competency assessments and feedback to staff. 5. Training plans. 6. Mechanism for monitoring and evaluation of training programmes. The assessor should verify the existence, implementation, effectiveness and performance of the RA’s competency framework, including the components detailed above. 3. Regulatory system 9 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual This includes verifying the quality (including consistency) and effectiveness of internal mechanisms to monitor and evaluate the performance and impact of the competency framework. Such monitoring and evaluation should be done regularly (for example, annually); and should align with the RA’s strategic plan and commitment to continual improvement. The assessor should also verify the existence, and adequate resourcing, of a system that ensures a coordinated and integrated approach across all regulatory activities. In all cases, this indicator should be evaluated alongside related GBT human resources indicators under each function; as well as GBT sub-indicators RS06.01, RS06.02, RS06.03, RS06.04, RS05.14 (4). Objective To ensure that the RA has established and implemented a competency framework that can be used to: • identify required qualifications for staff performing various regulatory and supportive activities • select staff with the qualifications and level of competency needed • assign regulatory work to scientific staff members according to their qualification • define training needs • plan and provide training designed to meet competency objectives • measure the impact of training and use this information to support continual improvement. Evidence to review The assessor should ask for and review: ƒ The competency framework manual (or equivalent document) for all applicable regulatory and operational functions. ƒ Evidence that all components of the competency framework have been implemented, including documents, records, reports or assessments to show that: a. descriptions of the required competencies are clear and fit for purpose b. competency assessment tools used and respective outputs are appropriate c. interventions to build and maintain competencies are relevant d. training has been provided and impact assessments carried out e. there are mechanisms to evaluate and demonstrate the effectiveness of the competency framework in conjunction with the organization’s continual improvement plans. ƒ Periodic (trend) reports and/or reviews of the competency framework’s performance, including subsequent actions taken for improvement (for example, through internal audits, management review, corrective and preventive actions that evaluates the framework periodically). ƒ Records of communications to staff about the competency framework, including the results of any competency assessments carried out. The assessor should liaise with other assessors involved in GBT and WLA performance evaluation in order to evaluate the quality and impact of the competency framework’s outputs. References Towards a global competency framework for regulators of medical products (8) and GBT sub-indicators RS06.01, RS06.02, RS06.03, RS06.04, RS05.14 (2). Rating scale Not implemented (NI): The RA does not have a comprehensive competency framework to cover both regulatory and operational functions. Various components of a competency framework may exist separately, but they are not connected or integrated properly to have an effective and forward-looking competency management plan. Partially implemented (PI): The RA has established most of the components of the competency framework. But there is inadequate evidence of a mechanism to evaluate the framework’s effectiveness in line with the organization’s continual improvement plan, and/ or there is inadequate evidence of actions taken for improvement. Implemented (I): The RA has established an effective and forward-looking competency framework for continual improvement, including all the required components as explained above under description, objective, and evidence to review. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. 10 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing candidate WLAs.  Table 4. PE indicators for RS: PE.RS.03 fact sheet PE.RS.03 The RA has implemented measures to monitor, evaluate and sustain the performance of the quality management system (QMS) Description A fit-for-purpose QMS facilitates the consistent, effective, transparent, and efficient performance of technical and administrative activities within the capacity, resources, needs and context of the RA. In all cases, the RA should be able to monitor and evaluate the performance of key operational aspects and areas of the QMS over time, by integrating results achieved by all different activities of the organization. The assessor should verify that there is a system and procedures in place to manage the performance of the QMS for all applicable regulatory and operational functions; and that it works effectively. The system and related procedures should cover all essential components of a QMS, including qualified staff and management, processes, resources (financial, infrastructure and tools), documentation, trend reports for key QMS outputs, management reviews and actions taken for continual improvement. Objective This indicator aims to ensure that the RA has established, implemented, and adequately resourced a system and procedures for measuring the performance of the QMS across all applicable regulatory and operational functions. Evidence to review The assessor should ask for and review: a. written procedures for measuring QMS performance b. records of resources assigned for measuring QMS performance (including staff, IT infrastructure, financial and so on). In addition, the assessor should ask for and review evidence of monitoring and evaluation towards continual improvement, including: ƒ Trend reviews and reports on non-conformances and subsequent corrective actions ƒ Reviews and revisions of processes and documents (such as manuals, standard operating procedures (SOPs), templates and so on) as per QMS requirements ƒ Reports and follow-ups on areas of improvement and (service) complaints ƒ Risk and opportunity management procedures and related actions taken ƒ Records to show top management's commitment to QMS performance (for example, meeting minutes detailing frequency and type of meetings held, and management’s participation) ƒ List of key performance indicators (or equivalent) for QMS in line with organization’s plan for continual improvement ƒ Periodic (trend) reports or reviews of QMS key performance indicators, and subsequent actions taken. To evaluate the QMS’ contribution to the RA’s overall performance, the assessor should liaise with other assessors; or review the benchmarking report and other WLA PE reports. Note: procedures and resources for measuring QMS performance may be described in the same document that details other key performance indicators for the RA. References WHO guideline on the implementation of quality management systems for national regulatory authorities (9); Implementing quality management systems in national regulatory authorities: examples and practices (10); GBT sub-indicators RS05.01 to RS05.14 (2) Rating Scale Not implemented (NI): The RA has no mechanisms, system and/or procedures to measure QMS performance. Partially implemented (PI): The RA has identified and implemented a system or mechanism for monitoring and evaluating QMS performance in critical areas of its functions and operations (criticality to be based on the needs of the RA with respect to the scope of WLA listing). 3. Regulatory system 11 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Implemented (I): The RA has developed and implemented a robust mechanism or system with relevant procedures for monitoring, evaluating QMS performance; and for implementing actions to sustain its performance. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing candidate WLAs.   Table 5. PE indicators for RS: PE.RS.04 fact sheet PE.RS.04 The RA has a mechanism, supported by adequate regulations, guidelines and/or SOPs, for sharing technical information, evaluation reports, or any other information about its regulatory decisions, with other authorities. Description Given its importance in building trust and enabling reliance, transparency in regulatory operations and decisions is a key focus area for WLAs. RAs seeking designation as WLAs are therefore expected to share information with other regulatory authorities and with the public, thereby enabling greater regulatory efficiencies and more informed decision-making at the regional and global level. In the best-case scenario, pertinent information is made publicly available by issuing public assessment outcomes, and shared proactively with other RAs with whom the candidate WLA has confidentiality agreements; however, in the process of progressively adopting more transparent and proactive approaches, information should be shared with other regulators, at least upon their request. At the same time, it is acknowledged that in some circumstances RAs may not be in a position to respond to all requests. In such cases, a prioritization process should be adopted based on criticality or novelty of products and impact on public health. In addition to information made publicly available, the assessor should verify that a mechanism is in place for facilitating the exchange of non-public information on regulatory decisions with other RAs, at least upon their request, and that this mechanism is made public. The mechanism should include details of any information exchange arrangement (e.g., confidentiality agreement), or legal requirements required to share information with other regulatory authorities. Information sharing mechanisms should ideally provide for the sharing of technical evaluation reports and at a minimum the sharing of summarized versions of reports or justifications of regulatory decisions. Reasonable efforts should be made by the RA to verify the relevance, accuracy and, as appropriate and feasible, comprehension of information exchanged. The information should be shared according to RA regulations, guidelines and/or SOPs, within a timeframe that responds to the needs of the requesting RA. Depending on the volume and nature of requests, responses may need to be prioritized based on perceived risk. Objective This indicator aims to ensure the existence of regulations, guidelines and/or SOPs that allow the RA to share information about their regulatory decisions particularly, but not limited to, those related to marketing authorization and regulatory inspections. The ultimate goal is to ensure the RA embraces and follows adequate transparency policies, principles and procedures. Evidence to review The assessor should ask for and review:  ƒ Documented regulations, guidelines or procedures to support information sharing. ƒ Templates for a memorandum of understanding, confidentiality agreement or other relevant information-sharing instruments. ƒ Any other evidence that a mechanism exists for sharing information with other regulatory authorities, including records of sharing information with a network of authorities through a private digital platform, using a common language; emails of information shared with relevant contacts upon request; time-embargoed information shared in advance of publication with regulators under confidentiality agreements; written records of confidential discussions held between regulators. 12 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ƒ A number of shared files, selected by the assessor, for refused or rejected MA applications.  ƒ A number of shared GxP inspection reports, selected by the assessor, for authorized and suspended/revoked applicants. ƒ Other type of information that has been shared with other regulators upon request (e.g., clinical trials assessments, laboratory testing results, other). References GBT sub-indicators RS01.06; RS09.03; RS09.04; RS09.05; RS09.06; MA 05.03; MA 05.04; RI06.04; CT05.02; LT07.01 (2) Rating Scale Not implemented (NI): A mechanism for sharing technical information with other regulators does not exist or is not properly communicated to the public or is not effectively applied. Partially implemented (PI): The RA developed a mechanism for sharing non-public information with other regulators, at least upon request; the mechanism has been properly communicated to the public and to other regulators, and it is effectively applied. Implemented (I): The RA has defined and established a mechanism for proactive sharing of information with other regulators. Exchange of non-public information requested by other RAs is also carried out, taking into consideration the nature and volume of requests. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing candidate WLAs.  In case no request for non-public information has been received by a RA which utilizes a reactive approach, this PE indicator can be scored as partially implemented, if adequate justification is provided. Information exchanged with other regulators supplements the timely publication of pertinent information on regulatory activities and assessment outcomes. 4. Registration and marketing authorization 13 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 4. Registration and marketing authorization 4.1 Methodology for PE of registration and marketing authorization PE of registration and marketing authorization (MA) is designed to assess registration and marketing authorization for specific products and activities, including: ƒ Products: new chemicals entities (new medicines), multisource/generic medicines, vaccines, biotherapeutics, similar biotherapeutic products. ƒ Processes: pre-submission procedure; submission, screening, and validation of the application; selection of regulatory pathway; administrative and scientific evaluation; advisory committee procedures and adoption of final decision; transparency and structure; post-approval actions (including renewals, variations, extensions of registration and marketing authorization, withdrawals and transfers of registration and marketing authorization); and effectiveness of the registration and marketing authorization process. As shown by the flowchart in Fig. 3, PE of registration and marketing authorization is considered to be fulfilled if, in addition to meeting the eligibility criteria, the RA demonstrates that it: a. fully implements all ML3 sub-indicators for registration and marketing authorization b. fully implements the three mandatory ML4 GBT sub-indicators for registration and marketing authorization (see section 4.2) c. acceptably meets three PE indicators for registration and marketing authorization (see section 4.3) d. has successfully undergone an expert review of Marketing Authorization Application (MAA) assessments (see section 4.4 and Annex 2). Fig. 3 Flowchart for PE of registration and marketing authorization End Are eligibility criteria met? Are all ML3 and mandatory ML4 sub-indicators met? Assess against MA PE indicators Are all MA PE indicators acceptably met? Conduct expert review of MAA Assessments Start No Yes No No Yes Yes Are all parts of the expert review acceptably met? Report negative conclusion & outcome Yes MA PE fulfilled Yes No 14 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual All RA files, records and reports used for this PE must be less than three years old. Selection of this documentation, including number and type, is to be made by the assessor. 4.2 Mandatory ML4 GBT sub-indicators for registration and marketing authorization To be designated as a WLA for registration and marketing authorization, an RA must fully implement the following three ML4 indicators, as defined in the GBT: ƒ MA04.05: An advisory or scientific committee, including external experts is involved in the review of registration and marketing authorization applications (as needed) (Note that the score can be considered as implemented if the RA can demonstrate that there is an established mechanism to allow access to an advisory/scientific committee when needed). ƒ MA05.03: A summary technical evaluation report for approved registration and marketing authorization applications is published and available to the public. ƒ MA06.02: Performance indicators for registration and registration and marketing authorization activities are established and implemented. 4.3 PE indicators for registration and marketing authorization An RA seeking designation as a WLA for registration and marketing authorization must be assessed against the three PE indicators for registration and marketing authorization detailed in Tables 6 –8 below. Once each indicator has been assessed, the score should be recorded in the PE indicators scorecard (see Annex 1). 4. Registration and marketing authorization 15 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 6. PE indicators for registration and marketing authorization: PE.MA.01 fact sheet PE.MA.01 The RA has a well-established pre-submission procedure, supported by adequate guidelines and SOPs, including pre-submission meetings and regulatory/scientific advice, as applicable Description The assessor should verify that a mechanism is in place (guidelines, procedures, instructions, interpretation guides) to provide scientific and regulatory guidance to manufacturers, including pre-submission guidance in advance of MAAs, that those activities are supported by adequate guidelines and/or SOPs, and that there is data available on the effectiveness of advice in relation to the quality of subsequent MAAs. Objective This indicator aims to ensure that a pre-submission framework, supported by adequate guidelines and SOPs, exists and is communicated to manufacturers. The RA provides scientific and regulatory advice upon request. Evidence to review The assessor should ask for and review guidelines, SOPs, instructions or interpretation guides on providing scientific and regulatory guidance. In addition, the assessor should select at least three files and review: ƒ records of scientific and regulatory advice provided pre-submission meeting minutes any data on the effectiveness of advice in relation to the quality of subsequent MAAs.  References GBT sub-indicators relating to registration and marketing authorization (2) Rating scale Not implemented (NI): The RA has not developed or implemented a well-established pre- submission mechanism, system and/or procedures for the pre-submission step, including pre-submission meetings and regulatory/scientific advice to manufacturers. Partially implemented (PI): The RA has recently developed or implemented a pre- submission mechanism, system and/or procedures for the pre-submission step, including pre-submission meetings and regulatory/scientific advice to manufacturers Implemented (I): The RA has defined, implemented and published a mechanism, system and procedures for the pre-submission step, including pre-submission meetings and regulatory/scientific advice to manufacturers. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for registration and marketing authorization. However, assessors should note that pre submission meeting may not be needed for all applications and RA should clarify when this kind of meetings is foreseen. Table 7. PE indicators for registration and marketing authorization: PE.MA.02 fact sheet PE.MA.02 The RA consistently complies with the procedures and timelines for marketing authorization activities. Description The assessor should verify that the RA has established procedures and timelines related to all steps of marketing authorization activities (including scheduling of pre-meetings, product eligibility evaluation, regulatory/scientific advice, submission, screening and validation, administrative and scientific evaluation, post-approval actions); and that these are consistently respected. The assessor should further verify that, in cases where timelines do not comply with established procedures, an acceptable justification or rationale is given (to show that non- compliance is not common practice). This indicator should be assessed alongside GBT sub-indicator MA04.06 (ML3). Objective This indicator aims to ensure that the RA consistently applies the requirements of its MA function, guidelines and formal procedures, including timeline requirements. The ultimate objective is to ensure that all steps are an enabler of the MA process, not an obstacle to access to medicines. 16 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evidence to review The assessor should ask for and review: ƒ guidelines, SOPs, instructions or equivalent document establishing procedures and timelines for registration and marketing authorization activities ƒ annual reports, trend analysis, and/or key performance indicators as applicable, to assess the consistency of implementation of such procedures ƒ at least three files, selected by the assessor (to check procedures followed, including timelines, for selected steps of the registration and marketing authorization process) ƒ at least three files, selected by the assessor, that do not comply with established procedures, including timelines (to check for presence of justification or rationale showing that non-compliance is not a common practice). References GBT sub-indicators MA 04.06 and MA 06.02 (2). Rating scale Not implemented (NI): The RA does not comply with the established procedures and requirements, including timelines, in the applications reviewed. Partially implemented (PI): The RA recently established the procedures and requirements, including timelines, for different steps of the marketing authorization process, thus compliance with requirements is found only in recent applications reviewed. Implemented (I): The RA complies with the established procedural requirements, including timelines, in the applications reviewed. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for MA.  Table 8. PE indicators for registration and marketing authorization: PE.MA.03 fact sheet PE.MA.03 The RA consistently publishes its regulatory actions on a registered product. Description The assessor should verify that regulatory actions taken on registered products are regularly published and made available to the public; and that this is supported by a regulation or a guidance. Objective This indicator aims to ensure that guidelines or regulations exist to allow the RA to publish its post-approval regulatory actions on registered products; that such guidelines and regulation are publicly available; and that post-approval regulatory actions are made public timely and consistently. Evidence to review The assessor should review: • The same files selected for previous PE indicators.  • The website of the RA (to check for publication of actions, including warning letters, renewals, variations, extensions of registration and marketing authorization, withdrawals and transfer of registration and marketing authorization). References GBT sub-indicators MA01.05, MA04.02, MA04.03, MA04.06 and MA06.02 (2). Rating Scale Not implemented (NI): There is no evidence of publication and public availability of regulatory actions taken on registered products after MA is granted by the RA. Partially implemented (PI): The RA drafted a procedure for or recently started publishing regulatory actions taken on registered products after MA is granted. Implemented (I): The RA publishes and makes available to the public post-approval regulatory actions on registered products. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable" – that is, this PE indicator always applies when assessing WLAs for registration and marketing authorization.   4. Registration and marketing authorization 17 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 9. Number of sub-indicators per evaluation criteria in the PE tool for registration and marketing authorization (PE.MA.04) Evaluation criteria No. of sub-indicators Application process 3 Assessment report 2.1 Quality of the report 11 2.2 Completeness of the report 4 2.3 Scientific rigour 3 2.4 Scientific opinions/Outcomes 1 Assessment follow-up 2 4.4 PE tool for MA: expert review of MAA assessments An RA that has acceptably met all mandatory ML4 GBT sub-indicators and MA PE indicators must successfully undergo an expert review of MAA assessments in order to be designated a WLA for registration and marketing authorization. For guidance on planning, preparing, conducting, and reporting on expert review of MAA assessments see the Expert review of assessments of marketing authorization applications and clinical trial applications (Annex 2), which includes the questionnaire for assessing the performance of marketing authorization activities (see Appendix A2.1). 4.4.1 Description and objective WHO uses expert review of MAA assessments to evaluate and document the performance of registration and marketing authorization in a medical products regulatory system. This entails a review by a WHO team of assessors of a representative number of MAA assessment reports. As a general rule, PE for registration and marketing authorization should include review of 2–3 assessment reports less than three years old and preferably relating to full applications. A risk-based approach will be applied in order to select the most representative product files, also considering the product categories within the scope of WLA. In preparation for its assessment, the RA should make available the assessment reports of MAAs, along with the respective product dossiers. This will include preliminary approval from the manufacturer for the sharing of confidential information. Translations of key documents may be requested by WHO, in agreement with the RA. The WHO experts should ensure that all guidelines used as reference are relevant and were up to date at the time the assessment was conducted by the RA. 4.4.2 Evidence to review The expert review of MAA assessment reports should be planned, prepared, conducted and reported according to the guidance in the Expert review of assessments of marketing authorization applications and clinical trial applications (see Annex 2). During the evaluation, the WHO team of assessors should use the questionnaire for expert review of MAA assessments (see Appendix A2.1) in order to assess and evaluate several areas of MAA assessments, including application process, quality, completeness, scientific rigour and outcomes of the assessment report, follow-up activities. Table 9 shows the number of sub-indicators for each evaluation criteria. 4.4.3 Rating scale The results of the expert review should be written up according to the guidance in Expert review of assessments of marketing authorization applications and clinical trial applications (Annex 2). These will be combined with results from the PE indicators scorecard (Annex 1) and submitted to the WHO Secretariat together with the other PE results for registration and marketing authorization to inform the final decision on WLA listing. 4.4.4 Limitations and remarks None. 18 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5. Vigilance 5.1 Methodology for PE of vigilance PE for vigilance (VL) is designed to assess the performance of an RA in conducting vigilance in relation to medical products. In addition to meeting the eligibility criteria, PE for VL is considered fulfilled if the RA demonstrates to: a) fully implement all ML3 sub-indicators for vigilance b) fully implement the four mandatory ML4 GBT sub-indicators for vigilance (see section 5.2) c) acceptably meet the seven PE indicators for vigilance (see section 5.3) d) successfully undergo a vigilance field visit (see section 5.4 and Annex 3. Performance against the PE indicators for vigilance is evaluated using a combination of self-assessment, remote review and onsite assessment during the field visit (see Fig. 4). Whereas the GBT indicators benchmark systemic aspects of the regulatory function (for example, established legislations, organization and governance, available resources, QMS, transparency), the PE indicators are seen as a proxy of performance, through qualitative and quantitative indicators of vigilance. The vigilance field visit helps to evaluate and assess vigilance and related activities in practice and establish whether these are properly in place. The collective evidence of these tools and methodologies determines the overall performance of the vigilance function. Fig. 4. Flowchart for PE of vigilance Are eligibility criteria met? Are all ML3 and mandatory ML4 sub-indicators met? Are VL PE indicators completed by RA (self-assessment)? WHO remote review of VL indicators self-assessment Conduct VL field visit, including onsite assessment of VL indicators Start No Yes No Yes Does the conclusion of field visit & evaluation support WLA listing for VL? Report negative conclusion & outcome VL PE fulfilled Yes No End 5. Vigilance 19 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5.2 Mandatory ML4 GBT sub-indicators for vigilance WLAs must fully implement the following ML4 sub- indicators for vigilance, as defined in the GBT: ƒ VL04.03: Standard procedures exist and are implemented for enforcement of the national vigilance system. ƒ VL04.07: With respect to vigilance data, assessment of the risk-benefit balance of medical products is regularly conducted. ƒ VL04.08: Active vigilance activities, as well as proactive monitoring programmes (when needed) have been developed and implemented. ƒ VL05.02: Performance indicators for vigilance activities are established and implemented. 5.3 PE indicators for vigilance To be listed as WLA for vigilance, an RA must be assessed against the seven PE indicators for vigilance (see Tables 10 to16 below). These indicators have been designed to enable an evaluation of the baseline situation and progress in the performance of key vigilance structures and processes. As far as possible, each indicator aims to be specific, measurable, achievable, realistic and timely. Data to assess performance against the indicators can be obtained from multiple sources, including: ƒ databases, including national database (census figures, registers) and pharmaceutical databases (sales, prescription, consumption) ƒ national pharmacovigilance centres ƒ immunization programmes ƒ hospital or clinic records ƒ surveys, and ƒ peer-reviewed publications. These data may be qualitative or quantitative. In some cases, data may be required over several years to identify and track trends. Once each indicator has been assessed, the score should be recorded in the PE indicators scorecard (see Annex 1). 20 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 10. PE indicators for vigilance: PE.VL.01 fact sheet PE.VL.01 Total number of adverse drug reaction reports received in the last three years (also expressed as number of adverse drug reactions per 100 000 persons in the population). Description This indicator serves to measure vigilance activity in the setting, including the awareness and willingness of health professionals and the public to report adverse drug reactions. Valid case reports should contain four core data elements, as per International Conference on Harmonization (ICH) E2A (11): 1. reporter 2. identifiable patient 3. suspected medical product adverse reaction. Trends in this indicator enable authorities to appreciate the effectiveness of measures taken to improve quantitative reporting. Measures expressed in relation to population size allow for comparisons across and within countries. The assessor should verify: ƒ The reporting trend over the last three years and ascertain that the methodology producing the reporting statistics has remained consistent over time and that the underlying population base has remained the same. Possible reasons for major deviations between annual reporting rates, such as technical development, expansion of products to be monitored or reporting base (direct patient reporting or mandatory reporting requirements for MA-holders), promotional activities or media attention on specific safety issues. The assessor may wish to look at and review data along with figures older than three years (if available). However, for the purpose of scoring assessors are advised to stick to the criteria stated under the rating scale. This indicator should be evaluated alongside GBT sub-indicator VL04.01 (2) . Objective This indicator aims to provide a crude measure of the attention paid to medical product- related harm in society and in the healthcare system. The extent of reporting also demonstrates a general understanding that the prevention of future harm from medical products must be based on observations of current harm, and that reporting is a prerequisite for learning. Evidence to review The assessor should ask for and review data from at least the last three years, including: ƒ absolute number of reports; and ƒ number of reports per 100 000 people in the population. These data should be available through the vigilance centre where reports are received and collated. References GBT sub-indicator VL04.01 (2) and WHO pharmacovigilance indicator CP1 (12). Rating scale Not implemented (NI): Data for this indicator are not available or show an unjustified negative (that is, decreasing) trend over the last three years. Partially implemented (PI): Data for this indicator show a negative (that is, decreasing), but justified, trend over the last three years. Implemented (I): Data for this indicator are available and show a stable or positive trend (that is, increasing). For an authority to be granted WLA status for vigilance, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable" – that is, this PE indicator always applies when assessing WLAs for vigilance. 5. Vigilance 21 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 11. PE indicators for vigilance: PE.VL.02 fact sheet PE.VL.02 Percentage of annual reports satisfactorily completed and submitted to the national vigilance centre in the last three years. Sub-indicator: Percentage of the reports satisfactorily completed and submitted to the national vigilance centre, percentage of reports committed to the WHO global database of individual case safety reports (VigiBase). Description This indicator reflects the completeness of reports received by the vigilance centre and points to reporters’ understanding of, and willingness to complete, the critical elements in adverse drug reaction forms (where low values of the indicator suggest high numbers of poor-quality reports). The sub-indicator reflects the centre’s contribution to global learning about the harm caused by medical products. Submitting reports to WHO is a requirement for full members of the WHO Programme for International Drug Monitoring. According to ICH E2A (11), valid individual case safety reports include at least: ƒ one identifiable patient ƒ one identifiable reporter ƒ one reaction/event and ƒ one suspect drug. Information about the patient and reporter (name and contact details) is confidential and should not be shared with VigiBase. The assessor should verify: ƒ the proportion of reports received that fail to meet international criteria for a valid individual case safety report (identified reporter, identifiable patient, suspected medical product, suspected reaction); ƒ the efforts made to contact original reporters of incomplete reports to get missing information and turn incomplete reports into valid individual case safety reports; ƒ the proportion of valid individual case safety reports in the national vigilance database that have been shared with VigiBase over the last three years; and ƒ any reports that have not been shared with VigiBase, including reasons for non- submission. This indicator should be evaluated alongside GBT sub-indicators VL04.01, VL04.02 and VL06.03 (2). Objective This indicator aims to ensure the completeness of reports received by the vigilance centre; and the centre’s commitment to sharing that information through global platforms. Evidence to review The assessor should ask for and review annual data from each of the last three years on: ƒ the total number of individual case safety reports received ƒ the number of incomplete reports received ƒ the number of reporters submitting incomplete reports that could be contacted, allowing incomplete reports to be turned into valid reports. The indicator can then be calculated as: (Number of complete reports received during one year / Total number of reports received during the same year) × 100 In addition, the assessor should check how many completed reports were submitted to VigiBase; and then calculate the value of the sub-indicator as: (Number of complete reports submitted to VigiBase during one year / Total number of complete reports added to the national database during the same year) × 100 References GBT sub-indicators VL04.01, VL04.02 and VL06.03 (2) and WHO pharmacovigilance core process (CP) indicators CP5 and CP5a (12) 22 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Rating scale Not implemented (NI): Data for this indicator or sub-indicator are not available or show unjustified negative (that is, decreasing) trend over the last three years. Partially implemented (PI): Data for this indicator and/or sub-indicator shows negative (that is, decreasing) but justified trend over the last three years. Implemented (I): Data for this indicator and/or sub-indicator are available and show stable or positive (that is, increasing) trend over the last three years. For an authority to be given WLA status for vigilance, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for vigilance.  Table 12. PE indicators for vigilance: PE.VL.03 fact sheet PE.VL.03 Number of regulatory actions taken over the last three years as a consequence of national vigilance activities including: ƒ number of product label changes (variation) ƒ number of safety warnings on medical products to (i) health professionals and (ii) general public. ƒ number of withdrawals of medical products ƒ number of other restrictions on use of medical products Description An effective vigilance system enables regulatory authorities to issue advice and take action that ensures the safe use of medical products. The assessor should verify the regulatory actions taken as a result of safety signals from the national vigilance system over the last three years, including the number of: ƒ product labelling changes (variations) ƒ safety warnings on medical products to (i) health professionals, (ii) general public ƒ withdrawals of medical products, and ƒ other restrictions on use of medical products. For each regulatory action taken, the assessor should verify: ƒ the time taken between signal identification and action taken ƒ the contribution of domestic safety data to the action taken, relative to foreign data and information in the literature (regulatory measures taken solely on the basis of information or data from other countries should not be counted), and ƒ any impact of the action taken that was followed up by the RA, for example by measuring a decrease in reporting of the problem or in the consumption of the implicated product. In addition, the assessor should verify the number of signals not leading to any regulatory actions. This indicator should be evaluated alongside GBT sub-indicator VL04.03 (2). Objective This indicator aims to provide a measure of regulatory decisions, based on vigilance activities, that are taken to ensure safety in the use of medical products. It also measures the functionality of the vigilance centre and the interface between the centre’s activities and those of the regulatory agency. Evidence to review The assessor should ask for and review documentation on the number and characterization of regulatory actions with breakdown and differentiation between actions based on national activities/data and actions based on activities/data from other countries. Note: regulatory measures taken solely on the basis of information or data from other countries should not be counted unless the regulatory authority presents a strong justification (e.g., small population of the country). References GBT sub-indicator VL04.03 (2) and WHO pharmacovigilance indicator core outcome CO2 (12). Rating scale Not implemented (NI): Data for this indicator are not available or show unjustified negative (that is, decreasing) trend over the last three years. Partially implemented (PI): Data for this indicator shows negative trend (that is, decreasing) but justified over the last three years. 5. Vigilance 23 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Implemented (I): Data for this indicator are available and show stable or positive (that is, increasing) trend over the last three years. For an authority to be granted WLA status for vigilance, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for vigilance. It is acknowledged that in the case of small regulatory authorities (mainly countries with a small population), national vigilance data may not be sufficient to initiate regulatory actions. In such cases it is advised that regulatory authorities present data related to this sub-indicator that differentiates between actions taken based on national vigilance data and actions taken based on vigilance data from other countries. In such situations, assessors will consider regulatory actions taken based on reliance and/or recognition approach. Table 13. PE indicators for vigilance: PE.VL.04 fact sheet PE.VL.04 Percentage of registered medical products with a vigilance plan and/or a risk management strategy from the MA-holders in the country. Description This indicator contributes to measuring the enactment of the regulatory requirements for the vigilance plan and/or risk management strategy of some registered medical products. The assessor should verify: ƒ the country’s regulatory requirements for MA-holders to submit a vigilance plan or risk management strategy (including when these first entered into force); and ƒ how many products registered during the last three years submitted a vigilance plan or risk management strategy. This indicator should be evaluated alongside GBT sub-indicator VL04.08 (2). Objective This indicator aims to provide a measure of how regulatory requirements for vigilance plans and risk management strategies are enacted for specific registered medical products. Evidence to review The assessor should ask for and review records from the vigilance centre and use them to calculate this indicator as: ƒ (Number of registered products with a vigilance plan and/or a risk management strategy/ total number of registered products in the same period) × 100 References GBT sub-indicator VL04.08 (4) and WHO pharmacovigilance process indicator P10 (12) Rating scale Not implemented (NI): Data for this indicator are not available or show an unjustified negative (that is, decreasing) trend over the last three years. Partially implemented (PI): Data for this indicator show a negative (that is, decreasing) but justified trend over the last three years. Implemented (I): Data for this indicator are available and show stable or positive (that is, increasing) trend over the last three years. For an authority to be given WLA status for vigilance, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for vigilance. Table 14. PE indicators for vigilance: PE.VL.05 fact sheet PE.VL.05 Percentage of registered medical products for which periodic benefit-risk evaluation reports (PBRERs) were submitted and evaluated by the RA as stipulated in the country over the last three years Description This indicator assesses how many registered medical products PBRERs submitted by MA- holders; and to what extent these have been reviewed by the RA. 24 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The assessor should verify: ƒ the country’s regulatory requirements for MA-holders to submit PBRERs (and when these requirements first entered into force); ƒ how many products registered during the last three years had submitted and evaluated PBRERs; and ƒ how many PBRERs submitted during the last three years have still not been evaluated. This indicator should be evaluated alongside GBT sub-indicator VL04.03 (2). Objective This indicator aims to assess the enactment of regulatory requirements for MA-holders to submit PBRERs. Evidence to review The assessor should ask for and review records from the vigilance centre; and use them to calculate this indicator as: ƒ (Number of registered medical products for which PBRERs were submitted and evaluated by the RA / total number of registered medical products for which PBRER is required to be submitted and evaluated in the same time period) × 100 References GBT sub-indicator VL04.03 (2) and WHO pharmacovigilance process indicator P11 (12). Rating scale Not implemented (NI): Data for this indicator are not available or show an unjustified negative (that is, decreasing) trend over the last three years; or the absolute value of this indicator for any one year over the last three years is less than 60%. Partially implemented (PI): Data for this indicator shows a negative (that is, decreasing) but justified trend over the last three years; or the absolute value of this indicator for any one year over the last three years is less than 90%. Implemented (I): Data for this indicator show stable or positive (that is, increasing) trend over the last three years; and the absolute value of this indicator for every year over the last three years is more than 90%. For an authority to be given WLA status for vigilance, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for vigilance. Table 15. PE indicators for vigilance: PE.VL.06 fact sheet PE.VL.06 Number of registered medical products for which post-marketing safety or effectiveness studies were required and evaluated over the last three years. Description The assessor should verify: ƒ the country’s regulations on granting a conditional product registration, subject to post- marketing safety or effectiveness studies (and when these regulations entered into force) ƒ how many of the products registered over the last three years required additional safety or effectiveness studies (including what the timeframe for submission was for each one and how many were submitted on time) and ƒ how many post-marketing safety studies were evaluated and approved by the RA. This indicator should be evaluated alongside GBT sub-indicator VL01.04 (2). Objective This indicator aims to evaluate regulatory oversight of the vigilance function for post- marketing safety and effectiveness studies. Evidence to review The assessor should request records and documentation from the vigilance centre and use them to calculate: ƒ Number of registered products with post-marketing safety or effectiveness studies evaluated and approved over the last three years. References GBT sub-indicator VL01.04 (2) Rating scale Not implemented (NI): Data for this indicator are not available or show an unjustified negative (that is, decreasing) trend over the last three years. 5. Vigilance 25 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Partially implemented (PI): Data for this indicator show a justified negative (that is, decreasing) trend over the last three years. Implemented (I): Data for this indicator are available and show stable or positive (that is, increasing) trend over the last three years. For an authority to be given WLA status for vigilance, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for vigilance. Table 16. PE indicators for vigilance: PE.VL.07 fact sheet PE.VL.07 Number of good vigilance practices regulatory inspections over the last three years. Description This indicator helps determine the extent to which MA-holders comply with regulatory requirements for vigilance, including the timely submission of required reports. The assessor should verify that the regulatory system includes requirements for MA- holders to maintain vigilance systems and to regularly report safety information to the vigilance centre. Regulations should also empower the national regulatory authority to carry out vigilance inspections. The enforcement date of this legislation should be noted. In addition, the assessor should verify the existence of: ƒ an up-to-date routine vigilance inspection plan following risk-based principles ƒ a separate emergency inspection plan for crises (for example, when the safety of patients or workers in manufacturing sites are under immediate threat) and ƒ routine and emergency site inspections (including the dates and locations visited over the last three years). This indicator should be evaluated alongside GBT sub-indicators VL01.02, VL01.04 and VL04.03 (2). Objective This indicator aims to ensure regulatory oversight of the legally required good vigilance practices of MA-holders. Evidence to review The assessor should randomly select a number of inspection reports from the last three years and review them, along with other documents from the vigilance centre, including: ƒ plan for routine vigilance inspections ƒ generic plan for crisis vigilance inspections, which should ideally be adapted whenever needed and ƒ records of vigilance inspections carried out the last three years. References GBT sub-indicators VL01.04 and VL04.03 (2) and WHO pharmacovigilance core indicator CO2 (12). Rating scale Not implemented (NI): Data for this indicator are not available (for example, a good vigilance practices inspection plan is not available or has not been implemented); or show an unjustified negative (that is, decreasing) trend over the last three years. Partially implemented (PI): Data for this indicator shows a negative (that is, decreasing) but justified trend over the last three years. Implemented (I): Data for this indicator are available and show stable or positive (that is, increasing) trend over the last three years. For an authority to be given WLA status for vigilance, this indicator should at least be scored as “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable"; that is, this PE indicator always applies when assessing WLAs for vigilance. 26 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 17. Areas and number of indicators covered in the Vigilance field visit assessment questionnaire (PE.VL.08) Vaccine vigilance system Systems, structure and stakeholder coordination Detection, reporting and data management Case investigation and analysis Risk assessment and management Information, education and commu- nication with concerned groups Human and financial resources Product utilization National 10 18 13 9 9 8 6 Sub-national 4 1 1 1 3 Health facility 4 1 1* 1 3 Medicine vigilance system National 5 6 2 4 9 7 Sub-national 4 4 4 8 4 Health facility 4 6 1 9 * adverse event(s) following immunization (AEFI) reporting 5.4 PE tool for vigilance: the vigilance field visit To be listed as WLA for vigilance, in addition to being evaluated against mandatory ML4 sub-indicators and vigilance PE indicators, an RA must successfully complete a vigilance field visit. For guidance on planning, preparing, conducting and reporting on a vigilance field visit see vigilance field visit (Annex 3), which includes the vigilance field visit assessment questionnaire (Appendix A3.1). 5.4.1 Description and objective The vigilance field visit is used to evaluate and document the performance of the vigilance function in a medical products regulatory system. The activity consists of a field visit made by a WHO team of assessors to several levels of the vigilance system (national, sub-national and health facility levels) to assess how vigilance operates and performs throughout the target country. The type and number of facilities to be visited is established according to a risk-based approach. The field visit may also include onsite assessment of the PE indicators for vigilance detailed above. 5.4.2 Evidence to review A vigilance field visit should be planned, prepared, conducted and reported in accordance with the Vigilance Field Visit (see Annex 3). During the visit, WHO assessors should use the vigilance field visit assessment questionnaire (Appendix A3.1) to assess and evaluate several areas of vigilance operations. These include: vigilance systems, structures and stakeholder coordination; detection, reporting and data management; case investigation and analysis; risk assessment and management; information sharing, education and communication with concerned groups; and human and financial resources. Table 17 presents a summary of the areas and number of indicators covered. 5.4.3 Rating scale The results of the field visit should be written up in accordance with the vigilance field visits (Annex 3). These will be combined with results on the PE indicators scorecard (Annex 1) and submitted to the WHO Secretariat along with the other vigilance PE results to inform the final decision on WLA listing. 5.4.4 Limitations and remarks Not applicable. 6. Market surveillance and control 27 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6. Market surveillance and control 6.1 PE methodology for market surveillance and control PE for market surveillance and control (MC) is designed to build on the GBT by further evaluating performance within specific areas of the market surveillance and control function. As the regulatory activities and outputs covered by the market surveillance and control function are heavily inter-connected and dependent on other regulatory functions, listing for the market surveillance and control function alone is not possible, and is only granted in conjunction with WLA status for registration and marketing authorization (MA), vigilance (VL), regulatory inspections (RI), and laboratory access and testing (LT). This means that in addition to meeting the eligibility criteria for market surveillance and control, any RA applying for WLA status in MC should also meet all the PE requirements for MA, VL, RI and LT. RAs applying for listing in any of the product categories, together with other relevant functions, must acceptably meet the requirements described below. As shown in the flowchart in Fig. 5 below, PE for market surveillance and control is considered to be fulfilled if the RA, in addition to meeting the eligibility criteria, demonstrates to: a. fully implement all ML3 sub-indicators for market surveillance and control b. fully implement two mandatory ML4 GBT sub- indicators for market surveillance and control (see section 6.2) and c. acceptably meet two PE indicators for market surveillance and control (see section 6.3). Are eligibility criteria met? Are all ML3 and mandatory ML4 sub- indicators met? Assess against indicators for market surveillance and control Are PE indicators for market surveillance and control acceptably met? PE fulfilled for market surveillance and control Report negative conclusion & outcome End Start No Yes No No Yes Yes Fig. 5. Flowchart for PE of market surveillance and control 28 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The assessment against market surveillance and control indicators should in the first instance be conducted as a self-assessment, with that self- assessment then being submitted to WHO for review before a final recommendation is made on WLA listing. 6.2 Mandatory ML4 GBT sub-indicators for market surveillance and control Being a WLA for market surveillance and control requires full implementation of the following ML4 indicators, as defined in the GBT (2): ƒ MC05.02: Database for product batches that have undergone surveillance along with their relevant testing results and regulatory actions is established and periodically reviewed. ƒ MC05.03: Performance indicators for market surveillance and control activities are established and implemented. 6.3 PE indicators for market surveillance and control In order for an RA to be listed as a WLA for market surveillance and control it must – in addition to meeting the mandatory ML4 GBT sub-indicators listed above – be assessed against two PE indicators for market surveillance and control and meet the acceptability criteria for each, as detailed in Table 18 and Table 19 below. Once each indicator has been assessed, the score should be recorded in the PE indicators scorecard (see Annex 1). 6. Market surveillance and control 29 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 18. PE indicators for market surveillance and control: PE.MC.01 fact sheet PE.MC.01 The RA has developed and implemented a risked-based post-market surveillance (PMS) plan. Description Having a risk-based approach to PMS is critical for assuring medical product quality, safety and effectiveness. The assessor should verify that a PMS plan exists and is being implemented, and that it clearly considers: ƒ all the stakeholders involved in implementing the plan ƒ each entity’s roles and responsibilities ƒ how to communicate and coordinate internal and external stakeholders ƒ the resources required to implement and maintain the plan ƒ what data analysis and management system to use, and ƒ how to ensure evidence-based follow up, including regulatory actions, enforcement and communication. The assessor should further evaluate the risk factors and rationale used to inform the plan’s approach to sampling and testing in order to ensure it uses a risk-based approach. The assessor should verify the quality of the PMS plan and consistency of implementation for the last 3–5 years. The assessor should also evaluate findings that require follow- up regulatory actions and assess whether these have been enforced by the RA when warranted. Note that this indicator covers all products circulated in the market, including products distributed through online pharmacies (if applicable). This indicator should be evaluated alongside all GBT sub-indicators for substandard and falsified products, as well as all market surveillance and control sub-indicators, especially MC02.02, MC04.04, MC05.02, MC06.02, and MC06.03. In reviewing the evidence on follow- up actions, the assessor should also look at GBT sub-indicators RS04.02–04 on rapid alert and recall. Objective This indicator aims to ensure that the RA has established and implemented an efficient risked-based plan for PMS related activities at the national level. Evidence to review The assessor should ask for and review the PMS plan and outputs from it, including documented evidence of: ƒ roles and responsibilities for the plan for internal and external stakeholders ƒ procedures and records showing efficient management and coordination of timely communication among the relevant stakeholders ƒ clearly assigned resources to implement the PMS plan ƒ the risk factors and rationale used to establish the sampling and testing plan for medical products (including selection of medical products for monitoring, the type and number of samples, the location and frequency of sampling, the type of controls, scope and number of laboratory tests performed etc.) ƒ statistical analysis and data management of vigilance data generated through PMS activities, and ƒ appropriate follow-up regulatory actions (including rapid alerts and product recalls) and enforcement actions where relevant. References GBT sub-indicators MC02.02, MC04.04, MC05.02, MC06.02 and MC06.03 (2) and the Guidelines on the conduct of surveys of the quality of medicines (13). Rating scale Not implemented (NI): The RA has not developed and implemented a comprehensive risk- based PMS plan, or has established only some components of the PMS plan. Implemented (I): The RA has developed and fully implemented an effective national and risk-based PMS plan including all required components as described under description, objective and evidence. For an authority to be given WLA status for market surveillance and control, this indicator must be scored as “implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable " – that is, this PE indicator always applies when assessing WLAs for market surveillance and control.   30 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 19. PE indicators for market surveillance and control: PE.MC.02 fact sheet PE.MC.02 The RA has implemented measures to monitor, evaluate and sustain the performance of PMS-related activities. Description In order to sustain the expected performance of the national PMS plan, an RA should be able to monitor and evaluate the performance of key operational aspects and areas of the plan. The assessor should therefore verify the existence, implementation and effectiveness of a monitoring and evaluation system for the PMS plan and make sure that it tracks the plan’s implementation and enables appropriate action towards continual improvement. In particular, this system should enable the monitoring and evaluation of: a) effectiveness of PMS activities, including: – implementation of planned PMS activities – timeliness of implementation (that is, timing between when a PMS activity is planned versus when it is implemented) – use of the rapid alert and recall system (including timelines and role in regulatory decisions) – periodic review of the capability and performance of the PMS system to detect and receive PMS events and reports b) use of PMS feedback and findings and other related outcomes to adopt: – strategies, regulatory activities and regulatory decisions – timely decisions, evidence-based enforcement and resource allocations – actions for continual improvement c) transparency and timeliness of reporting to: – the public – other stakeholders. Monitoring and evaluation activities should be carried out at least once a year. They should align with the organization’s regulatory PE framework, strategic plan and commitment to continual improvement. Monitoring and evaluation outcomes should be used when considering resource and training needs, output quality, the impact on regulatory decisions and plans for continual improvement. This indicator should be evaluated alongside PE indicator PE.MC.01, as well as GBT sub- indicators MC02.02, MC04.04, MC04.05, MC04.07, MC05.02, MC05.03, MC06.02, and MC06.03 (2). Objective This indicator aims to ensure that the RA has a robust and resourced system and procedures for measuring the effectiveness and impact of PMS activities in assuring medical products consumer safety. The outcome of this indicator is particularly useful in informing resource requirements for PMS and ensuring that appropriate use is being made of additional resources (such as staff with the required expertise, competencies, tools, finances etc.) in order to maintain PMS performance and support continual improvement. Evidence to review The assessor should review the system, processes and procedures the RA implements to regularly monitor and evaluate the performance and impact of the PMS plan. This includes requesting and reviewing: ƒ Documented and implemented procedures for measuring the performance of PMS- related activities, including: – key aspects of regulation, infrastructure, RA functions and structure – resources required (including human resources, materials, and financial resources) – planned versus implemented activities and non-compliances. ƒ Documented evidence of performance indicators or outputs for the PMS-activities. ƒ Periodical (trend) reports or reviews of the performance of the PMS activities, including identified areas for improvement. ƒ Follow-up actions taken to address areas for improvement. 6. Market surveillance and control 31 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ƒ Risk and opportunity management controls on PMS activities and actions taken to manage them. ƒ Evidence of appropriate communication and coordination activities and their respective timelines related to the regulatory actions and outcomes of the PMS related activities with all relevant internal and external stakeholders. References GBT sub-indicators MC02.02, MC04.04, MC04.05, MC04.07, MC05.02, MC05.03, MC06.02, and MC06.03 (2) and the Guidelines on the conduct of surveys of the quality of medicines (13). Rating scale Not Implemented (NI): The RA has no mechanisms, system and/or procedures to measure the performance of PMS-related activities. Partially implemented (PI): The RA has identified and implemented some aspects of a system or mechanism for measuring the performance of PMS-related activities in critical areas of its functions and operations (criticality to be based on the needs of the RA with respect to the scope of WLA listing). Implemented (I): The RA has developed and implemented a robust mechanism or system with relevant procedures for measuring the performance of the market surveillance and control activities and for implementing actions to sustain its performance. For an authority to be given WLA status for market surveillance and control, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable " – that is, this PE indicator always applies when assessing WLAs. 32 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 7. Licensing establishments 7.1 Methodology for PE of licensing establishments PE of licensing establishments (LI) is designed to complement the GBT by ensuring that the process of issuing licenses is based on, and complies with, quality standards for good practices (GxP). WLA status cannot be granted for licensing establishments function alone: it is only granted in conjunction with WLA status for regulatory inspection (RI). This means that any RA applying for WLA status for licensing establishments must, in addition to meeting the eligibility criteria for licensing establishments, also meet all the PE requirements for regulatory inspection. In practice, the two functions will be assessed together. As shown in the flowchart in Fig. 7, PE of licensing establishments is considered fulfilled if, in addition to meeting the eligibility criteria, the RA demonstrates to: a. fully implement all ML3 sub-indicators for licensing establishments b. fully implement two mandatory ML4 GBT sub- indicators for licensing establishments. 7.1 Mandatory ML4 GBT sub-indicators for LI WLAs for licensing establishments must fully implement the following ML4 indicators, as defined in the GBT (2): ƒ LI05.01: A database is established and regularly updated that includes all licensing applications received, approved, refused, suspended or withdrawn, along with the essential documentation for each application. ƒ LI05.02: Performance indicators for licensing activities are established and implemented. Are eligibility criteria met? Are all PE requirements for RI acceptably met? Are all ML3 and mandatory LI ML4 sub-indicators acceptably met? LI PE fulfilled Report negative conclusion & outcome End Start No Yes No No Yes Yes Fig. 6. Flowchart for PE of licensing establishments 8. Regulatory inspection 33 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8. Regulatory inspection 8.1 Methodology for PE of regulatory inspection PE of regulatory inspection (RI) is designed to complement the GBT and ensure that the process of issuing licenses is based on, and complies with, quality standards for GxP, including good manufacturing practices, good distribution practices and good clinical practices. WLA status for regulatory inspection function alone is granted to RAs only when the licensing establishment function is not applicable (see GBT licensing establishments fact sheets (2) for additional information). Where applicable, WLA status for regulatory inspection will be granted in conjunction with WLA status for licensing establishments. This means that in addition to meeting the eligibility criteria for regulatory inspection, any RA applying for WLA status in regulatory inspection must also meet all the PE requirements for licensing establishments (see section 7 above). In practice, the two functions will be assessed together. In addition to meeting the eligibility criteria, PE of regulatory inspection is considered to be fulfilled if the RA demonstrates to: a. fully implement all ML 3 sub-indicators for regulatory inspection b. fully implement five mandatory ML4 GBT sub- indicators for regulatory inspection (see section 8.3) c. successfully undergo an observed audit for good manufacturing practices and good clinical practices (see section 8.4 and Annex 4. 8.2 Recognition of (re-)assessment by Pharmaceutical Inspection Cooperation Scheme To make the best use of available resources and expertise and avoid duplication of activities already confirmed by relevant organizations or processes, the PE process for RI recognizes other evaluation mechanisms, in particular (re-)assessment by Pharmaceutical Inspection Cooperation Scheme (PIC/S). This means that, once the RA successfully demonstrates full implementation of all mandatory ML4 GBT sub-indicators, it will be checked for pharmaceutical inspection cooperation scheme membership. If it is a PIC/S member and has been subject to a PIC/S (re-) assessment for good manufacturing practices in the last five years, it will only need to complete an observed audit for good clinical practices (see Fig. 8). The PIC/S assessment report for activities relating to good manufacturing practices, demonstrating compliance to PIC/S standards, should be made available by the candidate WLA. 8.3 Mandatory ML4 GBT sub-indicators for regulatory inspection WLAs for regulatory inspection must fully implement the following ML4 indicators, as defined in the GBT: ƒ RI05.01: A database is established and regularly updated of all establishments which may be subject to inspection, along with their relevant regulatory decisions (certification and/or enforcement activities). ƒ RI05.03: Inspection reports are subjected to a regular and robust review by experts other than the designated inspection team. ƒ RI05.04: Inspection data and outcomes are systematically evaluated or interpreted. ƒ RI05.05: Performance indicators for regulatory inspection activities are established. ƒ RI06.02: The updated list or database of all inspected facilities along their regulatory decisions, actions and enforcement activities, is regularly published and publicly available. 8.4 PE tool for regulatory inspection: observed audit for GxP As part of the PE of regulatory inspection, an RA must complete an observed audit for GxP (namely, good manufacturing practices and good clinical practices). For guidance on planning, preparing, conducting and reporting on a GxP observed audit see the GxP observed audit for assessing the performance of the regulatory inspection function (Annex 4), which includes the inspectors’ evaluation form (Appendix A4.1), and the Pharmaceutical inspection cooperation scheme audit checklist: interpretation guide (14). 34 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Fig. 7. Flowchart for PE of regulatory inspection Are eligibility criteria met? Are PE requirements for LI under assessment and acceptably met? Are all ML3 and mandatory ML4 sub-indicators met? Is the RA a PIC/S member? Prepare and conduct observed audit for GMP Start No Yes No Yes Prepare and conduct observed audit for GCP Report negative conclusion & outcome Do the conclusions of observed audits support WLA listing for RI? NoHas the RA had a PIC/S (re-)assessment in the past five years? No Get PIC/S assessment data for GMP Yes No Yes RI PE fulfilled Yes End Yes No 8. Regulatory inspection 35 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8.4.1 Description and objective WHO uses an observed audit of good manufacturing practices and good clinical practices inspections to document and evaluate the performance of the regulatory inspection function in a medical products regulatory system. This entails a WHO team of experts observing routine inspections at authorized sites, carried out according to national references, regulations and guidelines. National references are expected to be at least equivalent to WHO GxP guidelines or other recognized guidelines. As a general rule, the PE exercise for regulatory inspection should include three audits for each GxP type (that is, three for good manufacturing practices and three for good clinical practices). Fewer audits may be justified using a risk-based approach (for example, because a limited number of sites are subject to GxP regulatory inspection; or the team of observers has former experience with the RA; or the RA is at an advanced stage of becoming a pharmaceutical inspection cooperation scheme member). 8.4.2 Evidence to review The observed audit for GxP should be planned, prepared, conducted and reported according to the GxP observed audits for assessing the performance of the regulatory inspection function (see Annex 4). During the audit, the WHO team of observers should use the observed audit inspectors’ evaluation form (see Appendix A4.1) to assess and evaluate five indicators through a set of sub-indicators. Table 20 shows the number of sub-indicators for each indicator. Table 20. Main areas covered by the Observed Audit Inspectors’ Evaluation Form: PE.RI.01 Indicators of the Observed Audit Inspectors’ Evaluation Form No. of sub- indicators 1. Inspection preparation process 11 2. Inspection conduct process 17 3. Inspection reporting process 9 4. Inspectors’ technical competency 6 5. Inspectors’ skills and attitude 11 8.4.3 Rating scale Results of the observed audit should be written up according to the GxP observed audit (Annex 4). These will be combined with results from the PE indicators scorecard (Annex 1) and submitted to the WHO Secretariat together with the other Regulatory Inspection PE results to inform the final decision on WLA listing. 8.4.4 Limitations and remarks None. 36 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 9. Laboratory testing 9.1 Methodology for PE of laboratory testing PE of laboratory testing (LT) is designed to assess laboratories that carry out medical product testing for the RA including chemical, biological, microbiological and any other relevant laboratories. This includes both national control laboratories (NCLs) and external laboratories that perform tests on behalf of the RA. The RA can apply to be a WLA for laboratory testing for one or more product categories. As shown in the flowchart in Fig. 9, PE of laboratory testing is considered fulfilled if the national control laboratory or external laboratory applying for WLA status, in addition to meeting the eligibility criteria demonstrates to: a. fully implement all ML 3 sub-indicators for laboratory testing b. fully implement three mandatory ML4 GBT sub- indicators for laboratory testing (see section 7.2) c. acceptably meet one laboratory testing performance evaluation indicator (see section 7.3) d. Successfully undergo an expert review of laboratory testing activities (see section 7.4 and Annex 5). Fig. 8. Flowchart for PE of laboratory testing Report negative conclusion & outcome End Is the laboratory: - a WHO PQ laboratory (for medicines, - a WHO-contracted laboratory (for vaccines), or - an OMCL attested by EDQM? Are eligibility criteria met? Are all ML3 and mandatory ML4 sub-indicators met? Assess against LT PE indicator Start No Yes No Are LT PE indicators acceptably met? Complete review of LT PE tool No LT PE fulfilled RI PE fulfilled Yes Yes No No No No 9. Laboratory testing 37 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 9.1.1 Recognition mechanisms To make the best use of available resources and expertise and avoid unnecessary duplication of activities already confirmed by relevant organizations or processes, the PE for laboratory testing recognizes other evaluation mechanisms. In particular, as shown in the flowchart in Fig. 9, the following laboratories can be considered to acceptably meet the PE requirements for being a WLA for laboratory testing, without undergoing further assessment through the PE for laboratory testing: ƒ WHO-prequalified laboratories (for medicines) ƒ WHO-contracted laboratories (for vaccines) ƒ Official medicines control laboratories attested by European Directorate for the Quality of Medicines & Healthcare. The accreditation against ISO/IEC 17025: 2017 General requirements for the competence of testing and calibration laboratories, while acknowledged to be valuable, is not formally recognized under the PE for laboratory testing. This means that even if a laboratory is ISO-accredited, it will still have to fully meet all eligibility criteria, as well as fully implement all mandatory ML4 sub- indicators for laboratory testing and complete the PE tool for laboratory testing. 9.2 Mandatory ML4 GBT sub-indicators for laboratory testing WLAs for laboratory testing must fully implement the following ML4 sub-indicators, as defined in the GBT: ƒ LT08.01: Updated database of all medical products batches that have undergone quality testing. ƒ LT08.03: Regular participation in proficiency schemes, collaborative studies and inter- laboratory comparisons. ƒ LT08.04: Performance indicators for laboratory testing activities are established. 9.3 PE indicator for laboratory testing For an RA to be listed as WLA for laboratory testing it must be assessed against one indicator, PE.LT.01, as detailed in Table 21 below; once the indicator has been assessed, the score should be recorded in the PE indicators scorecard (see Annex 1). 38 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 21. PE indicators for laboratory testing: PE.LT.01 fact sheet PE.LT.01 The laboratory defined an appropriate mechanism of external control to provide objective evidence of overall laboratory performance and competence in an ongoing manner Description There should be an analysis of the tests and analysts available in the laboratory, with a scheme in place to monitor performance and competence at a reasonable frequency (for example once a year), with justification of the strategy and frequency implemented. Where possible, tools to monitor performance should include participation in proficiency testing schemes organized by recognized providers (such as WHO, European Directorate for the Quality of Medicines & Healthcare, National Institute for Biological Standards and Control, National Institutes for Food and Drug Control). However, it is not always possible for laboratories to find a suitable proficiency testing scheme. In such cases other approaches may be used. For example, external mechanisms such as: collaborative studies organized by reference laboratories; collaborative studies between the national control laboratory and academia’s accredited laboratories or between the national control laboratory and manufacturer; inter-laboratory comparisons according to pre-determined conditions with exchange of samples to be tested; or independent comparison with another national control laboratory or manufacturer for the same batch of the product. Acceptable alternatives also include enhanced internal quality control procedures. For these, evaluation should use internally generated data. Note that for some non-complex testing methods individual proficiency testing schemes are of no added value as the main operational steps are covered by other individual proficiency testing schemes (for example, weighing operations during high performance liquid chromatography) or are related to the qualification of equipment (for example, friability). The laboratory should consider the availability of proficiency testing schemes based on product stream: if none are available, the laboratory should as a minimum implement blind testing and/or comparison of results with the manufacturer, must provide evidence that no proficiency testing scheme is available and must justify the alternative means used for assessing proficiency. For specific tests, the frequency of external control should be determined according to the number of tests performed each year, the number of analysts involved and their general level of experience, analyst turnover, the availability of reference standards, the criticality of the testing results, complexity of the technique, etc. External control should preferably be carried out at least once a year for all the laboratory’s relevant testing (“regular” participation). After receiving the results of the external control mechanisms, the laboratory should check the consistency of measurements’ accuracy, precision and uncertainty, and follow this up according to the QMS. If the result obtained is unsatisfactory, or on the borderline of unsatisfactory, the laboratory should investigate root causes, identify possible consequences, and define appropriate corrective actions. The investigation should cover various factors that can influence results. Objective This indicator aims to ensure that the laboratory has established an external control mechanism to evaluate the appropriateness and consistency of its laboratory activities. Evidence to review The assessor should ask for and review: ƒ Continuity of participation (percentage or ratio) in proficiency testing scheme for specific type of testing over a defined period of time (for example, past five years). ƒ Percentage or ratio of satisfactory results from proficiency testing scheme (satisfactory against total number of the same type of studies performed). ƒ Effective root-cause analysis performed (satisfactory against total number) following proficiency testing schemes. ƒ Effectiveness of corrective measures implemented following proficiency testing schemes. ƒ Number of external control mechanisms engaged in for specific type of testing per year. ƒ Requalification/education of the analysts after deviations detected during proficiency testing scheme to total pool of analysts’ qualifications for laboratory testing. ƒ Witten procedures to review of resources required to perform the proficiency testing scheme testing properly per resource type, or the number of initiatives taken (e.g.: method/ technique compliance with up-to-date guidelines, norms and standards, new equipment/ lab utensils introduction, software upgrades, environmental conditions interventions). 9. Laboratory testing 39 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ƒ Documented improvements related to specific testing implemented during proficiency testing scheme in a defined period of time (e.g.: past three years). References GBT sub-indicator LT08.03 (2); the Laboratory quality management system handbook (LQMS) (15); Alternatives to proficiency testing schemes (PTS) (16); Proficiency testing: guidelines on the level of participation and evaluation of performances in proficiency testing activities in the context of accreditation assessments (17). Rating scale Not implemented (NI): The laboratory(ies) has no mechanisms, system and procedures to ensure that the laboratory has established an external control mechanism to evaluate the appropriateness and consistency of its laboratory activities. Partially implemented (PI): The laboratory(ies) has identified and implemented a system or mechanism for monitoring and evaluating the appropriateness and consistency of its laboratory activities through external control but this is not applied in a consistent manner. Implemented (I): The laboratory(ies) has established mechanisms, system and procedures to ensure that the laboratory has established an external control mechanism to evaluate the appropriateness and consistency of its laboratory activities. For an authority to be given WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as “not applicable” – that is, this PE indicator always applies when assessing candidate WLAs for laboratory testing 9.4 PE tool for laboratory testing: expert review of laboratory activities Once an RA has acceptably met all mandatory ML4 GBT sub-indicators and the laboratory testing performance indicator, it must successfully undergo an onsite expert review of laboratory activities in order to be listed as a WLA for laboratory testing. See the Expert review of laboratory testing activities (Annex 5) for guidance on planning, preparing, conducting and reporting on the expert review of laboratory activities, including assessment questionnaires reported as Part A (QMS), Part B (Competence), Part C (Reporting) (Appendix A5.1). 9.4.1 Description and objective WHO uses the expert review of laboratory activities to document and evaluate the performance of the laboratory testing function in a medical products regulatory system. This entails observation by a WHO team of experts of routine analysis conducted in the National Control Laboratory and/or outsourced laboratory(ies), as applicable. PE for laboratory testing comprises a combination of evaluation activities based on: i) the checklist used to support self-inspections and peer audits as part of the WHO Prequalification programme (WHO Good practices for pharmaceutical quality control laboratories), and ii) the WHO Global Framework for Competency of Regulators, designed to ensure that the outcomes of a laboratory’s activities are documented according to a QMS, including quality and completeness of reports, scientific rigour of approaches, compliance with regulatory requirements and data integrity. In preparation for their assessment, laboratories should make available the full reports of any recognized international standard (including WHO prequalification, International Organization for Standardization ISO 17025 or similar) inspections/ audits (including corrective action plan). 9.4.2 Evidence to review The expert review of laboratory activities should be planned, prepared, conducted and reported in accordance with the Expert review of laboratory testing activities (see Annex 5). During the evaluation, the WHO team of assessors should use the questionnaire for the expert review of laboratory testing activities (see Appendix 5.1) to assess and evaluate several areas of laboratory activities, mainly related to QMS system, staff competency, and quality of testing reports. Table 22 summarizes the number of items covered. 40 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 22. Criteria and number of items covered in the PE tool for laboratory testing: PE.LT.02 Requirements/Criteria  No. of items  Part A QMS evaluation 11 Part B Competency assessment 47 B.1 Sample receipt and record initiation 7 B.2 Reagents/sample and standard preparation 8 B.3 Use of equipment 9 B.4 Running of sample analysis 9 B.5 Reporting of analysis 11 B.6 Storage of records 1 B.7 Stock management 1 B.8 Reviewing quality and technical records 1 Part C Analytical reports 23 C.1 Quality of the report 2 C.2 Completeness of the report 5 C.3 Scientific rigour 5 C.4 Compliance 1 C.5 Data Integrity 10 9.4.3 Rating scale The results of the expert review should be written up in accordance with the guidance for assessing the performance of laboratory testing activities (see Annex 5). These will be combined with results from the PE indicators scorecard (see Annex 1) and submitted to the WHO Secretariat together with the other PE results for laboratory testing to inform the final decision on WLA listing.  9.4.4 Limitations and remarks Not applicable. 10. Clinical trial oversight 41 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 10. Clinical trial oversight 10.1 Methodology for PE of clinical trials oversight PE of clinical trials oversight is designed to assess oversight of clinical trials for specific products and activities, including: ƒ Products: new chemicals entities (new medicines), multisource/generic medicines, vaccines, biotherapeutics, similar biotherapeutic products. ƒ Processes: clinical trial listing in a registry; submission and assessment of clinical trial application and clinical trial protocol approval, registration or licensing; transparency and structure; post-approval safety and compliance and reporting (pharmacovigilance, that is safety monitoring system, including adverse event assessment and safety reporting during trials); effectiveness of the clinical trials oversight process. In addition to meeting the eligibility criteria, PE of clinical trials oversight is considered fulfilled if the RA demonstrates to: a. fully implement all ML3 sub-indicators for clinical trials oversight b. fully implement two mandatory ML4 GBT sub- indicators for clinical trials oversight (see section 10.2) c. acceptably meet three PE indicators for clinical trials oversight (see section 10.3) d. successfully undergo an expert review of clinical trials application assessments (see section 10.4). All RA files, records and reports used for this performance evaluation must be less than three years old. Selection of this documentation, including number and type, is to be done by the assessor. 10.2 Mandatory ML4 GBT sub-indicators for clinical trials oversight WLAs for clinical trials oversight must fully implement the following ML4 indicators, as defined in the GBT: ƒ CT04.01 RA has access to an advisory committee for review of clinical trial applications and post‐ approval safety and compliance issues. ƒ CT06.02 Performance indicators for clinical trial oversight activities are established and implemented. 10.3 PE indicators for clinical trials oversight To be listed as WLA for clinical trial oversight, an RA must be assessed against three clinical trials PE indicators, as detailed below. Once each indicator has been assessed, its score should be recorded in the PE indicators scorecard (Annex 1). 42 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Are eligibility criteria met? Are all ML3 and mandatory ML4 sub-indicators met? Assess NRA against CT PE indicators Are all CT PE indicators acceptably met? Conduct expert review of CT Application Assessments Start No Yes No No Yes Yes Are all sections of the expert review acceptably met? CT PE fulfilled Report negative conclusion & outcome End No Yes Fig. 9. Flowchart for PE of clinical trials oversight 10. Clinical trial oversight 43 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table 23. PE indicators for clinical trials oversight: PE.CT.01 fact sheet PE.CT.01 The RA consistently complies with the timelines established in its guidelines and SOPs for assessing clinical trial applications. Description The assessor should verify that the RA has established timelines for assessing clinical trial applications, including post-approval procedural requirements, and that these are consistently respected. The assessor should further verify that, in cases where timelines do not comply with guidelines, there is an acceptable justification or rationale given (to show that non- compliance with timelines is not common practice). The assessment of this indicator should be conducted alongside GBT sub-indicator CT06.04. In particular, assessors should ensure that: ƒ Timelines for routine and non-routine clinical trial applications are established following well-defined criteria and are publicly available. ƒ Regulation and guidelines exist to ensure that clinical trial applications are consistently processed and assessed according to prescribed timelines. ƒ An internal tracking system for active monitoring of compliance with published timelines is used and outcomes are documented. Objective This indicator aims to ensure that the RA efficiently and consistently evaluates in a timely way a) clinical trial applications and b) post-approval clinical trial activities. When overall timelines are short, it is not expected that timeframes are established for each individual step of the clinical trial application assessment, as this would unnecessarily overload the process without tangible added value. The ultimate objective is to ensure that this step is an enabler, rather than an obstacle, to the whole clinical trial process. Evidence to review The assessor should ask for and review: ƒ guidelines, SOPs, instructions or equivalent document establishing timelines for assessing clinical trial applications ƒ at least three files selected by the assessor (to check timelines) ƒ at least three files selected by the assessor that do not comply with timeline requirements (to check for justification or rationale). ƒ In addition, the assessor should ask for and review: ƒ evidence of an internal monitoring system for tracking compliance with timelines. References GBT sub-indicators CT01.02 and CT06.04 (2). Rating Scale Not Implemented (NI): In the reviewed applications, the RA does not comply with established timelines for evaluation of clinical trial applications. Partially Implemented (PI): The RA recently established timelines for assessing clinical trial applications, and so compliance with requirements is only found in recent applications reviewed. Implemented (I): In the reviewed applications, the RA complies with the established timelines for evaluation of clinical trial applications. For an authority to be granted WLA status, this indicator must be scored as “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable" – that is, this PE indicator always applies when assessing WLAs for clinical trial oversight. Table 24. PE indicators for clinical trials oversight: PE.CT.02 fact sheet PE.CT.02 The list of clinical trial applications, including their current status, is publicly available or recorded in a domestic or international database. Description The assessor should verify that all clinical trial applications, and their current status, are listed in the public domain and that this is supported by a regulation or guideline. 44 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The assessor should also verify that regulations, guidelines or similar documents are in place requiring all clinical trial applications (including approved, terminated, suspended, withdrawn and other applications) to be listed in an easy-to-access domestic or international database and that those lists are regularly updated. Established guidelines should include guidance on what information should be listed and what publication mechanism should be used. Objective This indicator aims to ensure that guidelines or regulations are in place to mandate the RA to publish, and make publicly available, information about the clinical trial applications it receives, including their status. Evidence to review The assessor should ask for and review:  ƒ regulations, guidelines or equivalent documents on listing of clinical trial applications ƒ written guidance or SOPs on format and information of listing ƒ publicly available domestic and international databases with clinical trial listings ƒ at least three clinical trial application files, selected by the assessor (to check for compliance). References GBT sub-indicator CT05.02 (2) Rating scale Not Implemented (NI): The RA has not developed or implemented a well-established procedure for regular publication of information about clinical trial applications and their status in a domestic or international database; or it does not comply with the procedure in the applications reviewed. There is no evidence of regular and consistent publication of information about clinical trial applications and their status. Partially Implemented (PI): The RA recently developed a procedure for regular publication of the clinical trial applications and their status in a domestic or international database, thus compliance with requirements is found only in recent applications reviewed. Implemented (I): The RA has defined and implemented a well-established procedure for regular publication of the clinical trial applications and their status in a domestic or international database, and it complies with that procedure in the applications reviewed. The RA regularly and consistently publishes clinical trial applications and their status and makes this available to the public. For an authority to be granted WLA status, this indicator must be scored as at least “partially implemented”. Limitations and remarks This sub-indicator cannot be scored as "not applicable " – that is, this PE indicator always applies when assessing WLAs for clinical trials oversight. Table 25. PE indicators for clinical trials oversight: PE.CT.03 fact sheet PE.CT.03 The RA provides regulatory support to clinical researchers and sponsors to assist in the development of new therapies for patients. Description The assessor should verify that the RA gives clinical researchers and sponsors specific support. The RA should support initiatives by stakeholders (such as ethics committees) to improve the quality of clinical trials. The RA should also be actively involved in initiatives to create a positive regulatory environment for the development of new or improved therapies and should regularly review the effectiveness and impact of its clinical trials registration or license system on the development of new therapies, taking action towards improvement where possible. Objective This indicator aims to ensure that there are mechanisms in place to review how effective the clinical trials registration or license system is in supporting the development of safe, effective and quality-assured new therapies. Evidence to review The assessor should ask for and review documentation that can help answer the following questions, as applicable: 1. Is regulatory support for the development of clinical research in the country an objective of the agency? What actions are being taken in that regard? 2. Is regulatory support given to the development of new medicines/combination 10. Clinical trial oversight 45 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual therapies/advanced therapies? In what way? Is there particular regulatory support for first-in-human trials? 3. Has support been given to researchers in terms of risk adaptation in clinical trials? 4. Is the agency involved in regional or international harmonized procedures or initiatives for assessing multinational clinical trials? 5. Is there provision of direct regulatory support and advice to industry and academic sponsors? How is this done? 6. What is the extent of interaction and co-operation with national and local ethics committees? What particular achievements have resulted from this? 7. Are there reviews of the agency’s contribution to an efficient regulatory environment for national clinical research? What improvements have been made? References GBT sub-indicator CT01.03 (2) Rating Scale Not Implemented (NI): The RA does not take an active role in the development of new therapies in the country. Partially implemented (PI): The RA has recently developed or implemented a mechanism to provide regulatory support to clinical researchers and sponsors and assist in the development of new therapies for patients. Implemented (I): The RA is involved in national and international initiatives to create a positive regulatory environment for the development of new or improved therapies. There are strong links and cooperation with ethics committees, including joint initiatives to improve the quality and extent of clinical trials. Advice and assistance are provided, especially to academic sponsors. Where necessary and possible, changes have been made to the clinical trials registration or license system, and its effectiveness and impact on the development of new therapies for the country is regularly reviewed. For an authority to be given WLA status, this indicator must be scored as “partially implemented”. If the RA has been assessed throughout the Benchmarking of European Medicines Agencies IV cycle and showed to be rated as 4 or 5 in key performance indicator 12.1, the RA is considered to acceptably fulfil this indicator. Limitations and remarks This sub-indicator cannot be scored as "not applicable " – that is, this PE indicator always applies when assessing candidate WLAs for clinical trials oversight. 46 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 10.4 PE tool for clinical trial oversight: expert review of clinical trials application assessments Once an RA has acceptably met all mandatory ML4 GBT sub-indicators and PE indicators, the RA must undergo an expert review of clinical trial application assessments in order to be listed as WLA for clinical trials oversight. For guidance on planning, preparing, conducting, and reporting on expert review of CTA assessments see the Expert review of assessments of marketing authorization applications and clinical trial applications (Annex 2), which includes the questionnaire for assessing the performance of clinical trial activities (see Appendix A2.2). 10.4.1 Description and objective WHO uses expert review of clinical trials application assessments to evaluate and document the performance of the clinical trial oversight function in a medical products regulatory system. This entails a WHO team of assessors undertaking a review of a representative number of clinical trial application assessment reports. As a general rule, PE of clinical trials oversight should involve reviewing two or three application assessments less than three years old. A risk- based approach will be used to select the most representative product files, while also considering the product categories in the scope of the WLA. In preparation for the assessment, the RA should make available the clinical trial application assessment reports and the respective product dossiers. This will include preliminary approval from the sponsor for the sharing of confidential information. Translations of key documents may be requested by WHO, in agreement with the RA. The WHO experts should ensure that all guidelines used as reference materials are relevant and were up to date at the time the assessment was conducted by the RA. 10.4.2 Evidence to review The expert review of clinical trials application assessments should be planned, prepared, conducted and reported in accordance with the Expert review of assessments of marketing authorization applications and clinical trial applications in Annex 2. During the evaluation, the WHO team should use the questionnaire for assessing the performance of clinical trial activities provided in Appendix A2.2, to assess and evaluate various aspects of clinical trial application assessments, including the application process, the quality, completeness, scientific rigour and outcomes of the assessment report, and follow- up activities. The number of sub-indicators relating to each evaluation criteria is shown in Table 26. Table 26. Number of sub-indicators per evaluation criteria in the PE tool for clinical trials oversight (PE.CT.04) Evaluation Criteria No. of sub-indicators Application process 3 Assessment report 2.1 Quality of the report 11 2.2 Completeness of the report 4 2.3 Scientific rigour 6 2.4 Scientific opinions/outcomes 1 Assessment follow-up 2 10.4.3 Rating scale The results of the expert review should be written up according to the guidance in the Expert review of assessments of marketing authorization applications and clinical trial applications (see Annex 2). These will be combined with results from the PE indicators scorecard (see Annex 1) and submitted to the WHO Secretariat together with the other PE results for clinical trial oversight to inform the final decision on WLA listing. 10.4.4 Limitations and remarks None. 11. RA lot release 47 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 11. RA lot release 11.1 Methodology for PE of lot release PE of lot release is designed to assess laboratories that perform lot release for a regulatory authority, in vaccine-producing countries only. This includes both national control laboratories and external laboratories that perform tests on behalf of the RA. In all cases any national control laboratory or external laboratory applying for WLA status for lot release must, in addition to meeting the eligibility criteria, also ensure that all the laboratory testing (LT) activities performed for lot release (LR) meet the PE requirements for laboratory testing (see section 9 above). PE of lot release is considered fulfilled if, in addition to meeting the eligibility criteria, the national control laboratory or external laboratory demonstrates to: a. fully implement all ML 3 sub-indicators for lot release b. fully implement one mandatory ML4 GBT sub- indicator for lot release (see section 11.2). 11.2 Mandatory ML4 GBT sub-indicators for lot release ƒ LR06.04: Performance indicators for national lot release activities are established and implemented. Fig. 10. Flowchart for PE of lot release Are eligibility criteria met? Are all PE requirements for laboratory testing acceptably met? Are all ML3 and mandatory ML4 sub-indicators for lot release acceptably met. Start No Yes No Yes PE of lot release is fulfilled Report negative conclusion & outcome End No Yes 48 Manual for performance evaluation of regulatory authorities seeking designation as WHO-listed authorities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual References 1. Operational guidance for evaluating and publicly designating regulatory authorities as WHO-listed authorities. Geneva: World Health Organization; 2023. 2. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory systems of medical products, revision VI. Geneva: World Health Organization; 2021 (https://apps.who.int/ iris/handle/10665/341243, accessed 29 June 2021). 3. Good regulatory practices in the regulation of medical products. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fifty-fifth report. Geneva: World Health Organization; 2021: Annex 11 (WHO Technical Report Series, No. 1033; https://apps.who.int/iris/ handle/10665/340323/, accessed 6 March 2023). 4. Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans. Geneva: World Health Organization; 2021, in press. 5. WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce: questions and answers (Q&A): adopted guidance. WHO Drug Information. 2016;30(3):376–88 (https://apps.who.int/iris/ handle/10665/331012, accessed 29 March 2023). 6. Guidelines on the implementation of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fifty-fifth report. Geneva: World Health Organization; 2021: Annex 9 (WHO Technical Report Series, https://apps.who.int/iris/ handle/10665/340323, accessed 29 March 2023). 7. Resolution WHA50.3. WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce. Geneva: World Health Organization; 1997 (https://apps.who.int/iris/ handle/10665/179638, accessed 29 March 2023). 8. Towards a global competency framework for regulators of medical products. WHO Drug Information. 2019;33(1):6–9 (https://apps.who. int/iris/handle/10665/330894, accessed 17 May 2023). 9. WHO guideline on the implementation of quality management systems for national regulatory authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fifty-fourth report. Geneva: World Health Organization; 2020: Annex 13 (WHO Technical Report Series, No. 1025; https://apps.who.int/iris/ handle/10665/331814, accessed 17 May 2023). 10. Implementing quality management systems in national regulatory authorities: examples and practices. Geneva: World Health Organization; 2021 (https://apps.who.int/iris/ handle/10665/341942, accessed 17 May 2023). 11. ICH E2A. Clinical safety data management: definitions and standards for expedited reporting – scientific guideline. London: European Medicines Agency; 1995 (https://www.ema. europa.eu/en/ich-e2a-clinical-safety-data- management-definitions-standards-expedited- reporting-scientific). 12. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015 (https://apps.who.int/ iris/handle/10665/186642, accessed 30 June 2021). 13. Guidelines on the conduct of surveys of the quality of medicines. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fiftieth report. Geneva: World Health Organization; 2016: Annex 7 (WHO Technical Report Series, No. 996; https://apps. who.int/iris/handle/10665/255338, accessed 1 July 2021). 14. Pharmaceutical inspection cooperation scheme audit checklist: interpretation guide. Geneva: Pharmaceutical Inspection Convention/ Cooperation Scheme; 2020 (PS/W 31/2019; https://picscheme.org/docview/3549, accessed 1 July 2021). 15. Assessment – external quality assessment. In: Laboratory quality management system: handbook. Geneva: World Health Organization; 2011:113–24 (https://apps.who.int/iris/ handle/10665/44665, accessed 17 May 2023). References 49 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 16. Alternatives to proficiency testing schemes (PTS). Strasbourg: European Directorate for the Quality of Medicines; 2020 (PA/PH/QMCL (15) 50 R8; https://www.edqm.eu/en/d/1213806, accessed 18 August 2023). 17. Proficiency testing: guidelines on the level of participation and evaluation of performances in proficiency testing activities in the context of accreditation assessments. Brussels: Belgian Organization for Accreditation; 2011 (BELAC 2-106 Rev. 1; https://economie.fgov.be/sites/ default/files/Files/Publications/files/Belac-EN/2- 106-EN.pdf, accessed 12 January 2021).

Annex 1. PE indicators scorecard 52 PE indicators scorecard Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual PE indicators scorecard The PE indicators scorecard is designed to enable assessment against PE indicators in all functions, as applicable. In advance of the performance evaluation, the regulatory authority (RA) must complete the scorecard using the template below, providing a justification of each score in the relevant column and submitting it to the WHO Secretariat as part of the assessment of the performance of medical products. For full guidance on objectives and evidence to be provided in support of the applied regulatory practices, please refer to the relevant table (also referred to as “fact sheets”), available for each function listed in the Manual for the performance evaluation of regulatory authorities seeking designation as WHO- listed authorities (“The PE manual”). The WHO team will review the self-assessment and may adjust the scoring by adding a justification under the WHO team column, without altering the self-assessment justification entered by the RA. The WHO team will also provide an overall conclusion and recommendation for or against WLA listing. The results from the amended scorecard will be combined with results obtained through the PE tools to inform the final decision on WLA listing. Country: __________________ Institution: __________________ Dates: __________________ Assessors: __________________ ID Indicator Scoring Tick one box, as applicable RA input For self-assessment WHO assessor input For formal assessment PE.RS.01 The RA participates in the WHO certification scheme on the quality of pharmaceutical products moving in international commerce and issue certificate of pharmaceutical product. ☐ Not implemented ☐ Implemented PE.RS.02 The RA has established an effective competency framework. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.RS.03 The RA has implemented measures to monitor, evaluate and sustain the performance of the quality management system (QMS). ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.RS.04 The RA has a mechanism, supported by adequate regulations, guidelines and/or standard operating procedures (SOPs), for sharing technical information or any other non-public information about its regulatory decisions, with other authorities. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.MA.01 The RA has a well-established pre-submission procedure, supported by adequate guidelines and SOPs, including pre- submission meetings and regulatory/scientific advice, as applicable. ☐ Not implemented ☐ Partially implemented ☐ Implemented 53 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Indicator Scoring Tick one box, as applicable RA input For self-assessment WHO assessor input For formal assessment PE.MA.02 The RA consistently complies with the procedures and timelines for marketing authorization activities. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.MA.03 The RA consistently publishes its regulatory actions on a registered product. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.VL.01 Total number of adverse drug reaction reports received in the last three years (also expressed as number of adverse drug reactions per 100 000 persons in the population). ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.VL.02 Percentage of total annual reports satisfactorily completed and submitted to the national vigilance centre in the last three years Sub-indicator: Percentage of the reports satisfactorily completed and submitted to the national vigilance centre, percentage of reports committed to VigiBase ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.VL.03 Number of regulatory actions taken over the last 3 years as a consequence of national vigilance activities including: a. number of product label changes (variation) b. number of safety warnings on medical products to (i) health professionals and (ii) general public c. number of withdrawals of medical products d. number of other restrictions on use of medical products ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.VL.04 Percentage of registered medical products with a vigilance plan and/or a risk management strategy from the marketing authorization-holders in the country ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.VL.05 Percentage of registered medical products for which periodic benefit-risk evaluation reports (PBRERs) were submitted and evaluated by the RA as stipulated in the country over the last three years ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.VL.06 Number of registered medical products for which post- marketing safety or effectiveness studies were required and evaluated over the last three years ☐ Not implemented ☐ Partially implemented ☐ Implemented 54 PE indicators scorecard Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Indicator Scoring Tick one box, as applicable RA input For self-assessment WHO assessor input For formal assessment PE.VL.07 Number of good vigilance practices regulatory inspections over the last three years ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.MC.01 The RA has developed and implemented a risk-based post- market surveillance (PMS) plan ☐ Not implemented ☐ Implemented PE.MC.02 The RA has implemented measures to monitor, evaluate and sustain the performance of PMS-related activities. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.LT.01 The laboratory defined an appropriate mechanism of external control to provide objective evidence of overall laboratory performance and competence in an ongoing manner ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.CT.01 The RA consistently complies with the timelines established in its guidelines and SOPs for assessing clinical trial oversight applications. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.CT.02 The list of clinical trial oversight applications, including their current status, is publicly available or recorded in a domestic or international database. ☐ Not implemented ☐ Partially implemented ☐ Implemented PE.CT.03 The RA provides regulatory support to clinical researchers and sponsors to assist in the development of new therapies for patients. ☐ Not implemented ☐ Partially implemented ☐ Implemented Overall conclusion and recommendation by the WHO team: Please provide an overall conclusion of the PE indicators considering the guidance provided under each of the indicators (for example, in terms of acceptance for the purpose of WLA designation) and any consequent recommendation Annex 2. Expert review of assessments of marketing authorization applications and clinical trial applications 56 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Contents Code of conduct ..................................................................................................................................................................................................................... 57 Abbreviations ............................................................................................................................................................................................................................58 Glossary ..................................................................................................................................................................................................................................... 59 1. Introduction .........................................................................................................................................................................................................................61 2. Purpose .................................................................................................................................................................................................................................. 62 3. Scope .....................................................................................................................................................................................................................................63 4. Objectives and expected outcomes ...................................................................................................................................................................64 5. Deliverables ........................................................................................................................................................................................................................65 6. Overview of the expert review process ............................................................................................................................................................66 6.1 General principles ..............................................................................................................................................................................................66 6.2 Preparing for an expert review ..................................................................................................................................................................66 6.3 Conducting the expert review ..................................................................................................................................................................... 67 6.4 Expert review report .......................................................................................................................................................................................... 67 7. Roles and responsibilities ...........................................................................................................................................................................................68 7.1 The regulatory authority ..................................................................................................................................................................................68 7.2 WHO Secretariat (WHO headquarters, regional offices and country offices) .............................................................68 7.3 WHO focal point ..................................................................................................................................................................................................68 7.4 WHO team members ........................................................................................................................................................................................68 7.5 Regulatory authority participants ..............................................................................................................................................................69 8. Bibliography ........................................................................................................................................................................................................................ 70 9. Document Change History ........................................................................................................................................................................................ 71 Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities ......................................................................................................................................................................................................................................72 Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities .....................................................................................................................................................................................................................................86 57 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual WHO values and relies upon the normative and technical advice provided by leading subject matter experts in the context of its advisory/technical committees, meetings, and other similar processes. Such advice contributes to the formulation of the public health policies and norms that are promulgated by WHO for the benefit of its Member States. In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice: a. fully and honestly disclose all relevant interests and biases on the declaration of interests (DOI) Form that may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting, or group members at the outset, that is, unless this is done by the chairperson or WHO Secretariat b. spontaneously report any material changes to their disclosed interest on an ongoing basis during the period in which the expert serves the Organization c. respect the confidential nature of committee or meeting deliberations or of the advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO d. undertake not to engage in activities that may bring reputational harm to the WHO process that they are involved in e. undertake to represent their views in a personal and individual capacity with the best interest of WHO in mind as opposed to representing the views of their employers, other institutions, or governments f. actively and fully participate in discussions and deliberations within the relevant advisory group, committee, or meeting. WHO has zero tolerance towards sexual exploitation and abuse, sexual harassment (SEAH) and other types of abusive conduct (that is, discrimination, abuse of authority and harassment). Sexual exploitation, sexual abuse and sexual harassment are considered to be acts of serious misconduct and constitute a basis on which staff members, whether internationally or locally recruited, and contractors can be summarily dismissed. Code of Conduct 58 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Abbreviations CT clinical trial oversight CTA clinical trial application GBT Global Benchmarking Tool ICH International council for harmonisation of technical requirements for pharmaceuticals for human use MA (registration and) marketing authorization MAA marketing authorization application NCL national control laboratory PE performance evaluation QMS quality management system RA regulatory authority SOP standard operating procedures WHO World Health Organization WLA WHO-listed authorities Glossary 59 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Glossary The definitions below apply to the terms as used in the current document; they may have different meanings in other contexts.  Assessor To avoid any confusion, “assessor” refers to the regulatory authority (RA) official who performed the assessment of marketing authorization applications (MAA)/clinical trial applications (CTA). Expert The “evaluator” selected by WHO to perform the expert review. Expert review A process used by WHO to document and evaluate the performance of the registration and marketing authorization and clinical trial oversight functions in a medical products regulatory system, for the purpose of designation as a WHO-listed authority (WLA). Expert review is a desk-based activity and is conducted remotely; it comprises the evaluation by a WHO team of experts of MAA/CTA assessment reports issued by the RA for the purposes of authorizing a medical product or its use in a clinical trial, as applicable. Expert review plan A plan developed by the WHO focal point, in agreement with other WHO team members and WHO Secretariat, to detail different activities, timings, and assignments to be performed during the expert review. Expert review report for marketing authorization/ clinical trials A report, prepared in English, delivered by the WHO Team following the completion of the expert review. The expert review report provides an overview of the activities conducted, findings, and recommendations, if any. Performance evaluation (PE) indicator Indicator developed to assess and evaluate the performance of registration and marketing authorization and clinical trial oversight functions, as applicable, in the target country. Guidance for PE indicators is available in the form of “fact sheets”. PE tools for registration and marketing authorization and clinical trial oversight assessment Questionnaires used for evaluating the performance of the regulation and marketing authorization/clinical trials oversight functions, as applicable, of the target country (see Appendix A2.1 and Appendix A2.2). RA coordinator One or more experts, ideally familiar with the national medical products registration and marketing authorization/clinical trials oversight activities, nominated by the RA to represent it and to contribute to the expert review. WHO Expert A competent expert, who is familiar with WHO published regulations and guidelines in the area of marketing authorization and clinical trial of medical products, as relevant to the scope of expert review of registration and marketing authorization/ clinical trials applications, as applicable. WHO experts should have extensive (more than 7 years) experience and advanced skills in performing registration and marketing authorization/clinical trial oversight activities. WHO focal point WHO staff in charge of arranging and coordinating all activities related to the expert review. WHO Secretariat The WHO unit in charge of organizing the expert review. 60 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual WHO team (also called WHO experts) The team established by the WHO Secretariat in accordance with the relevant terms of reference (TORs) to perform the expert review for registration and marketing authorization/clinical trial oversight activities, as applicable. The WHO team is usually composed of two to three experts with different expertise to cover various aspects of the registration and marketing authorization/clinical trial oversight applications, as applicable. WHO-listed authority (WLA) A national regulatory authority or a regional regulatory system that has been documented to comply with all the indicators and requirements specified by WHO for listing based on an established benchmarking and performance evaluation process. A regulatory authority can be listed for one or more product categories or for one or more regulatory functions.  1. Introduction 61 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 1. Introduction Marketing authorization refers to a procedure for approval of a medical product for marketing after it has undergone a process of evaluation to determine the safety, efficacy and quality of the product and the appropriateness of the product information. The issuance of marketing authorizations – also referred to as product licensing or registration – is critical to any regulatory authority (RA). The objective of registration and marketing authorization as a regulatory function is to provide a system which ensures that only medical products that have been duly authorized by the RA are allowed to be manufactured, imported, distributed, sold or supplied to end-users. The process of assessing a medical product for registration and marketing authorization includes the review of data on quality, safety and efficacy submitted by the applicant. Imported and locally manufactured medical products should be subject to the same standards. However, evaluating the complex data used to support market authorization of new or novel medical products may require specialized resources and experience not readily available in the RA. RAs should also have the legal mandate to authorize, regulate and, if necessary, terminate clinical trials. The requirements, guidelines, procedures and forms necessary for this should be developed to be in line with country and region-specific guidelines as well as major international clinical trial guidance, including guidelines from the Declaration of Helsinki, the Nuremberg code, International Council on Harmonization, and the World Health Organization (WHO). The aim of the clinical trials oversight regulatory function is aimed at protecting the safety and rights of humans participating in clinical trials, ensuring that trials are adequately designed to meet scientifically sound objectives, and preventing any potential fraud and falsification of data. The registration and marketing authorization and clinical trial oversight functions are two of the common regulatory functions subject to assessment during the WHO benchmarking process, in the context of capacity building or WLA designation. This raised the need for comprehensive assessment and evaluation of the performance and functionality of the registration and marketing authorization and clinical trial oversight functions. In response to that need, WHO – in consultation with Member States, partners and regulatory experts – developed the process and methodology for conducting an expert review of marketing authorization applications (MAA) and clinical trial application (CTA) assessments. The methodology for conducting an expert review is described in this document, including defined roles and responsibilities. Following the guidance provided in this document will ensure consistency in the organization of expert reviews of MAA and CTA assessments, which will in turn contribute to quality output and proper interaction among the involved and interested parties. 62 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2. Purpose The purpose of this document is to: a. provide guidance to WHO, RAs and other interested parties on all aspects of the WHO expert review process and methodology, including the relevant procedures and timelines for planning, preparing, conducting, reporting and follow up, and providing templates for related documentation. b. define the roles and responsibilities of the WHO team assigned to perform an expert review of MAA and CTA assessments. c. describe the roles and responsibilities of the three levels of WHO (WHO headquarters, regional offices and country offices), as well as of the concerned RA, in this process. d. establish a level of rigour, consistency and uniformity in the procedure for expert review of MAA/CTA assessments, creating confidence in its outcomes. This document is subject to periodic review and revision as part of the quality system approach applied by WHO. It is designed to be read in conjunction with other relevant manuals, guidelines, standard operating procedures (SOPs) and work instructions. 3. Scope 63 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 3. Scope This document describes the process for initiating, planning, preparing, conducting, reporting, and following up on expert review of MAA and CTA assessments. It identifies the key critical steps that are needed in an expert review in order to be able to confirm that the performance of the registration and marketing authorization or clinical trial function, as applicable, complies with the relevant WHO standards and other internationally recognized requirements. This document applies equally to MAA and CTA assessments pertinent to medicines and biological products, including biotherapeutic products as well as vaccines. However, some requirements that are specific to one or the other may be noted in the questionnaire and PE indicators; all product-specific requirements are indicated as such in the respective documentation. 64 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 4. Objectives and expected outcomes The objectives and expected outcomes of expert review of MAA/CTA assessment, as applicable, are to: a. assess the performance of MA/CT activities and operations, conducted by the regulatory authority b. identify strengths and best practices of the MA/ CT activities, as applicable, performed by the RA c. provide feedback on the relevant GBT sub- indicators or PE indicators for designation as a WLA for the MA/CT function, as applicable. 5. Deliverables 65 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5. Deliverables After the expert review of MAA/CTA assessment has been completed, the following deliverables should be provided to the WHO Secretariat: a. Completed questionnaire containing scoring and experts’ input, prepared using the templates attached as Appendix A2.1 (for MAA) and/or Appendix A2.2 (for CTA) to this document, as applicable. b. Report of expert review (in English) to be delivered by the WHO team. c. Updated onsite assessment and evaluation of the PE indicators for the relevant function (following the template provided in the PE indicators scorecard (Annex 1), as part of the PE process for registration and marketing authorization and clinical trials oversight). 66 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6. Overview of the expert review process The expert review aims to assess the performance of a Regulatory Authority’s registration and marketing authorization or clinical trials oversight function, as applicable, with an emphasis on systems, structure and related activities – including the application process, assessment report and assessment follow up. 6.1 General principles A well-functioning registration and marketing authorization or clinical trials oversight function also relies on the other areas and components – such as legislation, regulatory requirements, infrastructure and resources, and the quality management systems (QMS). The expert review process focuses on some, but not all of the aspects that contribute to the effectiveness of the registration and marketing authorization/clinical trial oversight functions. It is not intended for use in standalone mode but is designed to be complemented by other tools and methodologies, such as GBT and/or PE indicators. It is therefore essential that these tools and methodologies are considered in combination. That means that consideration should be given to how GBT assessment contributes to and interacts with the expert review and the PE indicators. At the end of the assessment process, all of the evidence should be carefully considered. In practical terms, this means that the WHO team performing the expert review should be well briefed and aware of the outcomes of any earlier assessment. The expert review of MAA/CTA assessments is focused on actual activities and operations. This contrasts with and complements the GBT indicators, which are concerned with systematic aspects, and the PE indicators, which are concerned with quantitative and qualitative PE of the functions. All details of the review, including the participants and translation (if any) should be discussed and agreed in advance by WHO and the RA that is subject to the expert review. To facilitate the expert review, the RA should share a copy of the relevant procedures or standard operating procedures with WHO, preferably four weeks before the review. The WHO team should be given unlimited access to information, people, and assets relevant to the expert review of MAA/CTA assessments and should respect all applicable confidentiality arrangements and codes of conduct. The objective of expert review is not to assess MAA and CTA dossiers. Rather, it is to review how the RA assesses such applications. Expert review does not constitute, by any means, an endorsement by WHO of the authorization of the products concerned. The marketing authorization holder or clinical trial sponsor should refrain from misuse of the WHO expert review (such as for promotional purposes). 6.2 Preparing for an expert review 6.2.1 Selecting assessments for review The selection of which MAA/CTA assessment reports should be subject to the expert review should be agreed in advance between the RA and the WHO Secretariat. To facilitate this, the RA should provide the WHO Secretariat with a comprehensive list of MAAs and/or CTAs that have been received and assessed by the RA over the previous few years, indicating the type of procedure followed (for example, full vs. abbreviated, initial application or variation and so on) and the outcome (approved, rejected, withdrawn, suspended, completed, ongoing). The WHO experts should select for review a representative number (at least two) of MAA/CTA assessment reports issued by the RA, excluding assessment reports that are more than three years old. In the case of MAA, full applications should be ideally selected. As a guiding principle, a risk-based approach should always be used to select the most representative product files. For example, this could include one assessment report for each of the product categories in the WLA scope. Other considerations might include product criticality and/or complexity, as well as the availability of the authorization dossiers in a language that is understood by the WHO assessors. The assessments selected for review will be communicated to the RA well in advance of the start of the activity. 6.2.2 Briefing sessions The members of the WHO team selected for a specific expert review should be thoroughly briefed on the principles described in this document prior to the start of the activity. 6. Overview of the expert review process 67 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The WHO Secretariat should brief all experts remotely as part of the preparations for the expert review. The briefing should include details relating to: ƒ the context of the expert review, including objectives and expected outcomes ƒ the methodology for the review ƒ the availability of required documents ƒ access to and utilization of WHO secure information sharing platform ƒ roles and responsibilities of different experts, including specific task(s) and ƒ answers to questions raised and clarifications sought by experts. 6.3 Conducting the expert review By default, the expert review is designed to be a desk- based activity that should be conducted remotely. In exceptional circumstances and on a case-by-case basis, WHO may, in agreement with the RA, decide whether an onsite review of the registration and marketing authorization or clinical trials oversight function would be appropriate and feasible. In general, onsite expert review is discouraged. To facilitate the expert review process, the relevant RA coordinator(s) shall upload the following documents to the secure WHO information-sharing platform, in advance of the planned expert review activity: ƒ applicable national regulations/guidelines ƒ the (full) application dossiers of the products selected for review ƒ the MAA/CTA assessment reports issued by the RA ƒ the self-assessment of the expert review questionnaire (see Appendix A2.1 and Appendix A2.2, as applicable). The WHO team should verify that all the guidelines used as references were relevant and up to date when the RA conducted the assessment. At the start of the expert review process, a meeting should be arranged between the WHO experts and the relevant RA staff in order to obtain preliminary information about the dossier concerned, the pathways applied for authorization and the RA’s overall assessment process. The WHO experts will be given the possibility to raise two rounds of written queries to the RA officials, in order to clarify any issue or doubt they have identified in the assessment being reviewed. Responses to queries will then be discussed in dedicated virtual meetings arranged by the WHO Secretariat, in agreement with the WHO team and RA officials. The “questionnaires" should be viewed as an aide memoire for ensuring all critical elements are evaluated. Experts should also keep in mind that some of the listed criteria may be considered “not applicable”, depending on the application type. However, the overall status should be evaluated and scored. 6.4 Expert review report On completing the review, the WHO team should issue an expert review report (in English or bilingual), containing general information about its activities, findings (strengths, gaps, and areas for improvement) and recommendations, if any. This should include the completed questionnaire used to assess the performance of activities (provided in Appendix A2.1 and A2.2 to this document, as applicable). The finalized expert review report should be made available to WHO Secretariat within 14 working days of the last day of the activity. 68 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 7. Roles and responsibilities The expert review should be seen as a collaborative exercise to which several parties – including the RA, WHO Secretariat, and the WHO team of experts – are contributing. This section provides guidance on the roles and responsibilities of each of those parties. 7.1 The regulatory authority The RA is responsible for: a. discussing and agreeing with the WHO Secretariat the selection of file(s) that will be subject to the review b. designating one or more focal persons to coordinate the activities of the expert review c. granting the WHO team access to all relevant data and information throughout the expert review process d. sharing all necessary information and documentations (including national code/ regulations/guidelines, relevant procedures and data specific to the file(s) selected for the expert review) with WHO through the secure information-sharing platform or any other agreed means e. providing clarifications and explanations in response to questions from the WHO team. f. seeking and obtaining any necessary consent from the applicants in order to allow the relevant dossiers to be shared with WHO. 7.2 WHO Secretariat (WHO headquarters, regional offices and country offices) WHO headquarters (specifically the Regulatory Systems Strengthening Team), in collaboration with the relevant WHO regional and country offices, is responsible for: a. establishing and maintaining the tools and databases related to expert review b. establishing a roster of qualified experts c. training experts in order to ensure consistency and quality of the process as well as robustness of the assessment outcome d. discussing and agreeing with the RA the selection of file(s) that will be subject to the expert review e. establishing a dedicated country page for the expert review on the WHO information- sharing platform and uploading of all relevant documentation for access and archive purposes f. selecting from the roster of qualified experts the members of the WHO team who will perform the expert review on behalf of WHO g. designating the WHO team leader h. organizing any necessary contractual arrangements. 7.3 WHO focal point The WHO focal point is responsible for: a. leading and coordinating the expert review from the beginning to the end of the process (the WHO focal point may or may not participate in the assessment and performance evaluation of the relevant function during the expert review) b. briefing the WHO team members on various aspects of the expert review (including context, background, objectives, process and methodology, roles, and responsibilities) c. coordinating work among all members of the WHO team in order to ensure smooth and consistent completion of the abridged pathway assessment and avoid the duplication of effort and/or conflicts d. communicating with RA officials on behalf of WHO e. delivering the expert review report (although the expert review report should be ideally prepared by the entire WHO team, the responsibility for delivering the final report rests with the WHO focal point). 7.4 WHO team members The members of the WHO team are responsible for: a. reviewing and signing the relevant administrative documents, including invitation letter, confidenti- ality agreement, and declaration of interests form. 69 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 7. Roles and responsibilities b. respecting all applicable protocols, ethics, and codes of conduct. c. assessing and evaluating the performance of registration and marketing authorization/clinical trials oversight activities using the questionnaire in Appendices A2.1 and A2.2. d. identifying strengths as well as gaps and areas for improvement, if any e. preparing a detailed report on the expert review conducted, which should be provided to the WHO Secretariat within 14 working days of the last meeting with RA. The report may quote the various components/sections in the questionnaire. 7.5 Regulatory authority participants The RA participants are responsible for: a. establishing and maintaining communication between the WHO team, and the RA staff b. keeping senior management informed about the expert review c. helping to coordinate the expert review d. preparing any and all materials requested by the WHO team e. facilitating easy access of the WHO team to the requested documents, information, and persons f. providing the clarifications and explanations sought by the WHO team about systems and procedures used g. responding to the WHO team’s questions and calls for interview, if any. 70 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8. Bibliography Good review practices: guidelines for national and regional regulatory authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: forty-ninth report. Geneva: World Health Organization; 2015: Annex 9 (WHO Technical Report Series, No. 992, https:// apps.who.int/iris/handle/10665/176954, accessed 11 April 2023. Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans. Geneva: World Health Organization; 2021, in press. Registration and marketing authorization (MA) or clinical trials oversight (CT) function: Indicators and Fact Sheets. In: WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva, World Health Organization; 2021 (https://apps.who.int/iris/ handle/10665/341243, accessed 11 April 2023). WHO-listed authority (WLA). Geneva: World Health Organization (https://www.who.int/initiatives/who- listed-authority-reg-authorities, accessed 9 August 2023). 9. Document change history 71 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 9. Document change history Version no. Date of issue Main changes 1 June 2023 First version 72 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A2.1. About this questionnaire This questionnaire is designed to assess the performance of the medical products registration and marketing authorization function as part of an expert review of MAA assessments. The questionnaire comprises a mix of open-ended and close-ended questions designed to assess three main evaluation criteria: 1. the application process 2. the assessment report 3. assessment follow-up. Regulatory authority (RA) staff should answer the questions by completing the fields in the column headed “RA input”; this is the self-assessment element of the review. The WHO team should: a. complete the fields in the column headed “Expert input” b. select the appropriate checkboxes indicating the assigned “score”, and c. attach a copy of the completed questionnaire to the expert review report. Rating WHO uses the expert review to determine whether or not the RA can be considered to acceptably meet WLA requirements. In order for an RA to be granted WLA status for the registration and marketing authorization function, it must: a. achieve a “satisfactory” score in each of the three areas of the questionnaire. b. score “yes” in at least 85% of the components (excluding components that are not applicable). Expert review questionnaire: for assessing the performance of marketing authorization activities Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 73 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 1 Application Process (including pre-submission procedures, assessment, compliance with regulatory requirements and policies, and interactions with stakeholders) Evaluation of the application process should focus on all activities that could have an impact in the assessment. 1.1 Was there scientific/regulatory advice or other similar activities provided by the RA prior to the submission to support the success of a complete application with quality, if applicable? Were there pre-submission meetings with the company for this application arranged by the RA prior to the submission to support the success of a complete application, if applicable? Appropriate support and information have been provided to the sponsor by the RA for the success of an application submission. All actions were taken by the RA to allow a more predictable and clear process for applicants. The RA benefited from a complete application submission at the outset by the applicant. ☐ Yes ☐ No 1.2 Were the relevant documented procedures to support the full review process adequately followed? Was the submitted dossier compliant with relevant International Standards such as International Council for Harmonisation (ICH) Common Technical Document (CTD) format? ICH Guidelines, as applicable. WHO Guidelines, as applicable. Was the assessment team well formulated including involvement of all other relevant teams (for example, staff with specific expertise for the given therapeutic area is involved, other teams are involved as necessary, such as inspections, national control laboratory, etc.)? The RA adequately followed the relevant internal Guidelines and standard operating procedures (SOPs) for the review process. If those were not strictly followed in some cases, this was well justified for each of those cases and the RA ensured it did not impact the outcomes of the assessment. Interdisciplinary work between scientific staff was properly arranged for the given application. ☐ Yes ☐ No 1.3 Quality and consistency of the assessment, reports and decision- making. Overall, does the assessment report comply with local and international standards, defined by the RA to be applied upon? The RA has robust procedures in place for evaluation of the quality and consistency of the assessment, assessment reports and decision-making. An adequate system is in place for maintenance of regulatory memory. The assessment, assessment reports and decision-making at the RA is perceived to be consistently conducted and ensured. ☐ Yes ☐ No Questionnaire for expert review of MAA assessment Country: __________________ Institution:__________________ Dates:__________________ Assessors:__________________ 74 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Overall outcome of section 1 WHO Experts should use the above-listed items to qualitatively evaluate the overall application process Based on the above, WHO Experts conclude that overall section 1 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2 Assessment Report The WHO Experts are expected to have access to the entire dossier and consult it as needed to be able to evaluate the level of RA’s performance in this section. 2.1 Quality of the report 2.1.1 Considers context Does the assessment report consider the data and the conclusions from the applicant in the context of the proposed conditions of use and storage? Does it include perspectives from patients/patient associations, health- care professionals and other RAs’ analyses and decisions? Was there a mechanism/process activated to obtain opinion or advice from outside stakeholders, as necessary, in the adequate moments of the assessment and as per the RA guidelines establish? The assessment report considers all relevant data and conclusions from the applicant on the proposed conditions of use and storage. The assessment report also considers any feedback provided by patients/patient associations and health-care professionals as well as other RAs’ analyses and decisions. The RA adequately followed its guidelines in terms of consulting and requesting advice from external experts, healthcare professionals and patients/patients’ association, as necessary and as per its guidelines. ☐ Yes ☐ No Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 75 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.1.2 Balanced and evidence-based Is the assessment report objective and unbiased? Is the assessment report evidence-based? Does it reflect both updated scientific and regulatory state of the art? Does it integrate legislative, regulatory and policy frameworks with emerging science? Are the type and number of objections raised and clarifications requested supported by evidence? Are concerns categorized as major and minor (or similar, based on national guidance)? Is the classification appropriate and supported by scientific discussion? Is the assessment of responses provided by the applicant considered in the final decision? The assessment report is evidence-based and factual. It considers and integrates emerging scientific and regulatory aspects, and it is aligned with relevant legislative, regulatory and policy frameworks. It is based on updated and relevant technical guidelines. Specifically, the type and number of objections raised and clarifications requested are supported by appropriate evidence. The assessment of responses provided by the applicant is integrated into the final decision of the RA. ☐ Yes ☐ No 2.1.3 Depth Does the assessment report comprehensively highlight potential areas of concern, providing a detailed analysis of those? The assessment report properly highlights and deeply analyses potential areas of concern supported by adequate justifications and observations. ☐ Yes ☐ No 2.1.4 Investigates problems Does the assessment report provide both the applicant’s and the assessors’ in-depth analyses and findings of key scientific data? Does the assessor demonstrate the use of risk-based tools, analyses and synthesis skills to ask relevant questions where needed? The assessment report provides comprehensive analysis and findings of key scientific data. The assessor demonstrated the use of risk-based tools, analyses and synthesis skills, to ask relevant questions and make appropriate judgments, where needed. ☐ Yes ☐ No 76 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.1.5 Makes linkages Does the assessment report provide integrated analysis across all aspects of the application: quality, non-clinical; clinical; chemistry/ biocompatibility; manufacturing; and risk management plan? Does it include timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application? The assessment report provides good quality and integrated analysis of all relevant aspects of the application: non-clinical; quality; clinical; chemistry/ biocompatibility; manufacturing; and risk management plan. It includes timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application. ☐ Yes ☐ No 2.1.6 Thorough Does the assessment report reflect adequate follow-through of all the issues by the assessors? The assessment report reflects adequate follow-through of all issues raised by the assessors. ☐ Yes ☐ No 2.1.7 Utilizes critical analyses Does the assessment report assess the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application? Observations are well classified or categorized according to national agreed terms (such as major/minor)? The assessment report critically assesses the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application. Observations made throughout the report are categorized according to national agreed terms. ☐ Yes ☐ No 2.1.8 Well-documented Does the review report provide a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions and rationale for reaching a decision? Does it contain clear, succinct recommendations that can stand up to scrutiny by all the parties involved and could be leveraged by others? Are observations well described and detailed? Observations are well grouped or categorized? The assessment report provides a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions as well as rationale for reaching a decision. It contains clear recommendations and well-described, detailed and categorized observations. ☐ Yes ☐ No Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 77 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.1.9 Well-managed Does the review report apply project and quality management processes, including clearly defined steps with specific activities and targets? Were timelines well managed throughout the assessment, including for drafting the list of questions? Assessment report is finalized within the agreed timeframe? The assessment report applied adequate project and quality management processes, including clearly defined steps, targets and timelines. The timelines were well managed throughout the assessment procedure, and this is reflected in the report. The final report was complete within the established timelines, as the RA guidelines stipulate. ☐ Yes ☐ No 2.1.10 Peer reviews Was the assessment report subject to peer reviews? How is it completed and recorded? How are the comments of the peer reviewer handled? Are they documented and kept? The agency has an effective system for peer review of reports. The assessment report was subject to adequate and well-documented peer reviews. The comments provided by the peer reviewer were appropriately handled and addressed. When it is not applicable, a proper justification is provided. ☐ Yes ☐ No 2.1.11 Product information to the public Is the product information available to the public, such as information akin to summary of product characteristics, product information leaflets, product packaging and labelling, or other type of product information communication to the public, of good quality, easily readable? Does the product information appropriately reflect the conclusions from the assessment regarding the approved indication, posology, method of administration, contra-indication, precautions for use, interactions, shelf-life, storage conditions, the available information on safety, efficacy, potential risk and how to manage them? The product information, such as SmPC, leaflets or other types of product information communication to the public, is of good quality and easily readable. ☐ Yes ☐ No 78 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Overall outcome of sub-section 2.1 WHO Experts should use the above-listed items to qualitatively evaluate the overall quality of the report Based on the above, WHO Experts conclude that overall sub-section 2.1 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.2 Completeness of the report to provide a comprehensive and complete picture of the situation or sample under consideration. 2.2.1 Were all relevant parts/modules of the dossier reviewed? All relevant parts/modules of the dossier were reviewed and they are reflected in the assessment report. 2.2.2 Are all relevant regulations, standards and guidance referenced in the report, as necessary, linked to the respective observation? All relevant regulations, standards and guidance are referenced in the report, as necessary, linked to the respective observation. ☐ Yes ☐ No 2.2.3 Is the assessment report compliant with the content and format described in the relevant SOP? The assessment report is compliant with the content and format described in the relevant SOPs or guidelines. ☐ Yes ☐ No 2.2.4 Were the risk management plans considered and included in the assessment? Are the risk management plans adequate to address the potential risks? Risk management plans are part of the assessment report and are adequate from a qualitative point of view. ☐ Yes ☐ No Overall outcome of sub-section 2.2 WHO Experts should use the above-listed items to qualitatively evaluate the overall completeness of the report Based on the above, WHO Experts conclude that overall sub-section 2.2 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 79 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.3 Scientific rigour to ensure the application of the scientific approach for unbiased analysis and interpretation of the evidence or data High-quality scientific work provides a sound basis for appropriate consistent and harmonised opinions and decisions that affect public health Are the main critical features of the product, salient findings and those deficiencies that justify any questions intended for the applicant well described? Are the assessor’s own critical assessment and observations to the applicant data included, particularly with respect to scientific elements and adherence to specific guidance documents? Are cross-references adequately used to clearly indicate the origin of any information used in the report, such as to the specific parts of the dossier (for example, overview, summary, study reports), the references to the literature or other sources? Are those findings that need to be reflected in the summary product characteristics, Labels & Package Leaflet well emphasized? Are conclusions on the different scientific components well developed and described by the assessors? The WHO Experts are expected to look at the essential elements under each of the following sections considering 1) the specific category of the product (chemical (new or multisource) or biological (vaccines or biosimilars)) and 2) the type of module (quality, clinical or non-clinical). Experts should use the list of items provided for guidance but mainly, his/ her experience and judgement to analyse and evaluate the assessment conducted by the RA on each of the 3 areas for the specific type of product. The Expert should aim to answer specific technical questions from a qualitative point of view. The Experts should write a summary of findings for each of the sections (quality, clinical and non-clinical) on how the assessment was conducted by the RA (in terms of evidence assessed by the assessor, quality of such assessment and observations, and decision-making done by the assessor). 2.3.1 Clinical ICH common technical document Module 2 and 5 for new chemical entities, vaccines and biosimilars (for the last one, reduced clinical data will depend on proof of its similarity to an appropriate RBP through the comparability exercise – based on WHO guidance). If it is a multisource finished pharmaceutical product (FPP), based on WHO guidance, only demonstration of bioequivalence of the finished pharmaceutical product is required. This may necessitate the manufacturer carrying out a bioequivalence study and an assessment of the bioequivalence study (trial) information: the data generated should provide a bridge between the comparator product for which safety and efficacy data are available and the generic products for which such data are not available 80 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.3.1.1 Aspects to be considered in the assessment report: ƒ Good clinical research practice aspects ƒ Biopharmaceutics ƒ Bioequivalence studies, as applicable ƒ Clinical pharmacology ƒ Clinical efficacy ƒ Clinical safety ƒ Paediatric studies ƒ Risk Management Plan ƒ Pharmacovigilance system/post-marketing experience For each section, the discussion should identify the most important findings and deficiencies and how results agree. It should indicate if the data submitted fulfil the requirements (legal, guidelines, scientific advice). The major issues raised and how they were addressed should be reflected. Uncertainties should be considered by mentioning what the source of the uncertainty is (for example, missing data), what the item is that the assessment is uncertain about (for example, efficacy in a subgroup) and what the possible coping strategies are (for example, need to collect further data to reduce uncertainty; acknowledge through labelling changes). Both study design and results should be subject to the critical discussion. The following is a compilation of potential aspects to be addressed in such discussion. ƒ Adequacy of the study design (randomized active and placebo-controlled trials), with justification. ƒ Adequacy of the selected patient population (reflection on inclusion/exclusion criteria), including any age limit exclusion? ƒ Appropriateness of the comparator (for generics). In case of an active comparator, discussion on the relevance in view of the national/local established clinical practice guideline and treatment options. ƒ Demonstration of bioequivalence/bioavailability (BE/BA) when deemed necessary for multisource generic medicines. ƒ Critical discussion of the appropriateness of the choice of endpoints as well as the duration of the study considering regulatory guidance/ scientific advice (for example, validity of surrogate markers to replace hard endpoints; acceptability of a composite endpoint and its domains). ƒ Adequacy of the methods, conduct, analysis and reporting of results from main studies, as appropriate. Discussion on any particular issues raised regarding the study design. ƒ Accordance of the design with legal requirements, available guidelines, and scientific advice. ƒ Implications of any good clinical research practice inspection A brief statement about the conclusions in terms of establishing efficacy and safety that can be drawn from the documentation should be provided. ☐ Yes ☐ No Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 81 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Overall outcome of sub-section 2.3.1 WHO Experts should use the above-listed items to qualitatively evaluate the overall scientific rigour of the clinical part of the assessment Based on the above, WHO Experts conclude that overall sub-section 2.3.1 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.3.2 Quality ICH common technical document (CTD) Module 2 and 3 for new chemical entities, vaccines and biosimilars (for the last one, a comparison of the similar biotherapeutic products (SBP) and the reference biological product (RBP) with respect to quality represents an additional requirement to the “traditional” full quality dossier – comparability exercise - based on WHO guidance). If multisource, based on WHO guidance it is only required to demonstrate the quality of the active pharmaceutical ingredient(s), demonstrate the quality of the finished pharmaceutical products, demonstrate adherence to WHO Good Manufacturing Practices. 2.3.2.1 For the quality, the following aspects should be considered: ƒ Drug substance (CTD module 3.2.S) ƒ general information ƒ manufacture ƒ characterization ƒ Good manufacturing practices compliance ƒ control of drug substance ƒ reference standards of materials ƒ container closure system ƒ stability. Drug product (CTD module 3.2.P) ƒ description and composition of the drug product ƒ pharmaceutical development ƒ manufacture ƒ control of excipients ƒ Good manufacturing practices compliance ƒ control of drug product ƒ reference standards or materials ƒ container closure system, ƒ stability ☐ Yes ☐ No 82 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Elements from Appendices (CTD module 3.2.A) ƒ facilities and equipment ƒ adventitious agents safety evaluation ƒ novel excipients. Key aspects and summaries of relevant studies (including comparability, if applicable) that are essential in providing reassurance with regard to the quality of drug substances and drug product should be provided in the assessment report. The assessment report should include a general background of the product to identify the main critical features: active substance (for example, new chemical entity, known chemical active substance, biosimilar), if paediatric formulation has been/is to be developed, orphan status, indications, target population, posology, method of administration (for example, use of device), use of delivery/administration systems and preparation/reconstitution of product. It should be mentioned whether a CEP or active substance master file procedure or full information of the active substance in the dossier is used (when active substance master file procedure is used, restricted part with information which is protected by intellectual property rights or is otherwise sensitive should not be disclosed to the applicant). The report should be sufficiently detailed to allow for secondary assessment. Quality matters should relate to efficacy and safety consequences as much as possible. It should be indicated if there is any quality aspect either in the active substance or in the finished product which could lead to an impact on the Benefit- Risk Balance. Scientific argumentation in the assessment report should support the proposed questions and the report should emphasize those findings that need to be reflected in the summary product characteristics, labelling and package leaflet. A very brief summary of the conclusions drawn from the quality documentation should be provided. Overall outcome of sub-section 2.3.2 WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific rigor” of the quality part of the assessment Based on the above, WHO Experts conclude that the overall sub-section is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory Justification: (please provide text) 2.3.3 Non-clinical ICH CTD Module 2 and 4 structure for new chemical entities, vaccines and biosimilars (for biosimilars, reduced non-clinical data will depend on proof of its similarity to an appropriate reference biotherapeutic product through the comparability exercise – based on WHO guidance). This section is not required for a multisource finished pharmaceutical product. Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 83 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.3.3.1 For the non-clinical, the following aspects should be considered: ƒ Good laboratory practice (GLP) aspects ƒ pharmacology ƒ pharmacokinetics ƒ toxicology ƒ ecotoxicity/environmental risk assessment and ƒ Implications of the assessment of non-clinical data for the Safety Specification of the Risk Management Plan (RMP). For each section, the discussion should address the following points: ƒ Identify the most important findings and deficiencies. Describe how results agree. Summarize evidence for each conclusion. ƒ State whether the data submitted fulfils the requirements, comment on whether the non-clinical study programme was built up using the risk-based approach that is, with possible omission of in vivo studies. ƒ Describe the major issues raised and how they have been addressed. Consideration should be given to the following: ƒ data submitted in accordance with legal requirements, available guidelines and scientific advice ƒ any justifications for waiving certain studies or replacing original studies by literature data ƒ major issues raised (major objections and other important concerns) and how they were addressed ƒ assessment of all information in the product information and correspondence with the findings (particularly preclinical safety data but also contraindications, interactions, pregnancy and lactation, non-clinical pharmacodynamic properties, non-clinical pharmacokinetic properties, as relevant) ƒ key findings (or uncertainties) that should be part of the benefit- risk assessment, or biosimilarity assessment for biosimilars ƒ a very brief summary of the conclusions drawn from the non-clinical documentation should be provided. ☐ Yes ☐ No Overall outcome of sub-section 2.3.3 WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the non-clinical part of the assessment. Based on the above, WHO Experts conclude that the overall sub-section 2.3.3 is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 84 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.4.1 Scientific Opinion How is an overall benefit risk assessment generated for an application? Are the conclusions on risk-benefit analysis and overall assessment outcomes consistently and adequately reached and concluded, in line with the assessment report observations, concerns and evidence reviewed? Are the assessment outcomes adequately considering the Risk management plans? How did the assessors achieve an integrated opinion? Is there input or advice from scientific committees, or from external experts? How was this integrated into the scientific opinion? How were divergent views handled, if any? Overall, the assessment outcomes/opinions are aligned with the observations made throughout the assessment process. It reflects all observations and concerns as per those identified in the assessment report. All input received during the assessment is adequately reflected in the report and in the scientific opinion. Benefit-risk based decisions are inclusive, comprehensive, documented and consistent. In the positive scientific opinions, the benefits clearly outweigh the risks, based on sound scientific evidence. The production of the integrated opinion of assessors and their senior managers for the final decision-making by the agency (e.g., to the development and agreement on a positive opinion to authorize a medicinal product), is consistently and adequately achieved. ☐ Yes ☐ No Overall outcome of sub-section 2.4 WHO Experts should use the above-listed items to qualitatively evaluate the overall scientific opinion of the assessment. Based on the above, WHO Experts conclude that overall sub-section 2.4 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Overall outcome of section 2 WHO Experts should use the above-listed items to qualitatively evaluate the overall application process. Based on the above, WHO Experts conclude that overall section 2 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 3. Assessment follow-up 3.1 Are the post-approval actions well reflected in the product file? Further post-approval actions taken (if any) are adequately reflected in the product file ☐ Yes ☐ No Appendix A2.1. Expert review questionnaire: for assessing the performance of marketing authorization activities 85 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 3.2 In case of emergency approvals (or approvals provided under exceptional circumstances), are there follow-ups after introduction with respective reflection in the product file? In case of emergency approvals (or approvals provided under exceptional circumstances), there are appropriate follow-ups after introduction of the product, with respective reflection in the product file. ☐ Yes ☐ No Overall outcome of section 3 WHO Experts should use the above-listed items to qualitatively evaluate the overall “assessment follow-up.” Based on the above, WHO Experts conclude that overall section 3 is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Outcome of the performance evaluation of MA activities WHO Experts’ overall conclusion of the MA expert review The overall conclusion should be based on the evaluation and scoring achieved in each of the individual three afore-mentioned sections of the questionnaire. If one of these parts is found to be unsatisfactory according to the specific guidance provided, the overall outcome of the performance evaluation should be consequently scored as unsatisfactory Based on the collective evidence and findings of this MA expert review, the WHO Experts conclude that the performance of the MA activities, including application process, assessment report, and assessment follow up is: ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text 86 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A2.2. About this questionnaire The objective of this questionnaire is to assess the performance of the medical products clinical trial oversight function through an expert review of clinical trial-related activities.   The questionnaire includes a mix of open-ended and closed-ended questions, aimed at assessing three evaluation criteria: 1. application process 2. assessment report 3. assessment follow-up. Regulatory authority (RA) staff should answer the questions by filling in the fields in the column headed “RA input”; this is the self-assessment element of the review. The WHO team should: a. complete the fields in the column headed “Expert input” b. select the appropriate checkboxes indicating the “score” and c. attach a copy of the completed questionnaire to the report of the expert review of registration and marketing authorization. Rating WHO uses the expert review to determine whether or not the relevant RA can be considered to acceptably meet WLA requirements. For an authority to be given WLA status for clinical trials oversight, the RA must fulfil the following criteria: d. a) achieve a satisfactory score in each of the three areas of the questionnaire e. b) achieve not less than 85% of the components (excluding non-applicable components) scored Expert review questionnaire: for assessing the performance of clinical trial oversight activities Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 87 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manualEvaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 1 Application Process (including pre-submission procedures, assessment, compliance with regulatory requirements and policies, and interactions with stakeholders) The evaluation of the application process should focus on all activities that could impact the assessment. 1.1 Did the RA provide regulatory advice or other similar activities prior to the submission of the CTA to support the success of a complete application, if needed? The RA has provided appropriate support and information to the sponsor to enable a successful CTA submission. The RA took action to ensure a clearer and more predictable process for applicants. The RA benefited from the applicant submitting a complete application at the outset. ☐ Yes ☐ No 1.2 Were the relevant documented procedures to support the full CTA review process adequately followed? Was the assessment team well formulated, including involvement of all other relevant teams? (for example, did it involve staff with specific expertise for the given therapeutic area and other teams as necessary?) The RA adequately followed the relevant internal guidelines and SOPs for the review process. Roles and responsibilities were clearly defined and followed in accordance with the indications in the guidelines and SOPs. ☐ Yes ☐ No 1.3 Are there documented procedures, templates and checklists available to support the RA assessment process? Were these appropriately followed? Was the national guidance appropriately followed in relation to ethics, medical care and records, confidentiality and the conduct of clinical trials in the country? Was the submitted application based on relevant national and international standards and practices? Were international standards appropriately followed during the assessment of CTA? (For example, the Declaration of Helsinki, applicable WHO guidelines such as the WHO Guidance for organizations performing in vivo bioequivalence studies, and applicable ICH guidelines such as ICH Guidelines for good clinical practice E6 (R1)). The RA has documented procedures, templates and checklists in place for the CTA assessment process and these are adequately followed. If there were some cases where these were not strictly adhered to, this was well justified in each case and the RA ensured that it did not impact the outcomes of the assessment. Any lacking or missed information in the application was identified by the RA prior to scientific review, thus avoiding spending time and reviewing resources on an application that does not allow critical analysis, signal identification or regulatory decision-making. The processes followed for this application assessment are perceived to be compliant and aligned with the established procedures and international standards and practices. ☐ Yes ☐ No Questionnaire for expert review of clinical trial application (CTA) assessments Country:__________________ Institution:__________________ Dates:__________________ Assessors:__________________ 88 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Overall outcome of section 1 Guidance: WHO Experts should use the above-listed items to qualitatively evaluate the overall “application process” Based on the above, WHO Experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2 Assessment report The Experts are expected to be given access to the whole dossier and consult it as needed to be able to evaluate the level of RA’s performance in this section. 2.1 Quality of the report 2.1.1 Considers context Does the assessment report consider the data and the conclusions from the applicant? Does it include perspectives from patients/patient associations, health care professionals and other RAs’ analyses and decisions? Was a mechanism activated to obtain opinion and advice from relevant stakeholders, as necessary, at adequate points in the assessment and as established in the RA guidelines? The assessment report considers all relevant data and conclusions from the applicant. The assessment report also considers any feedback provided by patients/patient associations and health- care professionals as well as other RAs’ analyses and decisions. The RA adequately followed its guidelines in terms of consulting and requesting advice from external experts, healthcare professionals and patients/patients’ association, as necessary and as per its guidelines. ☐ Yes ☐ No Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 89 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.1.2 Balanced and evidence-based Is the assessment report objective and unbiased? Is the assessment report evidence-based? Does the assessment report reflect the updated scientific and regulatory state of the art? Does the assessment report integrate legislative, regulatory and policy frameworks with emerging science, such as the Declaration of Helsinki? Are the type and number of objections raised and clarifications requested supported by evidence? Are concerns categorized as major and minor (or similar, based on national guidance)? Is the classification appropriate and supported by scientific discussion? Is the assessment of responses provided by the applicant considered in the final decision? The assessment report is evidence-based and factual. It considers and integrates emerging scientific and regulatory aspects, and it is aligned with relevant legislative, regulatory and policy frameworks. It is based on updated and relevant technical guidelines. Specifically, the type and number of objections raised, and clarifications requested are supported by appropriate evidence. Assessment of the responses provided by the applicant is integrated into the final decision of the RA. ☐ Yes ☐ No 2.1.3 Depth Does the assessment report comprehensively highlight potential areas of concern, providing a detailed analysis of those? The assessment report properly highlights and deeply analyses potential areas of concern supported by adequate justifications and observations. ☐ Yes ☐ No 2.1.4 Investigates problems Does the assessment report provide both the applicant’s and the assessors’ in-depth analyses and findings of key scientific data? Does the assessor demonstrate the use of risk-based tools, analyses and synthesis skills to ask relevant questions where needed? The assessment report provides comprehensive analysis and findings of key scientific data. The assessor demonstrated the use of risk-based tools, analyses and synthesis skills, to ask relevant questions and make appropriate judgments, where needed. ☐ Yes ☐ No 90 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.1.5 Makes linkages Does the assessment report provide integrated analysis across all aspects of the application: quality, pre-clinical, clinical, GxP compliance, study protocol? Does it include timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application? The assessment report provides good quality and integrated analysis of all relevant aspects of the application: quality, pre- clinical, clinical, GxP compliance, study protocol. It includes timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application. ☐ Yes ☐ No 2.1.6 Thorough Does the assessment report reflect adequate follow-through of all the issues by the assessors? The assessment report reflects adequate follow-through of all issues raised by the assessors. ☐ Yes ☐ No 2.1.7 Utilizes critical analyses Does the assessment report assess the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application? Are observations well classified or categorized according to nationally agreed terms (such as major/minor)? The assessment report critically assesses the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application. Observations made throughout the report are categorized according to national agreed terms. ☐ Yes ☐ No 2.1.8 Well-documented Does the review report provide a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, as well as the assessors’ conclusions and rationale for reaching a decision? Does it contain clear, succinct recommendations that can stand up to scrutiny by all the parties involved and could be leveraged by others? Are observations well described and detailed? Are observations well grouped or categorized? The assessment report provides a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions as well as rationale for reaching a decision. It contains clear recommendations and well-described, detailed and categorized observations. ☐ Yes ☐ No Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 91 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.1.9 Well-managed Does the review report apply project and quality management processes, including clearly defined steps with specific activities and targets? Were timelines well managed throughout the assessment? Was the CTA assessment report finalized within the agreed timeframe? The assessment report applied adequate project and quality management processes, including clearly defined steps, targets and timelines. The timelines were well managed throughout the assessment procedure, and this is reflected in the report. The final report was completed within the established timelines, as stipulated in the RA guidelines. ☐ Yes ☐ No 2.1.10 Peer Reviews Was the assessment report subject to peer reviews? How is peer review completed and recorded? How are the comments of the peer reviewer handled? Are they documented and kept? The agency has an effective system for peer-review of reports. The assessment report was subject to adequate and well documented peer reviews. The comments provided by the peer reviewer(s) were appropriately handled and addressed. When peer review was not applicable, a proper justification for this is provided. ☐ Yes ☐ No 2.1.11 Information to the public Is the information to the public about the CTA assessment outcomes easily readable and clearly communicated? Is it aligned with the national guideline requirements? The information to the public about the CTA assessment outcomes are good quality, easily readable and clearly communicated to the target audience. ☐ Yes ☐ No Overall outcome of sub-section 2.1 WHO Experts should use the above-listed items to qualitatively evaluate the overall quality of the report. Based on the above, WHO Experts conclude that the overall sub-section is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.2 Completeness of the report to provide a comprehensive and complete picture of the situation or sample under consideration. 92 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.2.1 Were all relevant parts of the application file reviewed? All relevant parts of the dossier were reviewed and are reflected in the assessment report. ☐ Yes ☐ No 2.2.2 Are all relevant regulations, standards and guidance referenced in the report, as needed and linked to the relevant observation? All relevant regulations, standards and guidance are referenced in the report as needed and linked to the respective observation. ☐ Yes ☐ No 2.2.3 Is the assessment report a complaint to the content and format described in the relevant SOPs? Is the assessment report aligned with the registries’ information? The assessment report is compliant with the content and format described in the relevant SOPs or guidelines. It is also aligned with the published registries information. ☐ Yes ☐ No 2.2.4 Did the assessment report include analysis of the oversight of similar trials already running in other areas and indicates by which government/ organization? The assessment report includes analysis of any other trials already running in other areas and indicates by which government/ organization. ☐ Yes ☐ No Overall outcome of sub-section 2.2 WHO Experts should use the above-listed items to qualitatively evaluate the overall completeness of the report Based on the above, WHO Experts conclude that overall sub-section 2.2 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 93 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.3 Scientific rigour to ensure the application of the scientific approach for unbiased analysis and interpretation of the evidence or data High-quality scientific work provides a sound basis for appropriate consistent and harmonized opinions and decisions that affect public health. Are the main critical features of the clinical trial and salient findings well described? If there are any deficiencies that justify questions to the applicant, are these well described? Is the assessor’s own critical assessment and observations to the applicant data included, particularly with respect to scientific elements and adherence to specific guidance documents? Are cross-references adequately used to clearly indicate the origin of any information used in the report, such as to the specific parts of the dossier (for example, overview, summary, study reports), the references to the literature or other sources? Are the findings that need to be reflected in the summary product characteristics, Labels & Package Leaflet well emphasized? Are conclusions on the different scientific components well developed and described by the assessors? The Experts are expected to look at the essential elements under each of those sections considering a) the product scope – new chemical entities, multisource (bioequivalence studies), vaccines or biosimilars, and b) the type of scientific components. The Experts should use the list of items provided for guidance but mainly draw on their experience and judgement to analyse and evaluate the assessment conducted by the RA on each of the areas for assessment. The Experts should aim to answer specific technical questions from a qualitative point of view. The Experts should write a summary of their findings for each of the scientific areas on how the assessment was conducted by the RA (in terms of evidence assessed by the assessor, quality of such assessment and observations, and decision-making done by the assessor). 2.3.1 Pre-clinical data 2.3.1.1 Aspects to be considered in the assessment report: ƒ Comments on adequacy in relation to the proposed protocol (study) ƒ Demonstration of relevance of the animal model ƒ Nature of the target ƒ Pharmacodynamics ƒ Pharmaco- and toxicokinetics ƒ Safety pharmacology ƒ Toxicology 94 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Overall outcome of sub-section 2.3 WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the pre-clinical data of the assessment. Based on the above, WHO Experts conclude that the overall sub-section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.3.2 Quality 2.3.2.1 Aspects to be considered in the assessment report: Investigational medicinal product (IMP), including comparators, blinded comparators, blinded test products and placebos and (if applicable) the auxiliary medicinal products quality data. The assessment of manufacturing and import information of IMP. Comments on adequacy in relation to the proposed protocol (study) The assessment report should be sufficiently detailed to allow for secondary assessment. It should focus on the compliance with the requirements concerning the manufacturing and import of investigational medicinal products and auxiliary medicinal products as well as compliance with the labelling requirements. Information to be provided for investigational medicinal products (IMPs) should focus on the risk aspects and should consider the nature of the product, the state of development/clinical phase, patient population, nature and severity of the illness as well as type and duration of the clinical trial. IMPs based on innovative and/or complex technologies may need more detailed data to be submitted. ☐ Yes ☐ No Overall outcome of the sub-section 2.3.2 WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the quality part of the assessment Based on the above, WHO Experts conclude that the overall sub-section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 95 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 3. Assessment follow-up 2.3.3 Clinical (if any) 2.3.3.1 Aspects to be considered in the assessment report: Data from previous clinical trials and human experience (if applicable) Comments on adequacy in relation to the proposed protocol (study) ☐ Yes ☐ No Overall outcome of the sub-section 2.3.3 Guidance: WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the clinical part of the assessment Based on the above, WHO Experts conclude that the overall sub-section 2.3.3 is: Please tick one of the below checkboxes. ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.3.4 Investigational brochure 2.3.4.1 Aspects to be considered in the assessment report: ƒ confidentiality statement ƒ investigational product, physical chemical and pharmaceutical properties and formulation ƒ nonclinical studies ƒ effects in humans ƒ summary of data and guidance for the investigator ƒ Consideration should be given to the completeness and adequateness of the assessment of the investigator’s brochure. 96 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Overall outcome of the sub-section 2.3.4 WHO experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the investigational brochure assessment. Based on the above, WHO Experts conclude that the overall sub-section 2.3.4 is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.3.5 Good clinical practices, good laboratory practices and good manufacturing practices compliance 2.3.5.1 Aspects to be considered in the assessment report: the assessment of validity of official documentation stating GxP compliance (e.g., good manufacturing practices certificate) Overall outcome of the sub-section 2.3.5 WHO experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the good clinical practice, good laboratory practices and good manufacturing practices compliance assessment. Based on the above, WHO experts conclude that overall sub-section 2.3.5 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 2.3.6 Study Protocol – risk benefit analysis Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 97 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.3.6.1 Aspects to be considered in the assessment report: ƒ the trial design ƒ selection and withdrawal of subjects ƒ treatment of subjects ƒ assessment of efficacy ƒ assessment of safety ƒ discontinuation criteria for participants and stopping criteria ƒ statistics ƒ data handling and record-keeping ƒ ethics and local suitability and compliance, including protection of subjects and informed consent ƒ financing and insurance ƒ quality control and quality assurance, and ƒ publication policy. The assessment report should take into account: a) The anticipated therapeutic and public health benefits, including: the characteristics of and knowledge about the investigational medicinal products; the relevance of the clinical trial, including whether the groups of subjects participating in the clinical trial represent the population to be treated, or if not, the explanation and justification; the current state of scientific knowledge; whether the clinical trial has been recommended or imposed by regulatory authorities in charge of the assessment and authorization of the placing on the market of medicinal products; the reliability and robustness of the data generated in the clinical trial, taking account of statistical approaches, design of the clinical trial and methodology, including sample size and randomization, comparator and endpoints; and (b) The risks and inconveniences for the subject, including: the characteristics of and knowledge about the investigational medicinal products and the auxiliary medicinal products; the characteristics of the intervention compared to normal clinical practice; the safety measures, including provisions for risk minimization measures, monitoring, safety reporting, and the safety plan; the risk to subject health posed by the medical condition for which the investigational medicinal product is being investigated. Overall outcome of the sub-section 2.3.6 WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific rigour” of the study protocol assessment Based on the above, WHO Experts conclude that overall sub-section 2.3.6 is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 98 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 2.4 Assessment Outcomes & Decision-making 2.4.1 Assessment Outcomes How is an overall assessment generated for an application? Are the conclusions on analysis and overall assessment outcomes consistently and adequately reached and concluded, in line with the assessment report observations, concerns and evidence reviewed? How did the assessors achieve an integrated opinion/outcome? Is there input or advice from technical committees, or from external experts? How was this or other input from ethics committee integrated into the opinion/outcomes? How were divergent views handled, if any? Overall, the assessment outcomes/opinions are aligned with the observations made throughout the assessment process. It reflects all observations and concerns as per those identified in the CT application assessment report. All input received during the assessment is adequately reflected in the report and in the opinion/outcome. Those are inclusive, comprehensive, documented and consistent. The production of the integrated opinion/outcomes from the assessors and their senior managers for the final decision-making by the agency is consistently and adequately achieved. ☐ Yes ☐ No Overall outcome of the sub-section 2.4 Guidance: WHO Experts should use the above-listed items to qualitatively evaluate the overall “scientific opinion” of the report Based on the above, WHO Experts conclude that the overall sub-section 2.4 is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Overall outcome of section 2 WHO Experts should use the above-listed items to qualitatively evaluate the overall “application process” Based on the above, WHO Experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Appendix A2.2. Expert review questionnaire: for assessing the performance of clinical trial oversight activities 99 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input 3 Assessment follow-up 3.1 Are there any changes made to the initial submission well reflected in the CT file? Further changes to the initial submission (if any) are adequately reflected in the CT file ☐ Yes ☐ No 3.2 In case of emergency approvals (or expedited approvals provided under exceptional circumstances), are there follow-ups after CT licensing with respective reflection of any update in the CT file? In case of emergency approvals (or expedited approvals provided under exceptional circumstances), there are appropriate follow- ups after CT licensing with respective reflection of any update in the CT file. ☐ Yes ☐ No Overall outcome of section 3 WHO Experts should use the above-listed items to qualitatively evaluate the overall “assessment follow-up.” Based on the above, WHO Experts conclude that overall section 3 is: Please tick one of the checkboxes below. ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text 100 Expert review of assessments of marketing authorization applications and clinical trial applications Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Evaluation Criteria Performance goal(s) to be met Score RA input WHO expert input Outcomes of the performance evaluation of clinical trials oversight activities WHO Experts’ overall conclusion of the expert review of clinical trials The overall conclusion should be based on the evaluation and scoring achieved in each of the individual three afore-mentioned sections of the questionnaire. If any one of these parts is found to be unsatisfactory according to the specific guidance provided, the overall outcome of the performance evaluation must be consequently scored as unsatisfactory. Based on the collective evidence and findings of this expert review of clinical trials oversight, the WHO Experts conclude that the performance of the clinical trials oversight activities, including application process, assessment report, and assessment follow up is: ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Vigilance field visit for assessing the performance of the vigilance function Annex 3. 102 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Contents Code of conduct .................................................................................................................................................................................................................. 103 Abbreviations .......................................................................................................................................................................................................................... 104 Glossary ................................................................................................................................................................................................................................... 105 1. Introduction ...................................................................................................................................................................................................................... 106 2. Purpose .................................................................................................................................................................................................................................107 3. Scope ................................................................................................................................................................................................................................... 108 4. Objectives and expected outcomes ................................................................................................................................................................. 109 5. Deliverables ....................................................................................................................................................................................................................... 110 6. Overview of the vigilance field visit process ................................................................................................................................................. 111 6.1 General principles .............................................................................................................................................................................................. 111 6.2 Preparing for a vigilance field visit ........................................................................................................................................................... 111 6.3 Vigilance field visit conduct.......................................................................................................................................................................... 112 6.4 Vigilance field visit report .............................................................................................................................................................................. 112 7. Roles and Responsibilities ....................................................................................................................................................................................... 113 7.1 Relevant RA ............................................................................................................................................................................................................ 113 7.2 WHO Secretariat (WHO headquarters, regional and country offices) ............................................................................ 113 7.3 WHO Team leader ............................................................................................................................................................................................. 113 7.4 WHO Team member ........................................................................................................................................................................................ 114 7.5 RA participants ..................................................................................................................................................................................................... 114 7.6 Visited site(s) .......................................................................................................................................................................................................... 114 8. Bibliography ....................................................................................................................................................................................................................... 115 9. Document Change History ...................................................................................................................................................................................... 116 Appendix A3.1. Vigilance field visit assessment questionnaire: for assessing the performance of vigilance activities ..........................................................................................................................................................................................................117 Code of conduct 103 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual WHO values and relies upon the normative and technical advice provided by leading subject matter experts in the context of its advisory/technical committees, meetings, and other similar processes. Such advice contributes to the formulation of the public health policies and norms that are promulgated by WHO for the benefit of its Member States. In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice: a. fully and honestly disclose all relevant interests and biases on the declaration of interests (DOI) Form that may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting, or group members at the outset – that is, unless this is done by the chairperson or Secretariat. b. spontaneously report any material changes to their disclosed interest on an ongoing basis during the period in which the expert serves the Organization. c. respect the confidential nature of committee or meeting deliberations or of the advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO. d. undertake not to engage in activities that may bring reputational harm to the WHO process that they are involved in. e. undertake to represent their views in a personal and individual capacity with the best interest of WHO in mind as opposed to representing the views of their employers, other institutions, or governments. f. actively and fully participate in discussions and deliberations within the relevant advisory group, committee, or meeting. WHO has zero tolerance towards sexual exploitation and abuse, sexual harassment (SEAH) and other types of abusive conduct (that is, discrimination, abuse of authority and harassment). Sexual exploitation, sexual abuse and sexual harassment are considered to be acts of serious misconduct and constitute a basis on which staff members, whether internationally or locally recruited, and contractors can be summarily dismissed. Code of conduct 104 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Abbreviations GBT Global Benchmarking Tool PE performance evaluation QMS quality management system RA regulatory authority SOP standard operating procedure VL vigilance WHO World Health Organization WLA WHO-listed authorities Glossary 105 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Glossary The definitions given below apply to the terms used in the current document. These terms may have different meanings in other contexts. Assessment questionnaire The questionnaire/form/template used for the evaluation of the performance and practice of vigilance (VL) function at several administrative levels of the target country. Field visit agenda A plan developed by the WHO Team leader, in agreement with other WHO Team members and WHO Secretariat, to detail different activities, timings, and assignments to be performed during the conduct phase of the field visit. Field visit report A report prepared in English language which is delivered by WHO team following the predefined field visit report template. Field visit report provides an overview of the field visit activities, findings and recommendations, if any. RA participants One or more experts, ideally familiar with the national medical products vigilance system, who is/are nominated by the regulatory authority (RA) to represent it and to participate in the vigilance field visit. Performance evaluation (PE) Indicators Indicator developed to assess and evaluate the performance of the vigilance function at the target country. Guidance for PE indicators is available in the form of fact sheets. Team leader A competent expert in the area of medical products vigilance with team management skills. Team leader is designated by WHO Secretariat and may or may not be a WHO staff. Vigilance field visit A process, using a WHO developed practice, that helps to document and evaluate the level of performance of vigilance function of a national medical products regulatory system. The activity consists of a field visit made by WHO team to several layers of the vigilance system (e.g., national, sub-national and health facility levels) to assess the performance and functionality of vigilance throughout the target country. The field visit may comprise onsite assessment of performance evaluation (PE) indicators of vigilance function for the purpose of designation as a WHO-listed authority (WLA). WHO Secretariat The WHO unit in charge of organization of the vigilance field visit. WHO Team The team established by the WHO Secretariat as indicated in the respective terms of reference to perform the vigilance field visit. WHO team is usually composed of three experts including a designated team leader. WHO team may be accompanied by observers when needed. WHO team members (also called WHO assessor) A competent expert, who is familiar with WHO published regulations and guidelines in the area of medical products vigilance as relevant to the scope of vigilance field visit. WHO-listed authority (WLA) A national regulatory authority or a regional regulatory system that has been documented to comply with all the indicators and requirements specified by WHO for listing based on an established benchmarking and performance evaluation process. A regulatory authority can be listed for one or more product categories or for one or more regulatory functions. 106 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 1. Introduction The medical products vigilance function, defined as the science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any other medical product‐related problems, contributes significantly to ensuring that safe and effective medical products of high quality are used within the country. One of the common regulatory functions subject to assessment during the WHO benchmarking process, in the context of capacity building or WLA designation is the medical products vigilance function. This raised the need for comprehensive assessment and evaluation of the performance and functionality of the vigilance function. In response to this need, WHO, in consultation with Member States, partners and regulatory experts, developed the process and methodology for a vigilance field visit. The vigilance field visit is an essential part of benchmarking of regulatory systems for medical products and designation as a WHO-listed authority (WLA). The guidance provided in this document will ensure consistency when organizing vigilance field visits and clearly define roles and responsibilities, which will in turn contribute to quality output and proper interaction among the involved and interested parties. 2. Purpose 107 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2. Purpose The purpose of this document is to: a. provide guidance to WHO staff and experts, the concerned RA and other interest or involved parties, on all aspects of the WHO vigilance field visit process and methodology, including procedures and timelines for planning, preparing, conducting, reporting and follow up, and templates for related documentation. b. define the roles and responsibilities of the WHO team and team members assigned to perform a vigilance field visit. d. describe the roles and responsibilities of the three levels of WHO (WHO headquarters, regional offices and country offices), as well as the concerned RA, in this process. e. establish a level of rigour, consistency and uniformity within the process for expert review of laboratory testing activities and confidence in its outcomes. This document should be read in conjunction with other relevant manuals, guidelines, standard operating procedures (SOPs) and work instructions. This document is subject to periodic review and revision as part of the quality system approach applied by WHO. 108 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 3. Scope This document describes the process for initiating, planning, preparing, conducting, reporting on and following up a vigilance field visit. It identifies the critical and key steps involved during a field visit to confirm that the performance of the vigilance function complies with applicable WHO and other internationally recognized requirements. This document applies equally to field visit pertinent to medicines and biological products, including biotherapeutic products and vaccines. However, some particularities are noted within the questionnaire for vigilance performance assessment and PE indicators; all product-specific requirements are marked accordingly in the respective documentation. 4. Objectives and expected outcomes 109 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 4. Objectives and expected outcomes The objectives and expected outcomes of the vigilance field visit are to: a. assess the performance of vigilance function activities and operations conducted at the site(s) selected for the field visit b. assess the knowledge, competence and experience of the officials and staff involved in vigilance-related activities at the selected site(s) c. identify strengths and best practices in the vigilance activities performed at the selected site(s) d. identify areas that need further improvement for which a specific development plan might be needed e. provide feedback on the performance of the vigilance function in relation to the relevant GBT sub-indicators or WLA performance evaluation (PE) indicators. 110 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5. Deliverables After completion of the vigilance field visit, the following deliverables should be provided to the WHO Secretariat: a. A vigilance field visit report (in English) to be delivered by the WHO Team. b. If applicable, an updated onsite assessment and evaluation of PE Indicators following the relevant template (included in the PE indicators scorecard (Annex 1) as part of the vigilance PE process). 6. Overview of the vigilance field visit process 111 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6. Overview of the vigilance field visit process The aim of a vigilance field visit is to assess the performance of the vigilance function, with an emphasis on vigilance systems, structure and stakeholders as well as vigilance activities such as detection, reporting and data management, case investigation and analysis, risk assessment and management, information, education and communication with concerned groups, and human and financial resources. 6.1 General principles A well-functioning vigilance system relies on multiple areas and components, including, in addition to those mentioned above, legislation and regulatory requirements, infrastructure and resources, alert and crisis systems, surveillance programmes and quality management systems (QMS). A vigilance field visit focuses on some, but not all of these aspects and is designed to be complemented by other tools and methodologies (such as the Global Benchmarking Tool (GBT) and PE indicators). It is therefore essential that these tools and methodologies are considered together and not in isolation, and that consideration is given to how the GBT assessment contributes to and interacts with the vigilance field visit and PE indicators. At the end of the assessment process, all of the available evidence should be considered. In practical terms, this means that the WHO Team performing the vigilance field visit should be well briefed and aware of the outcomes of any earlier assessment. A vigilance field visit is concerned with assessing the actual activities and operations of the vigilance system in the field, across the target country. This complement the GBT, which is concerned with systematic aspects of the vigilance function, and the PE indicators, which are concerned with quantitative and qualitative performance evaluation of the vigilance function. The RA, WHO and if necessary, the site(s) subject to the vigilance field visit should discuss and agree, in advance, all details and aspects of the visit, including the participants, the observers and translation (if any). To help the WHO vigilance field visit to evaluate the vigilance function, the RA should share with WHO a copy of vigilance-related procedures or standard operating procedures, including reporting and communication forms, preferably at least two weeks before the visit. The WHO Team should be granted unlimited access to information, people and assets relevant to the vigilance field visit, while respecting all applicable confidentiality arrangements and codes of conduct. In terms of unlimited access to people, the WHO Team should have the right to interview employees without formally respecting hierarchical lines but should always demonstrate respect for the relevant organization’s culture and habits. 6.2 Preparing for a vigilance field visit 6.2.1 Site selection Selection of the site(s) or entities subject to the vigilance field visit should be decided by agreement between the RA and the WHO Secretariat. In order to help with this, the RA should, if required, provide the WHO Secretariat with a comprehensive list of sites (including names and addresses of entities) at a specific administrative level or geographical area. In principle, the site(s) should be selected from among those that are regularly involved in vigilance-related activities (such as reporting, investigation, response). Factors to consider when selecting site(s) include a) the complexity of activities or processes, b) the criticality of products and c) the geographic and multi-ethnic reach. The ultimate objective is to have a representative sample of vigilance activities and operations. Simulations or vigilance activities scheduled for the sole purpose of the field visit should not be considered. 6.2.2 Briefing session For each individual vigilance field visit, the members of the WHO Team should be selected from the roster of qualified experts and should be thoroughly briefed on the principles described in this document before the start of the visit. The WHO Secretariat or WHO Team leader should hold a remote briefing session for all team members in preparation for the visit. The briefing should include details of: ƒ the context of the field visit, including objectives and expected outcomes ƒ the methodology for the field visit ƒ the availability of required documents ƒ how to access and utilize the WHO secure information sharing platform 112 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ƒ the roles and responsibilities of different team members, including specific task(s) ƒ other logistical arrangements (such as travel and accommodation), and ƒ answers to questions raised and clarifications sought by WHO Team members. 6.2.3 Documentation review As part of their preparations for the field visit, each member of the WHO Team – no matter how experienced – will need to spend the time preparing by reading background documents. To facilitate the process of preparing for the visit, the relevant RA coordinator(s) should upload the following documents to the secure WHO information sharing platform, at least 10 days before the start of the field visit. To the extent possible, the WHO Team should review the following documents well in advance of the visit: a. quality manual along with all standard operating procedures (SOPs), particularly those related to the medical products vigilance function b. a copy of national vigilance code/regulations/ guidelines c. background documents about the institution/ entity/site/facility that is the subject of the vigilance field visit. 6.3 Vigilance field visit conduct By default, a vigilance field visit involves an onsite evaluation. In exceptional situations, and in agreement with the RA, WHO may consider conducting a remote vigilance field visit, if this is justified by the circumstances (for example, public health emergencies involving travel restrictions). The limitations of a remote field visit should be taken into account: a remote process cannot completely replace an on-site field visit. If necessary, WHO may organize a subsequent face-to-face (physical) mission to the RA in follow up to the remote field visit, once the reasons that necessitated the remote approach have been resolved. The activities of any such face-to-face field visit will be decided on a case-by-case basis. In general, a remote field visit is discouraged. The vigilance activities and operations subject to the field visit should take place in accordance with routine practice, as defined in the procedures of the RA and in accordance with the relevant RA Quality Management System (QMS). The WHO Team may ask questions, request documents from the representatives of the visited site(s) or request to interview of one or more of the staff working at the site(s). Document review alone is not usually sufficient to assure the degree to which documents accurately reflect work activities; document review should therefore always be combined with discussions, interviews, questions and most importantly observation. To the extent possible, the WHO Team should witness actual operations and activities. Records and documents should be selected carefully for review to ensure that they are representative and adequately characterize the programme, system, or process being assessed. For the purposes of evaluating and assessing vigilance processes, operations and practice, the WHO Team should make use of the Vigilance field visit assessment questionnaire found in Appendix A3.1 to the current document. The questionnaire should be considered as an aide memoire for ensuring all critical elements are evaluated. The agenda of the vigilance field visit should be respected, but may be adjusted if necessary. Changes to the agenda should be discussed with participants from the RA. The WHO Team should review the process and plan for the vigilance field visit with the participants from the RA at agreed intervals (at the end of each working day, for example). Other participants from the site(s) visited may join one or more of these meetings. The WHO Team should provide feedback on the strengths and gaps identified so far. Throughout the field visit, the WHO Team should make clear, accurate and legible notes. These notes should provide relevant, detailed facts that serve as a record of what was assessed and evaluated and can be used for development of the field visit report. Once the field visit is completed, the WHO team should hold a de-brief meeting with the RA participants, involving other representatives from the RA (such as top management) as appropriate. The purpose of the de-brief meeting is to inform attendees about the field visit activities and present the findings, including the identified strengths, gaps, areas to be improved and recommendations, if any. The WHO Team is encouraged to prepare a presentation indicating the main findings and recommendations of the vigilance field visit for the purpose of this debrief meeting. 6.4 Vigilance field visit report The WHO Team should issue a vigilance field visit report (in English or bilingual), presenting general information about the activities, findings (strengths, gaps, and areas for improvement) and recommendations, if any. A copy of the completed questionnaire used to assess the performance of vigilance activities, provided as Appendix A3.1 to this document, should also be attached to the report. The finalized vigilance field visit report should be made available to the WHO Secretariat within 14 working days from the last day of the visit. 7. Roles and Responsibilities 113 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 7. Roles and Responsibilities The vigilance field visit should be seen as a collaborative exercise to which several parties – including the RA, WHO Secretariat, WHO Team, and the visited site(s) – are contributing. This section is meant to provide guidance on the distribution of roles and responsibilities among the aforementioned parties. 7.1 Relevant RA The relevant RA is responsible for: a. discussing and agreeing with the WHO Secretariat the selection of the site(s) that will be subject to the field visit b. designating one or more focal person to coordinate the field visit related activities c. nominating the RA participant who will join the field visit d. sharing with WHO, through the secure information-sharing platform or any other agreed means, all necessary information and documentation including, national code/ regulations/guidelines, relevant procedures, data specific to the site(s) selected for the visit e. nominating which officials will be granted access to the WHO secure information-sharing platform f. communicating and coordinating with the visited site(s), including all necessary management and logistical arrangements g. granting the WHO Team access to all relevant data and information throughout the field visit h. providing clarifications and explanations in response to questions from the WHO Team i. seeking and obtaining any consent needed from any of the stakeholder(s) involved in order to allow the relevant information to be shared with WHO. 7.2 WHO Secretariat (WHO headquarters, regional and country offices) WHO headquarters (specifically the Regulatory Systems Strengthening Team), in collaboration with WHO regional offices and relevant country offices, is responsible for: a. establishing and maintaining the tools and databases relating to field visits b. establishing a roster of qualified experts c. training experts (in order to ensure consistency and quality of the process as well as robustness of the assessment outcome) d. discussing and agreeing with the RA the selection of the site(s) that will be subject to the field visit e. establishing a dedicated country page for the field visit on the WHO information- sharing platform and uploading of all relevant documentation for access and archive purposes f. selecting the WHO Team members from the roster of qualified experts to perform the field visit on behalf of WHO g designating the WHO Team leader h. organizing any necessary contractual arrangements. 7.3 WHO Team leader The WHO Team leader is responsible for: a. leading and coordinating the vigilance field visit from the beginning to the end of the process, including participating in the evaluation and assessment of the performance and functionality of the vigilance function during the field visit b. briefing the WHO Team members about the field visit, including context, background, objectives, process and methodology, roles and responsibilities as well as safety issues, if any c. coordinating work among all members of the WHO Team in order to ensure smooth and harmonized fulfilment of the field visit, while avoiding duplication of effort and/or conflict d. communicating with RA officials on behalf of WHO e. delivering presentations (although presentations during the meetings at the open and close of the field visit will ideally be made and handled by the WHO Team leader, these presentations will need to be prepared to include inputs from different WHO team members; the WHO Team leader may also invite any of the WHO Team members to present findings, provide clarifications, or 114 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual answer the questions of the RA or the visited site as needed). f. delivering the field visit report (although the overall field visit report should ideally be prepared by the entire WHO Team, responsibility for delivering the finally agreed report lies with the WHO Team leader). 7.4 WHO Team member The WHO Team members are responsible for: a. reviewing and signing the relevant administrative documents (including the invitation letter, confidentiality agreement, and declaration of interests’ form) b. making necessary travel arrangements (for example, booking flights and obtaining visa) as described in the invitation letter c. complying with immunization requirements and bringing with them a copy of their immunization certificates, if required d. respecting all applicable protocols, ethics and codes of conduct e. assessing and evaluating the performance of vigilance operations and activities using the questionnaire attached as Appendix A3.1 to the current document f. identifying strengths, gaps and areas for improvement, if any (the strengths and areas for improvement identified should be presented in the closing meeting of the visit) g. preparing a detailed report on the field visit, including general information about the activities conducted, findings (strengths, gaps, and areas for improvement) and recommendations for addressing the identified gaps, if applicable. The field visit report should be provided to the WHO Secretariat within 14 working days of the last day of the field visit. If possible, a draft of the report should be delivered by the WHO Team on the last day of the visit. The report may quote the different components/sections in the questionnaire. 7.5 RA participants The RA participants are responsible for: a. establishing and maintaining communication between the WHO Team and the visited site b. coordinating the field visit on-site c. discussing and considering any request for adjustment of the field visit agenda d. ensuring easy access of the WHO Team to the requested documents, information and persons, and e. providing clarifications and explanations, if sought by the WHO Team. 7.6 Visited site(s) The inspected site(s) is responsible for: a. preparing all materials requested by the WHO Team, if any, prior to the planned visit b. providing clarifications and explanations sought by the WHO Team about systems and protocols used for daily activities, and c. responding to the WHO Team’s questions and calls for interview, if any. 8. Bibliography 115 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8. Bibliography Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans. Geneva: World Health Organization; 2021, in press. Vigilance (VL): Indicators and factsheets. In: WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva, World Health Organization; 2021, Marketing Authorization and registration (MA) and Clinical Trial oversight (CT): indicators and factsheets (https://apps.who.int/iris/ handle/10665/341243, accessed 11 April 2023). WHO-listed authority (WLA). Geneva: World Health Organization (https://www.who.int/initiatives/who- listed-authority-reg-authorities, accessed 9 August 2023). WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015 (https://apps.who.int/iris/handle/10665/186642, accessed 11 April 2023). 116 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 9. Document change history Version no. Date of issue Main changes 1 April 2021 First version 2 June 2023 Revision including editorial changes of the document considering the experience gained during the pilot implementation in Q4/2022 and Q1/2023. Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 117 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A3.1. About this questionnaire ƒ The objective of this questionnaire is to aid assessment of the performance of the medical products vigilance function during a vigilance field visit. The questionnaire is not intended to assess the activities of healthcare centres, other than those related to the vigilance function, such as interaction with the regulatory authority. ƒ The questionnaire has two parts, with three sections to each part: Part A Assessment of vaccine vigilance systems Section 1 — national level Section 2 — sub-national level Section 3 — health facility level Part B Assessment of medicine vigilance systems Section 1 — national level Section 2 — sub-national level Section 3 — health facility level ƒ This questionnaire includes both “open-ended” and “closed-ended” questions. ƒ The WHO team should complete the relevant fields in this questionnaire and attach a copy of the completed questionnaire to the vigilance field visit report. ƒ To avoid overlapping inputs, whenever indicated, refer to the corresponding GBT or PE indicator to verify if an assessment has been already conducted and report only the outcome. ƒ Whenever possible, please attach to the questionnaire an electronic copy of the relevant documents reviewed during the field visit. Rating (for WLA purposes only) In the context of the WLA framework, WHO uses a vigilance field visit to determine whether the RA can be considered to acceptably meet WLA requirements. For an authority to be given WLA status for the vigilance function, the entire vigilance system should achieve a satisfactory score in each section, at national, sub-national and health facility level. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 118 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part A: Assessment of vaccine vigilance systems Section 1: National level This section targets assessment of the performance of the vaccine vigilance system at the national level, namely: 1) The regulatory authority, including the central vigilance centre, and 2) The National Immunization Programme (NIP) – also called the Expanded Programme on Immunization (EPI) ID Question Guidance and value range Related GBT or PE indicators WHO assessor input General information → Country: → Institution(s) assessed: → Persons met and interviewed: Systems, structure and stakeholder coordination A-1-01 Do you have a national vigilance centre? No need to address this question as it has been addressed by the Global Benchmarking Tool (GBT) or performance evaluation (PE) indicators. VL02.01 Please refer to the related GBT or PE indicator. A-1-02 If YES to question A-1-01, is the national vigilance centre a full or associate member of the WHO collaborating centre for international drug monitoring? Yes, No. PE.VL.06 A-1-03 Do you have a designated national focal point for vaccine adverse events following immunization (AEFI)? Yes, No. If Yes, provide contact information → At RA → At national immunization programme/EPI → At Ministry of Health VL02.01 A-1-04 Do you have written national AEFI surveillance guidelines? No need to address this question as it has been addressed by the GBT or PE indicators. VL01.06 Please refer to the related GBT or PE indicator. Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 119 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-05 Do the national AEFI guidelines fulfil the WHO recommended format? Guidelines include: • objectives of the system • list of AEFI to be reported • case definitions of AEFI to be reported • clear definitions of terminology relevant for analysis and response (e.g. adverse event versus adverse reaction; coincidental, program error, serious events, cluster events) • information on how to report (who, how, where, when) • all vaccines to be included in the reporting system (not only EPI vaccines) • procedure for analysing data • feedback procedure back to key players, parents, communities of findings and relevant actions • procedure for investigating and actions to be taken in case of serious AEFI or cluster events • definition of the people in charge VL01.06 A-1-06 Have these guidelines been communicated to staff at all levels? Select all that apply • national level • sub-national level • health facility level VL03.02 A-1-07 Do EPI and RA collaborate regularly to review vaccine safety issues? Select all that apply • notifying each other on AEFI • sharing AEFI reports • convening regular meetings between the institutions • being involved in or coordinating analysis of data • sharing report analysis or summaries • jointly participating in national AEFI committee reviews • other- please specify VL02.02 A-1-08 Do you have a national database or system for collating, managing and retrieving AEFI reports? No need to address this question as it has been addressed by the GBT or PE indicators. VL04.01 VL04.02 Please refer to the related GBT or PE indicator. A-1-09 Do you have a quality management system for vaccine pharmacovigilance activities? No need to address this question as it has been addressed by the GBT or PE indicators. RS05 Please refer to the related GBT or PE indicator. 120 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-10 Do you have a management system to ensure traceability of actions? No need to address this question as it has been addressed by the GBT or PE indicators. RS05 Please refer to the related GBT or PE indicator. Overall evaluation of the systems, structure and stakeholder coordination The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Detection, reporting and data management A-1-11 Do you have written procedures on actions to be taken in case of serious AEFI or cluster of AEFIs, e.g., standard operating procedures for reporting and case management? Yes, No If Yes, please provide document. VL04.01 VL04.02 A-1-12 Is it mandatory to report serious AEFI? At national level At sub-national level At health facility level VL04.01 VL04.02 A-1-13 Is it mandatory to report non-serious AEFI? At national level At sub-national level At health facility level VL04.01 VL04.02 A-1-14 At which level is the list of AEFIs eligible for reporting disseminated? Select all that apply • national level • sub-national level • health facility level • do not have a list of eligible AEFIs VL04.01 VL04.02 A-1-15 At which level is the current case definitions for AEFI reporting disseminated? Select all that apply • national level • sub-national level • health facility level • do not have a list of eligible AEFIs VL04.01 VL04.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 121 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-16 Which type of reporting tool do you use? Select all that apply. If “Yes” to any of the below, please attach a sample. a) line-listing of AEFI cases b) case-based reporting c) aggregate reporting d) other (specify type): VL04.01 A-1-17 Are the reporting tools being used standardized for the country? Yes, No. VL04.01 A-1-18 If “Yes” for I-1-16 (a) and/or (b), please indicate whether the following minimum information are collected. Select all that apply (a) in Line-listing of AEFI cases • event • place of the event • patient • vaccine • reporter (b) in case-based reporting • event • place of the event • patient • vaccine • reporter VL04.01 A-1-19 Do you have a specified time frame for reporting serious AEFIs? Yes, No. If Yes, specify the time frame: • 24-48 hr • # of days VL04.01 VL04.02 VL05.02 A-1-20 Do you have a specified time frame for reporting non-serious AEFIs? Yes, No. If Yes, please specify the time frame: e.g. • # of days • # of weeks • # of months VL04.01 VL04.02 VL05.02 122 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-21 What is the proportion of AEFI reported within the expected timelines in previous year? % of AEFI reports within the timelines • serious AEFI • non-serious AEFI PE.VL.04 VL05.02 A-1-22 What is the proportion of AEFI reports fully completed in previous year? % of AEFI reports fully completed (= no missing data) VL05.02 A-1-23 Do you receive AEFI reports from the private sector? Yes, No A-1-24 At which level(s) is data coding/entry performed? Select all that apply • at national level • at sub-national level VL03.02 A-1-25 Are AEFI reports forwarded from EPI/AEFI system to the RA/ pharmacovigilance centre? Yes, No PE.VL.06 A-1-26 Are AEFI reports forwarded from RA/pharmacovigilance centre to the EPI/AEFI system? Yes, No PE.VL.06 A-1-27 Summary of AEFI data for last year: Are AEFI rates (serious, non-serious) consistent with expected rates? Provide summary statistics available on AEFI data reported at national level during last year Yes, No VL05.02 A-1-28 Among the AEFI reports submitted to the national level, which ones are shared with WHO UMC? Please specify • all AEFIs (A), • only serious AEFI (S) • other (O) PE.VL.06 VL05.02 Overall evaluation of the Detection, reporting and data management The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 123 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input Case investigation and analysis A-1-29 Do you have written standard procedures for case investigation? Yes, No. If yes, please provide document VL04.02 A-1-30 If yes to question 29, at what level have they been disseminated? Select all that apply • national level • sub-national level • health facility level A-1-31 Do you have case investigation forms? Yes, No. If available, please provide form VL04.02 A-1-32 How many AEFI cases have been investigated in last year? Please provide number of AEFI cases investigated in the last year VL05.02 A-1-33 Is there a monitoring of peripheral (sub-national and health facility) levels to determine whether AEFI cases were reported and investigated according to National policy? Yes, No. If yes, please specify VL02.01 A-1-34 What proportion of AEFI case investigations started within 48 hours following reporting in the last year? % of cases investigated within 48 hours VL05.02 A-1-35 What proportion of preliminary investigation reports was available within 1 week from the start of investigation in last year? % of preliminary investigation reports available within 1 week VL05.02 A-1-36 What are the expected timelines for AEFI investigation reports? Select the correct one • <6 weeks • 6-12 weeks • >12 weeks VL05.02 A-1-37 What is the proportion of AEFI investigation reports available within the expected timelines? % of AEFI investigation report within the timelines VL05.02 A-1-38 Do you have access to appropriate resources to conduct AEFI investigation? No need to address this question as it has been addressed by the GBT or PE indicators. VL03.01 VL03.02 Please refer to the related GBT or PE indicator. 124 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-39 Of the AEFI investigation conclusions available, what proportion is supported by findings? For each kind of “finding” indicate an estimated proportion of <10%, 10 to <25%, 25 to <50%, 50 to <75% OR >=75% • Lab findings (positive or negative) on clinical specimen(s) • Post-mortem findings (among AEFI deaths) • Lab findings (positive or negative) for vaccine samples VL05.02 A-1-40 Do you have any of the following summary (analysis) reports of AEFIs? Select all that apply • monthly or quarterly summary reports • annual summary reports • other (specify type) A-1-41 If YES for any type of summary reports at the previous question, then specify at which level(s) such summary reports are prepared. Select all that apply a) monthly or quarterly summary reports • national level • sub-national level • health facility level b) annual summary reports national level • sub-national level • health facility level c) other (specify type) • national level • sub-national level • health facility level Overall evaluation of the Case investigation and analysis The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 125 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input Risk assessment and management A-1-42 Do you have a national vaccine safety committee(s) for: Select all that apply: • AEFI case investigation • AEFI causality assessment • both • neither VL04.06 PE.VL.03 A-1-43 Do you have written procedures and criteria for the selection of members of the national vaccine safety committee(s)? Yes, No if Yes please attach document PE.VL.03 A-1-44 Are confidentiality and conflicts of interest appropriately regulated within the national vaccine safety committee(s)? Yes, No Please specify PE.VL.03 A-1-45 Do you have documents that clearly define the roles and responsibilities of the national vaccine safety committee(s) members? Yes, No If yes please attach document (terms of references of national vaccine safety committee(s)) VL04.06 PE.VL.03 A-1-46 Do you have documented evidence (meeting reports) of regular meetings of national immunization safety committee(s)? Yes, No VL04.06 PE.VL.03 A-1-47 Do you use WHO classification of AEFI type (vaccine product related reaction, vaccine quality defect, immunization error, immunization anxiety reaction, coincidental event)? Yes, No. State if done and at what level of reporting system • at national level • at sub-national level • at health facility level VL04.02 A-1-48 If NO to 47, is there another system you use for causality classification for AEFIs? Yes, No. If Yes (another system is used for causality classification for AEFIs), give the name or reference for the system used. A-1-49 If initial causality categorization is done for at least some of the AEFI case reports at sub-national level (or below), do you have a routine system for review and validation or final categorization? Select all that apply: • national committee • at national level (e.g., AEFI focal point) 126 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-50 In case of serious vaccine-related AEFI detected in the past three years, were regulatory decisions taken according to RA guideline (suspension, recall, update of product leaflet…) Yes, No. If yes, please specify action taken Overall evaluation of the Risk assessment and management The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Information, education and communication (iec) with concerned groups A-1-51 Do you have any document(s) that provide(s) guidance on establishment of a communication system or communication plan relevant to vaccine safety/AEFIs? Yes, No. If Yes, specify type of document and the level(s) (e.g., national) to which it applies. Please attach the document. VL02.02 A-1-52 Do you have a communication unit at National level responsible for communication with concerned groups on vaccine safety/AEFIs? Yes, No. Please specify VL02.01 A-1-53 Do you have a designated spokesperson for media enquiries relevant to vaccine safety or AEFI? Yes, No. If yes, name, affiliation. VL02.01 A-1-54 Do you have a written communication plan in case of vaccine safety crisis? Yes, No. Please specify VL02.01 A-1-55 Does your organization regularly check the local, including social, media for reports of adverse events? Yes, No. A-1-56 Do you have information material/leaflets relevant to vaccine safety/AEFI issues developed for community, vaccinees and parents? Yes, No. Specify • Community • Vaccinees and parents Please make sure to request and check the materials, if any VL06.01 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 127 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-57 Do you provide/share information relevant to vaccine safety/AEFI to the private sector? Yes, No. VL06.01 A-1-58 How often do you share AEFI investigation outcomes with concerned groups Please indicate corresponding score: Almost never=1; Occasionally=2; Often=3; Almost always=4 • AEFI reporters • immunization staff/other health care providers • parents/vaccinees/community • media VL06.01 A-1-59 Please describe any vaccine safety crisis that recently occurred; use the checklist in next column as a guide to elements to include in your brief description. • what specific AEFI or vaccine safety issue it involved • date when it occurred • how promptly the situation was handled (timing of initial response) • whether you had a focal point or unit for communication • how promptly you responded to the community and AEFI reporters • if an investigation was conducted and how long it took to complete the investigation • what was the impact of this incident on your immunization programme (vaccine acceptance and/or coverage, resources and staff, other) Overall evaluation of the Information, education and communication with concerned groups The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text 128 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input Human and financial resources A-1-60 Is there a budget component specific for the AEFI surveillance system available? Select all that apply • at national level • at sub-national level • at health facility level A-1-61 Is there a specific budget line for AEFI case management (treatment of the person with suspected AEFI)? a) for routine immunization b) for immunization campaign If yes, specify: • name of document • service where document can be found A-1-62 Do you have pre-assigned investigation team(s) responsible for AEFI investigation when needed? Select all that apply • at national level • at sub-national level If YYes briefly describe the team composition (e.g., paediatrician, epidemiologist, Immunization supervisor etc.) of the persons in the pre-assigned team(s) VL03.01 A-1-63 What percent (%) of staff involved in AEFI surveillance (reporting, investigating or managing cases) have attended training relevant to AEFI/vaccine safety last year? For each level, indicate an estimated proportion of <10%, 10 to <25%, 25 to <50%, 50 to <75% OR >=75% • at national level • at sub-national level • at health facility level VL03.03 PE.VL.02 A-1-64 On average, in all training activities relevant to AEFI/vaccine safety conducted last year, what proportion of trainees/participants have been staff from the private sector (physicians and other health care workers)? Numerator = number of staff from private sector attended the training Denominator = total number of participants attended the training VL03.03 VL03.04 PE.VL.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 129 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-65 Is there a document where information on vaccine safety trainings is reported (including number of participants, course description/agenda)? Yes, No Specify: • Name of document • Training plan • Training report • Other, specify Service where document can be found PE.VL.02 A-1-66 Which type of training relevant to AEFI has been provided in the last year? Please describe VL03.03 A-1-67 Is updated information (including training materials) on AEFI detection and reporting procedure provided to health staff at all levels? Select all that apply • at national level • at sub-national level • at health facility level PE.VL.02 Overall evaluation of the Human and financial resources The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Vaccine utilization A-1-68 Current routine childhood immunization schedule. Please provide list, table or PowerPoint slide A-1-69 List of vaccines used in EPI programme in your country. Provide list of vaccines currently used in EPI programme A-1-70 Do you receive information on total # of doses distributed? Select all that apply • at national level • at sub-national level • at health facility level 130 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range Related GBT or PE indicators WHO assessor input A-1-71 Do you receive information on lot/batch # of doses distributed? Select all that apply • at national level • at sub-national level • at health facility level A-1-72 Do you receive information on total # of doses administered? Select all that apply • at national level • at sub-national level • at health facility level A-1-73 Do you receive information on lot/batch # of doses administered? Select all that apply • at national level • at sub-national level • at health facility level Overall evaluation of the Vaccine utilization The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Overall evaluation of the vigilance system at the national/central level The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 131 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input General information → Institution(s) assessed: → Persons met and interviewed: Systems, structure and stakeholder coordination A-2-01 Do you have contact information of designated national focal point for vaccine AEFI? Yes, No. If Yes, provide contact information A-2-02 Are you aware of the written National AEFI surveillance guidelines? Yes, No. A-2-03 Have these guidelines been communicated to your staff? Yes, No. A-2-04 Interview some staff using the respective guidance and ask if they have read the guidelines and assess their knowledge of the contents: Questions to guide the discussion • What is an AEFIs? • Do you have a list of AEFIs eligible for reporting? • Do you have AEFI case definitions for expected vaccine reactions? • How is the reporting done to this level, what forms are used? • How is the reporting from this level done? To whom? Routinely – nil reports? What frequency? Part A: Assessment of vaccine vigilance systems Section 2: Sub-national level This section targets assessment of the performance of vaccine vigilance system at the sub-national levels, namely: a) regional regulatory authorities (for example, at state or provincial levels), and b) Regional Immunization Programme Guidance: ƒ Identify critical issues to be assessed during the data collection process (from the information collected at the national level, background documents provided and the informal information gathered). ƒ Do not attempt to ask all the questions listed as discussion points. Instead, try to focus on the critical issues the team agreed after the data was collected at national and health facilities levels. ƒ You will have more success obtaining information if you try to establish an open dialogue with health staff and stakeholders and observe them while they are working. ƒ If necessary, this section can be repeated if several institutions at this level are being visited. If so, please clearly indicate the visited site/facility and its pertinent information. 132 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input • Communication mechanism (phone, fax, email?) • What is the time frame for reporting cases? • Do they know the local drug inspector(s)/ RA officials? • Were they involved in AEFI investigation for previous AEFIs • In the last year, were there any joint RA EPI meetings /trainings? • Is there an AEFI committee at this level? Is this functional? Who are the members? How frequently does this committee meet? How do you support the AEFI committee? Detection, reporting and data management A-2-05 • Review AEFI reports received from health facilities and investigations in the last year. Review numbers of AEFI reports received in the last year and compare to the number of reports received in the year before the last one. • Estimate the rate of AEFIs reported by comparing AEFIs with the number of doses of vaccine administered. • Look at the AEFI reports and data management process • Look at the AEFI reports submitted to national level. Questions to guide the discussion • Which AEFI (serious/non serious) are reported from operational level and how (forms, communication mechanism, frequency, timelines)? • How do you decide which cases should be reported as AEFI cases? • Do you have a list of AEFIs eligible for reporting? • Do you manage AEFI cases not reported to supervisor? If YES, do you refer to those when you find similar case? • Do you have AEFI case definitions for expected vaccine reactions? Ask what the expected vaccine reactions are for specific vaccines • Do you compile, analyse and interpret AEFI data you receive on a regular basis? How often? • Do you have procedure for analysing the data? • How do you decide which AEFI cases should be investigated? • Which AEFI are communicated to the national level? To whom? (EPI? Pharmacovigilance centre?) • Where do you register the AEFIs? Do you have a database or repository? Do you have designated personnel/data manager for data entry? • How do you proceed with reporting to national level? • Which forms/mechanism do you used? Can you please show me those forms? • To whom do you report, and when? • How do you send AEFI reports - electronically, hardcopy? • Is AEFIs reporting included into routine immunization reports to national level? • Have you ever received feedback from your supervisor/national level? How often do you receive feedback? • If you received a request to fill missing information to AEFI case, which you report to your supervisor, do you respond? If YES, how often? Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 133 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input Case investigation and causality assessment A-2-06 Review the availability of standard operating procedures (SOPs) for case investigation Questions to guide the discussion • Please describe what do you do when you receive a serious AEFI report from health facility? • Do you have written procedure for investigating and actions to be taken in case of serious AEFI or cluster events? • Who conducts the investigation? • Do you have standard form for investigation? Do you have SOP for specimen collection? Forms associated? • Assess whether the different types of AEFIs are known (vaccine reaction, vaccine quality defect, immunization error…)? • In the last year, how many AEFI have you investigated personally? What were the outcomes? Was there any impact on the program? • Are you familiar with the Brighton collaboration definition? • Who joins you for AEFI investigations? • How frequently do you conduct discussion of the results of investigated cases among your staff? • Have you ever conducted cross checking of investigation results among investigators for consistency of investigation? • If you find missing information in the reported AEFI, do you request reporter to provide such information? If YES, how often? Do they respond to your request? Information, education and communication (IEC) with concerned groups (AEFI reporter (that is, the person who reported AEFI, not only health care provider), parents, vaccinees, public, community, immunization staff, other health care providers, AEFI case, investigators, media etc..) A-2-07 Review IEC materials, including training materials (slides, booklets, SOPs), posters, leaflets. Questions to guide the discussion • Have you conducted training for AEFI investigation in the last year? How many? For whom? When? Can you please show me some of training materials? • Has AEFI reporting improved after training? • Do health workers feel comfortable reporting programme errors? Are they confident that they will not be blamed by the department? • Do parents/public report minor AEFI (e.g., fever/pain) first to the staff who vaccinated or to the medical officer? • Do you have information material/leaflets relevant to vaccines and AEFI to communicate to health care workers? 134 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input • In case of previous serious AEFI, were the results of the investigation shared with the vaccinee/ parents/community? How? By whom? How long after the event? • Do you receive regular information on vaccine safety and AEFI (newsletter, epidemiological bulletin…)? Do you share that information with health care workers? • Do you actively collect vigilance information? If yes, please specify what kind of information and how do you collect. • If you think correction of vigilance information distributed by Ministry of Health, EPI deemed necessary, do you provide your feedback to the source of the information? What frequency? Human and financial resources A-2-08 Staffing: Review staffing list for the facility and qualification Questions to guide the discussion • Ask staff if there are enough of the right kind of staff in the facility. If not, ask them to give you details. • How many posts are now vacant in the health facility? • In the last year, have you solicited the assistance of your supervisors/next level for AEFI investigations? A-2-09 Training: WHO team should review …. • information material on AEFI detection, reporting and management • training material and certificate Questions to guide the discussion • Have you ever attended a training on AEFI? If yes, which type of training, when was it? • Is updated information (including training materials) on AEFI detection and reporting procedure provided? • Do you maintain training record of your staff and if they were not attended regularly (at least once a year), do you encourage them to attend? • Do you organize regular training on AEFI for health care workers? If YES, could we look some of training materials? A-2-10 Supervision: health workers’ performance is regularly evaluated and feedback provided The WHO Team should review: • health worker performance ’s reports. • supervisor’s reports. Questions to guide the discussion • Ask the district officer/medical officer to tell you who has visited them from the national level. How often do they visit? What do the visitors do while they are in the facility? • Ask whether she/he conducts regular review/observation of health workers performance, how? how often? Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 135 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input Overall evaluation of the vigilance system at the sub-national level The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text 136 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part A: Assessment of vaccine vigilance systems Section 3: Health facility level This section is targeting the assessment of the performance of vaccine vigilance system at the sub- national levels, namely: a) immunization centre b) points of care (POC) Guidance: ƒ Identify critical issues to be assessed during the data collection process (from the information collected at the national and sub-national levels, background documents provided, and the “informal” information gathered). ƒ Do not attempt to ask all the questions listed as discussion points. Instead, try to focus on the critical issues the team agreed after the data was collected at national and health facilities levels. ƒ You will have more success obtaining information if you try to establish an open dialogue with health staff and stakeholders and observe them while they are working. ƒ If necessary, this section can be repeated in case of visiting several institutions at this level. If so, please clearly indicate the visited site/facility and its pertinent information. ID Question Guidance and value range WHO assessor input General information → Institution(s) assessed: → Persons met and interviewed: Systems, structure and stakeholder coordination A-3-01 Do you have contact information of the designated national focal point for vaccine AEFI? Yes, No. If Yes, provide contact information A-3-02 Are you aware of the written national AEFI surveillance guidelines? Yes, No. A-3-03 Have these guidelines been communicated to your staff? Yes, No. A-3-04 Interview some staff using the respective guidance and ask if they have read the guidelines and assess their knowledge of the contents: Questions to guide the discussion • What is an AEFI? • Do you have a list of AEFIs eligible for reporting? • Do you have AEFI case definitions for expected vaccine reactions? • How is the reporting done to this level, what forms are used? • How is the reporting from this level done? To whom? Routinely – nil reports? What frequency? • Communication mechanism (phone, fax, email?) • What is the time frame for reporting cases? • Do they know the local drug inspector(s)/RA officials? Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 137 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input • Were they involved in AEFI investigation for previous AEFIs? • In the last year, were there any joint RA EPI meetings /trainings? • Is there an AEFI committee at this level? Is this functional? Who are the members? How frequently does this committee meet? How do you support the AEFI committee? • Do you receive regular information on vaccine safety and AEFI (newsletter, epidemiological bulletin…)? Do you share that information with health care workers? • Do you actively collect vigilance information issued by Ministry of Health, EPI? If YES, please specify what kind of information and how do you collect. Detection and management A-3-05 • Review AEFI reports received from health facilities and investigations in the last year. Review numbers of AEFI reports received in the last year and compare to the number of reports received in the year before last one. • Estimate the rate of AEFIs reported by comparing AEFIs with the number of doses of vaccine administered. Questions to guide the discussion • Have you ever had AEFI at your health facility? • If yes, do you know what to do to help the patient with AEFI at the first minutes, when to call for emergency? • Do you have emergency kit? Can you please show me the kit? Have you been trained on how to use this? • Before each session, do you inform vaccinees/parents about possible adverse reaction after immunization? • How do you decide which cases should be reported as AEFI cases? • Do you have a list of AEFIs that should be reported? • Do you have AEFI case definitions for expected vaccine reactions? Ask health workers what the expected vaccine reactions for specific vaccines are AEFI reporting A-3-06 The WHO Team should look at and review: • the periodic reports (routine reports) sent from the institution • the AEFI reports sent and check the timelines and completeness, compare consistency with the onsite logbook/registry. Questions to guide the discussion • Have you ever reported an AEFI? • Where do you register the AEFIs? Do you have logbook, can I see it? • How do you proceed with reporting? • Which forms do you used? Can you show me those forms? • To whom do you report, and when? • How do you send AEFI reports: electronically, hardcopy? • Ask and check if AEFI reports are submitted on time. If not, why? • Do you include AEFIs reports into routine immunization reports to higher supervisory level? • If you ever reported AEFI case, have you received feedback from your supervisor(positive/ negative)? How often do you receive feedback? • If you receive a request to fill missing information to AEFI case, which you report to your supervisor, do you respond? If YES, how often? 138 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input Information, education and communication (iec) with concerned groups (AEFI reporter [person who reported AEFI, not only health care provider], parents, vaccinees, public, community, immunization staff, other health care providers, AEFI case, investigators, media etc.) A-3-07 Review IEC materials, including training materials (slides, booklets, SOPs), posters, leaflets. Questions to guide the discussion • Have you conducted training on AEFI investigation in the last year? How many? For whom? When? Can I see some of training materials? • Has AEFI reporting improved after training? • Do health workers feel comfortable reporting programme errors? Are they confident that they will not be blamed by the department? • Do parents/public report minor AEFI (e.g., fever/pain) first to the staff who vaccinated or to the medical officer? • Do you have information material/leaflets relevant to vaccines and AEFI to communicate to health care workers? • In case of previous serious AEFI, were the results of the investigation shared with the vaccinee/parents/ community? How? By whom? How long after the event? • Do you receive regular information on vaccine safety and AEFI (newsletter, epidemiological bulletin…)? Do you share that information with health care workers? • Are there any anti-vaccination groups communicating concerns about AEFI? Human and financial resources A-3-08 Staffing: Review staffing list for the facility and qualification Questions to guide the discussion • Ask staff if there are enough of the right kind of staff in the facility. If not, ask them to give you details. • How many posts are now vacant in the health facility? A-3-09 Training: The WHO Team should review …. • Information material on AEFI detection, reporting and management • Training material and certificate Questions to guide the discussion • Have you ever attended a training on AEFI? If yes, what type of training, when was it? • Is updated information (including training materials) on AEFI detection and reporting procedure provided? • Do you maintain training records of your staff and if training has not been attended regularly (at least once a year), do you encourage them to attend? • Have you been a resource person in trainings? A-3-10 Supervision: The WHO Team should review …. • supervisor’s reports. Questions to guide the discussion • Ask health workers to tell you who has visited them from the district/ regional office. How often do they visit? What do the visitors do while they are in the facility? Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 139 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Guidance and value range WHO assessor input Overall evaluation of the vigilance system at the health facility level The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text 140 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part B: Assessment of medicine vigilance systems Section 1: National level This section targets the assessment of the performance of medical product vigilance system at the national levels, namely: 1. Regulatory authority (RA)/national vigilance centre (please note that majority of the assessment of the RA/national vigilance centre is covered by the global benchmarking tool (GBT) as well as the PE indicators). 2. Central Health Programme (e.g., HIV, noncommunicable diseases, malaria, TB, tropical diseases, and others), if applicable. Note that not all countries run the health system though a public health programme (PHP). In the latter case, the relevant part of the below questionnaire would apply to Ministry of Health or its disease surveillance programme. In all cases, the public health programme, Ministry of Health and/or disease surveillance programme should be involved in the performance evaluation if they have an active role in pharmacovigilance within the country. ID Question Value range Guidance Related GBT or PE indicators WHO assessor input General information → Institution(s) assessed: → Persons met and interviewed: Systems, structure and stakeholder coordination B-1-01 Do you have a designated National focal point for medical product vigilance? Yes, No. If Yes, provide Terms of Reference and contact information • At RA • At public health programme • At Ministry of Health Identifying the post with the ultimate responsibility for the national medical product vigilance is essential VL02.01 B-1-02 Are guidelines for medical product vigilance included within the strategic and/or annual operational plans of your public health programme? No need to address this question as it has been addressed by the GBT or PE indicators. No need to address this question as it has been addressed by the GBT or PE indicators. VL01.06 VL02.02 B-1-03 Have these guidelines for medical product vigilance been communicated to staff at all levels? Select all that apply • national level • sub-national level • health facility level Availability of vigilance guidelines throughout the organization to be assured VL03.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 141 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-1-04 Do your public health programme and RA collaborate regularly to review medical product safety issues? Select all that apply • notifying each other on medical product safety issues • sharing medical product individual case safety reports • convening regular meetings between the institutions • being involved with or coordinating analysis of data • Sharing report analysis or summaries • Jointly participating in national medical product vigilance advisory committee reviews • other - please specify. Regular and close collaboration between public health programme and RA/vigilance function is essential VL02.02 B-1-05 Do you have a national system for collating, managing and retrieving reports of suspected adverse reactions to medical products? No need to address this question as it has been addressed by the GBT or PE indicators. No need to address this question as it has been addressed by the GBT or PE indicators. VL04.01 VL04.02 Overall evaluation of Systems, structure and stakeholder coordination The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Detection, reporting and data management B-1-06 Do you have written procedures on actions to be taken in case of serious medical product related safety concerns e.g., standard operating procedures for reporting and case management? Yes, No If Yes, please provide document An action plan for crisis management should be in place VL04.01 VL04.02 142 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-1-07 Which reporting tool do you use for individual case safety reports of medical products? Specify if different from tool used by national vigilance centre VL04.01 B-1-08 Is the reporting tool being used, standardized for the country? Yes, No If No record justification VL04.01 B-1-09 At which level(s) is data coding/entry performed? Select all that apply • At national level • At sub-national level VL04.02 B-1_10 Are all reported individual case study reports forwarded from the public health programme system to the RA/vigilance centre? Yes, No If No, provide justification Important to establish that all reports meeting the minimum criteria for completeness are shared with the RA/vigilance centre. No potentially embarrassing cases should be hidden. PE.VL.06 B-1-11 Are summary rates of individual case safety reports last year consistent with expected rates? Yes, No If No, provide justification Reasons for large annual variations should be investigated VL05.02 Overall evaluation of Detection, reporting and data management The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Case investigation and analysis B-1-12 Who is in charge of investigation of adverse drug events? How is quality of such investigation assured? Adverse drug events investigation report A high level of assurance should be established with respect to investigation of adverse drug events. PE.VL.04 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 143 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-1-13 How many active medical product safety surveillance studies have been conducted in the last three years (36 months) in your public health programme? Indicate type of study (e.g., cohort event monitoring, targeted spontaneous reporting) and stage of completion (e.g., initiated, on-going or completed) for each study Engagement in active safety surveillance indicates ambitions to learn about mechanisms and risk factors, enabling future prevention VL04.08 PE.VL.07 Overall evaluation of Case investigation and analysis The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Risk assessment and management B-1-14 Does your public health programme have representation in the national vigilance advisory committee? Select all that apply • for individual case safety reports causality assessment • individual case study reports signal investigation • other If YES, strengthens II-1-04 and documents a coherent vigilance system VL04.06 PE.VL.03 B-1-15 Have any medical product-related problem, detected in the past three years in your public health programme resulted in a regulatory decision by the RA (suspension, recall, update of product leaflet…)? Yes, No. If yes, please specify action taken If YES, supports impression of a functional and coherent national vigilance system. If NO, can be due to lack of actual safety concerns but also due to lack of communication. VL04.03 PE VL.09 B-1-16 How many medicine safety issues identified from outside sources were acted on at national level in the previous year? Outside sources refer to literature data or information from other countries Important for patient safety to be alert to new and relevant international data. Lack of identified such issues does not prove failure. PE VL09 144 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-1-17 What is the number of suspected product quality problems detected through the public health programme in the previous year? Record statistics if available If the vigilance system is considered to be an important component in the national fight against sub-standard and falsified medicines this question should be documented carefully, otherwise it is not critical PE.VL.08 Overall evaluation of Risk assessment and management The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Information, education and communication (iec) with concerned groups B-1-18 Do you have any document(s) that provide(s) guidance on establishment of a communication system or communication plan relevant to safety of medical products used in your programme? Yes, No. If Yes, specify type of document and the level(s) (e.g., national) to which it applies. Please attach the document. The availability of a communication system and plan for medical product safety is essential VL02.02 VL06.02 PE.VL.01 B-1-19 Do you have a communication unit at national level responsible for communication with concerned groups on safety of medical products used in your programme? Yes, No. Please specify Identification of the responsible office or manager for communication of medical product safety issues is required VL02.01 VL06.02 PE VL 01 B-1-20 Do you have a designated spokesperson for media enquiries relevant to the safety of medical products used in your programme? Yes, No. If yes, name, affiliation. A spokesperson for media questions should be identified VL02.01 VL06.02 PE VL01 B-1-21 Do you have a written communication plan in case of a safety crisis related to medical products used in your programme? Yes, No If Yes, specify the level(s) (e.g., national) to which it applies. Please attach the document. A crisis communication plan should be developed jointly between the public health programme and the RA VL02.01 PE VL01 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 145 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-1-22 Do you have information material/website, free telephone line etc. by which relevant safety information of medical products used in your programme is made available to the community? Yes, No. Specify • Community • Children and parents An information service should be available for the community, preferably developed in collaboration with the RA/vigilance function. VL06.01 PE VL01 B-1-23 How many public or community education activities relating to medical product safety were carried out by the public health programme in the previous year? Specify method of training and number of activities Follow-on question to II-1-22 VL 02.02 PE.VL.02 B-1-24 How many requests for information about medical product safety were received in the previous year? How many were addressed? Provide communication channels and numbers if available Not critical if statistics are not available VL 02.02 VL 06.01 B-1-25 How long does it take from when a medical product safety signal or significant safety issue is identified to when it is communicated to health workers and the public? Provide time estimate in number of days The efficiency of the regulatory system in terms of giving priority to actions to protect patients at risk is an important indicator to record. VL04.03 PE VL.09 B-1-26 Are pharmacovigilance data being considered when updating standard treatment guidelines for your PHP? Explain frequency and process of guideline update The main justification for vigilance activities is to improve future practices. The use of vigilance data to achieve this needs to be documented. VL 05.01 VL.06.02 Overall evaluation of information, education and communication with concerned groups The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Human and financial resources B-1-27 Is there an annual budget component specific for vigilance of medical products used in your programme? Specify public and donor funding 146 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-1-28 Is there a specific budget line for case management of patients affected by adverse effects of medical products used in your programme? Yes or No B-1-29 Do you have pre-assigned investigation team(s) responsible for investigation of suspected medical product related adverse reactions when needed? Yes or No Select all that apply • at national level • at sub-national level VL03.01 B-1-30 What percentage (%) of staff involved in patient management component of your programme have attended training relevant to safety surveillance of medical products last year? For each level, indicate an estimated proportion of <10%, 10 to <25%, 25 to <50%, 50 to <75% OR >=75% • At national level • At sub-national level • At health facility level Maintenance of system for continuous competence development in safety surveillance is critical for the long-term operation VL03.03 PE.VL.02 B-1-31 Is there a document where information on medical product safety surveillance training is reported (including number of participants, course description/agenda)? Yes, No Specify: • Name of document: • Training plan • Training report • Other, specify Documentation of safety surveillance training on an individual level should be required PE.VL.02 B-1-32 Which type of training relevant to medical product vigilance has been provided in the last year? Please describe Evidence of recent performance in competence development to be provided VL03.03 B-1-33 Is updated information (including training materials) on medical product safety surveillance, including detection and reporting procedures, provided to health staff at all levels? Select all that apply • at national level • at sub-national level • at health facility level Implementation and follow-on from II-1-03 PE.VL.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 147 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input Overall evaluation of Human and financial resources The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text. Overall evaluation of the vigilance system at the national level The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text 148 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part B: Assessment of medicine vigilance systems Section 2: Sub-national level This section targets assessment of the performance of the medicine vigilance system at the sub-national levels, namely: 1) regional regulatory bodies (e.g., at state or provincial levels) if applicable, and 2) regional health programme (e.g., HIV, noncommunicable diseases, malaria, TB, tropical diseases, and others), if applicable. ID Question Value range Guidance Related GBT or PE indicators WHO assessor input General information → Institution(s) assessed: → Persons met and interviewed: Systems, structure and stakeholder coordination B-2-01 Do you have contact information of designated national focal point for medical product vigilance? Yes, No. If Yes, provide Terms of Reference and contact information • at RA • at public health programme • at Ministry of Health It is essential to identify the post with ultimate responsibility for the national medical product vigilance in the public health programme VL02.01 B-2-02 Are guidelines for medical product vigilance included within the strategic and/or annual operational plans of your public health programme? No need to address this question as it has been addressed by the GBT or PE indicators. No need to address this question as it has been addressed by the GBT or PE indicators. VL01.06 VL02.02 B-2-03 Have these guidelines for medical product vigilance been communicated to staff at all levels? Select all that apply national level • sub-national level • health facility level Availability of vigilance guidelines throughout the organization to be assured VL03.02 B-2-04 Do your centre and national pharmacovigilance centre collaborate regularly to review medical product safety issues? Select all that apply • notifying each other on medical product safety issues • sharing medical product individual case safety reports • convening regular meeting between the institutions Regular and close collaboration between national pharmacovigilance centre and RA/vigilance function is essential VL02.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 149 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input • being involved with or coordinating analysis of data • sharing report analysis or summaries • jointly participating in national medical product vigilance advisory committee reviews • other – please specify Overall evaluation of Systems, structure and stakeholder coordination The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Detection, reporting and data management B-2-05 Do you have written procedures on actions to be taken in case of serious medical product related safety concerns e.g., standard operating procedures for reporting and case management? Yes, No If Yes, please provide document An action plan for crisis management should be in place VL04.01 VL04.02 B-2-06 Which reporting tool do you use for individual case safety reports of medical products? Specify if different from tool used by national vigilance centre VL04.01 B-2-07 At which level(s) is data coding/entry performed? Select all that apply • at national level • at sub-national level VL04.02 B-2-08 Are all reported individual case safety reports to your centre forwarded to the national pharmacovigilance centre? Yes, No If No, provide justification Yes, No If No, provide justification Important to establish that all reports meeting the minimum criteria for completeness are shared with national pharmacovigilance centre. No potentially embarrassing cases should be hidden. PE.VL.06 150 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Risk assessment and management B-2-09 Does your centre have representation in the national vigilance advisory committee? Select all that apply ƒ for individual case safety report causality assessment ƒ individual case study report signal investigation ƒ other If YES, strengthens II-1-04 and documents a coherent vigilance system VL04.06 PE.VL.03 B-2-10 Has any medical product-related problem, detected in the past three years in your centre resulted in a regulatory decision by the RA (suspension, recall, update of product leaflet…)? Yes, No. If yes, please specify action taken If YES, supports impression of a functional and coherent national vigilance system. If NO, can be due to lack of actual safety concerns but also due to lack of communication. VL04.03 PE VL.09 B-2-11 How many medicine safety issues identified from outside sources were acted on locally in the previous year? Outside sources refer to literature data or information from other countries Important for patient safety to be alert to new and relevant international data. Lack of identified such issues does not prove failure. PE VL09 B-2-12 What is the number of suspected product quality problems detected through the public health programme in the previous year? Record statistics if available If the vigilance system is an important component in the national fight against sub-standard and falsified medicines this question should be documented carefully, otherwise it is not critical PE.VL.08 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 151 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input Overall evaluation of Risk assessment and management The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Information, education and communication (iec) with concerned groups B-2-13 Do you have any document(s) that provide(s) guidance on establishment of a communication system or communication plan relevant to safety of medical products used in your programme? Yes, No. If Yes, specify type of document and the level(s) (e.g., national) to which it applies. Please attach the document. The availability of a communication system and plan for medical product safety is essential VL02.02 VL06.02 PE.VL.01 B-2-14 Do you have a communication unit responsible for communication with concerned groups on safety of medical products used in your programme? Yes, No. Please specify Identification of the responsible office or manager for communication of medical product safety issues is required VL02.01 VL06.02 PE VL 01 B-2-15 Do you have a designated spokesperson for media enquiries relevant to the safety of medical products used in your programme? Yes, No. If yes, name, affiliation. A spokesperson for media questions should be identified VL02.01 VL06.02 PE VL01 B-2-16 Do you have a written communication plan in case of a safety crisis related to medical products used in your programme? Yes, No If Yes, specify the level(s) (e.g., national) to which it applies. Please attach the document. A crisis communication plan should be developed jointly between the public health programme and the RA VL02.01 PE VL01 B-2-17 Do you have information material/website/ free telephone line etc. by which relevant safety information of medical products used in your program is made available to the community? Yes, No. Specify • Community • Children and parents An information service should be available for the community, preferably developed in collaboration with the RA/ vigilance function. VL06.01 PE VL01 152 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-2-18 How many public or community education activities relating to medical product safety were carried out by your centre in the previous year? Specify method of training and number of activities Follow-on question to II-1-22 VL 02.02 PE.VL.02 B-2-19 How many requests for information about medical product safety were received in the previous year? How many were addressed? Provide communication channels and numbers if available Not critical if statistics is not available VL 02.02 VL 06.01 B-2-20 How are you providing feed-back to internal individual reporters of medical product related case safety reports? Specify all that apply • acknowledgement (electronic/ paper/verbal, automatic or not) • feedback with case assessment • advise re. possible prevention no feedback Identify mechanisms available to stimulate, acknowledge and give feedback to reporters including the result of the local causality assessment made. VL04.02 Overall evaluation of Information, education and communication with concerned groups The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Human and financial resources B-2-21 Do you have pre-assigned investigation team(s) responsible for investigation of suspected medical product related adverse reactions when needed? Yes or No VL03.01 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 153 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-2-22 What percentage (%) of staff involved in patient management component of your centre have attended training relevant to safety surveillance of medical products last year? Indicate an estimated proportion of <10%, 10 to <25%, 25 to <50%, 50 to <75% OR >=75% Maintenance of system for continuous competence development in safety surveillance is critical for the long-term operation VL03.03 PE.VL.02 B-2-23 Is there a document where information on medical product safety surveillance training is reported (including number of participants, course description/agenda)? Yes, No Specify: • Name of document • Training plan • Training report • Other, specify Documentation of safety surveillance training on an individual level should be required PE.VL.02 B-2-24 Which type of training relevant to medical product vigilance has been provided in the last year? Please describe Evidence of recent performance in competence development to be provided VL03.03 Overall evaluation of Human and Financial resources The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text Overall evaluation of the vigilance system at the sub-national level The WHO Team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text 154 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part B: Assessment of medicine vigilance systems Section 3: Health facility level This section targets the assessment of the performance of medicine vigilance system at the health facility levels – that is, hospitals, polyclinics, or other points of care (POC). ID Question Value range Guidance Related GBT or PE indicators WHO assessor input General information → Institution(s) assessed: → Persons met and interviewed: Systems, structure and stakeholder coordination B-3-01 Does the health facility have a functional Drug and Therapeutics Committee (DTC) or equivalent, responsible for vigilance activities? Provide Terms of Reference for Drug and Therapeutics Committee and minutes from latest meetings The governance and management structure for medical product vigilance in the facility needs to be established VL02.01 B-3-02 Within the previous year, has the Drug and Therapeutics Committee carried out any vigilance activities or addressed medicine safety issues? Specify • kind of activity • purpose of activity • number of activities The level of recent activity and engagement in vigilance activities to be established VL02.01 B-3-03 Do you have designated focal point for medical product vigilance in the health facility? Yes, No. If YES, provide Terms of Reference and contact information The responsible person for medical product vigilance to be identified VL02.01 B-3-04 Which are the reporting lines between the vigilance focal point, the Drug and Therapeutics Committee and the management of the health facility? Should be clear from Terms of Reference Reporting lines in the management structure for medical product vigilance in the facility to be identified VL02.01 VL03.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 155 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input Detection, reporting and data management B-3-05 By which mechanisms are suspected medical product related adverse events or problems identified in your health facility? Specify all that apply • patient counselling and diagnosis • laboratory and other test results • systematic chart reviews etc. This is to identify all sources of reports of medical product adverse events within the facility VL04.01 B-3-06 How are suspected medical product related adverse events reported within your health facility? Specify all that apply • paper forms • web-based form • SMS • communication app • patient record system etc. Understanding the communication channels for medical product adverse events in the facility is essential and allows identification of possible gaps VL04.01 B-3-07 Which kind of medical product related problems are reportable? Specify all that apply • suspected adverse effects • lack of effect • medical product quality problem • medication errors The coverage and scope of the internal vigilance system is investigated, allowing detection of omissions VL04.01 B-3-08 Who is entitled (authorized) to reports medical product related adverse events in your health facility? Specify all that apply • anybody • assistant nurses • dentists • doctors • nurses • patients • pharmacists Allows identification of possible hierarchical hurdles in the sensitivity of the vigilance system if certain categories are kept from reporting directly VL04.01 156 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-3-09 How many medical product adverse event reports were recorded from the health centre in the previous year? Provide statistics, if available, preferably specified by category: a) adverse effect (pharmacological/biological) b) use-related events c) quality related effects The absolute numbers of reported adverse events provide a certain indication of the level of attention paid to vigilance activities. The numbers should be put in relation to the number of patients treated during the same period. If no use- related reports have been recorded questions should be asked about identification of medication errors (they do occur everywhere) VL04.01 PE VL04 B-3-10 How many medical products adverse event reports did the health facility submit to the national vigilance centre in the previous year? Provide statistics All reports of suspected adverse events recorded in the facility, that fulfil the completeness criteria, should also be submitted to the national vigilance Centre. Important to establish that no reports are left behind because of possible embarrassment (e.g., medication errors) VL04.01 PE.VL.06 Risk assessment and evaluation B-3-11 Which is the process for assessing validity and causality of individual medical product case safety reports received from within your health facility? Specify all that apply • verifying completeness of case details • consulting national or international literature or databases • use of decision support algorithm • expert committee consensus • no local verification or assessment • Describe classification system used Ascertain that efforts are made to verify the validity and completeness of case reports originating from the health facility. Routines should be in place for regular causality assessment and route-cause analysis if warranted. VL04.02 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 157 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input Information, education and communication (iec) with concerned groups B-3-12 How are you providing feedback to internal individual reporters of medical product related case safety reports? Specify all that apply • acknowledgement (electronic/paper/verbal, automatic or not) • feedback with case assessment • advise on possible prevention no feedback Identify mechanisms available to stimulate, acknowledge and give feed-back to reporters including the result of the local causality assessment made. VL04.02 B-3-13 How are you using the internally collected reports on medical product related safety problems in teaching staff how to contribute to safer patient care? Describe mechanism of collective learning from reports of medical related problems in terms of improved standard operating procedures (SOPs) and routines for safer patient therapy and care Identify mechanisms by which the health facility management or Drug and Therapeutics Committee use the information received through vigilance activities to continuously improve SOPs and routines for medical product handling to improve patient safety VL04.03 B-3-14 Are there mechanisms in place to disseminate vigilance or medical product safety information to members of staff of your health facility? Specify all that apply • newsletter • information bulletin • bulletin board • website • message app • phone line • other Document the different channels by which health facility management is providing up-to-date information related to safe use and vigilance of medical products to its staff members. Identify possible gaps. VL04.07 PE VL 01 B-3-15 How many requests for information about medical product safety were received by the health facility in the previous year? Provide statistics if available on: • Inquiries received • From inside facility • From external parties and community This question refers to the medical product safety information service provided internally to health facility staff and to the external community. This is not function considered as critical for a vigilance Centre, but has implications for the wider understanding of safe use of medicines VL06.01 158 Vigilance field visit for assessing the performance of the vigilance function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-3-16 How many healthcare workers has the facility trained on vigilance and safe use of medical products in the previous year (through in-service training)? Provide statistics and records of staff members trained Providing training in safe use of medical products and vigilance practices is considered an integral part of the function of a vigilance Centre. VL03.03 PE.VL.02 B-3-17 How many public or community education activities relating to medical product safety were carried out by the health facility in the previous year Specify method of education and number of events Assessors should verify if education of the public or community in medical product safety is part of the Terms of Reference of the health facility. This is not a given for all vigilance centres. If, YES, the educational activities should be recorded. VL03.03 PE.VL.02 B-3-18 How many training events/sessions related to medical product safety were conducted in the previous year? Specify • kind of activity • purpose of activity number of activities See II-3-16 above VL03.03 B-3-19 How many and which regular communications on medical product safety issues did the health facility receive from the national or regional vigilance centre in the previous year? Specify all that apply • Newsletters (printed/electronic) • Dear Health Professional letters • Other This question intends to verify the health facility perspective of the outreach activities of the national vigilance centre. Identify how often communications are received from the centre and how they are distributed in the health facility. Document any feed-back on the communications provided to the national centre. VL 06.02 PE.VL.01 Appendix A3.1. Vigilance field visit assessment questionnaire : for assessing the performance of vigilance activities 159 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ID Question Value range Guidance Related GBT or PE indicators WHO assessor input B-3-20 How many vigilance training sessions organized by the national or regional vigilance centre did staff of the health facility attend in the previous year? Specify number of training events and number of staff members attending? This is to document the engagement of the health facility and its staff in training sessions organized by the national or regional vigilance centres. Document local judgement on the quality of the training being offered. The question intends to verify the coherence of the national vigilance system. VL03.03 PE.VL.02 Overall evaluation of the vigilance system at the health facility level The WHO team concludes that the assessed areas are: Please tick one of the checkboxes below ☐ Satisfactory ☐ Unsatisfactory Justification: Please provide text

Annex 4. GxP observed audit for assessing the performance of the regulatory inspection function 162 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Code of conduct .................................................................................................................................................................................................................. 163 Abbreviations .......................................................................................................................................................................................................................... 164 Glossary ................................................................................................................................................................................................................................... 165 1. Introduction .......................................................................................................................................................................................................................167 2. Purpose ................................................................................................................................................................................................................................ 168 3. Scope ................................................................................................................................................................................................................................... 169 4. Objectives and expected outcomes ................................................................................................................................................................. 170 5. Deliverables ...................................................................................................................................................................................................................... 171 6. Overview of observed audit process ................................................................................................................................................................172 6.1 General principles ............................................................................................................................................................................................ 172 6.2 Preparation of the observed audit .......................................................................................................................................................... 172 6.3 Conducting the observed audit ................................................................................................................................................................ 173 6.4 Reporting of the observed audit ...............................................................................................................................................................174 7. Roles and responsibilities ......................................................................................................................................................................................... 176 7.1 Relevant RA/Inspectorate ............................................................................................................................................................................ 176 7.2 WHO Secretariat (headquarters, regional and country office) ............................................................................................ 176 7.3 WHO Team leader ............................................................................................................................................................................................ 176 7.4 WHO Team members (observers) .......................................................................................................................................................... 177 7.5 Inspection team leader ................................................................................................................................................................................... 177 7.6 Inspection team members ........................................................................................................................................................................... 177 7.7 Inspection site(s) ................................................................................................................................................................................................ 177 8. Bibliography ...................................................................................................................................................................................................................... 180 9. Document change history ....................................................................................................................................................................................... 180 Appendix A4.1. Observed audit inspectors’ evaluation form: for assessing the performance of regulatory i nspection activities ............................................................................................................................................................................................................. 182 Contents Code of conduct 163 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual WHO values and relies upon the normative and technical advice provided by leading subject matter experts in the context of its advisory/technical committees, meetings, and other similar processes. Such advice contributes to the formulation of the public health policies and norms that are promulgated by WHO for the benefit of its Member States. In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice: a. fully and honestly disclose all relevant interests and biases on the declaration of interests (DOI) form that may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting, or group members at the outset (unless this is done by the chairperson or WHO Secretariat). b. spontaneously report any material changes to their disclosed interest on an ongoing basis during the period in which the expert serves the Organization. c. respect the confidential nature of committee or meeting deliberations or of the advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO. d. undertake not to engage in activities that may bring reputational harm to the WHO process that they are involved in. e. undertake to represent their views in a personal and individual capacity with the best interest of WHO in mind as opposed to representing the views of their employers, other institutions, or governments. f. actively and fully participate in discussions and deliberations within the relevant advisory group, committee, or meeting. WHO has zero tolerance towards sexual exploitation and abuse, sexual harassment (SEAH) and other types of abusive conduct (that is, discrimination, abuse of authority and harassment). Sexual exploitation, sexual abuse and sexual harassment are considered to be acts of serious misconduct and constitute a basis on which staff members, whether internationally or locally recruited, and contractors can be summarily dismissed. Code of conduct 164 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Abbreviations CAPA corrective actions and preventive actions GBT Global Benchmarking Tool GxP good practices RA regulatory authority RI regulatory inspection WHO World Health Organization WLA WHO-listed authority Glossary 165 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Glossary The definitions given below apply to the terms used in the current document. These terms may have different meanings in other contexts. GxP For the purpose of this document, GxP refers to good manufacturing practices, good storage and distribution practices and good clinical practices. Inspectors’ evaluation form The form/template used for the evaluation of the inspection process and practice, including inspection preparation, conduct and reporting, as well as the competency, skills and attitude of the inspection team. Inspection findings Results of the inspection undertaken documented in a written report. In principle, the findings are compared against established guidelines, including regulations and guidelines. Based on the inspection findings, a conclusion can be made to indicate whether the inspected site conforms to the country’s legislation, regulations and code of practice or does not conform to these. The findings may be positive or negative. Negative inspection findings are usually referred to as deficiencies. Inspection report A report prepared in English or the local language with an English translation by the Regulatory Authority (RA) inspection team, which documents the different inspection activities performed along with the observations, deficiencies and findings of the inspection. The inspection report is usually prepared according to the predefined template/format at the relevant inspectorate. Inspection team The team established by the RA to perform the regulatory inspection as part of the provision of the national legislation and/or regulations enforced in the country relevant to different medical products and health technologies. In principle, an inspection should generally be performed by a team of inspectors; however, it may be conducted by a single inspector as well. For the inspection team, an inspector should ideally be appointed as a team leader. In addition, if the inspectorate procedures provide for it, the inspection team may include inspectors in training, observers or external consultants. Inspection team leader A trained, qualified inspector (according to well-established criteria) appointed or designated as such by the RA/Inspectorate. Inspection workplan A plan usually developed by the inspection team to detail different inspection process. The inspection plan should be prepared and cleared, if necessary, according to the procedures at the relevant inspectorate. Observed audit A process used by WHO to document and evaluate the level of performance of a national GxP regulatory inspection function. Observed audit may complement WHO benchmarking using the Global Benchmarking Tool for capacity-building purposes or the performance evaluation process (PE) for the purpose of designation as a WHO- listed authority (WLA). The activity consists of an observation made by WHO observers of a routine inspection at an authorized site. The regulatory inspection under observation should ideally be a routine inspection and executed according to national references, including regulations and guidelines. National references are expected to be at least equivalent to WHO good practice guidelines (e.g., good manufacturing practices, good distribution practices, and good clinical practices) and/or any other internationally accepted guidelines. Observed audit report A report prepared in English, which is delivered by WHO observers according to the predefined observed audit report template. An observed audit report provides an overview of the observed regulatory inspection along with details of findings and recommendations of the WHO observers on the inspection process and inspectors’ performance. 166 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual WHO observer A competent expert who is familiar with WHO published regulations and guidelines on the specific field of regulatory inspection as relevant to the inspection activities under observation (that is, good manufacturing practices, good distribution practices or good clinical practice). WHO observers should have extensive (more than 7 years) experience and advanced skills in conducting national and international regulatory inspections as regulatory inspectors or WHO auditors. WHO observers are also referred to as the WHO Team. WHO-listed authority (WLA) A national regulatory authority or a regional regulatory system that has been documented to comply with all the indicators and requirements specified by WHO for listing based on an established benchmarking and performance evaluation process. A regulatory authority can be listed for one or more product categories or for one or more regulatory functions. 1. Introduction 167 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 1. Introduction The inspection of establishments across the medical product supply chain is an essential regulatory function. The supply chain includes manufacturers, distributors, re‐packagers, re‐labellers, importers, agents, traders, wholesalers and retailers of medical products. The inspection of clinical research organizations and sponsors is also covered by this regulatory function, whose purpose is to ensure that operations at these establishments are carried out in accordance with approved standards, norms, and guidelines and are in compliance with the national medical products legislation and regulations. These, in turn, should be consistent with World Health Organization (WHO) recommendations and other internationally recognized guidelines. One of the common regulatory functions subject to assessment in the context of benchmarking for capacity-building or WLA designation is Good Practices (GxP) regulatory inspection. This raised the need for comprehensive assessment and evaluation of the performance and functionality of GxP regulatory inspection. In response to this need, WHO – in consultation with Member States, partners and regulatory experts – has developed the process and methodology for a WHO observed audit. Observed audit forms an essential part of benchmarking of regulatory systems for medical products (for good manufacturing practice inspections) and designation as a WHO-listed authority (for good manufacturing practice and good clinical practice inspections). Following the guidance provided in this documentwill ensure the necessary consistency when organizing observed audits, including by defining roles and responsibilities, which will in turn contribute to quality output and proper interaction among interested parties. 168 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2. Purpose The purpose of this document is to: a. provide guidance to WHO staff and observers, the relevant RA and other interested or involved parties on all aspects of the WHO observed audit process and methodology, including procedures and timelines for planning, preparing, conducting, reporting and follow up, and templates for related documentation b. define the roles and responsibilities of the observed audit team(s) and team members c. describe the roles and responsibilities of the three levels of WHO (WHO headquarters, regional offices and country offices) as well as the concerned RA in this process d. establish a level of rigour, consistency and uniformity within the process for expert review of laboratory testing activities and confidence in its outcomes. This document should be read in conjunction with other relevant manuals, guidelines, standard operating procedures (SOPs) and work instructions, as applicable. This document is subject to periodic review and revision as part of the quality system approach applied by WHO. 3. Scope 169 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 3. Scope This document describes the process of initiating, planning, preparing, conducting and reporting on an observed audit of good manufacturing practice and good clinical practice inspections. It identifies the key steps involved in an observed audit to confirm that the performance of the regulatory inspection function complies with applicable WHO and other internationally recognized requirements. This document applies to observed audits relating to both medicines (new chemical entities, multisource/ generic medicines) and biological products (biotherapeutic products, similar biotherapeutics and vaccines). 170 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The objectives and expected outcomes of an observed audit are to: a. assess the performance of regulatory inspectors to prepare, conduct and report on regulatory inspections of good manufacturing practices/ good clinical practices. b. assess the knowledge, competence, skills and attitudes of RA inspectors. c. identify strengths and best practices of the inspection activities performed. d. identify areas in need of further improvement and for which a specific training plan might be needed. e. provide feedback on the relevant GBT sub- indicators of the regulatory function. 4. Objectives and expected outcomes 5. Deliverables 171 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5. Deliverables On completion of the observed audit, the following deliverables shall be provided to the WHO Secretariat: a. Inspection report (prepared in English or in the local language with English translation) to be delivered by the RA inspection team following the predefined template/format of inspection reports. b. Observed audit report (in English) to be delivered by the WHO Team (of WHO observers). 172 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6. Overview of observed audit process The observed audit aims to assess the performance of the GxP regulatory inspection function with an emphasis on inspection activities and inspectors’ competency, skills and attitude. 6.1 General principles Two related activities take place concurrently: the inspection process by the inspection team on behalf of the RA and the observed audit by the observer(s) on behalf of WHO. It should be ensured that neither of the two processes negatively affects the other. This can be achieved through close collaboration between the inspection team and the WHO observer(s). A GxP regulatory compliance programme is not limited to the GxP inspection process. It also includes components such as the supporting infrastructure of legislative and regulatory requirements, GxP standards, inspection/enforcement resources and procedures, performance standards, alert and crisis system, analytical capability, surveillance programme and quality management systems. While the observed audit process focuses on activities conducted during GxP regulatory inspection, along with inspectors’ competency, skills and attitude, other components are covered by systemic assessment as part of benchmarking activities, such as the Global Benchmarking Tool (GBT). During the on-site inspection, it is expected that the inspection team and the observer(s) collaborate to ensure that the above-stated objectives are met. For this purpose, a briefing meeting between the inspection team and the observer(s) should be planned in advance and prior to the conduct of the inspection. The RA and site/firm should discuss and agree in advance on the process of the observed audit, roles of the observer(s) and translator (if any), and the number of observer(s) to be included in the inspection. To facilitate the WHO observed audit in evaluating the inspection process, a copy of the inspection process manual or regulating procedures or SOPs, including the RA procedure for the format and content of inspection reports, should be sent to WHO, preferably two weeks before the observed audit. Similarly, a copy of the inspected institution/entity/ site/facility Information master file, Quality manual or similar file/document should be shared with WHO observers as soon as available, preferably before the commencement of the inspection. It is not the objective of the observed audit to inspect the entities/firms or evaluate the level of implementation and consequently compliance with GxP. Observed audit does not constitute, by any means, a WHO inspection/audit of the site/firm. The site inspected by the RA should refrain from misuse of the WHO observed audit (e.g., for promotional purposes). Unrestricted access: WHO observers should have unlimited access to information, documents, people and assets of the inspected site/firm during the observed audit while respecting all applicable confidentiality arrangements and code of conduct. In terms of unlimited access to people, WHO observers may directly interview the firm’s employees at any hierarchical level while respecting the organization’s culture and habits. Discussion among RA inspectors and WHO observers related to the observed audit, including any major disagreement, should be made or resolved away from the inspected site/firm. 6.2 Preparation of the observed audit 6.2.1 Selection of sites or entities Selection of the site(s) for the observed GxP regulatory inspection should be agreed between the RA and WHO. The RA should provide WHO with the inspectorate routine inspection schedule (including names and addresses of entities, designated inspector(s) and tentative dates, if possible) in order to help in selection of the site(s). In principle, the site(s) should be selected from among those sites scheduled for inspection as per the annual or regular inspection plan. The number of the site(s) selected for observed audit should be done through a risk-based approach. Factors to consider include the criticality of the products or the complexity of activities or processes, number of licensed/authorized firms/sites, capacity, geographical distribution, national/international 6. Overview of observed audit process 173 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual exposure and earlier performance assessment experience of the relevant RA/Inspectorate. The ultimate objective is to have a representative sample of the inspection process at the concerned RA/ Inspectorate. Mock inspection, simulation or inspections scheduled for the sole purpose of observed audit should not be considered. 6.2.2 Briefing session The WHO observers selected from the roster of qualified experts for each individual observed audit should be thoroughly briefed on the principles described in this document prior to the start of the mission. The WHO team leader should brief all team members (that is, observers) remotely as part of preparation for the mission. The briefing should include details of the following: ƒ context of the observed audit, including the objectives and expected outcomes ƒ methodology of the observed audit process ƒ availability of the required documents ƒ access to and utilization of a WHO secure information-sharing platform ƒ roles and responsibilities of different observers, including in specific area(s), if any ƒ other related logistical arrangements (e.g., travel, accommodation). Answers to questions raised and clarifications sought by team members. 6.2.3 Documentation review Each team member, no matter how experienced, will need to spend the necessary time preparing for the observed audit, and reading the background documents. As part of the preparation for the observed audit, WHO observers should review the following documents, to the extent possible and where applicable, well in advance of the observed audit: a. RA or Inspectorate quality manual along with all relevant SOPs, particularly those related to inspection planning, preparation, conduct, reporting, enforcement, and follow up of corrective actions and preventive actions (CAPA) b. a copy of national GxP (good manufacturing practices, good clinical practices or good distribution practices) code/regulations/ guidelines c. previous inspection reports of the same firm selected for the observed audit along with the CAPAs, if any d. a background document about the institution/ entity/site/facility subjected to inspection (e.g., inspection site(s) information, site master file, investigator’s brochure, clinical study protocol (CSP), others as applicable) e. major changes at the inspection site since the last inspection f. list of inspectors designated by the RA to perform the GxP regulatory inspection, including their curricula vitae (CVs), job description, qualification and training overview g. inspection workplan (also called inspection agenda, inspection schedule or inspection programme of work) h. compliance history of the inspection site i. list of recalls, complaints, safety issues, among others, related to the site or products to be inspected j. recent regulatory or enforcement actions related to the site or products to be inspected, if any. To facilitate the preparation process for the observed audit, 10 days before the start of the observed audit at the latest, the relevant RA focal person(s) should upload the above-mentioned documents to the relevant secure WHO information-sharing platform. 6.3 Conducting the observed audit By default, GxP observed audit involves onsite evaluation. In exceptional situations, WHO, in agreement with the RA, may take into consideration to conduct a remote GxP Observed audit, in case of justified conditions (e.g., public health emergencies involving travel restrictions). Remote inspections should follow the applicable procedures developed for coordinating, preparing and conducting GxP inspections, but should also take into consideration the limitations imposed due to the use of a remote process and recognize that such a remote process 174 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual cannot completely replace on-site GxP inspections. If necessary, a face-to-face (physical) mission may be organized by WHO to the RA/Inspectorate to follow up remote inspections and remote observed audits once the reasons that called for the remote approach are resolved. In general, remote observed audit is discouraged. Site-specific issues (e.g., access restrictions due to safety/biosafety reasons) should not be considered as the sole justification for remote observed audits. The GxP regulatory inspection subjected to observed audit should take place in accordance with normal practice, as defined in the procedures of the RA/ Inspectorate and in accordance with the relevant RA quality system. The observer(s) should not take any active part in conducting or performing the inspection. However, if necessary, observers may ask further questions, request additional documents from the representatives of the inspected site or interview one or more of the staff working at the inspected site. The objective of such requests is not to evaluate the level of compliance at the inspected site. Rather, the objective of such requests by the observers is to help in comprehensively understanding the context of the regulatory inspection and evaluating the performance of the RA inspectors. The observer(s) may question the inspection team about findings made or not made by them during the inspection. The purpose of such questions is to evaluate the inspection process or the inspectors’ competency. For the purpose of evaluation of the process and practice of the inspection as well as the competency of the inspection team, the observer(s) should make use of the "Inspectors' evaluation form" attached as Appendix A2.1 of this document. Inspectors’ evaluation form developed for RI observed audit should be considered as an aide memoire for ensuring all critical elements are evaluated. Good knowledge and proper understanding are crucial for effective use of the Inspectors' evaluation form and, consequently, for the quality of the observed audit report, including the respective conclusions and recommendations. The GxP regulatory inspection should always be led and managed by the RA inspectors. At agreed intervals (e.g., end of each working day), the observer(s) should review the inspection process with the inspectors and give feedback on the strengths and gaps in their progress. Observers are not expected to deliver any judgement on single individuals, but rather to provide general feedback on the behaviour and the achievements of the inspection team, through the evaluation of each inspector. Throughout the observed audit, observers should make clear, accurate and legible notes. Such notes should provide relevant yet detailed facts that serve as a record of what is directly observed. The notes should be used for the formulation of the observed audit report. In the unfortunate situation that one or more critical findings, which are or may potentially have a negative public health impact, are overlooked by the inspection team but identified by the WHO team, the WHO team leader should be informed and act by reporting the issue to the proper managerial level at the RA, in close coordination with the WHO Secretariat. Once the inspection is completed, the WHO observer(s) should hold a debriefing meeting with the RA inspection team, involving, as appropriate, other representatives from the RA or the Inspectorate (e.g., top management). The purpose is to brief the attendees about the observed audit activities and present the observed audit findings, including the identified strengths, gaps, areas for improvement and recommendations (if any). This debriefing meeting should not include representatives of the inspected site and preferably not be held at the inspected site. For the purpose of the debriefing meeting with the inspection team and RA/Inspectorate representatives, the observer(s) are encouraged to prepare a presentation indicating the major findings and recommendations of the observed audit. 6.4 Reporting of the observed audit In conjunction with the observed audit, two sets of reports should be issued: an inspection report by the inspection team and an observed audit report by the observer(s). a) Inspection report The inspection team should provide an inspection report (prepared in English or in the local language with English translation) following the predefined template/format at the RA/Inspectorate. 6. Overview of observed audit process 175 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The content of the inspection report is expected to correspond to the latest WHO technical report series that is applicable or other internationally recognized guidelines or recommendations. The final inspection report should be ideally made available to WHO within 14 working days from the last day of the inspection; alternative timelines may be considered according to RA internal procedure, which should reasonably not exceed 1 month from the inspection close out date. b) Observed audit report The observer(s) should prepare an observed audit report (in English or bilingual). The finalized observed audit report should be made available to WHO within 21 working days from the last day of the observed audit or 7 working days from the date of receipt of the RA inspection report. A draft of the same should ideally be delivered by the observers on the last day of activity of the observed audit. The final observed audit report should be subjected to a thorough review by the WHO Secretariat according to the relevant procedures to ensure consistency in and robustness of the output. The final observed audit report should be shared with the respective RA/Inspectorate and uploaded to the relevant site of the WHO secure information-sharing platform for archiving purposes. 176 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 7. Roles and responsibilities Observed audit should be seen as a collaborative exercise to which several parties contribute, including the RA/Inspectorate, WHO Secretariat, inspection team, observers and inspection site(s). This section is meant to provide guidance on the roles and responsibilities of these parties. 7.1 Relevant RA/Inspectorate The RA/Inspectorate subject to observed audit is responsible for: a. discussing and agreeing with WHO on selection of the site(s) that will be subjected to the observed regulatory inspection b. designating the inspection team, including the inspection team leader, for the observed GxP regulatory inspection c. sharing with WHO, through the secure information-sharing platform or any other agreed means, all necessary information and documentation including, among others, national GxP code/regulations/guidelines, annual inspection schedule/plan, data specific to the selected regulatory inspection site (such as inspection site data, inspection team data), which will be subjected to the observed audit. d. nominating officials and granting them access to the WHO secure information-sharing platform e. communicating and coordinating with the inspection site(s), including on all necessary management and logistical arrangements f. confirming the regulatory inspection in writing at least 15 working days before the inspection date, along with the latest details of the inspection information g. providing the necessary clarifications and explanations in response to questions from WHO Team h. seeking and obtaining any necessary consent from any involved stakeholder in order to share the relevant information with WHO. 7.2 WHO Secretariat (headquarters, regional and country office) WHO headquarters (Regulatory Systems Strengthening team), in collaboration with WHO regional offices and relevant country offices, is responsible for: a. establishing and maintaining the tools and databases related to observed audit. b. establishing a roster of qualified experts. c. Training experts in order to ensure consistency and quality of the process as well as robustness of the assessment outcome. d. Discussion and agreement with RA on selection of the site(s) which will be subject to the observed audit. e. Establishing a dedicated country page on the WHO information sharing platform for the observed audit and uploading of all relevant documentation for access and archive purposes. f. Selection of the WHO observers from the roster of qualified experts to perform the observed audit on behalf of WHO. g. Designation the WHO team leader. h. Organization of any necessary contractual arrangements. 7.3 WHO Team leader The WHO Team leader is responsible for: a. Leading and coordinating the WHO observed audit from the beginning to the end of the process. The team leader will lead the team of observers and participate in the observation and evaluation of the inspection process. The team leader will lead, advise and guide the WHO Team. b. Briefing WHO observers on various aspects related to the observed audit, including context, background, objectives, process and methodology, as well as any safety issues relevant to the observed audit such as vaccination, if applicable. c. Coordinating work among all members of the WHO team in order to ensure smooth and harmonized execution of the WHO observed audit with avoidance of work duplication and/or conflicts. d. Communicating with RA officials on behalf of WHO. e. Delivering presentations: presentations made during the WHO observed audit opening and closing meetings will ideally be done by the WHO team leader. Nevertheless, different WHO team members will also help in preparing for these presentations and providing input, as necessary. Similarly, the WHO team leader may invite any of 7. Roles and responsibilities 177 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual the WHO observers to present the findings, provide clarifications, answer questions of the RA if needed. f. Delivering the WHO observed audit report: the overall report of the WHO observed audit shall ideally be prepared by all WHO team members; however, the responsibility of delivering the finally agreed report lies with the WHO team leader. 7.4 WHO Team members (observers) The observers are responsible for: a. Reviewing and signing the relevant administrative documents, including the invitation letter, confidentiality agreement and declaration of interests. b. Making necessary travel arrangements (e.g., booking flights and obtaining visas), as described in the invitation letter. c. Complying with the immunization requirements, if any, and bringing with them a copy of their immunization certificates. d. Respecting all applicable protocols and codes of ethics and conduct. e. Observing and evaluating the inspection process and inspectors’ performance, including planning, meetings, interviews, reviewed documents, inspection methodology, inspector’s competence and skill, as well as the leadership skills and duties of a team leader. Observers should use the "Inspectors' evaluation form" attached as Appendix A4.1 of this document to assess the performance of the inspection process. f. Identifying the strengths as well as gaps and areas for improvement, if any. The identified strengths and areas for improvement should be organized by the observers in a presentation. Preparing a detailed report on the observed audit conducted, including general information of the observed audit, activities, findings (strengths, gaps and areas for improvement) and recommendations, if applicable, to address the identified gaps. The observed audit report should be submitted to WHO by the observer within a maximum of 21 working days from the last day of the observed audit. If possible, a draft of the same should be delivered by the observers on the last day of the observed audit activity. The report may quote the different components/sections in the evaluation form. 7.5 Inspection team leader The RA inspection team leader is responsible for: a. Establishing and maintaining communication between the inspection team, the firm to be inspected and the WHO team or its representative in charge of the WHO observed audit. b. Planning and preparing the inspection workplan/ agenda/programme/schedule. c. Planning, coordinating and preparing the on-site visit and informing the inspection and WHO team accordingly. d. Briefing the inspection and WHO teams and arranging the distribution of duties of the inspection team as per the agreed programme. e. Leading the inspection team and coordinating with the WHO observers. Coordinating and finalizing the inspection report to be provided to the RA as well as WHO. 7.6 Inspection team members The inspection team members are responsible for: a. Assisting and collaborating with the WHO team. b. Ensuring the necessary flow and sharing of information between parties. c. Providing support on translation process if necessary and applicable. Respecting all applicable protocols and codes of ethics and conduct. 7.7 Inspection site(s) The inspection site(s) is responsible for: a. Communicating and coordinating all the necessary management and logistical arrangements with the RA. b. Providing documents or information requested by inspection team as well as WHO observers in a timely manner. c. Making available any information and/or documentation related to the site or product(s) (e.g., site master file, investigator’s brochure, others as applicable) required for the preparation and conduct of the inspection. d. Ensuring that relevant staff involved in the main activities or related activities are present and available during the inspection for interviews or clarification of issues identified. The following table provides an overview of the roles and responsibilities of the different parties involved in the observed audit, including shared responsibilities. 178 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Activity RA/ Inspectorate WHO Secretariat 1 WHO team leader WHO observers Inspection team leader Inspection team members Inspection site R = responsible; C = contributor; I = informed; NA = not applicable Draft the terms of reference of the observed audit, including objectives, scope, activities and timelines C R C I I I I Select sites for the observed audit C R C I I I I Designate the WHO team leader I R NA I I I I Designate WHO observers (team members) I R C NA I I I Establish a dedicated site under the relevant WHO information-sharing platform I R I I I I I Share information related to the inspection site (including uploading to the WHO information-sharing platform) R I I I I NA NA Designate the inspection team, including the team leader R I I I NA NA NA Nominate officials to access the WHO information- sharing platform R I I NA I I NA Inform the inspection site and coordinate the observed audit R I I NA NA NA NA Organize any necessary contractual arrangements for the WHO team - R I NA NA NA NA Organize any necessary overseas logistical arrangements for the WHO team I R NA NA NA NA NA Organize any necessary domestic logistical arrangements for the WHO team as well as the inspection team R C I I I I I Organize any necessary on-site or off-site translation services R R I I I I I 1 Also called WHO responsible officer 7. Roles and responsibilities 179 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Activity RA/ Inspectorate WHO Secretariat 1 WHO team leader WHO observers Inspection team leader Inspection team members Inspection site R = responsible; C = contributor; I = informed; NA = not applicable Brief WHO observers on various aspects related to the observed audit NA C R NA NA NA NA Lead the observed audit and coordinate with the RA/ Inspectorate on-site C NA R C I I I Deliver presentations during the opening and closing meetings of the observed audit, apart from representatives of the inspected site I NA R C I I NA Conduct the GxP observed audit I NA R R I I I Lead the inspection process, including inspection opening and closing meetings NA I I I R C I Conduct the GxP inspection NA I I I R R I Prepare the inspection report NA I I I R R I Deliver the inspection report, including uploading it to the WHO information-sharing platform R I I I R C I Prepare the observed audit report NA NA R R I I NA Deliver the observed audit report, including uploading it to the WHO information- sharing platform I NA R C I I NA Respect all applicable protocols, codes of conduct and ethics R R R R R R R 180 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8. Bibliography Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans. Geneva: World Health Organization, in press. Regulatory Inspections (RI): Indicators and factsheets. In: WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021. (https://apps.who.int/iris/ handle/10665/341243, accessed 6 April 2023). WHO-listed authority (WLA). Geneva: World Health Organization (https://www.who.int/initiatives/who- listed-authority-reg-authorities, accessed 9 August 2023). 9. Document change history 181 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Version no. Date of issue Main changes 6th 6 October 2015 NA 7th 31 March 2022 Revision of the document outline and expansion of the content in light of discussions related to performance evaluation process (process) of WLA 8th 30 June 2023 Further revision including editorial changes of the document considering the experience gained during the pilot implementation in Q3 and Q4/2022. 9. Document change history 182 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A4.1. Observed audit inspectors’ evaluation form: for assessing the performance of regulatory inspection activities About this evaluation form ƒ This form is intended to assess the performance of the medical products regulatory inspection (RI) function through an observed audit of regulatory inspections; it is not meant to evaluate the activities carried out by the inspected site. ƒ The form comprises five independent sections, aimed at assessing: a. Inspection preparation b. Inspection conduct c. Inspection reporting d. Inspectors’ technical competency e. Inspectors’ attitude and skills ƒ The form comprises a mix of “open-ended” and “closed-ended” questions. ƒ The WHO Team should complete the respective fields (WHO observers’ comments and conclusion) and attach a copy of the completed form (one for each observed audit) to the RI expert review report. Rating (for WLA purposes only) Within the context of the WLA framework, WHO uses the Observed Audit of GxP-related activities to determine whether or not the RA can be considered to acceptably meet WLA requirements. In order for an authority to be granted WLA status for RI, the relevant GxP inspectorate must fulfil the following criteria: ƒ Achieve a satisfactory score in each section, for each observed inspection. Appendix A4.1. Observed audit inspectors’ evaluation form: for assessing the performance of regulatory inspection activities 183 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Observed audit inspectors' evaluation form Country:__________________ Institution:__________________ Dates: __________________ RA Inspectors: __________________ WHO Observers/Experts:__________________ Observed audit evaluation criteria WHO observer input 1 Inspection preparation Relevant procedures should be applied across the inspection preparation process. Observers should assess and evaluate the adherence of the inspection team to the procedures. Please note that evaluation of the procedures themselves should be subject to review and assessment during the benchmarking process using WHO GBT. A risk-based approach should be considered while preparing for the inspection. 1.1 Inspection team is well formulated according to the relevant procedure(s) Guidance: National Control Laboratory (NCL) staff, product reviewers/assessors, product specialists, among others, may join the inspection team, if necessary. 1.2 Inspection team leader is assigned according to the criteria set in the relevant RA procedure Guidance: the assignment process should be based on a well-defined criterion (e.g., number of years of experience, level of administration/hierarchy, nomination by the RA/inspectorate, etc.) 1.3 Roles and tasks are well distributed among inspection team members Guidance: assigning specific roles and responsibilities should ideally consider qualifications, experience and skills important for the scope of the inspection. Also, as part of inspection preparation, the team leader can consider the possibility of splitting the team or working together, as part of risk-based decision and time management. 1.4. Inspected site(s) info (e.g., site master file, investigator’s brochure, protocol source documents, others as applicable) is reviewed Guidance: the format and content of these documents should match relevant WHO guidelines or relevant international guidelines. 1.5 Inspected product(s) information is reviewed (for example, monograph, marketing authorization file, investigational medicinal product dossier, clinical study protocol, recent related national control laboratory and quality control data, others as applicable). Note: particularly for good manufacturing practices inspection, recent related national control laboratory and quality control data should be reviewed. 1.6 Last relevant inspection report is reviewed along with the corrective and preventive actions (CAPA) undertaken, where applicable. Guidance: corrective and preventive actions here may refer to the detailed CAPA plan, CAPA evidence or CAPA evaluation/outcome according to national procedures and practices. 1.7 Recent related regulatory actions are reviewed Guidance: recalls, suspension of marketing authorization, suspension of clinical trial study, revocation, etc. 1.8 Recent related complaints, adverse drug reactions, and other vigilance outcomes are reviewed. Guidance: in order to properly meet this item, it is expected that other departments and units within the RA (for example pharmacovigilance centre, market control department, national control laboratory) provide some input as part of inspection preparation. 184 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Observed audit evaluation criteria WHO observer input 1.9 An inspection workplan is developed by the inspection team Guidance: the term workplan refers to the inspection programme, agenda or schedule of each single inspection activity and is not meant to refer to the overall inspection programme/schedule, which usually extends to several months. 1.10 Inspection workplan considered a risk-based approach 1.11 Inspection workplan is available to each inspection team member Guidance: the workplan should be shared among team members well in advance of the inspection either in paper or electronic format as part of inspection preparation. WHO observers’ conclusion of the overall inspection preparation Guidance: WHO observers should use the above-listed items to qualitatively evaluate the overall inspection preparation process. As preparation for the inspection, (i) the inspection team should be properly organized with clear roles and responsibilities of the team leader and members, (ii) the inspection team should have access to essential background documents and information, with input from other units within the RA. A risk-based inspection workplan should consequently be developed and shared among the inspection team. The WHO observers conclude that the overall inspection preparation is: (please tick one of the below checkboxes) ☐ Satisfactory ☐ Unsatisfactory Justification: (please provide text) 2 Inspection conduct Relevant procedures should be applied across the inspection conduct process. Observers should assess and evaluate the adherence of the inspection team to these procedures. Please note that evaluation of the procedures themselves should be subject to review and assessment during the benchmarking process using the WHO GBT. A risk-based approach should be considered while conducting the inspection. 2.1. Inspection opening meeting was conducted 2.2. The inspection team introduced themselves to the inspectee Guidance: introductions can be made by each of inspection team members or the team leader on behalf of the team. 2.3. Inspection team leader presented the scope, objectives and expected outcomes to the inspectee Guidance: this presentation can be verbal or using any sort of formal communication (e.g., inspection notification, paper or Micosoft Powerpoint) or, in particular situations, by a mixed methodology. 2.4. Inspection workplan was made available to the inspectee 2.5 The inspection plan/agenda was adjusted, where warranted, based on the findings of the inspection (a risk-based approach is applied) 2.6 Last inspection findings and related Corrective action/preventive action CAPAs, where applicable, were checked considering a risk-based approach Guidance: in terms of CAPA review, the on-site check should ensure that appropriate corrective actions include both short-term actions to address the immediate problem and long-term actions to prevent the recurrence of the problem, particularly for critical and major findings. Any CAPA pending from the last inspection should also be prioritized for on-site check. 2.7 Inspection facilitation aids (e.g., checklists, aide memoires) were used, if necessary Guidance: use of inspection facilitation aids is optional and may not always be justified or required. In all cases, it is essential to review the RA/Inspectorate policy on this aspect and check if the reviewed policy is followed in a consistent manner. 2.8 Inspection team focused on primary objectives and risk areas identified Appendix A4.1. Observed audit inspectors’ evaluation form: for assessing the performance of regulatory inspection activities 185 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Observed audit evaluation criteria WHO observer input 2.9 Updated guidelines, currently applicable to the inspection scope, were followed during the inspection Guidance: the guidelines followed, as applicable to the inspection scope, are the current ones in use, and not those that are obsolete or in transition/phasing in stages. 2.10 Actual operations were witnessed by the inspection team Guidance: as part of the inspection methodology, normally it is expected to observe activities or operations to confirm compliance with MAA, procedures and guidelines. Strong justification should be provided in case actual operations are not witnessed (in case of good clinical practice inspections at a clinical research organization or Sponsor, operations are not expected to be carried out and this criterion can be scored as NA). 2.11 Essential documentation was reviewed by the inspection team: Guidance: examples of essential documentation include product quality reviews, standard operating procedures, trend analysis, raw data, deviation reports, out-of- specification reports, and others. 2.12 Critical stages and parameters of the inspected processes were covered during t he inspection Guidance: during the inspection preparation phase, the critical stages and parameters should be identified and addressed accordingly during the inspection process. 2.13 Essential calibrations, qualifications and validations were assessed (not applicable in case of good clinical practice inspections of contract research organization/sponsor) 2.14 Critical changes since the last inspection were reviewed, where applicable 2.15 Systemic findings were identified, if any, out of isolated ones, was made 2.16 Daily feedback/debrief to the inspectee was done Guidance: the feedback/debrief may be done during or by the end of the working day or the morning of the next day. The debrief should ideally cover the main findings and pending issues. 2.17 Inspection closing/exit meeting is conducted WHO observers’ conclusion of the overall conduct of the inspection Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall inspection conduct process. A properly conducted inspection process should be considered as far as administrative procedure (e.g., opening, daily briefing and closing meeting) are respected and properly followed, the inspection team properly utilized a risk- based approach throughout the inspection conduct phase, including documentation review, observation of actual operations and interview of staff. The WHO observers conclude that the overall inspection conduct is: (please tick one of the below checkboxes) ☐ Satisfactory ☐ Unsatisfactory Justification: (please provide text) 3 Inspection reporting Relevant procedures should be applied across the inspection reporting process. Observers should assess and evaluate the adherence of the inspection team to the procedures. Please note that evaluation of the procedures themselves should be subject to review and assessment during the benchmarking process using the WHO GBT. Deficiencies should be factual, evidence- and risk-based, and supported by GxP requirements. 3.1. Deficiencies are well described and detailed Guidance: deficiencies should be written such that they provide a comprehensive understanding of the context and the exact deviation from GxP requirement. Formulation of the deficiencies and how they are written and grouped can also be a factor affecting the classification of the deficiency. 186 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Observed audit evaluation criteria WHO observer input 3.2. Deficiencies are well grouped and categorized Guidance: grouping and categorization of the deficiencies (may be also called findings, observations or non-conformities) should be well justified and consider the context of the deficiencies based on GxP chapters. Formulation of the deficiencies and how they are written and grouped can also be a factor affecting the classification of the deficiency. Categorization should not be confused with classification. The latter is addressed under item 3.3. 3.3. Deficiencies are well classified/ranked according to agreed definitions (e.g., critical, major and other/minor), according to internal procedures Guidance: ideally, deficiencies should be classified in order to help prioritization of the actions to address them. It is expected that the RA/Inspectorate will have a policy or a system of classification of the deficiencies and the same is respected by the inspection team. 3.4. Deficiencies and observations are supported with evidence 3.5 Observations are referenced to regulations and guidelines 3.6 The inspection report is finalized within the agreed time-frame 3.7 The inspection report adheres to the content and format described in the relevant standard operating procedures 3.8 Conclusion and overall compliance rating is in line with the inspection observations (in terms of the number and classification of deficiencies) 3.9 Inspectors’ recommendations to the RA in response to the inspection objective, if any, is consistent with the level of detected risks Guidance: recommendation is meant for the advice by inspectors with respect to the objective of the inspection (e.g., pre-approval inspection, licensing inspection, etc.). WHO observers’ conclusion of the overall reporting of the inspection Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall inspection reporting process. An inspection reporting process should be considered proper if the administrative procedures (e.g., format, timelines) are respected and properly followed, inspection deficiencies are well-described, classified and grouped, as applicable, and reference made to the relevant GxP guidelines. The WHO observers conclude that the overall inspection reporting is: (please tick one of the below checkboxes) ☐ Satisfactory ☐ Unsatisfactory Justification: (please provide text) 4 Inspectors’ technical competency 4.1. Inspection team members have the required background and education to carry out the assigned inspection, in accordance with the inspectorate’s procedures 4.2. Inspectors’ designation (lead, senior, junior) and qualification (initial training and observation) corresponds to the scope of the inspection Guidance: the concept of qualification entails all the minimum prerequisite competencies considered necessary by the inspectorate to qualify inspectors for a specified activity. It includes: (a) initial training (theoretical and practical) and (b) required experience (that is, junior vs senior; or qualification for specific product categories). 4.3. Inspection team members are aware of, knowledgeable in and actually utilizing relevant regulations and guidelines (Good manufacturing practices, good storage and distribution practices and good clinical practices) throughout the inspection process Appendix A4.1. Observed audit inspectors’ evaluation form: for assessing the performance of regulatory inspection activities 187 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Observed audit evaluation criteria WHO observer input 4.4. The depth of the inspection is appropriate and applies a risk-based approach Guidance: inspection depth is considered an expression of the competency of inspectors 4.5 Inspection team members can engage in scientific discussions and provide rational reasons 4.6 Inspection team members’ performance in the field of inspection is satisfactory Guidance: performance is meant to assess the overall inspection process, including critical thinking, root cause analysis, capability of taking decisions and immediate regulatory actions, when necessary, appropriateness ( justification) of such decisions. Inspectors should be skilled in making professional judgements based on facts and science. WHO observers’ conclusion of the overall inspectors’ technical competency Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall technical competency of the inspection team. A technically competent team can be described as one having a team leader and team members with proper qualifications (background, education, training and experience), and knowledge (GxP guidelines and requirements), as evidenced by their proper performance throughout the inspection process. The WHO observers conclude that the overall inspectors’ technical competency is: (please tick one of the below checkboxes) ☐ Satisfactory ☐ Unsatisfactory Justification: (please provide text) 5 Inspectors’ attitude and skills 5.1. Inspection team members follow the code of ethics and conduct Guidance: it should be noted that ethics, values and code of conduct may not be limited to the inspectors. Rather, a code may be applicable to all RA staff or even all civil servants. 5.2. Inspection team members communicate effectively among themselves during the whole inspection process 5.3. Inspection team members communicate effectively with the inspectee during the whole inspection process Guidance: team members are expected to lead the investigation, by maintaining control and pace of the discussion, and providing continuous and effective feedback to inspectee. 5.4. Inspection team members have good and proper questioning skills Guidance: in this context, “questioning skills” are intended more to evaluate investigational skills such as critical thinking, root cause analysis, for example, other than communication skills in general. 5.5 Inspection team members have good and proper listening skills Guidance: in this context, “listening skills” should be measured with respect to the capability of identifying issues, understanding the context and collecting evidence, and arguing or questioning to obtain clarification and arrive at conclusions. 5.6 Inspection team members take notes during the inspection process Guidance: it is important to take good notes during all communication, interaction or observation; these notes may or may not be part of the inspection documentation process. However, such notes will contribute to the final inspection report. 188 GxP observed audit for assessing the performance of the regulatory inspection function Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Observed audit evaluation criteria WHO observer input 5.7 Inspection team members manage their time well Guidance: time management is an important part of the inspection process, as the process of organizing and planning how to divide time between specific activities allows the fulfilment of the objectives and scope. 5.8 A good environment was observed among the inspection team and the inspectee during the inspection process 5.9 The team leader manages various aspects of the team well (e.g., conflicting opinions, redistribution of workload, coordination of the team assignments) 5.10 The team leader manages time effectively and respects the inspection workplan Guidance: please note that the workplan may be modified in order to address situations seen/found during the course of the inspection process, as applicable or necessary. 5.11 The team leader has sufficient ability to make final decisions Guidance: this is meant to assess the team leader’s skills and capabilities to resolve issues among the inspection team in case of any dissenting opinions. WHO observers’ conclusion of the overall inspectors’ attitude Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall skills and attitude of the inspection team. The team skills and attitude can be concluded as satisfactory if the team leader as well as team members are found to respect procedural arrangements, professionally conduct the inspection process in a positive environment among the team and with the inspectee, and the team shows advanced communication and time management skills. The WHO observers conclude that the overall inspectors’ attitude and skills are: (please tick one of the below checkboxes) ☐ Satisfactory ☐ Unsatisfactory Justification: (please provide text) Overall conclusion WHO observers’ overall conclusion of the inspection performance Guidance: the overall conclusion should be based on the evaluation of each of the individual five afore-mentioned indicators. If one of these indicators is found to be unsatisfactory, the overall conclusion should be consequently scored as unsatisfactory Based on the collective evidence and findings of this observed audit, the WHO observers conclude that the performance of the GxP regulatory inspection, including inspection preparation, conduct and reporting as well as inspectors’ technical competence, skills and attitude is: ☐ Satisfactory ☐ Not satisfactory Justification: (please provide text) Expert review of laboratory testing activities Annex 5.. 190 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Contents Code of conduct .................................................................................................................................................................................................................... 191 Abbreviations .......................................................................................................................................................................................................................... 192 Glossary ................................................................................................................................................................................................................................... 193 1. Introduction ...................................................................................................................................................................................................................... 194 2. Purpose ................................................................................................................................................................................................................................ 195 3. Scope ................................................................................................................................................................................................................................... 196 4. Objectives and expected outcomes ..................................................................................................................................................................197 5. Deliverables ..................................................................................................................................................................................................................... 198 6. Overview of the expert review process .......................................................................................................................................................... 199 6.1 General principles ............................................................................................................................................................................................ 199 6.2 Preparing for an expert review ................................................................................................................................................................ 199 6.3 Expert review conduct ...................................................................................................................................................................................200 7. Roles and Responsibilities ...................................................................................................................................................................................... 202 7.1 Relevant RA .......................................................................................................................................................................................................... 202 7.2 Relevant laboratory .......................................................................................................................................................................................... 202 7.3 WHO Secretariat (WHO headquarters, regional offices and country offices) .......................................................... 202 7.4 WHO team leader ........................................................................................................................................................................................... 202 7.5 WHO team member ......................................................................................................................................................................................203 7.6 RA/national control laboratory participants ....................................................................................................................................203 8. Bibliography ..................................................................................................................................................................................................................... 204 9. Document Change History...................................................................................................................................................................................... 205 Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities ........ 206 Code of conduct 191 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Code of conduct WHO values and relies upon the normative and technical advice provided by leading subject matter experts in the context of its advisory/technical committees, meetings, and other similar processes. Such advice contributes to the formulation of the public health policies and norms that are promulgated by WHO for the benefit of its Member States. a. In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice: b. fully and honestly disclose all relevant interests and biases on the declaration of interests (DOI) Form that may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting, or group members at the outset (unless this is done by the chairperson or WHO Secretariat). c. spontaneously report any material changes to their disclosed interest on an ongoing basis during the period in which the expert serves the Organization. d. respect the confidential nature of committee or meeting deliberations or of the advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO. e. undertake not to engage in activities that may bring reputational harm to the WHO process that they are involved in. f. undertake to represent their views in a personal and individual capacity with the best interest of WHO in mind as opposed to representing the views of their employers, other institutions, or governments. g. actively and fully participate in discussions and deliberations within the relevant advisory group, committee, or meeting. WHO has zero tolerance towards sexual exploitation and abuse, sexual harassment (SEAH) and other types of abusive conduct (that is, discrimination, abuse of authority and harassment). Sexual exploitation, sexual abuse and sexual harassment are considered to be acts of serious misconduct and constitute a basis on which staff members, whether internationally or locally recruited, and contractors can be summarily dismissed. 192 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Abbreviations ALCOA attributable, legible, contemporaneous, original and accurate GBT Global Benchmarking Tool ISO international organization for standardization LT laboratory testing NCL national control laboratory PE performance evaluation QMS quality management system RA regulatory authority SOP standard operating procedures TORs terms of reference WHO World Health Organization WLA WHO-listed authorities Glossary 193 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Glossary The definitions given below apply to the terms used in the current document. These terms may have different meanings in other contexts.  Expert The “evaluator” selected by WHO to perform the expert review. Expert review A process used by WHO to document and evaluate the performance of the laboratory testing function in a medical products regulatory system, for the purpose of WHO- listed authorities (WLA) designation. The activity consists in the observation, made by a WHO team of experts, of routine analysis conducted in the National Control Laboratory and/or outsourced laboratory(ies), as applicable. This expert review is a combination of evaluation activities conducted onsite, which can also include the assessment of the performance evaluation (PE) indicator for laboratory testing function. Expert review agenda A plan developed by the WHO Team leader, in agreement with other WHO Team members and WHO Secretariat, to detail different activities, timings, and assignments to be performed during the onsite expert review. LT expert review report A report prepared in English language which is delivered by WHO team following the completion of the expert review. Expert review report provides an overview of the lab activities, findings, and recommendations, if any. RA participants One or more experts, ideally familiar with the national medical products lab activities, who is/are nominated by the RA to represent it and to participate in the expert review. Performance Evaluation (PE) indicators indicator developed to assess and evaluate the performance of laboratory testing function at the target country. Guidance for PE indicators is available in the form of fact sheets. Performance Evaluation (PE) tools for laboratory testing assessment a set of three questionnaires (Tool A, Tool B and Tool C) used for the evaluation of the performance and practice of laboratory testing function at the target country. Team leader A competent expert in the area of medical products analytical testing with team management skills. Team leader is designated by WHO Secretariat and may or may not be a WHO staff. WHO expert A competent expert, who is familiar with WHO published regulations and guidelines in the area of medical products analytical testing, as relevant to the scope of laboratory testing expert review. WHO experts should have extensive (more than 7 years) experience and advanced skills in performing laboratory activities. WHO Secretariat The WHO unit in charge of organization of the visit to the laboratory. WHO Team The team established by the WHO Secretariat as indicated in the respective terms of reference (TORs) to perform the expert review for laboratory testing activities. The WHO team usually comprises two to three experts including a designated team leader. The WHO team may be accompanied by observers when needed. WHO-listed authority (WLA) A national regulatory authority or a regional regulatory system that has been documented to comply with all the indicators and requirements specified by WHO for listing based on an established benchmarking and performance evaluation process. A regulatory authority can be listed for one or more product categories or for one or more regulatory functions.  194 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 1. Introduction The laboratory testing regulatory function is intended to ensure that the Regulatory Authority (RA) is able to assess the quality of medical products by performing analytical quality tests on them, in a variety of circumstances, such as: a) confirmation of a manufacturer’s test results as a part of the evaluation for marketing authorization or for a variation to a marketing authorization b) lot release for certain products depending upon national regulations c) testing of products about which there has been a complaint or a report, or under investigation due to an adverse event d) checking and confirming the quality of medical products placed on the market and detecting substandard and falsified medical products, as part of the market surveillance function. In order to perform product testing, the RA must have access to suitable laboratories where these quality tests can be performed. Laboratory testing is one of the common regulatory functions subject to assessment in the context of benchmarking for capacity-building or WLA designation. This created a need for comprehensive assessment and evaluation of the performance and functionality of laboratory testing. In response to this need, WHO – in consultation with Member States, partners and regulatory experts – developed the process and methodology for an expert review of laboratory testing activities. Adherence to the guidance provided in this document will ensure the consistency required when organizing expert reviews of laboratory testing activities, including in the defined roles and responsibilities, which will in turn contribute to quality output and proper interaction among interested parties.  2. Purpose 195 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2. Purpose The purpose of this document is to: a. provide guidance to WHO, the relevant RA and other interested parties on all aspects of the WHO expert review process and methodology, including the relevant procedures and timelines for planning, preparing, conducting, reporting, and follow up, with templates for related documentation. b. define the roles and responsibilities of the WHO team assigned to perform expert review of laboratory testing. c. describe the roles and responsibilities of the three levels of WHO (WHO headquarters, regional offices and country offices), as well as the relevant RA and laboratory, in this process. d. establish a level of rigour, consistency, and uniformity within the process for expert review of laboratory testing activities and confidence in its outcomes. This document should be read in conjunction with other relevant manuals, guidelines, standard operating procedures (SOPs), and work instructions. This document is subject to periodic review and revision as part of the quality system approach applied by WHO. 196 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 3. Scope This document describes the process to initiate, plan, prepare, conduct, report upon and follow-up on expert review of laboratory testing activities. It identifies the critical and key steps involved in an expert review to confirm that the performance of the laboratory testing function complies with applicable WHO and other internationally recognized requirements. This document applies equally to laboratory testing pertinent to medicines and biological products, including biotherapeutic products, and vaccines. However, some specificities for product-specific activities may be noted within the questionnaire for laboratory testing performance assessment and PE indicators. All product specific requirements are marked as such in the respective documentation. 4. Objectives and expected outcomes 197 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 4. Objectives and expected outcomes The objectives and expected outcomes of the expert review of laboratory testing activities are: a. assessment of the performance of laboratory testing activities and operations, conducted at the national control laboratory and/or external laboratory, as applicable. b. assessment of knowledge, competence and experience of the officials and staff involved in laboratory testing related activities at the lab. c. identification of strengths and best practices of the laboratory testing activities performed at the lab. d. feedback to the relevant GBT sub-indicators or WLA performance evaluation (PE) indicators for the laboratory testing function. 198 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5. Deliverables After the expert review of laboratory testing activities has been completed, the following deliverables should be provided to WHO Secretariat: a. Compiled questionnaire containing scoring and experts’ input, following the template attached as Appendix A5.1 to this document. b. Report of expert review to be delivered by WHO Team. c. Updated onsite assessment and evaluation of the PE indicator for laboratory testing, if necessary, following the relevant template (included in the PE indicators scorecard (Annex 1) as part of the laboratory testing PE process). 6. Overview of the expert review process 199 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6. Overview of the expert review process The expert review aims to assess the performance of the laboratory testing function with an emphasis on laboratory structure and management, along with laboratory testing activities including quality management system, competency assessment, quality of analytical reports in terms of scientific rigour, compliance, and data integrity, among others. 6.1 General principles Other areas and components also contribute to well- functioning laboratory testing, such as legislation and regulatory requirements, infrastructure and resources. It is worth mentioning that expert review of laboratory testing activities focuses on some, but not all of the aspects relating to the laboratory testing function. The expert review is complemented by other tools and methodologies (for example GBT, PE indicators) in the assessment of the overall laboratory testing function. It is essential that these tools and methodologies are considered together and not in standalone mode (that is, with consideration of how GBT assessment contributes to and interacts with laboratory testing expert review and the PE indicators). At the end of the assessment process, careful consideration of should be given to the totality of evidence. In practical terms, this means the WHO team performing the expert review of laboratory testing activities should be well briefed and aware of the outcomes of any earlier assessments. The laboratory testing expert review process is concerned with the actual activities and operations of the national control laboratory and/or external laboratory performing the function. This contrasts with the GBT, which is concerned with systematic aspects of the laboratory testing function, and the PE indicators for laboratory testing which are concerned with quantitative and qualitative PE of the laboratory testing function. WHO, the RA, and the laboratory subject to the expert review should discuss in advance and agree on all details and aspects relating to the review, including the participants, the observers and interpretation (if any). To facilitate the evaluation of the laboratory testing function through the WHO expert review of laboratory testing, the RA should share a copy of the laboratory testing-related procedures or standard operating procedures with WHO, preferably two weeks before the review. The WHO team should be given unlimited access to information, people, and assets relevant to the expert review of laboratory testing, while respecting all applicable confidentiality arrangements and the code of conduct. In terms of unlimited access to people, the WHO team should have the right to interview employees without formally respecting hierarchical lines. However, the WHO team should always show respect for the culture and habits of the relevant organization. 6.2 Preparing for an expert review Laboratory selection If a regulatory system has access to more than one laboratory (for example, laboratories at regional level within the country or outsourcing of quality testing to external laboratories – either inside or outside the country – to perform the required tests on behalf of the RA, in replacement or in addition to the national control laboratory), an agreement should be sought between WHO Secretariat and the RA as to which laboratories should be visited and assessed. When needed, the RA should provide WHO Secretariat with a comprehensive list of laboratories (including name and address) and activities (tests, methods, products) in order to help in selection of the laboratories and tests which will be involved in the laboratory testing expert review. Factors to consider in the selection of laboratories and tests include the size of the laboratory, volume of testing, complexity of activities or processes, and criticality of products. The ultimate objective is to achieve a thorough understanding of the laboratory testing activities and operations. In all cases, the selection of which laboratories will be subject to the expert review of laboratory testing should be made using a risk-based approach. Under no circumstances should simulations or testing activities scheduled for the sole purpose of the expert review be considered. 6.2.1 Briefing session WHO team members, selected from the roster of qualified experts for each individual expert review, should be thoroughly briefed on the principles described in this document before starting the expert review. 200 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The WHO Secretariat or WHO team leader should brief all team members remotely as part of the preparations for the review. The briefing should include details related to: a. context of the expert review, including objectives and expected outcomes b. methodology of the review c. availability of required documents d. access and utilization of the WHO secure information-sharing platform e. roles and responsibilities of different team members, including specific task(s) f. other related logistical arrangements (such as travel, accommodation), and g. answers to questions raised and clarifications sought by team members. 6.2.2 Documentation review As part of the preparations for the expert review, the WHO team should review the following documents – ideally well in advance of conducting the activity. No matter how experienced, each member of the WHO team member will need to spend time preparing for the expert review by reading background documents. To facilitate the preparation for the expert review, the relevant RA coordinator(s) shall upload to the secure WHO information-sharing platform, at least 10 days before the start of the mission, the documents below: ƒ Quality manual, along with all standard operating procedures (SOPs), particularly those related to medical products laboratory testing function ƒ applicable national regulations/guidelines ƒ background documents about the institution/ entity/laboratory that is subject to the expert review ƒ self-assessment performed by the relevant laboratory using the applicable parts of the questionnaire (see Appendix A5.1). 6.3 Expert review conduct By default, an expert review of laboratory testing involves an onsite evaluation. In exceptional situations, WHO may, in agreement with the RA and the laboratory concerned, consider conducting an expert review of laboratory testing remotely, if circumstances require it (for example, public health emergencies involving travel restrictions). If necessary, WHO may subsequently organize a face- to-face (physical) mission to the RA to follow up on the remote expert review, once the reasons that necessitated the remote approach have resolved. In general, remote expert review is discouraged. The laboratory testing activities and operations subject to the expert review should take place in accordance with routine practice, as defined in the procedures of the RA and in accordance with the relevant RA Quality Management System (QMS). The WHO team may ask questions, request documents from the representatives of the visited laboratory(ies) or request an interview of one or more of the staff working at the laboratory(ies). Records and documents for review should be selected carefully to ensure that they are representative and adequately characterize the programme, system, or process being assessed. However, a document review alone is not usually sufficient to assure the degree to which documents accurately reflect work activities. Document review should therefore always be combined with discussions, interviews, questions and most importantly observation. To the extent possible, the WHO team should witness actual operations and activities. For the purposes of evaluation and assessment of the laboratory testing processes, operations and practice, the WHO team should make use of the "questionnaire" attached as Appendix A5.1 to this document. The “questionnaire" should be considered as an aide memoire for ensuring all critical elements are evaluated. Although the agenda for the expert review of the laboratory testing should be respected, it may be amended/adjusted if needed. Changes to the agenda should be discussed with the participants from the RA and the national control laboratory. At agreed intervals (at the end of each working day, for example), the WHO team should meet with the participants from the national control laboratory to review the process and plan for the expert review of laboratory testing. At such meetings, the WHO team should provide feedback on the strengths and gaps so far identified. Throughout the expert review, the WHO team should make clear, accurate and legible notes. Such notes should provide relevant yet detailed facts that serve as a record of what was assessed and evaluated. Such notes should ideally be used for formulation of the expert review of laboratory testing report. 6. Overview of the expert review process 201 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Once the expert review of laboratory testing is completed, the WHO team should hold a de- briefing meeting with the national control laboratory, involving, as appropriate, other representatives from the RA (such as top management). The purpose is to brief attendees about the activities conducted and present the findings, including the identified strengths, gaps, areas to be improved and recommendations, if any. 6.4 Expert review report The WHO team should issue an expert review report (in English or bilingual), containing general information of the activities, findings (strengths, gaps, and areas for improvement) and recommendations, if applicable, as well as the questionnaire used to assess the performance of laboratory testing activities (see Appendix A5.1). The finalized expert review report should be made available to WHO Secretariat within 14 working days from the last day of the activity. 202 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 7. Roles and Responsibilities The expert review should be seen as a collaborative exercise to which several parties, including RA, national control laboratory, WHO Secretariat, and WHO team are contributing. This section is meant to provide guidance on the roles and responsibilities among the aforementioned parties. 7.1 Relevant RA The RA concerned with the expert review is responsible for: a. discussing and agreeing with the WHO Secretariat the selection of the laboratory(ies) which will be subject to the review. b. designating one or more focal persons to coordinate the activities relating to the expert review. c. communicating and coordinating with the visited laboratory(ies), including all necessary management and logistical arrangements. d. granting the WHO team access to all relevant data and information throughout the expert review. e. provide the necessary clarifications and explanations in response to questions from the WHO team. Seek and obtain any consent necessary from any involved stakeholder in order to share the relevant information with WHO. 7.2 Relevant laboratory The national control laboratory or external lab concerned with the expert review is responsible for: a. sharing with WHO, through the secure information-sharing platform or any other agreed means, all necessary information and documentations including, among others, national code/regulations/guidelines, relevant procedures, data specific to the site(s) selected for the expert review. b. Nominate officials for granting them access to the WHO secure information sharing platform. c. Prepare all materials requested by the WHO team, if any, prior to the planned review. d. Provide clarifications and explanations, sought by the WHO team, of systems and protocols used for daily activities, and e. Respond to the WHO team’s questions and calls for interview, if any. 7.3 WHO Secretariat (WHO headquarters, regional offices and country offices) WHO headquarters (Regulatory Systems Strengthening Team), in collaboration with WHO regional offices and relevant country offices, is responsible for: a. establishing and maintaining the tools and databases related to expert review. b. establishing a roster of qualified experts. c. training experts in order to ensure consistency and quality of the process as well as robustness of the assessment outcome d. discussing and agreeing with the RA on selection of the laboratory(ies) which will be subject to the expert review. e. establishing a dedicated country page for the expert review on the WHO information sharing platform and uploading of all relevant documentation for access and archive purposes. f. selecting WHO team members from the roster of experts qualified to perform the expert review on behalf of WHO g. designating the WHO team leader h. organization of any necessary contractual arrangements. 7.4 WHO team leader The WHO team leader is responsible for: a. leading and coordinating the expert review of laboratory testing from the beginning to the end of the process. He/she will also participate in the evaluation and assessment of the performance and functionality of laboratory testing during the expert review. b. Briefing the WHO team members on various aspects related to the expert review including context, background, objectives, process and methodology, roles, and responsibilities as well as safety issues, if any. 7. Roles and Responsibilities 203 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual c. Coordinating work among all members of the WHO team to ensure smooth and consistent completion of the abridged assessment and avoid work duplication and/or conflicts. d. Communicating with RA officials on behalf of WHO. e. Delivering presentations – ideally, presentations during the opening and closing meetings of the expert review will be made and handled by the WHO team leader. Nevertheless, the preparation of these presentations as well as inputs from different WHO Team members would be necessary. Similarly, the WHO Team leader may invite any of the WHO team members to present the findings, provide clarifications, answer questions of the RA/NCL, if needed. f. Delivering the expert review report: the overall report of the expert review should ideally be prepared by all the WHO Team. However, the responsibility of delivering the finally agreed report lies on the WHO Team leader. 7.5 WHO team member The WHO team members are responsible for: a. reviewing and signing the relevant administrative documents including invitation letter, confidentiality agreement, and declaration of interests form. b. making necessary travel arrangements (e.g., booking flights and obtaining visas) as described in the invitation letter. c. complying with the immunization requirements and bring with them a copy of their immunization certificates, if necessary. d. respecting all applicable protocols, ethics and codes of conduct. e. assessing and evaluating the performance of laboratory testing operations and activities using the questionnaire attached as Appendix A5.1 to this document. f. identifying strengths as well as gaps and areas for improvement, if any. The identified strengths and areas for improvement should be presented in the visit closing meeting. g. preparing a detailed report on the assessment conducted (see section 6.5). The expert review report should be provided to WHO within 14 working days of the last day of the onsite assessment. If possible, a draft of the same shall be delivered by the WHO team on the last day of the onsite assessment. The report may quote the different components/sections in the questionnaire. 7.6 RA/national control laboratory participants The national control laboratory participants are responsible to: a. Establish and maintain communication between the WHO team, and the staff of the lab b. Keeping senior management informed on the expert review c. Coordinate the visit on-site d. Discuss and consider any request for adjustment of the agenda e. Prepare any and all materials requested by WHO Team f. Ensure easy access of the WHO team to the requested documents, information and persons g. Provide clarifications and explanations, sought by WHO team, of systems and protocols used for daily activities h. Respond to WHO team’s questions and calls for interview, if any. 204 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8. Bibliography Laboratory testing (LT): Indicators and factsheets. In: WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021. (https://apps.who.int/iris/ handle/10665/341243, accessed 6 April 2023). Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans. Geneva: World Health Organization, in press. Global Competency Framework for Regulators of medicines;. Geneva: World Health Organization, in press. WHO good practices for pharmaceutical quality control laboratories. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fifty fourth report. Geneva: World Health Organization; 2020: Annex 13 (WHO Technical Report Series, No. 1025; https://apps.who.int/iris/handle/10665/331814, accessed 17 May 2023). 9. Document change history 205 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 9. Document change history Version no. Date of issue Main changes 1 June 2023 First version 206 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities About this questionnaire ƒ This questionnaire is intended to assess the performance of the medical products laboratory testing (LT) function through an expert review of lab-related activities. ƒ The questionnaire comprises three independent parts: – Part A: the quality management system of the laboratory – Part B: staff competencies for each laboratory role – Part C: performance and documentation of laboratory activities ƒ The rating scale and passing score is different for each part, as indicated in the questionnaire. ƒ The questionnaire comprises both “open-ended questions” and “closed-ended questions”. ƒ The laboratory is requested to fill in the “lab input” column (self-assessment) for Part A and Part C of the questionnaire. ƒ The WHO team should complete the relevant fields (expert input and scores) and attach a copy of the completed questionnaire to the expert review of laboratory testing report. Rating WHO uses the expert evaluation and comments to determine whether the above-mentioned RA can be considered to acceptably meet WLA requirements. In order for an authority to be granted WLA status for LT, the national control laboratory or external laboratory must fulfil the following criteria: ƒ achieve a passing score in each component of Part A. ƒ achieve a satisfactory score in each of the eight areas of the matrix in Part B; and overall, score “YES” in not less than 85% of the components (excluding non-applicable components). ƒ be scored at “advanced level” in at least 80% of applicable items in Part C (that is, 18 out of 23) for each of the randomly selected reports; with no item scored below intermediate level. Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 207 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part A: Questionnaire for assessment of the Quality Management System of the National Control Laboratory and/or external laboratory The expert should evaluate and rate each of the requirements set out in the table below. All answers must be confirmed through the provision, review, and assessment of appropriate quality documents. These documents will include the quality manual, if one exists, standard operating procedures, test records, monitoring sheets and records of instrument qualification etc. Any non-applicable requirements should be clearly indicated with the term "N/A" (not applicable) and must be fully justified. Part A - QMS evaluation Country: __________________ Institution: __________________ Dates: __________________ Assessors: __________________ Requirement description Rating scale and score Pass or Fail Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring A.1 The scope of the laboratory’s activities is well described and established in a quality management system (QMS) Scale 0–3 Passing score: 3 ☐ 0 no scope in QMS ☐ 1 scope unclearly defined in QMS ☐ 2 scope clearly defined but not established in accordance with international standards ☐ 3 scope clearly defined, written, and established in accordance with international standards. 0-2 ☐ Fail 3 ☐ Pass A.2 A written and clear statement of the laboratory management’s intentions with respect to the standard of customer service it will provide Scale 0–2 Passing score: 2 ☐ 0 no statement ☐ 1 written statement but not clear ☐ 2 written and clear statement. 0-1 ☐ Fail 2 ☐ Pass 208 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Requirement description Rating scale and score Pass or Fail Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring A.3 A written and clear statement on the laboratory management’s commitment to comply with specific technical guidance e.g., ISO, WHO, official medicines control laboratory etc. Scale 0–2 Passing score: 2 ☐ 0 no statement ☐ 1 written statement but not clear ☐ 2 written and clear statement 0-1 ☐ Fail 2 ☐ Pass A.4 An organizational structure that clearly defines the extent and limits of responsibilities for operational and functional activities pertaining to quality. Scale 0–3 Passing score: 3 ☐ 0 structure available but does not describe operations ☐ 1 reporting structure defined but does not relate to quality and responsibilities are not well defined ☐ 2 structures for quality defined but responsibilities are not clear ☐ 3 structure for quality is clear and responsibilities are well defined. 0-2 ☐ Fail 3 ☐ Pass A.5 A clear structural outline for documents used in the laboratory QMS Scale 0–3 Passing score: 2 ☐ 0 no outline ☐ 1 outline available but does not follow reference guidance or requirement ☐ 2 outline available, in line with reference guidance but not fully complied with and no other templates described ☐ 3 outline available, in line with reference guidance, and no evidence of other templates found. 0-1 ☐ Fail 2-3 ☐ Pass A.6 A written policy for internal and external audits Scale 0–3 Passing score: 2 ☐ 0 no policy available ☐ 1 practised but no written policy available ☐ 2 written policies available, but not fully complied with ☐ 3 written policy available, in line with reference guidance or standard, and evidence of compliance. 0-1 ☐ Fail 2-3 ☐ Pass Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 209 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Requirement description Rating scale and score Pass or Fail Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring A.7 A written policy for implementing and verifying corrective actions and risk management system Scale 0–3 Passing score: 2 ☐ 0 no policy available ☐ 1 practiced but no written policy available ☐ 2 written policy available, but not fully complied with ☐ 3 written policy available, in line with reference guidance or standard, and evidence of compliance. 0-1 ☐ Fail 2-3 ☐ Pass A.8 A written policy for dealing with complaints Scale 0–3 Passing score: 2 ☐ 0 no policy available ☐ 1 practiced but no written policy available ☐ 2 written policy available, but not fully complied with ☐ 3 written policy available, in line with reference guidance or standard, and evidence of compliance. 0-1 ☐ Fail 2-3 ☐ Pass A.9 A written policy for performing management reviews of the QMS Scale 0–3 Passing score: 2 ☐ 0 no policy available ☐ 1 practiced but no written policy available ☐ 2 written policy available, but not fully complied with ☐ 3 written policy available, in line with reference guidance or standard, and evidence of compliance. 0-1 ☐ Fail 2-3 ☐ Pass A.10 A written policy for participating in appropriate proficiency testing schemes and evaluation of the results Scale 0–3 Passing score: 3 ☐ 0 no policy available ☐ 1 practiced but no written policy available ☐ 2 written policy available, but not fully complied with ☐ 3 written policy available, in line with reference guidance or standard, and evidence of compliance. 0-2 ☐ Fail 3 ☐ Pass A.11 A written policy to select service providers and suppliers Scale 0–3 Passing score: 2 ☐ 0 no policy available ☐ 1 practiced but no written policy available ☐ 2 written policies available, but not fully complied with ☐ 3 written policy available, in line with reference guidance or standard, and evidence of compliance. 0-1 ☐ Fail 2-3 ☐ Pass 210 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Overall conclusion for Part A → Number of PASS scores _________ out of 11 items (100% required to pass) Based on the above, WHO experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 211 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part B: Questionnaire for assessment of competence of staff at the national control laboratory and/or external laboratory The expert should review records and reports, or observe members of the workforce performing analyses, evaluating results, or doing other technical laboratory tasks. Alternatively, the expert can review internal documents of past internal competency assessments. The expert should request and review: a. records of written or oral tests, including those to evaluate training effectiveness b. work samples using established rubric c. results of internal competency assessments, where possible (done through intra-laboratory proficiency testing or comparisons using internally generated data d. trainers’ training records e. internal human resources qualification records f. problem logs and incident investigations. In addition, the expert should complete directly observed competency assessments of analysts, using the matrix below. For each analyst under observation, the expert should first define responsibility and job function to identify non-applicable components. Analysts should then be observed as they perform routine work processes and procedures, and the matrix should be used to determine if all steps were properly completed. Before performing a test, analysts are expected to have read all standard operating procedures, manuals, logs, work instructions and workstation tasks, as well as any other procedure-relevant documents. All procedures should be performed according to standard operating procedures; and competency should be based on how well the analyst under observation adheres to these. Any non-applicable requirements should be clearly indicated with the term "N/A" (not applicable) and must be fully justified. Part B - Directly observed competency assessments Country: __________________ Institution: __________________ Dates: __________________ Assessors: __________________ B.1. Sample receipt and record initiation Score WHO expert input Provide a justification for the scoring B.1.1 Did the responsible staff member complete the sample chain of custody control form? ☐ Yes ☐ No B.1.2 Did the responsible staff member check if sample information matches the analysis request form and check the general applicability of the test requested? ☐ Yes ☐ No B.1.3 If sample and analyses request form details do not match, was the responsible staff member able to apply sample acceptance and rejection criteria? ☐ Yes ☐ No B.1.4 If the samples do not match the general applicability of the request, did the responsible staff member note this under the remarks section? ☐ Yes ☐ No B.1.5 Did the responsible staff member assign laboratory reference numbers to the forms and samples correctly? ☐ Yes ☐ No B.1.6 Did the responsible staff member log in the samples, forms and documents in the appropriate database or laboratory information system? ☐ Yes ☐ No B.1.7 Did the responsible staff member capture and save records correctly? ☐ Yes ☐ No 212 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Overall score for section B.1 Guidance: WHO experts should use the above-listed items to qualitatively evaluate the overall “sample receipt and record initiation” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text B.2. Reagents/sample and standard preparation Score WHO expert input Provide a justification for the scoring B.2.1 Did the responsible staff member select appropriate test method (pharmacopeial/inhouse validate/ manufacturer)? ☐ Yes ☐ No B.2.1 Did the responsible staff member select appropriate test method (pharmacopeial/inhouse validate/ manufacturer)? ☐ Yes ☐ No B.2.2 Did the responsible staff member check availability of necessary equipment to do the test? ☐ Yes ☐ No B.2.3 Did the responsible staff member perform the appropriate daily or pre-use calibration or checks on equipment (e.g.: pH meter) and document it? ☐ Yes ☐ No B.2.4 Did the responsible staff member select appropriate reference material? ☐ Yes ☐ No B.2.5 Did the responsible staff member select appropriate equipment for testing? ☐ Yes ☐ No B.2.6 Did the responsible staff member carry out appropriate sample preparations and dilutions? ☐ Yes ☐ No B.2.7 Did the responsible staff member prepare media, mobile phases, reagents correctly? ☐ Yes ☐ No B.2.8 Did the responsible staff member label samples as per procedure? ☐ Yes ☐ No Overall score for section B.2 Guidance: WHO experts should use the above-listed items to qualitatively evaluate the overall “reagents/samples and standard preparation” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text B.3 Use of equipment Score WHO expert input Provide a justification for the scoring B.3.1 Did the responsible staff member perform bench clearing, cleaning and environmental monitoring (including safety precautions) and complete relevant logs? ☐ Yes ☐ No Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 213 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual B.3.2 If equipment initializing failed, did the responsible staff member inform the supervisor? ☐ Yes ☐ No B.3.3 Did the responsible staff member arrange samples according to the worksheet or work instruction? ☐ Yes ☐ No B.3.4 Did the responsible staff member check label samples as per procedure and equipment? ☐ Yes ☐ No B.3.5 Did the responsible staff member apply the correct sample on the correct slot of the worksheet/sample tray/map? ☐ Yes ☐ No B.3.6 Did the responsible staff member decontaminate devices used in sample transfer or dilution etc.? ☐ Yes ☐ No B.3.7 Did the responsible staff member use the correct pipetting technique? ☐ Yes ☐ No B.3.8 Did the responsible staff member measure the right quantities of sample, reference standards, controls and reagents? ☐ Yes ☐ No B.3.9 Did the responsible staff member use the correct method of sample extraction? ☐ Yes ☐ No Overall score for section B.3 Guidance: WHO experts should use the above-listed items to qualitatively evaluate the overall “equipment use” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text B.4 Running of sample analysis Score WHO expert input Provide a justification for the scoring B.4.1 Did the responsible staff member create a new method/ select an appropriate existing method? ☐ Yes ☐ No B.4.2 Did the responsible staff member place control/blank samples on the correct slot of the worksheet or sample tray? ☐ Yes ☐ No B.4.3 Did the responsible staff member select the correct protocol for running the test? ☐ Yes ☐ No B.4.4 Did the responsible staff member follow the correct sequence testing steps during processing? ☐ Yes ☐ No B.4.5 Did the responsible staff member address any “error” notification on the equipment? ☐ Yes ☐ No B.4.6 Did the responsible staff member record error logs on equipment use log and appropriate action taken? ☐ Yes ☐ No B.4.7 Did the responsible staff member follow correct gowning procedure before running the sample analysis? ☐ Yes ☐ No B.4.8 Did the responsible staff member respect aseptic technique and cross contamination risk minimization measures, where applicable? ☐ Yes ☐ No B.4.9 Did the responsible staff member apply safe procedure for handling hazardous substances and disposal of contaminated waste? ☐ Yes ☐ No 214 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Overall score for section B.4 Guidance: WHO experts should use the above-listed items to qualitatively evaluate the overall “running of sample analysis” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text B.5 Reporting of results Score WHO expert input Provide a justification for the scoring B.5.1 Did the responsible staff member follow quality control rules to review the test data and records? ☐ Yes ☐ No B.5.2 Did the responsible staff member review results of sample against the controls/standards? ☐ Yes ☐ No B.5.3 Did the responsible staff member interpret results according to the SOP including correctly using the correction factors, if needed? ☐ Yes ☐ No B.5.4 Did the responsible staff member review results against acceptance criteria or specifications on the request form? ☐ Yes ☐ No B.5.5 In case of out-of-specification results, was the procedure for out-of-specification investigations followed? ☐ Yes ☐ No B.5.6 Did the responsible staff member submit results to the supervisor for review and approval? ☐ Yes ☐ No B.5.7 Did the responsible staff member enter results in the laboratory information system/ data base correctly? ☐ Yes ☐ No B.5.8 Did the responsible staff member dispatch results correctly in their respective boxes/envelopes? ☐ Yes ☐ No B.5.9 Did the responsible staff member formally note or advise on the significance or give judgment with reference to the results including recommending corrective action when controls are unacceptable? ☐ Yes ☐ No B.5.10 Did the responsible staff member follow the correct sequence in filing the records? ☐ Yes ☐ No B.5.11 Did the responsible staff member follow the correct sequence in compiling reports? ☐ Yes ☐ No Overall score for section B.5 Guidance: WHO experts should use the above-listed items to qualitatively evaluate the overall “reporting of results” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 215 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual B.6 Storage of records Score WHO expert input Provide a justification for the scoring B.6.1 Did the responsible staff member store records in the appropriate area? ☐ Yes ☐ No Overall score for section B.6 WHO experts should use the above-listed items to qualitatively evaluate the overall “storage of records” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text B.7 Stock management Score WHO expert input Provide a justification for the scoring B.7.1 If there were any reagent or consumables replaced, did the responsible staff member verify these and complete the appropriate records? ☐ Yes ☐ No Overall score for section B.7 WHO experts should use the above-listed items to qualitatively evaluate the overall “stock management” process. → Based on the above, WHO experts conclude that the overall section is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text B.8 Reviewing quality and technical records (observe the analyst reviewing quality & technical records) Score WHO expert input Provide a justification for the scoring B.8.1 Did the responsible staff member record or demonstrate knowledge in reviewing the following records? a. Correct formulae and spreadsheets (where applicable used) – calculations and results derivation b. Review of internal quality control data c. Detecting data trends d. Recognition and interpretation of inconsistent results and test system problems and troubleshooting e. Corrected reports f. Equipment error logs g. Acceptance criteria/specifications and critical values ☐ Yes ☐ No Overall score for section B.8 Guidance: WHO experts should use the above-listed items to qualitatively evaluate the overall “reviewing quality and technical records” process. 216 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual → Based on the above, WHO experts conclude that the overall section is: Please tick one of the below checkboxes ☐ Satisfactory ☐ Not satisfactory ☐ Not applicable Justification: Please provide text Overall conclusion for Part B → Number of satisfactory areas (excluding non-applicable components) _________ out of 8 areas (100% required to pass) → % of components scored as “Yes”, excluding non-applicable components____________ (not less than 85% to pass) → Based on the above, WHO experts conclude that overall part B is: Please tick one of the checkboxes below ☐ Satisfactory ☐ Not satisfactory Justification: Please provide text Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 217 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Part C: Questionnaire for the assessment of analytical reports issued by the National Control Laboratory and/or external laboratory A representative number of work samples should be selected for evaluation. For each randomly selected work sample, the expert should evaluate and rate each of the items set out in the table below (Part C). This means that the expert will need to complete the same number of tables as the number of work samples selected. Any non-applicable requirements should be clearly indicated with the term “N/A” (not applicable) and must be fully justified. Part C – Assessment of analytical reports Country: __________________ Institution: __________________ Dates: __________________ Assessors: __________________ C.1 Quality of the report to communicate and facilitate comprehension by the reader Rating scale and score Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring C.1.1 Presentation ☐ Not acceptable: The flow of test data or discourse is not organized and difficult to follow. ☐ Basic: The test data or discourse are organized and easy to follow in some sections of the report. ☐ Intermediate: The test data or discourse are organized and easy to follow in most sections of the report. ☐ Advanced: The test data or discourse are organized and easy to follow, enhancing readability and comprehension in all sections the report. C.1.2 Critical features ☐ Not acceptable: The report does not include any analyst comments/ remarks on the critical features of the sample that may impact test results. ☐ Basic: The report includes the analyst’s comments/remarks on some of the critical features of the sample that may impact test results. ☐ Intermediate: The report includes the analyst’s comments/remarks on most of the critical features of the sample that may impact test results. ☐ Advanced: The report includes the analyst’s comments/remarks on all the critical features of the sample that may impact test results 218 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual C.2 Completeness of the report to provide a comprehensive and complete picture of the situation or sample under consideration Rating scale and score Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring C.2.1 Templates ☐ Not acceptable: The correct and approved templates and forms were not used or available (including out of trend, out of specification, deviation as applicable). ☐ Basic: The correct and approved templates and forms were almost never used, but were used sometimes (including out of trend, out of specification, deviation as applicable), with no justification when not used. ☐ Intermediate: The correct and approved templates and forms were almost always used, but not always (including out of trend, out of specification, deviation as applicable), with justification when not used. ☐ Advanced: The correct and approved templates and forms were always used (including out of trend, out of specification, deviation as applicable). C.2.2 Inclusion of invalid data ☐ Not acceptable: Only compliant (valid) data included in the report. No inclusion of “invalid” data or data from “rejection report” in case of re-test and re-sampling in the report. ☐ Basic: Compliant (valid) data included in the report. Inclusion of some, but not all, “invalid” data from re-sampling and re-testing in the report. ☐ Intermediate: Inclusion of all invalid data in the report, including from “rejection report” in case of re-test and re-sampling. But these data were not appropriately labelled as “invalid”. ☐ Advanced: Inclusion and appropriate labelling of all invalid data, including from “rejection report” in case of re-test and re-sampling. C.2.3 Environment ☐ Not acceptable: No records to show that environmental conditions such as temperature and humidity were checked or monitored while sample was handled, and testing conducted. Microbiological analysis tests are not performed under pharmacopeial conditions (laminar air flow, biosafety cabinet, isolator, clean room). Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 219 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ☐ Basic: Environmental conditions such as temperature and humidity were checked at some unspecified points, but there was no demonstration of monitoring at critical times while sample was handled, and testing conducted. The relevant standard operating procedure was not used. Microbiological analysis tests are partially performed under pharmacopeial conditions (laminar air flow, biosafety cabinet, but no clean room when applicable). ☐ Intermediate: Environmental conditions such as temperature and humidity were checked and monitored at all critical points while sample was handled, and testing conducted. But records were not complete for some critical conditions. Inadequate records of any other potentially interfering/testing activities at the time of the testing, if applicable. The relevant standard operating procedure was only partly followed. Microbiological analysis tests are performed under pharmacopeial conditions (laminar air flow and biosafety cabinet, isolator, in a clean room when applicable) but there is no regular monitoring or requalification. ☐ Advanced: Complete records to show that critical environmental conditions such as temperature and humidity were checked and monitored while sample was handled, and testing conducted. Records were made of any other potentially interfering/ testing activities at the time of the testing, if applicable. The relevant standard operating procedure was appropriately followed. Microbiological analysis tests are performed under pharmacopeial conditions (laminar air flow, biosafety cabinet, isolator, and clean room when applicable) and there is regular monitoring and annual requalification. C.2.4 Facilitating information ☐ Not acceptable: Information needed to facilitate testing and interpretation of results is incorrect or rarely included in the report. ☐ Basic: Some of the information needed to facilitate testing and interpretation of results is included in the report for simple issues. ☐ Intermediate: Most of the information needed to facilitate testing and interpretation of results where few factors or non-complex issues are involved is included in the report. ☐ Advanced: All the correct information needed to facilitate testing and interpretation of results, including all limitations, is included in the report. C.2.5 Report summary ☐ Not acceptable: The report summary is incomplete and lacks adequate detail. It does not outline essential test data, requiring the reader to make substantial reference to the source data and other references. 220 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ☐ Basic: The report is largely incomplete and lacks detail. It does not include significant or essential information, requiring the reader to corroborate information from source or other reference sources. ☐ Intermediate: Most of the essential information is provided in the report, requiring the reader to refer to the source test data only occasionally. There were some omissions of essential information, which could be referred from easily accessible records. Special techniques (e.g., dilution) were well reported. Deviations from specified (authorized) methods were recorded but not justified. ☐ Advanced: The report is complete and detailed, and consistently includes essential information necessary to understand the analyst’s conclusions. Special techniques (e.g., dilution) were well reported. Deviations from specified (authorized) methods were recorded, justified, and approved. C.3 Scientific rigour to ensure a scientific approach is applied for unbiased analysis and interpretation of the evidence or data Rating scale and score Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring C.3.1 Sample and media integrity ☐ Not acceptable: For microbiological analysis. Some of the following was observed during analyses: a) no evidence of use of negative and positive controls. b) records of satisfactory incubation conditions were not maintained. ☐ Basic: For microbiological analysis. All the following was observed during analyses: a) inadequate evidence of use of negative and positive controls. b) some records of satisfactory incubation conditions were maintained. ☐ Intermediate: For microbiological analysis. Some of the following was observed during analyses: a) some adequate evidence of use of negative and positive controls. b) full records of satisfactory incubation conditions were maintained. Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 221 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ☐ Advanced: For microbiological analysis. All the following were observed during analyses: a) adequate evidence of use of negative and positive controls and b) full records of satisfactory incubation conditions were maintained. C.3.2 Presence of conclusions ☐ Not acceptable: The analyst’s opinions and conclusions are largely absent from the report, if relevant (or placed in incorrect sections). It is difficult to understand the significance/relevance of the analyst’s discussion and conclusions. ☐ Basic: The analyst’s opinions and conclusions are present and accurately placed in some sections of the report, if relevant. It is often not clear what the basis for the analyst’s discussion and conclusions are. ☐ Intermediate: The analyst’s opinions and conclusions are present and accurately placed in most sections of the report, if relevant. The basis for the analyst’s discussion and conclusions is clear in most cases. ☐ Advanced: The analyst’s opinions and conclusions are present and accurately placed throughout the report, if relevant. The analyst noted anomalous or suspect data (e.g., chromatographic peaks) or observations (e.g., abnormal appearances of samples, reagents, solvents or solutions; reagents not dissolved completely; or out of trend results). Throughout the report it is clear to the reader what the analyst thinks of the information, approach, and conclusions. C.3.3 Validity of conclusions ☐ Not acceptable: The analyst’s conclusions are inconsistent with the test data generated; or are scientifically invalid. ☐ Basic: The analyst’s conclusions are sometimes inconsistent with the test data generated; or are sometimes scientifically invalid. ☐ Intermediate: The analyst’s conclusions are almost always consistent with the test data; and are scientifically valid. ☐ Advanced: The analyst’s conclusions are always consistent with the test data; and are scientifically valid. C.3.4 Alignment of conclusions ☐ Not acceptable: recommendations and conclusions were not in line with applicable regulatory requirements and internal policies. ☐ Basic: Some recommendations and conclusions were not in line with applicable regulatory requirements and internal policies 222 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual ☐ Intermediate: Most recommendations and conclusions were in line with applicable regulatory requirements and internal policies ☐ Advanced: All recommendations and conclusions were in line with applicable regulatory requirements and internal policies. C.3.5 Requirement interpretation ☐ Not acceptable: The analyst made significant misinterpretations of applicable regulatory requirements. ☐ Basic: The analyst made some misinterpretation of applicable regulatory requirements. ☐ Intermediate: The analyst shows an adequate level of interpretation of requirements for non-complex issues. ☐ Advanced: The analyst shows an expert-level interpretation of regulatory requirements, including complex issues. C.4 Compliance with regulatory requirements and policies Rating scale and score Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring C.4.1 Context ☐ Not acceptable: There is no evidence of consideration of the context- associated risk of the issue in any cases when applying policy, guidance, and procedures. ☐ Basic: There is some evidence of consideration of the context-associated risk of the issue in some cases when applying policy, guidance, and procedures. ☐ Intermediate: There is evidence of consideration of the context-associated risk of the issue in most cases when applying policy, guidance, and procedures. ☐ Advanced: There is sufficient evidence of consideration of the context- associated risk of the issues in all cases when applying policy, guidance, and procedures. Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 223 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual C.5 Data integrity to ensure data are attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring and available (ALCOA Plus) Rating scale and score Laboratory input Provide an overview of the system in place, referring to relevant supportive evidence WHO expert input Provide a justification for the scoring C.5.1 Critical test equipment ☐ Not acceptable: None of the critical test equipment (including balances, pipettes, glassware, and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. ☐ Basic: Some of the critical test equipment (including balances, pipettes, glassware, and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. There was some cross-reference to instrument identification, last calibration/ qualification date, and usage record (instrument record). ☐ Intermediate: Most of the critical test equipment (including balances, pipettes, glassware, and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. Some critical equipment was missed. But there was cross-reference to instrument ID, last calibration/ qualification date, and usage record (instrument record). ☐ Advanced: All the critical test equipment (including balances, pipettes, glassware, and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. This includes cross- reference to instrument ID, last calibration/ qualification date, and usage record (instrument record). C.5.2 Test methods ☐ Not acceptable: Some test methods and calculations followed were ambiguous with an unclear source. There was use of unapproved spreadsheets and formulae. Inappropriate rounding-off was used. ☐ Basic: Some ambiguous test methods and calculations were followed. It was not clear if the analyst used approved/ validated spreadsheets or approved formulae. There was inconsistent application of policies such as rounding off. ☐ Intermediate: All test methods and calculations were unambiguous and followed. Approved spreadsheets, formulae and policies were not always applied with no valid reason. ☐ Advanced: All test methods and calculations were unambiguous and followed. Approved spreadsheets, formulae and policies were consistently applied. All calculations were reviewed and approved. 224 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual C.5.3 Reagents and reference materials ☐ Not acceptable: There was no evidence or records showing that reagents and reference materials used in the test were appropriate for use (that is, prepared according to procedure and within expiry date). There was no reference to standards (e.g., solution preparations working standard, primary standard, chemical reference standard, system suitability test solution preparation and usage details). ☐ Basic: There was some evidence or records showing that reagents and reference materials used in the test were appropriate for use (that is, prepared according to procedure and within expiry date). There was no reference to standards (e.g., solution preparations working standard, primary standard, chemical reference standard, system suitability test solution preparation and usage details). ☐ Intermediate: There was evidence or records showing that reagents and reference materials used in the test were appropriate for use (that is, prepared according to procedure and within expiry date). There was some reference to standards (e.g., solution preparations working standard, primary standard, chemical reference standard, system suitability test solution preparation and usage details). ☐ Advanced: All reagents and reference materials used in the test were appropriate for use (that is, prepared according to procedure and within expiry date). There was always reference to standards (e.g., solution preparations working standard, primary standard, chemical reference standard, system suitability test solution preparation and usage details). C.5.4 Validation ☐ Not acceptable: There was no information on whether the test methods used were validated/ verified and authorized for use within the laboratory. ☐ Basic: There was some information on whether the test methods were authorized for use within the laboratory in the test. There was no information on whether the test methods were validated/ verified. ☐ Intermediate: There was adequate information that all test methods used were authorized for use within the laboratory. Although method validation/ verification was claimed, no reference or evidence was included. ☐ Advanced: There was adequate information that all test methods used were validated/ verified and authorized for use within the laboratory. There was reference to a validation report as evidence. Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 225 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual C.5.5 Sample and materials use ☐ Not acceptable: There was no information recorded to confirm: a) use of correct samples/ reagents/reference materials, including microbiological media; b) appropriate storage conditions for samples; c) storage of samples or media/ reagents for the correct time before being used. ☐ Basic: There was information recorded to confirm: a) use of correct samples/ reagents/reference materials, including microbiological media; b) appropriate storage conditions for samples. This information was not always found in the appropriate sections of records. ☐ Intermediate: Information was clearly recorded to confirm a) use of correct samples/ reagents/reference materials, including microbiological media and written details; b) appropriate storage conditions for samples and media/ reagents; c) storage of samples for the correct time before being tested. This information was found in the appropriate sections of records for analytical preparation at various stages; although some relevant documents/ reports were not easily recognizable and may affect traceability. ☐ Advanced: Information was clearly recorded using a continuous monitoring device to confirm a) use of correct samples/ reagents/reference materials, including microbiological media and written details; b) appropriate storage conditions for samples and media/ reagents; c) storage of samples for the correct time before being tested. This information was found in the appropriate sections of records for analytical preparation at various stages, with clear written evidence with document and record IDs to facilitate traceability and complete reconstruction of all tests. C.5.6 Conditions for testing ☐ Not acceptable: There was no record of whether the conditions for testing (e.g., system suitability and assay validity criteria) were met before and during analyses. ☐ Basic: There were some records of statements that conditions for testing (e.g., system suitability and assay validity criteria) were met before analyses. There was no record of the actual conditions. ☐ ntermediate: There were records and confirmation that conditions for testing (e.g., system suitability and assay validity criteria) were met before and during analyses. Missing or failed conditions were noted. ☐ Advanced: There were records and confirmation that conditions for testing (e.g., system suitability and assay validity criteria) were met before and during analyses. Missing or failed conditions were noted and assessed through risk analysis. 226 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual C.5.7 Blank spaces ☐ Not acceptable: No blank pages/spaces in worksheets were crossed out. ☐ Basic: Blank pages/spaces in worksheets were almost never crossed out, but sometimes crossed out. ☐ Intermediate: Blank pages/spaces in worksheets were almost always crossed out, but not always crossed out; and details such as signature, date and “n/a” were included. ☐ Advanced: Blank pages/spaces in worksheets were always crossed out; and details such as signature, date and “n/a” were included. C.5.8 Links to raw data ☐ Not acceptable: There was no reference or link to raw data as evidence of contemporaneous recording. ☐ Basic: There was an occasional reference or link to raw data as evidence of contemporaneous recording. There were no printed and signed outputs from electronic equipment that does not store data (e.g., some balances, pH meter etc). ☐ Intermediate: There was mostly adequate reference or link to raw data as evidence of contemporaneous recording, including printed and signed outputs from electronic equipment that does not store data (e.g., some balances, pH meter etc) and a link to audit trails. ☐ Advanced: There was always adequate reference or link to raw data as evidence of contemporaneous recording, including printed and signed outputs from electronic equipment that does not store data (e.g., some balances, pH meter etc) and adequate reference to audit trails. C.5.9 Re-testing and re-sampling ☐ Not acceptable: No justification given for a re-test or re-sampling. ☐ Basic: Justification given for re-testing and re-sampling (e.g., calculation error, power outage, equipment failure; testing errors). But relevant standard operating procedure was not used appropriately and there was no cross reference or link to other records as evidence. ☐ Intermediate: Justification given for re-testing and re-sampling (e.g., calculation error, power outage, equipment failure; and testing errors). Appropriate standard operating procedures used and some cross reference or link to other records as evidence (e.g., maintenance record not always provided). ☐ Advanced: Justification given for re-testing and re-sampling (e.g., calculation error, power outage, equipment failure; and testing errors). Appropriate standard operating procedures used and cross reference or link to other records as evidence, including maintenance records for power outages and equipment failure. Appendix A5.1 Expert review questionnaire: for assessing the performance of laboratory testing activities 227 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual C.5.10 ALCOA principles ☐ Not acceptable: The following was noted: a) evidence of unintentional or unauthorized data changes; b) records were not attributable, legible, original, and accurate; or c) copies were not verifiable as true; d) evidence of selective reporting; e) overwriting or deletion of original data. ☐ Basic: The following was noted: a) no evidence of unintentional or unauthorized data changes; b) records were sometimes not attributable, legible, original, and accurate; or c) where applicable, they were evidently true copies; d) no evidence of selective reporting; e) clear evidence of documentation of sequence of processes (e.g., sequence of injections); f ) some evidence of overwriting or deletion of original data. ☐ Intermediate: The following was noted: a) no evidence of unintentional or unauthorized data changes; b) records were attributable, legible, original, and accurate; or c) where applicable, they were evidently true copies; d) no evidence of selective reporting; e) modified or adjusted parameters were traceable or retrievable with some difficulty (e.g., use of manual integration); f ) clear evidence of documentation of sequence of processes (e.g., sequence of injections) g) not all printed records supported with unique identifiers. ☐ Advanced: The following was noted: a) no evidence of unintentional or unauthorized data changes; b) records were attributable, legible, original, and accurate; or c) where applicable, they were evidently true copies; d) no evidence of selective reporting; e) modified or adjusted parameters were easily traceable or retrievable (e.g., use of manual integration); f) clear evidence of documentation of sequence of processes (e.g., sequence of injections); g) all printed records were supported with unique identifiers; h) no overwriting or deletion of original data. Overall conclusion for part C Number of items scored at “advanced” level (excluding not applicable items) _________ out of 23 areas (80% to pass per each of the randomly selected report) Number of items scored at “not acceptable” or “basic” level ____________ (0 to pass) Based on the above, WHO experts conclude that the overall section is: Please tick one of the checkboxes below Justification: ☐ Satisfactory ☐ Not satisfactory Please provide text 228 Expert review of laboratory testing activities Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Outcomes of the Performance Evaluation of laboratory testing activities WHO experts’ overall conclusion of the expert review of laboratory testing The overall conclusion should be based on the evaluation and scoring achieved in each of the individual three afore-mentioned parts of the questionnaire. If one of these parts is found to be unsatisfactory according to the specific rating scale provided, the overall outcome of the performance evaluation should be consequently scored as unsatisfactory. Based on the collective evidence and findings of this expert review of laboratory testing, the WHO experts conclude that the performance of the laboratory testing activities, including QMS, staff competence as well as analytical test reports is: ☐ Implemented ☐ Not implemented Justification: Please provide text Annex 6. The abridged pathway to assess the performance of stringent regulatory authorities (for medicines) and highly performing regulatory authorities (for vaccines) seeking designation as a WHO-listed authority 230 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Code of conduct ................................................................................................................................................................................................................... 231 Abbreviations ..........................................................................................................................................................................................................................232 Glossary .................................................................................................................................................................................................................................. 233 1. Introduction ..................................................................................................................................................................................................................... 234 2. Purpose ............................................................................................................................................................................................................................... 235 3. Scope .................................................................................................................................................................................................................................. 236 4. Objectives and expected outcomes .................................................................................................................................................................237 5. Deliverables ..................................................................................................................................................................................................................... 238 6. Overview of the abridged assessment process .......................................................................................................................................................................................239 6.1 General principles ............................................................................................................................................................................................. 239 6.2 Abridged assessment ................................................................................................................................................................................... 242 6.3 PE report ................................................................................................................................................................................................................. 242 7. Roles and Responsibilities ...................................................................................................................................................................................... 243 7.1 Relevant RA .......................................................................................................................................................................................................... 243 7.2 WHO Secretariat (WHO headquarters, regional and country offices) .......................................................................... 243 7.3 WHO focal point ............................................................................................................................................................................................... 243 7.4 WHO team members .................................................................................................................................................................................... 243 7.5 RA participants ................................................................................................................................................................................................... 244 8. Bibliography ..................................................................................................................................................................................................................... 245 9. Document Change History .................................................................................................................................................................................... 246 Appendix A6.1 The abridged pathway tool ..........................................................................................................................................................247 Contents 231 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Code of conduct WHO values and relies upon the normative and technical advice provided by leading subject matter experts in the context of its advisory/technical committees, meetings, and other similar processes. Such advice contributes to the formulation of the public health policies and norms that are promulgated by WHO for the benefit of its Member States. In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice: a. fully and honestly disclose all relevant interests and biases on the declaration of interests (DOI) Form that may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting, or group members at the outset, that is unless this is done by the chairperson or WHO Secretariat b. spontaneously report any material changes to their disclosed interest on an ongoing basis during the period in which the expert serves the Organization c. respect the confidential nature of committee or meeting deliberations or of the advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO d. undertake not to engage in activities that may bring reputational harm to the WHO process that they are involved in e. undertake to represent their views in a personal and individual capacity with the best interest of WHO in mind as opposed to representing the views of their employers, other institutions, or governments f. actively and fully participate in discussions and deliberations within the relevant advisory group, committee, or meeting. WHO has zero tolerance towards sexual exploitation and abuse, sexual harassment (SEAH) and other types of abusive conduct (that is, discrimination, abuse of authority and harassment). Sexual exploitation, sexual abuse and sexual harassment are considered to be acts of serious misconduct and constitute a basis on which staff members, whether internationally or locally recruited, and contractors can be summarily dismissed. 232 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Abbreviations CT clinical trial oversight GBT Global Benchmarking Tool LR lot release LT laboratory testing MA (registration and) marketing authorization MC market (surveillance and) control ML maturity level NCL national control laboratory PE performance evaluation QMS quality management system RA regulatory authority RI regulatory inspections RS national regulatory system RRS regional regulatory system VL vigilance WLA WHO-listed authorities Glossary 233 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Glossary The definitions given below apply to the terms used in the current document. These terms may have different meanings in other contexts.  Abridged assessment In the context of the abridged pathway, it is the process used by WHO to document and evaluate the performance of Stringent Regulatory Authorities (SRAs) for medicines, and highly performing RAs for vaccines, for the purpose of WHO- listed authority (WLA) designation. The activity is desk-based and consists of an evaluation by a WHO team of experts of pre-selected GBT sub-indicators and PE indicators, as detailed in The abridged pathway tool (see Appendix A6.1). Abridged assessment plan A plan developed by the WHO focal point, in agreement with other WHO team members and WHO Secretariat, to detail different activities, timings, and assignments to be performed during the abridged assessment. Abridged assessment report/PE report A report prepared in English language which is delivered by WHO team following the completion of the abridged assessment. Abridged assessment report provides an overview of the activities conducted, findings, recommendations, if any. Performance evaluation (PE) Indicator Indicator developed to assess and evaluate the performance of different functions. Guidance for PE indicators is available in the form of fact sheets. RA coordinator One or more experts, ideally familiar with the national regulatory activities, who is/are nominated by the RA to represent it and to contribute to the abridged assessment. WHO focal point WHO staff in charge of arranging and coordinating all activities related to the abridged assessment. WHO Secretariat The WHO unit in charge of organizing the abridged assessment. WHO expert A competent expert, who is familiar with WHO published regulations and guidelines in the area of medical products, and indicated in the respective terms of reference (TORs) to perform the abridged assessment. WHO experts should have extensive (more than seven years’) experience and advanced skills in regulatory activities. WHO-listed authority (WLA) A national regulatory authority or a regional regulatory system that has been documented to comply with all the indicators and requirements specified by WHO for listing based on an established benchmarking and performance evaluation process. A regulatory authority can be listed for one or more product categories or for one or more regulatory functions.  234 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 1. Introduction The abridged evaluation pathway is applicable to regulatory authorities (RAs) eligible for the WHO abridged prequalification procedure – that is, stringent regulatory authorities (SRAs) for medicines and highly performing regulatory authorities (for vaccines) previously included in the Interim list of National Regulatory Authorities, published since 2019 on the WHO website. The abridged pathway takes into consideration the extensive knowledge gained with these RAs from a long history of engagement and collaboration with the Prequalification and other WHO regulatory programmes. These authorities are also widely recognized by the international regulatory community as leaders in regulatory science and the oversight of medical products. 2. Purpose 235 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 2. Purpose The purpose of this document is to: a. provide guidance to WHO and to the relevant RA, as well as to other interested parties, on all aspects of the WHO abridged assessment process and methodology, including the relevant procedures and timelines for planning, preparing, conducting, reporting, and follow up, and templates for related documentation. b. define the roles and responsibilities of the WHO team assigned to perform the abridged assessment of a RA. c. describe the roles and responsibilities of the three levels of WHO (WHO headquarters, regional offices and country offices) as well as of the relevant RA in this process. d. establish a level of rigour, consistency and uniformity within the abridged assessment process and confidence in its outcomes. This document should be read in conjunction with other relevant manuals, guidelines, standard operating procedures (SOPs), and work instructions. This document is subject to periodic review and revision as part of the quality system approach applied by WHO. 236 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 3. Scope This document describes the process to initiate, plan, prepare, conduct, report upon, and follow up on abridged assessments. It identifies the key steps involved in the abridged assessment, to confirm that the performance of the concerned RA complies with applicable WHO and internationally recognized requirements. This document equally applies to medicines and biological products, including biotherapeutic products as well as vaccines. 4. Objectives and expected outcomes 237 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 4. Objectives and expected outcomes The objectives and expected outcomes of the abridged assessment are to: a. assess the performance of regulatory activities and operations, conducted by the regulatory authority, to verify compliance with WHO or other internationally recognized requirements, as well as its own regulatory requirements. b. identify strengths and best practices performed by the RA to be shared with other regulators in the context of WLA initiative. 238 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 5. Deliverables After completion of the abridged assessment, the following deliverables should be provided to WHO Secretariat: a. A completed copy of The abridged pathway tool containing scoring and experts’ input of GBT sub- indicators and PE indicators (see Appendix A6.1). b. PE report to be delivered by WHO team. 6. Overview of the abridged assessment process 239 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6. Overview of the abridged assessment process The evaluation process takes into consideration the extensive knowledge gained by WHO with SRAs and highly performing RAs and foresees the use of The abridged pathway tool (Appendix A6.1), which derives from the combination of two components: a. a set of pre-selected GBT sub-indicators, and b. all PE indicators. None of the PE tools are included in The abridged pathway tool, as alternative evaluation mechanisms recognized by WHO already exist and provide sufficient evidence of the performance of RAs in all functions (for example, WHO Prequalification products, emergency use listing products, pharmaceutical inspection cooperation scheme membership, WHO contracted or prequalified labs, official medicines control laboratory members, Benchmarking of European Medicines Agencies assessment, others). 6.1 General principles WHO and the RA should discuss, in advance, and agree on all details and aspects related to the evaluation process, including the participants and the need for translation (if any). To facilitate the abridged assessment, a copy of the relevant evidence in support of regulatory practices should be shared with WHO in advance, preferably four weeks before the assessment. WHO team should have access to all information, people, and assets relevant to the abridged assessment, while respecting all applicable confidentiality arrangements and its code of conduct. 6.1.1 Pre-selected sub-indicators from Global Benchmarking Tool (GBT) The selection of GBT sub-indicators included in The abridged pathway tool (see Appendix A6.1) was based on the following criteria and considerations: ƒ A focus on regulatory functions which, together with the overarching regulatory system, are critical to ensuring the safety, efficacy and quality of products in international supply, and considered indicative of the performance of the system. ƒ In line with the PE framework, a focus on: – Good regulatory practices principles of consistency, flexibility, efficiency, and transparency as hallmarks of a WLA. – Good regulatory practices enablers addressing inter- and intra-organizational communication, collaboration and coordination, and the quality management system. As PE indicators are considered more comprehensive than GBT sub-indicators, GBT sub-indicators are excluded from The abridged pathway tool if they address topics covered by a PE indicator (this is known as “the principle of hierarchy”). Because of this approach, RAs are expected to thoroughly discuss and provide comprehensive evidence in support of each PE indicator, to enable proper understanding and assessment of the regulatory practices in place. Requirements of GBT sub-indicators mentioned in the “References” section of the relevant PE indicator table should be taken into consideration in the preparation of self-assessment. Similarly, sub-indicators under the regulatory system are considered overarching with respect to comparable sub-indicators in regulatory functions, and as such have been included in The abridged pathway tool instead of sub-indicators in individual functions. The abridged pathway tool also includes some ML3 and ML4 sub-indicators that are considered valuable for the purposes of collecting best practices but are not subject to WHO assessment and so are included “for information only”. However, candidate WLAs are still expected to thoroughly discuss and provide comprehensive evidence in support of “for information only” GBT sub-indicators. Additional information may be requested by WHO, should this be considered necessary to gain a full understanding of the applicable regulatory practices. All non-mandatory ML4 sub-indicators are excluded from the evaluation. Table 1 presents an overview of the number of GBT sub-indicators included in The abridged pathway tool obtained through the application of the above- mentioned criteria. Table 2 shows the total number of indicators in the abridged pathway tool, GBT and PE. 240 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table A6.1. Number of sub-indicators included in the abridged pathway, by maturity level and function (abridged/standard), compared to total number of GBT sub-indicators By maturity level (abridged pathway/standard process) By function (abridged pathway/ standard process) ML1 ML 2 ML 3 ML 4 Regulatory system 0/4 0/7 10/27 7/22 17/60 Registration and marketing authorization 0/6 0/2 3/23 2/4 5/35 Vigilance 0/5 0/3 6/14 2/4 8/26 Marketing surveillance and control 0/3 0/4 0/15 0/5 0/27 Licensing establishment 0/2 0/1 0/13 0/3 0/19 Regulatory inspections 0/3 0/2 0/13 3/8 3/26 Laboratory testing 0/2 0/2 4/18 2/6 6/28 Clinical trials 0/2 0/8 2/17 0/3 2/30 RA lot release 0/1 0/3 4/11 0/2 4/17 Total 0/28 0/32 29/151 16/57 45/268 6.1.2 PE Indicators All PE indicators are included in The abridged pathway tool as they are considered key for a proper definition of RAs’ performance. Table A6.2 presents the overall number of GBT sub-indicators and PE indicators included in The abridged pathway tool, distributed per function and maturity level. The distribution of “fully applicable” and “for information only” GBT sub-indicators is also detailed. 6.1.3 Preparing for the abridged assessment: briefing session The members of the WHO team, selected for the abridged assessment from the roster of qualified experts, should be thoroughly briefed on the principles described in this document prior to the start of the activity. The WHO Secretariat should brief all experts remotely as part of preparation for the review. The briefing should include details related to: 1. Context of the abridged assessment including objectives and expected outcomes 2. Methodology of the assessment 3. Availability of required documents 4. Access and utilization of WHO secure information sharing platform 5. Roles and responsibilities of different experts, including specific task(s), and 6. Answers to questions raised and clarifications sought by experts. 6. Overview of the abridged assessment process 241 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Table A6.2. Total number of indicators in the abridged pathway tool: GBT + PE Regulatory function ML3 Mandatory ML4 PE TOTAL For info only To be assessed For info only To be assessed RS Regulatory system 7 3 3 4 4 21 MA Registration and marketing authorization 1 2 1 1 3 8 VL Vigilance 1 5 0 2 7 15 MC Market surveillance and control 0 0 0 0 2 2 LI Licensing establishments 0 0 0 0 0 0 RI Regulatory inspection 0 0 2 1 0 3 LT Laboratory testing 3 1 1 1 1 7 CT Clinical trials oversight 1 1 0 0 3 5 LR RA lot release 1 3 0 0 0 4 TOTAL 14 15 7 9 20 65 Grand TOTAL 45 242 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 6.2 Abridged assessment By default, the abridged assessment is a desk-based activity, conducted remotely. To facilitate the process for abridged assessment, the relevant RA coordinator(s) shall upload to the relevant secure WHO information sharing platform, in advance of the planned abridged assessment, the following documents: 1. relevant national regulations/guidelines, along with any supporting information, or links to the relevant publicly available resources 2. self-assessment against the GBT sub-indicators and PE indicators listed in The abridged pathway tool (see Appendix A6.1). It is essential that relevant RA staff review the respective fact sheets and guidance provided for GBT sub-indicators and PE indicators constituting The abridged pathway tool prior to the preparation of the self-assessment. A meeting between WHO assessors and the relevant RA personnel should be arranged to reach a common understanding of the evaluation process and respond to any key questions from either party. The self-assessment report is evaluated by the WHO team of assessors through a desk-based review of the evidence provided. If needed, WHO assessors may share a list of questions or queries with the relevant RA personnel. Remote meetings can be arranged between the WHO team of assessors and representatives of RA to seek additional clarifications and confirmation. 6.3 PE report The WHO team issue the PE report (in English or bilingual), based on a review of evidence collected. The report includes: a. summary of the conducted activities and conclusions, b. the completed abridged pathway tool (that is, pre-selected GBT sub-indicators and the PE Indicators scorecard provided as Annex 1 to the Manual for the performance evaluation of regulatory authorities seeking designation as WHO-listed authorities). The finalized PE report should be made available to the WHO Secretariat within the agreed upon timeframe. The PE report is submitted to the Technical Advisory Group for WLAs (TAG-WLA), following the standard procedure (see section 7.5 of the Operational guidance for evaluating and publicly designating regulatory authorities as WLA). 7. Roles and Responsibilities 243 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual The abridged assessment should be seen as a collaborative exercise involving the RA, WHO Secretariat, and WHO team. This section is meant to provide guidance on the roles and responsibilities among the aforementioned parties. 7.1 Relevant RA The RA is responsible for: a. Designating one or more focal person to coordinate the abridged assessment related activities. b. Nominating officials for granting them access to the WHO secure information-sharing platform. c. Sharing with WHO, through the secure information sharing platform or any other agreed means, all necessary information and documentations including, among others, national code/regulations/guidelines, relevant procedures, and data specific to the document(s) selected for the assessment. d. Granting WHO team’s access to all relevant data and information throughout the abridged assessment. e. Providing the necessary clarifications and explanations, in response to questions from the WHO Team. f. Seeking and obtaining any necessary consent from any involved stakeholder in order to share the relevant information with WHO. 7.2 WHO Secretariat (WHO headquarters, regional and country offices) WHO headquarters (Regulatory Systems Strengthening Team), in collaboration with WHO regional offices and relevant country offices, is responsible for: a. establishing and maintaining the tools and databases related to abridged assessment b. establishing a roster of qualified experts c. training experts in order to ensure consistency and quality of the process as well as robustness of the assessment outcome d. establishing a dedicated country page on the WHO information sharing platform for the abridged assessment and uploading of all relevant documentation for access and archive purposes e. selection of the WHO team members from the roster of qualified experts to perform the abridged assessment on behalf of WHO. f. organization of any necessary contractual arrangements. 7.3 WHO focal point The WHO focal point is responsible for: a. Leading and coordinating the abridged assessment from the beginning to the end of the process. He/she may or may not participate in the performance evaluation of the RA. b. Briefing the WHO team members on various aspects related to the abridged assessment, including context, background, objectives, process and methodology, roles, and responsibilities. c. Coordinating work among all members of the WHO Team to ensure smooth and consistent completion of the abridged assessment and avoid duplication of effort and/or conflicts. d. Communicating with RA officials on behalf of WHO. e. Delivering the PE report: the overall report of the abridged assessment should ideally be prepared by all WHO team, however the responsibility of delivering the finally agreed report lies on the WHO focal point. 7.4 WHO team members The WHO team members are responsible for: a. reviewing and signing the relevant administrative documents including invitation letter, confidentiality agreement, and declaration of interests form. b. respecting all applicable protocols, ethics, and codes of conduct. 7. Roles and Responsibilities 244 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual c. assessing and evaluating the performance of RAs using the indicators listed in The abridged pathway tool, and provided in Appendix A6.1. d. identifying best practices. e. preparing a detailed report on the assessment conducted (see section 6.5). The PE report should be provided to the WHO within the agreed timeframe. The report may quote the different components/sections in the tool. 7.5 RA participants The RA participants are responsible for: a. establishing and maintaining communication between the WHO Team, and the RA staff b. keeping senior management informed on the abridged assessment c. contributing to the coordination of the abridged assessment d. preparing all materials requested by the WHO Team, if any e. ensuring easy access of the WHO Team to the requested documents, information, and persons f. making themselves available for meetings with the WHO Team, as required g. providing clarifications and explanations, if sought by the WHO Team. 8. Bibliography 245 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 8. Bibliography Manual for benchmarking of the national regulatory system of medical products and formulation of institutional development plans. Geneva: World Health Organization, in press. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021. Indicators and factsheets (https://apps.who.int/iris/handle/10665/341243, accessed 6 April 2023). WHO-listed authority (WLA). Geneva: World Health Organization (https://www.who.int/initiatives/who- listed-authority-reg-authorities, accessed 9 August 2023). 246 The abridged pathway Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 9. Document change history Version no. Date of issue Main changes 1 June 2023 First version 247 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A6.1 The abridged pathway tool: Appendix A6.1 The abridged pathway tool for assessing the performance of stringent regulatory authorities (for medicines) and highly performing regulatory authorities (for vaccines) seeking designation as WHO-listed authorities (WLA) About The abridged pathway tool This document is meant to facilitate the self- assessment and WHO assessment of regulatory authorities seeking designation as WLAs through the use of The abridged pathway tool. The abridged pathway tool has two components: Component 1: pre-selected GBT sub-indicators that primarily target good regulatory practices and quality management system (QMS). Component 2: all performance evaluation (PE) indicators. The RA is requested to provide: a) narratives describing the adopted regulatory practices in response to all GBT sub-indicators and PE indicators, and b) evidence in support of the applied regulatory practices. RAs may not score any of the indicators as “not applicable”. The WHO team of assessors should complete the relevant fields (WHO input and scores) and attach a copy of the completed components to the PE report. Rating WHO uses The abridged pathway tool to determine whether or not the RA can be considered to acceptably meet WLA requirements. For an authority to be designated as a WLA for the scope of the listing being sought, the RA must fulfil the following criteria: (a) achieve full implementation of GBT sub- indicators listed in component 1. (b) acceptably meet the criteria set out for each PE indicator, as part of component 2. Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 248 The abridged pathway Component 1: Pre-selected Global Benchmarking Tool (GBT) sub-indicators Note: sub-indicators intended to be “for information only” are indicated as such in the table below. Regulatory authorities (RAs) are expected to discuss thoroughly and provide comprehensive evidence in support of “for information only” GBT sub-indicators even if they are not subject to WHO assessment. WHO may request additional information, should this be considered necessary to obtain a full understanding of the applied regulatory practices. For full guidance on objectives and evidence to be provided in support of the applied regulatory practices, see the relevant GBT fact sheets for each sub-indicator, which are included in the WHO Global Benchmarking Tool for Evaluation of National Regulatory System of Medical Products (available at https://apps.who.int/iris/handle/10665/341243). Country: __________________ Institution: __________________ Dates: __________________ Assessors: __________________ 01-National regulatory system (RS) ML Scoring Please tick one box RA input For self-assessment WHO assessor input For formal assessment Indicator RS02 Arrangement for effective organization and good governance. Sub-indicator RS02.01 The structure and line of authority among, and within, all institutions that participate in the regulatory system is defined, documented and implemented. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator RS02.03 Scientific and advisory committees exist to advise the NRA on topics of scientific and regulatory interest and on future objectives and strategies. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Indicator RS03 Strategic plan with clarified objective in place Sub-indicator RS03.02 The NRA has established and declared its vision, mission and strategic priorities. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator RS03.05 The NRA is promoting good regulatory practices 4 Input relating to this sub-indicator is collected for information only. 249 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A6.1 The abridged pathway tool: Indicator RS04 Regulatory system is supported with leadership and crisis management plans Sub-indicator RS04.03 A rapid alert and recall system based on documented communication to the appropriate level of the distribution channel and with a feedback mechanism. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator RS04.05 Written criteria to cover circumstances in which the routine regulatory processes may not have to be followed in relation to crises and emergencies linked to a risk management plan. 3 Input relating to this sub-indicator is collected for information only. Indicator RS05 Quality management system (QMS) including the risk management principles are applied and realized Sub-indicator RS05.10 A mechanism to evaluate the satisfaction of internal and external customers and other interested parties is in place for system improvement 4 Input relating to this sub-indicator is collected for information only. Indicator RS06 Human resources to perform regulatory activities. Sub-indicator RS06.04 Documented mechanism to handle potential conflicts of interest for internal and external experts and committee members, to gather declarations of interest and to guarantee the update of these declarations for all regulatory functions. 3 Input relating to this sub-indicator is collected for information only. Indicator RS09 Mechanisms exist to promote transparency, accountability and communication. Sub-indicator RS09.01 The NRA participates in regional and/or global networks to promote convergence and harmonization efforts and expand its collaboration in the regulatory field. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 250 The abridged pathway Sub-indicator RS09.02 The information on laws, regulations guidelines and procedures is publicly available and is kept duly updated. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator RS09.03 Information on decisions related to regulatory activities is available to the public. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator RS09.04 Information on marketed medical products, authorized companies and licensed facilities is publicly available. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator RS09.05 All publicly available information is periodically reviewed and maintained. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator RS09.06 Appropriate mechanisms exist for management of confidential information. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator RS09.07 A code of conduct, which includes management of conflicts of interest, is published and enforced for internal and external staff, including members of the advisory committees. 3 Input relating to this sub-indicator is collected for information only. Indicator RS10 Mechanism in place to monitor regulatory performance and output. Sub-indicator RS10.01 Requirements established to monitor, supervise and review the performance of the NRA and affiliated institutions using key performance indicators (KPIs). 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator RS10.02 Reports on the regulatory activities and on the progression and status of resources are available at regular intervals. 4 Input relating to this sub-indicator is collected for information only. 251 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A6.1 The abridged pathway tool: 02-Registration and marketing authorization (MA) ML Scoring Please tick one box RA input For self-assessment WHO assessor input For formal assessment Indicator MA01 Legal provisions, regulations and guidelines required to define regulatory framework of registration and/or marketing authorization. Sub-indicator MA01.12 There are established guidelines that cover circumstances under which the routine MA procedures may not be followed (e.g., for public- health interest). 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Indicator MA04 Procedures established and implemented to perform registration and/or marketing authorization Sub-indicator MA04.05 An advisory/scientific committee, including external experts is involved in the review of MA applications as necessary. 4 Input relating to this sub-indicator is collected for information only. Sub-indicator MA04.10 The regulations and/or guidelines for good review practices are developed or recognized and implemented. 3 Input relating to this sub-indicator is collected for information only. Indicator MA05 Mechanism exists to promote transparency, accountability and communication. Sub-indicator MA05.02 Updated list of all medical products granted MA is regularly published and publicly available. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator MA05.03 A summary technical evaluation report for approved registration marketing authorization applications is published and available to the public. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 252 The abridged pathway 03-Vigilance (VL) ML Scoring Please tick one box RA input For self-assessment WHO assessor input For formal assessment Indicator VL02 Arrangement for effective organization and good governance. Sub-indicator VL02.02 Documented procedures and mechanisms are implemented to ensure the involvement, coordination and communication among all stakeholders relevant to vigilance activities. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Indicator VL04 Procedures established and implemented to perform vigilance activities. Sub-indicator VL04.01 Vigilance procedures and tools are in place and implemented for collection and assessment of adverse drug reactions and AEs. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator VL04.02 Vigilance procedures and tools are in place for investigation, interpretation of and response to adverse drug reactions and AEs. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator VL04.04 Risk approach is considered throughout different vigilance activities, including timely response to detected signals for risks or benefits. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator VL04.07 With respect to vigilance data, assessment of the risk-benefit balance of medical products is regularly conducted. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator VL04.08 Active vigilance activities, as well as proactive monitoring programmes (when needed) have been developed and implemented. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented 253 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A6.1 The abridged pathway tool: Indicator VL05 Mechanism in place to monitor regulatory performance and output. Sub-indicator VL05.01 Vigilance information is used in timely manner to amend existing regulatory decisions or to issue new regulatory decisions or actions. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Indicator VL06 Mechanism exists to promote transparency, accountability and communication Sub-indicator VL06.02 Mechanism for regular feedback to all stakeholders on vigilance events exists and is complemented with a risk communication plan. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented 06-Regulatory inspection (RI) ML Scoring Please tick one box Justification by RA For self-assessment WHO assessor input For formal assessment Indicator RI05 Mechanism in place to monitor regulatory performance and output. Sub-indicator RI05.03 Inspection reports are subjected to a regular and robust review by experts other than the designated inspection team. 4 Input relating to this sub-indicator is collected for information only. Sub-indicator RI05.04 Inspection data and outcomes are systematically evaluated or interpreted. 4 Input relating to this sub-indicator is collected for information only. Indicator RI06 Mechanism exists to promote transparency, accountability and communication. Sub-indicator RI06.02 The updated list or database of all inspected facilities along their regulatory decisions, actions and enforcement activities, is regularly published and publicly available. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 254 The abridged pathway 07-Laboratory testing (LT) ML Scoring Please tick one box Justification by RA For self-assessment WHO assessor input For formal assessment Indicator LT03 Laboratory activities implemented as per well-established plans and policies according to a Quality Management System (QMS) Sub-indicator LT03.01 Documented and implemented policy for testing exists that is based on the product’s risk. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator LT03.02 Documented and implemented policy exists on the validation, verification and transfer of analytical procedures. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator LT03.03 A policy is in place to establish and/or qualify all reference standards used in laboratory testing activities 3 Input relating to this sub-indicator is collected for information only. Indicator LT08 Mechanism in place to monitor regulatory performance and output. Sub-indicator LT08.01 There is an updated database of all medical products batches that have undergone quality testing. 4 Input relating to this sub-indicator is collected for information only. Sub-indicator LT08.02 Monitoring and trend analysis are carried out for laboratory testing results data of reference materials and medical products. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Sub-indicator LT08.03 Regular participation in proficiency schemes, collaborative studies and inter-laboratory comparisons. 4 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented 255 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A6.1 The abridged pathway tool: 08-Clinical trials oversight (CT) ML Scoring Please tick one box Justification by RA For self-assessment WHO assessor input For formal assessment 02 Arrangement for effective organization and good governance Sub-indicator CT02.02 Documented procedures are implemented to ensure the involvement and communication between all stakeholders relevant to CT 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Indicator CT04 Procedures established and implemented to perform clinical trials oversight. Sub-indicator CT04.04 There are defined roles for ECs at all levels (for example, national, subnational, or institutional). 3 Input relating to this sub-indicator is collected for information only. 09-NRA Lot release (LR) ML Scoring Please tick one box Justification by RA For self-assessment Justification by WHO team For formal assessment Indicator LR04 Procedures established and implemented to perform NRA lot release. Sub-indicator LR04.02 NRA or national control laboratory staff involved in lot release have access to marketing authorization relevant files and updates. 3 Input relating to this sub-indicator is collected for information only. Sub-indicator LR04.03 Analysis of lot-to-lot consistency is conducted. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual 256 The abridged pathway Indicator LR05 Mechanism for information-sharing exists to promote transparency and accountability. Sub-indicator LR05.02 Follow-up and communication with involved parties, including the manufacturer, on issues of data quality. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Indicator LR06 Mechanism in place to monitor regulatory performance and output. Sub-indicator LR06.03 Regulatory action taken in case of products non-compliance. 3 ☐ Not implemented ☐ Ongoing implementation ☐ Partially implemented ☐ Implemented Overall conclusion and recommendation by the WHO team: Please provide an overall conclusion of the GBT sub-indicators, considering the guidance provided under relevant factsheet, and any consequent recommendation. 257 Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 PE manual Appendix A6.1 The abridged pathway tool: Component 2: Performance evaluation (PE) indicators All PE indicators are included in The abridged pathway tool. The inputs by the RA, as well as the scores and justifications by the WHO team, should be reported in the PE indicators scorecard (see Annex 1 of the PE manual). Considering the principle of “hierarchy” (see section 6.1.1 above), RAs are expected to discuss thoroughly and provide comprehensive evidence in support of each PE indicator, in order to enable proper understanding and assessment of the regulatory practices in place. Full guidance on objectives and evidence that should be provided in support of the applied regulatory practices can be found in the relevant PE “fact sheets” (see Tables 2-25 of the PE manual). When preparing the self-assessment entries, consideration should be given to the requirements of the GBT sub- indicators mentioned in the “references” section of the relevant PE indicator table.

Regulation and Prequalification department World Health Organization 20 Avenue Appia 1211 Geneva 27 Switzerland https://www.who.int/teams/regulation- prequalification/