Evidence generation for development of health products: a practical guide for WHO staff

A practical guide for WHO staff Evidence generation for development of health products

A practical guide for WHO staff Evidence generation for development of health products Evidence generation for development of health products: a practical guide for WHO staff ISBN 978-92-4-008429-2 (electronic version) ISBN 978-92-4-008430-8 (print version) © World Health Organization 2023 Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization (http://www.wipo.int/amc/en/mediation/rules/). Suggested citation. Evidence generation for development of health products: a practical guide for WHO staff. Geneva: World Health Organization; 2023. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at https://iris.who.int/. Sales, rights and licensing. To purchase WHO publications, see https://www.who.int/publications/book-orders. To submit requests for commercial use and queries on rights and licensing, see https://www.who.int/copyright. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. Design by L’IV Com Sàrl iii Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Purpose of the document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Structure of the document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. Principles in evidence generation to inform public health recommendations . . . . . . . . . . . . . . . . . . . . . . 3 2.1 Assessing available evidence: WHO’s process for developing public health recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Producing the best evidence: basic principles of evidence generation . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Intended public health use of the end-product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3. Content of a guidance on evidence generation (GEG) document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1 Desired indication/intended use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Choice of trial population and study setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3 General study design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.4 Choice of endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.5 Suitable reference standards/comparators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.6 Clinical case definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.7 Case ascertainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.8 Sample size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.9 Statistical analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.10 Specific product-related issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.11 Special/vulnerable populations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.12 Other GRADE criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.13 Post-licensure studies, post-market surveillance and pharmacovigilance considerations . . . . . 17 3.14 Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.15 WHO prequalification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.16 Ethical aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.17 Role of this guidance in relation to WHO’s Coordinated Scientific Advice (CSA) procedure . . . . 20 iv Evidence generation for development of health products: a practical guide for WHO staff  4. Suggested procedure to develop guidance on evidence generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Useful websites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Annex 1: Respective prequalification product streams to encourage early contact with manufacturers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 vAcknowledgements This guidance was produced by the Emerging Technologies, Research Prioritization and Support (EPS) unit in the Research for Health Department, Science Division of the World Health Organization (WHO). This document is based on an initial draft by Vasee Moorthy. The development of this current version was coordinated by Sarah Charnaud and Christian Lienhardt (consultant), with support from Mercedes Perez Gonzalez and Blessing Amarachi Ezeocha under the guidance of Anna Laura Ross and Vasee Moorthy. Steering committee A steering committee of experts in evidence generation and use at WHO were instrumental in the conceptualization, development and review of the document. These were Lisa Askie (Quality, Norms and Standards Department), Vasee Moorthy (Research for Health Department), Deusdedit Mubangizi (Regulation and Prequalification Department) and Anna Laura Ross (Research for Health Department). WHO contributors Acknowledgements are due to the many WHO staff who contributed their time and expertise in providing insights, experiences and reviewing drafts. They are in alphabetical order: Rachel Baggaley (Global HIV, Hepatitis and STIs Programmes); Neerja Chowdhary (Mental Health and Substance Use); Joao Paulo De Souza (Maternal, Newborn, Child & Adolescent Health & Ageing); Karen Edmond (Maternal, Newborn, Child & Adolescent Health & Ageing); Amarachi Ezeocha (Research for Health); Geraldine Foster (Regulation and Prequalification); Nathan Ford (Global HIV, Hepatitis and STIs Programmes); Dongbo Fu (Health Promotion); Birgitte Giersing (Immunization, Vaccines and Biologicals); Kratu Goel (Health Systems Governance and Financing); Hebe Gouda (Health Promotion); Benedikt Huttner (Health Product Policy and Standards); Jan Kolacinski (Global Malaria Programme); Tanja Kuchenmuller (Research for Health); Olivier Lapujade (Regulation and Prequalification); Filip Meheus (Health Systems Governance and Financing); Vittal Mogasale (Health Systems Governance and Financing); Lorenzo Moja (Health Product Policy and Standards); Francis Moussy (Health Product Policy and Standards); Irena Prat (Regulation and Prequalification); Dominic Schuler (Regulation and Prequalification); Pura Rayco Solon (Quality Assurance, Norms and Standards); Matthias Stahl (Regulation and Prequalification); Kim Rok Ho (Quality Assurance, Norms and Standards); Jamie Rylance (Country Readiness Strengthening); Samuel Schumacher (Global Tuberculosis Programme); Rebekah Thomas Bosco (Quality Assurance, Norms and Standards); Marie Valentin (Regulation and Prequalification); and Lara Vojnov (Global HIV, Hepatitis and STIs Programmes). External experts WHO thanks the following experts who contributed expertise and experiences: Roger Chou, Oregon Health and Science University, Portland, United States of America; Claudia Denkinger, Heidelberg University Hospital (UKHD), Heidelberg, Germany; David Schellenberg, London School of Hygiene & Tropical Medicine, London, United Kingdom. Declarations of interest External contributors signed the WHO Declaration of Interest which were reviewed by the Emerging Technologies, Research Prioritisation and Support Team. Dr Claudia Denkinger declared receiving research support by means of grants, collaborations, sponsorships and other funding from nonprofit and government- run research organizations with an interest related to the subject of the work. The interests declared were assessed and deemed to be non-specific and did not constitute a conflict of interest. vi Abbreviations and acronyms AEs Adverse events APIs Active pharmaceutical ingredients ART Antiretroviral therapy BE Bioequivalence CONSORT Consolidated Standards of Reporting Trials CSA Coordinated scientific advice DST Drug susceptibility ECVP Evidence Considerations for Vaccine Policy EQUATOR Enhancing the Quality and Transparency Of health Research EVIPNet Evidence-informed Policy Network FPPs Finished pharmaceutical products GCP Good Clinical Practice GEG Guidance for evidence generation GLP Good Laboratory Practice GMP Global Malaria Programme GRADE Grading of Recommendations, Assessment, Development and Evaluations ICH International Council for Harmonisation of technical requirements for pharmaceuticals for human use ICP Immune correlates of protection IRB Institutional Review Board IVB Immunization, Vaccines and Biologicals IVDs In vitro diagnostics LMIC Lower-Middle-Income Countries viiAbbreviations and acronyms MedDRA Medical Dictionary for regulatory activities NRA National Regulatory Authorities NSP Normative and standard-setting products NTD Neglected Tropical Diseases PPCs Preferred Product Characteristics PQ Prequalification PQTm Prequalification Team medicines PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses PSFs Product summary files R&D Research and Development RCTs Randomized control trials REC Research Ethics Committee SAEs Serious adverse events SAGE Strategic Advisory Group of Experts on Immunization SRQR Standards for Reporting Qualitative Research STARD Standards for Reporting of Diagnostic Accuracy Studies TB Tuberculosis TGS Technical Guidance Series TPP Target Product Profile TRPs Target Regimen Profiles TSS Technical Specifications Series VCAG Vector Control Advisory Group WHA World Health Assembly WHO World Health Organization viii Purpose of the document This document describes the main elements that World Health Organization (WHO) technical departments should elaborate on when providing guidance on evidence generation to research and development (R&D) specialists who are developing health products to address critical unmet needs in public health. Structure of the document This guidance outlines the justification, considerations and concrete steps to develop guidance to researchers and developers on the evidence needed to inform recommendations on new products for public health. Section 1 sets out the background and justification to provide guidance on evidence needs. Section 2 provides principles and an overview of the route to developing guidance. Section 3 outlines the content to include in an evidence guidance document to be used by researchers and developers. Section 4 suggests a process by which a WHO technical department may develop evidence generation guidance.t 11Background WHO’s Science Division is developing optimized processes to enhance WHO activities in research prioritization, including target product profile (TPP) development and evidence generation. There is an urgent need to develop high-quality, effective health products for infectious and noncommunicable diseases in order to address the most compelling medical needs worldwide, but much of the research and development pipeline still focuses on the concerns of high-income countries. In some cases, where products are already available in high-income settings, additional data are required to guide decision-making on use in lower-income and lower-middle-income countries (LMICs). In other instances, entirely new health products are needed for conditions that are focused on lower-income settings. The Science Division supports WHO staff to develop cohesive guidance on the generation of evidence related to promising health products in order to increase the chances of positive outcomes from WHO guidelines and prequalification (PQ) processes. While WHO’s TPPs, preferred product characteristics (PPCs) and target regimen profiles (TRPs) provide high-level information on the desired characteristics of the product, the target population, and the key endpoints for safety and efficacy evaluation, more detailed guidance is needed to enable the development of suitable trial/study designs aimed at producing relevant evidence for WHO’s normative products such as guidelines. The absence of evidence may lead to unnecessary delays in guidelines or prequalification of specific health products and – potentially – to deaths, illness and financial consequences for persons and governments when global access is stalled. Such delays may also de-incentivize industry engagement in public-interest research and development, and may lead to additional costs for those funding public-interest developers. Therefore, providing the developers of health products with effective guidance on the technical information required to design pivotal clinical efficacy/validation trials with global access considerations in mind would assist the production of suitable evidence for WHO recommendations. Where a number of developers or funders are allocating resources to develop products in particular areas, it is more time- and resource- effective to provide guidance and feedback early in the process, avoiding the need for additional, costly and time-consuming studies. While the primary purpose of clinical trials is to obtain sufficient data on safety and efficacy to support licensure or authorization of a medical product a document that provides additional guidance to enable a smoother pathway to WHO guideline recommendations and WHO PQ is highly desirable. WHO has provided guidance on evidence generation in many areas – including on general vaccines (1), malaria and dengue vaccines (2), in vitro diagnostics (TSS and TGS)1,2, and COVID-19 therapeutics and vaccines (3). The guideline development process also identifies which research is needed to guide future guidelines. A standard procedure will provide assurance of good practice to other departments that wish to generate such guidance and pre-empt the evidence needed to inform normative recommendation processes. 1 The Technical Specifications Series (TSS) sets out the minimum validation and verification studies to be undertaken by the manufacturer in support of in vitro diagnostic (IVD) performance claims (See: https://extranet.who.int/pqweb/vitro-diagnostics/technical-specifications-series, accessed 30 May 2023). 2 The Technical Guidance Series (TGS) identifies standards and guidance that contain valuable information on a range of issues that are encountered in the manufacture, verification and validation of specific categories of IVD. The series should be read in conjunction with relevant international and national standards and guidance. (See: https://extranet.who.int/pqweb/vitro-diagnostics/technical-guidance-series, accessed 30 May 2023). 2 Evidence generation for development of health products: a practical guide for WHO staff  1.1 Scope This guidance highlights overarching areas in which WHO could specify the types of evidence required and the means by which evidence can be generated for a specific health product. The document that is to be produced on “Guidance for evidence generation” (GEG) will focus on the production of optimal quality evidence to inform decision making. The document will include elements that can enhance data in order to assess the balance of benefits and harms, as well as acceptability, feasibility and equity in line with WHO’s development of public health recommendations/guidelines as outlined in the WHO Handbook for guideline development (4). The procedure aims to identify the technical information required to design pivotal clinical efficacy or validation trials with global access considerations in mind for therapeutics, vaccines, diagnostics and vector control measures. By increasing clarity regarding the evaluation needs for products it is hoped to support harmonized evidence generation and to facilitate the WHO review process towards meeting the WHO goal of health for all. The objective of the Guidance for evidence generation (GEG) process is to produce a guidance document on how best to perform studies for the development of novel products (diagnostics, drugs, vaccines, vector control) and thus how to enable the generation of robust data and evidence to inform a WHO evidence review process. The ultimate goal is to avoid the situation whereby the limitations in design, execution and reporting of trials and studies lead to a low certainty of evidence about the true product performance, which can delay or impede decisions on policy and scale-up. 32Principles in evidence generation to inform public health recommendations 2.1 Assessing available evidence: WHO’s process for developing public health recommendations WHO has a core normative function: stakeholders look to WHO for guidance and standards on a broad range of health-related topics. In an environment of rapidly changing needs and contexts, and with the continuous emergence of new data, information and research evidence, WHO must provide trustworthy, up-to-date and impactful normative guidance. Scientific and technical normative and standard-setting products (NSPs) give guidance, inter alia, on what to do, how to carry out an action, how to measure or assess, and how to determine knowledge gaps or areas of uncertainty based on scientific evidence and advice from leading technical experts. NSPs aim to have an anticipated positive impact on health, individually or collectively, with positive implications for the use of resources. The process by which WHO produces public health guidelines and recommendations is one type of normative product and is described in the WHO Handbook for guideline development (4). In brief, the evidence for guideline development is assessed using the GRADE methodology, in order to systematically judge the quality of a body of evidence and estimate the strength of the recommendation that is to be derived from that evidence (5). Evidence is graded on the basis of study design, risk of bias, imprecision, inconsistency, and other considerations such as publication bias. Detailed criteria that are considered include the quality of the evidence, the balance of benefits and harms and the resource implications, as well as patients’ values and preferences, equity, acceptability and feasibility. An evidence-to-decision framework is generated to inform the specific WHO recommendation and to determine its direction and strength (6). The procedure for developing WHO normative statements across WHO departments is currently being updated on the basis of three high-level approaches, as follows: Better anticipate: This applies to activities that build up to and trigger the process for developing WHO recommendations – including horizon-scanning and developing or endorsing PPCs and TPPs – in order to stimulate innovation, guide product development and provide predictability to manufacturers with respect to the anticipated evaluation process for these new tools. Better develop: This applies to activities to develop WHO recommendations, including anticipating and advising on the generation of evidence by manufacturers and/or research groups to demonstrate that an intervention has a public health value; the assessment of these data by the relevant WHO advisory groups; and the formulation of recommendations by WHO. Optimize uptake: Once WHO guidance and recommendations are disseminated, their use and uptake should be monitored. These process enhancements are expected to enable WHO to identify and communicate unmet public health needs, to develop recommendations through an open and transparent process with shortened timelines, and to improve uptake. The present GEG document takes place within this endeavour. This document aims to promote the development of appropriate evidence that can inform normative products. 4 Evidence generation for development of health products: a practical guide for WHO staff  2.2 Producing the best evidence: basic principles of evidence generation This guidance is intended to be used by WHO technical departments wishing to provide guidance on evidence generation (e.g. for a class of products which is publicly available and can be used by any product developer, funder, government and others who would benefit from being aware of WHO preferences for data generation). This guidance is applicable where multiple product developers may engage and an open process is warranted to discuss key needs for data generation. This guidance can be read in conjunction with the WHO handbook for the development of normative products, with an overview in Volume III, Chapter 1: Basis for WHO normative statements, definitions and standards and Volume III, Chapter 2: Best evidence. Figure 1 illustrates the process for developing guidance on evidence generation. Fig. 1. Roadmap indicating steps and considerations when developing guidance on evidence generation to support recommendations on new public health products 52. Principles in evidence generation to inform public health recommendations Documents outlining the various components of clinical evaluation have been developed by various technical departments at WHO (e.g. IVB, GMP, NTD etc.), mostly focusing on regulatory-directed criteria (e.g. Guidelines on clinical evaluation of vaccines: regulatory expectations) (1). Some technical departments have gone further and have produced documents integrating technical, regulatory and policy development requirements – such as the joint GMP/NTD/PQ document on Norms, standards and processes underpinning development of WHO recommendations on vector control products (7), or the IVB document From vaccine development to policy: a brief review of WHO vaccine-related activities and advisory processes (8). Capitalizing on, and drawing lessons from, these documents and the expertise developed, some general principles can be developed to frame WHO’s GEG process, as follows: 1. Developing public health recommendations for a product requires more evidence than regulation of a product for introduction of health products into the market. Regulatory licensure alone is insufficient for policy and deployment. For WHO, international policy-making involves consideration of additional elements such as cost-effectiveness, programmatic fit and performance against outcomes that may not have been quantified during clinical trials for regulatory approval (such as the impact of a drug or vaccine on disease transmission at population level), as well as the capacity and feasibility of the product to be produced and taken up at population level. Product developers/manufacturers and funders need clarity on what data are needed to position a product for policy consideration that will ensure its use and scale-up and guarantee a return on investment. 2. It is important to contextualize the need for evidence generation. WHO technical departments should conduct appropriate scoping reviews and research gap analyses to map existing evidence and identify specific needs for further evidence. For instance, such reviews and analyses should provide clear indications on whether there is a need to gather evidence on new products for which existing evidence is thin, or whether evidence is available in the context of an existing guideline but conditions are changing and new data need to be assembled to update the recommendation. This has important consequences in terms of methods to be used to generate the requested evidence – such as the choice of comparator. In practical terms, it may be useful to perform a review of reviews or a systematic review to uncover the current best practices and gaps for each of the evidence headings to be included prior to starting a GEG document. (For further information, consult the WHO handbook for the development of normative products (9). Volume I, Chapter 3: How to develop WHO normative products: an overarching approach, and Volume III, Chapter 9: Keeping WHO normative products up-to-date. These documents are currently under development and accessed via the WHO intranet at https://intranet.who.int/homes/mst/qahandbook/). 3. Creating high-quality evidence on clinically relevant outcomes for population health. GEG represents the current understanding of what is likely to be important for global policy recommendations for a specific intervention or strategy, to inform policy and to ensure a product’s uptake and scale-up. GEG aims to serve as a foundation for future discussion between developers, manufacturers, funders and WHO, and cannot be considered as a formal WHO guideline at any point. Developers/manufacturers should be clear that producing evidence as requested does not exempt products from undergoing the independent evidence review process that is required for all products seeking WHO policy recommendation. 4. In addition to listing the required evidence, the GEG document should clearly mention the preferred methods to be used for producing this evidence where desirable. The GEG document should be considered as a methodological document that clarifies the required type and hierarchy of evidence and describes the best methods to be used to prevent (high) uncertainty of endpoints. Too often, the available evidence base for recommendations is not of sufficient quality to make strong recommendations. 6 Evidence generation for development of health products: a practical guide for WHO staff  Consequently, the GEG document should clearly address the following: • production of clinically relevant outcomes – in the absence of these outcomes, only previously validated surrogates can be used that are expected to show clinical relevance to allow public health decisions; • representativeness of the studies for generalizability – the study population and environment (including infrastructure) should be as similar as possible to the target population and intended setting for use; • that all data are collected using the best clinical and laboratory practice, as recommended by the International Council for Harmonisation of technical requirements for pharmaceuticals for human use (ICH) (10); • the use of the appropriate control or reference groups, as needed, and for diagnostics the use of appropriate reference methods, specimen-handling and characterization methods. In this respect, technical departments should see their guidance on evidence generation within the larger context of World Health Assembly resolution WHA75.8 Strengthening clinical trials to provide high-quality evidence on health interventions and to improve research quality and coordination, adopted in 2022, recognizing that “well-designed and well-implemented clinical trials are indispensable for assessing the safety and efficacy of health interventions”3. Technical Departments should refer to recently developed resources such as the Guidance for good randomized clinical trials (11) that aims to support individuals and organizations involved in the planning, conduct, analysis, interpretation, funding and oversight of trials, and the recommendation to researchers outlined in Clinical research in resource-limited settings (12). 5. The GEG document should clearly deal with quantitative and qualitative data related to the body of evidence needed for recommendation. The traditional hierarchy of evidence places randomized control trials (RCTs), and systematic reviews of RCTs, at the top of the evidence quality ladder. Nevertheless, the “usability” of the expected intervention is also a key element, and patients’ values and preferences contribute substantially to the evidence required for policy-making. Therefore, the GEG document should include sections on the production of relevant evidence for assessing values and preferences. The GEG document should also clearly indicate the need for modelling studies to estimate the expected public health and cost impact of the product, including cost-effectiveness analyses, and should clarify their contribution to the evidence review. For further information, consult: 1) WHO handbook for the development of normative products, Volume III, Chapter 6: Public hearing in the development of WHO normative products; 2) Lessons from the integration of citizen perspectives in policy deliberations, in: Implementing citizen engagement within evidence-informed policy-making: an overview of purpose and methods (WHO, 2022) (13); 3) useful tips on engagement with people (e.g. enabling participatory spaces in community workshops and stakeholder meetings) in Voice, agency, empowerment – handbook on social participation for universal health coverage (WHO, 2021) (14); and 4) WHO handbook for the development of normative products, Volume IV, Chapter 8: Use of mathematical modelling studies for WHO normative products. 3 See: https://apps.who.int/gb/ebwha/pdf_files/WHA75/A75_R8-en.pdf (accessed 30 May 2023). Of note, throughout the resolution “well-designed trials” refers to trials that are scientifically and ethically appropriate. For submission to medical product regulatory authorities, trials should adhere to International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use guidelines and some Member States may consider International Coalition of Medicines Regulatory Authorities guidelines. In order to generate evidence that is sufficiently robust to support decision-making, such as widespread use of therapeutics or preventives, trials should be designed, conducted, analysed and reported appropriately. 72. Principles in evidence generation to inform public health recommendations 6. PPCs, TPPs and TRPs are propitious points of entry for GEG although not essential. WHO technical departments can initiate a process to define the relevant method(s) and study design(s) that would be required to produce the expected evidence in line with the attributes listed in a TPP or similar document. An example is the Guidance for the evaluation of tuberculosis diagnostics that meet the World Health Organization (WHO) target product profiles produced for the development of TB diagnostics that directly followed the publication of several TPPs for TB diagnostics (15). The Global Malaria Program included clinical development considerations within the document describing the preferred product characteristics of the newest malaria vaccines (16). This requires that PPCs, TPPs and TRPs should be carefully developed, and that the definition of attributes and subsequent requirements should be mindful of the realities of scientific developments – e.g. without fixed targets that would be difficult to attain and may end up as disincentives. 7. It is expected that GEG documents will be reviewed and potentially revised regularly – e.g. 24–36 months after publication – in order to integrate new developments and ensure continued stakeholder alignment. In this respect, GEG revision may benefit from interaction with national regulatory authorities to take into account new developments in study design and methods or new information on research aimed at improving programme integration strategies. If not reviewed GEG documents should be considered expired 5 years after publication date. 2.3 Intended public health use of the end-product As part of use case guidance, it is critical to liaise with end-users, disease programme managers and policy- makers to assess how the envisaged health product may be integrated into existing health systems and control programmes. It is imperative that new medical products should provide added value to public health programmes. Therefore, the guidance should outline considerations for the design of pivotal clinical trials or studies that can provide relevant information to assess the suitability of new products for integration into existing health programmes. In this respect, it is important to relate to the work of the Evidence-informed Policy Network (EVIPNet) that embraces cutting-edge approaches to knowledge translation for better health policy-making4. 2.3.1 Vaccines As mentioned in the WHO guidelines on clinical evaluation of vaccines (1), “vaccination strategies depend on the magnitude and duration of protection provided by the vaccine and how this is thought to interact with the epidemiology and transmission of the pathogen under consideration together with pre-existing naturally acquired population immunity. As a rule, the use case of any vaccine should meet the most pressing public- health needs of disease-endemic countries….Consideration should be given as to whether the primary focus for use is on addition of a vaccine to routine immunization programmes, potentially supplemented by initial catch-up campaigns, or for mass vaccination campaigns. Other possible use cases include adolescent immunization, or reactive vaccination in the context of epidemics, among others.” 2.3.2 Therapeutics For medicines, consideration should be given to the clinical circumstances in which the medicine will be administered, and the stage of the disease at which the medicine is effective. For some indications, the desired benefit can accrue only if the medicine is administered in community or out-patient settings such as in mass drug administration efforts. For other indications, in-patient administration is more appropriate. 4 See: https://www.who.int/initiatives/evidence-informed-policy-network (accessed 30 May 2023). 8 Evidence generation for development of health products: a practical guide for WHO staff  The endpoints chosen should reflect those that are appropriate for the setting of the administration. There may or may not be specific subgroups of users for which the therapeutic is intended – e.g. whether the therapeutic is intended for adults, adolescents, pregnant and lactating women, children and infants. There may be specific use-cases of interest, such as for chronic versus acute disease, for management of risk factors, or for prevention of disease, including pre- or post-exposure prophylaxis. The use of combination therapy of more than one medical product may be explicitly desired in order, for instance, to avoid drug resistance, enhance response or reduce side-effects. 2.3.3 Diagnostics For diagnostics, consideration should be given to the level of the health system at which the diagnosis is to be used, the treatment possibilities that are available to address the disease, and the epidemiology and geographical distribution of the condition to be detected. More particularly, one should consider the operational characteristics of a diagnostic test (e.g. the time taken to perform the test, its technical simplicity or ease of use, and user acceptability), the connectivity solutions, the infrastructure required and the training needs (17). All these are important for translation of global policy into actionable implementation plans at the country level (18). The intended use and test purpose will determine the level and type of evidence that will need to be collected to support a regulatory and/or WHO PQ submission and the development of a policy recommendation. Data required to support the validation of an in vitro diagnostic device will be different, for instance, if the device is intended to be used for surveillance purposes or as a diagnostic tool for clinical management (19). Furthermore, the manufacturer’s validation and verification studies should be tightly linked to the risk management. 2.3.4 Vector control products WHO has developed a document describing the norms, standards and processes underpinning the development of WHO recommendations for vector control interventions (7). The document also includes high-level information on the PQ process, which is complementary to and coordinated with the development of WHO recommendations. Evidence generation should meet the criteria set out in these documents, with consideration given to context of use and evaluation of effectiveness. 93Content of a guidance on evidence generation (GEG) document It is proposed that the GEG document outlines the core technical information needed in general for the development of vaccines, therapeutics, diagnostics and vector control products, and then addresses the specific needs of each of these types of health products. The document will address the following elements: desired indication and intended use; choice of trial population and study setting; overall study design(s); endpoint selection: efficacy, safety, other endpoints; suitable reference standard/ comparators; clinical case definition; case ascertainment; sample size; statistical analysis; Core specific product-related issues: vaccines, diagnostics, therapeutics, vector control products; special/vulnerable populations; other GRADE criteria: patients’ values and preferences, acceptability, feasibility and equity, estimation of public health impact, cost and cost-effectiveness; post-licensure aspects, post- market surveillance and pharmacovigilance; reporting; WHO prequalification; and annexes (as needed). Optional Note: For therapeutics and vaccines, the document will focus on the definition and measurement of efficacy endpoints, evaluation of safety/tolerability, duration of follow-up and, for vaccines, the need for and role of immune correlates of protection, as well as the expected duration of protection. The document will focus primarily on clinical studies to be conducted in patients and the intended population, including consideration for early clinical trials in healthy volunteers. For diagnostics, the focus will be on study design for the measurement of accuracy and validity, as well as determination of operational characteristics 10 Evidence generation for development of health products: a practical guide for WHO staff  3.1 Desired indication/intended use Guidance should be provided on critical indications or test purposes which WHO considers the data submitted should be able to support. In the case where the guidance on evidence generation complements an existing PPC, TPP or TRP document, this will follow the indications/intended use already noted. 3.2 Choice of trial population and study setting Participants should ideally include a representative mix of the geographical areas and target populations presenting the highest disease burden. The guidance should specify whether the target group is global or if particular regions or countries should be targeted. Priority subgroups for data generation – such as infants, children, adolescents, pregnant and lactating women, women of childbearing age, older persons, or those who are HIV-infected or immunocompromised – should be specified where appropriate. The choice of the target population and study settings should ensure the broad generalizability of data produced. For regulatory purposes, developers should include any population that will be required to allow them to support their claims. 3.3 General study design The core of the study design guidance should be based on the outcome desired and the ability to make policy decisions on the evidence generated; it may not be necessary or desirable to specify the methodology. If it is desirable to specify study design it should be clear whether RCTs or other designs are envisaged for testing the product in the planned population. According to WHO’s Handbook for guideline development, for questions on the benefits and harms of interventions “high-quality randomized controlled trials (RCTs) provide the highest qual¬ity evidence with regard to causality and potential confounding” to allow for “like with like” comparisons (16). Non-randomized study designs including experimental designs (e.g. quasi- randomized trials and investigator-assigned cohort studies) as well as observational studies (e.g. before– after or paral¬lel-group cohort studies or surveillance data) can be included in the evidence to inform of benefits and harms. In indicating study design, the comparator group (placebo, other test/drug/vaccine) should be clearly indicated (including whether it should be a non-inferiority or superiority design) – with appropriate methodological considerations to substantiate the selection of design. In some cases, additional bioequivalence studies or immune correlate-based studies should be considered. These should be clarified in the guidance together with their rationale. In the case of therapeutics large-scale, adaptively designed platform trials can in some instances represent superior options for evidence generation due to their ability to include multiple interventions concurrently, answer policy-relevant questions quickly and pivot from one disease area to another as local priorities change – in addition to being faster and often less costly than other designs (20). With adaptive designs, researchers can monitor the incoming data and adapt the protocol on the basis of pre-established criteria as the study unfolds (e.g. by dropping or adding doses, adapting the size or duration of a trial, or enriching the study population by adding more of the types of patients who respond to the treatment being investigated) (17). Adaptive designs can benefit investigators as well as patients because the sooner it is proved that a drug or a dose of a drug either works or not, the sooner that drug can either be advanced or its evaluation in patients stopped. However, adaptive designs require much greater attention from statisticians and data safety monitoring committees, as these designs rely on rapid analysis and data transmission. It is recommended that WHO’s guidance for evidence generation should recognize this evolving change and support public health and policy-oriented platform trials. 113. Content of a guidance on evidence generation (GEG) document In the case of diagnostics, evidence has traditionally relied on accuracy studies. However, it is important to include both clinical impact and accuracy. The study design should include the clinical impact of having the intervention – for instance, does having the new test lead to more people being diagnosed and linked to treatment? For products which include an element of digital health, users should consult WHO’s guidance in Monitoring and evaluating digital health interventions (21) (WHO, 2016) – particularly Table 4.5 on study design and method considerations. 3.4 Choice of endpoints 3.4.1 Efficacy A key aspect of GEG is clear definition of the endpoints or outcomes that will be considered to be of relevant public health value by WHO in later-stage trials – on treatment, vaccines or diagnostics – in order to drive policy recommendations. In general, clinical cases of the disease of interest (to be diagnosed, treated or prevented) are considered appropriate endpoints of public health relevance while also being more frequent than severe cases or deaths, and thus it may be easier to detect an effect in reasonably sized trials. In the case of multiple possible endpoints or outcomes they should be prioritized in a transparent manner. Depending on the context, the GEG document should also provide guidance on classification of severe cases of disease because information on the impact on more severe outcomes will usually be important to WHO. Importantly, if surrogate markers of clinical endpoints are used, these must be fully validated as robustly predictive of clinical outcomes (e.g. achieving undetectable viral load in HIV infection). The GEG document should therefore clarify whether surrogate endpoints are considered as relevant endpoints by WHO for the formulation of policy in the specific context of the guidance document under development. Ultimately the GDG will decide what is critical to consider for guidelines, but the GEG document should aim to provide guidance to developers on what is likely to be considered most important based on the experience of the technical unit and currently-available information. For vector control products, the preferred endpoints are generally the incidence of disease and/or detection of new infections in humans. In situations where infection is chronic, or an infection frequently manifests subclinically, the prevalence of pathogen infection (or prior infection) is warranted. Prevalence may also be used as a secondary trial endpoint for those trials designed to collect incidence data. For diagnostics, the most common efficacy equivalent outcome will be test performance5. 3.4.2 Safety All clinical trials that are conducted pre-licensure or post-licensure should include an assessment of safety. It is critical that high-quality data on safety are collected in the relevant populations and age groups for which the biomedical intervention is intended (including high-risk and immunocompromised groups such as HIV- infected children or adults). Conduct and reporting should be in accordance with international standards and accepted case definitions. When the assessment of safety is a primary objective of a clinical trial – as in early phase 1 and 2 trials – it is usual for the primary analysis to be based on a specific safety endpoint (e.g. rates of serious adverse events [SAEs] or rates of predefined SAEs or adverse events [AEs] that may be of interest). WHO guidance 5 See: Global Harmonization Task Force document GHTF/SG5/N6:2012 on Clinical evidence for IVD medical devices at https://www.imdrf.org/sites/default/ files/docs/ghtf/final/sg5/technical-docs/ghtf-sg5-n6-2012-clinical-evidence-ivd-medical-devices-121102.pdf (accessed 1 June 2023). 12 Evidence generation for development of health products: a practical guide for WHO staff  on evidence generation is likely to target later-stage trials when the assessment of safety or specific aspects of the safety profile is a secondary objective. At this stage trials are not usually powered a priori to support statistical analyses of endpoints such as rates of all, or specific, AEs. Therefore WHO may need to provide guidance on the detection of specific AEs and management of the safety of trial participants, tailored to the trial population and to what is already known about the effects of the intervention (16). 3.4.3 Other endpoints The primary or co-primary endpoints of pivotal Phase 3 studies should primarily be meaningful to patients and also sufficient to support both regulatory and policy decisions to avoid research waste by conducting trials for different purposes. These elements should be included in the guidance document to ensure that regulatory and policy perspectives (e.g. prequalification, technical units and regulatory viewpoints) are included equally in the consultative process. There should be a discussion about the advantages and disadvantages of choosing different possible endpoints. Some endpoints may be more relevant to policy but are less feasible for measurement because of factors such as sample-size constraints due to increased costs of larger trials or laboratory capacity requirements. Some endpoints may require a high degree of laboratory capacity, and therefore the extent of strengthening of laboratory capacity in desired sites may be a relevant factor in the choice of endpoints. Also, some endpoints may be specific to certain types of health system or health-seeking behaviour; it is important that the selected endpoints are appropriate to the health systems of the settings included. Technical units should encourage developers to make use of WHO’s Coordinated scientific advice procedure (22) before finalizing key Phase 3 trials/clinical validation studies to ensure that their development plans include data that can satisfy both WHO PQ and policy recommendation requirements. 3.5 Suitable reference standards/comparators Guidance should be provided regarding expectations on choice of the best control. For therapeutics and vaccines this could be the current standard of care for treatment, or a currently-used vaccine or a placebo. The use of placebo alone in a control group is not usually appropriate where an alternative efficacious intervention is available, and ethics should be carefully considered. For diagnostics, the reference test or reference strategy should be selected carefully. For clinical impact outcomes of diagnostics, the comparator is likely to be standard of care and should be defined. The inclusion of WHO-prequalified tests as comparators in evaluation studies also provides the ability to benchmark and generate stronger evidence for WHO review (23). If necessary, a composite reference standard that combines several existing tests (e.g. phenotypic susceptibility testing plus genotypic sequencing for microbiological resistance testing) can be used. All efforts must be made to recruit a concurrent control group; the use of historical controls should be discouraged. 3.6 Clinical case definitions Guidance should clearly describe the clinical and laboratory criteria that must be met to define a case and predefined primary and secondary endpoints. The criteria should be sufficiently detailed to enable standardized collection of both clinical and laboratory data – including assay standardization or quality control elements. For infectious disease, the case definition should usually include the laboratory confirmation of the presence of the target pathogen. Definitions may be required for “mild”, “uncomplicated” or “early” disease and severe disease, as the extent to which the medical product prevents, treats or diagnoses severe disease is often an important consideration 133. Content of a guidance on evidence generation (GEG) document 3.7 Case ascertainment It is critical that the same methodology for case ascertainment and endpoints is applied consistently at all clinical sites throughout the duration of a trial. In trials designed to detect disease prevention, passive case ascertainment is usually based on trial subjects/caregivers presenting to or otherwise contacting a local health-care facility due to the onset of specific symptoms. Active case ascertainment usually requires frequent monitoring and contact with trial subjects/caregivers. Clear instructions must be provided for detection of a possible case – i.e. contact with designated health-care professionals, telephone contacts, investigations to be carried out once a case is confirmed, etc. For efficacy endpoints other than clinically apparent disease, it is essential for subjects to be monitored at regular intervals, and the frequency of these visits (with acceptable windows around the visits) should be clearly stated. 3.8 Sample size Estimating the required sample size is a crucial part of study design which ensures that studies are sufficiently powered to detect clinically relevant findings without burdening an excessive number of participants. WHO may wish to guide sample size calculations by indicating the most critical endpoint to which the study should be powered and any other pertinent assumptions to be used – e.g. what may be considered a clinically meaningful difference for policy-making and PQ determination. Guidance should clarify that the number of events of relevant endpoints is as important as the number of trial participants for policy processes. For instance, moderately large trials may be insufficient to generate adequate evidence if the incidence rate of the event of interest is so low that very small numbers of events are expected. Where incidence of an endpoint of greatest public health relevance is very low, a surrogate endpoint should be discussed together with the data needed to allow for interpretation of the surrogate. 3.9 Statistical analysis Guidance should insist on the need to provide a clear statistical analysis plan which should include sample size, the provision of disaggregated data by age or sex as appropriate, and any other analysis considerations (stratification, subgroup analyses, sensitivity analyses etc.) which can be done by referencing standard guidance. The data should be analysed in accordance with the protocol and statistical analysis plan developed prior to knowledge of the study results. Any analysis post hoc should be clearly identified as such. In RCTs, the main analyses should follow the “intention-to-treat” principle (16).Technical teams developing guidance on evidence generation may indicate whether interim analyses should be considered. In general, any prognostic features that are to be used in analyses of intervention effects in RCTs should be clearly recorded (and/or sample collected) before randomization (16). 14 Evidence generation for development of health products: a practical guide for WHO staff  3.10 Specific product-related issues 3.10.1 Vaccines Generation of evidence on aspects of vaccine-related safety should be carefully described, with clarification of the suitable safety database to be assembled pre-licensure (24). Sections may also be added on the measurement of immunogenicity, immune correlates of protection (ICP) and bridging studies, as relevant (1). In most cases, ICPs have been determined from vaccine efficacy trials that were initiated pre-licensure, often with long-term follow-up of subjects that extended into the post- licensure period. In the absence of such ICPs (e.g. tuberculosis), efficacy trial protocols should plan to collect sufficient information to allow for analyses of the relationship between immune parameters and protection against clinically apparent disease. At the minimum, this requires the collection of post-vaccination samples from a substantial subset of the vaccinated and control groups. Serial collection of samples over the longer term, along with follow-up surveillance for vaccine breakthrough cases, can also support identification of ICPs. A complementary process which focuses on evidence generation for policy is the WHO Evidence Considerations for Vaccine Policy (ECVP) framework. This is designed to facilitate early engagement and alignment across the stakeholders involved in vaccine development and those responsible for regulatory, policy, programmatic use and decisions on introduction. It includes regulatory, delivery and timeline trigger considerations for vaccine policy development. The ECVP framework aims to support collective delineation of the clinical trial and observational data or other evidence that is anticipated to be needed for policy decisions relating to new vaccines – i.e. indications where no vaccine currently exists or for new vaccine classes such as vaccines for new target populations – thus minimizing delays between vaccine licensure and policy formulation, adoption and introduction, particularly in LMICs. 3.10.2 Diagnostics A fundamental step in the validation of diagnostic tests is the assessment of their performance, including analytical and clinical performance. Analytical performance refers to the ability of an in vitro diagnostic device to detect or measure a particular analyte6. Analytical performance includes: stability of specimen(s); validation of specimens; metrological traceability of calibrator and control material values; precision (repeatability and reproducibility); analytical sensitivity, analytical specificity; high-dose hook effect; validation of the assay procedure; usability/human factors; and stability of the IVD and performance panels. It is generally evaluated by using contrived specimens, repository specimens or commercial panels (25). Clinical performance refers to the assay’s ability to provide results that are correlated with a particular clinical condition/physiological state in accordance with the target population and intended user (22). This includes diagnostic sensitivity and specificity and qualification of usability. Evaluation of clinical performance may be conducted prospectively or retrospectively using intended-use specimens collected from an intended-use population. All studies should conform with agreed principles that consider ethics, design, conduct, quality assurance measures and reporting and must rely on the use of a fully accepted/recommended reference method or composite reference as comparator (26). Depending on the index test and planned comparator tests, the study flow needs special consideration. For comparability purposes, the index test, comparator test, or reference standard should ideally be performed on the same specimen. In addition, it might be indicated to assess the clinical utility of a diagnostic test – i.e. the usefulness of the results provided by the test and their impact on patient-important outcomes, such as time to treatment initiation or mortality as well as elements such as acceptability, feasibility and 6 See: Global Harmonization Task Force document GHTF/SG5/N6:2012 on Clinical evidence for IVD medical devices at https://www.imdrf.org/sites/default/ files/docs/ghtf/final/sg5/technical-docs/ghtf-sg5-n6-2012-clinical-evidence-ivd-medical-devices-121102.pdf (accessed 1 June 2023). 153. Content of a guidance on evidence generation (GEG) document cost-effectiveness. While such studies collect important information for implementation concerns, they are not generally expected in a regulatory submission and are therefore rarely conducted by the manufacturers. However, these studies are of paramount importance in health technology assessment processes such as the development of policy at WHO. Lastly, WHO’s review considers the specific patient population targeted by the guideline, the level of the health system for implementation, and the needs and challenges related to distribution of and access to the new test, especially in resource-limited countries and among vulnerable groups. 3.10.3 Therapeutics Combination treatment regimens will be specifically required for some disease conditions such as tuberculosis. Guidance should be provided regarding which combinations WHO considers to be more desirable and which other drugs it is possible to co-administer – e.g. the need for no significant drug–drug interactions with first- line HIV treatment regimens in many target settings. 3.10.4 Vector control products WHO’s process for developing recommendations for new vector control interventions relies on evidence from well-designed and well-conducted trials with epidemiological endpoints to demonstrate the public health value of the intervention. The process includes establishing close interaction with the Vector Control Advisory Group (VCAG) early in the protocol development process in order to ensure that trial data meet WHO’s standards for determining public health value (27). To this end, WHO has produced norms and standards to guide the VCAG’s advice on the generation of epidemiological data and study designs for trials assessing the public health value of novel vector control interventions (7). To determine the epidemiological impact of a vector control intervention, the preferred endpoints are generally the incidence of disease and/or the detection of new infections in humans. In situations where infection is chronic, or an infection frequently manifests subclinically, use of the prevalence of pathogen infection (or prior infection) is warranted. Prevalence may also be used as a secondary trial endpoint for those trials designed to collect incidence data. Within WHO’s vector control evaluation process, a minimum of two trials with epidemiological endpoints conducted in two geographically separate settings is required to demonstrate that any observed public health value is replicable across settings. Trial durations should consider the characteristics of the intervention and its intended deployment, expected durability/residual efficacy and replacement intervals, and the contextualized epidemiology (e.g. pathogen transmission intensity) in the selected study sites. Trial durations should be selected that maximize the likelihood that the study objectives and targeted statistical power will be robustly achieved in order to strengthen the evidence that is used to inform the best WHO recommendation. Where appropriate, consideration may need to be given to other factors that might influence the long-term efficacy of an intervention (including behavioural or genetic adaptation to the existence of the intervention). 3.11 Special/vulnerable populations As needed, technical departments willing to develop a GEG document will have to consider the specificities of the desired product in relation to their use in various vulnerable populations for reasons related to comorbid conditions requiring concomitant treatment (e.g. HIV-infected persons using ART); dosing and toxicity (children, pregnant and lactating women, older persons); product formulation (children); access to care (migrants, refugees, prisoners) etc (28). 16 Evidence generation for development of health products: a practical guide for WHO staff  3.12 Other GRADE criteria For questions related to issues other than the benefits and harms of an intervention (e.g. feasibility, equity, acceptability, resource use), the best available evidence should be used and the choice of specific study designs will vary according to the topic and other factors. The WHO handbook for the development of normative products will include a chapter on evidence-to-decision frameworks encompassing GRADE ETD and WHO- INTEGRATE frameworks (Volume III, Chapter 5: Evidence to decision frameworks: an essential aid to developing normative statements). Technical departments may decide whether to indicate broad consideration of the following criteria in the guidance, or detail the methodologies to be used. Some potential areas to consider for evidence generation are listed below. 3.12.1 Patients’ values and preferences These are an integral part of the landscape of evidence generation as they enable us to better understand whether interventions are feasible and acceptable for implementation and therefore whether they will have a greater likelihood of uptake. Guidance should be given to direct researchers to the area of greatest likely concern, such as side-effects, visitation schedules, cost or stigma. Both quantitative and qualitative research may inform values and preferences. According to the WHO handbook for guideline development, “Evidence from qualitative research can be used to assess: (i) the extent to which the potential benefits or harms of an intervention are important to people (the relative importance of the outcomes); (ii) the extent to which certain interventions are more or less acceptable to different stakeholders (patients, care-givers, health-care providers etc.); (iii) the extent to which different interventions are more or less feasible to implement in different settings, based on people’s practical or day-to-day experiences with healthcare services; and (iv) the potential consequences of different interventions on equity across populations” (29, 30). 3.12.2 Acceptability, feasibility and equity Other aspects such as acceptability, feasibility and equity also form part of the criteria for formulating recommendations according to the GRADE Evidence to Decision Frameworks. Thus, the effect of tests on intermediate outcomes that imply impact on patient outcomes (e.g. reduced time to diagnosis and treatment) needs to be considered. Data from qualitative research can also provide valuable evidence on the feasibility of implementing specific interventions or options. 3.12.3 Estimation of public health impact It is important to evaluate the expected public health impact of an intervention. For instance, new diagnostics tests that have the potential to be used at lower levels of the health-care system may lead to diagnosing people at earlier stages of disease progression, which may accelerate treatment initiation, improve clinical outcomes and reduce disease transmission. To this end, mathematical modelling studies can provide useful information on the population-level impact of an intervention (31). WHO guidance may direct researchers to the most important factors to consider in modelling. 3.12.4 Cost and cost-effectiveness Economic analyses should also be part of the assessment to inform the WHO policy process (32). Where possible, researchers should consider collecting cost data alongside evidence-generating studies to assess feasibility and cost and to inform cost-effectiveness analyses which are likely to be performed by specialist researchers. Access, supply and pricing are critical considerations for WHO. Comparative cost-effectiveness data have often been important in medicine recommendations and are requested as part of the WHO Model 173. Content of a guidance on evidence generation (GEG) document list of essential medicines procedure. Studies are needed to clarify price, evaluate data requirements and assess how these will be used in decision-making. WHO may also provide guidance to researchers on the most critical choices to inform modelling, which may include comparison of different vaccination strategies, or when in the clinical pathway a test may be used. It may be beneficial to have independent modelling performed to avoid possible conflicts. WHO produced a guide to cost-effectiveness analysis called WHO-CHOICE (33). WHO has developed a resource package for analysing the health benefit packages or health technology assessment (34). Further support may be found in through the Economic Evaluation and Analysis website and team (35). 3.13 Post-licensure studies, post-market surveillance and pharmacovigilance considerations For most medical products, post-licensure/Phase 4 studies are essential to monitor their safety in routine use and practice and to collect information on rare adverse events that might have been missed during the clinical development path (Phase 1 to Phase 3 trials). Well-developed pharmacovigilance systems are required in areas where the product will be introduced – either in Phase 4 implementation studies or in routine use. Post-licensure studies may also be conducted in populations for which adequate data have not yet been collected, such as vulnerable populations, and in paediatric and pregnancy studies. GEG documents should note these points which will be of importance for evaluation and review by the Guideline Development Group and the Strategic Advisory Group of Experts on Immunization (SAGE). Post-licensure studies may also be used to demonstrate the generalizability of efficacy or effectiveness results, especially in situations where conditional licensure may have been granted on the basis of surrogate endpoints. The evaluation of post-market surveillance experiences can also highlight opportunities to improve a medical device (36). 3.14 Reporting The GEG process should be clear on the need to capture data as efficiently and reliably as possible and the need to comply with international standards for data reporting. For instance: The Consolidated Standards of Reporting Trials (CONSORT) website outlines the minimum set of recommendations for reporting randomized trials7. It also offers a standardized approach for presenting trial findings, which facilitates complete and transparent reports as well as critical appraisal and interpretation. The reporting of the diagnostic accuracy studies that assess clinical validity should follow the Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidance to ensure that all essential information about the study is provided (37). Safety data should be classified according to a standardized terminology (such as ICH MedDRA) to enable their categorization by system organ class or preferred term8. Systematic reviews should follow standard methods such as those described by Cochrane and be reported using the preferred reporting items for systematic review and meta-analysis (PRISMA) guideline (38). 7 See Consolidated Standards of Reporting Trials (CONSORT) at http://www.consort-statement.org (accessed 1 June 2023). 8 See the Medical Dictionary for Regulatory Activities (MedDRA) at https://www.meddra.org/ (accessed 1 June 2023). 18 Evidence generation for development of health products: a practical guide for WHO staff  The Standards for Reporting Qualitative Research (SRQR) aim to improve the transparency of all aspects of qualitative research by providing clear standards for reporting qualitative research (39). In addition, specific requests made by WHO-established Guideline Advisory or Technical Groups (e.g. SAGE9, VCAG10) in terms of data collection and evidence review should be carefully followed. Of note, there are also works by the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network11 that seeks to improve reporting of all health research, which can be profitably consulted. Useful guidance is also available from the Agency for Healthcare Research and Quality12. 3.15 WHO prequalification WHO’s PQ process aims to ensure that key health products (diagnostics, medicines, vaccines and immunization- related equipment and devices, and vector control products) for high burden diseases meet global standards of quality, safety and efficacy/performance in order to optimize the use of health resources and improve health outcomes13. The GEG document should indicate relevant prequalification considerations where appropriate. The PQ process consists of a transparent, scientifically sound assessment which includes dossier review, product testing, performance evaluation and inspection of manufacturing sites. This information, in conjunction with other procurement criteria, is used by the United Nations and other procurement agencies to make purchasing decisions regarding diagnostics, medicines and/or vaccines. WHO PQ covers medicines, in vitro diagnostics, vaccines, immunization devices and vector control products. Information on each of these product streams can be found through the website at https://extranet.who. int/pqweb/. An essential prerequisite for PQ is an indication that the class of medical product is of public health value through the assessment of the relevant WHO technical department. This assessment is usually provided in the form of a recommendation for use in WHO guidelines documents that have undergone quality oversight through the WHO Guidelines Review Committee or, in the case of vaccines, by SAGE. In liaison with the relevant PQ team, the GEG document should cross-reference any existing PQ-related guidance and ensure that other critical data generation requirements for PQ not covered in the GEG document are indeed being addressed – ideally before development of guidelines14. Links and information on relevant PQ streams can be found in Annex 1. 3.15.1 Vaccines Vaccines PQ aims to ensure that a sufficient supply of safe, effective vaccines is available for immunization programmes, including in emergency situations and in response to novel disease outbreaks. It assesses vaccines for PQ via a rigorous process of evaluation and applies WHO-established standards and norms of acceptable quality, safety and efficacy. These cover vaccine standards that address the manufacturing, licensing, quality control, labelling, transportation and storage of vaccines. 9 See the Strategic Advisory Group of Experts on Immunization (SAGE) at https://www.who.int/groups/strategic-advisory-group-of-experts-on- immunization (accessed 1 June 2023). 10 See the Vector Control Advisory Group (VCAG) at https://www.who.int/groups/vector-control-advisory-group (accessed 1 June 2023). 11 See the Equator network at https://www.equator-network.org/ (accessed 1 June 2023). 12 See the Agency for Healthcare Research and Quality (AHRQ) at https://www.ahrq.gov/ (accessed 1 June 2023). 13 See details of WHO’s prequalification process at https://extranet.who.int/pqweb/ (accessed 19 June 2023). 14 In this respect, developers will be encouraged to make use of the WHO Coordinated Scientific Advice (CSA) procedure at an early stage of development. 193. Content of a guidance on evidence generation (GEG) document Prequalification is required for procurement of the product by United Nations agencies and for financing by agencies such as Gavi. Registration by an NRA, or by the European Medicines Agency in the case of the centralized procedure for marketing authorization in the European Union, will be required prior to any consideration of PQ. A SAGE recommendation for use of vaccines against the specific pathogen is also required prior to official issuance of WHO PQ. WHO encourages vaccine developers and manufacturers to be aware of the WHO PQ process and to discuss the product and the regulatory requirements with the WHO PQ staff early in the process. It is important that developers and manufacturers understand WHO’s preferences for parameters that have a direct operational impact on immunization programmes. To facilitate early attention to these issues, WHO has published several documents that describe WHO preferences for vaccine presentations, packaging and programmatic suitability (see links in Annex 1). 3.15.2 Medicines Prequalification of medicines ensures that active pharmaceutical ingredients (APIs) and finished pharmaceutical products (FPPs) are safe and effective and that they meet stringent quality standards. The principal medicines PQ activities are: assessment of product dossiers (for FPPs) or master files (for APIs), assessment of manufacturing and clinical sites, and organization of quality control testing of products. These activities form part of WHO’s broad agenda which seeks to expand access to quality-assured medicines. WHO’s PQ currently covers medicines for HIV/AIDS, malaria, tuberculosis, reproductive health, hepatitis B and C, influenza, COVID-19, Ebola virus disease, infections in newborn and young infants and childhood pneumonia, multi-drug resistant bacterial infections, diarrhoeal diseases and selected neglected tropical diseases. WHO also recently initiated PQ of biotherapeutic products – selected products to treat certain types of cancer, as well as human insulin and insulin analogues for diabetes. In the area of generics, efficacy and safety will be documented by a bioequivalence study (or a waiver in certain circumstances). In the vast majority of cases, therefore, clinical efficacy studies will not be needed for PQ of generics. As the medicines PQ programme covers a range of disease areas, there is minimal specific WHO PQ-oriented guidance on clinical studies for specific diseases or product classes. Pre-submission meetings are mandatory for new applicants to medicines PQ, and specific details of requirements and expectations are provided in this way (see Annex 1). The PQ medicines team is ready to respond to questions from those wishing to develop products in line with WHO’s priorities which reflect unmet medical need. 3.15.3 In vitro diagnostics Prequalification of in vitro diagnostics (IVD) aims to promote and facilitate access to safe, appropriate and affordable diagnostics of good quality in an equitable manner. The focus is on IVDs for priority diseases that are appropriate for use in resource-limited settings. WHO IVD prequalification incorporates comprehensive assessment of individual IVDs through a standardized procedure to determine whether the product meets WHO PQ requirements. The assessment includes four components: review of a product dossier; performance evaluation, including operational characteristics; manufacturing site(s) inspection; and labelling review. IVD PQ covers a wide array of diagnostics for both endemic and epidemic diseases in LMICs, namely: HIV/AIDS, hepatitis B and C, human papillomavirus, malaria, glucose-6-phosphate dehydrogenase (G6PD) deficiency, cholera, syphilis, tuberculosis and SARS-CoV-2. The programme eligibility will shortly be expanded to tests for noncommunicable diseases. Manufacturers can request a meeting to answer questions prior to submitting the dossier (see Annex 1). 20 Evidence generation for development of health products: a practical guide for WHO staff  3.15.4 Vector control products A structure has been established within WHO to assess the public health value of vector control products with the involvement of PQT/VCP, GMP, NTD and an independent Vector Control Advisory Group (VCAG)15. PQ of vector control products focuses on the assessment of new products used for the prevention of vector-borne disease, including bednets, sprays and larvicides. PQ assesses vector control products and public health pesticide active ingredients to determine if they are manufactured to a sufficiently high quality standard and can be used safely and effectively. Products that meet PQ requirements are added to the WHO list of vector control products. WHO prequalification of vector control products primarily benefits populations most affected by major vector-borne (often also neglected tropical) diseases such as malaria, dengue fever and other arboviral diseases (Chikungunya, Zika virus), Chagas disease, lymphatic filariasis, visceral leishmaniasis and human African trypanosomiasis. Prequalification pathway: Irrespective of whether an intervention is first-in-class or whether a WHO recommendation supporting its use is already in place, all products must undergo the PQ evaluation if they are to be prequalified. The PQ pathway for vector control products is managed by PQT-VCP. The team assesses product dossiers, inspects manufacturing sites and, as appropriate, supports quality-control testing of products. Interventions without a WHO recommendation will follow the New Intervention Pathway, which complements the PQ pathway. The New Intervention Pathway is designed to help substantiate an intervention’s public health value and, in doing so, to support the development of an evidence base to inform deliberations on a WHO recommendation by a Guideline Development Group. 3.16 Ethical aspects Researchers must be reminded of the principles of conducting studies and trials with full respect for the rights and well-being of participants (40). Independent review of proposals for new research, through an Institutional Review Board (IRB), Research Ethics Committee (REC) or equivalent, is an important governance tool and can help ensure that appropriate steps are taken to protect the rights and welfare of participants. 3.17 Role of this guidance in relation to WHO’s Coordinated Scientific Advice (CSA) procedure The Coordinated Scientific Advice (CSA) procedure is used for developers working on a well-defined specific product that will meet WHO conditions for scientific advice. Consequently, if a single product developer would benefit from an early-stage interaction with WHO to ask specific questions or receive feedback on their product development plan, the WHO CSA procedure is applicable. This guidance is intended to be used where a WHO technical department wishes to provide generic guidance on evidence generation which is publicly available and can be used by any product developer, funder, government, or other stakeholder interested in providing evidence to support public health product assessment, who would benefit from being aware of WHO’s preferences for data generation. This guidance is therefore applicable where multiple product developers may engage and an open inclusive process is warranted to discuss key needs for data generation. 15 See Vector control product prequalification at https://extranet.who.int/pqweb/vector-control-products (accessed 1 June 2023). 213. Content of a guidance on evidence generation (GEG) document Box 1: Example Guidance for the evaluation of tuberculosis diagnostic that meet World Health Organization (WHO) Target Product Profiles: Achieving the targets of WHO’s End TB strategy will require improved tests for early and rapid detection of TB and for universal drug-susceptibility testing (DST) in order to reach more patients and accelerate the decline in TB incidence and mortality. In 2014, WHO developed target product profiles (TPPs) for most needed tests to close the TB diagnostic gap, defined as follows: 1) a rapid sputum-based test for detecting TB at the microscopy-centre level; 2) a rapid biomarker-based non-sputum test for detecting TB; 3) a triage test of referral test for identifying patients suspected of having TB; and 4) a test for rapid DST. These TPPs outline a wide range of requirements that need to be considered when developing new products, so that they can best meet the requested public health needs and be evaluated for potential use by WHO, following the clear set of procedures outlined in WHO’s Handbook for guideline development. While WHO’s TPPs provide high-level information on the most suitable characteristics of a given type of product, further detailed guidance can be produced on the suitable trial/study designs aimed at producing relevant evidence for WHO guideline development. This is essential since the evidence generated will be subject to thorough evaluation by WHO, using the GRADE assessment, to allow for the development of full evidence-based recommendations with a process that is fully documented and transparent. To allow this, WHO and its partners held a series of consultations in 2018 to develop guidance for the evaluation of TB diagnostics that meet the WHO TPPs. The objective was to provide clarity around the clinical evaluation needs for tests that have the potential to meet the TPP specifications published by WHO with a view to facilitating the evaluation of such tests and developing recommendations for use, as appropriate. These TPPs served as a starting point for the exercise. On this basis, guidance was developed by multidisciplinary writing committees (one for each TB diagnosis), using a step-by-step iterative consensus-building process, followed by a public consultation process. The guidance focused on the trial needs to address the analytical and clinical validity for the four high- priority TPPs that can substantially improve the current TB diagnostic cascade. For each type of test, the following specific topics were addressed: 1. the intended use of the test and its implication for the study design; 2. general study design considerations; 3. choice of population and setting; 4. issues pertaining to the index test (the test under investigation) itself; 5. reference standard and comparators; 6. flow and specimen issues; and 7. key issues beyond accuracy that should be considered (including impact studies and evaluation of benefits/harms). As an outcome of this undertaking, relevant design details and characteristics were provided for each of the above-listed topics. It is hoped that these will help in generating the appropriate evidence on their performance and operational characteristics to ensure that policy recommendations can be made, which is the first step for TB patients to access these tests. Studies following the outline provided in the above guidance documents are expected 1) to generate high-quality laboratory and clinical data regarding assay performance and operational characteristics and 2) to support WHO review and potential recommendation. • Denkinger CM, Schumacher SG, Gilpin C, Korobitsyn A, Wells WA, Pai M et al. Guidance for the evaluation of tuberculosis diagnostics that meet the World Health Organization (WHO) target product profiles: an introduction to WHO process and study design principles. J Infect Dis. 2019;220(S3):S91–8. https://doi.org/10.1093/infdis/jiz097. • High-priority target product profiles for new tuberculosis diagnostics: report of a consensus meeting, 28–29 April 2014, Geneva, Switzerland. Geneva: World Health Organization; 2014 (https://apps.who.int/iris/handle/10665/135617, accessed 2 June 22 4Suggested procedure to develop guidance on evidence generation The following is a non-binding suggestion as to how you may wish to proceed based on the WHO harmonized process for the development of Target Product Profiles. This procedure is aimed at WHO staff and includes the internal clearance procedures as of the publication date. 1. Determine the need for guidance on evidence generation – e.g. any of the below considerations: • Does WHO need to provide proactive advice on the health product? • Are multiple developers developing a similar product? • Will WHO need to prepare guidelines on the product – i.e. is it important to public health? • Does a TPP/PPC/TRP already exist which needs to be expanded upon? 2. Draft a one-page scope and submit planning clearance for another normative and standard setting product: • Review existing evidence – e.g. systematic reviews, review of reviews. • Identify evidence gaps. 3. Constitute a group with the relevant expertise: • Identify experts in the health topic and in the different methods – e.g. clinical trials, statistical analyses, cost-effectiveness, regulatory issues, programme officers and policy-makers. • If a TPP does not exist, ensure that end-user product preferences are taken into account to understand key questions for implementing the product. • The development group members should be screened following WHO standard procedures for experts with assessment of real or perceived conflict of interest (COI) using standard WHO Declaration of Interest (DOI forms). See information in the Compliance, Risk Management and Ethics office (CRE). Experts with declared interests that cannot be managed adequately should be excluded. The development group should not include representatives of for-profit industry entities. Staff from PDPs (Product Development Partnerships) and from private sector entities with an underlying conflict of interest, and procurement agencies should not be included in the TPP development group. As a general rule, entities that are themselves funded to develop or facilitate the development of medical or health products should not be involved in the decision-making process that takes place during the development of a WHO normative product. However, if relevant, their feedback should be sought at a later stage, for example, during a public comment phase. 4. Develop a version zero draft • Review relevant literature and current state of the art. • Review any harmonized measures for trials – e.g. outcome measures for trials. • Understand what evidence a Guideline Development Group will need to make the most robust decision. • Address all key criteria and add any extra headings relevant to the specific topic area. 234. Suggested procedure to develop guidance on evidence generation 5. Consult with the development group to refine the draft to version 0.1. 6. Consider a period of public consultation: • If using a period of public comment, ensure that the product reaches the key audience – i.e. policy- makers, trialists, funders. 7. Finalize guidance on evidence generation: • It is recommended to finalize by consensus. 8. Disseminate the GEG widely: • Ensure the GEG reaches all stakeholders. • Encourage funders to refer to the GEG when deciding on funding trials. • Consider linking the launch to a specific event. 9. Inform RFH/EPS to add to the database of research and development documents. 10. Review the document regularly, e.g. after 24–36 months. 24 Useful websites Guidance for good randomized clinical trial. (Online). Developed by the Good Clinical Trials Collaborative, May 2022 (https://www.goodtrials.org/the-guidance/guidance-overview/, accessed 2 June 2023). Clinical research in resource-limited settings. A consensus by a CIOMS Working Group. Geneva: Council for International Organizations of Medical Sciences (CIOMS); 2021 (https://cioms.ch/wp-content/uploads/2021/06/ CIOMS_ClinicalResearch_RLS.pdf, accessed 2 June 2023). Implementation of the 2022 WHA resolution on clinical trials: https://www.who.int/our-work/science-division/ research-for-health/implementation-of-the-resolution-on-clinical-trials 25 References 1. Guidelines on clinical evaluation of vaccines: regulatory expectations. In: WHO Expert Committee on Biological Standardization: sixty-seventh report. Geneva: World Health Organization; 2017: Annex 9 (WHO Technical Report Series, No. 1004; https://iris.who.int/handle/10665/255657, accessed 3 June 2023). 2. Guidelines for the clinical evaluation of dengue vaccines in endemic areas. Geneva: World Health Organization; 2008: WHO document WHO/IVB/08.12 ( https://iris.who.int/handle/10665/69850, accessed 3 June 2023). 3. Evaluation of COVID-19 vaccine effectiveness. Interim guidance. Geneva: World Health Organization;2021 (https://iris.who.int/handle/10665/363344, accessed 3 June 2023). 4. WHO handbook for guideline development, second edition. Geneva: World Health Organization; 2014 https://iris.who.int/handle/10665/145714, accessed 3 June 2023). 5. Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE et al. GRADE: grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008;336:1106–10. 6. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: introduction. BMJ. 2016;353:i2016. 7. Norms, standards and processes underpinning development of WHO recommendations on vector control. Geneva: World Health Organization; 2020 (https://iris.who.int/handle/10665/338030, accessed 3 June 2023). 8. From vaccine development to policy: a brief review of WHO vaccine-related activities and advisory processes. Technical document. Geneva: World Health Organization; 2017 (https://www.who.int/ publications/m/item/from-vaccine-development-to-policy-a-brief-review-of-who-vaccine-related- activities-and-advisory-processes-(2017), accessed 3 June 2023). 9. WHO handbook for the development of normative products. Geneva: World Health Organization; 2023 (https://intranet.who.int/homes/mst/qahandbook/, accessed 14 June 2023). 10. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). (website). Geneva: International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (https://www.ich.org/, accessed 13 August 2023). 11. Guidance for good randomized clinical trials. (Online). Developed by the Good Clinical Trials Collaborative, May 2022 (https://www.goodtrials.org/the-guidance/guidance-overview/, accessed 2 June 2023). 12. Clinical research in resource-limited settings. A consensus by a CIOMS Working Group. Geneva: Council for International Organizations of Medical Sciences (CIOMS); 2021 (https://cioms.ch/wp-content/ uploads/2021/06/CIOMS_ClinicalResearch_RLS.pdf, accessed 2 June 2023). 13. Implementing citizen engagement within evidence-informed policy-making: an overview of purpose and methods. Geneva: World Health Organization; 2022 ( https://iris.who.int/handle/10665/364361, accessed 14 June 2023). 14. Voice, agency, empowerment - handbook on social participation for universal health coverage. Geneva: World Health Organization; 2021 (https://iris.who.int/handle/10665/342704, accessed 22 June 2023). 26 Evidence generation for development of health products: a practical guide for WHO staff  15. Denkinger CM, Schumacher SG, Gilpin C, Korobitsyn A, Wells WA, Pai M et al. Guidance for the evaluation of tuberculosis diagnostics that meet the World Health Organization (WHO) target product profiles: an introduction to WHO process and study design principles. J Infect Dis. 2019;220(S3):S91–8. doi:10.1093/ infdis/jiz097. 16. Malaria vaccines: preferred product characteristics and clinical development considerations. Geneva: World Health Organization; 2022 (https://iris.who.int/handle/10665/362694, accessed 3 June 2023). 17. Whiting PF, Weswood ME, Rutjes AW, Reitsma JB, Bossuyt PN, Kleijnen J. Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol. 2006; 6:9. 18. Albert H, Nathavitharana RR, Isaacs C, Pai M, Denkinger CM, Boehme CC. Development, roll-out and impact of Xpert MTB/RIF for tuberculosis: what lessons have we learnt and how can we do better? Eur Respir J. 2016;48:516–25. 19. Clinical evidence for IVD medical devices – clinical performance studies for in vitro diagnostic medical devices. Global Harmonization Task Force, November 2012 (https://www.imdrf.org/documents/ghtf- final-documents/ghtf-study-group-5-clinical-safetyperformance, accessed 3 June 2023). 20. Bhatt DL, Mehta C. Adaptive Designs for Clinical Trials. N Engl J Med. 2016;375(1):65–74. doi:10.1056/ NEJMra1510061. 21. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. Geneva: World Health Organization; 2016 (https://iris.who.int/handle/10665/252183, accessed 22 June 2023). 22. Coordinated Scientific Advice for health product R&D. Geneva: World Health Organization (https://www.who.int/activities/optimizing-research-and-development-processes-for-accelerated-access- to-health-products/who-coordinated-scientific-advice-for-health-product-r-d, accessed 21 June 2023). 23. Takwoingi Y, Leeflang MM, Deeks JJ. Empirical evidence of the importance of comparative studies of diagnostic test accuracy. Ann Intern Med. 2013;158:544–54. 24. Global vaccine safety blueprint. Geneva: World Health Organization; 2012: WHO document WHO/ IVB/12.07 (https://iris.who.int/handle/10665/70919, accessed 3 June 2023). 25. Burd EM. Validation of laboratory-developed molecular assays for infectious diseases. Clin Microbiol Rev. 2010;23:550–76. 26. Banoo S, Bell D, Bossuyt P, Herring A, Mabey D, Poole F et al. Evaluation of diagnostic tests for infectious diseases: general principles. Nat Rev Microbiol. 2010;8:S17–29. 27. Standard operating procedures for Vector Control Advisory Group (VCAG) applicants. Geneva: World Health Organization; 2019 (https://apps.who.int/iris/handle/10665/274450, accessed 3 June 2023). 28. Gupta A, Hughes MD, Garcia-Prats AJ, McIntire K, Hesseling AC. Inclusion of key populations in clinical trials of new antituberculosis treatments: current barriers and recommendations for pregnant and lactating women, children, and HIV-infected persons. PLoS Med. 2019;16(8):e1002882. 29. Glenton C, Lewin S. Using evidence from qualitative research to develop WHO guidelines. In: WHO Handbook for guideline development, second edition, chapter 15. Geneva: World Health Organization; 2014 (https://iris.who.int/handle/10665/145714). 30. Downe S, Finlayson KW, Lawrie TA, Lewin SA, Glenton C, Rosenbaum S et al. Qualitative evidence synthesis (QES) for guidelines: Paper 1 – using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):76. doi:10.1186/s12961-019-0467-5. 27References 31. Egger M, Johnson L, Althaus C, Schöni A, Salanti G, Low N et al. Developing WHO guidelines: time to formally include evidence from mathematical modelling studies. F1000Research. 2017;6:1584. doi:10.12688/f1000research.12367.2. 32. Husereau D, Drummond M, Augustovski F, de Bekker-Grob E, Briggs AH, Carswell C et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS 2) statement: updated reporting guidance for health economic evaluations. Value Health. 2022;25(1):3–9.doi: 10.1016/j.jval.2021.11.1351. 33. Methods for the economic evaluation of health care interventions for priority setting in the health system: an update from WHO CHOICE. Int J Health Policy Manag. 2021;10(11):673–7. doi: 10.34172/ ijhpm.2020.244. 34. Resource guide for the use of health technology assessment and health benefit package design processes. Geneva: World Health Organization; Resource Guide on the use of Health Technology Assessment in Health Benefit Package Design Processes (https://www.who.int/teams/health-systems-governance-and- financing/economic-analysis/health-technology-assessment-and-benefit-package-design/resource- guide-for-the-use-of-hta-and-hbp-design-processes). 35. Health Systems Governance and Financing. Geneva: World Health Organization (https://www.who.int/ teams/health-systems-governance-and-financing, accessed 13 August 2023). 36. Guidance for post-market surveillance and market surveillance of medical devices, including in vitro diagnostics. Geneva: World Health Organization; 2020 (https://iris.who.int/handle/10665/337551, accessed 3 June 2023). 37. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. 38. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1. 39. O’Brien B, Harris I, Beckman T, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245–51. 40. World Medical Association Declaration of Helsinki. Ethical principles for medical research involving human subjects. Ferney-Voltaire: World Medical Association; 2022. 28 Annex 1: Respective prequalification product streams to encourage early contact with manufacturers Medicines: technical Advice to manufacturers The Prequalification Team medicines (PQTm) is available to provide technical and scientific advice to manufacturers and clinical research organizations. See: https://extranet.who.int/pqweb/medicines/technical-advice (accessed 20 June 2023). Medicines: pre-submission meeting This is required by PQTm for all new applicants submitting via the full assessment route – the goal is to help the applicant resolve critical issues before submission. A pre-submission meeting allows the PQT team to have an overview of the product and: 1) to ensure that the applicant is on the right track; 2) to provide general guidance on how to proceed; and 3) to provide guidance on identified issues that should be dealt with prior to submission. At the same time, it is an opportunity for the applicant to: 1) introduce and discuss the intended dossier; 2) raise questions and gain valuable feedback; and 3) address issues prior to submission. See: https://extranet.who.int/prequal/content/pre-submission-meetings. Vaccines: pre-submission meeting If considered necessary or desirable by either party, a discussion may be held between the manufacturer, the responsible NRA (if willing to participate) and WHO before the actual evaluation process starts. This pre-evaluation meeting should be scheduled as early as possible and should have a predefined agenda, addressing questions sent to WHO in advance by the manufacturer. Such meetings are important for discussing programmatic suitability issues and can be scheduled when requested by the manufacturer. See: https://extranet.who.int/pqweb/vaccines/manufacturers. In vitro diagnostics: pre-submission meeting The In Vitro Diagnostics Assessment Team requires all first-time applicants to request a pre-submission meeting prior to submitting their first submission for prequalification assessment. A meeting can be requested by contacting the team at diagnostics@who.int. Vector control products: early evaluation of public health value A structure has been established within WHO to assess the public health value of vector control products with the involvement of PQT/VCP, GMP, NTD and an independent Vector Control Advisory Group (VCAG). See: https://www.who.int/groups/vector-control-advisory-group. 29Annex 1: Respective prequalification product streams to encourage early contact with manufacturers Vector control products: pre-submissions meeting The pre-submission phase of the Vector Control Product (VCP) prequalification process covers the time from the manufacturer’s initial contact with WHO to the submission of the full product dossier for evaluation. Manufacturers interested in the prequalification of a VCP are invited to contact the PQT/VCP team prior to the submission of their application(s) in order to discuss the requirements of the prequalification process and ask any questions. At any time, the PQT/VCP staff can assist the manufacturer with preparation for the prequalification process through pre-submission meetings. See: https://extranet.who.int/pqweb/vector-control-products/pre-submission PQ Medicines: Website: https://extranet.who.int/prequal/content/what-we-do Procedure: https://extranet.who.int/pqweb/medicines/prequalification-procedures-and-fees PQ Vaccines: Website: https://extranet.who.int/pqweb/vaccines Procedure: https://extranet.who.int/pqweb/vaccines/procedures-fees-who-prequalification PQ Diagnostics: Website: https://extranet.who.int/pqweb/in-vitro-diagnostics Procedure: https://extranet.who.int/pqweb/vitro-diagnostics/procedures-and-fees-prequalification PQ Vector control products: Website: https://extranet.who.int/pqweb/vector-control-products Procedure: https://extranet.who.int/pqweb/vector-control-products/prequalification-procedures- and-fees-vector-control-products World Health Organization Science division Research for Health department (RFH) Emerging Technologies, Research Prioritisation & Support Unit Email: aross@who.int, researchsupport@who.int