In vitro diagnostic medical devices used for the quantitative detection of HIV-1 nucleic acid

Technical specifications series for submission to WHO prequalification – diagnostic assessment TSS-11 In vitro diagnostic medical devices used for the quantitative detection of HIV-1 nucleic acid In vitro diagnostic medical devices used for the quantitative detection of HIV-1 nucleic acid (Technical specifications series for submission to WHO prequalification – diagnostic assessment, TSS11) ISBN 978-92-4-005750-0 (electronic version) ISBN 978-92-4-005751-7 (print version) © World Health Organization 2023 Some rights reserved. This work is available under the Creative Commons Attribution- NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization (http://www.wipo.int/amc/en/mediation/rules/). Suggested citation. In vitro diagnostic medical devices used for the quantitative detection of HIV-1 nucleic acid. Geneva: World Health Organization; 2023 (Technical specifications series for submission to WHO prequalification – diagnostic assessment, TSS11). Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see https://www.who.int/copyright. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. TSS 11 Page | iii Contents ACKNOWLEDGEMENTS IV LIST OF CONTRIBUTORS IV ABBREVIATIONS V 1 INTRODUCTION 1 2 HOW TO APPLY THESE SPECIFICATIONS 2 3 OTHER GUIDANCE DOCUMENTS 2 4 PERFORMANCE PRINCIPLES FOR WHO PREQUALIFICATION 2 4.1 Intended use 2 4.2 Diversity of specimen types, users and testing environments and impact on required studies 3 4.3 Applicability of supporting evidence to IVD under review 3 5 TABLE OF REQUIREMENTS 6 Part 1: Analytical performance and other evidence 7 Part 2: Clinical evidence (clinical performance characteristics) 24 Part 3: Qualification of usability for near POC testing by healthcare professionals 27 6 REFERENCES 28 TSS 11 Page | iv Acknowledgements Acknowledgements are due to the many experts whose contributions made this publication possible. The document was prepared in collaboration with J. Saldanha; consultant, California, United States of America (USA); R. Luo, consultant, California, USA; S. Best, consultant, Melbourne, Australia; D. Healy, U. Ströher, In vitro diagnostics assessment team, Prequalification Unit and L Vojnov, Global HIV, Hepatitis and STI Department, WHO. This document was produced under the coordination and supervision of U. Ströher and I. Prat, In vitro diagnostics assessment team, Prequalification Unit, WHO, Geneva, Switzerland. List of contributors A technical consultation on WHO prequalification requirements for hepatitis C and HIV nucleic acid detection tests was held in Geneva, Switzerland from 27 to 29 May 2019 where meeting participants gave significant technical input. Meeting participants: P.N. Akolkar, Office of Blood Research and Review, Center for Biologics Evaluation and Research (CBER), Food and Drug Administration (FDA) Silver Spring, Maryland, USA 1 ; H.P.W. Bayer, consultant, Weinheim, Germany; S. Best, consultant, Melbourne, Australia; S. Carmona, Department of Molecular Medicine and Haematology, University of the Witwatersrand, Johannesburg, South Africa; C.I. Chime, Institute of Human Virology, Abuja, Nigeria; E. Ivanova, FIND, Geneva, Switzerland; S. Kamili, Division of Viral Hepatitis, Centers for Disease Control and Prevention (CDC), Atlanta, USA; J. Kress, Paul-Ehrlich-Institut, Langen, Germany; W. Leelawiwat, Thailand Ministry of Public Health-U.S. Centers for Disease Control and Prevention Collaboration, Nonthaburi, Thailand; P. Musasa Ncube, National Microbiology Reference Laboratory, Harare, Zimbabwe; J. Parry, consultant, Middlesex, United Kingdom; J. Saldanha, consultant, California, USA; S.M. Samiee, Reference Health Laboratory, Ministry of Health & Medical Education, Tehran, The Islamic republic of Iran; A. Tanuri, Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro, Brazil; K. Whitaker, Office of In Vitro Diagnostics and Radiological Health, Center for Devices and Radiological Health (CDRH), Food and Drug Administration (FDA) Silver Spring, Maryland, USA2; C. Zeh, Division of Global HIV & TB, Centers for Disease Control and Prevention (CDC), Atlanta, USA. WHO secretariat: M. Lanigan; A.L. Page; I. Prat; U. Ströher, In vitro diagnostics assessment Team, Prequalification Unit; R. Baggaley and L. Vojnov, Global HIV, Hepatitis and STI Department. Public comments were received for consideration from: the TC WG2 secretary on behalf of Global Harmonization Working Party (GHWP) (formerly Asian Harmonizatoin Working Party (AHWP)) TC working group 2; Alere Technologies GmbH, Jena, Germany; Agência Nacional de Vigilância Sanitária (ANVISA), Brasilia, Brazil; Center for Biologics Evaluation and Research (CBER), Food and Drug Administration (FDA) Silver Spring, Maryland, USA; Cepheid, Sunnyvale, California, USA; Global HIV, Hepatitis and STI Department, WHO, Geneva, Switzerland; and ISO/TC 212 Secretary on behalf of ISO/TC 212, International Standards Organization, Geneva, Switzerland. The document was developed with support from the Bill & Melinda Gates Foundation and UNITAID. 1 Participated via conference call 2 Participated via conference call TSS 11 Page | v Abbreviations ANOVA analysis of variance CI confidence interval EDTA ethylenediaminetetraacetic acid CRF circulating, recombinant form of HIV-1 IFU instructions for use IS International Standard IVD in vitro diagnostic IU international units LOD limit of detection ULOQ/LLOQ upper/lower limit of quantitation NA nucleic acid NAT nucleic acid amplification technology POC point of care ROC receiver operated curve spp. several species TSS Technical Specifications Series US FDA U.S. Food and Drug Administration WHO World Health Organization TSS 11 Page | 1 1 Introduction The purpose of this document is to provide technical guidance to in vitro diagnostic (IVD) medical device manufacturers that intend to seek WHO prequalification of quantitative nucleic acid amplification technology (NAT) tests for the quantitative detection of HIV-1 nucleic acid. For the purpose of this document, the verbal forms used follow the usage described below: • “shall” indicates that the manufacturer is required to comply with the technical specifications. • “should” indicates that the manufacturer is recommended to comply with the technical specifications, but it is not a requirement. • “may” indicates that the technical specifications are suggested methods to undertake the testing, but not requirements. A documented justification and rationale shall be provided by the manufacturer when the WHO prequalification submission does not comply with the required technical specifications outlined in this document. Where possible, WHO analytical and clinical performance study requirements are aligned with published guidance, standards and/or regulatory documents. Although references to source documents are provided, in some cases WHO prequalification has additional requirements. A full list of the individual studies is provided in chapter 5 (parts 1-3). • Part 1 lists the analytical performance studies that are required to assess the ability of the IVD to measure the relevant analyte(s) or measurand. • Part 2 lists the clinical performance studies that are required to support the clinical performance of an IVD and demonstrate that reasonable steps have been taken to ensure that a properly manufactured IVD, being correctly operated in the hands of the intended user, will detect the target analyte and fulfil its indications for use. • Part 3 lists the usability studies that are only required for IVDs with the intended use for testing at or near point of care (POC) by healthcare professionals trained in use of the test. For WHO prequalification purposes, manufacturers shall provide evidence in support of the clinical performance of an IVD to demonstrate that reasonable steps have been taken to ensure that a properly manufactured IVD, being correctly operated in the hands of the intended user, will detect the target analyte consistently and fulfil its indications for use. WHO prequalification requirements summarized in this document do not extend to the demonstration of clinical utility, i.e., the effectiveness and/or benefits of an IVD, relative to and/or in combination with other measures, as a tool to inform clinical intervention in a given population or healthcare setting. To demonstrate clinical utility, a separate set of studies is required. Clinical utility studies usually inform programmatic strategy and are thus the responsibility of programme managers, ministries of health and other related bodies in individual WHO Member States. Such studies do not fall under the scope of WHO prequalification. TSS 11 Page | 2 2 How to apply these specifications For purposes of WHO prequalification, HIV-1 assays with a claim to determine viral load (quantitative tests) shall comply with the specifications in Part 1 and Part 2 of this document. Part 3 only applies if, according to the IFU, testing is performed by health professionals trained in the use of the test in near POC settings. Semi-quantitative assays 3 shall be validated using the HIV-1 analytical performance studies described in Part 1 of TSS-12: “In vitro diagnostic medical devices used for the qualitative detection of HIV-1 and HIV-2 nucleic acid”. Depending on the intended use and claims of the semi-quantitative assay, the clinical performance studies described in this document (TSS-11) or TSS-12 might apply. Quantitative assays also seeking an HIV diagnostic claim, shall be validated using the HIV-1 requirements of TSS-12: “In vitro diagnostic (IVD) medical devices used for the qualitative detection of HIV-1 and HIV-2 nucleic acid” in addition to validation studies outlined in this document. 3 Other guidance documents This document should be read in conjunction with other WHO guidance documentation, including: WHO prequalification documents:4 • Technical Guidance Series for WHO Prequalification – Diagnostic Assessment. • Instructions for Compilation of a Product Dossier, WHO document PQDx_018. WHO Global HIV, Hepatitis and STI Department documents: • Consolidated guidelines on the use of antiretroviral drugs for treating and preventing HIV infection: recommendations for a public health approach, 2016.5 • HIV molecular diagnostics toolkit to improve access to viral load testing and infant diagnosis, 2019.6 4 Performance principles for WHO prequalification 4.1 Intended use An IVD submitted for WHO prequalification assessment shall be accompanied by a sufficiently detailed intended use statement. This should allow an understanding of at least the following: • The type of assay • What is detected or measured 3 Semi-quantitative assays are assays that do not give a result for the concentration of a given analyte in numerical values, such as 50 IU/mL or 2,000 genome equivalents/mL, but give an indication of the relative concentration of the analyte by e.g., reporting the Ct value for real-time PCR assays (which is inversely proportional to the concentration of the analyte 4 Available at: https://extranet.who.int/pqweb/vitro-diagnostics/guidance-documents 5https://apps.who.int/iris/bitstream/handle/10665/208825/9789241549684_eng.pdf;jsessionid=0BD31F601096C7EABEB7 FBE6506D9EC9?sequence=1 6https://apps.who.int/iris/bitstream/handle/10665/325961/9789241516211-eng.pdf?ua=1 TSS 11 Page | 3 • The clinical indication and function of the IVD (e.g., to quantitate RNA from individuals infected including all known subtypes of HIV-1, groups M, including the most prevalent, circulating recombinant forms (CRFs), such as A/E, A/G, B/G, B/F, HIV-1 groups N and O, and to monitor treatment of patients on anti-retroviral therapy) • The target population • The intended use environment and user (e.g., for professional use in a laboratory setting, and/or near POC7) • The intended specimen types (e.g., capillary whole blood, EDTA plasma etc.) • What the IVD reports (e.g., qualitative test, semi-quantitative, quantitative test) • Whether or not it includes automated components or is intended to be used with automated instruments • Any limitations to the intended use (e.g. identification or restrictions regarding age groups or other limiting characteristics) 4.2 Diversity of specimen types, users and testing environments and impact on required studies For WHO prequalification submission, clinical performance studies shall be conducted using the specimen types (e.g., plasma, serum, capillary or venous whole blood, dried blood spot (DBS) specimens) that are claimed in the instructions for use (IFU). For DBS specimens, the validated brand(s) of filter paper shall be stated in the IFU. Prequalified HIV NAT assays in low- and middle-income countries are likely to be used by a range of users in different geographical regions: • laboratory professionals 8 either in centralised testing laboratories or at/near POC, • laboratory professionals in health care settings not experienced in nucleic acid testing, • health professionals trained in the use of the test at or near POC. Depending on the intended use of an IVD, analytical and clinical performance studies shall be designed to consider not only the diversity of knowledge and skills across the population of IVD users, but also the likely operational settings in which testing will occur. It is a manufacturer’s responsibility to ensure that the risk assessment for an IVD reflects the intended operational settings, including laboratory or service delivery complexity, user expertise, training received and testing population. 4.3 Applicability of supporting evidence to IVD under review Analytical and clinical performance studies shall be undertaken using the specific, final (locked-down) version of the assay intended to be submitted for WHO prequalification. For 7 In some juridictions, the concept “near-patient testing” is used instead of “point of care testing”. Either term may be used in the intended use statement. 8Medical technologists, medical laboratory technicians or similar, who have received a formal professional or paraprofessional certification or tertiary education degree. TSS 11 Page | 4 WHO prequalification, design lock-down is the date that final documentation, including quality control and quality assurance specifications, is signed off and the finalized method is stated in the IFU. Where this is not possible, a justification shall be provided, and additional supporting evidence may also be required. This may occur in the case of minor variations to design where no impact on performance has been demonstrated (see WHO document PQDx_121 Reportable Changes to a WHO Prequalified In Vitro Diagnostic Medical Device).9 If the protocol section of the IFU has been changed in any way, both the protocol provided to laboratory for clinical performance studies as outlined in Part 2 of this document and that in the final version of the IFU intended for users shall be provided with the submission for WHO prequalification assessment. The version of the IFU used for verification and validation studies submitted for WHO prequalification assessment shall be stated. If the test procedure in the IFU is changed in any way after completing performance verification and validation studies the change(s) shall be reported to WHO, including a rationale for the change, and an explanation of why the study results support the claimed performance. Specific information is provided in this document for the minimum numbers of lots required for each study. Where more than one lot is required, each lot shall comprise different production (or manufacturing, purification, etc.) runs of critical reagents, representative of routine manufacture. It is a manufacturer’s responsibility to ensure, via risk analysis of its IVD that the minimum numbers of lots chosen for estimating performance characteristics considers the variability in performance likely to arise from the interlot diversity of critical components and their formulation or from changes that could occur during the assigned shelf life of the IVD. Differences found between lots during the analytical and clinical performance studies shall be reported. Where the manufacturer supplies additional instrumentation required to conduct the assay, safety and performance data shall be provided in the dossier for all instrumentation supplied by the manufacturer and required to conduct the assay. The accurate quantitation of HIV-1 RNA shall be determined using a suitable reference method that has been calibrated against the WHO International Standard for HIV-1 RNA. For WHO purposes this should be a method that currently is at a developed stage of technical capability based on the relevant consolidated findings of science, technology and experience (commonly referred to as state of the art). Justification for the choice of method, shall be provided such as a laboratory- based assay that has undergone comprehensive pre-market assessment and has been stringently regulated and approved by a recognized regulatory authority.10 Estimation (and reporting) of IVD performance shall include the rate of invalid test results and the two-sided 95% confidence interval around the estimated values for key performance metrics. The cause of the invalid results should be reported if available (such as sample issues e.g., age of sample, storage conditions, inadequate sample volume), instrument hardware 9 http://apps.who.int/iris/bitstream/handle/10665/251915/WHO-EMP-RHT-PQT-2016.01- eng.pdf;jsessionid=30D5BF0B09FFDA3B38A1698E65C8B496?sequence=1 10 See document PQDx_173 Abridged Assessment for the list of recognized stringent assessments by regulatory authorities TSS 11 Page | 5 and/or software error, operator error. Data should be presented in clear and understandable format. It is unlikely that clinical specimens will be available in the volumes required for all analytical performance studies. Therefore, it is acceptable to use contrived specimens, for example, cell culture-derived virus that is characterized and sequenced, or a clinical specimen spiked into the appropriate matrix, i.e., a matrix that has been claimed in the intended use of the IVD (e.g., human plasma, whole blood), and which has been prepared in a validated and standardized manner for such analytical performance studies. In addition, dilutions of a high- concentration clinical specimen may be used, if they are in an appropriate matrix (e.g., plasma, whole blood) for certain studies such as limit of detection (LOD) studies. The material chosen should use the entire assay system from specimen preparation to interpretation. For certain analytical performance studies in part 1 of this document, it may be acceptable to generate evidence of performance using a single subtype of HIV-1 group M as a surrogate for performance of other subtypes claimed in the intended use of the device. A justification for this approach shall be provided, for example, by providing data to demonstrate that the primers and/or probes chosen are effective for all subtypes or variants claimed in the IFU. Furthermore, for analytical performance studies described in part 1 it may be also possible to carefully design protocols that will generate useful data for more than one of the required studies, provided the specific criteria for each requirement are met by the study (e.g., number of replicates, concentration of analyte, specimen types, etc.). For example, precision testing and whole system failure testing could be combined in a single study. Studies which may fall in this category are indicated in the appropriate sections of part 1. If the validation of specimens (section 1.2.1) shows equivalency between specimen types, some analytical performance studies (as indicated in this document) may use a representative specimen type only. However, if no equivalence between specimen types is shown, validation shall be conducted in all specimen types where indicated in the TSS. Clinical performance studies shall be based on testing human specimens only sourced from population cohorts reflective of the intended use. Independent of the outcome of the equivalency study (section 1.2.1), all claimed specimens types need to be considered in the clinical performance study, unless otherwise stated in part 2. The use of well-characterised repository specimens and panels may be acceptable if they are relevant to the IVD under assessment, taking into consideration: • storage conditions (e.g., including age of the specimen, temperature logs, freeze- thaw cycles if applicable); • the stability of the nucleic acid target; • selection bias. Studies that comprise the testing of left-over specimens by research and development staff at a manufacturer’s facility shall not, on their own, be considered sufficient to meet the clinical performance study requirements summarized in this document. TSS 11 Page | 6 5 Table of Requirements PART 1 ANALYTICAL PERFORMANCE AND OTHER EVIDENCE 1.1 Stability of specimens(s) 1.1.1 Specimen collection, storage and transport 1.2 Validation of specimens 1.2.1 Demonstration of equivalence of specimen types 1.3 Metrological traceability of calibrator and control material values 1.3.1 Metrological traceability of calibrators and control material values 1.4 Accuracy of measurement 1.4.1 Trueness 1.4.2 Precision (repeatability and reproducibility) 1.5 Analytical sensitivity 1.5.1 Limit of detection 1.5.2 Limits of quantitation 1.6 Analytical specificity 1.6.1 Potentially interfering substances and medical conditions 1.6.1.1 Endogenous 1.6.1.2 Exogenous 1.6.2 Cross-reactivity 1.7 Measuring range of the assay 1.7.1 Linearity 1.8 Validation of the assay procedure 1.8.1 Validation of primer and probe choice 1.8.2 Whole system failure rate 1.8.3 Carry-over contamination 1.9 Usability/human factors 1.9.1 Flex studies/robustness 1.9.2 Software validation 1.10 Stability of the IVD 1.10.1 Claimed shelf-life (including shipping stability) 1.10.2 In-use stability 1.11 Performance panels 1.11.1 Subtype panels PART 2 CLINICAL EVIDENCE (CLINICAL PERFORMANCE CHARACTERISTICS) 2.1 Clinical sensitivity and specificity 2.1.1 General requirements for clinical sensitivity and specificity studies 2.1.2 Clinical sensitivity 2.1.3 Clinical specificity PART 3 QUALIFICATION OF USABILITY FOR NEAR POC TESTING BY HEALTHCARE PROFESSIONALS 3.1 Qualification of usability for near POC testing by healthcare professionals 3.1.1 Label comprehension (including IFU) 3.1.2 Results interpretation TSS 11 Page | 7 Part 1: Analytical performance and other evidence Aspect Testing requirements Notes on testing requirements Source Documents 1.1 Stability of specimens(s) 1.1.1 Specimen collection, storage and transport 1. Real time studies shall be conducted for each claimed specimen type considering: 2. Storage conditions (for example, duration at different temperatures, temperature limits, freeze/thaw cycles etc.). For DBS specimens, humidity shall also be assessed 3. Transport conditions 4. Intended use (see note 1) 5. Specimen collection and/or transfer devices intended to be used with the IVD 6. The testing panel shall contain (see note 2, 3) • 10 discrete weak positive specimens (approx. 3 x LLOQ) for each specimen type • The testing panel shall include 1 isolate of HIV-1 group M 7. Testing shall be conducted in 1 lot 1. Evidence shall be provided which validates the maximum allowable time between specimen collection and its processing or addition to the IVD in the setting where testing takes place 2. Contrived specimens may be used 3. In case the use of archived specimens is considered for Part 2 of this document, evidence of stability in the conditions in which the specimens have been stored shall be demonstrated e.g., by re-testing a subset of specimens with an approved test to verify that the same result is obtained compared to the sample result prior to freezing 4. Acceptance criteria shall confirm that claimed specimen types transported, processed and stored under recommended conditions will give expected results. Separated EDTA plasma and centrifuged whole blood in a plasma preparation tube are considered different specimen types in this context 5. If dried blood spots (DBS) are a claimed specimen type, the details of the filter paper (brand, product code) shall be specified and the use and stability validated 6. Unless all specimens are expected to be processed as fresh samples within a specified time frame, the IVD performance shall be established for each storage condition at the beginning and end of the stated period TGS 3 (1) TSS 11 Page | 8 Aspect Testing requirements Notes on testing requirements Source Documents 1.2 Validation of specimens 1.2.1 Demonstration of equivalence between specimen types 1. The relationship between IVD performance in claimed specimen types shall be established: 2. At least 25 positive and 25 negative specimens shall be tested for each claimed specimen type (see notes 1-4) 3. If testing is claimed in specimens other than blood derived specimens, then testing shall be conducted in at least 50 paired specimens (positive and negative) 4. For assays that detect more than 1 subtype but do not differentiate the subtype, the justification as to whether to study each subtype shall be risk- and evidence-based 5. 1 replicate of each specimen for each specimen type shall be tested 6. Testing shall be conducted in 1 lot 1. If multiple specimen types are claimed, (e.g., serum, plasma, venous whole blood, capillary whole blood; DBS), then equivalence shall be demonstrated using paired specimens; in other words, demonstrated in each specimen type for all specimens 7. Similarly, if equivalence is claimed in specimens collected into multiple anticoagulants, each specimen shall be collected into each of the claimed anticoagulants 8. Specimens should be chosen that have low to moderate concentrations of the analyte (including levels of analyte at relevant medical decision points) 9. It is known that IVDs may show different sensitivities between specimen types 10. Analytical sensitivity studies (see section 1.5) may also contribute to evidence regarding equivalence of specimen types 11. The established relationship between IVD performance in claimed specimen types (plasma, blood, DBS) shall be taken into account in the design of subsequent studies. For example, if the studies show that 2 or more of the claimed specimen types are equivalent, then not all specimen type need to be tested in some of the subsequent studies (where indicated) IMDRF IVD MA ToC (2) TSS 11 Page | 9 Aspect Testing requirements Notes on testing requirements Source Documents 1.3 Metrological traceability of calibrator and control material values 1.3.1 Metrological traceability of calibrator and control material values 1. As applicable, the traceability of the provided calibrator and control materials to a validated reference material shall be demonstrated (e.g., WHO International Standard (IS) for HIV-1) or a secondary standard calibrated from it 1. The version of the IS used shall be stated 2. Appropriate to the IFU, the low positive external control should contain target nucleic acid at levels approximately 3 x LLOQ. The high positive external control should contain target nucleic acid at levels in the upper half of the linear range of the assay 3. An internal control shall be added to each specimen before sample extraction so that all stages of the test, from extraction to final target detection can be verified. An internal control consists of defined nontarget sequences of the same type of nucleic acid, (RNA or DNA) as the target, which are extracted and amplified simultaneously with the test sample. Therefore, the test should be able to clearly identify the amplified products (amplicons) of the internal control and the target 4. Some quantitative assays may include an internal calibrator which is assayed simultaneously with the target to allow for quantitation of specimens based on algorithms that compare signal generated from the internal calibrator and target sequence to a pre-generated standard curve. In such cases, the establishment of the stored/pre-generated standard curve shall be validated. If calibration constants are used, the interval between recalibration of the pre-generated standard curves and calibration constants shall also be validated WHO (3) MM06-A2 (4) European Pharmacopoeia (5) TGS-6 (6) TSS 11 Page | 10 Aspect Testing requirements Notes on testing requirements Source Documents 1.4 Accuracy of measurement 1.4.1 Trueness The trueness of the IVD shall be demonstrated by comparison of the performance of the IVD with an established quantitative, HIV-1 method (see note 1). 1. The following specimen panel shall be tested by the IVD and the reference method: • A total of at least 100 HIV-1 samples with viral loads covering the entire linear range of the IVD and including as many different HIV-1 subtypes as possible 2. A minimum of 2 lots shall be used for testing 1. The reference method shall be authorized for use by a recognized stringent regulatory authority and acknowledged in the literature as representing state of the art 3. Either contrived specimens with varying viral loads or clinical specimens may be used 4. Correlation of results between the IVD and the established method shall be demonstrated statistically 5. Trueness may also be established during the clinical performance study in part 2 of this document CLSI EP09C 2018-A (7) CLSI EP15-A3 (8) 1.4.2 Precision (repeatability & reproducibility) 1. Both repeatability (see note 1) and reproducibility (see note 2) shall be estimated using panels with defined analyte levels. 2. The members of the repeatability and reproducibility panel should include (see note 3 and 4): • 1 x negative specimen • 1 x low positive specimens (approx. 3 x LLOQ) • 2 specimens in the middle and upper range of the linearity span 3. Each panel member shall be tested: • in 5 replicates • using 3 different lots (see notes 7 & 10) • over 5 days (not necessarily consecutive) with 1 run in that day (alternating morning/afternoon) • at each of 3 different testing sites (see note 6, 10) • by 1 operator/site (see note 5, 6) 1. Within run 2. Between -run, -lot, -day, -site, operator • Note: A run will be defined depending on the IVD’s throughput: if the platform can accommodate all specimens in a single run, i.e., in the same test plate, the replicates will be run together. If the assay can only accommodate a smaller set or a single specimen(s), a run will be defined as a testing session carried out on the same instrument/module 3. Only a single subtype of HIV-1 groups M is required to be tested 4. Specimens with target levels of analyte may be contrived 5. The testing panel should be the same for all operators, lots and sites 6. If operators are considered a significant source of test result variation (for example, with tests that have a significant TGS 3 (1) CLSI EP05-A3 (9) TSS 11 Page | 11 Aspect Testing requirements Notes on testing requirements Source Documents • by operators representative of intended users • unassisted • using only those materials provided with the IVD (e.g., IFU, labels and other instructional material 4. All claimed specimen types shall be tested. If equivalence between claimed anticoagulants has been demonstrated, only 1 anticoagulant is required to be tested. proportion of manual manipulations), then at least 2 operators/site shall be used 7. Each lot shall comprise different production (or manufacturing, purification, etc.) runs of critical reagents, representative of routine manufacture 8. The number of invalid tests shall be reported 9. Results shall be statistically analyzed by ANOVA or other methods to identify and isolate the sources and extent of any variance. In addition, the percentage of correctly-identified, incorrectly-identified and invalid results shall be tabulated for each specimen and be separately stratified according to each of site, lot, etc. 10. To understand irregularities in results obtained, at least 2 lots should be tested at each of the 3 testing sites 11. Alternative methods used to establish repeatability and reproducibility performance of the assay shall be discussed with WHO in advance of dossier submission 1.5 Analytical sensitivity 1.5.1 LOD Analytical sensitivity as expressed by LOD, shall be determined as follows: 1. Testing 20-24 replicates of at least 8 serial 0.5log10 dilutions of suitable biological reference materials (e.g., WHO 1st IS for HIV-1 group M, subtype B) or a secondary standard calibrated against the appropriate IS (see note 2). The serial dilutions shall be chosen so that the RNA concentration span the LOD of the IVD 1. Analytical sensitivity shall be estimated by determining the lowest concentration for which the rate of detection is 95% (95% LOD) with 95% confidence intervals (CI) (e.g., by Probit analysis) taking into account lot to lot variation 2. The WHO IS for HIV-1 group M or a secondary standard calibrated against the appropriate IS shall be used 3. For other HIV-1 groups for which an IS is not available, such as HIV-1 group O and HIV-1 group N, either the reference CLSI MM06-A2 (4) European Commission decision on CTS (10) CLSI EP17-A2 (11) TSS 11 Page | 12 Aspect Testing requirements Notes on testing requirements Source Documents 2. The replicate testing shall be conducted on 3 different days (see note 4) (8 replicate tests on each day) 3. Using 2 lots 4. At least 2 dilution series shall be tested 5. The analytical sensitivity shall be determined for all claimed specimen types 6. LOD shall be determined for each HIV-1 group for which claims are made (see note 3) material shall be quantitated with a suitable HIV-1 quantitative assay, authorized for use by a recognized stringent regulatory authority or if an approved quantitative assay is not available, e.g., for HIV-1 group N, a log dilution series of each specimen shall be tested 4. For low through-put instruments, the number of testing days may be increased 5. Where an IS is available for a particular analyte, the sensitivity of the IVD for that analyte shall be expressed in International Units for this analyte 6. The relationship between analytical sensitivity for the different claimed specimen types (e.g., EDTA plasma and whole blood) shall be established. The design of subsequent analytical performance studies shall take that relationship into account to ensure that results of different specimen types, when included in the study, can be evaluated TSS 11 Page | 13 Aspect Testing requirements Notes on testing requirements Source Documents 1.5.2 Limits of quantitation Lower and upper limits of quantification (LLOQ, ULOQ) shall be established for HIV-1 group M. 11 1. For the LLOQ determination, a minimum of 15 (3 days, 5 replicates/day) replicate tests of a multi-member dilution panel of a suitable biological reference material (e.g., WHO HIV-1 IS or a secondary standard calibrated against it) shall be tested 2. For the ULOQ, determination, a dilution series prepared from a high-titre clinical specimen or cell-culture derived HIV-1 isolate shall be tested in a similar manner (see note 1) 3. The concentrations of the dilution panel shall go beyond the claimed LLOQ and ULOQ 4. Using 2 lots 5. LLOQ and ULOQ shall be estimated by determining the lower and upper concentrations that can be determined within the accuracy expected (predefined) (see note 2) 6. For subtypes/circulating recombinant forms/groups other than HIV-1 group M, genotype B from which the IS is prepared, LLOQ shall be confirmed for HIV-1 subtypes/groups and circulating recombinant forms 1. In order to determine the ULOQ accurately, it may be necessary to use a high titre parent material (clinical specimen or cell culture-derived virus) calibrated against the IS and spiked into plasma to obtain large volumes of high titre material 2. Predefined criteria for acceptable accuracy (precision & trueness) at the LLOQ and the ULOQ shall be provided 3. The version of the IS used shall be stated 4. For HIV-1 groups for which an IS is not available, such as HIV- 1 group O and HIV-1 group N, either the reference material shall be quantitated with a suitable HIV-1 quantitative assay, authorized for use by a recognized stringent regulatory authority or if an approved quantitative assay is not available, e.g., for HIV-1 group N, a log dilution series of each specimen shall be tested European Pharmacopoeia (5) CLSI EP05-A3 (9) European Commission decision on CTS (10) CLSI EP17-A2 (11) PQDx_018 (12) 11 Limit of quantitation (LoQ): the lower and upper concentrations at which precision & trueness are within specified criteria TSS 11 Page | 14 Aspect Testing requirements Notes on testing requirements Source Documents (CRFs) for which claims are made. LLOQ shall be determined as follows: • At least 4 specimens of the predominant subtypes circulating in the populations for which claims are made, if available, shall be tested • At least 2 specimens of all the other claimed subtypes/groups, if available, shall be tested (see note 4) • Dilutions of the test samples at 0.5 x LLOQ, at the LLOQ, and at 3 x LLOQ shall be tested • 10 replicates/dilution shall be tested • Using 2 lot 7. All claimed specimen types shall be tested 1.6 Analytical specificity 1.6.1 Potentially interfering substances and medical conditions The potential for false results arising from interference by the substances/conditions listed below shall be determined. 1. Testing confirmed HIV-negative specimens (see note 4) both spiked (using the predominant HIV-1 subtype M) and unspiked 2. A minimum of 100 specimens shall be tested 3. Testing a minimum of 5 – 10 specimens per substance/condition 4. Spiking with HIV-1 at approx. 3 x LLOQ 5. Testing shall be conducted in 1 claimed specimen type (see note 7) 1. The risk assessment conducted for an IVD shall identify substances at medically relevant levels for which the potential for interference can reasonably be expected for the analyte being detected in the areas of intended use and not simply rely on published lists of such compounds and conditions which might be of limited relevance in resource limited settings • By conducting appropriate risk assessment, testing can be conducted on specimens spiked with the substances/ European Commission decision on CTS (10) CLSI EP07 (13) CLSI EP37 (14) FDA (15) TSS 11 Page | 15 Aspect Testing requirements Notes on testing requirements Source Documents 1.6.1.1 Endogenous The interference of endogenous substances in human plasma on the performance of the device shall be investigated. Endogenous substances shall be spiked at abnormally high levels compared with healthy individuals such as: Triglycerides, haemoglobin, unconjugated bilirubin, albumin Human genomic DNA (see note 1) Autoimmune diseases/markers • Anti-Nuclear Antibodies • Rheumatoid factor • Systemic lupus erythematosus conditions identified as likely to be significant and testing of potentially irrelevant substances/conditions avoided • Under some circumstances stringent risk evaluation may eliminate the requirement to test some of the items in the lists but any such decision shall be documented in any submissions to WHO and taken into account in the risk- benefit statements 2. Any observed interference/crossreactivity shall be further investigated and performance limitations of the IVD reported in the IFU 3. Results shall be reported with respect to each condition and not be reported as an aggregate of the total number of specimens tested in the study 4. Prior to spiking, specimens should be confirmed to be HIV- negative prior to testing with a suitable HIV-1 qualitative or quantitative assay, authorized for use by a recognized stringent regulatory authority 5. Where clinical specimens from individuals are unavailable, a negative specimen shall be spiked with the microbe of interest. Cross reactivity with other microbes shall be tested at the highest concentration available (a minimum of 105 plaque forming units/mL for viruses and 106 colony forming units/mL for bacteria) 6. Where either the scientific literature and/or risk analysis identifies the potential for false results in co-infected individuals (e.g., decreased sensitivity or specificity), further 1.6.1.2 Exogenous The interference of exogenous substances on the performance of the device shall be investigated. Exogenous substances shall be spiked at >3 times the peak plasma concentration levels such as 1. Common over the counter analgesia medications (aspirin, paracetamol, ibuprofen) 2. Medicines, relevant to the populations intended to be tested including treatment of HIV, hepatitis C, B, tuberculosis, malaria, herpes simplex virus 3. Medicines used to treat infections common in the region where the IVD will be used 4. Medicines or biologicals that increase circulating nucleic acid (see note 1) TSS 11 Page | 16 Aspect Testing requirements Notes on testing requirements Source Documents 5. Presence of nucleic acid-based medicines and metabolites and binding substances (see note 1) investigation shall be undertaken using both HIV-negative and HIV-positive specimens 7. Other organisms may be needed to be tested in the relevant specimen type when specimen matrices other than whole blood, serum and plasma are claimed. An individual risk assessment shall be conducted for these specimen types 8. In-silico analysis may be performed as part of the risk assessment but does not replace wet-lab testing 1.6.2 Cross-reactivity Determination of the potential for false results arising from cross-reactivity with other organisms unrelated to HIV or disease states commonly found in regions of the intended use of the IVD, including, where possible, at least 3-5 each of: 1. Potential cross-reacting organisms found in the blood (including viruses, bacteria, parasites & fungi) (notes 1, 2, 5, 7, 8) • Plasmodium spp • Leishmania • Trypanosoma cruzi • Trypanosoma brucei • Chikungunya virus • Hepatitis A, B, C • Cytomegalovirus • Epstein-Barr virus • Herpes simplex 1 & 2 • HTLV I & II • Mycobacterium tuberculosis • Staphylococcus aureus • Propionibacterium acnes • Staphylococcus epidermis • Staphylococcus haemolyticus • Yeast infections (e.g., Candida albicans) • Pneumocystis TSS 11 Page | 17 Aspect Testing requirements Notes on testing requirements Source Documents 2. Using 1 claimed specimen type 1.7 Measuring range of the assay 1.7.1 Linearity The linear range shall be established using: 1. A dilution series with a minimum of 7 concentrations that span and is 20 to 30% wider than the expected upper and lower limits of the measuring range shall be tested 2. At least 3 replicates shall be tested at each concentration 3. Using 1 lot 4. All claimed specimen types shall be tested 5. All major subtypes/CRFs that are predominant in populations for which claims are made and all other subtypes, if available, that are claimed in the IFU shall be included in the dilution series to establish the upper part of the measuring range 1. The parent material used to make the dilution series should be quantitated by testing multiple replicates with a suitable HIV-1 quantitative assay authorized for use by a recognized stringent regulatory authority 2. The upper part of the measuring range may be established using dilution series of cell-culture derived HIV-1 isolates or a high titre clinical specimen 3. Data for establishing the linear range may be taken from analytical sensitivity estimations (section 1.5), providing all required concentrations of HIV-1 RNA are tested 4. The test results shall be analyzed using appropriate statistical tools (e.g., Deming Regression Analysis) to demonstrate correlation between the IVD results and the nominal concentrations of the analyte CLSI EP06-A (16) 1.8 Validation of the assay procedure 1.8.1 Validation of primer and probe choice 1. For each claimed analyte (subtype), evidence supporting the choice of critical reagents (primers and probes sequences) shall be provided 1. A rationale for selection of primers and probes including specific sequences used to be provided, including: • Justification for alignments made to generate consensus sequences or best-fit modifications made to existent sequences e.g., to permit maximum homology to several strains, and • Information on size, GC content, melting temperatures, hairpin or other secondary structures if any, and the IMDRF IVD MA ToC (2) FDA (15) Stanford database (17) TSS 11 Page | 18 Aspect Testing requirements Notes on testing requirements Source Documents nucleotide position on the genome map of the primers and probes 2. For assays designed to detect or quantitate multiple HIV subtypes or variants, data should be provided to demonstrate that the primers and or probes chosen are effective for all subtypes or variants identified in the label 3. Evidence that HIV drug resistant mutation (see HIV drug resistance mutation Stanford database) have been taken into account in the design of primers and probes and that the location of the mutation does not affect amplification and or correct quantitation. 1.8.2 Whole system failure rate The potential for false negative results in low positive specimens shall be determined: 1. A panel of 20 specimens containing HIV-1 (approx. 3 x LLOQ) shall be tested (see note 1) 2. The panel shall be randomized and tested on • 5 consecutive days (to give a total of 100 test results) • Using 1 lot • With 1 user 3. The whole system failure shall be determined in the most viscous specimen type claimed, e.g., whole blood 1. This may be conducted as part of precision studies 2. Replicate contrived specimens should be prepared using a single subtype specimen diluted in the appropriate matrix. 1.8.3 Carry-over contamination The potential for false positive results due to carry-over shall be investigated. 1. The specimen panel shall contain 40 alternating high positive (≥ 106 IU or copies/mL) (see note 1) and negative specimens 1. Either clinical specimens (individual or pooled) or high titre cell culture derived virus diluted in the appropriate matrix shall be used. 2. If DBS specimens are a claimed specimen type and require manual punching/excision of the DBS prior to extraction, European Commission decision on CTS (10) Haeckel R (18) TSS 11 Page | 19 Aspect Testing requirements Notes on testing requirements Source Documents 2. Only 1 claimed subtype is required 3. Only 1 specimen type should be used. The specimen type with the highest viscosity (e.g., whole blood versus plasma) shall be chosen as the test specimen. In addition, DBS specimens should also be tested if claimed in the IFU (see note 2) 4. The panel shall be tested: • In at least 5 different runs • On 3 different days • By at least 2 users • Using 1 lot 5. For testing platforms that can only accommodate a single specimen, testing shall be conducted on a single instrument: • At least 4 tests per run • Using alternating high-positive (≥ 106 IU/mL or copies/mL) and negative specimens • A total of 10 runs • At least 2 users • Using 1 lot those manual handling steps shall also be evaluated in the carry-over contamination studies 3. Whether the IVD is a microtiter plate based system or a single use device, appropriate studies that will detect carry-over contamination should be designed. 1.9 Usability/human factors 1.9.1 Flex studies/ robustness Evidence is required to demonstrate that the conditions recommended in the IFU are validated and how they were verified. 1. The influence of the following factors on expected results (both detected and non-detected) shall be considered as applicable (see note 1, 2, 3, 4): 1. Refer to WHO document PQDx_018 “Instructions for compilation of a product dossier” for other flex studies that may be relevant, taking into consideration the broad range of operational and environmental conditions consistent with intended use PQDx_018 (12) IEC 62366-1:2015 (19) TSS 11 Page | 20 Aspect Testing requirements Notes on testing requirements Source Documents • Specimen and/or reagent volume • IVD instrument sturdiness (including the effect of non- level work surface) • Lighting, humidity and barometric pressure (simulating high altitude) • Handling contamination (e.g., from latex, powder, hand lotion, sweat, and/or soap, etc.) • Operating temperature 2. Instrumentation (both extraction and amplification) including: • Ruggedness (e.g., the effect of vibration from other instruments) (see note 5) • Impact of dust and mould on componentry (e.g., optics) • Impact of power/voltage fluctuation 3. Using 1 lot 4. Where different specimen types are claimed, flex studies shall use the most challenging specimen type (for example, whole blood) 5. Studies investigating the impact of specimen volume/specimen adequacy (e.g., DBS) shall be conducted in all specimen types 6. The specimen panel shall contain: • 1 negative specimen • 1 HIV-1 group M specimen (approx. 3 x LLOQ) 2. The risk assessment conducted for an IVD shall identify factors which have potential to affect the performance of the assay 3. The factors should be investigated in ways that not only reflect, but also exceed, likely operating conditions in lower- and middle-income countries so that the limitations of the device can be understood 4. Additional factors may be relevant for point-of-care devices or devices requiring significant manual interventions (e.g., manual DBS extraction). These factors may include errors during sample collection, sample handling and loading, and handling of relevant test components after sample application 5. For the purposes of this document, ruggedness means the ability to resist environmental shocks of a variety of kinds. 6. Robustness testing generally takes the form of statistically designed experiments to evaluate the effect of simultaneous “small but deliberate changes” in method parameters. 7. Since assay and analyzer parameters are locked down in a closed system and cannot be changed, there should be evidence that these parameters have been optimized TSS 11 Page | 21 Aspect Testing requirements Notes on testing requirements Source Documents 1.9.2 Software validation Software validation (including verification of built-in fail- safe and alert mechanisms) 1. If software is utilized for amplification, detection, and calculation of quantitative results, validation of such software for the intended function should be provided FDA (15, 20) 1.10 Stability of the IVD 1.10.1 Claimed shelf-life (including shipping stability) 1. Stability studies shall be evaluated for the shelf life of the test kit. The following conditions shall be investigated: • Conditions to mimic extremes of conditions (temperature, humidity, pressure) exposed to during transport • Storage temperature and humidity range • Operating temperature and humidity range 2. At least 3 lots (see note 3) shall be tested 3. The stability testing panel shall consist of the following contrived specimens: • 1 HIV-1 group M low positive specimen (approx. 3 x LLOQ) and medium positive specimen • 1 HIV-1 group O low positive specimen (approx. 3 x LLOQ) and medium positive specimen where applicable (see note 2) • 1 negative specimen 4. Each panel member shall be tested in at least triplicate at each time point/condition ((see note 1) 5. All claimed specimen types shall be tested (see exception note 6) 1. Justification for the number of replicates shall be based on the stability study set up, statistical analysis of the data and a prior knowledge of the assay’s performance 2. When more than 1 part of the genome is targeted by primers, each region shall be monitored separately during stability evaluation. If the assay contains more than 1 of each primer/probe combination, then each primer/probe combination needs to be assessed for stability 3. Each lot should comprise different production (or manufacturing, purification, etc.) runs of critical reagents, representative of routine manufacture (see chapter 4.3) 4. Experiments should be statistically designed to allow evaluation of any interactions between environmental conditions 5. The number of invalid tests with each kit lot shall be reported 6. If both venous and capillary whole blood is claimed, then only 1 of these specimen types need to be assessed. If equivalence between claimed anticoagulants has been demonstrated, only 1 anticoagulant is required to be tested 7. Claims for stability shall be based on the second-last successful data point from the least stable lot, with, if lots are different, a statistical analysis showing that the bulk of lots ISO 23640:2011 (21) CLSI EP25-A (22) TGS 2 (23) ASTM D4169-14 (24) TSS 11 Page | 22 Aspect Testing requirements Notes on testing requirements Source Documents will be expected to meet the claimed life. For example: for testing conducted at 3, 6, 9, 12 and 15 months, if stability was observed at 15 months, then the maximum stability claim can be 12 months 8. Determination of shipping stability shall be performed using simulated extreme stress conditions, ensuring that application of those conditions is consistent and controlled 9. Accelerated studies do not replace the need for real time studies 10. Multiple Instruments may be used to allow simultaneous testing at each time point 1.10.2 In-use stability (open pack or open vial stability) 1. Minimum of 1 lot shall be tested using a stability testing panel composed of (see note 3): • 1 negative specimen • 1 HIV-1 group M specimen low positive (approx. 3x LLOQ) and medium positive specimen 2. Replicate testing of each panel member (see note 1) 3. Testing shall be conducted using the most challenging specimen type (e.g., whole blood) 4. All labile components (e.g., buffers vials, sealed cartridges, etc.) shall be evaluated 5. On-board stability shall be tested for an IVD used with an instrument 1. Justification for the number of replicates shall be based on the stability study set up, statistical analysis of the data and a prior knowledge of the assay’s performance 2. When more than 1 part of the genome is targeted by primers, each region shall be monitored separately during stability evaluation. If the assay contains more than 1 of each primer/probe combination, then each primer/probe combination needs to be assessed for stability 3. Contrived specimens may be used 4. In-use stability of labile components shall be conducted using components in their final configuration TSS 11 Page | 23 Aspect Testing requirements Notes on testing requirements Source Documents 1.11 Performance panels 1.11.1 Subtype panels 1. The ability of the device to detect all claimed HIV-1 subtypes shall be demonstrated by testing well- characterized subtype panels available from commercial companies and Regulatory Authorities (e.g., NIBSC12) 2. The panels shall include (see note 1): 3. At least 10 specimens of the common HIV-1 subtypes and circulating recombinant forms • At least 10 specimens from groups other than HIV-1 group M • At least 3 specimens each of the less common subtypes 1. Suitable performance panels should consist of members which have been well characterized and quantitated by other recognized HIV molecular assays (see chapter 4.3) 2. The panel may comprise a combination of both clinical and cell culture –derived specimens diluted in plasma TGS 3 (1) 12 NIBSC: National Institute for Biological Standards and Control, United Kingdom https://www.nibsc.org/ TSS 11 Page | 24 Part 2: Clinical evidence (clinical performance characteristics) Aspect Testing requirements Notes on testing requirements Source documents 2.1 Clinical sensitivity and specificity 2.1.1 General requirements for clinical sensitivity and specificity studies Testing shall be conducted: 1. using specimens from all sections of the population for which claims are made in the IFU, e.g., adults and paediatric individuals 2. Using specimens from at least 2 different, geographically diverse regions 3. By a variety of intended users representing relevant intended use settings (e.g., primary, provincial) and at different test settings 4. Using at least 2 lots 5. Specimens shall be tested from individuals infected with all the common subtypes found in the region as well as any other claimed subtypes 6. Using all claimed specimen types (see exception note 12) 7. All specimens shall be tested by the reference assay (see notes 7, 8, 11) 8. Specimens with discrepant results shall be further evaluated. Where possible, follow-up testing shall be done (see note 3) 9. The procedure for selection of study specimens, how these represent the intended use population and how bias has been addressed shall be clearly described (see note 6) 1. Clinical performance shall be established using specimens that correspond directly to claims made in the IFU 2. All results shall be included in the final data analysis • Problematic specimens, those with unexpected results but which otherwise meet selection criteria for a study, shall not be systematically excluded from analysis 3. Discrepant results should be resolved as much as possible, however performance characteristics shall be based on the original result. Comparison with a similar device detecting the same genomic target is insufficient for resolution of discrepant results 4. All invalid results shall be recorded and evaluated in comparison to the reference result. Invalid results should be reported as individual categories (e.g., internal control failure, extraction failure, etc.) and not aggregated. Invalid results should be analyzed separately in the final performance calculations 5. Up to 25% of the test specimens with undetectable virus loads and 25% of test specimens with detectable/quantifiable virus loads may be well-characterised archived specimens, assuming that freezing specimens has been validated during analytical performance studies (see Section 1.1.1). The viral loads of the archived specimens should be reconfirmed European Commission decision on CTS (9) WHO (25) TSS 11 Page | 25 2.1.2 Clinical sensitivity and specificity 1. Testing shall be conducted using specimens from at least 1500 individuals on anti-retroviral therapy (ART) (see notes 5, 13) 2. Of these 1500 patients on ART, the viral loads in approx. 10% are expected to rebound resulting in approx. 1350 specimens with undetectable viral loads and 150 with detectable/ quantifiable viral loads 3. The total number of specimens with detectable/quantifiable viral loads shall be at least 200 consisting of: • Approx. 150 patients on ART (see above) • The remainder of positive specimens can be sourced from HIV positive patient prior to starting ART 4. The specimens shall be collected from at least 2 sites 5. The specimens shall be collected from unselected individuals on ART at each of the sites before testing using a test approved by a recognized regulatory authority. If > 25% of test specimens used in the clinical study are archived specimens, this shall be discussed with WHO in advance of submission 6. Criteria for the selection of specimens are required to be explained (e.g., testing of consecutive patients). In addition, any archived specimens used in the study shall be tested in a randomized, blinded manner interspersed with an appropriate number of negative specimens 7. The comparator/reference method shall be an assay as described in chapter 4.3 of this document 8. Sensitivity and specificity for detecting virological failure at the WHO threshold shall be estimated 9. Estimates of diagnostic sensitivity and specificity shall be reported with 95% confidence intervals 10. Clinical sensitivity and specificity shall be calculated for each specimen type and not for the aggregated data 11. Clinical performance study protocols shall specify how results in the IVD under evaluation and the comparator assay will be compared and how results in the 2 assays will be statistically determined to be equivalent or not (e.g., Bland Altman analysis) 12. If equivalence between specimens derived from venous whole blood (e.g., plasma collected in different anticoagulants, serum) has been demonstrated, then only 1 of the equivalent specimen types needs to be evaluated 13. Alternative clinical performance study designs to reach the required specimen numbers (undetectable viral load and detectable/quantifiable viral load) to establish clinical TSS 11 Page | 26 performance may be acceptable, but shall be discussed with WHO in advance of dossier submission TSS 11 Page | 27 Part 3: Qualification of usability for near POC testing by healthcare professionals Aspect Testing requirements Notes on testing requirements Source documents 3.1 Qualification of usability for near POC testing by healthcare professionals 3.1.1 Label comprehension (including IFU) 1. Testing of subjects to assess ability of intended users to correctly comprehend key messages from packaging and labelling that relate to near POC testing: • Understanding key warnings, limitations and/or restrictions, including correct methods and equipment • Proper test procedure • Test result interpretation 2. Studies shall include at least 15 intended users, users including those whose native language may not be the language of the IFU if necessary, to demonstrate comprehension of key messages in the user population 1. IFU and labelling should be clear and easy to understand. Use of pictorial instructional material is encouraged IEC 62366-1:2015 (19) Backinger CL and Kingsley PA (28) European Parliament IVD regulations (29) 3.1.2 Results interpretation 1. For near POC tests, intended users to interpret the results of contrived IVDs to assess their ability to correctly interpret pre-determined test results or error messages. Contrived tests should be made to demonstrate all potential test result interpretations. 2. Testing subjects to consist of at least 15 intended users including those whose native language may not be the language of the IFU if necessary, from at least 2 geographically diverse populations to demonstrate correct interpretation of simulated test results and error messages. 1. Study group may include subjects recruited as part of the label comprehension study TSS 11 Page | 28 6 References 1. World Health Organization; 2017. Licence: CC BY-NC-SA 3.0 IGO. World Health Organization. (2017). Principles of performance studies (TGS3). World Health Organization. https://apps.who.int/iris/handle/10665/258985. License: CC BY-NC-SA 3.0 IGO 2. In Vitro Diagnostic Medical Device Market Authorization Table of Contents (IVD MA ToC). IMDRF; 2019. http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-190321-ivd- mdma-toc-n13.pdf accessed 28 October 2021 3. World Health Organization & WHO Expert Committee on Biological Standardization (2016: Geneva, Switzerland). (2017). WHO manual for the preparation of secondary reference materials for in vitro diagnostic assays designed for infectious disease nucleic acid or antigen detection: calibration to WHO International Standards, Annex 6, TRS No 1004. https://apps.who.int/iris/handle/10665/255657. License: CC BY-NC-SA 3.0 IGO 4. Quantitative Molecular Methods for Infectious Diseases, Second Edition. CLSI MM06-A2. Wayne, PA: Clinical and Laboratory Standards Institute; 2010. 5. European Pharmacopoeia vol. 20, no. 3 20621 2008. 2.6.21 Nucleic Acid Amplification Techniques. 6. World Health Organization. (2017). Panels for quality assurance and quality control of in vitro diagnostic medical devices (TGS6). World Health Organization. https://apps.who.int/iris/handle/10665/259408. License: CC BY-NC-SA 3.0 IGO 7. Measurement Procedure Comparison and Bias Estimation Using Patient Samples; Approved Guideline - Third Edition. CLSI EP09c-A3. Wayne, PA: Clinical and Laboratory Standards Institute; 2018. 8. User Verification of Precision and Estimation of Bias, Third Edition. CLSI EP15-A3. Wayne, PA: Clinical and Laboratory Standards Institute; 2014. 9. Evaluation of Precision of Quantitative Measurement Procedures; Approved Guideline - Third Edition. CLSI EP05-A3. Wayne, PA: Clinical and Laboratory Standards Institute; 2019. 10. 2009/886/EC: Commission Decision of 27 November 2009 amending Decision 2002/364/EC on common technical specifications for in vitro diagnostic medical devices (notified under document C(2009) 9464) (Text with EEA relevance). 11. Evaluation of Detection Capability for Clinical Laboratory Measurement Procedures; Approved Guideline – Second Edition. CLSI document EP17-A2. Wayne, PA: Clinical and Laboratory Standards Institute; 2012. 12. World Health Organization. (2014). Instructions for compilation of a product dossier: prequalification of in vitro diagnostics programme. World Health Organization. https://apps.who.int/iris/handle/10665/329995. License: CC BY-NC-SA 3.0 IGO 13. Interference Testing in Clinical Chemistry. Approved Guideline, Third Edition. CLSI EP07-Ed03. Wayne, PA: Clinical and Laboratory Standards Institute; 2018. 14. Supplemental Tables for Interference Testing in Clinical Chemistry, First Edition. EP37. Wayne, PA: Clinical and Laboratory Standards Institute; 2018. 15. Guidance for Industry In the Manufacture and Clinical Evaluation of In Vitro Tests to Detect Nucleic Acid Sequences of Human Immunodeficiency Viruses Types 1 and 2; U. S. Food and Drug Administration Center for Biologics Evaluation and Research (CBER), 1999. TSS 11 Page | 29 16. Evaluation of Precision of Quantitative Measurement Procedures; Approved Guideline - Third Edition. CLSI EP05-A3. Wayne, PA: Clinical and Laboratory Standards Institute; 2019. 17. Shafer RW(2006). Rationale and Uses of a Public HIV Drug-Resistance Database. Journal of Infectious Diseases 194 Suppl 1:S51-8. 18. Haeckel R. Proposals for the description and measurement of carry-over effects in clinical chemistry. Pure Applied Chemistry 1991; 63:301-306. 19. IEC 62366-1:2015. Medical devices -Part 1: Application of usability engineering to medical devices. International Electrotechnical Commission; 2015. 20. General Principles of Software Validation; Final Guidance for Industry and FDA Staff; U. S. Food and Drug Administration https://www.fda.gov/media/73141/download accessed 28 October 2021 21. ISO 23640:2011. In vitro diagnostic medical devices - Evaluation of stability of in vitro diagnostic reagents. Geneva: International Organization for Standardization; 2011. 22. Evaluation of Stability of In Vitro Diagnostic Reagents; Approved Guideline. CLSI EP25-A. Wayne, PA: Clinical and Laboratory Standards Institute; 2009. 23. World Health Organization. (2019). Establishing stability of in vitro diagnostic medical devices (TGS2). World Health Organization. https://apps.who.int/iris/handle/10665/259742. License: CC BY-NC-SA 3.0 IGO 24. ASTM International. ASTM D4169-14. Standard Practice for Performance Testing of Shipping Containers and Systems. West Conshohocken, PA: ASTM International; 2014. 25. World Health Organization. (2019). Toolkit: HIV molecular diagnostics toolkit to improve access to viral load testing and infant diagnosis: HIV treatment and care. World Health Organization. https://apps.who.int/iris/handle/10665/325961. License: CC BY-NC-SA 3.0 IGO 26. Unitaid. UNAIDS data 2019 https://www.unaids.org/sites/default/files/media_asset/2019- UNAIDS-data_en.pdf accessed 28 October 2021. 27. https://www.avert.org/professionals/hiv-around-world/sub-saharan-africa/overview accessed 28 October 2021. 28. Backinger CL and Kingsley PA. Write It Right: Recommendations for Developing User Instructions for Medical Devices Used in Home Health Care. HHS Publication FDA 93-4258;. Silver Spring, MD: Food and Drug Administration; 1993 Available at: https://www.qualysinnova.com/download/files/write-it-right.pdf accessed 28 October 2021. 29. Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in-vitro diagnostic medical devices. O. J. E. U. 2017 L 117/176. 3.