Systematic review of statistics on causes of deaths in hospitals: strengthening the evidence for policy-makers

Bull World Health Organ 2014;92:807–816 | doi: http://dx.doi.org/10.2471/BLT.14.137935 Systematic reviews 807 Systematic review of statistics on causes of deaths in hospitals: strengthening the evidence for policy-makers Rasika Rampatige,a Lene Mikkelsen,b Bernardo Hernandez,c Ian Rileya & Alan D Lopezd Introduction The poor state of health information systems and, particularly, mortality statistics in many countries is widely documented in the literature and in country reports to the World Health Organization (WHO).1–5 However, health systems worldwide depend on reliable information about causes of mortality to be able to respond effectively to changing epidemiological circumstances. Such responses depend critically on accurate data to guide decision-making.6 Within a health information system, accurate and timely data on the cause of death are fundamental for programme and policy development7 and for measuring change in the magnitude and distribution of ill-health and disease in populations.8 Assessments of vital registration systems in low- and middle-income countries consistently reveal substantial weaknesses in the generation of cause-of-death statistics.9–11 In a recent editorial, we drew attention to the fact that even hospital statistics on cause of death cannot be assumed to be correct – a fact that is not widely appreciated by governments and other users of these data.12 The gold standard for cause-of-death reporting is to have the cause certified by a medical practitioner using the rules and procedures of the International classification of diseases and related health problems (ICD), which is currently avail- able in its tenth revision (ICD-10).13 Although most countries with statistical systems for cause of death now use the ICD classification for coding, not all countries have introduced the international standard certificate for reporting cause of death. Furthermore, physicians often do not receive adequate training in standard ICD death certification practices. It is, therefore, not surprising that comparative assessments commonly find that the quality of medical certification of the cause of death is poor.3,8 Cause-of-death statistics of poor quality have limited policy utility and may even seriously mislead policy debates.14 In most developing countries, more than half of all deaths occur outside hospitals. Since out-of-hospital deaths are rarely medically certified, most of the physician-certified deaths come from hospitals.3,4 Can we automatically assume that the cause assigned to a death in hospital is accurate? Unfortunately, even in countries where hospital data are the only source of cause-of-death information, data quality is rarely evaluated. Research in different countries has repeatedly identified sub- stantial misclassification of the cause of death of people who die in hospitals – with attendant implications for the use of cause-of-death data in informing policy.14–20 To carry out a validation study of cause-of-death data col- lected in hospitals, we need a gold standard against which the hospital cause-of-death reports can be compared. While autop- sy findings provide the ideal gold standard for cause-of-death evaluations, this approach is prohibitively expensive, rarely applied and likely to be based on a biased sample of deaths assigned to coroners. It would not be practical to carry out autopsies for all of the deaths occurring in a country – or even for all of the hospital deaths in a country.19 Instead, researchers have reviewed the medical records of people who have died in hospitals as a reference standard for validating the accuracy of the causes of deaths recorded by the hospitals. Although all hospitals have medical records for their patients, such records are rarely used for carrying out routine assessments of the extent and nature of any diagnostic misclassifications among Objective To systematically review the reliability of hospital data on cause of death and encourage periodic reviews of these data using a standard method. Methods We searched Google Scholar, Pubmed and Biblioteca Virtual de la Salud for articles in English, Spanish and Portuguese that reported validation studies of data on cause of death. We analysed the results of 199 studies that had used medical record reviews to validate the cause of death reported on death certificates or by the vital registration system. Findings The screened studies had been published between 1983 and 2013 and their results had been reported in English (n = 124), Portuguese (n = 25) or Spanish (n = 50). Only 29 of the studies met our inclusion criteria. Of these, 13 had examined cause of death patterns at the population level – with a view to correcting cause-specific mortality fractions – while the other 16 had been undertaken to identify discrepancies in the diagnosis for specific diseases before and after medical record review. Most of the selected studies reported substantial misdiagnosis of causes of death in hospitals. There was wide variation in study methodologies. Many studies did not describe the methods used in sufficient detail to be able to assess the reproducibility or comparability of their results. Conclusion The assumption that causes of death are being accurately reported in hospitals is unfounded. To improve the reliability and usefulness of reported causes of death, national governments should do periodic medical record reviews to validate the quality of their hospital cause of death data, using a standard. a University of Queensland, School of Population Health, Brisbane, Australia. b LM Consulting, Brisbane, Australia. c Institute for Health Metrics and Evaluation, University of Washington, Seattle, United States of America. d University of Melbourne, Melbourne School of Population and Global Health, Building 379, 207 Bouverie Street, Carlton 3053, Victoria, Australia. Correspondence to Alan D Lopez (email: alan.lopez@unimelb.edu.au). (Submitted: 1 March 2014 – Revised version received: 18 May 2014 – Accepted: 7 July 2014 – Published online: 16 September 2014 ) Systematic revi w Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935808 Systematic reviews Reliability of statistics on cause of death in hospitals Rasika Rampatige et al. hospital deaths. In part, this reflects a lack of awareness of the existence of such misclassification or – because there is no standard method and framework for carrying out routine evaluations of the quality of cause-of-death data – a lack of knowledge of how such misclassification might be identified. Here, we perform a systematic review of studies that used medical records to assess the quality of hospital cause-of-death data to ascertain the pattern and extent of diagnostic misclassification of the cause of death. We propose a standard method for the future use of medical record reviews for assessing the accuracy of hospital-based cause-of-death data. Systematic review Search strategy We searched for published articles on studies that used medical record reviews to validate cause-of-death data for hospital deaths in English, Spanish or Portuguese, between 1983 and 2013. Search terms and databases are shown in (Fig. 1). All 199 studies identified from the initial investigation were screened for specific content and only the 29 studies that met our inclu- sion criteria were subjected to further scrutiny (Fig. 1, Appendix A available from: http://www.uq.edu.au/hishub/ docs/Appendix%20A_final.pdf ). EsA- sentially, to be included in our system- atic review, a study had to be primary research, published in a peer-reviewed journal and involve the validation of cause-of-death data originating from one or more hospitals against reference cause-of-death data obtained by review of the corresponding medical records. Results The 29 selected studies were categorized according to timing, geographical loca- tion, scope and basic study features, age at death and range of causes of death included (Table 1). Wherever possible, we also assessed methodological issues such as the characteristics of the review- ers, the quality of the medical records used and the diagnostic facilities avail- able in the study hospitals. Since the methods, scopes and objectives of the selected studies were diverse, no review protocol was systematically applied and no comparative analysis of our findings was possible. Only nine of the 29 selected studies were published after 2005. The number of deaths included in each study ranged from 23 in a study in Mexico15 to 3316 in a nationally representative study of cause-of-death accuracy in Thai hos- pitals.17 More than half of the selected studies (n = 18) were carried out in the Americas; Mexico contributed three studies and Brazil six. There was only one study from the Pacific – in Tonga20 – and one from Africa – in South Africa.14 Eighteen studies considered all causes of death while the remainder (n = 11) investigated selected causes only. The large majority (n = 24) covered all age groups. However, two studies included only adult deaths, one study focused on deaths in the elderly and another two studies focused on infant deaths (Table 1). The main purpose of most of the medical record reviews was to identify the degree of misclassification of cause Fig. 1. Flowchart for the selection of studies on medical record reviews to validate cause-of-death data for hospital deaths Screening of studies using inclusion criteria: • primary research • published between 1983 and 2013 in a peer-reviewed journal • cause of death reported from a hospital was validated against cause identified through independent review of patient’s medical records Criteria used for extracting information from studies: • description of the study • characteristics of record review • assessment of quality of medical records to support cause of death diagnosis • reproducibility of the cause of death data Via PubMed and Google Scholar n = 112 Manual search for additional articles n = 12 Manual search for additional articles n = 12 Via PubMed and Biblioteca Virtual de la Salud n = 63 English articles n = 124 English studies n = 12 Spanish studies n = 10 All selected studies n = 29 Portuguese studies n = 7 All screened articles n = 199 Portuguese articles n = 25 Spanish articles n = 50 Search for English terms: “validity of cause of death reporting”, “accuracy of hospital death reporting”, “quality of underlying cause of death certification”, “validation of death certification”, “medical record review to validate cause of death”, “validating cause of death reported in hospitals”, “quality of hospital cause of death”. Search for Spanish terms: “validez del reporte de causas de muerte”, “precisión del reporte de muertes hospitalarias”, “calidad de la certificación de causa básica de muerte”, “validación de certificados de defunción” , “revisión de registros medicos para validar causas de muerte” , “validación de causas de muerte en hospitales”, “calidad de información de causas de muerte en hospitales” and Portuguese terms: “qualidade do preenchimento de atestados de óbito”, “confiabilidade dos atestados de óbito”, “precisão da causa básica de morte da”, “declaração de obito”, “qualidade da certificação medica da cause de morte”, “validação de declarações de obito”. Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935 809 Systematic reviews Reliability of statistics on cause of death in hospitalsRasika Rampatige et al. of death at the individual level – by comparing the hospital or vital registra- tion diagnosis with a reference diagnosis based on a review of the dead patient’s medical records. A misclassification matrix is the primary outcome of several studies.3,16–18,20,22,27 An example of a mis- classification matrix, based on empirical research carried out in Colombo, Sri Lanka,22 is shown in Table 2. Selected findings from the misclassification ma- trices obtained in different countries are presented in Box 1. The misclassifica- tion matrix produced in a study that is nationally representative can be used to derive a series of correction factors that can be applied to routine cause-of-death data – from vital registration systems – to estimate the probable true cause- of-death pattern in the study country. In Thailand, for example, cause-specific mortality fractions that had been cor- rected in this manner were applied to the numbers of registered deaths in 2005 – which had been adjusted for underre- porting – to estimate the probable true pattern of causes of death in the country in that year.44 For some causes of death, such as human immunodeficiency virus infection/acquired immunodeficiency syndrome and ischaemic heart disease, the corrected numbers of deaths in this study were 3- to 4-fold higher than the numbers recorded in the vital registra- tion system – with huge implications for Thailand’s health policies. Fifteen of our selected studies16–18,20–23,27,28,31–33,38,39,44 provided information to correct cause- specific mortality fractions based on vital registration data – where these fractions were known or suspected to be incorrect. The utility of any medical record review depends on the ability of the reviewers to identify the underlying cause of death correctly from the medi- cal records.45 The reviewers in most of our selected studies (n = 18) were phy- sicians who had – reportedly – been trained in death certification practices (n = 15). One study simply reported that professionals had been used as reviewers, while nine studies provided no information about reviewer quali- fications. A Brazilian study34 that used both a physician and a researcher was classified as physician-review in this analysis. The number of reviewers per study depended on sample size and ranged from one in each of four studies to 84 in the Thai study.17 The accuracy of the reference cause-of-death diagnosis depends on the quality of information available in the medical records. Ten of the reviewed studies included such an assessment – in which the records were categorized ei- ther simply, as adequate or inadequate, or, with more qualification, as excellent, good, average, weak or poor. One study excluded deaths when the correspond- ing medical records were judged to be incomplete.23 The remainder provided no information about the quality of the medical records used. In nine of our selected studies, standard diagnostic criteria were pre- determined – i.e. the clinical evidence required to classify a death as being due to a particular cause was specified in advance of the review.14,16,20,23,25,38,40,41,43 In case of diagnostic uncertainty, nine studies used a physician panel to discuss diagnostic problems. Three studies or- dered verbal autopsies when the cause of death was in doubt. One study referred diagnostic problems to a second physi- cian, and two others referred to external experts. Fourteen studies provided no information about the resolution of diagnostic uncertainty. The diagnostic capacity of hospitals will clearly influence the accuracy of their cause-of-death diagnoses, both in vital registration systems and in medi- cal record reviews. Only the Chinese study18 specifically mentioned that urban hospitals were selected to ensure that the study hospitals had adequate diagnostic facilities. None of the other studies included in our analysis referred to hospital diagnostic capacity. The correct assignment of the underlying cause of death on the death Table 1. Key characteristics of the 29 studies selected for inclusion in the reviewa Characteristic Selected studies, no. (%) Year of study After 200516,17,20–26 9 (31.0) 2000–200414,18,27–32 8 (27.6) 1990–199915,33–42 11 (38.0) Earlier than 199043 1 (3.4) Location Americas15,16,21,24–26,29–34,37–42 18 (62.1) Europe23,28,35,36 4 (13.8) Asia17,18,22 3 (10.3) Pacific20 1 (3.4) Africa14 1 (3.4) Middle East27,43 2 (7.0) Sample size (no. of deaths) 0–9915,40 2 (6.9) 100–49914,21,24,25,29–32,36–38,42,43 13 (44.8) 500–149920,22,26–28,33,35,39,41 9 (31.0) More than 150016–18,23,34 5 (17.3) Scope of the study All conditions14–18,20,22,24–26,28,30,34,35,38,39,42,43 18 (62.2) Cardiovascular conditions and/or diabetes29,32,37 3 (10.3) All non-accidental deaths21 1 (3.4) Ill-defined and vague causes27 1 (3.4) Deaths with legal implications23,36 2 (7.0) Neonatal causes33 1 (3.4) Cancer31,40,41 3 (10.3) Age groups included in the study All14–18,20,22–27,31,32,35–43 24 (82.8) Adultsb,21,30 2 (7.0) Elderlyb,28 1 (3.4) Infants aged < 1 year34 1 (3.4) Neonatesb,33 1 (3.4) a Summary of the articles is available in Appendix A (Available from: http://www.uq.edu.au/hishub/docs/ Appendix%20A_final.pdf ). b Ages not defined in relevant article or articles. Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935810 Systematic reviews Reliability of statistics on cause of death in hospitals Rasika Rampatige et al. Ta bl e 2. M isc la ss ifi ca tio n of ca us es o f d ea th s, al l a ge s, bo th se xe s c om bi ne d, Co lo m bo , S ri La nk a, 2 01 2 Vi ta l r eg ist ra tio n di ag no sis Di ag no sis b as ed o n m ed ica l r ec or ds re vi ew (n o. o f d ea th s) Ce rt ai n in fe ct io us an d pa ra sit ic di se as es Al l ca nc er s Di ab et es m el lit us Ot he r di se as es of th e ne rv ou s sy st em Hy pe r- te ns iv e di se as es Is ch ae m ic he ar t di se as es Ce re br o- va sc ul ar di se as es Ot he r he ar t di se as es Pn eu m on ia Ch ro ni c lo w er re sp ira to ry di se as es Ot he r di se as es of th e re sp ira to ry sy st em Di se as es of th e liv er Di se as - es o f th e sk in Ex te rn al ca us es Al l ot he r ca us es To ta l Ce rt ai n in fe ct io us an d pa ra sit ic di se as es 9 3 1 0 0 0 0 2 0 1 0 1 0 2 4 23 Al l c an ce rs 1 34 1 0 0 2 3 0 0 4 0 2 0 0 3 50 D ia be te s m el lit us 3 3 34 1 4 22 9 1 3 2 0 3 1 1 7 94 O th er d ise as es of th e ne rv ou s sy st em 2 0 0 3 0 2 0 0 1 0 1 2 0 0 0 11 H yp er te ns iv e di se as es 4 0 2 0 12 9 10 0 1 3 0 1 0 0 2 44 Isc ha em ic h ea rt di se as es 2 0 9 0 2 54 5 3 1 5 0 4 0 1 3 89 Ce re br ov as cu la r di se as es 0 1 1 1 2 1 17 1 0 1 1 1 0 0 1 28 O th er h ea rt di se as es 1 4 3 0 2 17 4 21 1 4 1 3 1 0 8 70 Pn eu m on ia 0 1 1 2 0 1 1 0 9 2 0 0 0 0 6 23 Ch ro ni c lo w er re sp ira to ry di se as es 1 1 1 0 0 7 1 1 1 13 0 0 0 0 2 28 O th er d ise as es of th e re sp ira to ry sy st em 0 1 1 1 0 1 1 0 1 0 3 2 0 1 3 15 D ise as es o f t he liv er 4 2 2 1 2 2 0 3 0 1 0 39 1 3 60 D ise as es o f t he sk in 0 1 0 0 0 1 0 0 0 0 0 1 0 1 4 Ex te rn al c au se s 3 3 3 0 2 3 0 1 0 1 0 2 0 1 5 24 Al l o th er c au se s 2 2 3 2 1 5 3 2 1 1 0 3 0 0 14 39 To ta l 32 56 62 11 27 12 7 54 35 19 38 6 64 3 6 62 60 2 D at a so ur ce : B as ed o n da ta c ol le ct ed b y Ra m pa tig e et a l.2 2 Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935 811 Systematic reviews Reliability of statistics on cause of death in hospitalsRasika Rampatige et al. certificate requires not only that the sequence and relationship of patho- logical conditions on the certificate are correctly certified but also that the underlying cause of death is correctly coded in terms of the ICD. Despite the critical role of coding, only three studies assessed coding accuracy.17,26,37 All studies reported some degree of diagnostic misclassification. Due to the use of different classification lists, the misclassification reported in these studies are not directly comparable. For the studies that have used ICD-10’s three-character codes for comparison of cause of death, the range of misclas- sification varied from 2524 to 62%.16 The concordance at the level of ICD-10’s three-character codes was 41.4% in Sri Lanka22 while concordance at the level of the Basic Tabulation List in the ninth revision of the ICD was found to be 77% in Sweden.35 In South Africa, concordance at the level of ICD-10’s mortality tabulation level was 55.3%.14 The pattern and extent of misclassifica- tion found in the studies were gener- ally reported either as matrices or as percentages. Other metrics that were used included simple concordance- based measures of sensitivity and specificity, and measures that attempt to correct for the probability of get- ting a concordant diagnosis merely by chance, such as Kappa statistics and chance-corrected concordance.46 More usefully, the analysis of misclassifica- tion matrices allows specific insight into the patterns of cause-of-death misallocation and is thus a critical in- put in activities designed to strengthen health information systems as well as in the correct interpretation of national mortality statistics. Common causes of death that were frequently misclas- sified included ischaemic heart disease, stroke, diabetes mellitus and external causes of injury. In a study based on a medical record review in the Islamic Republic of Iran, ill-defined causes of death were mainly reclassified to ischaemic heart disease (33.5%) and cerebrovascular accidents (17.1%)27 while a study in Kuwait reported that original death certificates underesti- mated cerebrovascular disease by 69%, diabetes mellitus by 60% and ischaemic heart disease by 33%.43 Surprisingly, in our selected studies, ill-defined causes of death were frequently reported even in hospital settings. The policy impact of allocating these non-specific codes to more definitive causes are reported in two studies;21,44 key findings from a meta-analysis of misclassification patterns can be found in the article by Rampatige et al.47 Framework for reviewing medical records Given the importance and cost of accu- rate cause-of-death statistics for health policy and priority setting – and the effort and cost of undertaking medi- cal record reviews – evaluation studies using this method could benefit enor- mously from some simple, evidence- based lessons. Based on this systematic review and experience with our own field research, we provide a synthesis of some of the key issues that should guide national applications of medical record reviews.16–18,20,22,27 Our systematic review confirms that a variety of approaches have been used to assess the accuracy of cause-of- death data. Rather than trying to evalu- ate whether there is one approach that is better than the others, we have tried to draw lessons from each study about the specific steps that added value to the study and could be repeated elsewhere. Building on these findings, we propose a framework that addresses the critical choices that need to be carefully consid- ered when using medical record reviews for assessing cause-of-death data quality (Box 2). Further detail on each of the topics included in our framework has been published elsewhere – with specific guidance on the selection of hospitals and cases, the development and use of diagnostic criteria, interpreting mis- classification matrices, evaluating the quality of the source documentation, and the choice and training of the re- viewers used.47 Discussion Medical record reviews can and have been carried out to serve many different purposes. Perhaps the most common ap- plication is the independent assessment of the reliability of hospital cause-of- death data – particularly when expert review or other use of mortality data has revealed these to be deficient. Another common purpose is to assess whether deaths from specific causes – e.g. various cancers or traffic accidents – are being reliably recorded in hospital settings. If such an assessment involves a reason- ably representative national sample of hospital deaths, correction factors can be calculated from the misclassification Box 1. Selected findings based on reported misclassification matrices for causes of hospital deaths in four countries China Rao et al.18 have shown that, ischaemic heart disease was undercounted in the official statistics by 31% because of the systematic misclassification of true cases of ischaemic heart disease to stroke, diabetes, pneumonia or other forms of heart disease. Hepatitis deaths were found to be frequently misclassified to other liver diseases, and pneumonia was found to be excessively and often incorrectly selected, from a list of respiratory diseases, as the underlying cause of death. Islamic Republic of Iran Khosravi et al.27 have found that the true cause-of-death pattern of the population was found to be considerably different from the pattern of causes reported by the vital registration system in the country. The ill-defined causes reported by the routine death registration system for many deaths among young and middle-aged adults were primarily reclassified, after review, to ischaemic heart disease, stroke and injuries. In half of the study sample, injury deaths had been classified as senility or unknown in the vital registration system – thus greatly underestimating the importance of external causes of hospital deaths. Ill-defined causes of death at an age of ≥ 70 years were largely reclassified, after review, to ischaemic heart disease and stroke. Sri Lanka Rampatige et al.22 have revealed major misclassification errors in identifying deaths caused by vascular diseases or diabetes. Of the deaths caused by ischaemic heart disease, 30% had been misclassified to diabetes or another heart disease and 25% of the deaths due to diabetes mellitus had been misclassified as various diseases of the circulatory system. Thailand Pattaraarchachai et al.17 also reported massive misclassification of major causes of death. Cases of septicaemia – commonly reported in the vital registration system – were reassigned to cerebrovascular disease, human immunodeficiency virus infection/acquired immunodeficiency syndrome and pneumonia. Ill-defined causes were identified as true cases of ischaemic heart disease, other heart disease, chronic obstructive pulmonary disease or stroke. The study also found gross underdiagnosis of diabetes by the vital registration system. Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935812 Systematic reviews Reliability of statistics on cause of death in hospitals Rasika Rampatige et al. matrices and applied to national cause- of-death data to estimate the likely true cause-specific mortality pattern in the population. Despite an extensive literature review, we were only able to identify a small number of peer-reviewed studies that had used medical records to inves- tigate the quality of cause-of-death data (n = 29). Given the critical role of cause- of-death data in informing debates around national health policies, this is both surprising and disappointing. Our main observation – of extensive misclas- sification of causes of death in hospitals wherever studies have been carried out – indicates that inaccurate cause-of- death data are pervasive. However, it is important to consider potential biases that might have affected the limited number of studies that we were able to analyse. First, we were only able to re- view articles that had been published in a language in which at least one member of our review team was competent – i.e. English, Portuguese or Spanish. It is possible that articles published in other languages might have led to different conclusions about the pattern and extent of cause-of-death misclassification in hospitals. More comprehensive studies covering other major language groups would be welcome but there is no reason to suspect that these would not reinforce our main findings – which seemed fairly consistent irrespective of the language of the article involved. Thus, our conclu- sions and recommendations for policy action should still be valid. It is also possible that the literature we screened was affected by selective biases – such as publication bias – but we suspect that this would only operate to suppress the publication of more extreme findings about misclassification. Finally, the studies we selected had diverse aims and the set of causes of death investigated in each study was presumably biased towards the priorities of the interest groups involved. We found only one study that involved the comprehensive investigation of cause-of-death mis- classification across all major causes of death. It is therefore quite likely that the extent of cause-specific misclassification across all countries is different to that indicated by the studies we selected. Again, however, that does not alter our fundamental conclusion that hospital statistics on causes of death – whatever the universe of causes investigated – are likely to be grossly inaccurate in many countries and ought to be the focus of an immediate policy response. All of the studies we selected for review assessed the quality of the hospital-reported cause-of-death data by comparing them with cause-of-death data derived by reviewing medical re- cords. However, the lack of a standard framework or method for conducting and reporting the findings of medical record reviews has resulted in substan- tial variation in the approaches used and difficulties in interpreting and compar- ing results. Building on these findings and our own empirical experience, we propose a series of steps to guide future studies on the accuracy of cause-of- death data that involve the review of Box 2. Proposed framework and key elements for conducting medical record reviews Select hospitals to be reviewed • Determine scope of investigation • Obtain agreement for hospital cooperation • Conduct census of diagnostic facilities available in included hospitals. Select diagnostic categories and develop diagnostic criteria • Set up a small expert group of physicians to develop standard diagnostic criteria • Establish a list of the diseases that are the most important for the review • Develop and pre-test diagnostic criteria on a sample of medical records. Select sample death certificates • Determine sample size • Determine the sampling method and identify the number of death certificates to be included in the study • Draw the sample of death certificates from the vital registration database or hospital mortality register • Retrieve corresponding medical records from the hospitals • Validate the quality of coding – according to the tenth revision of the International Classification of Diseases and Related Health Problems (ICD-10) – for the sample. Select physicians to rediagnose causes of death • Provide training in cause of death certification. Trace the relevant medical records • Decide on criteria to assess the quality of the records • Decide on rules to determine which records can be used and which are too incomplete • Reassess the sample size because of losses arising from poor or untraceable records • Prepare a summary of the quality and availability of medical records and the corresponding maintenance practices. Review medical records • Design form for extracting information from medical records. An example of such medical data extraction forms is given in the study by Rampatige et al.47 • Diagnose cause of death using pre-defined standard diagnostic criteria • Code the new cause according to ICD-10 • Check that coding is correct. Compare the cause of death originally reported from vital registration with that finally assigned after medical record review, and analyse findings • Determine the extent of misclassification • Draw up a misclassification matrix for all ages and both sexes – and by age and sex, if numbers allow – to identify patterns of misclassification • Reassign the ill-defined and misclassified causes based on the misclassification matrix • Compare the new distribution of causes of death with the original, to identify major discrepancies. Write final report and initiate policy dialogue to strengthen health information system • Describe the study design and methodology, including sampling strategy • Discuss findings and implications • Propose improvement steps as needed – e.g. improved certification, coding and keeping of medical records. Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935 813 Systematic reviews Reliability of statistics on cause of death in hospitalsRasika Rampatige et al. medical records. By following these steps, it should be possible to validate the quality of cause-of-death data reported by hospitals and vital registration sys- tems more reliably. The synthesis of lessons learnt, sum- marized in Box 2, outlines a clear and tested method and identifies the most important steps in the process – e.g. the design of the study, development of diagnostic criteria, selection of the sample, review of medical records, and the analysis and interpretation of find- ings. However, the framework that we have proposed – like all frameworks – has limitations. In particular, it would not be feasible to apply our framework in settings where medical records are poorly kept or where the diagnostic ca- pacity of hospitals is generally poor. An understanding – by hospital and health administrators – of the importance of studies based on our framework for policy-makers is crucial, as is a com- mitment, of those responsible for the development of national health informa- tion systems, to implement any resultant recommendations. While the proposed framework needs to be tested in its entirety in further empirical studies, we consider it sufficiently robust to be applied in its current state in different settings. We recommend its periodic application in all countries, to identify deficiencies in national mortality statistics. Using the misclassification matrix that the study method yields, health authorities should be able to identify the diseases that are commonly misclassified as causes of death and also ascertain whether the causes of death that are being recorded in hospitals are of sufficient quality to be fit for purpose. This knowledge can and should be used to guide strategies for the strengthening of health information sys- tems – e.g. increasing awareness among medical associations about the value of doctors correctly certifying deaths, im- proving the keeping of medical records in hospitals, and ensuring that cause-of- death coders are appropriately trained. Systems for the determination and recording of the causes of hospital deaths represent a purposeful and costly investment that countries make. It is imperative that such systems perform to the standard required to support good public policy. ■ Funding: Funding for this research was provided by the former Australian Agency for International Development under a grant provided to the University of Queensland, Brisbane, Australia, to establish a health information systems knowledge hub. Competing interests: None declared. صخلم ةسايسلا عانصل تانّيبلا زيزعت :تايفشتسلما في ةافولا بابسأب ةصالخا تاءاصحلإل يجهنم ضارعتسا تايفشتسلما تانايب ةيقوثولم يجهنم ضارعتسا ءارجإ ضرغلا هذله ةيرود تاضارعتسا ءارجإ عيجشتو ةافولا ببسب ةصالخا .ةدحوم ةقيرط مادختساب تانايبلا Pubmedو Google Scholar في ثحب ءارجإ مت ةقيرطلا لىع لوصحلل Biblioteca Virtual de la Saludو ةيلاغتبرلاو ةينابسلإاو ةيزيلجنلإا تاغللاب ةبوتكلما تاعوضولما انمقو .ةافولا ببس نأشب تانايبلل ققتح تاسارد نع تغلبأ يتلا يبطلا لجسلا تاضارعتسا تمدختسا ةسارد 199 جئاتن ليلحتب قيرط نع وأ ةافولا تاداهش في هنع غلبلما ةافولا ببس نم ققحتلل .ةيندلما لاوحلأا ليجست ماظن 2013و 1983 ينب اهصحف مت يتلا تاساردلا شرن مت جئاتنلا وأ )124 = ددعلا( ةيزيلجنلإا ةغللاب اهجئاتن نع غلابلإا متو فوتست لمو .)50 = ددعلا( ةينابسلإا وأ )25 = ددعلا( ةيلاغتبرلا ةسارد 13 تلوانتو .انب ةصالخا جاردلإا يرياعم ةسارد 29 ىوس - نياكسلا ديعصلا لىع ةافولا طمانأ ببس تاساردلا هذه نم ينح في - بابسلأاب ددحلما تايفولا لدعم بسن حيحصت ةيغب في نيابتلا هجوأ ديدحتل ىرخلأا ةسارد ةشرع تسلا ءارجإ مت .هدعبو يبطلا لجسلا ضارعتسا لبق ةنيعم ضارمأ صيخشت لىع ئطاخ صيخشت نع ةددحلما تاساردلا مظعم تغلبأو يربك توافت كانه ناكو .تايفشتسلما في ةافولا بابسلأ يربك وحن قرطلا تاساردلا نم ديدعلا فصت لمو .ةساردلا تايجهنم في وأ اهجئاتن راركت ةيلباق مييقت هعم ىنستي فاك ليصفتب ةمدختسلما .اهتنراقم بابسأ نع غلابلإا ةقد ضاترفلا ساسأ دجوي لا جاتنتسلاا اهنع غلبلما ةافولا بابسأ ةيقوثوم ينسحتلو .تايفشتسلما في ةافولا ةيرود تاضارعتسا ةينطولا تاموكلحا يرتج نأ يغبني ،اتهدئافو متي يذلا ةافولا تانايب ببس ةدوج نم ققحتلل ةيبطلا تلاجسلل قيقحتل ةدحوم ةقيرط مادختساب ،تايفشتسلما ةطساوب هب غلابلإا .ليودلا ديعصلا لىع اهتنراقم نكمي جئاتن 摘要 系统回顾医院死因统计数据 : 强化政策制定者的证据 目的 系统回顾医院死因数据的可靠性并鼓励使用标准 方法定期回顾这些数据。 方 法 我 们 搜 索 Google Scholar、Pubmed 和 Biblioteca Virtual de la Salud 中报告死因数据验证研究的英语、西 班牙语和葡萄牙语文章。我们分析了 199 个研究的结 果 , 这些研究利用医疗记录回顾来验证死亡证明报告 的死因或由生命登记系统报告的死因。 结果 筛选的研究发表在 1983 年和 2013 年之间 , 其研 究结果以英语(n = 124)、葡萄牙语(n = 25)和西班牙语(n = 50) 发表。只有 29 项研究满足我们的入选标准。这 其中 ,13 项研究在人口水平调查死因模式 ( 旨在纠正 死因别死亡率分数 ), 而另外 16 项识别特定疾病的诊 断在病历回顾前后的差异。大多数的所选择的研究报 道了大量对医院死亡原因的错误诊断。研究方法存在 很大差异。许多研究未详细描述所用的方法 , 不足以 评估结果的再现性和可比性。 结论 医院准确报道死因的假设是没有根据的。为改善 死因报告的可靠性和实用性 , 各国政府应该使用标准 方法完成定期的病历回顾来验证医院死因数据的质量 , 以取得国际可比性的结果。 Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935814 Systematic reviews Reliability of statistics on cause of death in hospitals Rasika Rampatige et al. Résumé Examen systématique des statistiques sur les causes de décès en milieu hospitalier: renforcement des preuves pour les responsables politiques Objectif Examiner systématiquement la fiabilité des données hospitalières sur la cause des décès et encourager l’examen périodique de ces données en utilisant une méthode standard. Méthodes Nous avons utilisé Google Scholar, Pubmed et Biblioteca Virtual de la Salud pour rechercher des articles en anglais, en espagnol et en portugais, qui ont rapporté des études de validation des données sur la cause des décès. Nous avons analysé les résultats de 199 études qui ont utilisé des examens de dossiers médicaux pour valider la cause de la mort déclarée sur les certificats de décès ou par le système d’enregistrement de l’état civil. Résultats Les études passées au crible ont été publiées entre 1983 et 2013, et leurs résultats ont été publiés en anglais (n = 124), en portugais (n = 25) ou en espagnol (n = 50). Seules 29 études ont rempli nos critères d’inclusion. Parmi ces études, 13 ont étudié les schémas des causes de décès au niveau de la population – en vue de corriger les fractions de mortalité selon l’âge – alors que les 16 autres ont été entreprises pour identifier les contradictions dans le diagnostic des maladies spécifiques, avant et après l’examen du dossier médical. La plupart des études sélectionnées ont rapporté des erreurs importantes de diagnostic pour les causes de décès dans les hôpitaux. Il y avait une grande variation dans les méthodologies d’étude. De nombreuses études ne décrivaient pas suffisamment les méthodes utilisées pour pouvoir évaluer la reproductibilité ou la comparabilité de leurs résultats. Conclusion L’hypothèse selon laquelle les causes de décès sont déclarées de manière précise dans les hôpitaux est sans fondement. Pour améliorer la fiabilité et l’utilité des causes de décès déclarées, les gouvernements nationaux devraient effectuer des examens périodiques des dossiers médicaux afin de valider la qualité des données sur la cause des décès dans leurs hôpitaux, en utilisant une méthode standard permettant d’obtenir des résultats comparables au niveau international. Резюме Систематический обзор статистических данных о причинах смертей в больницах: подкрепление доказательств для ответственных лиц Цель Провести систематический обзор достоверности больничных данных о причинах смерти и способствовать проведению периодических обзоров таких данных, используя стандартный метод. Методы Был выполнен поиск статей на порталах Google Scholar, PubMed и Biblioteca Virtual de la Salud, написанных на английском, испанском и португальском языках, в которых приводились проверочные исследования данных о причинах смерти. Были проанализированы результаты 199 исследований, в которых были проведены обзоры медицинских записей для подтверждения причин смерти, указанных в свидетельствах о смерти или в записях актов гражданского состояния. Результаты Отобранные исследования были опубликованы в 1983-2013 гг. и их результаты были представлены на английском (n = 124), португальском (n = 25) или испанском (n = 50) языках. Только 29 из этих исследований соответствовали критериям включения. Из них в 13 исследованиях рассматривались причины смертей на популяционном уровне — с целью коррекции долей конкретных причин смертности, а еще 16 были проведены для выявления расхождений в диагностике конкретных заболеваний до и после проверки медицинской документации. В большинстве из выбранных исследований сообщалось о значительных ошибках в установлении причин смерти в больницах. Методологии исследований значительно различались между собой. Многие исследования не содержали достаточно подробного описания используемых методов, чтобы можно было оценить воспроизводимость или сопоставимость их результатов. Вывод Предположение, что в больницах ведется точная отчетность о причинах смерти, является необоснованным. Для повышения достоверности и полезности данных о сообщаемых причинах смерти национальные правительства должны проводить периодические обзоры медицинских записей с целью проверки качества данных о причинах смерти, предоставляемых больницами, используя стандартный метод для достижения международной сопоставимости результатов. Resumen Revisión sistemática de las estadísticas sobre las causas de muerte en los hospitales: fortalecimiento de la evidencia para los creadores de políticas Objetivo Revisar sistemáticamente la fiabilidad de los datos de los hospitales sobre las causas de las muertes y promover revisiones periódicas de estos datos por medio de un método estándar. Métodos Se realizaron búsquedas de artículos en inglés, español y portugués en Google Scholar, Pubmed y la Biblioteca Virtual de la Salud sobre estudios que informaran acerca de la validación de datos sobre las causas de las muertes. Analizamos los resultados de 199 estudios que habían empleado revisiones de registros médicos para validar las causas de las muertes registradas en los certificados de defunción o por el sistema de registro civil. Resultados Los estudios seleccionados se habían publicado entre 1983 y 2013 y sus resultados aparecían en inglés (n = 124), portugués (n = 25) o español (n = 50). Solo 29 de los estudios cumplieron los criterios de inclusión. De estos, 13 habían examinado las causas de muerte en la población con el fin de corregir las fracciones de mortalidad por una causa concreta, mientras que los otros 16 pretendían identificar las discrepancias en el diagnóstico de enfermedades específicas antes y después de la revisión de los historiales médicos. La mayoría de los estudios seleccionados informaron acerca de diagnósticos sustancialmente incorrectos de las causas de muerte en los hospitales. Las metodologías de estudio mostraron una amplia variación. Muchos estudios no describieron los métodos utilizados de manera suficientemente detallada como para poder evaluar la reproducibilidad o la comparabilidad de sus resultados. Conclusión La suposición de que los hospitales notifican adecuadamente las causas de la muerte es infundada. A fin de mejorar la fiabilidad y la Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935 815 Systematic reviews Reliability of statistics on cause of death in hospitalsRasika Rampatige et al. utilidad de las causas de muerte notificadas, los gobiernos nacionales deberían llevar a cabo revisiones periódicas de los historiales médicos con objeto de validar la calidad de los datos hospitalarios sobre las causas de muertes por medio de un método estándar para lograr resultados comparables a nivel internacional. References 1. Gaete-Darbó A. Appraisal of vital statistics in Latin America. Milbank Mem Fund Q. 1964;42(2):86–117. doi: http://dx.doi.org/10.2307/3348659 PMID: 5889230 2. Mahapatra P, Shibuya K, Lopez AD, Coullare F, Notzon FC, Rao C, et al.; Monitoring Vital Events. Civil registration systems and vital statistics: successes and missed opportunities. Lancet. 2007 10;370(9599):1653–63. doi: http://dx.doi.org/10.1016/S0140-6736(07)61308-7 PMID: 18029006 3. Rao C, Lopez AD, Yang G, Begg S, Ma J. Evaluating national cause-of-death statistics: principles and application to the case of China. Bull World Health Organ. 2005;83(8):618–25. PMID: 16184281 4. Polprasert W, Rao C, Adair T, Pattaraarchachai J, Porapakkham Y, Lopez AD. Cause-of-death ascertainment for deaths that occur outside hospitals in Thailand: application of verbal autopsy methods. Popul Health Metr. 2010;8(1):13. . doi: http://dx.doi.org/10.1186/1478-7954-8-13 PMID: 20482760 5. World Health Organization. Annual epidemiological and vital statistics 1939-1946. Part 1: vital statistics and causes of death. New York: Columbia University Press; 1952. 6. Health Metrics Network. Assessing national health information systems: an assessment tool. Geneva: World Health Organization; 2008. 7. Shibuya K, Scheele S, Boerma T. Health statistics: time to get serious. Bull World Health Organ. 2005;83(10):722. PMID: 16283042 8. Ruzicka LT, Lopez AD. The use of cause-of-death statistics for health situation assessment: national and international experiences. World Health Stat Q. 1990;43(4):249–58. PMID: 2293493 9. Strengthening civil registration and vital statistics for births, deaths and causes of death: resource kit. Geneva: World Health Organization; 2013. Available from http://www.uq.edu.au/hishub/docs/Resource%20Kit/ CRVS_ResourceKIt_active_content.pdf [cited 2013 Aug 29]. 10. Report of the regional assessment study of civil registration and vital statistics systems in Africa. Addis Ababa: United Nations Economic Commission for Africa; 2012. Available from: http://www.uneca.org/sites/ default/files/page_attachments/regional-asessment-reporten.pdf [cited 2014 Jul 18]. 11. Regional strategy for the improvement of civil registration and vital statistics systems. Cairo: Eastern Mediterranean Regional Office of the World Health Organization; 2013. Available from: http://applications.emro.who.int/docs/ RC_Techn_paper_2013_10_15084_EN.pdf?ua=1 [cited 2014 Jul 18]. 12. Rampatige R, Mikkelsen L, Hernandez B, Riley I, Lopez AD. Hospital cause-of-death statistics: what should we make of them? Bull World Health Organ. 2014;92(1):3–3A. doi: http://dx.doi.org/10.2471/BLT.13.134106 PMID: 24391292 13. International statistical classification of diseases and related health problems. 10th revision. Geneva: World Health Organization; 2007. Available from: http://apps.who.int/classifications/apps/icd/icd10online [cited 2011 Jul 18]. 14. Burger EH, Groenewald P, Bradshaw D, Ward AM, Yudkin PL, Volmink J. Validation study of cause of death statistics in Cape Town, South Africa, found poor agreement. J Clin Epidemiol. 2012;65(3):309–16. doi: http:// dx.doi.org/10.1016/j.jclinepi.2011.08.007 PMID: 22169084 15. González-Medina A, Martínez-Natera O. Concordancia entre el diagnóstico de defunción y el diagnóstico anatomopatológico por necropsias en el Hospital Central Militar. Rev Sanid Milit. 2001;55(3):95–101. Spanish. 16. Hernández B, Ramírez-Villalobos D, Romero M, Gómez S, Atkinson C, Lozano R. Assessing quality of medical death certification: Concordance between gold standard diagnosis and underlying cause of death in selected Mexican hospitals. Popul Health Metr. 2011;9(1):38. doi: http://dx.doi. org/10.1186/1478-7954-9-38 PMID: 21816103 17. Pattaraarchachai J, Rao C, Polprasert W, Porapakkham Y, Pao-In W, Singwerathum N, et al. Cause-specific mortality patterns among hospital deaths in Thailand: validating routine death certification. Popul Health Metr. 2010;8(1):12. doi: http://dx.doi.org/10.1186/1478-7954-8-12 PMID: 20482759 18. Rao C, Yang G, Hu J, Ma J, Xia W, Lopez AD. Validation of cause-of-death statistics in urban China. Int J Epidemiol. 2007;36(3):642–51. doi: http:// dx.doi.org/10.1093/ije/dym003 PMID: 17329316 19. Shojania KG, Burton EC, McDonald KM, Goldman L. Changes in rates of autopsy-detected diagnostic errors over time: a systematic review. JAMA. 2003;289(21):2849–56. doi: http://dx.doi.org/10.1001/jama.289.21.2849 PMID: 12783916 20. Carter K, Hufanga S, Rao C, Akauola S, Lopez AD, Rampatige R, et al. Causes of death in Tonga: quality of certification and implications for statistics. Popul Health Metr. 2012;10(1):4. doi: http://dx.doi.org/10.1186/1478-7954- 10-4 PMID: 22390221 21. de Carvalho MH, Álvarez-Hernández G, Denman C, Harlow SD. Validity of underlying cause of death statistics in Hermosillo, Mexico. Salud Publica Mex. 2011;53(4):312–9. doi: http://dx.doi.org/10.1590/S0036- 36342011000400005 PMID: 21986787 22. Rampatige R, Gamage S, Peiris S, Lopez AD. Assessing the reliability of causes of death reported by the Vital Registration System in Sri Lanka: medical records review in Colombo. HIM J. 2013;42(3):20–8. PMID: 24067238 23. Gotsens M, Marí-Dell’Olmo M, Rodríguez-Sanz M, Martos D, Espelt A, Pérez G, et al. [Validation of the underlying cause of death in medico-legal deaths]. Rev Esp Salud Publica. 2011;85(2):163–74. Spanish.doi: http:// dx.doi.org/10.1590/S1135-57272011000200005 PMID: 21826379 24. Rodriguez Almada H, Ciriacos C, Piñeyrúa M, Logaldo R, González González D. Calidad del registro en el certificado de defunción en un hospital publico de referencia. Montevideo, Uruguay. Rev Med Urug (Montev). 2010;26:216–23. Spanish. 25. Durruthy Wilson O, Sifontes Estrada M, Martínez Varona C, Olazábal Hernández A. [From the death certificate to the protocol of necropsy: basic causes of death]. Rev Arch Méd. 2011;15(3):542-52. Spanish.doi: http:// dx.doi.org/10.1016/0895-4356(90)90094-6 PMID: 2254765 26. Fajardo S, Aerts DR, Bassanesi SL. [Accuracy of the Mortality Information System team in the specification of underlying cause of death in a State capital in southern Brazil]. Cad Saude Publica. 2009;25(10):2218–28. Portuguese.doi: http://dx.doi.org/10.1590/S0102-311X2009001000012 PMID: 19851621 27. Khosravi A, Rao C, Naghavi M, Taylor R, Jafari N, Lopez AD. Impact of misclassification on measures of cardiovascular disease mortality in the Islamic Republic of Iran: a cross-sectional study. Bull World Health Organ. 2008 Sep;86(9):688–96. doi: http://dx.doi.org/10.2471/BLT.07.046532 PMID: 18797644 28. Alpérovitch A, Bertrand M, Jougla E, Vidal J-S, Ducimetière P, Helmer C, et al. Do we really know the cause of death of the very old? Comparison between official mortality statistics and cohort study classification. Eur J Epidemiol. 2009;24(11):669–75. doi: http://dx.doi.org/10.1007/s10654-009-9383-2 PMID: 19728117 29. Agarwal R, Norton JM, Konty K, Zimmerman R, Glover M, Lekiachvili A, et al. Overreporting of deaths from coronary heart disease in New York City hospitals, 2003. Prev Chronic Dis. 2010;7(3):A47. PMID: 20394686 30. Casco Jarquín A. MitaAlbán LC. [Clinical study of the discrepancy between death diagnosis and the findings of autopsy in the Hospital Calderón Guardia]. Acta Med Costarric. 2003;45(1):15–9. Spanish. 31. Nunes J, Koifman RJ, Mattos IE, Monteiro GT. [Reliability and validity of uterine cancer death certificates in the municipality of Belém, Pará, Brazil]. Cad Saude Publica. 2004;20(5):1262–8. Portuguese.doi: http://dx.doi. org/10.1590/S0102-311X2004000500020 PMID: 15486669 32. Piñón Pérez J, Hernández Amaro Y, Lores Echevarría C, Lemus Sarracino A, Alvarez Miranda C. Certificados de defunción versus estudios necrópsicos en el infarto miocárdico agudo. Rev Ciencias Medicas Pinar del Rio. 2006;10(3):61–70. Spanish. 33. Carvalho ML, Silver LD. [Reliability of the certification of the basic causes of neonatal deaths: implications for the study of preventable mortality]. Rev Saude Publica. 1995;29(5):342–8. Portuguese.doi: http://dx.doi. org/10.1590/S0034-89101995000500002 PMID: 8731273 34. Niobey FM, Cascão AM, Duchiade MP, Sabroza PC. [The quality of the filling-in of death certificates of children below one year of age in the metropolitan area of Rio de Janeiro]. Rev Saude Publica. 1990;24(4):311–8. Portuguese.doi: http://dx.doi.org/10.1590/S0034-89101990000400009 PMID: 2103649 Bull World Health Organ 2014;92:807–816| doi: http://dx.doi.org/10.2471/BLT.14.137935816 Systematic reviews Reliability of statistics on cause of death in hospitals Rasika Rampatige et al. 35. Johansson LA, Björkenstam C, Westerling R. Unexplained differences between hospital and mortality data indicated mistakes in death certification: an investigation of 1,094 deaths in Sweden during 1995. J Clin Epidemiol. 2009;62(11):1202–9. doi: http://dx.doi.org/10.1016/j. jclinepi.2009.01.010 PMID: 19364635 36. de Arán Barés M, Pérez G, Rosell J, Molina P. [Exactness of mortality statistics by external and natural causes of death with medico-legal intervention in Catalonia, 1996]. Gac Sanit. 2000;14(5):356–62.Spanish. PMID: 11187453 37. Granero R, Ortiz R, Gómez-Marín O, Isaacura C, Goyo A, Rodríguez J. [Validation of “myocardial infarction” as cause of death on death certificates in Barquisimeto, Lara state, Venezuela]. Cad Saude Publica. 1997;13(3):383– 7. Spanish.doi: http://dx.doi.org/10.1590/S0102-311X1997000300013 PMID: 10886877 38. Laclé-Murray A. Causas de muerte en una cohorte de diabéticos tipo 2 de Costa Rica y la calidad de su certificado de defunción. Acta Med Costarric. 2012;54(1):23–30. Spanish. 39. Cordeiro R, Peñaloza ERO, Cardoso CF, Cortez DB, Kakinami E, Souza JJ, et al. [Validity of information on occupation and principal cause of death certificates in Botucatu, Sao Paulo]. Cad Saude Publica. 1999;15(4):719–28. Portuguese.doi: http://dx.doi.org/10.1590/S0102-311X1999000400006 PMID: 10633194 40. Monteiro GT, Koifman RJ, Koifman S. [Reliability and accuracy of reported causes of death from cancer. II. Accuracy of stomach cancer reported in the municipality of Rio de Janeiro County, Brazil]. Cad Saude Publica. 1997;13 Suppl 1:53–65. Portuguese.doi: http://dx.doi.org/10.1590/S0102- 311X1997000500006 PMID: 10886924 41. Schnitman A. [Analysis of the reliability of the declaration of cancer as the basic cause of death in Salvador, Brazil]. Rev Saude Publica. 1990;24(6):490– 6. Portuguese.doi: http://dx.doi.org/10.1590/S0034-89101990000600007 PMID: 2103070 42. García Pérez D, Muñoz C. Calidad de la certificación de la causa básica de la muerte. Estado Lara 1995. Bol Med Postgrado. 2001; 8(2). Spanish. 43. Moussa MAA, Shafie MZ, Khogali MM, el-Sayed AM, Sugathan TN, Cherian G, et al. Reliability of death certificate diagnoses. J Clin Epidemiol. 1990;43(12):1285–95. doi: http://dx.doi.org/10.1016/0895-4356(90)90094-6 PMID: 2254765 44. Porapakkham Y, Rao C, Pattaraarchachai J, Polprasert W, Vos T, Adair T, et al. Estimated causes of death in Thailand, 2005: implications for health policy. Popul Health Metr. 2010;8(1):14. doi: http://dx.doi.org/10.1186/1478-7954- 8-14 PMID: 20482761 45. Maudsley G, Williams L. Death certification–a sad state of affairs. J Public Health Med. 1994;16(3):370–1. PMID: 7999397 46. Murray CJL, Lozano R, Flaxman AD, Vahdatpour A, Lopez AD. Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. Popul Health Metr. 2011;9(1):28. doi: http:// dx.doi.org/10.1186/1478-7954-9-28 PMID: 21816106 47. Rampatige R, Mikkelsen L, Hernandez B, Riley I, Lopez AD. Assessing the reliability of hospital-based cause-of-death statistics: evidence-based guidelines for country application [Working paper 37]. Brisbane: University of Queensland; 2013. Available from: http://www.uq.edu.au/hishub/wp37 [cited 2013 August 30].