Variability of the diagnosis of stroke by clinical judgement and by a scoring method

Variability of the diagnosis of stroke by clinical judgement and by a scoring method S. HATANO 1, on behalf of the participants in the WHO Collaborative Study on the Control of Stroke in the Community 2 Existing criteria for the diagnosis of acute cerebrovascular disease (stroke) have not been satisfactory in epidemiological studies. Variability of the diagnosis of stroke, which had not been studied before, was investigated in a WHO collaborative study. Intra-observer and inter-observer variation of the diagnosis of stroke was studied by means of 45 case reports drawn at randomfrom among those included in the study. Diagnosis ofstroke and of the type of stroke was made by clinical judgement and by a scoring method. The clinical diagnosis ofstroke was more consistent and more comparable than the diagnosis of the type of stroke. Inter-observer agreement in clinical diagnosis was improved by using the score method. It is often thought that acute cerebrovascular disease appears with clear-cut neurological deficits of sudden onset, and that it is not very difficult to diagnose clinically. The diagnosis of stroke, how- ever, is based primarily on neurological examina- tions, which are subject to observer variation, and on clinical history taking, which varies with the memory and powers of communication of the patient or his family. In any cooperative epidemio- logical study, the reliability of diagnosis needs to be checked. This is even more true for cooperative epidemiological studies in which results from various centres are pooled. We have attempted, in the WHO Collaborative Study on the Control of Stroke in the Community (1, 2), to assess the extent of agreement in the clinical diagnosis of stroke. The direct measurement of observer bias in neuro- logical examinations of the patient and in history- taking on the spot was precluded because of the remoteness of the participating centres and the lan- guage barrier. Case reports (in English) were there- fore collected and distributed to the centres, which were asked to make a diagnosis in exactly the same manner as they were doing for their own stroke registers, in order to evaluate the comparability of the diagnoses in practice. It was thought opportune also to test a less subjective method: the Kyushu University score 1 Department of Epidemiology, Tokyo Metropolitan Institute of Gerontology, 35 Sakaecho, Itabashi-ku, Tokyo 173, Japan. 2 Listed on page 538. developed by Ikeda (3) for the differential diagnosis of stroke. The method is based on scores for selected neurological signs and a few personal particulars, such as age, sex, and history of hypertension (Annex). The purpose of the present paper is to report the results of the tests on the variability of diagnosis of stroke by clinical judgement and by the scoring method. MATERIAL AND METHODS Forty-five case histories were drawn at random from the WHO stroke register: 3 cases with stroke from each of the 12 participating centres, 4 cases from among patients registered initially as possibly having had stroke but later excluded from the register because another diagnosis had been established, and 5 cases from among stroke patients without paralysis of the limbs.a The latter two categories were added because their frequency differed among the centres and varying diagnosis of these cases might have resulted in biased estimates of the incidence of stroke. Of the 45 case reports, 15 were submitted twice, after having been " camouflaged " by irrelevant changes, e.g., in age, sex, or the style of the report, for the purpose of testing observer consistency. Thus a These 45 case histories were supplied by the following 12 centres: Akita (Japan), Colombo (Sri Lanka), Copen- hagen (Denmark), Dublin (Ireland), Espoo (Finland), Fukuoka (Japan), Gothenburg (Sweden), Moscow (USSR), Osaka (Japan), Rohtak (India), Japan National Railways, Tokyo (Japan), and Zerifin (Israel). 3549 - 533 - BULL. WORLD HEALTH ORGAN., Vol. 54, 1976 S. HATANO a total of 60 case reports was circulated in two series of 30 with an interval of about 6 weeks. The centres made their clinical diagnoses in the same way as is done in daily practice for the WHO stroke register: first, it was established whether the disease was an acute cerebrovascular disease or not, and then the diagnosis of the type of stroke was made both by clinical judgement and by the Kyushu University score (3). Sixteen centres made a clinical diagnosis for each of 60 case reports and one centre for only 30 reports. Diagnosis based on the Kyushu University score was made for 60 case reports by 8 centres and for 30 reports by 2 centres. RESULTS Clinical diagnosis ofstroke For the 15 cases submitted for diagnosis twice, intra-observer consistency as regards the presence or absence of stroke was between 87% and 100%, being 98% on average (Table 1). The number of cases excluded because they were found not to be due to stroke varied from 0 to 11 (out of the 60 cases) between observers. The relative frequency with which stroke was diagnosed by each observer thus ranged from 87% to 107% of the average figure (100%=913 strokes out of the total of 976 diagnoses). Frequency of types of stroke as diagnosed by the centres The frequency of various types of stroke in 60 or in 30 test cases according to the diagnosis by each observer, together with the frequency of types of stroke for all the cases registered at each centre, is shown in Fig. 1. The frequency of stroke of undetermined type tended to be higher in the test series than was found Table 1. Consistency of clinical and score diagnosis No. of Diagnosis of stroke or not Diagnosis of type of stroke Methods of diagnosis No. of No. of repeatexaminers cases tests a agreed disagreed agreement agreed disagreed agreement Clinical diagnosis 16 15 232 228 4 98.3% 181 51 78.0% Score diagnosis b 8 15 118 - - - 87 26 77.0 % a Some examiners did not make a diagnosis for all the test cases. b The score method is used to determine the type of stroke when stroke is present. Table 2. Consistency of diagnosis of type of stroke in a duplicate test with 15 cases Diagnosis by clinical judgement Diagnosis by the Kyushu University score(16 observers) (8 observers) Second diagnosis Second diagnosis Total Total SAH CH cI Un Ex NA SAH CH cI Un Ex NA SAH 14 1 1 1 17 SAH 8 2 10 ." CH 1 7 6 4 27 . CH 4 11 11 1 27 0 o t, ci 2 104 13 1 1 121 <, ci 1 6 68 1 76 n Un 1 7 11 40 3 62 n Un 1 1 ,z Ex 6 5 11 * Ex 1 3 4 NA 1 1 2 NA 2 2 Total 15 27 122 59 10 7 240 Total 13 17 83 1 4 2 120 Abbreviations: SAH, subarachnoid haemorrhage; CH, intracerebral haemorrhage; Cl, cerebral infarction; Un, stroke of undetermined type; Ex. cases excluded later because they were found not to be due to stroke; NA, no diagnosis made. 534 DIAGNOSIS OF STROKE in practice in the stroke registers maintained by the participating centres. A certain parallelism in the frequency of types of stroke was also observed. Intracerebral haemorrhage was diagnosed more fre- quently both in the test cases and in practice at the 50 ioe0 CENTRE r GOTHENBURG . COPENHAGEN EII... ..,,,111111111---..-..-...-....-..-.. DUBLIN l..-....a...n u s§.:.:.-.-.:.:... NORTH .-:- .:::: -:*:-: --::::-.. KARELIA sa " rb ul i_ :::.: ::.. ,...-...8.-....... -.s**+-6-, -,.,,.,aX Qmuuw* ......... --.-=.--.-.......................... ZA6REB ZAGREB~~~~~............... ZERIFIN ....................... IBADAN .-..... SAKU ~~~~~~. i. : : :~~~~~.:.... ........-..-...... AKITA ............ SAKU FUKUOKA ............. OSAKA JAP.NAT .-.-. RAILWAYS, TOKYO -- ; ,;,-,,-,,----;,---;..... .. ......'.'.;.-;;,... -;.. ;--NROHTAK - - - .... C O L O M B O[ > .... . ... ..............COLOMBO ALL CENTRES t* TYPES OF STROKE WHO 761082 Subarachnoid haemorrhage * Intracerebral haemoffhag Cerebral infarction * Stroke of undetermined type Fig. 1. Frequency of various types of stroke, according to clinical diagnosis made in registered cases (upper line) and test cases (lower line) at 16 centres. Ibadan, Japan National Railways, and Osaka cen- tres, and less frequently at the Dublin, North Karelia, and Saku centres. The frequency of the diagnosis of stroke of undetermined type was very high at Dublin and Gothenburg for both test cases and locally registered cases, and at Copenhagen and Espoo for test cases only. When this diagnosis is fre- quent, the ratio of haemorrhagic to thromboembolic strokes becomes uncertain. Consistency of clinical diagnosis of types of stroke The rate of intra-observer agreement on the type of stroke ranged from 40% to 100%, being 78% on the average (Table 1). The consistency with which various types of stroke were diagnosed is shown in Table 2. The diagnosis of subarachnoid haemor- rhage and cerebral infarction was fairly consistent. Consistency was much lower for intracerebral haemorrhage and stroke of undetermined type, and in cases later found not to be due to stroke and consequently excluded from the register. If a stroke of undetermined type had been regarded as com- patible with any type, the number of consistent diagnoses would have increased from 181 to 218 out of 232 diagnoses (78% to 94%). Consistency of diagnosis of type of stroke by the Kyushu University score In 12 out of 15 duplicate case reports, either the sex or the age was altered so that the total score could not be identical. However, in the remaining 3 cases, each of which was diagnosed at 6 centres (18 diagnoses in all), complete agreement of the score was observed only 4 times. The final diagnoses based on the score neverthe- less revealed a rate of intra-observer agreement almost the same as that for the clinical diagnoses (Table 1). The scoring method may be used only for the diagnosis of the type of stroke; it cannot be used for diagnosing stroke per se. Frequent shifts of diagnosis between different types of stroke were observed in the present series (Table 2), owing partly to lack of training in the use of the method, and partly to modifications of age and sex in the reports. Inter-observer differences and agreement Except for a few cases in which the diagnosis was confirmed by autopsy, a diagnosis could not be determined with absolute certainty. Therefore a diagnosis agreed upon by the majority of the parti- cipating examiners was taken as the standard. The 535 S. HATANO Table 3. Diagnosis in 15 cases with a duplicate test, made by the largest number of observers, and rate of agreement with the majority Clinical diagnosis a Score diagnosisb Majority Number of diagnoses Rate of M Number of diagnoses Rate of diagnosis c (type un- ae diagnosis c agreementagreed disagreed determined) 7 agreed disagreed % ci 21 11 (10) 65.6 ci 17 1 94.4 ci 30 3 (3) 90.9 ci 16 2 88.9 ci 25 8 (8) 75.8 ci 17 1 94.4 ci 31 2 (2) 93.9 cI 18 0 100.0 ci 18 15 (14) 54.5 cI 17 1 94.4 ci 23 10 (10) 69.7 ci 15 1 93.8 not stroke 18 9 (3) 66.7 ci 4 1 -d SAH 31 2 (1 ) 93.9 SAH 18 0 100.0 CH 22 11 (9) 66.7 CH 13 5 72.2 cI 17 16 (12) 51.5 cI 16 2 88.9 Un 10 23 (10) 30.3 CH 12 6 66.7 CH 16 17 (11) 48.5 CH 13 5 72.2 ci 1 5 1 8 (14) 45.5 ci 1 5 3 83.3 ci 21 1 1 (1 0) 65.6 ci 1 6 2 88.9 ci 27 6 (6) 81.8 ci 1 6 2 88.9 Total 325 165 (123) 66.3 Total 223 32 87.5 a Sixteen centres diagnosed twice and one centre once. b Seven centres diagnosed twice and three centres once. c For abbreviations, see Table 2. d The score method cannot be used in a case found not to be due to stroke. majority consisted of 31 %-100% of the examiners, depending on the amount of information in the case report and its diagnostic value. Two-thirds of the clinical diagnoses agreed with the majority diagnosis. The rate of agreement of various observers with the majority diagnosis ranged from 220% to 85% of cases. The rate of agreement was 91 % for subarachnoid haemorrhage; it was only about 43 % when the majority diagnosis was stroke of undetermined type. The rate of agreement for other types was in between. When the diagnosis was made according to the score, the rate of agreement was generally high, 87% agreeing with the majority diagnosis. The agreement rate was 90% for cerebral infarction, 85% for subarachnoid haemorrhage, and 76% for intracere- bral haemorrhage. Agreement between clinical and score diagnoses The clinical diagnosis made by the largest number of examiners was taken as the reference diagnosis. As an example of the agreement and disagreement of diagnoses in individual cases, the clinical and score diagnoses for 15 duplicate case histories are pre- sented in Table 3. The score diagnosis agreed well with the clinical diagnosis except that the rate of inter-observer agreement was higher for the former than for the latter. Validity of diagnosis Autopsy was carried out in only 5 of 45 cases. The diagnoses made in these 5 cases by clinical judge- ment and by the score are shown in Table 4. In a case supplied by the Japan National Railways 536 DIAGNOSIS OF STROKE Table 4. Autopsy diagnosis and diagnosis made clinically and by the score method Clinical diagnosis a Score diagnosis a AutopsyCdiagnosis supplying Number of diagnoses Rate of Number of diagnoses Rate of reportyCHsupplying correct correct SAH CH Ci Lin Ntok otal digoi SAH CH Cl Total digoi cerebral thrombosis cerebral haemorrhage not stroke(glioblastoma) Copenhagen 0 1 21 10 0 32 Gothenburg 0 12 1 3 1 17 Gothenburg 1 3 0 1 11 16 65.6 0 1 17 18 94.4 70.6 0 10 0 10 100.0 68.8 2 3 0 5 b subarachnoid haemorrhage Gothenburg 16 0 0 0 0 16 100.0 8 0 0 8 100.0 subarachnoid haemorrhage Tokyo 3 2 10 1 0 16 81.3 1 0 7 8 100.0 & cerebral infarction c (Japan National Railways) a For abbreviations, see Table 2. b If the clinical diagnosis showed the absence of stroke, the score method was not applicable. c The patient had two separate attacks. A valid diagnosis for either one was taken as correct. centre, the patient had two attacks: a slight attack of cerebral thrombosis and later a fatal attack of sub- arachnoid haemorrhage. Both diagnoses were there- fore regarded as correct. In all 5 cases, a correct diagnosis was made by many observers, both clini- cally and by the score method. The valid diagnosis was more frequently established by the score method than by clinical judgement. DISCUSSION The clinical diagnosis of stroke-i.e., its presence, irrespective of type, or its absence-was highly con- sistent (Table 1). The numbers of cases judged by the different observers not to be due to stroke, in a sample of 60 cases from the WHO collaborative study, resulted in variation between -13% and +7% from the average number of stroke cases. This degree of variability may be permissible when there are far greater differences in incidence between centres. Since the sample included 15% of problem- atic cases, the variability in practice would be much smaller. The diagnosis of stroke itself therefore appeared to be comparable. Validation of diagnosis was made by autopsy in a small number of cases. Considering the limitations of the information supplied, the frequency of correct diagnosis was remarkably high, so that the standard of diagnosis for the collaborative study as a whole was thought to be satisfactory. Intra-observer consistency and inter-observer agreement were much lower for the diagnosis of the type of stroke. The test cases were not representative of stroke cases from any one centre, but were intended to be representative, as far as possible, of cases registered in all the centres. A certain similarity in the frequency with which certain types of stroke were diagnosed in the test and in registered cases was observed (Fig. 1). The lack of uniformity in the criteria used by the various centres for determining the type of stroke may explain the variability in diagnosis of the type of stroke. The comparison of the frequency of particular types of stroke therefore requires cautious interpretation. The type of stroke was left undetermined in the clinical diagnosis in many of the test cases because the information given in case reports is generally less vivid than that gathered in real-life situations, in which a patient can be examined directly. When the same cases were submitted twice "blindly ", the diagnosis frequently shifted to and from stroke of undetermined type (Table 2). This may reflect the observer's uncertainty or cautious- ness and need not be considered as a diagnostic error, but this variation reduced the rate of intra- and inter-observer agreement of diagnosis. By intro- ducing a uniform diagnostic method, such observer variation may be reduced and the comparability of diagnoses improved. The Kyushu University score was tested as a possible example of such a method. Diagnosis by the 3 537 538 S. HATANO score method agreed well with clinical diagnosis except in stroke of undetermined type, which was reclassified according to the score as intracerebral haemorrhage or cerebral infarction. The rate of agreement with autopsy diagnosis was the same as in clinical diagnosis, but with less inter-observer variation. With the score method, the type of stroke is left undetermined when the scores calculated for two types are equal. Otherwise the majority of cases are categorized without reservation-i.e., the score allows less chance of leaving the diagnosis undeter- mined, even when insufficient information is pro- vided. Not all the items are essential for scoring, and scores calculated with fewer items are still likely to provide a diagnosis for each type of stroke. How- ever, it would seem to be necessary to estimate the probability of arriving at a valid diagnosis and to set the minimum number of items, or minimum differ- ence in scores, required for that purpose. Since the score was calculated on the basis of identical case reports, there should have been no disagreement between observers. However, the scores and the resultant diagnoses did differ. Accord- ing to the participants' reports, this was due to insufficient briefing about the method, which made it difficult to choose the right scores. This disadvantage may be easily overcome. The scoring method was based on the experience in one Japanese hospital, and could be adapted for wider application if cases from other places were included in it. Another limitation to the applicability of the method is the fact that the score was produced from the clinical records of cases in which autopsy was performed, and thus may not be valid for diagnosis in milder cases. However, this reservation is relevant to any kind of diagnostic procedure including clini- cal judgement, when validated by necropsy findings only. Recent progress in the use of a computer-aided transaxial scanner may help to overcome such ob- stacles to the development of a diagnostic method applicable also to surviving patients. Although there is room for improvement, the score method may serve as a basis for developing a standardized method for the differential diagnosis of types of stroke. An attempt to develop it further would therefore seem to be worth while. * * * K. Aho, Central Hospital, Kotka, Finland. B. C. Bansal, Department of Medicine I, Medical College, Rohtak, India. L. Geltner, Asaf Harofe Government Hospital, Tel Aviv University Medical School, Zerifin, Israel. P. Harmsen, Department of Neurology, Sahlgren's Hospital, Gothenburg, Sweden. S. Hatano, Department of Epidemiology, Tokyo Metropolitan Institute of Gerontology, Tokyo, Japan. K. Isomura, Saku Central Hospital, Nagano, Japan. S. Kojima, Central Institute of Health, Akita, Japan. Y. Komachi, Osaka Centre for Adult Diseases, Japan. T. Kondo, Central Health Institute, Japan National Railways, Tokyo, Japan. A. Makinskij, Institute of Neurology, Academy of Medical Sciences of the USSR, Moscow, USSR. J. Marquardsen, Department of Neurology, Fre- deriksberg Hospital, Copenhagen, Denmark. T. Omae, Faculty of Medicine, Kyushu University, Fukuoka, Japan. B. 0. Osuqtokun, Faculty of Medicine, University of Ibadan, Nigeria. J. B. Peiris, Neurological Unit, General Hospital, Colombo, Sri Lanka. Z. Poljakovic, Centre for Cerebrovascular Diseases, Zagreb, Yugoslavia. P. Puska, North Karelia Project, University of Kuopio, Finland. A. Radic, Medico-Social Research Board, Dublin, Ireland. K. Salmi, North Karelia Central Hospital, Joensuu, Finland. V. E. Smirnov, Institute of Neurology, Academy of Medical Sciences of the USSR, Moscow, USSR. T. Strasser, Cardiovascular Diseases, World Health Organization, Geneva, Switzerland. DIAGNOSIS OF STROKE 539 RESUMI VARIABILITE DU DIAGNOSTIC DES ACCIDENTS VASCULAIRES CEREBRAUX D'APRtS LE JUGEMENT CLINIQUE ET D'APRES UNE MhTHODE DE NOTATION Le taux de concordance dans le diagnostic des acci- dents vasculaires c6r6braux a ete evalu6 sur la base des observations relatives a 45 cas choisis au hasard dans le registre OMS des accidents vasculaires c6r6braux. Ces observations, dont 15 ont 6t6 fournies en double, ce qui porte le nombre total a 60, ont e diffusees parmi 17 exa- minateurs dans 16 des centres participant a l'6tude collec- tive OMS sur la lutte contre les accidents vasculaires carebraux dans la collectivite. Outre le diagnostic clinique, on a 6prouv6 la methode de notation de l'Universit6 Kyushu pour le diagnostic diff6rentiel des accidents vasculaires cer6braux d'apres ces observations. Le diagnostic clinique de ces accidents (quel qu'en soit le type) concordait d'un observateur a l'autre et d'une fois a l'autre. Ces resultats montrent que les taux d'inci- dence des accidents vasculaires cerebraux est comparable parmi les centres participants et probablement dans d'autres centres qui utilisent les memes criteres pour dia- gnostiquer ces accidents. En ce qui concerne la classifi- cation des l'accidents vasculaires c6rebraux, le diagnostic dtait moins constant et dependait de l'attitude person- nelle. L'hemorragie subarachnoidienne semble avoir ete diagnostiquee de facon relativement plus uniforme. La methode de notation a fourni le m8me diagnostic que le jugement clinique mais avec des taux de concordance sup6rieurs et ce diagnostic etait 6galement fond6 dans 5 cas v6rifies par autopsie. Les discordances dans le dia- gnostic de differents types d'accidents vasculaires c6r6- braux etaient dues en partie a un manque d'entrainement a la m6thode de notation et en partie aux modifications apportees, dans les observations, aux donnees relatives a I'age et au sexe. REFERENCES 1. HATANO, S. WHO Chronicle, 26: 456 (1972). 2. HATANO, S. Bulletin ofthe World Health Organization, 54: 541-553 (1976). 3. IKEDA, H. Fukuoka medicaljournal, 59: 818 (1968) (in Japanese, with English summary). 540 S. HATANO Annex KYUSHU UNIVERSITY SCORES FOR DIFFERENTIAL DIAGNOSIS OF STROKE Scores for each type Item class a of stroke Cl CH SAH 1. Sex male 2 3 0 female -2 -5 0 2. Age 70+ 6 0 0 60-69 0 1 0 50-59 1 4 0 S49 -17 -4 0 3. Blood pressure 200+/110+ 7 8 0 before stroke 160-199/95-109 -1 1 0 140-159/90-94 -1 -9 0 6139/< 89 -5 -7 0 4. History of yes 3 -2 0 previous stroke no -1 0 0 5. Blood pressure 200+/110+ 0 4 0 after onset 160-199/95-109 3 1 0 of stroke 140-159/90-94 -2 -13 0 -< 139/< 89 -4 -15 0 6. Vomiting yes -4 1 0 no 3 -4 0 7. Consciousness coma -4 0 0 level semicoma and somnolence 4 0 0 normal 6 -5 0 8. Conjugated yes, to healthy side 12 14 0 eye deviation yes, to paretic side 8 12 0 yes, but no hemiparesis -13 -13 0 no 0 -1 0 9. Anisocoria yes, larger on paretic side 0 1 0 yes, smaller on paretic side 8 15 0 yes, but no hemiparesis -13 -13 0 no 1 -2 0 10. Ught reflex lost or slow 0 3 0 normal 0 -6 0 11. Corneal reflex lost -3 3 0 normal 2 -6 0 12. Speech disorder present 4 5 0 absent -4 -9 0 13. Neck stiffness present -6 -1 0 absent 19 14 0 14. Motor deficit tetraplegia 0 1 1 0 hemiparalysis 3 4 0 hemiparesis 1 -1 0 absent -9 -17 0 15. Sensory present 4 6 0 deficit absent -3 -19 0 16. Cerebrospinal xanthochromia +, opening pressure > 200 -8 -2 0 fluid ,, +, ,, ,, < 200 -13 4 0 , 200 11 0 0 < 200 19 0 0 Sum b a Classification is made according to the maximum impairment up to two weeks from the onset of stroke. b The scores for each type are summed up. The total score for subarachnoid haemorrhage is always 0. The type with the largest total score is taken as the most likely diagnosis.