Disability after encephalitis: development and validation of a new outcome score

584 Research Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357 Introduction Neurological disability is a major problem among children in resource-poor countries but the true burden of disability is un- known because there is no simple and reliable way of measuring it.1 The ability to measure disease burden is especially important for Japanese encephalitis, which is a major cause of death and disability in Asia. The disease is caused by the mosquito-borne flavivirus, Japanese encephalitis virus, and is spreading. Recently, there have been large outbreaks in India and Nepal and it is estimated that there are 20 000 to 175 000 cases globally each year.2–5 Although vaccines against Japanese encephalitis have been available for many years, they have not been widely used, partly because policy-makers lack information about disease burden.4,6,7 Moreover, the proportion of patients reported to have severe sequelae after infection varies widely, from 19 to 71%.8–11 A major reason for this uncertainty is the lack of a standard method for assessing the outcome of Japanese encephalitis and other forms of acquired brain injury among children in resource- poor countries. Even in industrialized countries, tools for assessing disability in children are not as well developed as for adults.12 The gold- standard method requires a large multidisciplinary team and involves multiple lengthy assessments over an extended period of time. Although some tools have recently been redeveloped for use in resource-poor settings, they often still require lengthy assessments by trained personnel.13,14 We set out to develop a simple score for assessing disability in children affected by Japa- nese encephalitis that can be applied by health-care workers with minimal training. We focused on whether the disability was likely to make a child dependent on others, because this is the key issue in terms of disease burden, as well as the single most important parameter for the children themselves. The score we developed, which has become known as the Liverpool Outcome Score,15 was field-tested at two sites in south-eastern Asia: Bellary in India and Sibu in Malaysia. It is also now being used in Bangladesh, Cambodia, Indonesia,16,17 the Lao People’s Democratic Republic and Viet Nam (S Hills, et al. unpublished data, 2008). Methods Setting The new post-encephalitis disability assessment score was developed, piloted and tested in two different clinical settings Une traduction en français de ce résumé figure à la fin de l’article. Al final del artículo se facilita una traducción al español. .ةلاقلما هذهل لماكلا صنلا ةياهن في ةصلاخلا هذهل ةيبرعلا ةمجترلا Objective To develop a simple tool for assessing the severity of disability resulting from Japanese encephalitis and whether, as a result, a child is likely to be dependent. Methods A new outcome score based on a 15-item questionnaire was developed after a literature review, examination of current assessment tools, discussion with experts and a pilot study. The score was used to evaluate 100 children in Malaysia (56 Japanese encephalitis patients, 2 patients with encephalitis of unknown etiology and 42 controls) and 95 in India (36 Japanese encephalitis patients, 41 patients with encephalitis of unknown etiology and 18 controls). Inter- and intra-observer variability in the outcome score was determined and the score was compared with full clinical assessment. Findings There was good inter-observer agreement on using the new score to identify likely dependency (Κ = 0.942 for Malaysian children; Κ = 0.786 for Indian children) and good intra-observer agreement (Κ = 1.000 and 0.902, respectively). In addition, agreement between the new score and clinical assessment was also good (Κ = 0.906 and 0.762, respectively). The sensitivity and specificity of the new score for identifying children likely to be dependent were 100% and 98.4% in Malaysia and 100% and 93.8% in India. Positive and negative predictive values were 84.2% and 100% in Malaysia and 65.6% and 100% in India. Conclusion The new tool for assessing disability in children after Japanese encephalitis was simple to use and scores correlated well with clinical assessment. Disability after encephalitis: development and validation of a new outcome score Penny Lewthwaite,a Ashia Begum,b Mong How Ooi,c Brian Faragher,d Boon Foo Lai,c Indunil Sandaradura,a Anand Mohan,c Gaurav Mandhan,b Pratibha Meharwade,b S Subhashini,b Gulia Abhishek,b Asma Begum,b Srihari Penkulinti,b M Veera Shankar,b R Ravikumar,b Carolyn Young,e Mary Jane Cardosa,f V Ravi,g See Chang Wong,c Rachel Kneenh & Tom Solomona a Brain Infections Group, University of Liverpool, 8th Floor Duncan Building, Daulby Street, Liverpool L69 3GA, England. b Vijayanagar Institute of Medical Sciences, Bellary, India. c Sibu Hospital and Institute of Health and Community Medicine, Universiti Malaysia, Sarawak, Malaysia. d Liverpool School of Tropical Medicine, Liverpool, England. e Neuro-rehabilitation Unit, The Walton Centre NHS Foundation Trust, Liverpool, England. f Institute of Health and Community Medicine, Universiti Malaysia, Sarawak, Malaysia. g National Institute of Mental Health and Neurological Sciences, Bangalore, India. h Royal Liverpool Children’s NHS Trust, Liverpool, England. Correspondence to Penny Lewthwaite (e-mail: pennylewthwaite@doctors.org.uk). (Submitted: 23 August 2009 – Revised version received: 29 November 2009 – Accepted: 4 December 2009 – Published online: 19 April 2010 ) Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357 585 Penny Lewthwaite et al. Development of post-encephalitis outcome score Research representative of locations across Asia where Japanese encephalitis occurs: (i) the Vijayanagar Institute of Medical Sciences, which is a government hospital in Bellary in southern India with basic diagnostic facilities but no paediatric intensive care facilities that serves the city (population: 0.5 million) and district (population: > 2 million) of Bellary;18 and (ii) Sibu Hospital, which is a referral hospital in Sarawak, Malaysia, with full intensive care facilities that serves the town of Sibu (population: 250 000) and the central region of Sarawak (popula- tion: 650 000).19 The outcome score A pilot version of the outcome score based on 20 questions was developed after a literature review and the examination of assessment tools used in developed and resource-poor countries, including the Ten Questions screening questionnaire for childhood disability, the Denver II child development screening test, the Pae- diatric Evaluation of Disability Inventory (PEDI) and three other assessment tools (Fig. 1).20–23 Written informed consent was obtained from the parent or guard- ian of each child. Approval for the study was granted by the ethics committees of the University of Liverpool in the United Kingdom of Great Britain and Northern Ireland and the Vijayanagar Institute of Medical Sciences in India, and the direc- tor of health of the state of Sarawak and the hospital director of Sibu Hospital in Sarawak. Participants After a pilot study in 2006 involving 51 children in India, the score ques- tionnaire was revised and applied in its current 15-question format in 2006 to a cohort of children who had had Japanese encephalitis in Sibu, Malay- sia,11 and to controls. The question- naire is available at: http://liv.ac.uk/ neuroscience/brain-infections/educa- tion_presentations.htm Subsequently in 2007, after minor modification and clarification of the questions, the score questionnaire was applied to a further cohort of children with suspected Japa- nese encephalitis (defined according to the World Health Organization surveil- lance standard definition)24 in Bellary, India. Children were invited to attend a follow-up assessment by post in Bel- lary and via the radio message system in Sarawak. Japanese encephalitis was con- firmed using an enzyme-linked immu- nosorbent assay on cerebrospinal fluid and serum, and patients who tested negative were classified as having acute encephalitis syndrome of unknown etiology. These patients may also have had Japanese encephalitis but, because of sample timing, we were unable to confirm this. Controls were selected at both sites from the siblings of patients who were assessed using the new score and from children who were well and attending the outpatient department for non-neurological conditions. Application of the score The score questionnaire requires the as- sessor to ask the child’s parent or carer to answer direct questions about the child’s ability to perform various daily activities or functions, such as speaking and feed- ing, in comparison with other children of the same age in their community. It was decided to compare children with others because expected norms vary enormously across communities and no normative data are available. The child is also observed performing simple motor functions, as described in the question- naire, available at: http://liv.ac.uk/ neuroscience/brain-infections/educa- tion_presentations.htm For each question, a set of possible answers scored from 2 to 5 is provided. A child whose response to a particular question is completely normal would score 5 for that question. One having minor sequelae that are reported, for example, as mild behavioural problems would score 4. A child having moderate sequelae that affect function but would not lead to dependence (e.g. difficulty walking) would scores 3. A child whose Fig. 1. The process of devising a new outcome score for assessing disability after Japanese encephalitis, India and Malaysia, 2005–2007 Literature review 20-question score devised Examination of existing disability assessment tools: - Ten Questions - Denver II - Paediatric Evaluation of Disability Inventory - draw-a-person test - AMPS - an unvalidated score from Viet Nam Opinion from developed and developing country experts: - paediatric neurologists - psychologists - occupational therapists Clinical assessment by paediatric neurologist with an AMPS occupational therapy assessment Pilot on 51 children in India - 39 with prior JE (of 107 invited) - 12 with AES of unknown etiology (of 139 invited) Multidisciplinary team reviews data, redundant items are dropped, questions are reworded and additional questions are added 15-question score applied to a cohort of children in Sibu, Malaysia - 56 with prior JE (of 72 invited) - 2 with AES of unknown etiology - 42 controls Minor modification and clarification of question wording 15-question score applied to a cohort of children in Bellary, India - 36 with prior JE (of 105 invited) - 41 with AES of unknown etiology - 19 controls AES, acute encephalitis syndrome; AMPS, Assessment of Motor and Process Skills; JE, Japanese encephalitis. Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357586 Penny Lewthwaite et al.Development of post-encephalitis outcome score Research impairment is so great that it would lead to dependence in that setting (e.g. being unable to walk in rural India) would score 2. Although impairments do change with time, particularly during childhood, it is difficult to predict the change.11 Consequently, for the purposes of the as- sessment tool, the child is classified on the basis of the individual evaluation alone. The final outcome score for each child, which ranges from I to V, corresponds to the lowest individual score recorded for any single question in the completed score sheet. For example, children whose impairment is severe enough in one domain to make them dependent will be dependent however well they might score in other domains. A score of I is given if the child has died; children who died were not considered further in this study. A score of II corresponds to a low- est single question score of 2 and indicates severe sequelae. Correspondingly, a score of III indicates moderate sequelae, IV indicates minor sequelae and V indicates full recovery. Although the assessment tool can identify the specific domains in which each child has difficulty, for the purposes of health economic and epidemiological analyses it is more useful to dichotomize children as either “dependent” or “inde- pendent” (i.e. likely to be capable of inde- pendent living). Children with a score of II were classed as dependent, while those with a score of III to V were classed as in- dependent. Scores in individual domains could also be examined and a total score ranging from 33–75 could be derived from the sum of all the individual scores, but these parameters were not assessed in this study. Local doctors were trained to use the new outcome assessment tool by discussing cases and with the aid of a PowerPoint (Microsoft, Redmond, United States of America) teaching tool. In both India and Malaysia medical edu- cation and training is mainly conducted in English, hence English versions of the questionnaire forms were used.15 Although less than ideal, this was felt to be a practical approach as more than 20 languages are in use in Sibu and more than 6 languages, in Bellary. It was not felt appropriate to translate the writ- ten questionnaire into local languages as the written format of some of the languages used is a more formal format than that used in everyday speech. The new outcome score questionnaire was applied by junior physicians who were not otherwise involved in the study. To investigate the inter-observer and intra-observer variability in the outcome score each child was assessed twice by each of two independent assessors. For practical reasons, these assessments were performed on the same day. However, assessors were unlikely to remember the classification they had given earlier be- cause so many children were assessed in a single day: in Sibu the median number per day was 7 (range: 4–10); in Bellary, it was 6 (range: 2–9). The new outcome score was validated by comparing each child’s score with the results of a full clinical consulta- tion carried out on the same day. The consultation comprised an assessment by a physician, including history-taking and developmental and full neurological examinations, and an examination by a specialized occupational therapist using the Assessment of Motor and Process Skills,25 which has been validated interna- tionally and cross-culturally for children aged 3 years and older. For children aged under 3 years, the doctor’s assessment alone was performed. Children were classified on the basis of the clinical assess- ment as having “severe” sequelae, which were likely to make the child dependent, or “moderate”, “minor” or “no” sequelae. The latter three categories were compat- ible with independent living. The clinical assessors were blinded to the outcome score and vice versa. In both India and Malaysia, the pres- ence of Japanese encephalitis virus infec- tion was confirmed using standard local assays for detecting Japanese encephalitis virus-specific immunoglobulin-M anti- body, as described previously.26,27 Statistical analysis To give a measure of item redundancy and the internal consistency of the ques- tionnaire, Cronbach’s α was determined during development of the assessment tool for both the pilot 20-question and the final 15-question scores.28 Inter- and intra-observer agreement for the new outcome score and the comparison between the new score and full clinical assessment were all assessed using the kappa (Κ) statistic and 95% confidence intervals (CIs) were computed using the large-sample modified formula.29 The sensitivity, specificity and positive and negative predictive values of the new as- sessment score relative to full clinical as- sessment were determined and their 95% CIs were computed using exact binomial formulae. Predictive validity was calcu- lated as the correlation between the new score and clinical assessment. Data were analysed using SPSS version 15 (SPSS Inc., Chicago, United States of America). Results The new outcome score Cronbach’s α was determined for the data on all children assessed using the pilot questionnaire in India and the results were used to revise the questionnaire and to produce the current 15-question version, shown in the questionnaire, available at: http://liv.ac.uk/neuroscience/brain- infections/education_presentations.htm In Sibu, Malaysia, of the 72 children (78%) invited for a follow-up assessment, 56 attended and were evaluated using the 15-question outcome score. The children were assessed a median of 69 months (range: 6–114) after their acute illness. Their median age was 11 years (range: 5–20; interquartile range, IQR: 8–13) and 18 (32%) were female. Forty-two con- trol children (median age: 8 years; range: 3–18; IQR: 6–10) were also assessed, as were two children who had initially been diagnosed with Japanese encephalitis but who were subsequently classified as having acute encephalitis syndrome of unknown etiology after a review of virology results. The score questionnaire was then used in Bellary, India, in a cohort of 36 children with prior Japanese encephalitis (median age: 8.5 years; range: 4–15; IQR: 6–11; 19 [53%] female) and 41 with acute encephalitis syndrome of unknown etiology (median age: 7 years; range: 2–17; IQR: 5–10; 22 [55%] female). These children were assessed a median of 15 months (range: 1–38) after acute illness. In addition, 19 healthy con- trol children were also assessed (median age: 7 years; range: 3–13; IQR: 4–11; 10 [53%] female). Each assessment took approximately 10 minutes for individuals experienced in using the new score questionnaire. In total, 779 assessments were made with the new clinical score in 196 children. If problems were identified, they were discussed with carers and referrals were made to local agencies, where available. Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357 587 Penny Lewthwaite et al. Development of post-encephalitis outcome score Research Redundancy in the score questions In the Malaysian cohort, Cronbach’s α for the 15-question outcome score was 0.927 for all observers and children com- bined and 0.894 for children who had had Japanese encephalitis. In addition, in the Indian cohort, Cronbach’s α was 0.787 for all observers and children com- bined and 0.849 for those who had had Japanese encephalitis, 0.708 for controls and 0.585 for those with acute encepha- litis syndrome of unknown etiology. No significant improvement in the internal consistency of the questionnaire could be made by excluding any of the 15 items and inter-item correlations were acceptable (data not shown). Inter- and intra-observer agreement In the Malaysian cohort, there was very good inter-observer agreement (Κ = 0.714) on the outcome score when children were classified according to the severity of their sequelae; the intra- observer agreement was also very good (Κ = 0.943). Moreover, when children were classified according to the dichoto- mous outcome of being dependent (i.e. a final outcome score of II) or independent (i.e. a final outcome score of III, IV or V), inter-observer agreement was very good (Κ = 0.942) and intra-observer agreement was perfect (Κ = 1.000). In the Indian cohort, inter-observer agreement was moderate (Κ = 0.584) and intra-observer agreement was good (Κ = 0.799) when the severity of sequelae was examined and inter- and intra-observer agreement were good (Κ = 0.786) and very good (Κ = 0.902), respectively, when the dichotomous outcome was examined. Details of these results are shown in Table 1, Table 2, Table 3 and Table 4. Validation Outcomes obtained using the 15-ques- tion score and clinical assessment were compared (Table 5). Four scores were used for each child: one from each of the two assessments carried out by each of the two observers. When the outcome compared was the severity of the sequelae, Table 1. Inter-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Sibu, Malaysia, 2006 Observer 1 Observer 2 Likely dependence Final outcome score (sequelae) Final outcome score (sequelae) Likely dependence II (severe) III (moderate) IV (mild) V (none) Total Depen- dent Indepen- dent Total Dependent II (severe) 18 2 0 0 20 18 2 20 Independent III (moderate) 0 23 4 3 30 0 180 180IV (mild) 0 0 9 13 22 V (none) 0 3 5 120 128 Total 18 28 18 136 200 18 182 200 Kappa value 0.714 (95% CI: 0.622–0.806)b 0.942 (95% CI: 0.862–1.000)b CI, confidence interval. a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V). b Kappa values were interpreted as follows: 0.0–0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement.30 Table 2. Intra-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Sibu, Malaysia, 2006 Observer 1 Observer 2 Likely dependence Final outcome score (sequelae) Final outcome score (sequelae)b Likely dependenceb II (severe) III (moderate) IV (mild) V (none) Total Depen- dent Indepen- dent Total Dependent II (severe) 19 0 0 0 19 19 0 19 Independent III (moderate) 0 28 0 1 29 0 179 179IV (mild) 0 0 18 1 19 V (none) 0 1 3 127 131 Total 19 29 21 129 198 19 179 198 Kappa value 0.943 (95% CI: 0.897–0.988)c 1.000c CI, confidence interval. a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V). b Two repeat scores were omitted and the inter-observer agreement for that item was calculated by comparing one score with the average of two from the second observer. c Kappa values were interpreted as follows: 0.0–0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement.30 Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357588 Penny Lewthwaite et al.Development of post-encephalitis outcome score Research a moderate level of agreement was found between the new score and clinical assess- ment: Κ = 0.544 for the Malaysian cohort and Κ = 0.467 for the Indian cohort. When the outcome compared was the child being dependent or independent, very good agreement was found, with Κ = 0.906 and Κ = 0.762 for the Malay- sian and Indian cohorts, respectively. The sensitivity and specificity of the new score in identifying children likely to be dependent, as determined by clinical assessment, were 100% (95% CI: 89.1–100) and 98.4% (95% CI: 96.5–99.4), respectively, in Malaysia and 100% (95% CI: 91.2–100) and 93.8% (95% CI: 90.7–96.0), respectively, in India. The positive predictive values were 84.2% (95% CI: 68.7–94.0) and 65.6% (95% CI: 52.3–77.3) for the Malaysian and Indian cohorts, respectively, and the negative predictive values were 100% (95% CI: 98.6–100) and 100% (95% CI: 98.5–100), respectively. Overall only 3.8% of children categorized as independent on clinical assessment were incorrectly classified by the outcome score as dependent. Discussion The inability to measure disability using a simple tool has been identified as one of the key reasons for the lack of data on disease burden among children living in poor countries.1 The resulting gaps in knowledge mean that there is often insufficient evidence to drive changes in public health policy.31 Nothing provides a better example of this problem than the failure to control Japanese encephalitis over the past 40 years. Without good data on disease burden, the impetus to implement vaccination programmes has been haphazard. As more vaccines become available and as they become cheaper, countries will have to make important decisions about public health priorities.32,33 In particular, simple reliable ways of measuring disability are needed for diseases such as Japanese encephalitis, whose morbidity rate is much higher than the 8–30% mortality rate.10,11 Our aim was to design and validate a disability assessment tool that can be applied relatively quickly and easily by a range of health-care workers in different Table 3. Inter-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Bellary, India, 2007 Observer 1 Observer 2 Likely dependence Final outcome score (sequelae) Final outcome score (sequelae) Likely dependence II (severe) III (moderate) IV (mild) V (none) Total Depen- dent Indepen- dent Total Dependent II (severe) 25 4 4 2 35 25 10 35 Independent III (moderate) 0 31 7 4 42 1 153 154IV (mild) 0 10 14 15 39 V (none) 1 3 6 63 73 Total 26 48 31 84 189 26 163 189 Kappa value 0.584 (95% CI: 0.495–0.674)b 0.786 (95% CI: 0.666–0.906)b CI, confidence interval. a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V). b Kappa values were interpreted as follows: 0.0–0.2, poor agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement.30 Table 4. Intra-observer agreementa for new 15-question outcome score for assessing post-encephalitis disability in children, Bellary, India, 2007 Observer 1 Observer 2 Likely depen- dence Final outcome score (sequelae) Final outcome score (sequelae)b Likely dependenceb II (severe) III (moderate) IV (mild) V (none) Total Depen- dent Indepen- dent Total Dependent II (severe) 28 0 0 1 29 28 1 29 Independent III (moderate) 2 38 3 4 47 4 155 159IV (mild) 2 5 25 7 39 V (none) 0 0 3 70 73 Total 32 43 31 82 188 32 156 188 Kappa value 0.799 (95% CI: 0.729–0.868)c 0.902 (95% CI: 0.818–0.987)c CI, confidence interval. a Agreement is shown for outcomes classified both in terms of four severity levels of sequelae and in terms of a dichotomous outcome: dependent (i.e. final outcome score: II) or independent (i.e. final outcome score: III–V). b One repeat score was omitted and the inter-observer agreement for that item was calculated by comparing one score with the average of two from the second observer. c Kappa values were interpreted as follows: 0.0–0.2 poor agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 good agreement; and 0.81–1.00 very good agreement.30 Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357 589 Penny Lewthwaite et al. Development of post-encephalitis outcome score Research settings. None of the currently available scores, such as the Ten Questions, Denver II or PEDI score, meets this need. The Ten Questions was devised as a community screening tool to identify children who should be referred for neurological as- sessment but is too nonspecific for use as an assessment tool.34 The Denver II tool assesses disability in children and is widely used in Europe and North America.21 However, it is usually applied by paediatri- cians and requires at least 35 minutes. It is also dependent on the cultural setting, though it has recently been adapted for use in Malawi.14 The PEDI is another well-established and widely-used tool. However, it was designed for use in the developed world.20 Finally, the World Health Organization Disability Assess- ment Schedule II (WHO DAS II), which is in development, assesses patients’ needs, functioning and outcomes but is designed for an adult population.35 In developing the new outcome score we faced considerable challenges and had to accept many compromises. We had to accept that a scoring system would never match an assessment performed over sev- eral months by a multidisciplinary team. However, it would still be better than the disease outcome “discharged alive” so often recorded in hospital notes in many parts of rural Asia. We chose to focus on a single disease, Japanese encephalitis, because it is one of the most important causes of acquired brain injury in Asian children. However, the brain injury resulting from Japanese encephalitis is very similar to that associated with other infectious or noninfectious causes, such as trauma. Most disability assessment tools are generic and, with further validation, the new score can perhaps be used more generally across the spectrum of acquired neurological disability. One limitation of our study was that the proportion of children that responded to a request to attend a follow-up assess- ment was limited, especially in rural India, where distances to hospital are great. However, we felt it was important to de- velop the score in real-life settings where it will be used in practice rather than in the logistically easier, but less relevant, set- ting of a large teaching hospital. We were concerned that sicker and more disabled children might not be able to attend follow-up assessments, but our visits to rural villages to track down nonattendees indicated that it was those who recovered fully that were less likely to attend. Ideally the new score would have been compared with a full multidisciplinary team assess- ment performed over several visits, but again this was not practical: even assess- ment by the clinician and occupational therapist took 60–90 minutes. One of the challenges was to develop a single scoring system that could be ap- plied in a wide range of age groups, in different settings and in areas where there are no normative data. Our solution was to ask the caregiver to compare the child with other children of the same age in the same community. Although this is a crude measure that is dependent on the caregiver, a parent’s judgement of a child’s level of development and abilities is usually correct. This approach allows for cultural differences across Asia; for example, Indian children feed themselves at a younger age than Malaysian children. Cultural differences and the child’s living conditions could mean that an inability to walk would make the child dependent in one setting, for example in rural India, but not another, for example in urban Malaysia, where wheelchairs are available. We felt this was a pragmatic approach because, when looking at disease burden, the impact of a disability is more impor- tant than neurological observations or biological dysfunction. We did not attempt to classify or quantify disablement in terms of im- pairment, disability (i.e. in activity) or Table 5. Outcomes obtained with new 15-question outcome score for assessing post-encephalitis disability compared with the outcomes of clinical assessment in 196 children in Malaysia and India, 2006–2007 Outcome score assessmenta Clinical team assessment Likely dependenceb Final outcome score (sequelae) Final outcome score (sequelae) Likely dependenceb II (severe) III (moderate) IV (mild) V (none) Total Depen- dent Indepen- dent Total Malaysia Dependent II (severe) 32 0 2 4 38 32 6 38 Independent III (moderate) 0 28 19 11 58 0 362 362 IV (mild) 0 9 25 6 40 V (none) 0 7 46 211 264 Total 32 44 92 232 400 32 368 400 Kappa value 0.544 (95% CI: 0.473–0.616)c 0.906 (95% CI: 0.832–0.980)c India Dependent II (severe) 40 11 7 3 61 40 21 61 Independent III (moderate) 0 23 49 19 91 0 318 318 IV (mild) 0 6 51 13 70 V (none) 0 0 37 120 157 Total 40 40 144 155 379 40 339 379 Kappa value 0.467 (95% CI: 0.400–0.534)c 0.762 (95% CI: 0.666–0.858)c CI, confidence interval. a Most children were assessed twice using the 15-question outcome score by each of two observers. Overall, there were 779 assessments in 100 children in Malaysia and 96 in India. b Children were classified as likely to be dependent if their final outcome score was II and as independent if their final outcome score was III–V. c Kappa values were interpreted as follows: 0.0–0.2 poor agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 good agreement; and 0.81–1.00 very good agreement.30 Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357590 Penny Lewthwaite et al.Development of post-encephalitis outcome score Research handicap (i.e. in participation). Rather, we developed an assessment tool that identifies children who, after having Japanese encephalitis, suffer a loss of functional ability compared to their peer group. For practical reasons, junior physi- cians applied the tool in our study, though other health-care workers have now used it without difficulty (unpublished observations). The tool is, if anything, oversensitive in predicting disability, but only 3.8% of children were incorrectly classified as dependent by the outcome score. We felt this was a reasonable pro- portion since we wanted to ensure that no dependent child was missed. Recent data show that children with Japanese encephalitis may improve or deteriorate many months after the initial insult.11 Consequently, further work needs to be done in following up a prospective cohort to determine the correlation between the outcome score at hospital discharge with that 3 months and 3 to 5 years later. This information will enable us to determine the time at which the new outcome score will give the best prediction of long-term outcome. In addition, the test–retest reliability of the score now needs to be examined, as does its sensitivity to change over time and its potential for use in acute brain injury due to other causes. In summary, we have developed a simple outcome score for detecting dis- ability in children affected by Japanese encephalitis, a common cause of acquired neurodisability in Asia. Although the tool has limitations, its ability to identify chil- dren with “likely disability”, as judged by the clinical team, was good, with good to very good inter- and intra-observer agree- ment. It is now being used in several Asian countries affected by Japanese encephali- tis and should be suitable for modification to assess acquired neurodisability due to other causes in children in resource-poor countries. ■ Acknowledgements We are grateful to all the patients and their carers for assisting with this work. Thanks are also due to staff and patients at the Vijayanagar Institute of Medical Sci- ences, Bellary, who assisted with the study, and, in particular, the Medical Director and also Kailash Soni and Thomas Schulz. In Malaysia, we thank the hospital staff and patients at Sibu Hospital. We also thank JM Lewthwaite. We also thank Julie Jacobson and Susan Hills of the PATH Japanese Encephalitis Project for their support and encouragement, Mary Gainsborough for help with the initial development of the score and Janet Daly for help with the manuscript. Mong How Ooi and Rachel Kneen also work with the Brain Infections Group, University of Liverpool, Liverpool, United Kingdom and Tom Solomon is Chair of Neuro- logical Science at the Walton Centre for Neurology and Neurosurgery, Liverpool, United Kingdom. Funding: The project was supported by the Bill & Melinda Gates Foundation- funded Japanese encephalitis programme at PATH. Mong How Ooi is funded by the Wellcome Trust in the United Kingdom. Tom Solomon is funded by the United Kingdom Medical Research Council. Competing interests: None declared. صخلم جئاتنلل ديدج زرح ةيقودصم قيثوتو دادعإ :يغامدلا باهتللاا دعب ةقاعلإا يغامدلا باهتللاا نع ةجتانلا ةقاعلإا ةدش مييقتل ةطيسب ةادأ دادعإ ضرغلا هيرغ لىع ةلاع كلذل ةجيتن لفطلا يرصي نأ لماتحاو نيابايلا دعب ًادنب 15 نم نّوكم نايبتسا لىإ ًادانتسا جئاتنلل ديدج زرح دُعأ ةقيرطلا ،ءابرخلا ةشقانمو ،ةيلاحلا سايقلا تاودأ صحفو ،ةيبطلا تافلؤلما ةعجارم 56( ايزيلام في لفط 100 مييقتل زرحلا مدختساو .ةيدايترا ةسارد ءارجإو و ،فورعم يرغ يغامد باهتلاب ينضيرم و ،نيابايلا يغامدلا باهتللااب ًاضيرم يغامدلا باهتللااب ًاضيرم 36( دنهلا في ًلافط 95 و )دهاوشلا نم ًلافط 42 .)دهاوشلا نم ًلافط 18 و ،فورعم يرغ يغامد باهتلاب ًاضيرم 41 و ،نيابايلا زرح ليجست في بقارم لك ىدلو ينبقارلما ينب تافلاتخلاا ديدحت ىرجو .لماشلا يريسرلا مييقتلاب زرحلا نروقو ،جئاتنلا ديدجلا زرحلا مادختسا لىع ينبقارلما ينب ديج قفاوت كانه ناك تادوجولما Κ= ؛ينيزيلالما لافطلأل Κ= 0.942( ةلاعلإل لافطلأا ةجاح لماتحا ديدحتل دحاولا بقارملل ةبسنلاب ديج قفاوت كانه ناكو )دونهلا لافطلأل 0.786 ديدجلا زرحلا ينب قفاوتلا نأ لىإ ةفاضلإاب .)لياوتلا لىع 0.902 و Κ= 1.000( اتناكو .)لياوتلا لىع 0.762 و Κ= 0.906( ًاضيأ ًاديج ناك كيينيلكلإا مييقتلاو ةلاعلإل مهتجاح حجرلما لافطلأا ديدحت في ديدجلا زرحلا ةيعونو ةيساسح مّيقلا تناكو .دنهلا في 93.8% و 100% و ،ايزيلام في 98.4% و 100% ماه و 65.6% و ،ايزيلام في 100% و 84.2% يه ةيبلسلاو ةيباجيلإا ةينهكتلا .دنهلا في 100% باهتللااب ةباصلإا دعب لافطلأا ينب ةقاعلإا مييقتل ةديدجلا ةادلأا جاتنتسلاا عم ًاديج ًاطابترا زرحلا طبتراو مادختسلاا ةيرسي تناك نيابايلا يغامدلا . يريسرلا مييقتلا Résumé Invalidité post-encéphalitique : conception et validation d’une nouvelle échelle d’évaluation Objectif Développer un outil simple pour évaluer la gravité de l’invalidité résultant de l’encéphalite japonaise et, en conséquence, la probabilité de dépendance de l’enfant. Méthodes Une nouvelle échelle d’évaluation basée sur un questionnaire de 15 questions a été développée après une analyse de la littérature, l’examen des outils d’évaluation actuels, une discussion avec les experts et une étude pilote. L’échelle a été utilisée pour évaluer 100 enfants en Malaisie (56 patients atteints d’encéphalite japonaise, 2 patients atteints d’encéphalite d’étiologie inconnue et 42 contrôles) et 95 en Inde (36 patients atteints d’encéphalite, 41 patients atteints d’encéphalite d’étiologie inconnue et 18 contrôles). La variabilité inter et intra-observateurs dans l’échelle d’évaluation a été déterminée et l’échelle a été comparée avec une évaluation clinique complète. Résultats Il a été observé une bonne correspondance inter-observateurs dans l’utilisation de la nouvelle échelle pour identifier la dépendance probable (Κ=0,942 pour les enfants malais; Κ=0,786 pour les Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357 591 Penny Lewthwaite et al. Development of post-encephalitis outcome score Research enfants indiens), ainsi qu’une bonne correspondance intra-observateurs (Κ=1,000 et 0,902, respectivement). De plus, la correspondance entre la nouvelle échelle et l’évaluation clinique a été bonne (Κ=0,906 et 0,762, respectivement). La sensibilité et la spécificité de la nouvelle échelle d’évaluation de la probabilité de dépendance des enfants ont été de 100 % et de 98,4 % en Malaisie et de 100 % et de 93,8 % en Inde. Les valeurs prédictives positives et négatives ont été de 84,2 % et de 100 % en Malaisie et de 65,6 % et de 100 % en Inde. Conclusion Le nouvel outil pour l’évaluation de l’invalidité des enfants après encéphalite japonaise a été simple à utiliser et les résultats présentent une bonne corrélation avec l’évaluation clinique. Resumen Incapacidad tras la encefalitis: desarrollo y validación de una nueva escala de resultados Objetivos Diseñar una herramienta sencilla para valorar la gravedad de la incapacidad causada por la encefalitis japonesa y la posibilidad de que un niño sea dependiente como consecuencia de la misma. Métodos Se ha elaborado una nueva escala de resultados, basada en un cuestionario de 15 puntos, realizado tras una revisión bibliográfica, en el estudio de las herramientas de valoración actuales, en el debate con expertos y en un estudio preliminar. La escala se empleó para evaluar a 100 niños en Malasia (56 pacientes con encefalitis japonesa, 2 pacientes con encefalitis de etiología desconocida y 42 controles) y 95 en India (36 pacientes con encefalitis japonesa, 41 pacientes con encefalitis de etiología desconocida y 18 controles). Se determinó la variabilidad interobservador e intraobservador en la escala de resultados y se comparó la escala con una valoración clínica completa. Resultados Hubo un consenso interobservador favorable respecto a la utilización de la nueva escala para identificar la posible dependencia (Κ = 0,942 en el caso de los niños malasios; Κ = 0,786 para los niños indios) y un consenso intraobservador favorable ( Κ = 1,000 y 0,902, respectivamente). Además, el consenso entre la nueva escala y la valoración clínica también fue bueno (Κ = 0,906 y 0,762, respectivamente). La sensibilidad y la especificidad de la nueva escala para identificar a los niños que pueden ser dependientes fue del 100% y del 98,4% en Malasia, y del 100% y del 93,8% en India. Los valores predictivos positivos y negativos fueron del 84,2% y del 100% en Malasia, y del 65,6% y del 100% en India. Conclusiones La nueva herramienta de valoración de la incapacidad infantil tras la encefalitis japonesa fue fácil de usar y los resultados estaban relacionados con la valoración clínica. References 1. Mung’ala-Odera V, Newton CR. Identifying children with neurological impairment and disability in resource-poor countries. Child Care Health Dev 2007;33:249–56. doi:10.1111/j.1365-2214.2006.00714.x PMID:17439437 2. Immunization and vaccine development: Japanese encephalitis. New Delhi: World Health Organization Regional Office for South-East Asia; 2006. Available from: www.searo.who.int/en/section1226/section2073.asp [accessed 20 January 2010]. 3. Tsai TF. New initiatives for the control of Japanese encephalitis by vaccination: minutes of a WHO/CVI meeting, Bangkok, Thailand, 13–15 October 1998. Vaccine 2000;18(Suppl 2):1–25. doi:10.1016/S0264- 410X(00)00037-2 PMID:10821969 4. Solomon T. Control of Japanese encephalitis — within our grasp? N Engl J Med 2006;355:869–71. doi:10.1056/NEJMp058263 PMID:16943399 5. Halstead SB, Jacobson J. Japanese encephalitis. Adv Virus Res 2003;61:103–38. doi:10.1016/S0065-3527(03)61003-1 PMID:14714431 6. Beasley DW, Lewthwaite P, Solomon T. Current use and development of vaccines for Japanese encephalitis. Expert Opin Biol Ther 2008;8:95–106. doi:10.1517/14712598.8.1.95 PMID:18081539 7. Ding D, Kilgore PE, Clemens JD, Wei L, Zhi-Yi X. Cost-effectiveness of routine immunization to control Japanese encephalitis in Shanghai, China. Bull World Health Organ 2003;81:334–42. PMID:12856051 8. Richter RW, Shimojyo S. Neurologic sequelae of Japanese B encephalitis. Neurology 1961;11:553–9. PMID:13741420 9. Kumar R, Mathur A, Singh KB, Sitholey P, Prasad M, Shukla R et al. Clinical sequelae of Japanese encephalitis in children. Indian J Med Res 1993;97:9– 13. PMID:8387460 10. Rayamajhi A, Singh R, Prasad R, Khanal B, Singhi S. Study of Japanese encephalitis and other viral encephalitis in Nepali children. Pediatr Int 2007;49:978–84. doi:10.1111/j.1442-200X.2007.02495.x PMID:18045307 11. Ooi MH, Lewthwaite P, Lai BF, Mohan A, Clear D, Lim L et al. The epidemiology, clinical features, and long-term prognosis of Japanese encephalitis in central Sarawak, Malaysia, 1997–2005. Clin Infect Dis 2008;47:458–68. doi:10.1086/590008 PMID:18616397 12. Jones P, Drummond A. Occupational therapy for children with acquired brain injury: a review of the literature. Br J Occup Ther 2005;68:324–30. 13. Carter JA, Lees JA, Murira GM, Gona J, Neville BG, Newton CR. Issues in the development of cross-cultural assessments of speech and language for children. Int J Lang Commun Disord 2005;40:385–401. doi:10.1080/13682820500057301 PMID:16195197 14. Gladstone MJ, Lancaster GA, Jones AP, Maleta K, Mtitimila E, Ashorn P et al. Can Western developmental screening tools be modified for use in a rural Malawian setting? Arch Dis Child 2008;93:23–9. doi:10.1136/ adc.2006.095471 PMID:17379661 15. Japanese encephalitis teaching resources and outcome score. Liverpool: University of Liverpool; 2009. Available from: http://liv.ac.uk/neuroscience/ brain-infections/education_presentations.htm [accessed 28 January 2010]. 16. Liverpool outcome score for assessing children at discharge. Seattle: PATH; 2009. Available from: http://www.path.org/vaccineresources/details. php?i=676 [accessed 28 February 2009]. 17. Maha MS, Moniaga VA, Hills SL, Widjaya A, Sasmito A, Hariati R et al. Outcome and extent of disability following Japanese encephalitis in Indonesian children. Int J Infect Dis 2009;13:e389–93. doi:10.1016/j. ijid.2009.01.009 PMID:19346144 18. Health: general information and important statistics. Bellary: National Informatics Centre, India; 2010. Available from: http://www.bellary.nic.in/ Health.htm [accessed 31 May 2008]. 19. Department of Statistics. Population and vital statistics. Yearbook of statistics. Sarawak 2006. Kuala Lumpur: DS;2006. 20. Haley SM, Coster WJ, Ludlow LH, Haltiwanger JT, Andrellos PJ. Pediatric evaluation of disability inventory: development, standardization, and administration manual, version 1.0. Boston: Trustees of Boston University, Health and Disability Research Institute; 1992. 21. Frankenburg WK, Dodds J, Archer P, Shapiro H, Bresnick B. The Denver II: a major revision and restandardization of the Denver Developmental Screening Test. Pediatrics 1992;89:91–7. PMID:1370185 22. Durkin MS, Davidson LL, Desai P, Hasan ZM, Khan N, Shrout PE et al. Validity of the ten questions screened for childhood disability: results from population- based studies in Bangladesh, Jamaica, and Pakistan. Epidemiology 1994;5:283–9. doi:10.1097/00001648-199405000-00005 PMID:7518697 Bull World Health Organ 2010;88:584–592 | doi:10.2471/BLT.09.071357592 Penny Lewthwaite et al.Development of post-encephalitis outcome score Research 23. Solomon T, Dung NM, Kneen R, Thao le TT, Gainsborough M, Nisalak A et al. Seizures and raised intracranial pressure in Vietnamese patients with Japanese encephalitis. Brain 2002;125:1084–93. doi:10.1093/brain/awf116 PMID:11960897 24. WHO recommended standards for surveillance of selected vaccine- preventable diseases. Geneva: World Health Organization; 2003. Available from: http://www.who.int/vaccines-documents/DocsPDF06/843.pdf [accessed 29 January 2010]. 25. Robinson SE, Fisher AG. A study to examine the relationship of the Assessment of Motor and Process Skills (AMPS) to other tests of cognition and function. Br J Occup Ther 1996;59:260–3. 26. Cardosa MJ, Wang SM, Sum MS, Tio PH. Antibodies against prM protein distinguish between previous infection with dengue and Japanese encephalitis viruses. BMC Microbiol 2002;2:9. doi:10.1186/1471-2180-2-9 PMID:12019028 27. Ravi V, Desai A, Balaji M, Apte MP, Lakshman L, Subbakrishna DK et al. Development and evaluation of a rapid IgM capture ELISA (JEV-Chex) for the diagnosis of Japanese encephalitis. J Clin Virol 2006;35:429–34. doi:10.1016/j.jcv.2005.11.004 PMID:16406800 28. Streiner DLN, Norman GR. Health measurement scales: a practical guide to their development and use. 3rd ed. Oxford: Oxford University Press; 2003. 29. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull 1969;72:323–7. doi:10.1037/h0028106 30. Altman DG. Practical statistics for medical research. Padstow: TJ International; 1993. 31. Maulik PK, Darmstadt GL. Childhood disability in low- and middle-income countries: overview of screening, prevention, services, legislation, and epidemiology. Pediatrics 2007;120(Supplement 1):S1–55. doi:10.1542/ peds.2007-0043B PMID:17603094 32. Solomon T. New vaccines for Japanese encephalitis. Lancet Neurol 2008;7:116–8. doi:10.1016/S1474-4422(08)70004-8 PMID:18207104 33. Japanese encephalitis vaccines. Wkly Epidemiol Rec 2006;81:331–40. PMID:16933380 34. Durkin MS, Hasan ZM, Hasan KZ. The ten questions screen for childhood disabilities: its uses and limitations in Pakistan. J Epidemiol Community Health 1995;49:431–6. doi:10.1136/jech.49.4.431 PMID:7650469 35. World Health Organization Disability Assessment Schedule II (WHODAS II). Geneva: World Health Organization; 2001. Available from: http://www.who.int/ icidh/whodas/ [accessed 25 October 2009].