Validating a method of psychiatric case identification in Jamaica

Validating a method of psychiatric case identification in Jamaica T. W. HARDING 1 There is an urgent need for simple, reliable instruments for psychiatric research in developing countries. Limited resources, however, make the development of instruments de novo difficult. A simple methodfor establishing the validity of a method of case identi- fication in a new population is described. A shortened version of a standard general health questionnaire was shown to be an effective method of identifying non-psychotic disorders in Jamaica, subject to a recalibration. The limited level of resources available for health care in developing countries accentuates the need for well conducted research into the requirements for and the effectiveness of interventions. In the mental health field one of the main constraints to research is the lack of suitable instruments, since almost all instru- ments for case detection, symptom rating, and stan- dardized diagnosis have been developed and valid- ated in the industrialized countries. The procedure for developing, establishing the reliability of, and validating such instruments from scratch is lengthy and expensive. The experience in the Cornell-Aro study of psychiatric disorders in part of Western Nigeria (7) gave some confidence that psychiatric methods developed in one setting could be adapted for use in a completely different culture. More recently the International Pilot Study of Schizo- phrenia (12) has demonstrated that a standard set of research instruments could be developed for use in widely different cultural settings and could pro- duce reliable and valid results. It therefore seems reasonable to concentrate considerable effort on methods for validation and adaptation of research instruments developed in one country for use in a new sociocultural setting. This paper describes a method whereby the val- idity of a simple method of case detection originally developed and tested in the United Kingdom was tested in Jamaica. 1 Lecturer, Department of Psychiatry, University of the West Indies, Mona, Kingston, Jamaica. Present address: Medical Officer, Office of Mental Health, World Health Organization, Geneva, Switzerland. The problem of case identification Case identification is a central problem in psychi- atric research, particularly in surveys of the general population and in prevalence studies of patients attending general hospitals, health centres, or general practitioners. In some studies, psychiatrists have been used as case identifiers (2) and the reliability of this method can be increased by the use of structured interviews and rating scales (4, 5). It is nevertheless an expensive method and in many parts of the world could not be used because of a shortage of trained psychiatrists. An alternative is the identi- fication of cases by psychiatrists on the basis of information collected by research assistants with special training, as in the midtown Manhattan survey (11) and the Stirling County survey (6). In these surveys, lengthy, highly structured interviews have been used. Although the Stirling County survey method was later adapted for a survey in part of Western Nigeria (7), it would be difficult to use in many parts of the world because of the training involved and the need for extensive psychiatrist's time. A structured interview that can be administered by a non-psychiatrist and directly yields a score related to psychiatric morbidity is therefore an attractive alternative. The Health Opinion Survey (9) is one such method, but Leon & Climent (8) have concluded that neither this method nor the use of a symptom checklist is suitable for case finding in prevalence studies, if psychiatric examination is taken as the criterion. Potentially the simplest and most attractive method is the use of a self-administered questionnaire in 3512 - 225 - BULL. WORLD HEALTH ORGAN., Vol. 54, 1976 T. W. HARDING case identification, particularly if the questionnaire is relatively brief, acceptable, and comprehensible. The Cornell Medical Inventory (1) has been used to identify cases but the misclassification rate is unacceptably high (10). The General Health Questionnaire (GHQ) The introduction by Goldberg (3) of a self-admin- istered questionnaire, frequently referred to as the General Health Questionnaire (GHQ), which has been validated in general practice and medical out- patient populations in the United Kingdom and in a general practice population in Philadelphia, USA, and for which sensitivity varies between 85% and 96% and specificity varies between 80% and 88%, is therefore of great interest, since it provides a highly effective tool for the identification of non- psychotic psychiatric illness in a variety of situations. The GHQ exists in two forms, the 30-item and 60-item versions. In both versions respondents are asked to reply to a series of questions by underlining one of four possible responses. All questions are prefaced by the phrase " Have you recently . .. ? ". Examples of questions and possible responses are: - Have you recently lost much sleep over worry? Not at all/No more than usual/Rather more than usual/Much more than usual - Have you recently been feeling hopeful about your own future? More than usual/About same as usual/Less so than usual/Much less than usual For each question there is a score of 0 (for either of the first two possible responses for each question) or 1 (for either of the latter two possible responses), giving a maximum possible score of 30 and 60 for the 30-item and 60-item versions, respectively. Validation in new settings If the GHQ (or a similar instrument) can be shown to be a valid method in a variety of settings and countries, there would be a greatly increased poten- tial for epidemiological and evaluative research. The need for such a simple and reliable method of case identification is particularly great in the developing countries since research is needed to indicate the need for mental health care, in patients attending general clinics and health centres, and to evaluate the results of low cost intervention. In such a situ- ation research methods must be relatively simple and inexpensive as well as reliable and must yield valid results. Because of the limited resources in such countries, it would be important to devise simple and inexpensive ways to demonstrate the validity of an instrument without resorting to the complex series of steps involved in the original development of the instrument and its validation (3). There was clearly a need for such an instrument in Jamaica but skilled epidemiological and psychi- atric manpower was not sufficiently available either to devise epidemiological instruments de novo or to undertake rather complex validation procedures based on a general population sample as, for exam- ple, used by Leon & Climent (8) in assessing the Health Opinion Survey (HOS) as a screening instru- ment in Colombia. For these reasons a two-stage validation pro cedure was devised as follows. Stage 1. This was carried out in an outpatient psychiatric clinic, using new patients attending for psychiatric assessment and a control group of rela- tives accompanying patients to non-psychiatric hos- pital clinics. This stage was designed to indicate any striking problems of acceptability, comprehension, or administration of the GHQ and to provide some indication as to whether the GHQ discriminated be- tween " cases " and " normals ". Only if the instru- ment proved promising in this first stage was the more exacting second stage to be undertaken. Stage 2. This was carried out in a rural health centre using a standardized psychiatric interview as the criterion for the case detecting effectiveness of the GHQ. METHODS The 30-item version of the GHQ was used since it was easier to administer to illiterate patients. The first stage of the study was carried out in outpatient clinics of the University Hospital of the West Indies, Mona, and the second stage was to be initiated only if the GHQ satisfied the criteria of acceptability (a refusal rate of less than 5%), ease of administration (to be administered by a clerk in a mean time of less than 20 min), and discrimination (an overall misclassification rate of less than 20%) during the study's first stage. In the first stage, all new patients over 18 years of age attending the regular weekly psychiatric out- patient clinic of the University Hospital of the West Indies during a six-week period were asked to com- plete the GHQ. Most patients were referred from other clinics of the hospital or from the " Casualty Department ", which in fact functions as a primary health care clinic as well as an Accident and Emer- gency Department. A few patients were referred by 226 PSYCHIATRIC CASE IDENTIFICATION IN JAMAICA general practitioners. The GHQ was filled in by patients while waiting to be seen by a psychiatrist. In the case of illiterate patients, the questions and possible answers were read by a clerk. Where doubt was expressed about the meaning of any question, an explanation was given by the clerk using his own words. The GHQ score was not available to the psychiatrist during the subsequent consultation. Since the questionnaire is designed to detect " non- psychotic psychiatric illness ", only those patients in whom a definite diagnosis of one or other form of psychoneurosis was made at the diagnostic inter- view by a trained psychiatrist were included in sub- sequent analysis. All other patients (e.g., with func- tional psychosis, organic brain disease, mental retar- dation, alcoholism, personality disorder, or doubtful diagnosis) were excluded from further consideration. The patient sample consisted, therefore, of adults attending a general hospital psychiatric clinic as new patients in whom a definite diagnosis of a psycho- neurotic illness was made by a trained psychiatrist. The control group was drawn from relatives of patients attending the medical and surgical out- patient clinics of the same hospital. Relatives were asked if they would agree to complete a simple written or verbal test; 85% of those approached agreed. A series of 4 screening questions was then asked as follows: Are you less than 18 years old? Are you receiving treatment for an illness of any kind? Is your general health poor or are you concerned about any troubles at present? Have you suffered from nerves or depression? Any relative answering " Yes " to one or more of these questions was excluded. The GHQ was then administered in the same way as for the patient group. The control group, therefore, consisted of adults, accompanying relatives to hospital outpatient clinics, who were not receiving any medical treat- ment and who perceived themselves as healthy. The second stage of the validity study was carried out at the Rural Health Centre in Lawrence Tavern, situated 24 km from Kingston, the island's capital. The centre serves a population of approximately 10 000 people of predominantly African origin in an agricultural area notable for its steep terrain. A " doc- tor's clinics" was held at the centre twice weekly. While they were waiting to see the doctor, all patients over the age of 18 years attending the clinic were asked by an attendant (with no formal training beyond third year in secondary school) to fill in a questionnaire with her help as part of a " health study ". The two attendants involved were both told to read the instructions on the questionnaire to each patient twice and then to ask whether they had been understood. If in any doubt, the instructions were read again and, if necessary, explained. Those patients who seemed keen to complete the questionnaire them- selves were allowed to do so, but for those who expressed any doubt, the questions and possible answers were read by the attendant. If the patient hesitated, the question was read again and, if necess- ary, " explained in the attendant's own words ". The first response given by the patient was recorded by the attendant. The patients then saw the clinic doctor as usual. Finally patients were seen by a psychiatrist for a standardized psychiatric assessment (4) which allows patients to be assigned to a position on a six-point scale (0-5), of which categories 0 and 1 were taken as indicating no significant psychiatric disturbance and categories 2, 3, 4, and 5 as indi- cating significant psychiatric disturbance. Since the standardized interview took approximately three times longer than the clinic doctor's consultation, only one patient in three could be included. Patients were therefore selected by the clinic doctor to be seen by the psychiatrist using a random method based on a pack of cards. RESULTS The first stage Acceptability. A total of 145 patients and relatives were asked to complete the questionnaire with or without the help of an attendant: 112 were waiting to be seen at the psychiatric clinic (of these, 31 were subsequently diagnosed as psychoneurotic and there- fore included in the validation study), and 33 were relatives of non-psychiatric patients. The numbers refusing to complete the questionnaire are shown in Table 1. The overall refusal rate was therefore 4.1 %. If only those patients and controls otherwise qualify- ing for the validation study are considered, the refusal rate was only 3 %. Administration. All questionnaires were given to patients by hospital clerks routinely assigned to the clinics. Instructions for these clerks consisted of a 45-min introduction and test run. The number of patients for whom questionnaires could not be com- pleted is shown in Table 1 (reasons given by clerks and attendants included "unable to understand ", " deaf and illiterate ", and "stupidity "). The overall rate for inability to complete questionnaires was 4.3% but if those who did not otherwise qualify for the 227 T. W. HARDING Table 1. Acceptability of questionnaires in stage 1 a No. asked Unable to No. for whom No. of Group to complete Refusals complete questionnaire completed questionnaire for other reasons was read questionnaires Psychiatric clinic patients: Psychoneurotic 31 (100 %) 1 (3 %) - 12 (39%) 30 (97 %) Other diagnoses 81 (100%) 4 (5%) 5 (6%) 47 (58%) 72 (899%) Control group 33 (100 %) 1 (3 %) 1 (3 %) 14 (42 %) 31 (94 %) Total 145 (100%) 6 (4%) 6 (4%) 73 (50%) 133 (92%) a Percentages are given to the nearest whole number. validation study are excluded it was 1.6%. The mean time taken to complete the questionnaires was 18 min (standard deviation 12 min). Discrimination. GHQs were completed for 31 con- trols and 30 patients diagnosed as psychoneurotic. The scores are shown in Fig. 1. This shows that using a cut-off point of 3/4 as originally advocated by Goldberg (3), 9 controls were misclassified as "cases" and 3 "cases" misclassified as " non- cases,": an overall misclassification rate of 19%. Since the majority of misclassifications arose from 30- 20- GHQ score (30 item) 10 - 0- n-31 n=30 0 OS0 0 0 S 0@ 0 0 ~~0@ * 0@ @ - 000 000 *- 0.~~ 000___ - eT1e@ 56I- Normals Clinic Neurotics Fig. 1. GHQ scores in two groups: 'normals' and patients attending psychiatric clinic diagnosed as having psychoneurosis. relative lack of specificity, rather than lack of sensitivity, there was clearly a possibility of using a higher cut-off point and reducing the overall misclassification rate. Assessment offirst stage. The results of the first stage indicated that the GHQ was acceptable to subjects, relatively easy to administer, and discrimin- ated reasonably well between patients and controls. It was therefore decided to proceed to the second stage, and particularly to investigate the need for a change in cut-off point for use of the questionnaire in Jamaica. The second stage A total of 35 patients attending Lawrence Tavern clinic completed GHQs and had a standardized psy- chiatric interview: 3 patients completed the GHQ themselves and 32 with the help of an attendant. There were no refusals (either for completion of the GHQ or for psychiatric interview). Eleven patients were found to have " significant psychiatric disturb- ance " and 24 had " no significant psychiatric disturbance ". GHQ scores for patients in these two groups are shown in Fig. 2. The effect of varying cut-off point on sensitivity and specificity is shown in Table 2. This shows that using 3/4 as cut-off point, sensitivity and specificity are 100% and 54%, respectively, confirming the results of the first stage that indicated relatively low specificity using this cut-off point. If a higher cut-off point is used, however, acceptable specificity as well as sensitivity is achieved. On the basis of these scores the best cut-off point for most purposes would be 5/6 with sensitivity and specificity of 9100 and 84%, respectively. 228 PSYCHIATRIC CASE IDENTIFICATION IN JAMAICA 30- 20 - GHQ score (30 item) 10- 0- nf24 n-ll 0 00 S 0 B .0* * 0 * 00__ -- -fewu -- - - __ __ _ 0000*0 --vs- ---- - 0@ *0 0 OSO0 No psychiatric disturbance Significant psychiatric disturbance Fig. 2. GHQ scores of 35 patients attending Lawrence Tavern clinic and given standardized psychiatric interview. Time and resources for validation study This validation study was carried out over a period of 5 months (including planning, data collection, and analysis). The principal resource was the time of the investigator which totalled 174 hours (including travel to and from Lawrence Tavern), which is equivalent to slightly less than 1 day per week for 5 months. There was also an increase in work for the hospital clerks and the clinic attendants which totalled 56 man-hours (5 people were involved) but this work was performed in normal working time without an increase in staffing. There was no dif- ficulty in gaining the cooperation of clerks or attend- ants. There was also a marginal increase in work for the clinic doctor calculated at 3 hours (including general discussion and briefing). Cooperation and agreement was required from the patients and con- trols, psychiatric colleagues in the outpatient clinic, and those responsible for the rural health clinic. Other resources required were: transport to and from Lawrence Tavern (a total of430 km), secretarial help (10 hours), supplies of GHQ forms and forms for the standardized psychiatric assessment. DISCUSSION Table 2. Effect of different cut-off points on sensitivity and specificity of the General Health Questionnaire in a rural health clinic population Cut-off point Sensitivity Specificity 2/3 100% 33% 3/4 100% 54% 4/5 91 % 71 % 5/6 91 % 84% 6/7 73 % 88% 7/8 64% 92 % Validity, sensitivity, and specificity Using the cut-off point of 5/6 and pooling data from both stages of the study, results are available for a total of 96 persons-41 psychiatrically ill and 55 not psychiatrically ill. The sensitivity of the test was 85% and the specificity 85% with an overall misclassification rate of 15%. On this basis, the val- idity of the GHQ for use in Jamaica appears to be established. The direct implication of the results is that the GHQ can be used as a case identification instrument in the island of Jamaica. This extension of the use of the GHQ to a totally different setting with a high rate of illiteracy led to surprisingly few problems of administration or acceptability. It suggests that the GHQ might well be an effective instrument in a wide variety of English-speaking countries and could be translated for use elsewhere. The main reason for describing this study in some detail, however, is the difficulty in adapting estab- lished epidemiological research methods for use in developing countries. A WHO Expert Committee on Organization of Mental Health Services in Develop- ing Countries (13) has recently recommended that increased evaluative and epidemiological research is a prerequisite for the improvement of mental health services in developing countries. In such countries research tools must be relatively simple and inexpen- sive if they are to be widely used and validation must be feasible without extensive deployment of resources. This study provides information with which a realistic assessment of the time and resources required to carry out such a validation exercise can be made. A two-stage procedure is put forward as 229 230 T. W. HARDING the most practical method, since it allows an initial assessment within the hospital setting before pro- ceeding to the more time-consuming and compli- cated study of health clinic or general populations. One psychiatrist devoting one day per week, with support and cooperation from psychiatric col- leagues, clinic staff, and patients can readily carry out such a study in less than 6 months. In this way, additional information, for example the prevalence of psychiatric disorders in a rural health clinic popu- lation, may also be generated (although such infor- mation has not been included in this paper). The GHQ appears particularly well suited to validation in other settings in view of its brevity, acceptability and simple form of scoring. Recali- bration is a useful option when research tools are being adapted for use in new settings. A test that produces a single score with sufficient steps to allow changes in cut-off point therefore lends itself par- ticularly well to such adaptation. ACKNOWLEDGEMENTS Encouragement and assistance were provided by Dr A. Davies (formerly Director, United Kingdom Medical Research Council, Epidemiology Research Unit, Mona, Kingston) and Professor M. H. Beaubrun (Department of Psychiatry, University of the West Indies, Mona, Kingston). RESUME VALIDATION D'UNE MATHODE D'IDENTIFICATION DES CAS PSYCHIATRIQUES A LA JAMAIQUE Dans les pays en voie de developpement, oiu les res- sources disponibles pour proteger la sante sont tres limitees, des travaux de recherche pertinents et convena- blement men6s sont grandement n&eessaires pour definir le fondement de toute planification sanitaire rationnelle. Dans le domaine de la sante mentale, les outils requis a cet effet font en grande partie defaut puisque la plupart des instruments (pour le depistage des malades, le classe- ment des sympt6mes, le diagnostic et la determination du degre d'incapacit6) ont et6 mis au point et valides dans les pays industrialises. De plus, elaborer de tels instruments en partant de zero est une tache longue et cofiteuse. Aussi a-t-on besoin de toute urgence de me- thodes qui permettent de les adapter et de les revalider. Cet article decrit une methode pour adapter et reva- lider une formule d'identification des cas a la Jamaique en utilisant un questionnaire simple (le Questionnaire general sur 1'etat de sante) mis au point a l'origine au Royaume-Uni puis largement employe en Amerique du Nord. Dans le cadre d'une procedure en deux etapes, cet instrument a d'abord et mis a l'epreuve dans des services hospitaliers de consultations externes ou l'on a deter- mine qu'il permettait, d'une maniere acceptable et satis- faisante, de faire la distinction entre les malades atteints de troubles psychiatriques dejA diagnostiques et les sujets en bonne sante. Ensuite, des malades adultes se presen- tant dans un centre de sante rural et pris dans l'ordre des arrivees ont complet6 le questionnaire et ils ont e, eux aussi, passes au crible au moyen d'une entrevue psychiatrique type. L'etude a abouti aux conclusions suivantes: a) le Questionnaire general sur l'etat de sante (version a 30 roubriques) peut etre utilise comme instrument de detection des cas a la Jamalque, sous reserve d'un nouvel etalonnage, sa sensibilite etant superieure a 90% et sa specificite superieure a 80%; b) il est possible d'utiliser chez des populations anal- phabetes (en faisant appel a des assistants ayant requ une formation simple) des questionnaires qui ont et mis au point en vue d'etre completes par les sujets eux- memes dans des populations alphabetes; c) une procedure en deux etapes constitue un moyen economique d'adapter et de revalider des instruments de detection des cas en vue de leur emploi dans de nouvelles populations. REFERENCES 1. BRODMAN, K. ET AL. The Cornell index health questionnaire manual. Ithaca, Cornell University Press, 1956. 2. ESSEN-MOLLER, E. Individual traits and morbidity in a Swedish rural population. Acta psychiatrica scan- dinavica, supplement 100 (1956). PSYCHIATRIC CASE IDENTIFICATION IN JAMAICA 231 3. GOLDBERG, D. P. The detection of psychiatric illness by questionnaire. London, Oxford University Press, 1972. 4. GOLDBERG, D. P. ET AL. A standardized psychiatric interview for use in community surveys. British journal of preventive and social medicine, 24: 18-23 (1970). 5. KENDELL, R. E. ET AL. The reliability of the ' present state examination '. Socialpsychiatry, 3: 123-8 (1968). 6. LEIGHTON, D. C. ET AL. The character of danger. New York, Basic Books, 1963. 7. LEIGHTON, A. H. ET AL. Psychiatric disorders among the Yoruba. Ithaca, Cornell University Press, 1963. 8. LEON, C. A. & CLIMENT, C. E. Assessment of instru- ments for studying the prevalence of mental disorder. Social psychiatry, 5: 212-215 (1970). 9. MACMILLAN, A. M. The health opinion survey: tech- nique for estimating prevalence of psychoneurotic and related types of disorder in communities. Psychological reports, 3: 325-329 (1957). 10. SHEPHERD, M. ET AL. Psychiatric illness in general practice. London, Oxford University Press, 1966. 11. SROLE, L. ET AL. Mental health in the metropolis- the midtown Manhattan study, Vol. 1, New York, McGraw-Hill, 1962. 12. The international pilot study of schizophrenia. Vol. 1, Geneva, World Health Organization, 1974 (WHO Offset Publication No. 2). 13. WHO Technical Report Series No. 564, 1975 (Report of a WHO Expert Committee on Organization of Mental Health Services in Developing Countries).