Water, sanitation, and hygiene evaluation issues.

Letters Water, sanitation, and hygiene evaluation issues Editor – The article on Programme Saniya in Burkina Faso (1) raises im- portant generic issues for the evaluation of water, sanitation and hygiene educa- tion initiatives. According to the UK Department for International Develop- ment, ‘‘on scores of occasions water and sanitation projects have commis- sioned epidemiological or demographic evaluations of health benefits ... such studies are time-consuming, expensive, fraught with methodological defects, and frequently produce misleading or ambiguous results’’ (2). Funding agen- cies are unlikely to spend their money on such evaluation unless a health impact is explicitly stated as a project objective. Some difficulties arise from how the evidence is defined, such as setting unrealistic targets for demon- strating change, and not adequately differentiating between process, impact and outcome indicators. However, regarding the health promotion materi- als themselves, there is no debate that their design must be based on local practices and culture; this normally requires initial qualitative research and subsequent use of surveys to assess behaviour change, such as reported from Burkina Faso. We believe that the apparent lack of effectiveness of many such projects, as noted by the authors in their literature review, has less to do with the appropriateness of the educational materials and more to do with evaluation design and measure- ment issues. Even in the case of the Burkina Faso experience, it is curious that while indicators show relatively static levels (perhaps even a fall-off) for programme coverage, direct obser- vation of hygiene behaviour of mothers and children reveals a positive increase between baseline (1995) and post-inter- vention (1998) surveys. The Aga Khan University (AKU) is currently involved in the outcome evaluation of a 5-year (1997–2001) water, sanitation, and health and hygiene education programme targeting 100 villages (>100 000 population) in the Northern Areas and Chitral, Pakistan. The project is being imple- mented by the Water and Sanitation Extension Programme (WASEP) of the Aga Khan Building and Planning Services, with external donor funding. Villages were selected into the project in phases, based on pre-set criteria as defined by WASEP, incorporating a participatory approach to enhance sustainability after the project. Com- pared with the Burkina Faso setting, the project area differs greatly in the pre-intervention level of water and sanitation facilities (e.g. <10% of pre-intervention village households had a latrine). In northern Pakistan, villages are served by gravity-flowwater supplies using an intricate system of irrigation channels fed by melting snow and ice; in this traditional system, there is intermingling of water for animal and human use. The terrain is mountainous, with villages routinely becoming isolated. WASEP delivers an integrated package of interventions for three components — water, sanitation, and hygiene education; thus, it addresses not only hygiene behaviour through education, but also provides engineering solutions to improve water quantity and quality. Interventions are targeted at three levels: the village (e.g. improved water supply), the household (e.g. pro- vision of a latrine), and primary schools (health and hygiene education sessions). Interventions are conducted in phases among selected villages, and in stages within villages. As in Burkina Faso, the WASEP project experienced a shortage of donor funds in the fourth and fifth years, slowing implementation of some activities, particularly health and hygiene education; expectations did not diminish regarding the projected health impact. Throughout implementation, WASEP itself has monitored its engi- neering and educational interventions, using several methods: periodic house- hold surveys of knowledge, attitude and practice; direct observation of household sanitation; fortnightly diarrhoeal disease surveillance of households; and a water quality mon- itoring system. Even so, WASEP has opted for an independent evaluation by AKU, with the assistance of the Aga Khan Health Services, Pakistan. The main focus at this time is to assess the health outcome of the project, using a case-control study of diarrhoea as recommended by the World Health Organization (3). We also plan an integrated evaluation exercise, to encompass both engineering and hygiene aspects of the project. As in the case of Programme Saniya in Burkina Faso, this will include data collected at baseline and throughout implementation. n Debra Nanan1 & Franklin White2 Conflicts of interest: none declared. 1. Curtis V, Kanki B, Cousens S, Diallo I, Kpozehouen A, Sangare´ M, et al. Evidence of behaviour change following a hygiene promotion programme in Burkina Faso. Bulletin of the World Health Organization 2001;79:518-27. 2. Department for International Development (DFID). Guidance manual on water supply and sanitation programmes. London: WELL and London School of Hygiene and Tropical Medicine for DFID; 1998. 3. Cousens SN, Mertens TE, Kirkwood BR, Smith PG, Feachem RGA. Case-control studies of common childhood diseases: the example of diarrhoea. London: Macmillan Education Ltd for the World Health Organization; 1995. Impact of the Bosnian conflict on the health of women and children Editor — In 1990, Bosnia and Herze- govina enjoyed the economy, health status and health care of a middle- income country. The war from 1992 to 1995 left the country’s resources devastated and, from a population of 4.5 million, an estimated 250 000 people (including 16 000 children) are believed 1 Lecturer, Department of Community Health Sciences, The Aga Khan University, Karachi, Pakistan (email: debra.nanan@aku.edu). Correspondence should be addressed to this author. 2 Professor & Chair, Department of Community Health Sciences, The Aga Khan University, Karachi, Pakistan. 75Bulletin of the World Health Organization 2002, 80 (1) # World Health Organization 2002 to have died in the conflict or are missing (1). Health services, especially those supporting women and children, were severely disrupted, with over 35% of facilities destroyed or heavily dam- aged. To assess the impact of the conflict on the health status of women and children, we compared the following traditional indicators with information from the previous decade. Data onwomen of reproductive age (15–49 years) and children (0–15 years) in Bosnia and Herzegovina were col- lected from two main sources: routine official reporting systems and a nation- ally representative survey of households, women and children carried out in mid-2000 (2). Numerator data from the official reporting systems were extracted for the year 1991 and the most recent year for which complete infor- mation was available. Denominator data were available from the 1991 census; best available state estimates were used for the most recent year. The national survey covered 10 772 households, with a response rate of >98%; it was carried out using standardized United Nations Children’s Fund (UNICEF) methodology, described elsewhere (3). The results suggest that the health of women and children in Bosnia and Herzegovina has not worsened in the last decade— in fact several indicators, including infant mortality rate and maternal mortality ratio, show improvement. These find- ings are extremely surprising, given four years of war and its consequences. Although the second half of the decade has seen rebuilding of essential services, the economic situation and health and other services remain far below pre-war standards (1). Underlying factors suggest that the data quality may be poor. Official data collection systems are under consider- able pressure and there is little systema- tic effort to assess or improve data quality. The unclarity is compounded by significant concerns about the denominator population data used to calculate rates, with uncertainty surrounding true figures for deaths and refugee and migrant populations. Despite being widely used, we also question the appropriateness of relying solely on traditional indicators such as infant and maternal mortality rates to assess the impact of the war — selective primary care can improve these indica- tors even when the general health status of the population deteriorates (4). The limitations of relying on such indicators, tested under ‘‘developing country paradigms’’, in responding to complex emergencies in more-developed countries has already been high- lighted (5). If our results indeed reflect a true stability in the health of this population, possible explanations include a good pre-war health status, resilience of the socialist primary health care, education and other basic systems that continued to function, and significant levels of external aid for postwar reconstruction (US$ 5.1 billion during the period 1995–99) (1). These results are important because they suggest the possibility that good primary health care systems and adequate, targeted external assistance can protect the health of vulnerable populations such as women and children against the adverse effects of war. However, credence cannot be given to these claims as it may take many years for the impact of a conflict to be visible in traditional indicators, or it may be masked by an overall deterioration in the data collection systems. More research is needed to produce indicators that can adequately evaluate population health status, the resilience of local communities, and the protective effects of humanitarian assistance in conflict situations. This is especially relevant as these indicators routinely form the basis for international assistance. n Sanjay Kinra,1 Mary E. Black,2 Sanja Mandic,2 & Nora Selimovic3 Conflicts of interest: none declared. 1. Independent Bureau for Humanitarian Issues. Human development report: Bosnia and Herzegovina 1998. Sarajevo: United Nations Development Programme; 1999. 2. United Nations Children’s Fund Bosnia and Herzegovina. Bosnia and Herzegovina multiple indicator cluster survey 2000. Sarajevo: United Nations Children’s Fund; 2001 (in press). 3. Division of Evaluation, Policy and Planning. End-decade multiple indicator survey manual. New York: United Nations Children’s Fund; 2000. 4. Ugalde A, Selva-Sutter E, Castillo C, Paz C, Canas S. Conflict and health. The health costs of war: can they be measured? Lessons from El Salvador. BMJ 2000;321:169-72. 5. Spiegel PB, Salama P. Emergencies in developed countries: are aid organisations ready to adapt? The Lancet 2001;357:714. The role of RT-PCR assay of oral fluid for diagnosis and surveillance of measles, mumps and rubella Editor — The potential of oral fluid to replace serum for specific antibody detection for use in evaluating popula- tion immunity levels for important vaccine-preventable virus infections was recently discussed in the Bulletin (1). The important role of genotyping virus strains in surveillance of vaccine-pre- ventable diseases to identify ‘‘escape mutants’’ and to investigate epidemio- logically linked cases is well established. We report here an investigation of the rate of the virus shedding in oral fluid in patients with serological evidence of recent measles, mumps and rubella (MMR). While other clinical specimens such as blood, nasopharyngeal aspirate (NPA), throat swab and urine are recommended for testing for this purpose, oral fluid offers considerable compliance advantages to the patient and is easy and economical to collect. In all, 1047 oral fluid samples were tested by reverse transcription poly- merase chain reaction (RT-PCR) assay for MMR: 966 samples were collected as part of the salivary surveillance scheme in the United Kingdom (2) and had been previously tested for antibody (3), and 81 samples were obtained specifically for molecular studies from confirmed cases. All samples were collected using the Oracol device (Malvern Medical Developments, Worcester, UK) (4). The results (Table 1) support MMR virus genome detection by RT-PCR in oral fluid samples collected during the first 14 days after onset of symptoms. 1 Department of Social Medicine, University of Bristol, Canynge Hall, Whiteladies Road, Bristol BS8 2PR, England (email: Sanjay.Kinra@bristol.ac.uk). Correspondence should be addressed to this author. 2 UNICEF, Sarajevo, Bosnia and Herzegovina. 3 Agency for Statistics, Sarajevo, Bosnia and Herzegovina. Letters 76 Bulletin of the World Health Organization 2002, 80 (1) In contrast, MMR-specific IgM (3) may not be detected for several days after the onset of symptoms. The most important factor influencing the suc- cessful detection of viral genome by RT-PCR was the timing of sample collection, and detection rate decreased with time after onset of symptoms. A higher rate of RT-PCR positivity was also found in those samples which had not previously been used for detection of specific antibody, suggesting that oral fluids contain low levels of virus and that subjecting samples to cycles of freeze–thawing may damage genome copies. A further observation was that the proportion of oral fluid samples in which measles virus genome was detected was significantly higher than that for mumps and rubella (P<0.001). This suggests a longer lasting viraemia for measles, or that the immune system may take longer to clear measles than mumps or rubella viruses. Oral fluid samples from isolated cases that are to be used for molecular studies should ideally be collected within 7 days of onset of illness and stored at –70 oC until required for testing. During large outbreaks or epidemics where large numbers of samples may be collected this is not so critical, and samples obtained and used for other purposes (e.g. antibody detection) may also be used for molecular studies. However, the likelihood of detection of virus genome is much reduced. We have shown that oral fluid samples are a safe and practical alter- native to blood, NPA, throat swabs and urine for molecular studies involving MMR. Sequencing of the PCR amplicon also enables the characterization of viral pathogens without cell culture, which is less sensitive and can be difficult and time-consuming. These results, together with those on antibody detection, reinforce the value of oral fluid samples for surveillance of MMR and highlight the need for more detailed studies to exploit fully the use of this method in surveillance programmes. n L. Jin,1 A. Vyse,1 & D.W.G. Brown1 Conflicts of interest: none declared. 1. Nokes DJ, Endquselassie F, Nigaru W, Vyse AJ, Cohen BJ, Brown DWG, et al. Has oral fluid the potential to replace serum for the evaluation of population immunity levels? A study of measles, rubella, and hepatitis B in rural Ethiopia. Bulletin of the World Health Organization 2001;79:588-95. 2. Gay N, Ramsay M, Cohen B, Hesketh L, Morgan- Capner P, Brown D, et al. The epidemiology of measles in England and Wales since the 1994 vaccination campaign. Communicable Disease Report. CDR Review 1997;7:R17-21. 3. Perry KR, Brown DW, Parry JV, Panday S, Pipkin C, Richards A. Detection of measles, mumps and rubella antibodies in saliva using antibody capture radioimmunoassay. Journal of Medical Virology 1993;40:235-40. 4. Brown DWG, Ramsay ME, Richards AF, Miller E. Salivary diagnosis of measles: a study of notified cases in the United Kingdom, 1991-3. BMJ 1994;308:1015-7. 1 Enteric, Respiratory and Neurological Virus Laboratory, Central Public Health Laboratory, 61 Colindale Avenue, Colindale, London NW9 5HT, England. Correspondence should be addressed to Dr Jin at this address (email: ljin@phls.org.uk). Table 1. Measles, mumps and rubella RT-PCR results on 1047 oral fluid samples from confirmed and clinically diagnosed cases by time from onset of clinical symptoms Virus Collection RT-PCR result (%) with days after onset Total RT-PCR purposea positive (%) 0–7 days 8–14 days >14 days Not known Measles surveillance 28/52 (54) 27/67 (40) 36/174 (21) 19/100 (19) 110/393 (28) RT-PCR 52/54 (96) 1/1 (100) – – 53/55 (96) Mumps surveillance 46/67 (69) 11/55 (20) 3/183 (1.6) 4/49 (8) 64/354 (18) Rubella surveillance 10/24 (42) 1/24 (4) 3/108 (2.8) 2/63 (3.2) 16/219 (7) RT-PCR 12/16 (75) 3/10 (30) – – 15/26 (58) Total 1047 a Confirmed cases were collected for routine surveillance for antibody testing, and clinically diagnosed cases specifically for RT-PCR. Contributions are welcome for the Letters section, in response to articles that have appeared in the Bulletin or on matters of major public health importance. Letters are usually between 400 and 850 words, with a maximum of six references; they will be edited and may be shortened. We prefer to receive all responses electronically, either sent to the editorial office (by email: bulletin@who.int or on diskette) or submitted through our web site. Authors should give their current appointments and full addresses, with a telephone or fax number or email address for the corresponding author. We ask authors to declare any conflict of interest. Letters 77Bulletin of the World Health Organization 2002, 80 (1)